Well, I find this post looks good, but a like lot of 'data for developers' posts it's just a list of tools. As if a collection of tools banded together actually makes your customer successful.
What's missing?
1. There's nothing about deployment. How do I take this collection of tools and code and actually deploy it into production, or actually regression test it functionally? How do I make a small change in a database table and not have a massive regression? How do you do that automatically? How do you do it quickly?
2. It's cursory on testing. One of the biggest differences from a software developer to a data engineer is that your data providers give you crap data all the time. It could break. How do you test data? How do you get adequate test coverage? These things are essential for software developers and are actually doubly essential for data engineers and building analytics systems.
3. It's what success looks like. It's not just about a collection of tech; it's about making your customers successful. What does it mean to deliver good insight? How do you do it? How do you measure customer success, and measure your success? As a team, you wouldn't talk about software engineering without mentioning DevOps or DORA metrics. There's nothing here about that.
The SAT, like general intelligence, is half hard work and half inherited ability, from what I've read. Characteristics that you have no control over, like height, intelligence, race, or sex, should not determine your wealth. It should really be about hard work and character; that's fair. At least in the U.S., there are too many industries like law, venture capital, or VC-invested startup founders, where the pedigree of your school is what matters to your success.
Neither statement is true, unfortunately. The reports that you read in the media don't reflect the state of the world. In the US, most plumbers and SWE make between 100-200K. The way to wealth is to start one's own business. Hence, hard work and character.
The dude has obviously never met a multimillionaire owner of an auto body shop, a house demolition business, or a plumber. Did they do well in high school calc class? No. Are they richer than the author? Yes
Blowing up the cognitive hierarchy is a gift that AI gives us. Let's move into an age where hard work and character matter more than your SAT score at 17.
Many data teams often find themselves as 'tool jockeys' instead of becoming true engineers. They primarily learn some company data, and then rely on drag-and-drop or YML configuration functionality within the constraints of the tool's environment.
Their organization often insists they must use standard tools, and their idea of a good job is that the task works fine within their personal version. No automatic testing, no automated deployment, no version control, and handcrafted environments. And then they get yelled at when things break and yelled at for taking too long. And most DEs want to quit the field after a few years.
The real question is not that DE and software engineering are converging. It's why most DEs don't have the self-respect and confidence to engineer systems so that their lives don't suck.
Why is Japanese software so bad? Chip shortage or not using the card purchase UI at train stations is confusing, even in English.
As a US software engineer, I can not figure out how a country that can make great hardware (e.g. high-speed trains), has a beautifully minimalist design aesthetic (e.g. wabi-sabi), and lots of talented artists and engineers can produce such crappy user interfaces. Or for that matter, beyond a few games is irrelevant in the world software markets. Does anyone have an idea?
Optimizing for unit economics vs. top-line growth are opposites in building a company. In the years of free investor money (2020, 1999, etc), companies choose growth over unit cost because their boards and investors are trying to drive huge valuations. This is true in software, farming, and many industries.
sigh