While I generally agree, I think there's a point in these 2 statements that can easily misinterpreted.
>It is likely that the Data Scientist role is in a long term decline...
Also
> Data science is in decline and vaguely defined
Reading this, you can think that "Data Science" jobs are decreasing. But I don't think that's true.
Let's just say that it's 2017 and I hire a team of 3 people with the job title of Data Scientist. One ends up focusing on the data side, one on modeling+analysis, and one on building the infrastructure. In 2023, I decide to change the job titles so one of them is now a Data engineer, one is now a Data Scientist, and one is now a ML Engineer to match what is happening in the job market.
It's still 3 jobs with 3 people doing the same thing. So the number of jobs aren't decreasing, but their titles are more specific. Overall, the number of "Data Science" jobs are still doing up.
Somebody will say "But that's exactly what the author said." But I think people who are new(ish) to this field might read it as "Data Science Jobs are decreasing." So I'm making this comment.
> skills such as data mining and visualisation are also out of favour.
Honestly, I just don't believe this. It's possible that as job descriptions are filled with different buzzwords, people just leave these out. For visualization it's also possible that there is a bigger focus on keywords of an established BI tool (e.g. PowerBI) instead of ad-hoc charts in matplotlib or ggplot. But some degree of data mining and visualization is useful, even to Data Engineers.
>With 10+ years in DS, I've always felt that best DS were always basically software engineers that knew math and were more interested in prototyping cool machine learning product than maintaining production infrastructure. Unfortunately this always accounted for a small fraction of DS I interacted with.
I've been a DS for 10+ years, and I feel the exact opposite. The worst "Data Scientists" I've worked with are all ex Software Engineers who seem to assume that business problems are really computation problems. So they find convenient ways to ignore the human aspects (e.g. trying to figure out why the data is a mess) and gravitate to using more complex algorithms and breaking down the problem to an achievable programming pipeline that runs in production, but the results are of low value. But it looks awesome on a resume.
Are you right or am I right about SWEs turned DS? I have no idea. But one quality that IMHO is important is the interest in actually looking at data and asking questions, which is much rarer than most people realize.
There might be a lesson from history on where an "AI" startup needs to focus if they want to succeed.
Starting around 2012, there was a huge hype around ML. Lots of startups on selling "ML." If you look today, the majority of the startups that just sold ML algorithms or ML in a box are pretty much gone, or were acquired for amounts where only the founders made enough money to pay off their car loan on a Toyota Camry. At least one of the most highly regarded Unicorns in 2014 now is in the "Wait, that company still exists?" category.
The companies that survived or got acquired for a decent amount were the companies that used ML to actually solve a tangible problem. They were not selling ML, but a solution to some problem that happened to use ML - even if their marketing focused on the ML part.
I wouldn't be surprised to see the same with AI companies.
A big company (e.g. Google) can quickly release a similar product for an LLM because that's an area where the big company was already doing work and a lot of the knowledge to build it is published or known. But if a startup is targeting a focused problem in a specific industry, a big company can't just wake up one day and get the information needed to solve that problem in a short amount of time.
One of the ways I think about this type of problem is by asking "You want to use computation to extract a signal from this data. What's that signal worth to you in business ROI dollars?"
If Domain Expertise + Feature Engineering + ML model can get you 90% of the way there and it runs on a tiny cloud instance that takes 30 minutes to train, is a DL based approach that pushes you to 91% worth it from an ROI instance if takes a 4xGPU cluster 2 days to train it, not to mention inference costs? Especially if you need to explain what the model is doing?"
This above is exactly the situation I'm in now with my job. I'm on the "Get useful stuff to production so we can save money" side of things, and we have R&D teams who try to approach the same problems using DL and all the latest methods. At least for the use cases our team focuses on, they haven't been able to do more than set $$$ on fire via GPUs. For us, Domain Knowledge + Good Data Engineering is the secret.
I think ML is going to be around for a long time because it works, even though DL is dominating the news right now. Just because a neurologist can also diagnose and treat common medical conditions (e.g a pneumonia), that doesn't mean we need every doctor to be neurologist.
I think one thing to keep in mind is that there are specific use cases where the cost of using DL isn't worth the improvement in accuracy (if there is one) from a business ROI perspective.
I know somebody who works in the insurance industry on a text classification use case. The business impact of this use case is important as it's used as part of the claims process. The team he's on has tried a lot of different things, but feature engineering + domain expertise + a particular tree ML model has provided the best performance for the lowest overall cost. They are very open to trying new things, but a DL approach simply hasn't been worth it.
>But (especially in Enterprise B2B) they're not coders, and they're not product managers, and they are not quite salesperson "enough" to qualify for evolving to any of those career positions.
I was in PreSales for a long time and IME, this isn't true (excluding SWEs). Different SEs have different personalities and different inclinations. Some of these personality traits means they have to stop being an SE and do something else to maximize their career satisfaction.
A lot of SEs stay as SEs and simply grow as technical experts or industry experts and then go into a "Principal" SE role. I think these tend to be the type of people who like go deep into something and their strengths are in going deep. The ones who get tired of the sales part can go into developer relations, solution development, etc where they are part of building solutions, but they are removed from the heavy sales parts of it. You can certainly get paid enough as a Principal level that money is removed as a reason for changing career paths.
There's then the type of SE who is strong technically and is good at understanding what a customer is really asking for and translating that into a solution to a specific use case. These are the ones who can be good product managers. I put myself in this category and the product SWEs I worked with usually liked talking to me because I could frame a customer ask in a concrete technical way, which was something a lot of Product Managers struggled with. I had more than one instance where a Product Manager said if I wanted to switch to Technical Product Management they could find a path to make that happen.
There's then the type of SE who is okay technically, but who really cares about customers succeeding in the long term. They can do well in customer success roles, or even technical support and management roles in either of those areas.
There's then type of SE who is really enjoys the "educating" part of sales. They can go into the Training/Education side of things.
Finally, there's the type of SE who has a great personality, but they are poor technically and not very detail oriented when it comes to anything technical. I've seen these become sales people or move into account management.
I've worked with SEs who have moved into Sales roles, Product Management, Training, Customer Success, and PreSales Management, Developer Relations, etc.
Regarding SWE, I do agree with you that transitioning from SE to SWE is harder. I've worked with SWEs who became SEs but couldn't transition back. Once you've climbed out of the trenches and seen the big picture it's hard to say I'm going to go back in the trenches, especially at a large B2B company where there are 1000s of SWEs. It's easier if you were a specialist (e.g. in Data Science) and the SWE job is a specialist role.
>How do you evolve SEs in your org?
One thing I've seen is that many SE managers are sales people at heart, which makes them terrible at evolving SEs. They only reward and recognize the SEs who have the traits that overlap with sales people. So they can't even recognize how to evolve their SEs. Even if the SE manager gets it, the Directors/VPs, etc are all ex-sales people and they don't get the value of focusing on these other things.
IMO the way to evolve SEs is to connect them with somebody from a different org (that can use the SEs strengths) and have them work together. The SE will learn about how people do a different job, and that other person gets a direct line into what SES are seeing in the field. It's generally a net positive for the company.
First time I'm hearing about this one, thanks for the recommendation. Unlike Calculus or even a typical one semester Statistics course, probability is one of those topics where you need to see a lot of problems to really grok anything. The only way is to see a lot of solved problems and think about why that's the right answer.
Even highly recommended books (e.g. by Blitzstein) don't have enough solved problems, so it's nice to there's a problem focused book out there.
In the last 5 years I've moved from Data Science Manager to Principal (IC role, but basically the external facing technical lead of the team) and now Senior. When I add up all the positives and negatives at work and at home, I think I'm the most content I've been in a long time.
One piece of advice I can give is to make a list of concrete things that are making you think about becoming an IC. And by concrete I don't mean just a word like "Stress." List out the things that are causing that (e.g. Unpredictable rushed deadlines). If it's useful, create a mental Venn diagram of what overlaps and pull out some larger themes. From the list, pick the top 2 - 3 and ask yourself if you are really going to get that as an IC.
Example: For me, one thing that was causing me an immense amount of stress was a highly unpredictable schedule - last minute meetings, fires, people with no ability to prioritize, etc. At the top of my list was being able control my schedule to a reasonable degree. IME that's really hard to do as a Manager or even a Lead/Principal, so it made sense to me to move down to Senior and find a job that allowed for that.
That being said, there are plenty of IC jobs where you sit in pointless meetings all day and get called into meetings at the last minute. It's not like moving down is a guarantee of anything, so it's important to really identify if moving down is going to get you what want. HN has plenty of anecdotes of crappy IC jobs.
As far as pay, I took a small haircut from Manager -> Principal and then a slightly bigger haircut from Principal -> Senior. My spouse works so we still have more than enough to pay bills, meet retirement goals, and have something left over for other stuff. I'm not paid even close to what FAANG people are making, but I suspect lots of people I work with on the business side of things would love to be in my salary band.
>But it also (1) helps keep bad content off of the platform, so users aren't exposed to it, (2) lowers the number of human reviewers who come into contact with it, which is improves their jobs, and (3) frees up budget for whatever improvements need to be made to this whole workflow.
I think reading that this type of solution was created, and person who worked on it was laid off, makes me very sad as a Data Scientist.
I enjoy working as a Data Scientist, but I struggle a lot with the field. Lots of jobs are mostly about grabbing eyeballs or selling something. Some jobs are just total bullshit. Even the ones where you're doing something concrete (e.g. keeping a machine running), some days you still wonder if it really matters in the long run.
But with some of these social media safety topics, it can feel like a job has some meaning beyond just shuffling numbers around on an spreadsheet.
So it's disappointing to hear that people with the skills to create something like that are fired.
>But I also think there's a lot more opportunity in using statistics/ml/data science in the top line than most companies practice.
I consider myself a fairly honest Data Scientist, in the sense that I like it when I can map what I'm doing to the value it delivers. I know some other great people I've worked with who are like this as well.
This is anecdotal, but all of us have hated working with many top line people because there's some really fuzzy mapping from goal to value (since value is realized in the long term), and some of the people are champion bullshitters. I don't need to explain sales people. But marketing, corporate strategy, and even upper product management - they drove us crazy because their standard of being data driven was absolutely not consistent with how we thought about things at all. All of it was because the mapping from project to revenue was over years, not quarters. And it was all projections.
Compare this to bottom line people, where the mapping from project to cost savings is on a shorter time frame. The types of personalities this attracts is different.
Maybe the growth hacking stuff at software companies is different and you can focus on revenue growth and still connect what you are doing to that. I've never worked in that role so I don't know.
>I genuinely do not understand the logic of cutting this team to save costs.
I've been in a situation where a company was under pressure, was trying to make a big pivot, and there where multiple rounds of layoffs.
At one point I could only make sense of it by picturing a somewhat blind lumberjack getting an order that says "There's a forest that needs 15% of trees cut. Go cut." Good trees get get, bad trees get cut. Thankfully we are not trees and if we get cut we can move on. We don't die just because we got chopped down.
It's possible to have teams that save millions of dollars a year and not be worth it to keep them.
For the sake of argument, let's say a statistics team has 5 people.
Cost of Employee at FB, including insurance, office space, 401K match, salary, bonuses = 250K/year (probably very conservative).
Cost of Data and Software Infrastructure to support them (including people to respond to Infrastructure support tickets), let's just be very conservative = 100K/year.
Cost of People Management overhead to support them. Includes salary of at least one manager, not to mention the time of a program manager, project manager, product manager, or whomever else. Let's just say 500K/year.
Total = 1.85 Million/Year.
Let's say this team of 5 people comes up models that save the company $4M a year. I once had a VP tell me that to justify a Data Scientist on the team, they needed to have a savings of 10X what they cost the company to have that person on staff. I know this logic and math seems very weak and hazy. Mapping costs is a strange thing. But this is how some decision makers think, and this is how people get cut.
I work in the field of Data Science and one upsetting reality has started to sink into my mind over the last year.
In a business there is top line and bottom line. There are a lot Statistics/ML/Data Science jobs that are about moving that bottom line. You build something to optimize something to reduce costs.
The value provided by the bottom line people is less visible than the value of top line people. The easiest way to move the move the bottom line is by just getting rid of people. So when the axe falls the bottom line people get cut and it's hard to understand why.
It's the same thing as people say about fires. When you put out a fire you are a hero. When you prevent the fire in the first place, everybody thinks it's business as usual and nobody understands why you are needed.
IMO the reason behind this is that a lot of "data science" driven decisions are short term decisions. So you can look at something on a PowerPoint, not really care if it's wrong unless you personally will get fired if it turns out to be wrong, and back out of it a quarter later when it turns out to be wrong. IME there's no shortage of justifications or pivoting when it comes to a decision you made a quarter ago. The consequences are relatively small, so the caring is only bravado, not really caring.
When it comes to disastrous long term decisions, there's plenty of time to get input from multiple stakeholders. I always remember the armies of companies who went chasing after Hadoop because Big Data was going to transform something or the other. All the stakeholders were on board, from the CEO and CTO to IT and Engineering management. How much money and time got flushed down the toilet trying to implement and extract value from data with Hadoop. They only people who paid the consequences were the employees at Hadoop companies who thought their stock options would be worth something.
Data Engineers are the people who take raw data (e.g. what lands in S3) and put that into data systems that can be used by other systems (e.g. Dashboards) and people (e.g. Analyst, Data Scientists, BI people). Data Engineers clean data, but they are really looking at cleaning out systemic issues (e.g. some data that is missing in one field is in another field, and that needs to be consolidated) and not the scrutinized row-by-row cleaning that Data Scientists end up doing. Data Engineers also do the data steps (e.g. creating a performant stored query) required to support things like business KPIs and reporting.
ML Engineering has a lot more variety based on the company and org, but generally it's about building an automated pipeline that includes ML. In smaller orgs you do everything - build a data pipeline, train a model, deploy that model, score new data, etc. In larger orgs, ML Engineers take a model built by somebody else and make it run at scale while meeting certain SLAs (e.g. making recommendations on a social media website).
>One extremely frustrating aspect of plant meat is that they tried to aggressively push out traditional veggie burgers on restaurant menus.
The homogenization of veggie burgers into one particular type of patty is the reason I don't even try Burger places anymore. It used to be that if you had 5 Burger places near you, you'd get 5 very different veggie burgers (e.g. black bean, quinoa, mixed veggies, all with a different mix of spices). Now you basically get the same veggie burger at every place, just with different dressings/sauces.
I suspect it's better in big cities (e.g. NYC) where there's enough of a customer base for a veggie patty and a engineered meat patty on the menu. But in other places, I think the customers numbers are just too small. So I understand the logic from a business perspective, but for some customers it just sucks to have lost the veggie patty.
>I'm happily doing mostly data engineering + stakeholder management instead of hyperparameter tuning.
Agreed. I actually like that in DS you can have a job where you are involved in the end-to-end of a business problem and that you need to have a mix of skills (e.g. Act like a Product Manager and an Engineer) to succeed. And it's not just "Today I get to crank out yet another tile on the Kanban board."
>Where will that leave people trying for either ML engineer or more data analysis focused DS roles? Or even just everyday utility scripting/automating powers? Can even the latter two can be largely replaced by hybrid finetuned multi modal language/code models. I feel a bit lost and like I have wasted my time.
IME most people have a bit of culture shock when they first get an ML/DS role. When you learn ML/DS, there's a huge focus on the coding and mathematical parts of the job, and not on much else. When you get a job, suddenly you're exposed to everything else and you realize why every data scientist says that ML is only a tiny part of their job.
There's a lot of variety in DS jobs, but one of the things I try to explain (when somebody asks me) is that the real skill of every DS job I've had is "Taking a vaguely defined business problem and working with stakeholders to come up with a solution that happens to use code, math, and charts as the path to that solution." In reality, to succeed in as a Data Scientist, you have act like a Product Manager+Product Owner+Data Analyst+Software Engineer+Data Engineer+Data SME.
This is why so many attempts at "We can make your Business People Data Scientists" products haven't taken over. You might be able to take over some of the boring parts (e.g. AutoML, doing the parts that Data Scientists hated doing anyway), but I've never seen a piece of software that could tell the business users that they don't have the right data to measure what they are trying to measure.
I've even seen commercial AutoML solutions lead to business people realizing they need to hire Data Scientists. This is because once you use AutoML, you realize you need somebody who actually understands the data and the process to really trust the results.
If "hybrid finetuned multi modal language/code models" can replace what competent people in ML/DS can do, that technology is going to replace a lot more professions than just ML/DS. There's going to be a job apocalypse for lots of professions, including SWEs.
I think from a career standpoint, ML/DS is a bit of a mess because it's a new field and businesses are still trying to figure out the best way to get value out of it. So there are a lot of pain points for people working in the field. Compare that with Software Engineer, which is older and a bit more mature. But I still think ML/DS is a field worth getting into if you can sort through the noise of which jobs are good and which are crap.
IME the core issue here is that there are far more SWE jobs than ML/DS jobs, which leads to a far greater variety of SWE jobs, which means people are better able to find SWEs jobs where they are able to find a job that matches their job environment. If you want to coast, there are SWE jobs for that. If you want to work 12 hour days on the bleeding edge, there are jobs for that as well.
>A lot of the time you could train model prepare pipeline and test enviroment for 3-6 months before you get good enough result to push it into production. And it can get extremely stressful rly fast if you care about that
Some of the the major job challenges in ML/DS are that the field is new, the number of jobs are far fewer than SWEs, the teams are small, and the people who can exert control of you have wildly unrealistic expectations of what it takes to build a useful model. So it's easier to land in a crappy job where stakeholders add to your stress because you are not "delivering" according to their definition of "I thought ML would solve all my business problems in 2 weeks."
I personally prefer ML/DS to what I see a lot of SWEs do. Yes, it can take months to get something working (or maybe it will fail after all that effort), but I'm also involved in the end to end of what I'm building and I get be part of defining the the "what", "why", and "how" and not just crank out yet another story on a Kanban board. What really helps me in my job is having a management chain that pushes back and educates stakeholders on why things take time and why that time is required to have a certain level of quality.
There are plenty of SWEs (on HN and elsewhere) who say they feel like factory workers. Somebody else makes a bunch of decisions and the SWEs simply churn out features and stories. So the quick wins turn into an endless flood of drudgery, just like how when you learn to fix a toilet you feel great, but if you fix 20 toilets a day, it probably becomes monotonous pretty quickly.
>The last thing I want to do with my time is read data science and ML blogs and newsletters which are guaranteed to be 95%+ crap that's either irrelevant to me, wrong, useless or plagiarized from something I've already read.
Are you me? This is exactly how I feel. I've given up on trying to find a good source of Data Science news or articles.
If we're talking about ML models (e.g. Clustering Algorithms and Metrics), once a year I'll try to select a few books published in the last year read them. They'll always be some new ideas and algorithms, and I'll get to refresh my knowledge on existing topics as well.
Throughout the year I'll usually hear (HN/LinkedIn/colleagues) about related books (e.g. Designing Data Intensive Applications) and read them.
That's more than enough to keep me employed and ahead of the game at my job. I'm not a researcher nor a Twitter/LinkedIn "thought leader", so most of what I need to know needs to be tested and validated and not bleeding edge.
>It is likely that the Data Scientist role is in a long term decline...
Also
> Data science is in decline and vaguely defined
Reading this, you can think that "Data Science" jobs are decreasing. But I don't think that's true.
Let's just say that it's 2017 and I hire a team of 3 people with the job title of Data Scientist. One ends up focusing on the data side, one on modeling+analysis, and one on building the infrastructure. In 2023, I decide to change the job titles so one of them is now a Data engineer, one is now a Data Scientist, and one is now a ML Engineer to match what is happening in the job market.
It's still 3 jobs with 3 people doing the same thing. So the number of jobs aren't decreasing, but their titles are more specific. Overall, the number of "Data Science" jobs are still doing up.
Somebody will say "But that's exactly what the author said." But I think people who are new(ish) to this field might read it as "Data Science Jobs are decreasing." So I'm making this comment.
> skills such as data mining and visualisation are also out of favour.
Honestly, I just don't believe this. It's possible that as job descriptions are filled with different buzzwords, people just leave these out. For visualization it's also possible that there is a bigger focus on keywords of an established BI tool (e.g. PowerBI) instead of ad-hoc charts in matplotlib or ggplot. But some degree of data mining and visualization is useful, even to Data Engineers.