Ask HN: Is it just me, or do most data scientists hate their jobs?
11 comments
Is every data scientist who's unhappy unhappy in a different way?
I think:
1. The average data scientist thinks they are too good to have a reproducible build, use version control responsibly, or use software engineering practices to put their practice on wheels. You can get them to make a report but they hate the thought of making a system that makes reports. I always hear things like "I don't want to get my hands dirty"
2. I tell people (especially those who would hire or supervise a data scientist) that they need to make a training set of 1000 to 10,000 examples if they want to train a model. Often that's the last I hear from them. Otherwise I hear some guff about they are too good to do that. Good luck getting data from Kaggle to fit your business problem.
What anecdotes do you have?
I think:
1. The average data scientist thinks they are too good to have a reproducible build, use version control responsibly, or use software engineering practices to put their practice on wheels. You can get them to make a report but they hate the thought of making a system that makes reports. I always hear things like "I don't want to get my hands dirty"
2. I tell people (especially those who would hire or supervise a data scientist) that they need to make a training set of 1000 to 10,000 examples if they want to train a model. Often that's the last I hear from them. Otherwise I hear some guff about they are too good to do that. Good luck getting data from Kaggle to fit your business problem.
What anecdotes do you have?
Per point 1, after interviewing a good few candidates in the space I think that most of them just don't have the backgound to effectively handle version control (it's more complicated when working with evolving datasets), automated reporting, or reproducible builds. They might have a CS background, but not the software engineering skills to execute any of those tasks effectively. So, whenever there's a time constraint from management they're going to skip those more formal software engineering practices to the organizations longer term detriment.
Actually "data rich products" are complicated because they involve evolving code and data. There are tools that are good at one but not tools that are good at both.
For instance Docker is a problem instead of a solution if part of your system is a several-gigabyte language model that changes frequently.
For instance Docker is a problem instead of a solution if part of your system is a several-gigabyte language model that changes frequently.
I tend to agree with tools either being good at code or data. I'm sure eventually there will be a solution, but for now it's a lot of bespoke tooling.
Docker can be a headache at times, but being able to maintain a consistent environment is very handy. For the large binary resources (trained models, dataset, etc) I've found it easy enough to just use docker volumes to mount read only resources. Other people have resorted to leaving read only assets on the local network which might be fast enough for your needs. As long as you're not unintentionally copying data, mounting a TB of data read only takes no time in my experience.
Docker can be a headache at times, but being able to maintain a consistent environment is very handy. For the large binary resources (trained models, dataset, etc) I've found it easy enough to just use docker volumes to mount read only resources. Other people have resorted to leaving read only assets on the local network which might be fast enough for your needs. As long as you're not unintentionally copying data, mounting a TB of data read only takes no time in my experience.
regarding point 1, I actually feel it's the opposite in my circles.
for myself and many of my unhappy colleagues, we would love to push code to prod and build software but we're cut off from that part of the job due to the growth of data engineering and ML engineering roles as their own disciplines/roles.
instead we're left with the boring one-off 'make a slide deck report and some bar charts' type work instead getting the privilege of getting our hands dirty and actually building something.
i guess that just goes to your point that we're all unhappy in different ways.
for myself and many of my unhappy colleagues, we would love to push code to prod and build software but we're cut off from that part of the job due to the growth of data engineering and ML engineering roles as their own disciplines/roles.
instead we're left with the boring one-off 'make a slide deck report and some bar charts' type work instead getting the privilege of getting our hands dirty and actually building something.
i guess that just goes to your point that we're all unhappy in different ways.
Having worked at a DS consultancy (as a dev) I suspect (from an outsiders view) its because very few DS’s are building something or really reaching a meaningful end point with a project? Might just be from the projects my old firm took on but that was the impression I got. Churn in the DS team was through the roof as well, so that indicates unhappiness (with the company at least)
This was my experience working in data science. I was extremely unhappy with the impact/results that I was able to deliver in my organization. Almost all of my analytics focused projects ended with a presentation to executives who promptly forgot about it and made decisions based on intuition anyway. It's draining to repeatedly push and push to a finish line that has no impact other than self-aggrandizement. Again, maybe this was just my organization - I'm not making statements about data science as a whole.
Over time I found that I love coding/delivering useful solutions to users - so I gradually positioned myself onto progressively more pure dev projects. I'm starting my first "real" software engineering role in a couple weeks, and I couldn't be more excited. :)
Over time I found that I love coding/delivering useful solutions to users - so I gradually positioned myself onto progressively more pure dev projects. I'm starting my first "real" software engineering role in a couple weeks, and I couldn't be more excited. :)
Basically, you can't control anything by giving people data. You have to let people run your code on their devices. Now you're in the driver's seat.
This and the GP's description of presenting data to executives reminded me strongly of the role of the Shakespearean clown/fool, like in King Lear. Someone with a licence to speak truth to the King/Queen, often in riddles they don't fully understand, wisely reminding them of the right or prudent option.
There isn’t enough talented DS management. They end up being managed by people who don’t know what is going on 4/5 times.
I've been in different Data Science roles. For most of them my output was code and code going to production (which meant different things in different places). For one of my jobs my output was mostly PowerPoint and I left that job quickly because of what mattcdrake discussed. I was also in Data Science consulting for a software company for a long while, so I got to see what a lot other Data Science teams were doing.
One huge challenge for many Data Scientists is that they have a lot of incompatible expectations from all the different teams they work with. The stakeholder wants a model in 3 weeks, but doesn't understand what a model is what it means to build one. The executive wants to see ROI on their Data Science Investment because it's built into their 12 month financial plan. The Data Engineering team provides access to the data, but don't have the resources to help with all the data issues you are seeing. The Subject Matter Expert is trying their best to help, but they don't understand why you can't easily implement what they are saying. The project manager wants an estimate, but you have no idea how fast any of these other parties are going to move or issues are going to be resolved.
Then there is your manager. How do they evaluate your performance exactly? Models deployed per month? Reports per week? Data Issues fixed a quarter, even though nobody but you seems to understand why those issues need to be resolved?
What if you do a lot of great work but the model sucks anyway because of stuff out of your control? As a Software Engineer, your manager can still say "great job" when you ship, even if it fails with users. Nobody blames a Data Analyst if a report they create shows something is bad.
It's hard to feel satisfied in a job where you face these conflicting expectations. I think in other roles (e.g. Software Engineer, Data Analyst, Data Engineer), the expectations are more consistent because you intersect with fewer people and your work is better defined. Even if the metrics are crap (e.g. Lines of Code), at least they are consistent.
I'm not unhappy in my current Data Scientist job. I'm also a bit older and jaded, so to me "not unhappy" is pretty good. One of the big reasons I'm not unhappy is because my output is code and not slides/reports.
One huge challenge for many Data Scientists is that they have a lot of incompatible expectations from all the different teams they work with. The stakeholder wants a model in 3 weeks, but doesn't understand what a model is what it means to build one. The executive wants to see ROI on their Data Science Investment because it's built into their 12 month financial plan. The Data Engineering team provides access to the data, but don't have the resources to help with all the data issues you are seeing. The Subject Matter Expert is trying their best to help, but they don't understand why you can't easily implement what they are saying. The project manager wants an estimate, but you have no idea how fast any of these other parties are going to move or issues are going to be resolved.
Then there is your manager. How do they evaluate your performance exactly? Models deployed per month? Reports per week? Data Issues fixed a quarter, even though nobody but you seems to understand why those issues need to be resolved?
What if you do a lot of great work but the model sucks anyway because of stuff out of your control? As a Software Engineer, your manager can still say "great job" when you ship, even if it fails with users. Nobody blames a Data Analyst if a report they create shows something is bad.
It's hard to feel satisfied in a job where you face these conflicting expectations. I think in other roles (e.g. Software Engineer, Data Analyst, Data Engineer), the expectations are more consistent because you intersect with fewer people and your work is better defined. Even if the metrics are crap (e.g. Lines of Code), at least they are consistent.
I'm not unhappy in my current Data Scientist job. I'm also a bit older and jaded, so to me "not unhappy" is pretty good. One of the big reasons I'm not unhappy is because my output is code and not slides/reports.
I worked in IT for an energy utility company and saw DS teams work and presentation. The DS team was given mandates like optimise the maintenance schedule for a coal fired power plant. Optimise the turning on/off hydro electricity based on water flow, weather, electricity market rates. Optimise bids to electricity market (works like a stock market). Automatically evaluate Solar panel installation suitability from aerial photos of roof and tree cover, maps, sun angle, location etc. Aren't they interesting problems to solve.
Half of a counterpoint, I fall under the data science umbrella and any current dissatisfaction boils down to application domain (and anything that falls into tech worker gripes which aren't specific to data sciences at all). Given the right organization there's interesting problems to solve, plenty of results to visualize, important company directions to point out, enjoyable literature surveys, etc.
I've seen plenty of roles out there where it's more data analyst style work which appears to be more plug and chug work. None of those appeal to me as well it takes the scientist bit out of the role. No real exploration, you know? Analytics companies sit somewhere in-between and seem to have some similar grinds, but once you get to the level where the data becomes very complex the problem space becomes humongous. At that stage things are more limited by where management decides they want to have things explored than the role itself.
The general domain has problems and can be an ethical minefield, but it's at least interesting enough that I plan on sticking around for a while under various job titles in the same space.
I've seen plenty of roles out there where it's more data analyst style work which appears to be more plug and chug work. None of those appeal to me as well it takes the scientist bit out of the role. No real exploration, you know? Analytics companies sit somewhere in-between and seem to have some similar grinds, but once you get to the level where the data becomes very complex the problem space becomes humongous. At that stage things are more limited by where management decides they want to have things explored than the role itself.
The general domain has problems and can be an ethical minefield, but it's at least interesting enough that I plan on sticking around for a while under various job titles in the same space.
"After an initial honeymoon phase, I'll get hit by a strong combo of frustration/boredom/cynicism and want to quit."
I feel the same way as a software dev.
I'm a beekeeper. You can't really make enough to support yourself with beekeeping alone. Bulk honey prices are garbage, mostly due to imports. You can make decent money selling locally as an artisanal product, but depending on the region and distribution channels the ceiling for your sales might be pretty low. In my region, I would probably need 4 yards with 20 hives each, several miles from each other, and in areas with good natural features (nectar sources, water, accessible) to make revenue of about $40-50k. I've heard people say it takes 3 years to break even. I think that's true as hobbyist beekeepers, but probably more like 5 years as a large scale operation. Local pollination services can be another option to make money.
I feel the same way as a software dev.
I'm a beekeeper. You can't really make enough to support yourself with beekeeping alone. Bulk honey prices are garbage, mostly due to imports. You can make decent money selling locally as an artisanal product, but depending on the region and distribution channels the ceiling for your sales might be pretty low. In my region, I would probably need 4 yards with 20 hives each, several miles from each other, and in areas with good natural features (nectar sources, water, accessible) to make revenue of about $40-50k. I've heard people say it takes 3 years to break even. I think that's true as hobbyist beekeepers, but probably more like 5 years as a large scale operation. Local pollination services can be another option to make money.
It's because most stuff that engineers and DS work on is to make money for a company. Most of the stuff is internal tools, something used by internal staff, etc. And when you "finish" the job it's not something physical and hard that you can say "wow look at that, look at what we built".
So basically you're working on a team without much autonomy (there's only so many ways to fit in your contribution or your model), there's very little connection to the end product even though you worked hard, and most of the time you're working alone just coding something out.
So: 1. Little autonomy 2. No grand physical end product 3. Little deep social interaction
So basically you're working on a team without much autonomy (there's only so many ways to fit in your contribution or your model), there's very little connection to the end product even though you worked hard, and most of the time you're working alone just coding something out.
So: 1. Little autonomy 2. No grand physical end product 3. Little deep social interaction
What are your theories for why it’s unhappy? I always assumed it would be more fulfilling than software engineering.
It's just you. Any actual data scientist knows that only 49.35% hate their job.
I've had positions at a few different sized companies (big public company vs small startup) working on different types of products (B2B/enterprise vs B2C), and it doesn't matter. After an initial honeymoon phase, I'll get hit by a strong combo of frustration/boredom/cynicism and want to quit.
It could be that I'm burning out or just have a short attention span, but this seems like this is a pervasive issue among DS positions. Almost every DS I talk to (coworkers as well as those in other companies) is unhappy or disillusioned about their role. There are plenty of employees at these companies that genuinely enjoy their work, yet there isn't a single DS I know where I think "I can envision this person doing this work for a couple decades and making a full career out of it until they retire". Everybody is trying to transition to a different type of role (engineering, product management, people management) or just dropping out of tech completely to become a beekeper or something.
I have lots of theories for why this specific role seems so unhappy, but I wanted to take a step back from that and get some broader takes on whether this phenomenon is real. Are DSers -- the job title billed as the 'sexiest job' a few years ago -- generally unhappy in their work? Or is this my own confirmation/sampling bias?