The Future of Data Science
9 comments
There is an oversupply of people who only know how to train models and throw them over a wall to have someone else sort out the mess.
There is an undersupply of technically skilled people who have the knowledge and experience to sort this mess out and bring it into a production system (I'm talking about data engineers and machine learning engineers).
I think this will correct itself in the sense that those who only did some online courses will not be able to find a data science job so they will either retrain themselves to become a data engineer or they will pick a different field (maybe the field they came from initially. I suppose having domain knowledge AND some data science skills could be very useful).
There is an undersupply of technically skilled people who have the knowledge and experience to sort this mess out and bring it into a production system (I'm talking about data engineers and machine learning engineers).
I think this will correct itself in the sense that those who only did some online courses will not be able to find a data science job so they will either retrain themselves to become a data engineer or they will pick a different field (maybe the field they came from initially. I suppose having domain knowledge AND some data science skills could be very useful).
Agreed, the engineering side is where valuable growth is/will be. Also from the limited sample I know of (people I know who are hiring managers) the candidate pool for data scientists lacks people with genuine analytical ability. Training models is easy, thinking the right way about data is hard.
There's also an undersupply of managers and executives that know the right way to build and manage these kinds of development efforts. They think trial & error of data products is something we can do via Scrum or somesuch. And they don't understand anything about what is being built or have the data literacy to question/examine/learn from the data insights of data products
My feeling is that it's the same as with trading - trading can be very profitable. Same goes for data science. Unless you really know what you are doing, you can get very wrong, without knowing it.
Both data science and trading have low entry barriers - you can easily find what appears to be opportunities or a working strategy / model. It's much harder to find something that will stand the test of time. In both disciplines you can easily get the feeling you are doing something right, while in reality you are going in a very wrong direction - finding spurious correlations, overfitting or worse.
I think most of those data scientists don't really know their craft.
Both data science and trading have low entry barriers - you can easily find what appears to be opportunities or a working strategy / model. It's much harder to find something that will stand the test of time. In both disciplines you can easily get the feeling you are doing something right, while in reality you are going in a very wrong direction - finding spurious correlations, overfitting or worse.
I think most of those data scientists don't really know their craft.
I bet "data science" as a term will soon be forgotten. Do you remember "data mining"? Same thing will happen to "data science". And the next thing will be even sillier, like "knowledge science".
[1]: https://www.knowledgescientist.org/
[2]: https://dmccreary.medium.com/from-data-science-to-knowledge-...
[1]: https://www.knowledgescientist.org/
[2]: https://dmccreary.medium.com/from-data-science-to-knowledge-...
> Where does this trend go? Will there an over-supply of data scientists soon?
There already is an oversupply at least for beginner-level candidates. The trend will be that more people who switch to data science will have to eventually give up because they can't find a job. And that companies will increase their requirements for academic credentials even further (a PhD is already a minimum requirement at some places).
The field itself will be fine, there are plenty of interesting and difficult problems to work on.
There already is an oversupply at least for beginner-level candidates. The trend will be that more people who switch to data science will have to eventually give up because they can't find a job. And that companies will increase their requirements for academic credentials even further (a PhD is already a minimum requirement at some places).
The field itself will be fine, there are plenty of interesting and difficult problems to work on.
Much like software 20 years ago, many orgs will jump on the buzzword bandwagon, few will make the organizational changes needed to do it well and realize a fuller transformation.
It’s easy to hire someone to do “machine learning” off in the corner, it’s harder to adopt data literacy through your org and get people to thoughtfully use data to make decisions. Including knowing the value and limits of the data you’re looking at. And this last item is truly needed to really leverage the data whether it’s analysis or more complex machine learning.
It’s easy to hire someone to do “machine learning” off in the corner, it’s harder to adopt data literacy through your org and get people to thoughtfully use data to make decisions. Including knowing the value and limits of the data you’re looking at. And this last item is truly needed to really leverage the data whether it’s analysis or more complex machine learning.
Not really a big issue to be honest. The market will sort itself out.
[deleted]
We're a boutique consultancy that's been doing this for about seven years. Most of the people who apply have a lot of misconceptions. The usual Coursera MOOC and Kaggle challenges on a nice, clean, CSV with clear objectives and well defined metrics, train a model and boom, build a Flask application and "deploy the model" for the most adventurous. Obviously, very few have an idea of the work it took to even define the problem, and rarely about the work it took to get to that "clean CSV", let alone how that model will actually deliver value or its lifecycle.
Yes, the overwhelming majority thinks that. I can't blame them really. Most of the content out there about this is by data virgins who've never touched actual data, mostly Tweeting/blogging/Mediuming/YouTubing/audience building. This is due to the fact that relatively few are actually doing this in real life. ML projects can be quite expensive and you're either in a large organization's ML team or, like us, a company that large organizations hire either for lack of an internal team, or to support their team. That is, if you want to work on actual problems, with real data, and real stakes, real infrastructure questions, etc.
Recently, though, there has been an uptick in the amount of MLOps content. That, too, has been innundated by people writing for "that site on Medium".
That is the way it is for now and the field is immature. There will be a lot of churn. Most people will give up. Some will keep up and learn and actually get to do that and evolve. So, one may be dismissive and it would be understandable. But we could also see it from a stand-point of a conversion funnel, and all that is driving people through that funnel. The content may irk me, but if it's driving someone to the field and that person sticks and learns... Sysadmin, webmaster, webdev, IT... all went through something similar.
We have been building our own platform[0] to leverage what we have learned and at least remove some of the toil from our plate and ease off a bit of the hiring pressure resulting of that shortage. Our PhD people needed to tap on someone's shoulder to set up or fix their environment, or deploy their model. We're removing taps on shoulders, reducing the cardinality of the profiles we need, and the skills required to operate. These issues can kill a small company, as you either had to hire people who could do it all, or people who were specialized. And you're left with people jumping around putting out fires on several projects at a time. It can drive many people crazy.
- [0]: https://iko.ai
Yes, the overwhelming majority thinks that. I can't blame them really. Most of the content out there about this is by data virgins who've never touched actual data, mostly Tweeting/blogging/Mediuming/YouTubing/audience building. This is due to the fact that relatively few are actually doing this in real life. ML projects can be quite expensive and you're either in a large organization's ML team or, like us, a company that large organizations hire either for lack of an internal team, or to support their team. That is, if you want to work on actual problems, with real data, and real stakes, real infrastructure questions, etc.
Recently, though, there has been an uptick in the amount of MLOps content. That, too, has been innundated by people writing for "that site on Medium".
That is the way it is for now and the field is immature. There will be a lot of churn. Most people will give up. Some will keep up and learn and actually get to do that and evolve. So, one may be dismissive and it would be understandable. But we could also see it from a stand-point of a conversion funnel, and all that is driving people through that funnel. The content may irk me, but if it's driving someone to the field and that person sticks and learns... Sysadmin, webmaster, webdev, IT... all went through something similar.
We have been building our own platform[0] to leverage what we have learned and at least remove some of the toil from our plate and ease off a bit of the hiring pressure resulting of that shortage. Our PhD people needed to tap on someone's shoulder to set up or fix their environment, or deploy their model. We're removing taps on shoulders, reducing the cardinality of the profiles we need, and the skills required to operate. These issues can kill a small company, as you either had to hire people who could do it all, or people who were specialized. And you're left with people jumping around putting out fires on several projects at a time. It can drive many people crazy.
- [0]: https://iko.ai
The chefs could be originally data scientists who moved to culinary science profession.
Where does this trend go? Will there an over-supply of data scientists soon?
It’s getting worrisome.