Joined Microsoft in the early 2010s, and Google recently in the 2020s. I see the same bad company culture traits in both cases (incompetent & feuding middle managers, silos of information, promotion based on launches not business impact, hired too many people, etc.).
I think one big difference is that Microsoft at the time had clearly fallen behind competitors, while Google hasn't yet, or not to the same extent. I believe this failure created enough humility at Microsoft that I found many people & teams to be open to new ideas in terms of work processes & culture. Implementing change was harder, but having the conversation wasn't.
I see very little of that openness or humility at Google at any level, I suppose because there hasn't been a major business threat to force a change in mindset, or to let go of long-tenured ineffective leaders. It's been disappointing, because I would have expected a company with a lot of supposedly intelligent people wouldn't need external threats to avoid creating the bad culture common to big old companies.
To me Google work culture in 2023 looks a lot like the Microsoft work culture from 2010, but most can't accept that reality.
Be honest and humble. The business won't work as you expect. Keep an open mind and adapt. Ego will prevent this from happening, don't let it.
Be crystal clear why your business needs to exist. Not why you think it should exist. Deeply understand the customer pain points, business domain, competitors, and find your niche. Sales and product development get easier with this.
Deeply understand your cash flows months in advance. Identify warning signals sooner rather than later. Things won't magically get better, act now.
I haven't listened to this specific podcast, but Ezra Klein went on paternity leave a couple months ago. He didn't host this episode, per the description.
Seems like he's prioritizing something other than career at the moment?
Agree with the general idea. Data engineers are like the essential workers of the data world, people who today may not receive the appropriate level of appreciation that they deserve.
I think the glut of data scientist occurs because we clump so many different skills and disciplines under the single term "data scientist". Data scientists today come from so many different backgrounds that the definition means something different to everyone. Because of this, the surface area of possible skills that could be expected of a data scientist is vast, to the point where it's pretty unlikely to be sufficiently competent in all of them, let alone a majority.
I'd like to go back to a world where we had a little more specificity about what kind of data scientist you are (e.g. I had no problem with terms like statistician and data miner), which could help ground expectations that others have of us, and it'd also help clearly define the scope of various career paths for the next generation.
- It's televised Sunday mornings (at least here in America), and it's way more interesting than anything else on at that time.
- It's a soap opera/drama, but socially acceptable for men to watch. You've got all the classic story tropes there (intra-/inter-team rivalries, underdog/favorite, hotshot newcomer/grizzled veteran, rich vs poor, etc.)
- People are inherently intrigued by man + machine collaborations, kids too. I didn't appreciate this until I hung around my friends' kids, and saw their amazement at any moving vehicle. Then I remembered, yes, I was once one of those kids.
With that said, I think F1 as a motorsport is pretty boring.
- If you're in it for the racing, I'd rather watch Indycar, which has the same or better racing with open-wheeled cars at 1/10th the price.
- If you're in it to see car technology translate to your road car, probably should watch a GT series instead.
- If you're in it for the technological innovation, none of those things get shared to viewers. As a data person, I would love to see how the sausage gets made, like how they translate learnings from the simulator to the real track, or the the trial & error in tweaking their aero or engine configs, CFD, etc. But for obvious sporting reasons, they can't. Like you can notice that Mclaren's nose is different, you just don't know why. Knowing why is the more interesting part IMO.
In your mind, how many years is a few years? I watch Formula E, but it's hard to imagine it becoming the pinnacle of motorsport anytime soon.
Formula E's definitely made progress: e.g. they no longer have to run two separate cars just to finish the race, and they are decently quick. But they're still pretty far away in other aspects: 45 minute race, on temporary circuits purposefully designed with many tight turns to allow for braking regen. Battery technology has to improve quite a bit if you want to see electric cars turning laps at Spa at standard race distances.
Econometrics is the application of statistical techniques on economics-related problems, typically to understand relationships between economic phenomena (e.g. income) and things that might be associated with it (e.g. education).
Machine learning is typically defined as a way to enable computers to learn from data to accomplish tasks, without explicitly telling them how.
Both fields can use logistic regression, regularization, and gradient descent to accomplish their goals, so in that sense there's no distinction.
But IMO there is a difference in their primary intention: econometrics typically focuses on inference about relationships, machine learning typically focuses on predictive accuracy. That's not to say that econometrics doesn't consider predictive accuracy, or that machine learning doesn't consider inference, but it's usually not their primary concern.
The first half of the essay aligns with your takeaway. But the second half does start to delve into how to identify useful obsessions, starting with "But there are some heuristics you can use to guess whether an obsession might be one that matters." Which sounds like an optimization, and which seems at least somewhat at odds with a major starting premise that you mentioned ("They're not doing it to impress us or to make themselves rich, but for its own sake.").
I agree with OP, the thesis makes sense assuming the primary goal of one's life is discovering world-changing ideas (what he's calling "genius"). That's understandable from a VC perspective since that's basically what VCs do. But most of us here aren't VCs, and while idea discovery is a great goal to have even if you're not a VC, it doesn't have to be the primary one.
Like others have said, the technical infrastructure is usually a manifestation of the people processes of the corporation. I think it's valuable to kinda ignore the technical stuff initially, and instead first understand the requirements of your customers. It's totally possible that the current system, as weird as it is, might satisfy your customers' requirements best. Unlikely, but it's possible.
But given that they hired you, chances are they know the current system isn't great, and they didn't possess the domain knowledge to fix it. I'd guess you & your company are aligned high-level that change is needed. It's just a matter of making sure you can align your ideas with the short- and long-term goals of the company, usually with a convincing story explaining how your technical changes drive business value.
For example, you mention that data exploration is hard, and I'm inferring this is a problem because you have multiple consultants independently scouring your datasets. If so, you could communicate to your customers that you can reduce consultant onboarding time from 5 days to 2 days (made that up) if you invested in aggregated datasets or a centralized data warehouse. If you can translate this to a dollar figure (like consultant hourly rate), that's even better.
As for what part you tackle first, I'd suggest finding a problem everyone knows about, but is straightforward for you to solve. Goal is to display immediate value, and gain the trust of the people around you. You don't solve the systemic problem immediately, but the trust you gain is currency you use months from now to really invest in the system. Because truth is, higher-ups rarely value invisible things like data quality or maintainability, they respond very positively to shiny new graphs and numbers.
FWIW I don't know if it's just me, but I feel like the bulk of data science is the ugly pipeline and architectural decisions you're facing now. I read people doing interesting modeling & machine learning work, but I keep wondering how much work went into getting the data into a modeling-ready state. I haven't worked at a company where the % of data team effort going to pipelines is less than, say, 80%.
I think one big difference is that Microsoft at the time had clearly fallen behind competitors, while Google hasn't yet, or not to the same extent. I believe this failure created enough humility at Microsoft that I found many people & teams to be open to new ideas in terms of work processes & culture. Implementing change was harder, but having the conversation wasn't.
I see very little of that openness or humility at Google at any level, I suppose because there hasn't been a major business threat to force a change in mindset, or to let go of long-tenured ineffective leaders. It's been disappointing, because I would have expected a company with a lot of supposedly intelligent people wouldn't need external threats to avoid creating the bad culture common to big old companies.
To me Google work culture in 2023 looks a lot like the Microsoft work culture from 2010, but most can't accept that reality.