Don’t Let Metrics Undermine Your Business(hbr.org)
hbr.org
Don’t Let Metrics Undermine Your Business
https://hbr.org/2019/09/dont-let-metrics-undermine-your-business
7 comments
This is a recipe for mediocrity. Great things are created by people following their gut, not their metrics.
Jobs didn’t have a data scientist telling him to make the iPhone. A data scientist would have told Notch to make a Call of Duty clone instead of Minecraft. There was no data scientist behind Gates, Warren, Zuckerberg, Musk, or anyone of their ilk.
Looking back in history, some great discoveries and inventions were held back because the “data” said it wasn’t possible. Everybody thought ulcers were caused by stress and bad food, then Barry Marshall came along and literally trusted his gut by eating a bunch of bacteria to prove that’s how ulcers were formed.
At best, data science helps you squeeze a couple extra bucks from someone else’s ingenuity. Once Jobs died, the data scientists got their hands on the iPhone and chopped it up into different models. And it was the data scientists who made your Facebook feed ad-ridden junk. And at worst, data science is destroying our society. It’s the data scientists who are driving this insane data vacuum that is plaguing the internet and our daily lives. You thwarted the 2016 election by weaponizing data against America. You did it again with brexit. No doubt you will be out in full force for every major political event for the next few decades, sullying the discourse of the electorate.
Bit of a rant... sorry. I hate this over-reliance on data and the smugness it’s captors carry.
Jobs didn’t have a data scientist telling him to make the iPhone. A data scientist would have told Notch to make a Call of Duty clone instead of Minecraft. There was no data scientist behind Gates, Warren, Zuckerberg, Musk, or anyone of their ilk.
Looking back in history, some great discoveries and inventions were held back because the “data” said it wasn’t possible. Everybody thought ulcers were caused by stress and bad food, then Barry Marshall came along and literally trusted his gut by eating a bunch of bacteria to prove that’s how ulcers were formed.
At best, data science helps you squeeze a couple extra bucks from someone else’s ingenuity. Once Jobs died, the data scientists got their hands on the iPhone and chopped it up into different models. And it was the data scientists who made your Facebook feed ad-ridden junk. And at worst, data science is destroying our society. It’s the data scientists who are driving this insane data vacuum that is plaguing the internet and our daily lives. You thwarted the 2016 election by weaponizing data against America. You did it again with brexit. No doubt you will be out in full force for every major political event for the next few decades, sullying the discourse of the electorate.
Bit of a rant... sorry. I hate this over-reliance on data and the smugness it’s captors carry.
I would say it's a recipe for continuation rather than innovation. You're unlikely to invent anything revolutionary just by looking at metrics, but you're also unlikely to keep a business going without them. It's easy to look at something like the iPhone and say, "metrics didn't invent that!" but metrics absolutely put millions of phones in the hands of customers, built an effective production and supply chain, optimized the whole experience, and turned Apple into a trillion dollar company. As with most things, there's a balance and a place for each.
I disagree that metrics helped Apple scale, but we’d have to wade in to a semantic debate to settle this.
When you have a killer idea, scaling is obvious; just do more. More people, more machines, more materials, etc. You don’t need a number to tell you how much more to scale, because you always want the max anyways. Tim Cook was told there weren’t enough CNC machines in the world to manufacture the unibody MacBook. So he bought the company that makes CNC machines. What’s the point of metrics when you can shift the entire world to meet your needs? Great things are like that.
Metrics are for mediocre products and services. We use them in banking because people fucking hate the banking system and we have to pull teeth to get customers to spend money. McDonald’s uses metrics to gauge how shitty they can make their beef without people noticing. Hollywood uses metrics to decide which movie franchise to reboot or make a sequel of. Metrics are a sign that you are done innovating and you are squeezing the sponge before someone else’s innovation dethrones yours.
When you have a killer idea, scaling is obvious; just do more. More people, more machines, more materials, etc. You don’t need a number to tell you how much more to scale, because you always want the max anyways. Tim Cook was told there weren’t enough CNC machines in the world to manufacture the unibody MacBook. So he bought the company that makes CNC machines. What’s the point of metrics when you can shift the entire world to meet your needs? Great things are like that.
Metrics are for mediocre products and services. We use them in banking because people fucking hate the banking system and we have to pull teeth to get customers to spend money. McDonald’s uses metrics to gauge how shitty they can make their beef without people noticing. Hollywood uses metrics to decide which movie franchise to reboot or make a sequel of. Metrics are a sign that you are done innovating and you are squeezing the sponge before someone else’s innovation dethrones yours.
> You don’t need a number to tell you how much more to scale, because you always want the max anyways.
And you bankrupted the company building factories you didn't need, because iPhone demand has its limits.
And you bankrupted the company building factories you didn't need, because iPhone demand has its limits.
i mostly agree with you. great intuition and experience can largely reduce the need for metrics. the problem is that not every business decision is clear cut and people who have to make them are not always Jobs or Gates.
what metrics give you is a way to give the average joe (which is most of us) a way to take calculated risks when there are multiple stakeholders each with their own opinion of what should be done.
metrics - like statistics - can be used effectively as a guide or can be used obsessively like dogma.
what metrics give you is a way to give the average joe (which is most of us) a way to take calculated risks when there are multiple stakeholders each with their own opinion of what should be done.
metrics - like statistics - can be used effectively as a guide or can be used obsessively like dogma.
[deleted]
1. Steve Jobs didn't make the iPhone.
2. Apple has a very large Product Marketing team who do extensive quantitive and qualitative research on what products they should make, features should be delivered etc. This is the same for Microsoft, Facebook etc. I worked at Apple and each team will have a Product Owner/Program Manager assigned who decides what is done where as Engineers decide how it is done.
3. Data Scientists aren't solely responsible for releasing more iPhone models or how many ads to show. Those sort of decisions are cross-functional involving Finance, Product, Marketing, Strategy etc.
4. Data Scientists are basically engineers. They largely do what they are told to do by the "business". So not sure why you seem to think they are root of all evil.
2. Apple has a very large Product Marketing team who do extensive quantitive and qualitative research on what products they should make, features should be delivered etc. This is the same for Microsoft, Facebook etc. I worked at Apple and each team will have a Product Owner/Program Manager assigned who decides what is done where as Engineers decide how it is done.
3. Data Scientists aren't solely responsible for releasing more iPhone models or how many ads to show. Those sort of decisions are cross-functional involving Finance, Product, Marketing, Strategy etc.
4. Data Scientists are basically engineers. They largely do what they are told to do by the "business". So not sure why you seem to think they are root of all evil.
I think you’re missing the forest for the trees in my comment. I’m not really criticizing Data Science, the job function, but the field in general and the obsession over it.
Your last point kind of weighs in on a point I didn’t quite get to; that data can be found to reinforce any decision. If business wants to do X, they will have the data scientists prove to them that X is good. Data can be a huge force of tyranny in this way, this is where my disdain for the field comes from.
Your last point kind of weighs in on a point I didn’t quite get to; that data can be found to reinforce any decision. If business wants to do X, they will have the data scientists prove to them that X is good. Data can be a huge force of tyranny in this way, this is where my disdain for the field comes from.
Sure is was JFK that got us to the moon, not the scientists and engineers doing calculations, experiments and looking at the data.
Both vision and drive and the ability to execute are essential.
What happened at Apple was engineers came to Jobs with a prototype and he saw the potential and drove it relentlessly forward. Same for the Mac - he saw what Xerox had done and recognized the potential.
Yes he trusted his own judgement - but I'd say if you saw either Xerox prototype in the age of terminals or a touch device in the age of fiddly keyboards it doesn't take a genius to see the potential - it just takes a hero ( courage ) to bet on it.
Finally, on the dark arts of advertising, moving from the 'gut' to being data driven - sadly I'd say that's what has made it more effective as a tool.
Both vision and drive and the ability to execute are essential.
What happened at Apple was engineers came to Jobs with a prototype and he saw the potential and drove it relentlessly forward. Same for the Mac - he saw what Xerox had done and recognized the potential.
Yes he trusted his own judgement - but I'd say if you saw either Xerox prototype in the age of terminals or a touch device in the age of fiddly keyboards it doesn't take a genius to see the potential - it just takes a hero ( courage ) to bet on it.
Finally, on the dark arts of advertising, moving from the 'gut' to being data driven - sadly I'd say that's what has made it more effective as a tool.
> Finally, on the dark arts of advertising, moving from the 'gut' to being data driven - sadly I'd say that's what has made it more effective as a tool.
Part of the reason metrics work so well in marketing is because a lot of accepted industry practices are inherently malicious acts towards other human beings. So unlike other optimization processes, you don't care all that much about side effects of optimizing to metrics. For instance - do customers hate your guts for your advertising spam? Doesn't matter - the product becomes associated with a strong emotion, making it more likely some of them will buy it few months from now. Mission accomplished.
Part of the reason metrics work so well in marketing is because a lot of accepted industry practices are inherently malicious acts towards other human beings. So unlike other optimization processes, you don't care all that much about side effects of optimizing to metrics. For instance - do customers hate your guts for your advertising spam? Doesn't matter - the product becomes associated with a strong emotion, making it more likely some of them will buy it few months from now. Mission accomplished.
> do customers hate your guts for your advertising spam?
> Doesn't matter - the product becomes associated with a strong emotion, making it more likely some of them will buy it few months from now.
Interesting, does that really work? Got data you can point to, to back that one up?
I take your broader point of them not really caring - we are seeing that in data driven politics in the UK right now.
Obviously this host/parasite interaction will evolve - but at the moment it seems the population doesn't have many effective defenses. It's a bit like the opioid epidemic or obesity epidemic - currently people are being given the politics they apparently want.
Interesting, does that really work? Got data you can point to, to back that one up?
I take your broader point of them not really caring - we are seeing that in data driven politics in the UK right now.
Obviously this host/parasite interaction will evolve - but at the moment it seems the population doesn't have many effective defenses. It's a bit like the opioid epidemic or obesity epidemic - currently people are being given the politics they apparently want.
> Interesting, does that really work?
It's the basis of the concept of "brand awareness", on which many billions of dollars are spent every year.
I'll give you a single data point as an example, and I'm sure you'll be able to relate. There's a certain on-line currency exchange operating in Poland that has a tendency to run really cringy, annoying ads in cinemas, and those ads somehow get played 3 or 4 times in a row in the span of 5 minutes. After sitting through them a couple times I got really, really sick of them, and had pretty negative feelings associated with the brand. Fast-forward three years, I have my own business now and foreign customers, so I find myself needing an on-line currency exchange. Guess which service I checked out first?
It's the basis of the concept of "brand awareness", on which many billions of dollars are spent every year.
I'll give you a single data point as an example, and I'm sure you'll be able to relate. There's a certain on-line currency exchange operating in Poland that has a tendency to run really cringy, annoying ads in cinemas, and those ads somehow get played 3 or 4 times in a row in the span of 5 minutes. After sitting through them a couple times I got really, really sick of them, and had pretty negative feelings associated with the brand. Fast-forward three years, I have my own business now and foreign customers, so I find myself needing an on-line currency exchange. Guess which service I checked out first?
I understand brand awareness - just was skeptical that companies went out and out to be annoying to stick in the memory - but I think you are right some do.
You need both, I agree that given metrics don't predict the possible futures that depart significantly from the current state, you do need to support innovation without numbers to always back those ideas up. For every iPhone there is for example the wireless technology that powers the iPhone, which needed a huge amount of data driven decisions to get mobile masts in the right places, reliably send and receive data... The battery tech. Each of those innovations was the product of some good ideas combined with data and improvements that lead to the next innovations with countless failed innovations along the way.
I think you place too much emphasis on the "genius" that combines these novel ideas, and I'm not saying it isn't required, but I think there's a lot of these ideas that die and a lot more data behind them than is talked about.
For the iPhone to work it had to to be at a time where the tech was right, the price of components could enable it to be sold at an acceptable price-point, the UX had to be tested on countless people.
To be more succinct, the right amount of optimisation, data driven improvement and innovation combine with the right timing.
Ceaseless optimisation can lead a good product to become horrible, but you need numbers.
Surely you can see the symbiosis between things like data driven battery improvement and optimisation and the possiblity of something like the iPhone. The iPhone would not exist without data driven decisions.
It wouldn't exist without vision and risk taking either.
I think you place too much emphasis on the "genius" that combines these novel ideas, and I'm not saying it isn't required, but I think there's a lot of these ideas that die and a lot more data behind them than is talked about.
For the iPhone to work it had to to be at a time where the tech was right, the price of components could enable it to be sold at an acceptable price-point, the UX had to be tested on countless people.
To be more succinct, the right amount of optimisation, data driven improvement and innovation combine with the right timing.
Ceaseless optimisation can lead a good product to become horrible, but you need numbers.
Surely you can see the symbiosis between things like data driven battery improvement and optimisation and the possiblity of something like the iPhone. The iPhone would not exist without data driven decisions.
It wouldn't exist without vision and risk taking either.
Did you think my post was smug? Because I'm saying what you're saying. I agree with you. My post was just to say it's a tough pill to swallow. Bitter medicine is still medicine.
Except I wouldn't say data science is destroying our society. Maybe data science done the blind lazy way is, but shitty science isn't representative of good science.
Except I wouldn't say data science is destroying our society. Maybe data science done the blind lazy way is, but shitty science isn't representative of good science.
There is something about numbers that seems to push us into rational mode and totally subjugate the emotional side of us that does great without data, and allows for us to act in abundance. My best idea is to display something like standard deviation and a delta between actual and the target metric, perhaps as low resolution as green-yellow-red to the employee responsible for the metric. It may hypothetically be somebody's job it adjust the formula and never reveal more detail than that, in this scenario.
A concrete example, if at a restaurant we expect meals to be delivered within 7 minutes and we see the nightly average is currently 8:08 then some team members might start pushing to go faster and cut corners, compromising discipline and training effort--even if the company's values include accuracy and precision above speed. The values are laid down by the visionary founders, but the only metric that the workers have is an exact number and target. Somewhere along the chain between founding and the last hands on the food the values can be lost, and I feel like chasing the metric has a lot to do with it. Everyone pushes to make the number "because" of some factor, in my opinion probably pressure from above or a colleague.
Pardon me if I've abused any terms, thanks. The factor that I am sure of is entering scarcity mode rather than abundance mode, and I'd like to know more about how reading metrics affects us and whether there are known methods to systemically remove metrics from view such that we can remain true to all of the values outside of the metric. If we're short of a target then there is a gap in training or experience; nobody should be so driven by metrics that it overrides their acting in abundance.
I know this can be said to be coming from fantasy land but I've come from working in a number of domains in a short period of time, I feel like there's something in here even if it's hard to find.
A concrete example, if at a restaurant we expect meals to be delivered within 7 minutes and we see the nightly average is currently 8:08 then some team members might start pushing to go faster and cut corners, compromising discipline and training effort--even if the company's values include accuracy and precision above speed. The values are laid down by the visionary founders, but the only metric that the workers have is an exact number and target. Somewhere along the chain between founding and the last hands on the food the values can be lost, and I feel like chasing the metric has a lot to do with it. Everyone pushes to make the number "because" of some factor, in my opinion probably pressure from above or a colleague.
Pardon me if I've abused any terms, thanks. The factor that I am sure of is entering scarcity mode rather than abundance mode, and I'd like to know more about how reading metrics affects us and whether there are known methods to systemically remove metrics from view such that we can remain true to all of the values outside of the metric. If we're short of a target then there is a gap in training or experience; nobody should be so driven by metrics that it overrides their acting in abundance.
I know this can be said to be coming from fantasy land but I've come from working in a number of domains in a short period of time, I feel like there's something in here even if it's hard to find.
I agree with you. I feel a like optimizing metrics often leads to an overall worse product, but which, thanks to the actions of a minority of users who respond strongly to the change (while the rest grumble but don't change their behaviour much), gives the impression of having achieved an improvement.
I'm not so sure. Sounds like a urban legend "idea guys" want to believe.
For me that depends on the level you are looking at: Tactical, Operational, Strategic. The first two have to be data and metrics driven, that's where you make or loose money right now.
On the strategic level your performance metrics are giving only a baseline for your capabilities, strength and weaknesses. That should only be affecting the implementation of a given strategy and never the definition of it.
And as always, if a metrics system is ill defined from the start it will steer you in the wrong direction. Like a broken speedometer in a car. And we didn't even start to talk about the corresponding incentive and bonus program yet...
On the strategic level your performance metrics are giving only a baseline for your capabilities, strength and weaknesses. That should only be affecting the implementation of a given strategy and never the definition of it.
And as always, if a metrics system is ill defined from the start it will steer you in the wrong direction. Like a broken speedometer in a car. And we didn't even start to talk about the corresponding incentive and bonus program yet...
Schlitz beer is a great example of the downfall of managing metrics from a strategic level. They began cutting cost/quality of ingredients in their beer to compete with Anheuser Busch. "Salami slicing the company to death."[0]
>* the steps from A to B and from B to C might have been tiny and unnoticeable, but the steps from A to M added up to a big leap.* [1]
The metrics looked great, getting more beer out of less ingredients, in less time. Until the 70's when they hit a tipping point and drinkers stopped buying, tanking the company.
[0] https://en.wikipedia.org/wiki/Salami_slicing [1] https://beerconnoisseur.com/articles/how-milwaukees-famous-b...
>* the steps from A to B and from B to C might have been tiny and unnoticeable, but the steps from A to M added up to a big leap.* [1]
The metrics looked great, getting more beer out of less ingredients, in less time. Until the 70's when they hit a tipping point and drinkers stopped buying, tanking the company.
[0] https://en.wikipedia.org/wiki/Salami_slicing [1] https://beerconnoisseur.com/articles/how-milwaukees-famous-b...
I think the whole "data-driven" ideology is getting worn out.
It's especially bad on a lot of data science teams because there can be a lack of serious management experience and know-how if the team has a background mostly in academia. Academia is full of extremely dysfunctional behavior, even moreso than corporations, which is shockingly hard to pull off!
Everything described in this article is old hat for experienced, knowledgeable executives, people in the know have been talking about these ideas for at least half a century (dating back to Drucker, Deming and others).
While I agree with the authors' recommendations, I disagree about their diagnosis. I think Wells Fargo's main problem was a lack of internal controls, which is a boring, but necessary part of any large business.
It's especially bad on a lot of data science teams because there can be a lack of serious management experience and know-how if the team has a background mostly in academia. Academia is full of extremely dysfunctional behavior, even moreso than corporations, which is shockingly hard to pull off!
Everything described in this article is old hat for experienced, knowledgeable executives, people in the know have been talking about these ideas for at least half a century (dating back to Drucker, Deming and others).
While I agree with the authors' recommendations, I disagree about their diagnosis. I think Wells Fargo's main problem was a lack of internal controls, which is a boring, but necessary part of any large business.
This speaks to a concept in data science I like to call "measurability bias" - the entire field of data science is biased toward making decisions on things that are, well, measurable. If you try to apply data science to a problem whose objective function doesn't have robust representative measurements tied to it, you are bound to fail. Being able to read subtle facial cues, measure the energy in a room, basically all of the subtle empathic interactions required to gauge customer satisfaction... we're not there yet.
Nobody has ever been fired for believing data.
Unless it is data about political stuff like religion, race or gender. I bet someone has even gotten fired for believing in evolution.
Bringing up Culture War ideas like this in unrelated topics is a good way to derail them
This article represents one of the problems I have with HBR and "business school" literature in general.
It's a rehash of tried and true principles that have been described numerous times before almost always without attribution to original sources.
What's wrong with that?
Well, for one the authors are passing off work that is clearly not their own. Fine, whatever, it's a b-school publication, nobody takes their intellectual output seriously anyways.
But the more important problem is that you lose a sense of where this knowledge has come from, where it's been applied, how it's been applied, and other important context.
In other words, reading this article is basically a waste of time. It's better to stick to a handful of classic business texts (Drucker's a good place to start and yes, he goes over all the recommendations these authors make in Effective Executive). Reading Kaplan on the Balanced Scorecard is another good place to start.
It's a rehash of tried and true principles that have been described numerous times before almost always without attribution to original sources.
What's wrong with that?
Well, for one the authors are passing off work that is clearly not their own. Fine, whatever, it's a b-school publication, nobody takes their intellectual output seriously anyways.
But the more important problem is that you lose a sense of where this knowledge has come from, where it's been applied, how it's been applied, and other important context.
In other words, reading this article is basically a waste of time. It's better to stick to a handful of classic business texts (Drucker's a good place to start and yes, he goes over all the recommendations these authors make in Effective Executive). Reading Kaplan on the Balanced Scorecard is another good place to start.
> But the more important problem is that you lose a sense of where this knowledge has come from, where it's been applied, how it's been applied, and other important context.
Thank you for saying this!
A thought I have had in the past few years is "there's been nothing new in management since Drucker and Deming."
I go back and forth because some people have done a nice job of updating their ideas/applying them to modern disciplines, which is valuable.
However, what you said about where that knowledge has come from/been applied/how/context is sorely missing.
That these ideas are still not common knowledge leads to so many unnecessary mistakes.
Thank you for saying this!
A thought I have had in the past few years is "there's been nothing new in management since Drucker and Deming."
I go back and forth because some people have done a nice job of updating their ideas/applying them to modern disciplines, which is valuable.
However, what you said about where that knowledge has come from/been applied/how/context is sorely missing.
That these ideas are still not common knowledge leads to so many unnecessary mistakes.
Seems to me Surrogation is very similar to The Cobra Effect / Perverse Incentives (https://www.geckoboard.com/learn/data-literacy/statistical-f...) , or the McNamara fallacy: https://www.geckoboard.com/learn/data-literacy/statistical-f...
Both very real problems that have existed forever. The solution, as the article suggests, is not relying on just a single metric, and not losing sight of your wider strategy. There’s a really good article on Reforge about the problems with the recent trend of a single “North Star Metric”: https://www.reforge.com/blog/north-star-metric-growth. Stacey Barr’s another thought leader on the subject: https://www.staceybarr.com/measure-up/a-single-kpi-that-meas....
Metrics absolutely cannot replace strategy and vision. It needs to be a core value of the company that you’d rather do the right thing by the wider mission than drive a metric at any cost. Managers at every level need to live up to that. This is harder the bigger the company. It’s one of the things OKRs, love or hate them, try and help with. Like with everything it’s whether you comply with the spirit or the letter of the law. And whether people get rewarded for latter!
Both very real problems that have existed forever. The solution, as the article suggests, is not relying on just a single metric, and not losing sight of your wider strategy. There’s a really good article on Reforge about the problems with the recent trend of a single “North Star Metric”: https://www.reforge.com/blog/north-star-metric-growth. Stacey Barr’s another thought leader on the subject: https://www.staceybarr.com/measure-up/a-single-kpi-that-meas....
Metrics absolutely cannot replace strategy and vision. It needs to be a core value of the company that you’d rather do the right thing by the wider mission than drive a metric at any cost. Managers at every level need to live up to that. This is harder the bigger the company. It’s one of the things OKRs, love or hate them, try and help with. Like with everything it’s whether you comply with the spirit or the letter of the law. And whether people get rewarded for latter!
It's interesting to ponder how a metric, through its design, is a psychological brain hack that trumps rational thought and common sense. Perhaps it's the gamification aspect that gets us?
An example I think everyone can relate to is in education where grading is a rich source of perverse incentives. In developing minds, no less. Yet the metric persists.
(Just to be clear, the alternative to grading is not no feedback, but personalized, situation-specific feedback. Freeing yourself from a bad metric can open doors.)
An example I think everyone can relate to is in education where grading is a rich source of perverse incentives. In developing minds, no less. Yet the metric persists.
(Just to be clear, the alternative to grading is not no feedback, but personalized, situation-specific feedback. Freeing yourself from a bad metric can open doors.)
This reminds me of Goodhart's Law: "When a measure becomes a target, it ceases to be a good measure."
It seems like in the case of employees at a company, just knowing that a parameter is being measured can result in it implicitly becoming a target even when there is no explicit focus on it (especially if it is the only or one of a small number of metrics that is discussed).
I have observed that in general most humans are very bad at considering the incentives created by rules and rewards they impose or seek to impose.
It seems like in the case of employees at a company, just knowing that a parameter is being measured can result in it implicitly becoming a target even when there is no explicit focus on it (especially if it is the only or one of a small number of metrics that is discussed).
I have observed that in general most humans are very bad at considering the incentives created by rules and rewards they impose or seek to impose.
Goodhart’s Law really only applies to measures that aren't measuring the actual figure of merit but a proxy that is incorrectly assumed to be inextricably tied to it.
This is very common, though, and a real problem associated with it is people pick proxies without clearly documenting (or even thinking through) what they want to measure, why they pick a particular proxy, and how the proxy might fall. As a result, people remote from the decision (including the original decision maker) tend to incautiously apply the proxy because they aren't even aware that it is not the actual figure of merit.
This is very common, though, and a real problem associated with it is people pick proxies without clearly documenting (or even thinking through) what they want to measure, why they pick a particular proxy, and how the proxy might fall. As a result, people remote from the decision (including the original decision maker) tend to incautiously apply the proxy because they aren't even aware that it is not the actual figure of merit.
The point of the law is that virtually every measure is a proxy to some extent. Finding a measure that exactly matches what you really want to optimize is almost impossible.
Metrics are great for optimizing your business not for deciding which one to build.
The problem talked about in the article is that it's not easy to express your optimization goals in terms of concrete metrics, and your optimization process will always optimize those metrics, not your real goals.
> The problem talked about in the article is that it's not easy to express your optimization goals in terms of concrete metrics,
Largely, because what you are wanting to optimize is something like current value of the future income stream of the firm. So a proxy is taken, which inevitably has bias, and you need to understand why hat that big is likely to be a continuously sanity check your results for reason to think that you may being caught in that bias. Which you probably can't measure well (though maybe something has improved since you adopted the proxy), because if you could you'd have a better proxy to start with.
Largely, because what you are wanting to optimize is something like current value of the future income stream of the firm. So a proxy is taken, which inevitably has bias, and you need to understand why hat that big is likely to be a continuously sanity check your results for reason to think that you may being caught in that bias. Which you probably can't measure well (though maybe something has improved since you adopted the proxy), because if you could you'd have a better proxy to start with.
True, though I was thinking about something else still: unless you're a completely amoral alien mind, even your for-profit goals have some hidden value judgements and preferences attached. Those extra constraints are incredibly hard to capture as explicit goals and thus impossible to properly measure in a quantified way. So if you now turn the optimization knob up to 11 and just let it work for a while, you might be appalled by what it comes up with. Or if not you, the people on the receiving end of your optimizations definitely will.
Yes, the art is always to pick the right metrics.
The problem is when companies choose qualitative and thus interpretational metrics.
Just stay to the simple easily trackable and quantitative ones and the make it your job to interpret them qualitatively.
The problem is when companies choose qualitative and thus interpretational metrics.
Just stay to the simple easily trackable and quantitative ones and the make it your job to interpret them qualitatively.
Most real-life goals can't be easily expressed with few "easily trackable" metrics. Take example from the article, customer satisfaction. That's half of the problem; the other half is that metrics can be gamed, and they'll end up being gamed accidentally or for profit if you're not careful. Like, the easiest way to make a system stable is to make it so painful to use that people don't use it - and if they don't use it, they can't break it. Or, the example of return tracking from the TFA.
I read a lot about how data-driven companies measure this and that, often through questionable, privacy-violating measures. What I don't read about is how do these companies ensure the metrics are actually valid - that they're correctly sampling the population[0], or that they're measuring what the authors think they're measuring, or that they're not being misreported or otherwise gamed (very common if the value of a metric impacts someone's career or even workload).
--
[0] - e.g. voluntary surveys usually don't, telemetry increasingly doesn't either as more and more people are aware of it and disable it.
I read a lot about how data-driven companies measure this and that, often through questionable, privacy-violating measures. What I don't read about is how do these companies ensure the metrics are actually valid - that they're correctly sampling the population[0], or that they're measuring what the authors think they're measuring, or that they're not being misreported or otherwise gamed (very common if the value of a metric impacts someone's career or even workload).
--
[0] - e.g. voluntary surveys usually don't, telemetry increasingly doesn't either as more and more people are aware of it and disable it.
That's what I am saying.
Don't choose things that can't be tracked quantitatively. Track things quantitatively and then make qualitative decisions.
At Square, we had "Make Commerce Easy". This was qualitative and thus up for interpretation. It became even vaguer when it became "Economic empowerment".
These things can't and shouldn't be tracked as if they are quantitative metrics but instead used as guiding stars.
So the trick is just to separate the two and as I said. Use data metrics for optimizing and not for creating or measuring of qualitative goals.
Don't choose things that can't be tracked quantitatively. Track things quantitatively and then make qualitative decisions.
At Square, we had "Make Commerce Easy". This was qualitative and thus up for interpretation. It became even vaguer when it became "Economic empowerment".
These things can't and shouldn't be tracked as if they are quantitative metrics but instead used as guiding stars.
So the trick is just to separate the two and as I said. Use data metrics for optimizing and not for creating or measuring of qualitative goals.
> Track things quantitatively and then make qualitative decisions
Decisions are often inherently quantitative, e.g., you allocate an actual dollar amount to a budget. Or a project go/no-go is a crisp binary decision. I think you mean don't make decisions directly based on quantitative metrics but on qualitative assessments informed by quantitative metrics.
Decisions are often inherently quantitative, e.g., you allocate an actual dollar amount to a budget. Or a project go/no-go is a crisp binary decision. I think you mean don't make decisions directly based on quantitative metrics but on qualitative assessments informed by quantitative metrics.
Hmm maybe.
I don't distinguish that hard the second I get into the qualitative so many factors (like experience) inform the quantitative work.
Sometimes the qualitative goals is created because of the quantitative data.
What you are saying is certainly also what I mean but I don't think I only mean that but I see what you are saying and yeah it might just be my english.
I don't distinguish that hard the second I get into the qualitative so many factors (like experience) inform the quantitative work.
Sometimes the qualitative goals is created because of the quantitative data.
What you are saying is certainly also what I mean but I don't think I only mean that but I see what you are saying and yeah it might just be my english.
I see your point now, thanks for the clarification.
But this advice is so hard to implement. It's like the whole null hypothesis p-value debacle going on in academic publishing right now. Sure we can agree it's no good, but there's no standard alternative.
Standards are powerful. They break lots of ties. All else equal, if you have a hunch and I have a measurement tied to objective reality, people will side with me. Nobody ever got fired for choosing Java. Nobody ever got fired for doing the thing that made The Metrics go up.
It's so reassuring to just say "We still disagree, so let's just test it." We're submitting our dispute to adjudication via Science by resolving via The Metrics! It's much harder to be objective without these kinds of numbers, and I like being objective! I want the scientific method to tell me I did the right thing and I'm a good person.
Because without that, when we're trying to figure out what to do, it's just us, you know? It feels like guessing, and that's hard.