Business Advice Plagued by Survivor Bias(blog.asmartbear.com)
blog.asmartbear.com
Business Advice Plagued by Survivor Bias
http://blog.asmartbear.com/blog/business-advice-plagued-by-survivor-bias.html
13 comments
I wish it was still on the front page. Understanding bias is, in my book, key, key, key to an effective understanding (and navigation) of reality.
It's one reason to pursue 'smaller' opportunities with lower variance of return expectations but higher median returns -- for example many niche businesses -- than industries with gigantic return expectations.
This neatly encapsulates much of my current business strategy.
This neatly encapsulates much of my current business strategy.
Warren Buffett talks about this in stock investing: you're more likely to find a bargain in the under-researched companies, because there's greater variance between the stock price and the stock value (price inefficiencies).
He's publicly claimed to be able to double a million dollars in a year with this approach (it won't work for his $40+ billion, because there aren't enough under-researched stocks to buy that are also bargains).
He's publicly claimed to be able to double a million dollars in a year with this approach (it won't work for his $40+ billion, because there aren't enough under-researched stocks to buy that are also bargains).
Do you read business blogs where the author has failed three times without success?
Am I the only one who regularly reads blog posts by an entrepreneur analysing how he's failed and what he would do differently next time? They pop up on HN quite often.
Am I the only one who regularly reads blog posts by an entrepreneur analysing how he's failed and what he would do differently next time? They pop up on HN quite often.
Are you talking about www.thefailingpoint.com? That's me, and I am happy to share my experiences. They have been in the massive up and down times. Success and failure. I find it therapeutic to write, and since my old engineering team is in Y!Combinator this summer, I got the religion about sharing what I know. They asked me for a compendium of all the stupid mistakes I had made so that they wouldn't make them. =) I joked it was too much work, but it turns out that it makes for some good essays, and I really hope to turn it into a published work. For now, I am posting all the essays online for all to read for free. Chapter 1 and 2 are almost fully posted. Please send feedback.
Yes and I love reading those... They are usually much more interesting and insightful
I admit an amount of hyperbole when I implied that you NEVER read such blogs, and of course it's good that you do!
In general I think it's fair to say that the more popular blogs and books about startups talk about successes rather than failures, so I'll defend my general point.
I did take some license in how I stated it, and thanks for pointing out that it's too harsh and that there ARE good sources of info like that.
In general I think it's fair to say that the more popular blogs and books about startups talk about successes rather than failures, so I'll defend my general point.
I did take some license in how I stated it, and thanks for pointing out that it's too harsh and that there ARE good sources of info like that.
Very Nassim Taleb. These are great books NT's written that discuss our susceptibility to survivorship bias and other rational failings we have: Fooled by Randomness and The Black Swan.
Yes! You're right I'm hardly the first people to talk about such things. The Black Swan is a classic although oddly enough right now on the front page of HN there's a critique of Black Swan that pokes holes in it. :-)
But yes it's true there's lots of good material on these subjects for those who want to explore further. Hopefully my piece whetted your appetite -- it's not the final word!
But yes it's true there's lots of good material on these subjects for those who want to explore further. Hopefully my piece whetted your appetite -- it's not the final word!
This article makes everything pg says about the reasons behind Viaweb's success sound rather suspect, doesn't it?
Pretty much every startup success story has an "and then we got spectacularly lucky" part to it. In Viaweb's case, it's more like "and then, for the nth time, we almost died, then somehow didn't", but the idea is the same. It's one of the great unmentionables in the startup world, and understandably so: succeed and you want the credit; fail and any mention of luck sounds like sour grapes.
I've heard a lot of startup founders talk about a lot of startups, and only Marc Andreessen (of Netscape fame) both realized and admitted to the large role luck nearly always plays.
I've heard a lot of startup founders talk about a lot of startups, and only Marc Andreessen (of Netscape fame) both realized and admitted to the large role luck nearly always plays.
I should add that great startup success stories almost always involve founders who were awesome as well. (Netscape and Viaweb both qualify on this count.) It's just that being awesome isn't enough. Traf-O-Data had awesome founders, but it failed; those same founders, Bill Gates and Paul Allen, had better luck in their second venture.
Wasn't it always suspect?
It depends of what you consider suspect. I always saw PG's account of the Viaweb days in the light of the .COM 1.0 bubble.
Its funny - but for so many startup success stories, bubbles are the pink elephant in the room.
When you try to draw lessons from success during downturns, the lessons are very different, and yet most thought leaders come from bubbles. This deeply effects the quality of the advice out there - most of it comes from people who are to some extent unknowing products of bubbles - but who attribute all of their success to actions they took that have universal applicability. When it often ain't so.
For this reason, for me, 'startup credibility' goes up tremendously when it comes from a serial entrepreneur, or from someone that has actually built a cash-starved company to success during a downturn. "What can I do with very little?" and "What can I do if it is possible to raise $1 million?" are fundamentally different questions - and the same person isn't necessarily going to be skilled in both situations unless they've demonstrated that ability.
Whereas this comment - it has the legitimacy of coming from a complete failure :D
Finally... if your community leaders are state employees that can't say anything negative in public... you really need to be aware of that before you take their optimism at face value. Take to time to query them in private, to find out what they actually think. It might save you some mistakes and some grief.
When you try to draw lessons from success during downturns, the lessons are very different, and yet most thought leaders come from bubbles. This deeply effects the quality of the advice out there - most of it comes from people who are to some extent unknowing products of bubbles - but who attribute all of their success to actions they took that have universal applicability. When it often ain't so.
For this reason, for me, 'startup credibility' goes up tremendously when it comes from a serial entrepreneur, or from someone that has actually built a cash-starved company to success during a downturn. "What can I do with very little?" and "What can I do if it is possible to raise $1 million?" are fundamentally different questions - and the same person isn't necessarily going to be skilled in both situations unless they've demonstrated that ability.
Whereas this comment - it has the legitimacy of coming from a complete failure :D
Finally... if your community leaders are state employees that can't say anything negative in public... you really need to be aware of that before you take their optimism at face value. Take to time to query them in private, to find out what they actually think. It might save you some mistakes and some grief.
Oddly, self-help gurus seem to enjoy the opposite phenomenon (i.e. those who can't, teach self-help seminars).
I remember at one point this guy Richard(?) Kiyosaki had a book "Rich Dad, Poor Dad." In it, you basically found out that Kiyosaki was (a) lying his tits off or (b) a terrible investor. You found out that he'd been bankrupt, and had run several companies into the ground. That he invested in mining companies in Russia whose value would swing 50% up and down from day-to-day - but that this didn't bother him... Wacko.
Yet, somehow, the book was a total phenomenon and made the guy huge amounts of money.
I remember at one point this guy Richard(?) Kiyosaki had a book "Rich Dad, Poor Dad." In it, you basically found out that Kiyosaki was (a) lying his tits off or (b) a terrible investor. You found out that he'd been bankrupt, and had run several companies into the ground. That he invested in mining companies in Russia whose value would swing 50% up and down from day-to-day - but that this didn't bother him... Wacko.
Yet, somehow, the book was a total phenomenon and made the guy huge amounts of money.
I agree there's too much advice from people who haven't been there themselves.
My own article shows that advice from people who HAVE been there (like me) is still suspect, but at least it's MORE interesting than people who have ONLY failed.
Failing, learning, and then succeeding just feels like a better place than just failing, SAYING you learned, and then teaching.
My own article shows that advice from people who HAVE been there (like me) is still suspect, but at least it's MORE interesting than people who have ONLY failed.
Failing, learning, and then succeeding just feels like a better place than just failing, SAYING you learned, and then teaching.
I bet success does feel better than failure. And not because you get more credibility, either :)
At least he failed the first time (websites for galleries), and learnt from it. There was also a control in the form of other similar businesses at the time.
Doesn't it make you less certain of the reasons anyone gives for any success?
The article quotes Peter Norvig:
When a published paper proclaims "statistically, this could only happen by chance one in twenty times," it is quite possible that similar experiments have been performed twenty times, but have not been published.
It occurred to me that this means there is a real advantage to academia's "publish or perish" phenomenon. It means that when an experiment's results are not what was hoped, those results are still likely to get published. The experimenter would like to have his name on some ground-breaking results, of course, but, failing that, he still must have publications. So he publishes anyway. Not so true in industry.
Are you sure about that? Journals don't (in general) publish negative results. The result of the pressure to publish is not that negative results get published; it's that the scientist has an incentive to create positive results. Not necessarily through data falsification, but through choosing experiments very likely to generate "results", which, unfortunately, also implies that the information content of such experiments is necessarily lower than it could be if the scientist had more freedom to perform an experiment that had a higher likelihood of failure.
Googling "journal of negative results" gives a few hits. I think it's probably possible to get negative results published, but I doubt that they look good on the resume.
The takeaway here should be: if the book doesn't systematically examine the differences between winners and losers, drop it and walk away. Sentence by sentence, it's making you dumber.
So forget In Search of Excellence. Try books like How To Be a Star at Work, which is based on a pretty rigorous study of employee productivity.
So forget In Search of Excellence. Try books like How To Be a Star at Work, which is based on a pretty rigorous study of employee productivity.
Talk about survivor bias: http://www.dailymail.co.uk/news/article-1165768/The-man-surv...
Jumping on books marketed at airports to harried executives isn't really valid. There's plenty of actual business research done by professionals (and they've heard of "survivorship bias, along with all the other threats to validity). It's like looking at all the "For Dummies" books in your local book store's computer section and deciding that CS research is poor.
And Harvard Business School Case Studies are meant for teaching, and their function is analogous to science textbooks that leave out all the dead ends and fruitless research for pedagogical reasons.
And Harvard Business School Case Studies are meant for teaching, and their function is analogous to science textbooks that leave out all the dead ends and fruitless research for pedagogical reasons.
"Planes with bullets in the cockpit or fuel tanks didn't make it home"
Such a brilliant piece of lateral thinking - thanks for sharing.
Such a brilliant piece of lateral thinking - thanks for sharing.
This is a good point. At the same time, I believe there are actual differences between businesses that affect their chance of success.
So the trick is to separate the 'accidental' features of successful businesses that don't increase their chances of success from the 'essential' features that do.
I wish the article went into how one might do this...
So the trick is to separate the 'accidental' features of successful businesses that don't increase their chances of success from the 'essential' features that do.
I wish the article went into how one might do this...
One important way referenced in the article is to evaluate failures as well as successes for the likelihood of a trait: for example Jim Collins describes certain features that successful companies have, but does not evaluate the pool of losers to see if they also featured the same trait.
Another example is when evaluating various corporate traits against stock market performances, many people will choose a list of companies that exist today and then study their history to identify correlations of various traits. This omits those that no longer exist today, and it is a common problem among, for example, beginning hedge fund equity analysts. Instead, one should study the cohort from a random sample as existed at the beginning of the study, not at the end, and then study their performances from that point forward.
Another example is when evaluating various corporate traits against stock market performances, many people will choose a list of companies that exist today and then study their history to identify correlations of various traits. This omits those that no longer exist today, and it is a common problem among, for example, beginning hedge fund equity analysts. Instead, one should study the cohort from a random sample as existed at the beginning of the study, not at the end, and then study their performances from that point forward.
Agreed with mediaman here.
Select both successes AND failures. Where they seem similar, those things may not teach us much. Where there is a pattern shown ONLY IN ONE SIDE (more or less) there might be a lesson.
I have a fear that there's isn't a lot you can do to guarantee success, but I admit this is just my feeling, not based on any facts or stats.
Select both successes AND failures. Where they seem similar, those things may not teach us much. Where there is a pattern shown ONLY IN ONE SIDE (more or less) there might be a lesson.
I have a fear that there's isn't a lot you can do to guarantee success, but I admit this is just my feeling, not based on any facts or stats.
The winners write the histories.
This was a great read. I guess I have underestimated the intelligence of bears!
It's one reason to pursue 'smaller' opportunities with lower variance of return expectations but higher median returns -- for example many niche businesses -- than industries with gigantic return expectations. The latter attracts far more entrants, and while somebody gets rich, the median player does not. But this isn't clear to new entrants because of survivorship bias.
Somewhat related to this is the data availability bias. Big opportunities tend to have much better data sets, and so are easier to study and receive more attention. Many niche industries are closely private and are smaller, so do not have much publicly available data, making study difficult and attracting less attention. Therefore like moths to light bigger industries receive more study and more entrants even though their median expected economics are worse.