Ask HN: Survivor Bias. Why don't we focus more on startups that fail?
21 comments
The National WWII Museum implies that your WWII anecdote is the inverse of what actually happened. That is, that returning planes were shot in the fuselage, and a WWII statistician recognized survivorship bias in proposed reinforcement and concluded that the areas not hit on survivors should be reinforced.
See: http://www.nww2m.com/2012/11/scitech-tuesday-abraham-wald-se...
http://en.wikipedia.org/wiki/Abraham_Wald
See: http://www.nww2m.com/2012/11/scitech-tuesday-abraham-wald-se...
http://en.wikipedia.org/wiki/Abraham_Wald
Ah. Smart statistician. Thanks for that.
We pay a lot of attention to the causes of failure. Much of what I've written about startups is about pitfalls and mistakes. I don't name names, but I don't think you need to.
Hmm yea that's a good point. I guess my thought was that founders are almost unilaterally interested in learning from successes over failures. For example, Startup School doesn't consist of a bunch of speakers who have only failed at starting startups because not many people would want to hear from Joe-Schmoe who failed. Maybe it has some who failed once or twice and then succeeded, but they still suffer from survivor bias.
hi pg.
i don't know you, this website, or anything about anything, but i felt a duty to inform a fellow internet user that someone has commissioned others to compile your comments for this site. here's the link: https://www.elance.com/j/scrape-gather-all-comments-person-o...
it could be totally fine, or creepy, or it could just be you wanting your stuff compiled, but i felt it my duty to tell a fellow internet user that someone wanted their stuff compiled.
if i'm breaking any sort of rules, again, sorry, i don't know anything about this site or how to navigate it; i also obviously don't know how to pm.
cheers.
i don't know you, this website, or anything about anything, but i felt a duty to inform a fellow internet user that someone has commissioned others to compile your comments for this site. here's the link: https://www.elance.com/j/scrape-gather-all-comments-person-o...
it could be totally fine, or creepy, or it could just be you wanting your stuff compiled, but i felt it my duty to tell a fellow internet user that someone wanted their stuff compiled.
if i'm breaking any sort of rules, again, sorry, i don't know anything about this site or how to navigate it; i also obviously don't know how to pm.
cheers.
google for "paul graham". That's pg
There are an infinite number of ways to fail and only a few ways to succeed. If you learn from failure, you have learned one way that doesn't work out of an infinite number. If you learn from success, you have learned one way that does work out of a small finite number.
Obviously there are some ways to fail that are much more common than others, but those tend to be based on lack of action, e.g. failure to actually ask users if they want to use your product before building it. There is plenty of material on those common sorts of failures, it's just usually phrased constructively, i.e. "how to do SEO" rather than "we died because we posted duplicate content on every page of our site."
Obviously there are some ways to fail that are much more common than others, but those tend to be based on lack of action, e.g. failure to actually ask users if they want to use your product before building it. There is plenty of material on those common sorts of failures, it's just usually phrased constructively, i.e. "how to do SEO" rather than "we died because we posted duplicate content on every page of our site."
There are an infinite number of ways to succeed and an infinite number of ways to fail.
Think of the raven paradox (look it up on wikipedia). Finding something that isn't black and isn't a raven should be evidence that all ravens are black.
Why isn't it? Because there are an infinite number of things that aren't black, and a finite number of ravens. You could actually count all of the ravens in the world but you can't count all of the things in the world that aren't black.
Startups are not like this. Failures compare to successes at a rate of, say, 8 out of 10. Both classes are of the same order of magnitude. You need to examine both to find discriminating variables.
Think of the raven paradox (look it up on wikipedia). Finding something that isn't black and isn't a raven should be evidence that all ravens are black.
Why isn't it? Because there are an infinite number of things that aren't black, and a finite number of ravens. You could actually count all of the ravens in the world but you can't count all of the things in the world that aren't black.
Startups are not like this. Failures compare to successes at a rate of, say, 8 out of 10. Both classes are of the same order of magnitude. You need to examine both to find discriminating variables.
The hard part is pinning down the cause of a successful startup. Most people just point at highly visible things, and make a claim like "StartupX is successful because the founders worked extremely hard, the office culture was well-developed, and there were lots of team building activities." The problem is that this ignores the 5,000 other startups that did all those same things, but failed. Perhaps it turns out that StartupX really succeeded because they had a sales guy with lots of good connections, etc.
You have a good point here.
More prevalent in the financial industry, you have a lot of successful retirees who hold talks and write books what kind of tea they drank. Statistically nobody beats the market, so these people had very likely just luck.
Causation is not relation, everybody knows that. But it is damn hard to figure out what and often the people themselves are biased. I would say a big percentage of tips you get this way are totally useless, but to figure out which one of them depends on your judgement.
More prevalent in the financial industry, you have a lot of successful retirees who hold talks and write books what kind of tea they drank. Statistically nobody beats the market, so these people had very likely just luck.
Causation is not relation, everybody knows that. But it is damn hard to figure out what and often the people themselves are biased. I would say a big percentage of tips you get this way are totally useless, but to figure out which one of them depends on your judgement.
I wrote up my failed YC startup here: http://blog.paulbiggar.com/archive/why-we-shut-newstilt-down...
It was definitely a good move. More people read that blog post than heard about my startup in the first place, it got covered in loads of tech press, and people still ask me about it 2 years later. I get an email every few months thanking me for writing it, and I'm asked to do the occasional interview about it too.
It also allowed me to introspect a lot of what went wrong, and what I could have done better. One of the major reasons my current startup (https://circleci.com) is doing well is what I learned from the deconstruction.
It was definitely a good move. More people read that blog post than heard about my startup in the first place, it got covered in loads of tech press, and people still ask me about it 2 years later. I get an email every few months thanking me for writing it, and I'm asked to do the occasional interview about it too.
It also allowed me to introspect a lot of what went wrong, and what I could have done better. One of the major reasons my current startup (https://circleci.com) is doing well is what I learned from the deconstruction.
Just wanted to echo common sentiments that your blog post was a very well-written and yet personal treatise on how and why startups fail.
Another good pair of posts comes from Chad Etzel (previously Notifo): http://blog.jazzychad.net/2011/05/02/startups-are-hard.html
http://blog.jazzychad.net/2011/09/08/fouling-out-moving-on.h...
Another good pair of posts comes from Chad Etzel (previously Notifo): http://blog.jazzychad.net/2011/05/02/startups-are-hard.html
http://blog.jazzychad.net/2011/09/08/fouling-out-moving-on.h...
When I joined the startup scene 3 years ago, I mostly saw success stories. Now that I've been through my own failed startup, I see many failure stories.
I believe that two things have changed:
1. The field is maturing, so more people are talking about failures, the downsides of entrepreneurship, etc.
2. I'm growing as a business person, so I'm reading a larger variety of things, not just stories that reinforce my beliefs.
So it's possible that, like me, you're so focused on reading positive startup-reinforcing articles, that you're missing out on some of the articles which focus more on the failures and things to avoid.
Oh, and definitely read pg's articles - they have a lot of truth in them, compiled from one smart person who's seen many startups both fail and succeed, and who is specifically looking for the reasons and patterns behind it all.
I believe that two things have changed:
1. The field is maturing, so more people are talking about failures, the downsides of entrepreneurship, etc.
2. I'm growing as a business person, so I'm reading a larger variety of things, not just stories that reinforce my beliefs.
So it's possible that, like me, you're so focused on reading positive startup-reinforcing articles, that you're missing out on some of the articles which focus more on the failures and things to avoid.
Oh, and definitely read pg's articles - they have a lot of truth in them, compiled from one smart person who's seen many startups both fail and succeed, and who is specifically looking for the reasons and patterns behind it all.
I think there's two major components to this:
1) Writing about your failures sucks. My first startup got to 6 figure revenue within 18 months, but we really struggled to grow past that. I could write a full debrief on why we failed, but it's pretty low on my "things I want to do" list. I'd imagine quite a few founders with failed startups would have a similar viewpoint.
2) A lot of failures just aren't that exciting. I'd venture a guess that most startups fail because they gradually lose their passion for the problem. When you aren't growing for an extended time, it wears on you. A lot. The only way you survive is if you're obsessed with the problem you're solving. Those that aren't would just slowly fade out.
1) Writing about your failures sucks. My first startup got to 6 figure revenue within 18 months, but we really struggled to grow past that. I could write a full debrief on why we failed, but it's pretty low on my "things I want to do" list. I'd imagine quite a few founders with failed startups would have a similar viewpoint.
2) A lot of failures just aren't that exciting. I'd venture a guess that most startups fail because they gradually lose their passion for the problem. When you aren't growing for an extended time, it wears on you. A lot. The only way you survive is if you're obsessed with the problem you're solving. Those that aren't would just slowly fade out.
I'd note that this bias extends beyond startups; see http://jeps.efpsa.org/blog/2012/06/01/falsification-of-previ...
studies of failures don't just reveal things that increase failure rates, they prevent false positives.
Imagine X is some innovation that is being talked up because recently several companies had or used X to succeed.
If 19 out of 20 times X makes a startup more likely to fail or if X has no meaningful effect on failure rates at all, or if X only helps in situation Y, but you only hear about the times that people succeeded with X, you might try to use X, to no benefit or to actual harm to your business.
If startup success is rare, then trying to generalize only from instances of it may lead to errors. We have more instances of failures, therefore we can learn a great deal about commonalities in startups that fail. Of course the smart thing to do is to make a hypothesis about X, and compare startups with X that failed and succeeded to startups without X that failed and succeeded to see if there is a significant effect on the rate.
I'd note that errors of judgment about what causes startups to do well or fail get fed into the investor system. If X is useless or harmful, but your investors think it's a good idea, you will have problems.
studies of failures don't just reveal things that increase failure rates, they prevent false positives.
Imagine X is some innovation that is being talked up because recently several companies had or used X to succeed.
If 19 out of 20 times X makes a startup more likely to fail or if X has no meaningful effect on failure rates at all, or if X only helps in situation Y, but you only hear about the times that people succeeded with X, you might try to use X, to no benefit or to actual harm to your business.
If startup success is rare, then trying to generalize only from instances of it may lead to errors. We have more instances of failures, therefore we can learn a great deal about commonalities in startups that fail. Of course the smart thing to do is to make a hypothesis about X, and compare startups with X that failed and succeeded to startups without X that failed and succeeded to see if there is a significant effect on the rate.
I'd note that errors of judgment about what causes startups to do well or fail get fed into the investor system. If X is useless or harmful, but your investors think it's a good idea, you will have problems.
I think the answer is pretty simple: failed startps are insular. Founders who are able to discuss them won't, either because they've moved on, or because it's painful, embarrassing etc. Discussing failures would be more useful, I imagine, than reading about successes. If such a startup graveyard existed, I'd probably visit quite often.
- The failure may be a sore-spot for those involved, so they might not want to do a postmortem.
- Unless insiders are willing to speak up, there may not be too much to be said from the outside.
- A lot of start-ups that fail are uninteresting. What can be said about the tons of TODO web apps that have probably been created and failed.
- Those involved may be reluctant to speak out because they view the lessons-learned as their leg up on the competition when they get into their next start-up.
I agree that there are a lot of interesting things to discuss regarding failed start-ups, though.
- Unless insiders are willing to speak up, there may not be too much to be said from the outside.
- A lot of start-ups that fail are uninteresting. What can be said about the tons of TODO web apps that have probably been created and failed.
- Those involved may be reluctant to speak out because they view the lessons-learned as their leg up on the competition when they get into their next start-up.
I agree that there are a lot of interesting things to discuss regarding failed start-ups, though.
I think on a person-to-person basis there's a lot of talk about startups and companies that have failed. People who have worked with startups that ended up going under have no discomfort talking about the causes and errors, which subsequent companies learn from. I think the experiences and lessons are passed on fairly well, it's just that this exchange happens out of the public spotlight.
For those interested in a scholarly study of common failure modes for various fashionable Grand Business Strategies of the 80s and 90s, read Billion Dollar Lessons by Carroll and Mui: http://www.amazon.com/Billion-Dollar-Lessons-Inexcusable-Bus...
Good point. I was thinking that a lot of the characteristics of great entrepreneurs are really just things that make them more volatile. So for a % of the people they will be more successful, but for another percentage of the people with the same characteristics they will be a lot less successful.
I have no evidence to back this up, just food for thought.
I have no evidence to back this up, just food for thought.
I think another reason is that no one likes to acknowledge the uncomfortable truth that a significant factor in the success or failure of tech startups is pure chance. It's the same reason so much of recorded history is false. It seems to be hardwired into human experience to prefer a good narrative over truth.
Once you've failed (and maybe, given your post-mortem), what else is there to do? You move on to working on something else.
To extend your analogy: during WW2, they didn't study the same plane every time a plane was shot down. They studied a the one that was hit, and moved on to the next one after that.
To extend your analogy: during WW2, they didn't study the same plane every time a plane was shot down. They studied a the one that was hit, and moved on to the next one after that.
Your missing the point of the analogy. The planes that should have been studied were the same planes that never RTB'ed so they could not be studied.
There is a notable exception: http://thefailcon.com/ "FailCon is a one-day conference for technology entrepreneurs, investors, developers and designers to study their own and others' failures and prepare for success."
Availability heuristic? Successful startups survive, do well and people hear about them more (news coverage, PR, using their products etc.) It is hard to discuss a company when you've never even heard of them.
In my case, NDAs, mostly. Plus, as a savvy recruiter once told me, the Bay Area is big, but the Valley is small.
Agreed. Probably an equal number of lessons to learn from the failures as from those still slogging it out.
Failure is bad for SEO. Unless the failure is massive.
Many startups fail for uninteresting reasons.
Many startups fail for uninteresting reasons.
Well it is my opinion that failure stories are far better than successful stories because success is difficult to replicate but failures can be easily avoided.
same reason we like heroes so much, even though most heroes didn't do things on their own.
I agree. It makes for a lot more interesting story too.
During WWII planes that made it back from combat missions were analyzed to be reinforced. They found that these planes were shot in the wings and tail so they reinforced those areas. Really though, planes shot in the fuselage were much less likely to make it back so they should have reinforced that area. They were biased by trying to avoid the problems the survivors faced.
So my question is: why don't we try to learn more from startups that fail instead of focusing so much on the successes?