1. The models predicted survival, and the business survived
2. Predicted survival, but the business failed
3. Predicted failure, but the business survived,
4. Predicted failure, and the business failed
Here's how our results stacked up:
#1. 69% of observations (it wobbles around 66%, but was 69% in the last update)
#2. 31% of observations (notice #1 + #2 = 100%. Out of 100% of the times we predicted survival, we were right 69% and wrong 31%)
#3. 12%
#4. 88% (again, #3 + #4 = 100%. Out of all the times we predicted failure, we were right 88% of the time and wrong 12%).
Again, there's more luck when you predict failure, so it's really around 66% for both positives and negatives when you dig deeper into the stats.
Hi Y'all, Thomas here (guy in article). Just want to start by saying (1) this is a very intelligent thread, and (2) I didn't write the article, was just interviewed for it. You never know what's going to be written, no matter what you say.
Here's how the models really play out.
We compare our accuracy against the 10 year survivorship benchmark of 25% (not the 5 year). When you look at small businesses, VC-backed, and corporate ventures (ex. new products coming out of companies), the 10 year survival rate is around 25%, plus or minus 10% depending on the industry.
Our models have made thousands of predictions for around nine years now - all the predictions were live, real-time and forward looking (no back-testing included here). From those predictions, around 3,400 have matured to date. That is, only around 3,400 of the results have happened - the businesses have either become big successes (ex. Uber) or failed. In our research, we have to actually wait for businesses to live or die to test our accuracy.
From the roughly 3,400 predictions that have matured, we were right 66% of the time when predicting survivors, and 88% of the time when predicting failures. When we scratched beneath the surface, we were really around 66% accurate in both cases (just most businesses fail, which is why gloomy predictions were 22% more accurate - just a function of dumb luck since most things die).
So we consider our algorithms to be 66% accurate, which is much more accurate than anything we're aware of in human history (remember, the baseline we're compared against is 25%).
If you do a statistical analysis (to make sure our predictions weren't just luck), the models maintained a statistically significant correlation with 99% confidence. There was less than 1 chance in over 500,000 that the results were a function of luck (definitely not a coin toss).
We've used these models in venture, and our performance puts us in the top 5% of all VC funds for our vintage years, so we've monetized these models effectively with real dollars and made considerable gains.
I hope this gives folks a better sense for how it works. There's been a very emotional backlash to the Wired article today (not accusing this thread, just thinking of some others) and it's weird because it's just basic scientific research. Pretty drab stuff on most days, but apparently offensive to some people. Not sure why. We're using statistics to improve venture and startup odds, just like stats have been used to improve just about every other field humans have ever taken seriously. Seems obvious that stats are similarly useful in the startup world, and my dream has always been to help more businesses use stats to succeed.
Anyway, definitely a lot more controversy and emotion than I would have expected. Otherwise pretty basic science, not claiming perfection, just striving for improvement, using data as best we can, etc. I hope at least some folks see this for what it is - nothing out of the ordinary in any other domain of science. Why should entrepreneurship be any different?
Possible outcomes:
1. The models predicted survival, and the business survived 2. Predicted survival, but the business failed 3. Predicted failure, but the business survived, 4. Predicted failure, and the business failed
Here's how our results stacked up: #1. 69% of observations (it wobbles around 66%, but was 69% in the last update) #2. 31% of observations (notice #1 + #2 = 100%. Out of 100% of the times we predicted survival, we were right 69% and wrong 31%) #3. 12% #4. 88% (again, #3 + #4 = 100%. Out of all the times we predicted failure, we were right 88% of the time and wrong 12%).
Again, there's more luck when you predict failure, so it's really around 66% for both positives and negatives when you dig deeper into the stats.