Crowdsourced Data mining, forecasting and bioinformatics via competitions(kaggle.com)
kaggle.com
Crowdsourced Data mining, forecasting and bioinformatics via competitions
http://www.kaggle.com/
16 comments
You are right -- there are some neat aspects to this model.
However, it also tends to devalue the work invested by the analysts. They are doing the work for essentially a lottery ticket -- winner take all. That's the reason that it can be so cost effective to the company running the competition -- they don't absorb the costs of the failures (or less optimal approaches).
Those costs have to be absorbed by someone. In this model, they are eaten by the analysts, who (generally) don't have enough resources to cover those costs. Due to that, I don't think this model is sustainable or will really catch on.
OTOH, in some (ideal) academic endeavors, having multiple groups compete for more funding or for a prize has certainly benefitted the sciences. In that case, however, the competition was more friendly than zero sum. Also, I believe the different sides tend to share information more than hoard it, yielding lessons learned to the entire group from one team's failures.
However, it also tends to devalue the work invested by the analysts. They are doing the work for essentially a lottery ticket -- winner take all. That's the reason that it can be so cost effective to the company running the competition -- they don't absorb the costs of the failures (or less optimal approaches).
Those costs have to be absorbed by someone. In this model, they are eaten by the analysts, who (generally) don't have enough resources to cover those costs. Due to that, I don't think this model is sustainable or will really catch on.
OTOH, in some (ideal) academic endeavors, having multiple groups compete for more funding or for a prize has certainly benefitted the sciences. In that case, however, the competition was more friendly than zero sum. Also, I believe the different sides tend to share information more than hoard it, yielding lessons learned to the entire group from one team's failures.
My experience is the opposite. I entered a number of competitions last year. For a data scientist, the opportunity to access clean data sets and interesting real-world problems, as well as to see how your techniques compare to others', is really compelling. (At least it was for me!)
I won a couple of competitions, and discovered that the competitive environment pushed me to create new algorithms and develop new ideas that I otherwise would not have. (Normally I would have thought my initial answers were "good enough" - but in the extremely competitive environment of Kaggle that is never true!)
And of course there were many more competitions I did not win. Honestly, I learnt much more from these - because by the end of the comp I knew quite a bit about the problem domain, and had tried a few ideas out, so when I then read the winning papers it gave me heaps of new ideas and insights that I could use in future projects and competitions.
I became so interested in the company that I invested in it, and then started helping out here and there, and finally joined full-time and this week have moved to SF (from Melbourne, Australia) and am now Kaggle's President and Chief Scientist.
I won a couple of competitions, and discovered that the competitive environment pushed me to create new algorithms and develop new ideas that I otherwise would not have. (Normally I would have thought my initial answers were "good enough" - but in the extremely competitive environment of Kaggle that is never true!)
And of course there were many more competitions I did not win. Honestly, I learnt much more from these - because by the end of the comp I knew quite a bit about the problem domain, and had tried a few ideas out, so when I then read the winning papers it gave me heaps of new ideas and insights that I could use in future projects and competitions.
I became so interested in the company that I invested in it, and then started helping out here and there, and finally joined full-time and this week have moved to SF (from Melbourne, Australia) and am now Kaggle's President and Chief Scientist.
It depends on what you mean by "sustainable." Is it going to create a sustainable business model that will displace the whole analyst labor/consulting market? No. A significant portion? Probably not. Will it create opportunities for firms needing analysis to get it done cheaply and for non-traditional or inexperienced analyst to show their stuff? For sure.
Might it lead to totally new ways of approaching problems that would never have been discovered otherwise? Most likely.
So it's not going to replace analyst in a niche that is already hot, but it will be a valuable component of that ecosystem.
I wish them the best of luck and I'm already thinking of ways my company could use this.
Might it lead to totally new ways of approaching problems that would never have been discovered otherwise? Most likely.
So it's not going to replace analyst in a niche that is already hot, but it will be a valuable component of that ecosystem.
I wish them the best of luck and I'm already thinking of ways my company could use this.
That's fair. I originally had the word "devalue" in the comment, but I pulled it out. Why? No one is forcing the analyst to participate, but instead, they absorb the costs to get exposure or growth (or fun). Whether that's a fair tradeoff is up to each person, but given the number of folks participating in Netflix competitions, these, and others out there, a group of smart, analytic types feel the tradeoff is worth it.
For the Netflix competition in particular, a lot of the entrants were being "paid" in research papers, grants, and academic salary, because it was a high-profile competition in a theoretically not-entirely-settled area (collaborative filtering), so even many non-winning entries could get published papers out of it. I don't think that can be infinitely replicated, in part because once there are dozens of contests, it's less likely any one will be as high profile, and in part because not every ML contest is as academically interesting (the Netflix setting was sort of "weird" from the perspective of traditional statistical models, not being a straightforward regression problem, whereas some of these are pretty straightforward).
I'm not (yet) a highly paid consultant in this area, but I still don't think it's worth my time to participate in most of these Kaggle contests. Even the $3 million prize for the healthcare prediction isn't worth it if you think you have a chance of beating current best-practice (and think you can sell your time or have a friend who can sell your time to corporations).
How does Intellectual Property work in competitions like these? Are the entrants allowed to use proprietary methods? Do they give up IP by participating? Do the hosts of competitions gain any rights to IP or its usage?
It's not immediately clear by skimming the legal terms of service. I couldn't find a FAQ.
It's not immediately clear by skimming the legal terms of service. I couldn't find a FAQ.
See section 9 of our terms: http://www.kaggle.com/pages/terms
In general, the competition host gets a license to the winning algorithm in exchange for a winner accepting the prize money.
We also have private competitions that do not appear to you unless you're invited. In these competitions hosts can offer appearance fees just for participating as well as access to more restricted data.
In general, the competition host gets a license to the winning algorithm in exchange for a winner accepting the prize money.
We also have private competitions that do not appear to you unless you're invited. In these competitions hosts can offer appearance fees just for participating as well as access to more restricted data.
Would "kaggle" be able to handle patient data? Would "kaggle" sign data use agreements with hospitals that are interested in a shared task? There is a growing number of medical data mining competitions, e.g.:
https://www.i2b2.org/NLP/Coreference/PreviousChallenges.php
But the data mining challenge delivery systems in medicine are scattered. Mostly because of inability to create a secure and centralized web service.
https://www.i2b2.org/NLP/Coreference/PreviousChallenges.php
But the data mining challenge delivery systems in medicine are scattered. Mostly because of inability to create a secure and centralized web service.
Very interesting project! Can anyone recommend some good data mining for dummies tutorials/books/etc?
I recommend the video at http://blog.kaggle.com/2011/03/23/getting-in-shape-for-the-s...
I gave more recommendations at http://stackoverflow.com/questions/598726/overwhelmed-by-mac...
(Disclaimer: I work at Kaggle)
EDIT: Fixed second link
I gave more recommendations at http://stackoverflow.com/questions/598726/overwhelmed-by-mac...
(Disclaimer: I work at Kaggle)
EDIT: Fixed second link
Sweet, thanks! I would very much appreciate if you guys could somehow motivate the great data mining talent on your site to produce tutorials. I would be even willing to pay there for some training course because traditional tutorial providers (Lynda etc) have apparently not yet discovered this as a potential subject.
It's something we're interested in doing. I think Jeremy's video (that I linked to first) is a good start.
In addition, by working in the field, I think I'm in a better position to write blog tutorials like my TrueSkill one: http://www.moserware.com/2010/03/computing-your-skill.html
It's our hope that we can encourage a lot of our community to post insights of their approaches. That's what we already do on our blog with our "How I Did It" posts like http://blog.kaggle.com/2011/10/19/deceitful-beast-william-cu...
In addition, by working in the field, I think I'm in a better position to write blog tutorials like my TrueSkill one: http://www.moserware.com/2010/03/computing-your-skill.html
It's our hope that we can encourage a lot of our community to post insights of their approaches. That's what we already do on our blog with our "How I Did It" posts like http://blog.kaggle.com/2011/10/19/deceitful-beast-william-cu...
This is great. I'm a practicing medical doctor and a coder. My friend and I are going to give the healthcare one a go. If anything it's good fun and we'll learn more about the real world application of data mining to medicine. Thanks for the links.
From a stats-heavy perspective, Hastie/Tibshirani/Friedman's Elements of Statistical Learning is pretty good, and now free: http://www-stat.stanford.edu/~tibs/ElemStatLearn/
Similar to Mechanical Turk, we've managed to create a completely different value structure for some amazing work by smart folks... mostly by making it a competition. Great exposure for winners, sure, but these prizes are pretty minimal.
http://www.kaggle.com/c/GiveMeSomeCredit, for example, has a total basket of US$5K (only US$3K for first place) for a model predicting credit scores (in this case, likelihood to default or have financial distress). Folks I talk to who do this type of work professionally tend to charge far more than that to create these models.
For the company sharing this data, of course, big win: They get a cheap, potentially fantastic new model, and the creator gets some good exposure and some cash. But if these take off, they can really change the economics of how this work is created.