Pragmatic Theory, Commendo, Yahoo and AT&T hit the Netflix grand prize(netflixprize.com)
netflixprize.com
Pragmatic Theory, Commendo, Yahoo and AT&T hit the Netflix grand prize
http://www.netflixprize.com/leaderboard?att
31 comments
I believe you are correct:
http://www.netflixprize.com//rules
To qualify for the Grand Prize the RMSE of a Participant’s submitted predictions on the test subset must be less than or equal to 90% of 0.9525, or 0.8572 (the "qualifying RMSE"). After three (3) months have elapsed from the start of the Contest, when the RMSE of a submitted prediction set on the quiz subset improves beyond the qualifying RMSE an electronic announcement will inform all registered Participants that they have thirty (30) days to submit additional candidate prediction sets to be considered for judging.
http://www.netflixprize.com//rules
To qualify for the Grand Prize the RMSE of a Participant’s submitted predictions on the test subset must be less than or equal to 90% of 0.9525, or 0.8572 (the "qualifying RMSE"). After three (3) months have elapsed from the start of the Contest, when the RMSE of a submitted prediction set on the quiz subset improves beyond the qualifying RMSE an electronic announcement will inform all registered Participants that they have thirty (30) days to submit additional candidate prediction sets to be considered for judging.
[deleted]
AT&T confirms their accomplishment here:
http://www.research.att.com/~volinsky/netflix/bpc.html
This starts a 30 day period during which all the teams can submit their final versions. After that, I believe Netflix will run the algorithms against the unreleased set of movie rating data.
This starts a 30 day period during which all the teams can submit their final versions. After that, I believe Netflix will run the algorithms against the unreleased set of movie rating data.
Since the current teams were using the squared error ratings on the standard test data (not this new test data) to combine their old algorithms, it's quite possible that in one month we will learn that the prize has not yet been won. As I understand the rules, anyway.
No. from the rules -
"The RMSE for the first "quiz" subset will be reported publicly on the Site; the RMSE for the second "test" subset will not be reported publicly but will be employed to qualify a submission as described below. The reported RMSE scores on the quiz subset provide a public announcement that a potential qualifying score has been reached and provide feedback to Participants on both their absolute and relative performance."
The final call announcement would not go out unless a team beat 10% on the test set. That announcement went out at 4:30pm PST today.
It is still quite possible that the entries can be rejected if the reviewers find the system description lacking or if we are unable to reproduce their result. That clock starts after the 30 days runs out.
The final call announcement would not go out unless a team beat 10% on the test set. That announcement went out at 4:30pm PST today.
It is still quite possible that the entries can be rejected if the reviewers find the system description lacking or if we are unable to reproduce their result. That clock starts after the 30 days runs out.
I'd love to see if their work stands up to all the new ratings data generated by netflix since the prize was announced.
Is it possible that there is over-fitting going on?
Is it possible that there is over-fitting going on?
(disclaimer: I work on the cinematch team)
it's rather important to us that the solutions aren't over fit so we've worked very hard to make it hard if not impossible to do.
it's rather important to us that the solutions aren't over fit so we've worked very hard to make it hard if not impossible to do.
1) There's not enough information returned by the submission process (a single RMSE number per day).
2) The information you do get is on half of your submission, but it is the other half that counts. So you get no information back on that.
[deleted]
I forgot to answer your first question. We released a new prediction engine a few months ago (http://blog.netflix.com/2009/05/more-accurate-star-predictio...)...
I think it's safe to assume that we wouldn't have used new algorithm(s) if they didn't work well on our ~3B ratings.
I think it's safe to assume that we wouldn't have used new algorithm(s) if they didn't work well on our ~3B ratings.
The only candidate I think to make this a surprising finish is the 'Grand Prize Team' which has another of the long-time leaders 'Gravity' involved and has structured itself to accept any and all team members who care to join.
Probably no one would have won this contest if it wasn't for the fact that many of the top teams published their results, and so there was cross-pollination of ideas. Pragmatic Theory didn't contribute in this manner. Kudos to them for making I think a significant improvement on the best of the published models however.
Probably no one would have won this contest if it wasn't for the fact that many of the top teams published their results, and so there was cross-pollination of ideas. Pragmatic Theory didn't contribute in this manner. Kudos to them for making I think a significant improvement on the best of the published models however.
I hope someone will take the time to code an open source version of the published algorithm, or something close to it!
Netflix gets a non-exclusive license from the winner, so they can sell it to other parties, and it seems like something that various parties would be willing to pay for.
An open source version would be a significant act of charity. It would be nice to see older versions or versions that were crippled enough to be education without watering down the reward that the winnners receive.
An open source version would be a significant act of charity. It would be nice to see older versions or versions that were crippled enough to be education without watering down the reward that the winnners receive.
Algorithms have already been published for the Progress Prizes for 2007 and 2008.
Open sources projects have already been created in Python, Java, C, and C++. None of these projects include the full algorithms from the 2007 or 2008 papers.
The projects can be found from links on the Netflixprize forum:
http://netflixprize.com
Open sources projects have already been created in Python, Java, C, and C++. None of these projects include the full algorithms from the 2007 or 2008 papers.
The projects can be found from links on the Netflixprize forum:
http://netflixprize.com
Its not just AT&T, its 4 groups working together including Commendo(http://www.commendo.at/), Pragmatic Theory(http://pragmatictheory.googlepages.com/), Yahoo Research(http://research.yahoo.com/), and AT&T Statistics Research Team(http://www.research.att.com/%7Estat)
Agreed. PragmaticTheory team deserves a special commendation since last month they developed an enhancement that was included into the winner's submission and pushed it into the #1 position. Read about PT on Wired: http://www.wired.com/epicenter/2009/06/1-million-netflix-pri...
You're right, I tried to update the headline but it wouldn't take -- sorry about that
Did anyone here take a crack at this? I had some ideas that I wanted to try this summer but now its too late. If anyone already has a model built and made submissions using the KNN and SVD methods perhaps we can talk. I thought of a different approach that was not discussed in any of the papers, forums or websites that I read. I know its a long shot but it could happen... yhfin at yahoo dot kom.
Congratulations to BellKor's Pragmatic Chaos!
Actually receiving the prize will stimulate more people to do ML research. I would be happy to see some other AI competitions similar to IOI, IMO or TopCoder with big publicity.
Netflix are pioneers in such challenges and have the opportunity to be some of the founders of such future competitions.
Actually receiving the prize will stimulate more people to do ML research. I would be happy to see some other AI competitions similar to IOI, IMO or TopCoder with big publicity.
Netflix are pioneers in such challenges and have the opportunity to be some of the founders of such future competitions.
Very cool. For a long while, 10% improvement seemed like an asymptote.
Has Netflix announced any follow-up challenges?
Has Netflix announced any follow-up challenges?
Technically completing 10% gives you 30 days to submit another algorithm to do better than this algorithm.
So, all the submitted algorithms are going to be open sourced now, right? Right?!
In all seriousness, can anyone enlighten me as to who owns the submissions now?
In all seriousness, can anyone enlighten me as to who owns the submissions now?
The teams who created the solutions own them, but they're obligated to license them to Netflix. It doesn't have to be an exclusive license though. Quite generous on Netflix's part if you ask me.
> but they're obligated to license them to Netflix
Technically they are obliged to license them only if they are about to receive the prize.
Technically they are obliged to license them only if they are about to receive the prize.
Plus points on the baby-blue matrix background.
In any case, I'd love to see Netflix start another challenge round. A million dollars isn't a whole lot of money for them, it brings in a lot of publicity, and provides a 'fun' way to get some research done in the data-mining field.