BelKor has allegedly won Netflix Prize despite being second on the leaderboard(netflixprize.com)
netflixprize.com
BelKor has allegedly won Netflix Prize despite being second on the leaderboard
http://www.netflixprize.com/community/viewtopic.php?pid=9237#p9237
2 comments
Yes, the difference is negligible, but you obviously can't change rules of the game when it has already finished.
And to be clear: The winner of the contest is not determined by the number that you see listed on the leaderboard. The number on the leaderboard [1] is on the Quiz dataset, the winner is determined by running against the Test dataset (which is kept private by Netflix). As stated by Yehuda on team BellKor: "our team is top contender for winning the Grand Prize, as we have a better Test score than The Ensemble."
This turnaround does not surprise me. BellKor's Pragmatic Chaos took their sweet time getting to 10%+ - and in doing so they were very sure not to overfit [2] the data (making their solution a much more generic and viable solution to the dataset). It's my guess that The Ensemble rushed quickly to 10%+ and overfit their data like mad (which yielded high numbers on the public dataset, but evidently does not translate to a generic solution).
I'm looking forward to seeing the final papers published by BellKor, et. al. - they're going to be a fascinating read, regardless.
1: http://www.netflixprize.com/leaderboard 2: http://en.wikipedia.org/wiki/Overfitting
This turnaround does not surprise me. BellKor's Pragmatic Chaos took their sweet time getting to 10%+ - and in doing so they were very sure not to overfit [2] the data (making their solution a much more generic and viable solution to the dataset). It's my guess that The Ensemble rushed quickly to 10%+ and overfit their data like mad (which yielded high numbers on the public dataset, but evidently does not translate to a generic solution).
I'm looking forward to seeing the final papers published by BellKor, et. al. - they're going to be a fascinating read, regardless.
1: http://www.netflixprize.com/leaderboard 2: http://en.wikipedia.org/wiki/Overfitting
I don't think that is correct. The numbers on the leaderboard are not the scores on the training set.
The netflix prize uses three datasets: the training set, the leaderboard set, and the test set. The training set is distributed to everyone. The test set is totally secret. Access to the leaderboard set is only by submitting results (once per 24 hours) and looking at the results. It is not at all trivial to "overfit" to this leaderboard set. (It could be done by, e.g. submitting results with slight tweaks to the algorithm parameters, but this would take a lot of time since you can only submit every 24 hours. Also, you would basically have to do it consciously.)
The netflix prize uses three datasets: the training set, the leaderboard set, and the test set. The training set is distributed to everyone. The test set is totally secret. Access to the leaderboard set is only by submitting results (once per 24 hours) and looking at the results. It is not at all trivial to "overfit" to this leaderboard set. (It could be done by, e.g. submitting results with slight tweaks to the algorithm parameters, but this would take a lot of time since you can only submit every 24 hours. Also, you would basically have to do it consciously.)
A lot of people have been tripped up by this, and it's a VERY important distinction. I hope Netflix posts more about the final (Test) results from each when they officially announce the winner.
Given that further work on the dataset has continued to produce rapidly diminishing returns, one would assume that prolonging the contest would be likely to result in the teams being closer, not further apart.
Winner takes all contests seem to be zero sum games to me, in a certain sense. The pie is a fixed size, and one guy gets all of it. I think I'd rather sink my time into business or open source stuff.
Saying a winner-takes-all contest is "zero sum" in any sense is utterly nonsensical, as it's clearly not zero sum since one group wins $1,000,000.
Also, "one guy" doesn't necessarily get all of it, as team co-operation is implicitly encouraged as it increases the chances of winning. Additionally, the game is strictly skill-based, so the more skilled you are the more likely you are to win, which may encourage you to work in the contest. And that's excluding all the additional externalities involved.
Also, "one guy" doesn't necessarily get all of it, as team co-operation is implicitly encouraged as it increases the chances of winning. Additionally, the game is strictly skill-based, so the more skilled you are the more likely you are to win, which may encourage you to work in the contest. And that's excluding all the additional externalities involved.
> Saying a winner-takes-all contest is "zero sum" in any sense is utterly nonsensical
I thought that zero sum just meant that the total score of all players is a constant, regardless of the strategies chosen?
I thought that zero sum just meant that the total score of all players is a constant, regardless of the strategies chosen?
> clearly not zero sum
Ok, so call it "fixed sum" or whatever else is technically correct - that's the point I'm trying to make. If one guy wins, another guy necessarily loses. That's not true in plenty of other things.
> the more skilled you are the more likely you are to win
Yes, but the second and third guys are quite skilled too, and win nothing (and yes, we are excluding the externalities since we're talking about the general case, not this specific one).
Ok, so call it "fixed sum" or whatever else is technically correct - that's the point I'm trying to make. If one guy wins, another guy necessarily loses. That's not true in plenty of other things.
> the more skilled you are the more likely you are to win
Yes, but the second and third guys are quite skilled too, and win nothing (and yes, we are excluding the externalities since we're talking about the general case, not this specific one).
Actually, the top teams have already started negotiating with other companies for the technology that they came up with. Knowledge is never zero-sum.
I wrote contest, not knowledge. The knowledge is external to the contest. Some contests, like this one, have lots of external benefits to the top guys, but "winner takes all contest" in the abstract sense seems to me to be a zero sum game, so I stand by what I said: it's generally better to sink your time into something where the "external stuff" is actually the main benefit.
The term "zero sum" does not really make sense here. Netflix obviously thinks they're getting more than what they're spending on the contest and the winning team obviously thinks they're receiving more than what they've invested. So between those two parties, there is a net gain as is the case in most voluntary transactions.
At best, what you're saying is that other competitors have wasted their time, but I don't think that's true either.
At best, what you're saying is that other competitors have wasted their time, but I don't think that's true either.
> Netflix ... here
For the third time: I am talking about the abstract case, not 'here', and I am talking about the point of view of the competitors, not the entity running the contest.
> At best, what you're saying is that other competitors have wasted their time, but I don't think that's true either.
They haven't won anything, and unless there are externalities, they have wasted their time. How can they not have? They received nothing.
So, what I said is true: the contest is a "zero" or "fixed" sum game, and the other competitors gain only through externalities. You can downmod that all you like, but this more or less concurs with the definition of "zero sum game" in wikipedia.
For the third time: I am talking about the abstract case, not 'here', and I am talking about the point of view of the competitors, not the entity running the contest.
> At best, what you're saying is that other competitors have wasted their time, but I don't think that's true either.
They haven't won anything, and unless there are externalities, they have wasted their time. How can they not have? They received nothing.
So, what I said is true: the contest is a "zero" or "fixed" sum game, and the other competitors gain only through externalities. You can downmod that all you like, but this more or less concurs with the definition of "zero sum game" in wikipedia.
The reason you are having to dogmatically repeat your idiosyncratic use of the term "zero sum" is that you are using it in such a narrow sense that it adds nothing to what you are saying. A typical example of a zero sum game is a futures contract which is contrasted with a non-zero sum game like stock investing.
If you exclude externalities like all parties in a transaction being happier with the outcome or placing present value on some possible future outcome, then there is no net positive sum game.
If you exclude externalities like all parties in a transaction being happier with the outcome or placing present value on some possible future outcome, then there is no net positive sum game.
My view is that a lot of contest type things are not a good deal for the participants, and that it's a good idea to recognize that, going in. If there are external benefits, well, then it may be worth it. Otherwise, you're probably better off investing your time in something that's positive sum, such as, say, a startup. I would have thought that people here would get that.
Sure, if you're only weighing the cash prize. Contests like this often involve competitors spending more than the prize, certainly in total among all entrants, but sometimes even by the single winner.
But it's clear there are many other motivations at play: the excitement of the competition, the chance to learn based on a real-world problem and dataset, the fame available even to those who reach goals other than final victory. And many of the results can be published as papers or open-source software, as well.
But it's clear there are many other motivations at play: the excitement of the competition, the chance to learn based on a real-world problem and dataset, the fame available even to those who reach goals other than final victory. And many of the results can be published as papers or open-source software, as well.
A contest like this one (with such a large prize at stake) should have seen this coming and instituted a different rule for their deadline. Something like "24 hours after the last submission." Therefore you wouldn't have someone sniping the win like it seems has happened.
But this isn't an auction. You can't just conjure up a better result once you know you've been surpassed. Belkor etc was probably working just as hard all month & the ensemble just got lucky.
Just like in ping-pong, extend the game if it is that close until there is a clear winner.
With a winner takes all contest like this there has to be a clear winner.