When the 'Wisdom of Crowds' turns on itself: IMDB Edition(news.cnet.com)
news.cnet.com
When the 'Wisdom of Crowds' turns on itself: IMDB Edition
http://news.cnet.com/8301-13515_3-10000650-26.html
21 comments
If you allow people to vote after they have seen the rankings, you have the wisdom of mobs. The only way to get the wisdom of crowds is to ask people before they have seen other people's opinions.
I upmodded your post because it already had +9!
I wonder what would happen on this site if the comments and stories were still sorted by points but the actual number was hidden (or perhaps only hidden until after you vote but I don't see why they need to be there period).
That would completely change how I vote. I vote by comparing a post's score to the amount I would give it if I could pick a value. If they're significantly different, I vote accordingly.
If you're really curious checkout Perlmonks.org. It's been around for a decade and has consistently maintained a community that more closely resembles this site than any other I've seen. Blind voting has been in use there for a very long time and has worked quite well. You're also alloted a limited number of points per day based on your karma (XP) level.
>> (or perhaps only hidden until after you vote but I don't see why they need to be there period)
That's an interesting point--why does every site need to quantify for the user the particular score of a comment? Why not indicate popularity by simply ordering the comments?
Is it a trust issue?
That's an interesting point--why does every site need to quantify for the user the particular score of a comment? Why not indicate popularity by simply ordering the comments?
Is it a trust issue?
Karma -> karma whoring -> pageviews -> profit?
Slashdot no longer displays numeric karma and you can disable display of comment scores.
Slashdot no longer displays numeric karma and you can disable display of comment scores.
You could also give other clues besides ordering - without revealing the numbers 'in your face'.
What sort of other clues?
For example color-shading. While it may still be possible to re-extract numbers from the colors --- psychological colors may have another effect than numbers.
Or perhaps something like Nethack does: http://www.steelypips.org/nethack/pray.html#p0
That's sort of like what Slashdot does for karma.
"Karma: Excellent"
It seems like in this case we have a temporal error. A popular movie generated buzz now and skewed a result that people pronounce loosely as of all time.
Batman might be the best movie out recently but in 2 years time, it will be a dim and fading memory while the Godfather will retain its appeal as it has already proven that it can.
IMDB just needs a temporal filter to keep the latest trend from skewing the universal result.
Batman might be the best movie out recently but in 2 years time, it will be a dim and fading memory while the Godfather will retain its appeal as it has already proven that it can.
IMDB just needs a temporal filter to keep the latest trend from skewing the universal result.
But it only takes 1 spoiler to publicize the results to ruin it for most everyone else.
I think that the problem is similar to the eternal September problem. I'm sure that the people who had been voting and contributing to the ratings on IMDB had been involved on the site for a while. Now, there's a big influx of people on IMDB that are voting for their favorite movie. I bet that many of the people who are voting Dark Knight up have never seen the Godfather or Shawshank.
Same thing happened to my wife in film school. She went to film school because she loved the classics of the 30's - 50's. She was stunned to find out that the kids she was going to school with thought that Austin Powers was the best film ever.
Same thing happened to my wife in film school. She went to film school because she loved the classics of the 30's - 50's. She was stunned to find out that the kids she was going to school with thought that Austin Powers was the best film ever.
This mob mentality, brought on by a significant event (in this case, the release of The Dark Knight), causes a total breakdown in a system that is otherwise fairly accurate, useful, and trustworthy.
I wouldn't call this a "total breakdown in the system". I would call it a minor blip. I mean, ultimately the system recommends what movies to watch. If someone isn't going to watch the Godfather now that it's ranked #3 instead of #1, then the person is flawed, not the system.
Otherwise, this was an interesting post. I have been astonished that TDK is #1. Didn't realize people were also downvoting other movies to keep it there.
I wouldn't call this a "total breakdown in the system". I would call it a minor blip. I mean, ultimately the system recommends what movies to watch. If someone isn't going to watch the Godfather now that it's ranked #3 instead of #1, then the person is flawed, not the system.
Otherwise, this was an interesting post. I have been astonished that TDK is #1. Didn't realize people were also downvoting other movies to keep it there.
The flaw in the system isn't that The Godfather was bumped down. The flaw is that people are rating the Godfather down to the point at which it changed its ranking with respect to other movies.
And you're right, it's not a major flaw for a movie ranking system. But let's suppose something similar to the IMDB system(which would be an approval/range voting system) is used in an election, and the first place gets President, the second choice getting Veep. With an election, you don't have real-time rankings like IMDB, but you have something similar--polls. And that's even worse, since there's a few points of inaccuracy. Strategically worded polls could change the results of an election.
And you're right, it's not a major flaw for a movie ranking system. But let's suppose something similar to the IMDB system(which would be an approval/range voting system) is used in an election, and the first place gets President, the second choice getting Veep. With an election, you don't have real-time rankings like IMDB, but you have something similar--polls. And that's even worse, since there's a few points of inaccuracy. Strategically worded polls could change the results of an election.
But that change is a blip in and of itself. Godfather has been bumped down before. Happened with Lord of the Rings. It climbs back up after the initial wave.
Score Voting (aka Range Votiing) and Approval Voting are actually the best voting methods, especially when you take strategic voting into account.
http://rangevoting.org/StratHonMix.html
In this case, even if the election result changed, you'd still get a candidate who was rated relatively well by the average voter, so the results would not be that much worse. Now consider that other voting methods react much worse to strategic voters, _and_ that score voting is as good with a great number of strategic voters as other systems are with 100% honest voters, and this whole argument falls flat.
In this case, even if the election result changed, you'd still get a candidate who was rated relatively well by the average voter, so the results would not be that much worse. Now consider that other voting methods react much worse to strategic voters, _and_ that score voting is as good with a great number of strategic voters as other systems are with 100% honest voters, and this whole argument falls flat.
If it's a minor blip, how do you propose they now fix the problem? I see it is as so systemic and flawed that the only solution is to scrap it altogether.
so a system that has worked fine for over a decade should be completely scrapped because of one unprecedented event?
So how do you fix these outliers? Dark Knight is still #1 and The Godfather is still #3.
My point is you don't. The system is still working fine. If Godfather sank to #300, then there would be something to fix. Even with all of the shenanigans of Dark Knight fans, the result on Godfather's rankings is a minuscule blip that could be explained by chance and should have no effect on whether someone thinks Godfather is a movie worth watching.
This also happened when Wall-E came out.
In any case: yes, bugs should be fixed.
In any case: yes, bugs should be fixed.
Wisdom of Crowds was a good book. The sequel, Attention Span of Crowds, was significantly shorter.
Have they ruled out the possibility that the people marketing the movie are gaming the system? Even if voting was restricted to established IMDB users, surely a reasonably well connected Hollywood marketing person could round up a few hundred of those among friends and friends of friends.
http://www.imdb.com/title/tt0468569/ratings is the vote breakdown.
Hmm, those numbers are definitely too big to be influenced by anyone's network. Judging from the other top-ranked films, it looks as if IMDB's userbase is in the middle of changing.
I don't think they need to game the system. The lukewarm or negative reviews on RottenTomatoes are being bombarded with genuine fanboy vitriol. (Many of the commenters, at least on the first page of comments, have joining dates that suggest they have been on RT for a while)
http://www.rottentomatoes.com/m/the_dark_knight/?critic=crea...
It is quite amusing in a sad way actually, seeing them trash the likes of David Denby and Joe Morgenstern.
To be fair, those reviews were both fairly terrible. Fanboyism is never justified, in my opinion, but I agree with the respondants' sentiments.
IMDB's ranking system is one of the least sophisticated out there, which is surprising given that they're owned by Amazon these days. I assume it's still just the same couple of guys that started it left to run it in relative independence. You really don't need to game their system much; it's just bad technology.
Maybe the majority of people would indeed get more entertainment out of "The Dark Knight" than out of "The Godfather"?
You're right.
However, if "The Dark Knight" bumping off "The Godfather" was all that was happening, that wouldn't be a problem.
What's happening, though, is that the release of "The Dark Knight" made "The Shawshank Redemption" have a higher rank than "The Godfather". This means that the ranking system has lead to the rating system being used strategically.
The implications of this for approval voting would be an interesting corollary. The polls could influence themselves.
However, if "The Dark Knight" bumping off "The Godfather" was all that was happening, that wouldn't be a problem.
What's happening, though, is that the release of "The Dark Knight" made "The Shawshank Redemption" have a higher rank than "The Godfather". This means that the ranking system has lead to the rating system being used strategically.
The implications of this for approval voting would be an interesting corollary. The polls could influence themselves.
I've seen The Godfather. I've seen The Dark Knight. Now, maybe I'm just missing the point of that Godfather thing, but I'd say The Godfather had more dramatic tension and The Dark Knight had more philosophical depth.
No, stop laughing, I'm serious. Tell me what point The Godfather makes.
No, stop laughing, I'm serious. Tell me what point The Godfather makes.
Blockbusters popular with the avid Internet users have always started at incredibly high ratings only to slowly fally to the right place. The LOTR movies were in top-10 once too.
I suppose IMDB might try to account for the initial voting wave being raving fans and dampen the initial votes, but why would they? Personally I've reloaded the "Dark Knight" IMDB page half a dozen times to see if it was still at #1.
The most popular topic on the IMDB message boards is "This does not deserve to be in top 250" or "Why is this not in Top 250 why some-other-movie is not?". People like to keep an eye on rankings and to comment on them.
The main practical use for the ratings IMHO is to get a list of movies within each genre sorted by the rating, to see if you missed something people thought was great.
I suppose IMDB might try to account for the initial voting wave being raving fans and dampen the initial votes, but why would they? Personally I've reloaded the "Dark Knight" IMDB page half a dozen times to see if it was still at #1.
The most popular topic on the IMDB message boards is "This does not deserve to be in top 250" or "Why is this not in Top 250 why some-other-movie is not?". People like to keep an eye on rankings and to comment on them.
The main practical use for the ratings IMHO is to get a list of movies within each genre sorted by the rating, to see if you missed something people thought was great.
exactly -- imdb has an internet and populist skew. As long as you take this into account, their ratings are fine and predictable. Gaming rankings is practically an official internet pastime, so it's not unexpected that this event is taking place, although I'm surprised it built up enough steam without (presumedly) official co-ordination to drop the Godfather a few places -- that is notable. I'm sure politicians are taking notes as we speak.
IIRC, the same thing happened at the releases of Spiderman and the Royal Tenenbaums. Hype does funny things to men.
If you think of this system as a game and in this instance a number of users have figured out a way to act within the rules of the game to achieve an outcome inconsistent with IMDB's intended one, then IMDB can simply change the rules to get what they think is best!
It wouldn't be too difficult to observe the behavior of the offending members (vote for TDK, within x hours vote down Godfather...) and simply setup criteria to disallow these actions. You could also set up an algorithm to detect screwy behavior to alert a developer or block that action for a duration.
And hey, we throw error messages for repeated button clicks and such so that click-happy users or spam bots don't take down the system...can't that kind of precaution be extended if necessary? "We know you're in love with TDK, but do you really hate The Godfather...come back in a month if you still do!"
It wouldn't be too difficult to observe the behavior of the offending members (vote for TDK, within x hours vote down Godfather...) and simply setup criteria to disallow these actions. You could also set up an algorithm to detect screwy behavior to alert a developer or block that action for a duration.
And hey, we throw error messages for repeated button clicks and such so that click-happy users or spam bots don't take down the system...can't that kind of precaution be extended if necessary? "We know you're in love with TDK, but do you really hate The Godfather...come back in a month if you still do!"
Soon you get websites with people saying "Rate The Godfather 1 before rating TDK 10."
That's funny, because the book which coined the term is about exactly this - how to make Wisdom of the Crowds work.
The problem is that people discuss these concepts without reading the book and thus miss out on the subtleties. The long tail is another common victim.
Disclaimer: I didn't read the book.
Disclaimer: I didn't read the book.
I'm (slowly) working on the summary here:
http://www.squeezedbooks.com/book/show/23/the-wisdom-of-crow...
Contributions welcome!
http://www.squeezedbooks.com/book/show/23/the-wisdom-of-crow...
Contributions welcome!
I don't see why it wouldn't be possible to build into the system some of the same logic that obviously makes it clear to everyone that there is a problem with these votes.
A computer might not be able to appreciate the goodness of The Godfather, but it can be taught that a movie that has consistently been voted great for decades, does not suddenly fall out of favor in a month.
A computer might not be able to appreciate the goodness of The Godfather, but it can be taught that a movie that has consistently been voted great for decades, does not suddenly fall out of favor in a month.
The problem is not the Dark Knight fans, so much as the algorithms being used to rank the movies. How can you take advantage of crowd wisdom if you don’t control for spoofing, spam and mob voting? It’s surprising that there wasn’t already in place a process for dealing with such a common phenomenon.
[deleted]
The point of top-k rankings is to put items in the proper order. As far as the crowds are concerned, this is no failure. Sorry your favorite movie lost its spot.
over time it should return to a 'fair' position, once temporary disequilibrium in the 'crowd' (induced by marketing, hype, whatever) is resolved. It's a shame people vote 'strategically', not to reflect their true sentiments - but whatever. Over time, with a diverse enough crowd, 'wisdom of the crowd' systems are effective and reasonably unbiased.
doubtless IMDB will enjoy the free publicity.
doubtless IMDB will enjoy the free publicity.
AFter the first weekend, Dark Knight had 9.7 average with almost a hundred thousand votes. That's just crazy.
Nah. 9.7 was with only 3,000. 100,000 only hit a few days ago.
[deleted]
I'm astonished. The internet hordes usually have impeccable taste in film and music. "Beverly Hills Chowahwah" is not even in the top 20?!!
Should be easy to cull these votes from the system. Anyone voting Dark Knight at 10 and Godfather at 1 should get nixxed from the system.
In fact, anyone voting Godfather less than 8 should be nixxed.
In fact, anyone voting Godfather less than 8 should be nixxed.
See, but it's all about opinion. Some people honestly like Dark Knight better, because it's far more gripping and easier to get into. And those people have every RIGHT to give Godfather a 1, if people that gave Godfather a 10 are giving The Dark Knight a 1 in return. The Godfather people started this days before the Dark Knight people came in.
Well anyone voting Godfather at 10 and Dark Knight at 1 should also be nixxed then.
Why not just let the two cancel each other out?
Why not just let the two cancel each other out?
This reminds me of the recent articles about the netflix optimization challenge. Voting systems like imdb might be improved with some simple heuristics that take into account the absurdity of some people's votes.
If a user makes votes that are < 2.0 for a movie that the entire community for years has rated above 9.0, then probably that user's votes should all be ignored, because they are totally uninformed.
If a user makes votes that are < 2.0 for a movie that the entire community for years has rated above 9.0, then probably that user's votes should all be ignored, because they are totally uninformed.
What if you assume everything is average and you give more weight toward the finally rating for those individual ratings that are further from the norm? This means people couldn't game it by following the crowd, following the crowd gives the entire crowds rating less weight.