Ask HN: The HN karma paradox
7 comments
The problem is that a single upvote-downvote axis has to capture two different, sometimes orthogonal, judgments: (1) is this comment valuable or detrimental to the discussion?; and (2) do I agree or disagree with this comment?
This entanglement is unavoidable as long as there's only one score per comment, given the interface and established practice, at HN and similar sites.
It might work to have a second axis/score that's specifically agree/disagree. (This could take the form of left-agree and right-disagree arrows, perhaps on the right of the comment-meta-line.) Then a comment could be be wildly agreed-with without offering a karmic windfall to repetitions of obvious popular sentiments, or wildly disagreed-with without the current undercurrents of censorship (sinking/fading-out) and karmic punishment via disagreement-downvotes.
Personally, I would expect to find the comments with both net upvotes (valuable) and rightvotes (disagreement) to be most interesting -- because they capture challenging minority viewpoints, but well-presented.
This entanglement is unavoidable as long as there's only one score per comment, given the interface and established practice, at HN and similar sites.
It might work to have a second axis/score that's specifically agree/disagree. (This could take the form of left-agree and right-disagree arrows, perhaps on the right of the comment-meta-line.) Then a comment could be be wildly agreed-with without offering a karmic windfall to repetitions of obvious popular sentiments, or wildly disagreed-with without the current undercurrents of censorship (sinking/fading-out) and karmic punishment via disagreement-downvotes.
Personally, I would expect to find the comments with both net upvotes (valuable) and rightvotes (disagreement) to be most interesting -- because they capture challenging minority viewpoints, but well-presented.
The problem is that a single upvote-downvote axis has to capture two different, sometimes orthogonal, judgments
I would like to see a HN-like site with a 2 part Karma system.
It would appear as three numbers A and S:N where A = Agree, S = Signal and N = Noise. S:N would not really be a ratio, and the Karma calculated from the three numbers could be any function, and wouldn't necessarily just consider S:N as a ratio.
But voting would effectively be a 4-choice radio box, so you could only pick one of the 4.
I suspect that those who try to game this system and censor comments by voting Noise as opposed to Signal will get overwhelmed by the Agree/Disagree voting on Signal.
I also suspect that Signal could just add to Karma, and Noise could just subtract from it. This might be able to exhibit the same dynamics as up/down voting on HN.
Another idea is to put the Noise vote button away from the up/down voting arrows and put it next to "Flag". The Signal control should still be next to Agree/Disagree, however. (Maybe a little star or sunburst? Wait, I know, a Sine Curve! Noise could have a square wave icon next to it.)
I proposed this before on HN, but I can't find it with SearchYC.
I would like to see a HN-like site with a 2 part Karma system.
It would appear as three numbers A and S:N where A = Agree, S = Signal and N = Noise. S:N would not really be a ratio, and the Karma calculated from the three numbers could be any function, and wouldn't necessarily just consider S:N as a ratio.
But voting would effectively be a 4-choice radio box, so you could only pick one of the 4.
- Agree (+A +S)
- Disagree (-A +S)
- Signal (+S)
- Noise (+N)
I suspect that those who try to game this system and censor comments by voting Noise as opposed to Signal will get overwhelmed by the Agree/Disagree voting on Signal.
I also suspect that Signal could just add to Karma, and Noise could just subtract from it. This might be able to exhibit the same dynamics as up/down voting on HN.
Another idea is to put the Noise vote button away from the up/down voting arrows and put it next to "Flag". The Signal control should still be next to Agree/Disagree, however. (Maybe a little star or sunburst? Wait, I know, a Sine Curve! Noise could have a square wave icon next to it.)
I proposed this before on HN, but I can't find it with SearchYC.
Signal/noise is another good way to express the 'valuable' axis -- terms with a long history in net discussions.
I would still like to be able to agree/disagree without giving a 'signal'/karma boost. A comment might be true, but trivial/superfluous/repetitive -- that's an agree-but-no-signal. (In the case of snark or offtopic trolling, I might even say I agree-but-that's-noise.) Similarly, something that's both a rule-transgression (noise) and egregiously wrong (such as dangerous misinformation) should get disagee-and-noise.
I would still like to be able to agree/disagree without giving a 'signal'/karma boost. A comment might be true, but trivial/superfluous/repetitive -- that's an agree-but-no-signal. (In the case of snark or offtopic trolling, I might even say I agree-but-that's-noise.) Similarly, something that's both a rule-transgression (noise) and egregiously wrong (such as dangerous misinformation) should get disagee-and-noise.
Noise voting should take precedence over agree voting.
Perhaps an even better idea would be to not have the Agree/Disagree voting add directly to signal, but for the score to have some affect on Karma and comment visibility through some function.
Also, this scheme might be to your liking: http://news.ycombinator.com/item?id=613520
Perhaps an even better idea would be to not have the Agree/Disagree voting add directly to signal, but for the score to have some affect on Karma and comment visibility through some function.
Also, this scheme might be to your liking: http://news.ycombinator.com/item?id=613520
It might be interesting for you to look at the rating system that's used on the site RSDN.ru (Russian Software Developer Network). The site has been around for quite some time and they've formulated a very balanced approach to rating.
On that site one has 7 buttons to rate a message: 1, 2, 3, '+', '-', '+1' and ':)'
1, 2 and 3 affect score (=karma) of the message and its author based on the level of the person who rates it (i.e. 2 from a 5th level person would add 5*2 = 10 etc)
'+' and '-' show that you agree/disagree with that comment, but do not affect the score.
'+1' adds 1 point to score independent of rater's level.
':)' shows that you find this comment funny.
I translated rules from this page http://rsdn.ru/forum/Info.aspx?name=info.forum.rating and as you may see, there are more rules that help to prevent excessive you-rate-me-I-rate-you behaviour, keep score up-to-date (your level depends only slightly on your all-times score but greatly depends on the marks that you've got during the last month) etc.
If people are interested, I can translate the rest of the rules.
On that site one has 7 buttons to rate a message: 1, 2, 3, '+', '-', '+1' and ':)'
1, 2 and 3 affect score (=karma) of the message and its author based on the level of the person who rates it (i.e. 2 from a 5th level person would add 5*2 = 10 etc)
'+' and '-' show that you agree/disagree with that comment, but do not affect the score.
'+1' adds 1 point to score independent of rater's level.
':)' shows that you find this comment funny.
I translated rules from this page http://rsdn.ru/forum/Info.aspx?name=info.forum.rating and as you may see, there are more rules that help to prevent excessive you-rate-me-I-rate-you behaviour, keep score up-to-date (your level depends only slightly on your all-times score but greatly depends on the marks that you've got during the last month) etc.
If people are interested, I can translate the rest of the rules.
Interested!
This has some of the same qualities as Slashdot, where you tag one of several reasons on your moderation.
EDIT: I just thought of a more user-directed mod/metamod system along the lines of Slashdot, but this way would be more Web 2.0. (As opposed to web 1.99a like Slashdot.)
Have a choice of pre-defined tags (about 10), each with a karma score attached to them. These tags would be long the lines of: "Funny" "Interesting" "Insightful" "Logical Fallacy" "Troll" Users could choose to either moderate a comment, which would be attaching 1 or 2 tags to a comment and thus altering the comment's visibility rating either-or meta-moderating, which would be voting on the fairness of moderation. So you could see what tags are attached to a comment and vote fair/unfair on each of them.
Now here's the rub: you get Karma by fairly moderating. (And lose it by unfairly moderating.) So while moderating changes a comment's visibility, only meta-moderating affects Karma.
Karma would be rewarded by the number of tags one could apply to a comment.
EDIT: More ideas. Users should be able to define their own tags, but only the admins would decide what point score should be attached to them!
This has some of the same qualities as Slashdot, where you tag one of several reasons on your moderation.
EDIT: I just thought of a more user-directed mod/metamod system along the lines of Slashdot, but this way would be more Web 2.0. (As opposed to web 1.99a like Slashdot.)
Have a choice of pre-defined tags (about 10), each with a karma score attached to them. These tags would be long the lines of: "Funny" "Interesting" "Insightful" "Logical Fallacy" "Troll" Users could choose to either moderate a comment, which would be attaching 1 or 2 tags to a comment and thus altering the comment's visibility rating either-or meta-moderating, which would be voting on the fairness of moderation. So you could see what tags are attached to a comment and vote fair/unfair on each of them.
Now here's the rub: you get Karma by fairly moderating. (And lose it by unfairly moderating.) So while moderating changes a comment's visibility, only meta-moderating affects Karma.
Karma would be rewarded by the number of tags one could apply to a comment.
EDIT: More ideas. Users should be able to define their own tags, but only the admins would decide what point score should be attached to them!
A very interesting example, thanks!
Some of these ideas are are instances of the pattern: redirect more of what would otherwise be a cluttering one-word 'attaboy!' or 'haha!' or 'this doesn't belong' response into a single-click and running-tally.
I see that RSDN.ru has done that with ':)'; Facebook has done it with 'Like'; HN and others have done it with 'flag' (though this flagged tally is usually non-public).
Some of these ideas are are instances of the pattern: redirect more of what would otherwise be a cluttering one-word 'attaboy!' or 'haha!' or 'this doesn't belong' response into a single-click and running-tally.
I see that RSDN.ru has done that with ':)'; Facebook has done it with 'Like'; HN and others have done it with 'flag' (though this flagged tally is usually non-public).
I find it funny that you'd vote a comment "right" in order to indicate that you found it "wrong". ;-) I'd rather reverse them, so you mark evil comments as sinister...
I've struggled and gone back and forth on that, and it's somewhat arbitrary -- compared to the obvious up-down orientation for signal/noise votes. The fact that questionnaires and poll-charts are more likely to put 'yes/agree' on the left and 'no/disagree' on the right swayed my proposed orientation, but I could be flipped.
I also think rather than just a net tally, results could be shown as a teeny tiny inline horizontally stacked barchart -- a sparkline of sorts -- with the overhang of the leading sentiment in its vote-direction. For example, with left-agrees, this would be more agrees than disagrees:
I also think rather than just a net tally, results could be shown as a teeny tiny inline horizontally stacked barchart -- a sparkline of sorts -- with the overhang of the leading sentiment in its vote-direction. For example, with left-agrees, this would be more agrees than disagrees:
++++++++
----
+++
--------
edit: yeah, I'm pretty sure I agree too, I'm not sure why I wrote so much...
I think strict Up-Down voting ultimately regresses to a Boo/Yay judgement; despite whatever benevolent community coercion exists to color the vote along some more complex institutional axis. (In so far as we're talking about a free public forum... substitute virtually any conception of "free" you like. e.g. if this were a pay-for community we could probably implicitly afford to have a higher expectation for the "respect" for institutional rules.)
I think the concept of providing additional axis of voting could lead to very interesting community conversations. I think having a voting system that was usably different would drive adoption of the additional institutional rules (e.g. voting via the four cardinals would be interesting enough to be "forthwhile" somewhere on the web) .
something like: ( ) I liked this comment. YAY! ( ) I'd like this comment to get a response. YAY for conversation! ( ) I do not think this comment needs a response. BOO for conversation! ( ) I disliked this comment. BOO!
* Though I'm not sure what to make of the third one. These may not be great exemplars.
I think strict Up-Down voting ultimately regresses to a Boo/Yay judgement; despite whatever benevolent community coercion exists to color the vote along some more complex institutional axis. (In so far as we're talking about a free public forum... substitute virtually any conception of "free" you like. e.g. if this were a pay-for community we could probably implicitly afford to have a higher expectation for the "respect" for institutional rules.)
I think the concept of providing additional axis of voting could lead to very interesting community conversations. I think having a voting system that was usably different would drive adoption of the additional institutional rules (e.g. voting via the four cardinals would be interesting enough to be "forthwhile" somewhere on the web) .
something like: ( ) I liked this comment. YAY! ( ) I'd like this comment to get a response. YAY for conversation! ( ) I do not think this comment needs a response. BOO for conversation! ( ) I disliked this comment. BOO!
* Though I'm not sure what to make of the third one. These may not be great exemplars.
I think that this would make for a very interesting experiment -- have two up/down arrows next to each post -- one for "I Agree/Disagree" and one for "This is Valuable/Worthless".
After all, the single up/down does combine several dimensions into one judgment.
After all, the single up/down does combine several dimensions into one judgment.
Agreevoted.
Maybe instead of cardinals, you could have:
(Up Arrow) "?", "!" (not necessarily mutually exclusive) (Down Arrow)
But what would be more interesting is to not force any meaning on them whatsoever. And wait and see if some sort of meaning emerges from the community.
(Up Arrow) "?", "!" (not necessarily mutually exclusive) (Down Arrow)
But what would be more interesting is to not force any meaning on them whatsoever. And wait and see if some sort of meaning emerges from the community.
Yes, exactly. I kind of assumed that adding a second voting axis was a nonstarter, but that would indeed provide more useful information about each comment.
I think it's more that long, detailed, well-supported points tend to get more karma than simple one-liners. IIRC, PG posted once that there was a big correlation between comment length and score.
I've noticed much less of a correlation between popularity and score than I would've expected - certainly much less than at Reddit, where you can write a detailed page-long comment that completely demolishes the parent's argument, and then have it sit at -1 because voters on Reddit reflexively downvote articles that don't fit their preconceptions. Here, that same comment is often at +10 or more.
I've noticed much less of a correlation between popularity and score than I would've expected - certainly much less than at Reddit, where you can write a detailed page-long comment that completely demolishes the parent's argument, and then have it sit at -1 because voters on Reddit reflexively downvote articles that don't fit their preconceptions. Here, that same comment is often at +10 or more.
> IIRC, PG posted once that there was a big correlation between comment length and score.
Yes, he wrote that. I wrote a small script to test that proposition. It does not hold up. If there is a correlation, it is only very weak.
You can have the source, if you wish. The script's written in Python.
Yes, he wrote that. I wrote a small script to test that proposition. It does not hold up. If there is a correlation, it is only very weak.
You can have the source, if you wish. The script's written in Python.
I'm interested eru - could I see the script?
I have now sanitized the program. Anyone else wants to play with it?
PG, is it OK to download such massive amounts of items?
PG, is it OK to download such massive amounts of items?
On Reddit, it's the short, witty comments that get the most votes. Unfortunately.
Well thought out, detailed, long comments often end up with just a few points, or worse, in the negative, because of "too long, didn't read".
Well thought out, detailed, long comments often end up with just a few points, or worse, in the negative, because of "too long, didn't read".
I'd definitely agree that it's far worse at Reddit. And I hear you on comment length, but I took care not to mention length because it's possible for well-written, long comments to make a widely accepted point.
If post A and post B are equally convincing and interesting, but B is twice as long, then A is clearly superior.
They are rarely equally convincing and interesting, though.
There are many interesting subtleties to voting and karma systems, such as the one we use here on HN. Another one is that a post that has a child that glorifies the parent "Great insight, bla.bla." will inevitably be voted up. You can probably find more if you look a bit.
You assume humans are rational beings and vote accordingly. But they aren't, they are emotional beasts that are hard to control, and are full of strange biases and ideas about what is right and wrong.
See for instance this list of cognitive biases from wikipedia: http://en.wikipedia.org/wiki/List_of_cognitive_biases Your paradox is explained by one of the biases listed, I'll leave it to you to figure out which one ;-)
It's all about psychology.
You assume humans are rational beings and vote accordingly. But they aren't, they are emotional beasts that are hard to control, and are full of strange biases and ideas about what is right and wrong.
See for instance this list of cognitive biases from wikipedia: http://en.wikipedia.org/wiki/List_of_cognitive_biases Your paradox is explained by one of the biases listed, I'll leave it to you to figure out which one ;-)
It's all about psychology.
When did I assume humans are rational beings and vote accordingly? What I was getting at is that widely accepted points get lots of upvotes as people express their agreement, but more controversial points get more of their upvotes because of the comment's quality. For the controversial points, this is offset by downvotes from people expressing disagreement. See gojomo's comment for a possible solution.
I'm not saying any voting system can be perfect, but this particular problem is explained by the fact that multiple judgments are being captured by a single voting axis, as gojomo said.
I'm not saying any voting system can be perfect, but this particular problem is explained by the fact that multiple judgments are being captured by a single voting axis, as gojomo said.
Reading my parent comment I see that it may look a bit arrogant, sorry that wasn't intentional, I merely think the topic is interesting, and hope to add some value to the conversation :-)
I think that basically what's going on here is that a simplistic voting system, such as the one we use here, and also one that carries more parameters and dimensions will always just be a simplistic overlay on an extremely nuanced and varied set of psychological rules that humans use for evaluating their surroundings, thus making it hard to accurately have karma map onto a users insightfullness (in lack of a better word) I'm pretty sure I could get a lot of karma by using psychology tricks to make people upvote me, but it wouldn't be very interesting. Or ethical for that matter.
So I absolutely agree with you: Karma isn't a measure of a users insightfullness, there's probably a correlation but it's weakened by a lot of parameters, such as the ones you describe, age of the account, and lots of other factors.
And hey, Welcome to HN, hope you like it here :-)
I think that basically what's going on here is that a simplistic voting system, such as the one we use here, and also one that carries more parameters and dimensions will always just be a simplistic overlay on an extremely nuanced and varied set of psychological rules that humans use for evaluating their surroundings, thus making it hard to accurately have karma map onto a users insightfullness (in lack of a better word) I'm pretty sure I could get a lot of karma by using psychology tricks to make people upvote me, but it wouldn't be very interesting. Or ethical for that matter.
So I absolutely agree with you: Karma isn't a measure of a users insightfullness, there's probably a correlation but it's weakened by a lot of parameters, such as the ones you describe, age of the account, and lots of other factors.
And hey, Welcome to HN, hope you like it here :-)
I agree; there's no perfect voting system. This is similar to what you said, but people getting lots of upvotes for saying something many others agree with probably means at least slightly fewer unpopular views are being posted.
Thanks for the welcome! I know the account is new but I've actually been here for a long time. :)
Thanks for the welcome! I know the account is new but I've actually been here for a long time. :)
Soon you will realize the pointlessness of the score system (no matter how it's implemented), and from there it's a short distance to leaving it behind and getting down to business (whatever is business to you). Treat HN as a class which you need to graduate from and it will make a lot more sense.
I'm not saying I expect the score system to be perfect and "fair". That would be impossible. But it isn't pointless either, because it helps us sort the articles and comments. If it really were pointless, as you say, then even the single voting axis that exists today should be removed, right?
> it helps us sort the articles and comments.
Are you sure about that? I find it roughly as many good comments with low votes as I find with high votes. My most subjectively valuable comments remain at +2 whereas my smart-ass comments get upmodded uncomfortably high. Same with stories - half of the stories I find valuable are lingering at low scores.
Voting seems to be about social status, values and group identity, not about content quality. There are people who try to vote quality but I rarely see them rising above the normal voters.
I suggest you give it a try and take notes while reading both high and low rated stories/comments. Maybe you will come to different conclusions than I.
Are you sure about that? I find it roughly as many good comments with low votes as I find with high votes. My most subjectively valuable comments remain at +2 whereas my smart-ass comments get upmodded uncomfortably high. Same with stories - half of the stories I find valuable are lingering at low scores.
Voting seems to be about social status, values and group identity, not about content quality. There are people who try to vote quality but I rarely see them rising above the normal voters.
I suggest you give it a try and take notes while reading both high and low rated stories/comments. Maybe you will come to different conclusions than I.
Are we talking "karma for a single comment/post" or "karma for a user"? The reason I ask is that I've been using HN since close to the beginning and never once have I looked at a user's karma score. User karma has no meaning to me, good or bad. Post/comment karma matters in the sorting and what appears on the front page so, to me, it's the only thing that I care about.
So is your discussion more about user or post/comment karma? I suspect the latter.
So is your discussion more about user or post/comment karma? I suspect the latter.
I guess I had both in mind. The post/comment karma for the reasons you mentioned, but also user karma - not just because people look at a user's karma score, but because it's used to determine other things. If it's used for even more purposes in the future (vote weighting, for example) its importance will only increase.
HN and most karma-based forums follow the bandwagon effect. The exception is probably slashdot, due to it's different way of power assignment and levels of voting
What kind of point is widely accepted and false?
Rather than provide you with specific examples, which by definition many people would disagree with, I'll just ask you this: Isn't there something you believe to be true which most people don't?
Edit: I can also suggest historical examples. How many things were once widely accepted to be true, but are today widely accepted to be false? Assuming today's view is correct, these things were at one time widely accepted and false.
Edit: I can also suggest historical examples. How many things were once widely accepted to be true, but are today widely accepted to be false? Assuming today's view is correct, these things were at one time widely accepted and false.
Pretty much the basis of the excellent show "QI" which Stephen Fry presents. Check it out.
The existence or lack thereof of God. No matter which way you believe on that, there's a large portion of society that feels the complete opposite way.
For example (but that's kind of a meta-point): If you select any skill (say driving) and ask a sample of people in which quantile they place themself, you are very unlikely to get a uniform distribution. (Like: "90% of people think themself above median drivers.")
But they're often right. Most people don't think themselves above-median drivers, they think themselves above-average (mean) drivers, i.e. if you take all the car accidents and divide by the number of drivers, they've been involved in fewer than that number. And they'd probably be right: the median number of car accidents is significantly lower than the mean, because a small number of bad drivers account for a large number of car accidents. Not everything is uniformly distributed - in fact, most things aren't.
I know that most things are strangely distributed. Even the normal distribution is far from uniform. So I was carefully phrasing my comments in terms of quantiles. Quantiles are evenly distributed by definition.
People think themself "better than the typical guy" i.e. above median.
People think themself "better than the typical guy" i.e. above median.
"People only use 10% of their brains"
It appears that a comment which makes a point that's already widely accepted, whether true or false, tends to get more upvotes than a comment which is more controversial - again, whether true or false. The controversial yet true comments are usually more interesting and valuable. (Maybe that applies even to false controversial comments.)
Fortunately, I think most people here care more about saying what they want to say than about getting more upvotes. So I'm not saying something needs to be fixed or that I have a much better idea. Still, I think it's a reason to be cautious about taking someone's karma score very seriously. Forgive me if I'm just saying things you already know.