Adventures in Cognitive Biases(web.mit.edu)
web.mit.edu
Adventures in Cognitive Biases
http://web.mit.edu/~xiac/www/advbiases/
17 comments
Yeah, many of the questions I would have just said "Not sure" if I was asked and then I would google the answer.
Yeah, this is a fair criticism. There are explicit bounds encoded in each problem and when those conflict with your estimates, you will be frustrated.
Haven't got a frigging clue isn't a fair thing to say though. The whole point of the exercise is that most people have a significant misalignment going on between their modelling of reality and the track record of their modelling ability.
Exercises like figuring out how much milk a cow produces in a week or how fast a horse runs require you to consider what information you DO have suggests about upper and lower bounds and things like that, and then correctly realizing the confidence of your models is the other half of the equation.
You are pretty sure, I would imagine, that a cow doesn't produce 10,000 gallons of milk in a week. You are pretty sure that a 'heavy' user of a vending machine might use three a day and buy a bunch of snacks and a drink or two, and that 'heavy' users maximally represent Y% of the population, with high confidence.
You can use these intuitions to guide your estimates, and you do in fact do so on a regular basis. The exercise is calibrating those estimates against their empirical accuracy.
Haven't got a frigging clue isn't a fair thing to say though. The whole point of the exercise is that most people have a significant misalignment going on between their modelling of reality and the track record of their modelling ability.
Exercises like figuring out how much milk a cow produces in a week or how fast a horse runs require you to consider what information you DO have suggests about upper and lower bounds and things like that, and then correctly realizing the confidence of your models is the other half of the equation.
You are pretty sure, I would imagine, that a cow doesn't produce 10,000 gallons of milk in a week. You are pretty sure that a 'heavy' user of a vending machine might use three a day and buy a bunch of snacks and a drink or two, and that 'heavy' users maximally represent Y% of the population, with high confidence.
You can use these intuitions to guide your estimates, and you do in fact do so on a regular basis. The exercise is calibrating those estimates against their empirical accuracy.
The point here is: The website asked me about my beliefs. It did NOT ask me to work out the correct solution. And if i believe i cannot possibly give a useful estimate then that should be a viable answer i can give.
Your example is especially relevant since i don't even know what a gallon is. I guess it's maybe a liter or something? Who knows. Either way: Without actual research there is no more accurate modelling of my belief than "i don't know", period.
Your example is especially relevant since i don't even know what a gallon is. I guess it's maybe a liter or something? Who knows. Either way: Without actual research there is no more accurate modelling of my belief than "i don't know", period.
Similarly, some of the answers were unnecessarily vague. For instance, I actually happened to read just yesterday a blurb about racing horse speed, and I knew that while racehorses can reach around 30mph, the very fastest horses can apparently reach sprinting speeds (briefly) of around 50mph. So which answer does it want? (This was especially frustrating since for most of the other answers I had no friggin' clue...)
They asked how many "reincarnations" there have been of the Dalai Lama. I think, and maintain, that the answer is 0, as I don't think "reincarnation" is possible. Had they instead said, "How many people have there been that have been called 'The Dalai Lama'", I would have answered differently.
Since this is a game about cognitive biases, hosted at mit.edu, I maintained hope that they, too, would be this pedantic about semantics, but, sadly, no.
Since this is a game about cognitive biases, hosted at mit.edu, I maintained hope that they, too, would be this pedantic about semantics, but, sadly, no.
If they had asked for an estimate of how many people died when the first Death Star was destroyed, would you answer be 0 because there was no actual Death Star?
No, because such questions, to me, implicitly acknowledge we are talking about a fictional world. The question about the Dalai Lama, however, is a question about this world.
Another example that I would say 0 to is, how many times did Jesus Christ come back to life? But I would say 1 to, according to the Bible, how many times did Jesus Christ come back to life? The difference between Star Wars and the bible is that there are not large groups of people who insist that Star Wars is real, so I feel safe in assuming an implicit, "According to this fictional thing". I consider the "reincarnations of the Dalai Lama" to fall under the bible category.
Another example that I would say 0 to is, how many times did Jesus Christ come back to life? But I would say 1 to, according to the Bible, how many times did Jesus Christ come back to life? The difference between Star Wars and the bible is that there are not large groups of people who insist that Star Wars is real, so I feel safe in assuming an implicit, "According to this fictional thing". I consider the "reincarnations of the Dalai Lama" to fall under the bible category.
"No, because such questions, to me, implicitly acknowledge we are talking about a fictional world. The question about the Dalai Lama, however, is a question about this world."
Or, perhaps you're implicitly talking about the world in which the Dalai Lama is reincarnated. There's not much difference to your distinction.
The correct answer is whatever is commonly excepted by Buddhists, I guess, or perhaps there is an official number. 0 would be a trick answer.
Or, perhaps you're implicitly talking about the world in which the Dalai Lama is reincarnated. There's not much difference to your distinction.
The correct answer is whatever is commonly excepted by Buddhists, I guess, or perhaps there is an official number. 0 would be a trick answer.
> Since this is a game about cognitive biases, hosted at mit.edu, I maintained hope that they, too, would be this pedantic about semantics, but, sadly, no.
It's also very disappointing to have the site claim that there is one unique "right answer" to questions that are ambiguous. For example, the site claims that 30 mph is the "right answer" to "how fast can a race horse run", but Googling shows plenty of examples of race horses running 40 mph or faster. I think they meant something more like "what is the average speed of race horses when clocked during races", but that's not what the question said.
It's also very disappointing to have the site claim that there is one unique "right answer" to questions that are ambiguous. For example, the site claims that 30 mph is the "right answer" to "how fast can a race horse run", but Googling shows plenty of examples of race horses running 40 mph or faster. I think they meant something more like "what is the average speed of race horses when clocked during races", but that's not what the question said.
Yeah, they worded the question poorly.
I answered 13 since the current Dalai Lama is the 14th and the first wasn't reborn. Also, to get technical he wasn't reincarnated, but reborn, which is an actual distinction Buddhists make.
I answered 13 since the current Dalai Lama is the 14th and the first wasn't reborn. Also, to get technical he wasn't reincarnated, but reborn, which is an actual distinction Buddhists make.
I answered 0 gallons of milk, because I don't believe in cows.
Kidding aside, I get your frustration. Trivia questions by necessity have the format of "one short question" = "one short specific answer".
Real facts usually don't work this way. Historians argue or change the date of various historical events over time as more data emerges. The Solar System "lost" a planet few years ago, and so on. "Facts" are in constant flux.
There's also a complicated, detailed context, there are a lot of "depends" in a real answer. Say, "how fast a horse runs"... domesticated & trained or wild, what breed, on a road or on a meadow, with horseshoes or without, etc.
So when faced with trivia questions the last thing I could say is I feel "overconfident" answering. Especially when the answer is a number. The designated correct answer is just an arbitrary data point that the author stumbled upon earlier in a book or online and wrote down as the answer.
It's apocrypha masquerading as facts.
Kidding aside, I get your frustration. Trivia questions by necessity have the format of "one short question" = "one short specific answer".
Real facts usually don't work this way. Historians argue or change the date of various historical events over time as more data emerges. The Solar System "lost" a planet few years ago, and so on. "Facts" are in constant flux.
There's also a complicated, detailed context, there are a lot of "depends" in a real answer. Say, "how fast a horse runs"... domesticated & trained or wild, what breed, on a road or on a meadow, with horseshoes or without, etc.
So when faced with trivia questions the last thing I could say is I feel "overconfident" answering. Especially when the answer is a number. The designated correct answer is just an arbitrary data point that the author stumbled upon earlier in a book or online and wrote down as the answer.
It's apocrypha masquerading as facts.
The overconfidence/underconfidence issue is similar to that expressed in the book 'How to measure anything'. The author claims to have trained people to become better estimators, by presenting them estimation challenges, and providing feedback about how often the actual value was within their stated range. IIRC people were asked to present a range for which they were 80% confident. If the answer was within the range too often (e.g. 90%) then their ranges were too wide, and if not often enough, they were overconfident. Over time, most subjects improved their calibration.
I started (yesterday) working on a simple web app to train people in this way. It's not yet ready to try out, but you can bookmark it here: https://github.com/rahimnathwani/measure-anything
I started (yesterday) working on a simple web app to train people in this way. It's not yet ready to try out, but you can bookmark it here: https://github.com/rahimnathwani/measure-anything
I think it's cute, but the overconfidence bias bit in the beginning is frustratingly geo-centric.
It's all general knowledge questions using imperial measurements, dollar spending power and american supermarket stats.
This needs a shrug button.
It's all general knowledge questions using imperial measurements, dollar spending power and american supermarket stats.
This needs a shrug button.
Exactly, I would have pressed the shrug button quite a bit here.
Oddly enough I did get the cat one spot on.
Oddly enough I did get the cat one spot on.
Yah, I feel like I'm being accused of being overconfident when my answer is "I don't know"
The math here is problematic. I happened to know how many 'reincarnations' of the Dalai Lama there have been (hint: he's routinely referred to as the [redacted] Dalai Lama, much like Louis the Sun King gets a number). This gave me ten thousand points. Other guesses give me fractions of a point.
What am I supposed to learn from that?
edit: checking again, ten thousand, not a thousand. corrected.
What am I supposed to learn from that?
edit: checking again, ten thousand, not a thousand. corrected.
as an astrophysicist, this one annoyed me: https://www.dropbox.com/s/udoug25cpd2zpiy/Screenshot%202014-...
Haha. I did the same exact thing.
US cows produce twice as much milk as the average, I guess.
And their assumption that everything is to be normally distributed is weird, and a distinct cognitive bias in and of itself.
yes, at the very least you should first move the circle vertically to define the bounds, then move the circle horizontally to create the shape of the distribution.
It has potential but they need to fix two things:
-make standard units available: liters, kilograms, meters;
-allow users to express full range of my beliefs; I have no idea how many reincarnations of Dalai Lama there was I guessed 1000 and wanted to set a straight line from 1 to infinity (or something going down to 0 slowly). It didn't allow me to; it's not overconfidence from my side, it's a bug on your side :)
-make standard units available: liters, kilograms, meters;
-allow users to express full range of my beliefs; I have no idea how many reincarnations of Dalai Lama there was I guessed 1000 and wanted to set a straight line from 1 to infinity (or something going down to 0 slowly). It didn't allow me to; it's not overconfidence from my side, it's a bug on your side :)
Do you really have no idea?
Do you believe in a likely lower bound for the time gap between reincarnations? And an upper bound for when the first one was? Or do you believe that both of these could be arbitrarily small (large, resp.)?
Do you believe in a likely lower bound for the time gap between reincarnations? And an upper bound for when the first one was? Or do you believe that both of these could be arbitrarily small (large, resp.)?
I think this is a very interesting game with an important message. However, the UX could be more engaging by having a more vertical leveling up structure. I felt as if I wanted to get to the next level but the onslaught of questions seemingly never ended- after a bit this becomes disengaging, specially when you aren't shown how many more units are needed to graduate.
I'm not sure who this game is aimed at, but if you're trying to teach cognitive biases to people with no training in psychology you really need to add a tutorial to the beginning. I would have loved to go further with the game, but I became frustrated at the very first question when I was confronted with this:
"Drag down the circle so that your belief covers all answers that you think are reasonable. This diagram is your belief graph."
I have no idea what a "belief graph" is, or how to use one. I though I had answered the question by entering a number into the field, but then I'm presented with this strange graph.
Perhaps I'm in the minority around here when it comes to my knowledge of this domain, but if this is a game intended for laypeople, you need to be really clear and assume I know nothing at all about your methods and practices. Explain to me what a belief graph is and how to use it. Either have a pre-game tutorial, or use tool tips.
That's my 2 cents.
"Drag down the circle so that your belief covers all answers that you think are reasonable. This diagram is your belief graph."
I have no idea what a "belief graph" is, or how to use one. I though I had answered the question by entering a number into the field, but then I'm presented with this strange graph.
Perhaps I'm in the minority around here when it comes to my knowledge of this domain, but if this is a game intended for laypeople, you need to be really clear and assume I know nothing at all about your methods and practices. Explain to me what a belief graph is and how to use it. Either have a pre-game tutorial, or use tool tips.
That's my 2 cents.
It becomes clear once you progress through a few of the examples what this means.
You don't need tutorials and tool-tips, you just need to be able to learn by doing.*
*and users who don't shy away on the first hint of confusion.
You don't need tutorials and tool-tips, you just need to be able to learn by doing.*
*and users who don't shy away on the first hint of confusion.
Yes.
The problems absolutely illustrate the concept if you have a reliable overconfidence confidence bias. You do not need a background in psychology to understand "Hey, I feel 90% confidence, but am being surprised nearly 1 in 3 times. There's some kind of miscalibration going on there."
The problems absolutely illustrate the concept if you have a reliable overconfidence confidence bias. You do not need a background in psychology to understand "Hey, I feel 90% confidence, but am being surprised nearly 1 in 3 times. There's some kind of miscalibration going on there."
The green cab question was changed from the original in a way that (I think) changes the logic.
http://heatherlench.com/wp-content/uploads/2008/07/bar-hille...
In the original, witnesses are only 80% accurate reporting color. In this version it's altered so that it's simply asserted that 80% of cabs in the neighborhood of the accident are green.
With that revision it seems to me that you would not update your priors but actually replace them. "The mix of cabs is 85/15. Oh wait, it's actually 20/80 the other way."
Edit: Ah, never mind. The paper gives both versions, specifically because the in second version you are supposed to throw away the prior.
http://heatherlench.com/wp-content/uploads/2008/07/bar-hille...
In the original, witnesses are only 80% accurate reporting color. In this version it's altered so that it's simply asserted that 80% of cabs in the neighborhood of the accident are green.
With that revision it seems to me that you would not update your priors but actually replace them. "The mix of cabs is 85/15. Oh wait, it's actually 20/80 the other way."
Edit: Ah, never mind. The paper gives both versions, specifically because the in second version you are supposed to throw away the prior.
It was fun but the end was too much of a cliffhanger for my taste.
There should be a choice of metric/imperial and some of the questions were a little meh. I liked the Bayesian part.
There should be a choice of metric/imperial and some of the questions were a little meh. I liked the Bayesian part.
I'd be interested to see the results of the first part of this. I've heard of those studies (perhaps they were just stories?) where groups of people collectively estimated things like number of jelly beans in a jar and were actually pretty accurate on average.
Obviously it's not going to be like that, because some of these things people can actually know - but I'm still curious.
Obviously it's not going to be like that, because some of these things people can actually know - but I'm still curious.
If you can find that, provide a link. I strongly doubt people are particularly good (without applying some sort of rigor) at estimating beans in jars.
This effect comes from the effect of regression to the mean. Some people will vastly overestimate, while some people will vatsly underestimate, thus cancelling out.
It holds as long as the people used are very diverse (not under the same cognitive bias) http://www.socialresearchmethods.net/kb/regrmean.php
It holds as long as the people used are very diverse (not under the same cognitive bias) http://www.socialresearchmethods.net/kb/regrmean.php
If you ask people to estimate the number of protons inside the sun, you might develop a peak around some sorts of numbers, but it's not likely to be a particularly good estimate.
Well, yeah, we all have a cognitive bias in that we've never had to comprehend a scale at 10^35-10^45 before (that's my belief graph btw).
the fascinating thing about those studies is the implication that many kinds of human knowledge are (normally?) distributed, and you can ask questions that's hard for an individual to get right but easy for a group (crowdsourcing). somewhat akin to how one brain cell doesn't contain knowledge, but a (large) collection of them do. it's like there's built-in fault tolerance. =)
I thought the test at the end was way too difficult given the simple introduction to calculating probabilities.
I felt like the test at the end was more of a test of the adventure itself: if many people are off target, there's probably more that should be taught.
Yes, I came here to say the same thing.
I am going to challenge that belief by asking you to do the following:
Try solving the blue and red pots problem, given only a single observation.
Now try again, this time with two observations.
Try solving the blue and red pots problem, given only a single observation.
Now try again, this time with two observations.
What do you think about Daniel Kahneman's 'Thinking Fast and Slow'?
Arbitrary weird GUI control that doesn't work on mobile and doesn't have an accessible fallback mode, even though the information content can be easily expressed with short simple text: yep.
I think the interaction is very interesting and engaging, lots of potential there.
But the only thing I learned is to look up answers to trivia questions on Google, and to drag chart bars to match the text above.
Overconfidence isn't the problem people have when obviously faced with questions they don't know the answer to (I've never milked a cow, and I don't intend to), and the inferences the protagonist was making from his trip were very suspicious and arbitrary, so that was disengaging in a tale about using proper logic.
But the only thing I learned is to look up answers to trivia questions on Google, and to drag chart bars to match the text above.
Overconfidence isn't the problem people have when obviously faced with questions they don't know the answer to (I've never milked a cow, and I don't intend to), and the inferences the protagonist was making from his trip were very suspicious and arbitrary, so that was disengaging in a tale about using proper logic.
They're hypothetical inferences, but the downside to using actual realistic examples is that even when they're meticulously researched, people nitpick them to a degree that baffles me. Like, asserting "but the probability of the evidence is either 0 or 1, so it's clearly impossible to ever make any inferential reasoning work".
The examples here are well chosen to eliminate the usual attacks on inference. They're ODD. But they're not invalid.
The examples here are well chosen to eliminate the usual attacks on inference. They're ODD. But they're not invalid.
Dude, you cheated.
Not only cheated, but also destroyed an opportunity to learn something valuable.
GP can, however, take other calibration tests elsewhere.
Here is one: http://calibratedprobabilityassessment.org/
Don't cheat this time. There is a valuable lesson here.
GP can, however, take other calibration tests elsewhere.
Here is one: http://calibratedprobabilityassessment.org/
Don't cheat this time. There is a valuable lesson here.
I wimped out, the list was too long and it is really hot. http://imgur.com/OB890bo at least my line is mostly straight, ;).
Lots of those questions were ambiguous within a couple hundred miles, at least in my mind.
Lots of those questions were ambiguous within a couple hundred miles, at least in my mind.
while i honestly answered all of the questions in the story, i have to admit i did a view source on the codeword because i didn't have the patience to work out the problems in the google doc before seeing how the story ends (disappointingly, i might add).
There's one major gripe i have though, the UI doesn't let me accurately express my beliefs.
On the question "How many dollars does the average American spend at vending machines during a year?" i thought it might be somewhere around 100, between 10 and 1000. However i could only get it as far as 38 - 162, which didn't allow me to cover the right answer. Optimally though i'd like to be able to answer "i haven't got a friggin' clue".
On the other hand, i guess it demonstrates a massive cognitive bias on the side of the makers of this: Not everyone is confident in knowing the answer to something.
Difficulties on other questions: