Christopher Chabris Should Calm Down(slate.com)
slate.com
Christopher Chabris Should Calm Down
http://www.slate.com/articles/health_and_science/science/2013/10/malcolm_gladwell_s_david_and_goliath_he_explains_why_christopher_chabris.html
14 comments
That's fair, but I found this portion of Gladwell's response interesting:
> Chabris does not like this study. He thinks it involved too few subjects and that its findings were not replicated by a subsequent study. Alter and Oppenheimer disagree. They say that the version of desirable difficulty that they explore has been confirmed on numerous other occasions. This is the kind of intramural argument about the nature and value of evidence that social scientists have all the time.
So who am I supposed to trust here? Who has the higher epistemic ground, Chabris or Alter & Oppenheimer? The problem is that I don't know what the scientific consensus is on the validity of that study. I'm not sure whose side I should be taking.
Are Alter & Oppenheimer bad scientists trying to pass off a poor study as robust scholarship? Or is there a valid difference of opinion here?
> Chabris does not like this study. He thinks it involved too few subjects and that its findings were not replicated by a subsequent study. Alter and Oppenheimer disagree. They say that the version of desirable difficulty that they explore has been confirmed on numerous other occasions. This is the kind of intramural argument about the nature and value of evidence that social scientists have all the time.
So who am I supposed to trust here? Who has the higher epistemic ground, Chabris or Alter & Oppenheimer? The problem is that I don't know what the scientific consensus is on the validity of that study. I'm not sure whose side I should be taking.
Are Alter & Oppenheimer bad scientists trying to pass off a poor study as robust scholarship? Or is there a valid difference of opinion here?
It's not uncommon for researchers to dispute follow-up studies that fail to reproduce their results. Their reputations are at stake, if it points to methodological errors (which seems to be the case here).
The problem here is Gladwell didn't even hint that there is significant doubt about the methodology of study he cited. Note his dismissive term for it: a "intramural argument". Chabris just "doesn't like" the study, as if it's a shirt in a color that doesn't suit him.
These are not equivalent things. It's not just a difference of opinion or who tells a better story or who has more letters after their name. What matters is the science behind it, and Gladwell is intentionally perverting it in the service of a better story. That's a huge disservice to the public when it comes to popularizing science.
The problem here is Gladwell didn't even hint that there is significant doubt about the methodology of study he cited. Note his dismissive term for it: a "intramural argument". Chabris just "doesn't like" the study, as if it's a shirt in a color that doesn't suit him.
These are not equivalent things. It's not just a difference of opinion or who tells a better story or who has more letters after their name. What matters is the science behind it, and Gladwell is intentionally perverting it in the service of a better story. That's a huge disservice to the public when it comes to popularizing science.
> Alter and Oppenheimer disagree
But they're not the ones who can decide whether subsequent studies replicated their findings or not! It's up to the scientific community as a whole to decide.
The problem is, no one is really interested in replicating earlier findings, and even less in invalidating them. What people want is produce new and interesting (unexpected, curious) results that gets the authors noticed.
Many such studies never get replicated (see the famous one about how talking to students about old age makes them walk slower), and nobody cares.
So this part of Gladwell's response is even more disingenuous than the rest.
But they're not the ones who can decide whether subsequent studies replicated their findings or not! It's up to the scientific community as a whole to decide.
The problem is, no one is really interested in replicating earlier findings, and even less in invalidating them. What people want is produce new and interesting (unexpected, curious) results that gets the authors noticed.
Many such studies never get replicated (see the famous one about how talking to students about old age makes them walk slower), and nobody cares.
So this part of Gladwell's response is even more disingenuous than the rest.
This piece was written by Gladwell in response to Chabris. Between the two of them, Chabris, as a PhD holder and tenured professor, clearly has the higher ground when it comes to understanding the science in question.
Yes, Gladwell mentions Alter and Oppenheimer in his writing, but remember that the knock against Gladwell is that he selectively brandishes science when it suits his narrative. He's just doing that again here.
The fundamental complaint from Chabris is that Gladwell did not even acknowledge the existence of any controversy on this subject in his book. He simply picked the study he liked best and wrote about it as if it were fact.
Yes, Gladwell mentions Alter and Oppenheimer in his writing, but remember that the knock against Gladwell is that he selectively brandishes science when it suits his narrative. He's just doing that again here.
The fundamental complaint from Chabris is that Gladwell did not even acknowledge the existence of any controversy on this subject in his book. He simply picked the study he liked best and wrote about it as if it were fact.
> he cited studies that have been proven wrong in the service of a story.
That's a substantial overstatement. Gladwell didn't cite studies that have been proven wrong, he cited studies that have not been consistently reproducible. That's still a problem, but it's a very significantly different one.
That's a substantial overstatement. Gladwell didn't cite studies that have been proven wrong, he cited studies that have not been consistently reproducible. That's still a problem, but it's a very significantly different one.
Especially in social science world where it's difficult to have rigor in experiment design. Saying: "Gladwell didn't quote one study which according to one social scientist (but not other ones) made initial findings less likely" would be more accurate.
Again, the man told interesting story backed by some studies and opinions of somewhat respected scientists in the field. He also gave references so readers could investigate. By doing this he is ahead of vast majority of pop-science/business books out there already.
I have to say, I always take the criticisms of Gladwell very seriously. I read his first three books several years ago and have constantly felt like a sucker over the years as his detractors have grown louder.
And yet I find his responses very convincing, especially this one and the response to the Ask A Korean! article. I don't know, maybe I'm just easily swayed or maybe it speaks more about how persuasive Gladwell is.
And yet I find his responses very convincing, especially this one and the response to the Ask A Korean! article. I don't know, maybe I'm just easily swayed or maybe it speaks more about how persuasive Gladwell is.
I've read his books as well and definitely feel like I've been lied to. I almost feel like he was playing on stereotypes and I fell for it. Of course things aren't as simple as he presents them in outliers, but man did I fall for it at first.
>Of course things aren't as simple as he presents them in outliers, but man did I fall for it at first.
Yea, I had just graduated from college and was binging on popsci. I felt really smart because all the popsci authors seemed to reference each other and when I saw the references I thought I was very well read. Obviously, I don't think that anymore.
This is an embarrassingly accurate description of me at that time: http://xkcd.com/863/
Yea, I had just graduated from college and was binging on popsci. I felt really smart because all the popsci authors seemed to reference each other and when I saw the references I thought I was very well read. Obviously, I don't think that anymore.
This is an embarrassingly accurate description of me at that time: http://xkcd.com/863/
haha. That's a good one.
You "felt like a sucker" and from another comment "feel like I've been lied to"?
Do other people's comments and criticisms make you change your mind about something that easily? If you thought it was a good book when you read it, why would anything anyone else says change that? (Short of, say, Gladwell made up the entire thing or has been rampantly plagiarizing without attribution.)
Gladwell's books are very well-written narrative combining summarized research studies, anecdotal stories and histories to support his overall thesis. It is not a scientific study. He "connects the dots" of what he believes to be the central theme of his book. I haven't seen anywhere that he "lied".
There will always be detractors to any non-fiction author, especially one so popular. (See Nassim Taleb's books as another example.) If you can find a non-fiction book without major detractors, it probably isn't very good.
Do other people's comments and criticisms make you change your mind about something that easily? If you thought it was a good book when you read it, why would anything anyone else says change that? (Short of, say, Gladwell made up the entire thing or has been rampantly plagiarizing without attribution.)
Gladwell's books are very well-written narrative combining summarized research studies, anecdotal stories and histories to support his overall thesis. It is not a scientific study. He "connects the dots" of what he believes to be the central theme of his book. I haven't seen anywhere that he "lied".
There will always be detractors to any non-fiction author, especially one so popular. (See Nassim Taleb's books as another example.) If you can find a non-fiction book without major detractors, it probably isn't very good.
I felt lied to because he cites facts that aren't true, uses questionable sources, and draws conclusions more based on gut feeling and speculation than hard science. He does all this while presenting it as science. The book is sold as non-fiction after all. I got caught up in the hype and naively assumed he must know what he's talking about, but clearly there are issues with his methods.
Superficially, he is quite convincing, but when you look at a given statement or claim in isolation, it's common to find anything from an insufficiency of support, to outright howling fallacy.
It's an interesting illustration of his "narratives often begin in one place and end in another" that his Slate piece begins with calling his detractors ill-read and careless ("I didn’t want to debate them. I just wanted them to read my book all the way to the end."), and ends with the implicit argument that criticizing his work is the same thing as calling his adherents stupid ("I was simply saying that all writing about social science need not be presented with the formality and precision of the academic world...My point was that the people who read my books appreciate this.").
It's an interesting illustration of his "narratives often begin in one place and end in another" that his Slate piece begins with calling his detractors ill-read and careless ("I didn’t want to debate them. I just wanted them to read my book all the way to the end."), and ends with the implicit argument that criticizing his work is the same thing as calling his adherents stupid ("I was simply saying that all writing about social science need not be presented with the formality and precision of the academic world...My point was that the people who read my books appreciate this.").
"The kinds of people who read books in America seem to have no problem with my writing. But I am clearly a bee in the bonnet of some of the kinds of people who review books in America."
That's a nice example of the sort of false dichotomy that infuriates Gladwell's critics.
That's a nice example of the sort of false dichotomy that infuriates Gladwell's critics.
This bothered me more than anything else in the response. Just because your book is popular does not mean it is not misleading.
I'm dense today, I think. I'm afraid I don't see the false dichotomy. I see "kinds of people who read books in America" and "some of the kinds of people who review books in America". Is there a 3rd/nth group I'm not seeing - perhaps "people who read books and who also review books"?
Genuine question
Genuine question
To review a book it is not strictly necessary to have read it, at least according to a few targeted authors, including this case ("I just wanted them to read my book all the way to the end").
Ahhh - thank you. I just couldn't figure out what I was missing haha
Aren't some of these arguments against Gladwell along the same lines as people complaining about Mythbusters' lack of rigor? Well, then I defer to zombie Feynman.
http://xkcd.com/397/
I'm not saying Gladwell is right, but Chabris' argument amounts to pulling one referenced study out of context and saying "SEE HOW TERRIBLE THIS IS." He's complaining about sample size... with a single anecdotal example?
Also, social scientists are kind of cute when they get all angry.
http://xkcd.com/397/
I'm not saying Gladwell is right, but Chabris' argument amounts to pulling one referenced study out of context and saying "SEE HOW TERRIBLE THIS IS." He's complaining about sample size... with a single anecdotal example?
Also, social scientists are kind of cute when they get all angry.
I completely agree that bringing a helpful idea to a broader audience is quite valuable. The problem, however, lies in the combination of lack of precision and lack of attention span that creates artifacts of conventional wisdom that rattle around for years and years without question.
Some day, someone's grandkid is going to be spouting something about "10,000 hours" and not even know where it comes from.
Some day, someone's grandkid is going to be spouting something about "10,000 hours" and not even know where it comes from.
> Some day, someone's grandkid is going to be spouting something about "10,000 hours" and not even know where it comes from.
Which will replace the even more flawed understanding of "born with it".
We all go through life with flawed mental models, which aren't perfect but are better than a complete lack of understanding. Over time, we make those models in the general consciousness better, even if they continue to lag behind true scientific understanding.
Which will replace the even more flawed understanding of "born with it".
We all go through life with flawed mental models, which aren't perfect but are better than a complete lack of understanding. Over time, we make those models in the general consciousness better, even if they continue to lag behind true scientific understanding.
I'd say just the opposite. Zombie Feynman's point isn't "lay off the popularizers". It's that the essence of science is testing ideas.
I haven't read a Gladwell book, let alone bothered to run down all the relevant science. But if Gladwell's legion of critics are right, then he's doing the opposite of Mythbusters. They are teaching people to experiment; he's turning ideas into appealing stories without any interest in testing them.
I haven't read a Gladwell book, let alone bothered to run down all the relevant science. But if Gladwell's legion of critics are right, then he's doing the opposite of Mythbusters. They are teaching people to experiment; he's turning ideas into appealing stories without any interest in testing them.
The difference is that Mythbusters is doing their own (informal,just for fun) scientific investigations, while Gladwell is selectively misrepresenting other people's (serious, rigorous) scientific investigations so people take his essays more seriously.
"Misrepresenting" is your postulation perhaps. Either way the spirit of backing up your thesis with peer reviewed research is totally fine in my book.
The citations aren't there to trick people into thinking things are more serious than they are. They are meant as foundation and support to the theses of the essays. What more could you ask for? Seems ad hominem.
The citations aren't there to trick people into thinking things are more serious than they are. They are meant as foundation and support to the theses of the essays. What more could you ask for? Seems ad hominem.
The citations don't actually support the theses [1], so they make a pretty poor foundation.
[1] An example that Chabris points out: "The Law of the Few."
[1] An example that Chabris points out: "The Law of the Few."
Although I appreciate Gladwell's argument that he combines anecdotal narrative and scientific observation (usually he provides the former, someone else the latter), I don't think that the average reader of the books is clear on the limitations of this approach. I'm afraid many people come away that Gladwell has discovered some brand new, proven thing, and instead of taking the book as a series of illustrations of an idea Gladwell finds compelling and wants to share, they take it as settled science performed meticulously by the man. That's not a criticism that is limited to Gladwell, either - many 'pop science' books do the same, and are mistakenly understood as hard science. I try to point out problematic articles and books whenever I can, but it's not easy to convince someone that really wants to believe, for instance, that adversity (or borderline insanity) is the driver of creativity. And don't get me started on the misappropriation of neuroscience research.
> I don't think that the average reader of the books is clear on the limitations of this approach
The average reader may not, but at least they've been compelled to pick up a book, read it, synthesize it and draw their own conclusions - either agreeing or otherwise. That's better than nothing.
An above-average reader (whatever that is) might read it and be annoyed. A below-average reader might read it and either not understand it and take it as gospel.
What's the problem? Truly.
The average reader may not, but at least they've been compelled to pick up a book, read it, synthesize it and draw their own conclusions - either agreeing or otherwise. That's better than nothing.
An above-average reader (whatever that is) might read it and be annoyed. A below-average reader might read it and either not understand it and take it as gospel.
What's the problem? Truly.
"at least they've been compelled to pick up a book, read it, synthesize it and draw their own conclusions"
I suspect it's much more common that readers believe they have received a reasonable review of the topic. Gladwell's critics are saying that he presents his work as such, but it is not. There's the problem.
I suspect it's much more common that readers believe they have received a reasonable review of the topic. Gladwell's critics are saying that he presents his work as such, but it is not. There's the problem.
I wrote what I think the problem is: "instead of taking the book as a series of illustrations of an idea Gladwell finds compelling and wants to share, they take it as settled science performed meticulously by the man."
I think people often aren't synthesizing and drawing their own conclusions (it is generous to apply this IMHO uncommon quality to the average departure-lounge-pick-up reader). They are reading Gladwell like a textbook.
I think people often aren't synthesizing and drawing their own conclusions (it is generous to apply this IMHO uncommon quality to the average departure-lounge-pick-up reader). They are reading Gladwell like a textbook.
If that's the case then Gladwell isn't at fault – the reader is.
We already have gospel that readers (good ones, not just below-average ones) don't have comprehension of but still believe as faith. Do we really need more?
Responding to reasoned concerns about rigor with ad hominems in the Internet age is not a recipe for success.
> My point was that the people who read my books appreciate this. They are perfectly aware of the strengths and weakness of the narrative form. They know what a story can and can’t do, and they understand that narratives sometimes begin in one place and end in another.
The point is that they don't and they aren't and they can't - many people believe that Gladwell is more than a storyteller, and take his scientific gallavantry as truth. It's irresponsible to ignore that fact, but it's necessary for Gladwell's business model to do so - people read his works primarily not because he is a good storyteller, but because they believe he is telling the truth, as backed up by science.
> My point was that the people who read my books appreciate this. They are perfectly aware of the strengths and weakness of the narrative form. They know what a story can and can’t do, and they understand that narratives sometimes begin in one place and end in another.
The point is that they don't and they aren't and they can't - many people believe that Gladwell is more than a storyteller, and take his scientific gallavantry as truth. It's irresponsible to ignore that fact, but it's necessary for Gladwell's business model to do so - people read his works primarily not because he is a good storyteller, but because they believe he is telling the truth, as backed up by science.
There is something ugly about Gladwell's response. Gladwell insinuates that Chabris has a rabid vendetta against Gladwell and is trying to make a name for himself as an iconoclast. It might be true, but I would have liked the response to have taken a high road.
1) He emphasizes that Chabris has written <bold>three</bold> critiques of him as of date and will probably write a fourth
2) He calls the request for debate silly, making Chabris look agressive.
3) Notes that "I clearly drive Chabris crazy"
Everybody loses when the conversation turns personal.
1) He emphasizes that Chabris has written <bold>three</bold> critiques of him as of date and will probably write a fourth
2) He calls the request for debate silly, making Chabris look agressive.
3) Notes that "I clearly drive Chabris crazy"
Everybody loses when the conversation turns personal.
I think Chabris should "calm down" as soon as Gladwell starts publishing his works under the "science fiction" genre.
Is there anything wrong with reading and enjoying Gladwell for his prowess as a storyteller? And not reading his books as scientifically rigorous works? Whatever you feel about Gladwell, he is an extremely effective storyteller.
Of course not, the problem is that he presents himself and his books as authoritative and accurate, and a lot of people believe him.
One of the points that always seems to be lost in these criticisms and rebuttals and debates is that Gladwell spawns conversion. And that's a very good thing. Nothing should be taken as sacrosanct on either side. It's good to prod and examine these issues and bring more views to the table. Hopefully that leads to more questions... not dogma.
Spawning conversations on the basis of poor or misleading evidence is not a worthwhile endeavor. While nothing should be taken as sacrosanct, one needs to have solid evidence or a reasonable argument before they start prodding and examining issues.
Just saying "Hey, what if?" is not an incredible benefit to society if the claims are not properly testable.
Just saying "Hey, what if?" is not an incredible benefit to society if the claims are not properly testable.
I respectfully disagree. Dialog is a good thing. He may very well be wrong at times though I don't believe he's attempting to mislead people.
Personally, I never want to stop saying, "What if". That question has led to amazing things.
Personally, I never want to stop saying, "What if". That question has led to amazing things.
Politicians do this all the time. Fox News and the mainstream media make their careers out of it. I think we as a society have forgotten how to deconstruct an argument and fact check the specifics.
Deconstructing arguments is difficult, and fact-checking is incredibly time consuming with often little immediately benefit (besides the smug sensation of knowing more that it often provides).
This is why popularizers and journalists have an incredible responsibility to provide this deconstruction and fact checking on behalf of the public, and it's why there is so much outrage, particularly from the scientific community, when they fail.
This is why popularizers and journalists have an incredible responsibility to provide this deconstruction and fact checking on behalf of the public, and it's why there is so much outrage, particularly from the scientific community, when they fail.
It's not hard to start a conversation. Doing so by using selective data and unsupported conclusions is not a good thing.
> Kahneman understood my book. Why couldn’t these guys?
Writers should not be blaming their readers for misconceptions and misunderstandings.
> The first striking thing about all three of Chabris’ reviews of David and Goliath is how much attention he pays to a study that I mention at the beginning of my chapter on dyslexia.
Many of the readers of pop-science books do not know how to properly read research papers. They don't have access to the libraries; they don't have the stats knowledge; they don't know how to pick out the decent studies from the poor studies.
It'd be good if pop-sci writers were doing this work for the readers, or at least helping readers to do the work.
Writers should not be blaming their readers for misconceptions and misunderstandings.
> The first striking thing about all three of Chabris’ reviews of David and Goliath is how much attention he pays to a study that I mention at the beginning of my chapter on dyslexia.
Many of the readers of pop-science books do not know how to properly read research papers. They don't have access to the libraries; they don't have the stats knowledge; they don't know how to pick out the decent studies from the poor studies.
It'd be good if pop-sci writers were doing this work for the readers, or at least helping readers to do the work.
I agree - I think criticism of a book like this should both acknowledge that it tells a compelling story and may open readers' eyes to new ideas — while at the same time pointing out that those ideas don't necessarily have good scientific grounding because of reasons A, B, and C that the reader would not otherwise encounter, not being scientifically minded or not having access to relevant literature.
Chabris dismisses the Alter & Oppenheimer study because no "exact" replications have been done that "he has found," but then proceeds to give weight to a similar study, which failed to replicate the findings, even though that study itself was not an exact replica, because the test questions were not appropriate for the sample:
"In my review, I criticized Gladwell for describing this experiment at length without also mentioning that a replication attempt with a much larger and more representative sample of subjects did not find an advantage for difficult typefaces. One of the original study's authors wrote to me to argue that his effect is robust when the test questions are at an appropriate level of difficulty for the participants in the experiment, and that his effect has in fact been replicated “conceptually” by other researchers. However, I cannot find any successful direct replications—repetitions of the experiment that use the same methods and get the same results—and direct replication is the evidence that I believe is most relevant."
I'm not sure who to believe here. Legitimate criticism exists, but for every successful author, there is one who seeks to piggy back on their fame by attacking it, and for every sensational scientific study, there is more to be gained by disproving it than by replicating it.
"In my review, I criticized Gladwell for describing this experiment at length without also mentioning that a replication attempt with a much larger and more representative sample of subjects did not find an advantage for difficult typefaces. One of the original study's authors wrote to me to argue that his effect is robust when the test questions are at an appropriate level of difficulty for the participants in the experiment, and that his effect has in fact been replicated “conceptually” by other researchers. However, I cannot find any successful direct replications—repetitions of the experiment that use the same methods and get the same results—and direct replication is the evidence that I believe is most relevant."
I'm not sure who to believe here. Legitimate criticism exists, but for every successful author, there is one who seeks to piggy back on their fame by attacking it, and for every sensational scientific study, there is more to be gained by disproving it than by replicating it.
Ugh:
"The kinds of people who read books in America seem to have no problem with my writing. But I am clearly a bee in the bonnet of some of the kinds of people who review books in America."
Which I interpret as: Because I have high sales figures, that clearly proves that experts are easily upset fussbudgets who you shouldn't listen to.
"The kinds of people who read books in America seem to have no problem with my writing. But I am clearly a bee in the bonnet of some of the kinds of people who review books in America."
Which I interpret as: Because I have high sales figures, that clearly proves that experts are easily upset fussbudgets who you shouldn't listen to.
[deleted]
"I was simply saying that all writing about social science need not be presented with the formality and precision of the academic world. There is a place for storytelling, in all of its messiness."
Chabris didn't say that Gladwell was informal or merely imprecise. He said he cited studies that have been proven wrong in the service of a story.
In other words, Gladwell did not merely present a scientific conclusion in an informal, lightly supported manner. He presented false conclusions as if they were true, invoking the authority of "science", because it told a better story.
That is an abuse of science and very worthy of scientists being a little "uncalm" about it.