Why I retracted my Nature paper: correcting the scientific record(retractionwatch.wordpress.com)
retractionwatch.wordpress.com
Why I retracted my Nature paper: correcting the scientific record
http://retractionwatch.wordpress.com/2013/06/19/why-i-retracted-my-nature-paper-a-guest-post-from-david-vaux-about-correcting-the-scientific-record/
13 comments
From the horse's mouth: only 6 out of 53 "landmark" oncology studies were significantly reproducible [1]. Bayer did a similar study, found that they couldn't reproduce the results of about 2/3s of the papers they tried to use for drug discovery R&D [2].
This is a near universal consequence of academia's incentives and extends into many of the sciences (although biology is most susceptible due to its complexity).
[1] http://www.nature.com/nature/journal/v483/n7391/full/483531a...
[2] http://www.nature.com/nrd/journal/v10/n9/full/nrd3439-c1.htm...
This is a near universal consequence of academia's incentives and extends into many of the sciences (although biology is most susceptible due to its complexity).
[1] http://www.nature.com/nature/journal/v483/n7391/full/483531a...
[2] http://www.nature.com/nrd/journal/v10/n9/full/nrd3439-c1.htm...
>This is a near universal consequence of academia's incentives and extends into many of the sciences (although biology is most susceptible due to its complexity).
It is not simply a matter of incentives (although they play a part). Another issue is pure publication bias. If you get 'landmark' results, a paper is much more likely to publish then if you got the results everyone would expect. In many ways this is one of the necessary functions of the journals, as it allows people to look at the new stuff without sifting through many articles of expected stuff, but it does create a bias.
Add to that the incentives of needing to get published, and constrained resources, and a willingness to publish what could be sub-par methodologys (of course they are par), and it is clearly not the best system.
It is not simply a matter of incentives (although they play a part). Another issue is pure publication bias. If you get 'landmark' results, a paper is much more likely to publish then if you got the results everyone would expect. In many ways this is one of the necessary functions of the journals, as it allows people to look at the new stuff without sifting through many articles of expected stuff, but it does create a bias.
Add to that the incentives of needing to get published, and constrained resources, and a willingness to publish what could be sub-par methodologys (of course they are par), and it is clearly not the best system.
From a Bayesian perspective, papers that are demonstrating phenomena that are both (1) novel and (2) significant are the ones that are most likely to be false positives. If you have a journal built around publishing (1) and (2), you're bound to have an excess of studies that can't be reproduced.
That's a very insightful way of looking at it. If Nature wanted to get out of this unfortunate position, it would have to counteract this with stricter requirements on the articles that it publishes. It could require that papers be replicated before publication; it could require the release of any relevant code and data; it could be really diligent about publishing failure-to-reproduce papers. This would be a hassle, but would probably make Nature more prestigious, so I'm sure they could get a lot of researchers to go along with whatever requirements they want.
Nature's and Science's shenanigans are not publishing papers that turn out to be wrong, it's that (1) when contrary data comes up they refuse to publish it, as in this case, and (2) really questionable editorial policies, where articles get peer-reviewed by the wrong audience, and on-the-face obviously-wrong to anybody in the field papers still get published.
Every journal publishes novel and statistically significant findings, and probably encounter similar rates of type 1 error as Nature and Science. It's just that Nature and Science are more susceptible to human politicking and don't seem to face up to it.
Every journal publishes novel and statistically significant findings, and probably encounter similar rates of type 1 error as Nature and Science. It's just that Nature and Science are more susceptible to human politicking and don't seem to face up to it.
> This would be a hassle, but would probably make Nature more prestigious...
Prestige is a relative thing, and there's nothing more prestigious than Nature (or Science). Your suggestions are good, but I think they would be more likely to be adopted first by a journal that isn't Nature or Science, that possibly wanted to compete with them.
Prestige is a relative thing, and there's nothing more prestigious than Nature (or Science). Your suggestions are good, but I think they would be more likely to be adopted first by a journal that isn't Nature or Science, that possibly wanted to compete with them.
You're probably right. Nature and Science are Nature and Science; they don't need to improve on the status quo because they're already sitting on top.
I hope some lesser journal becomes greater, and soon.
I hope some lesser journal becomes greater, and soon.
This is true. I join AAAS in order to get access to Science and some of the related journals and have become quite disillusioned at their practice of either hiding, or outright omitting, papers or studies that contrast something that appeared 'first' in their pages. I was always taught that a scientist should never fear being wrong, they should only fear being not right. The discourse and disagreements, the questions and reproduction things. make things more real.
Science publishing is a classical case of the Market for Lemons [1]. The authors are like used car salesman in that they know so much more about the results in their paper than their readers and referees. Even if just a minority of authors abuses this, the value of publishing a paper plummets and honest authors don't bother any longer with the whole process. The well has been poisoned.
[1] http://en.wikipedia.org/wiki/The_Market_for_Lemons
[1] http://en.wikipedia.org/wiki/The_Market_for_Lemons
What I find the most surprising in this story, without having to take sides about the merits of any of the discussed papers, is that the initial refutation of the work of Bellgrau et al. was reviewed by Bellgrau et al.
If this fact is true, then there is a clear conflict of interest: Bellgrau et al. should have declared it and declined to review, or, barring this, the editors should have noticed it and ignored the review.
If this fact is true, then there is a clear conflict of interest: Bellgrau et al. should have declared it and declined to review, or, barring this, the editors should have noticed it and ignored the review.
If a paper is being contradicted by another, it's quite common to send a copy for review to the original authors. Who better to look for holes in a new paper than someone who's expert in the field and about to have a big problem on their hands?
The editors are the ones who make the go/no-go decision. In my field, I've seen papers that are faulty and have been rejected by experts be accepted by major journals. There's a lot that goes into the go/no-go decision by an editor; I don't envy that job.
If the article is correct, then the editors at Nature may have made the wrong decision. Most of us are sufficiently un-expert in the field to make any conclusion.
The editors are the ones who make the go/no-go decision. In my field, I've seen papers that are faulty and have been rejected by experts be accepted by major journals. There's a lot that goes into the go/no-go decision by an editor; I don't envy that job.
If the article is correct, then the editors at Nature may have made the wrong decision. Most of us are sufficiently un-expert in the field to make any conclusion.
I believe in this case, where two outside reviewers recommended acceptance and the original authors recommended rejection, that the appropriate action for the editors was to get the two outsider reviewer's opinion on the original author's report. Even better, a third outside reviewer should be contacted to assess the original author's report.
I'd even call for greater numbers of referees. There are big practical problems with doing so, but the uncertainty associated with the traditional two or three referees is >50% ( sqrt(3)/3 ). An ensemble of 7-8 referees may produce a more reliable result.
Refereeing is already a huge time sink. I don't believe doubling the referee burden is a possibility without a major incentive changes.
Agreed completely. Doing it right takes a lot of time.
Bellgrau deserves the right to know that there was a challenge to his paper, even to answer questions raised in the refutation, but it should all be done in the open after some point.
For those who are not familiar with the topic: retracting a news and views was like giving the journal a (quite deserved) middle finger.
Fascinating issue and the rest of the Retraction Watch website is eye opening!
I rarely read academic journals anymore but I had always assumed the top journals did a good job of filtering out bad papers and quickly retracting those that were later disproven. Like every human activity it is not so simple.
http://retractionwatch.wordpress.com/
Edit: Warning this is blog addictive - I just spent a solid hour reading. It's kinda science gossip but still important.
I rarely read academic journals anymore but I had always assumed the top journals did a good job of filtering out bad papers and quickly retracting those that were later disproven. Like every human activity it is not so simple.
http://retractionwatch.wordpress.com/
Edit: Warning this is blog addictive - I just spent a solid hour reading. It's kinda science gossip but still important.
Now that we have online publishing, and all scientists around the world appear to have email and Web access, we can do much better. Jelte Wicherts is a young professor of psychology who would like to clean up peer review and scientific publication practices in that discipline and other disciplines. In Frontiers of Computational Neuroscience (an open-access journal) he and his co-authors give a set of general suggestions
Jelte M. Wicherts, Rogier A. Kievit, Marjan Bakker and Denny Borsboom. Letting the daylight in: reviewing the reviewers and other ways to maximize transparency in science. Front. Comput. Neurosci., 03 April 2012 doi: 10.3389/fncom.2012.00020
http://www.frontiersin.org/Computational_Neuroscience/10.338...
on how to make the peer-review process in scientific publishing more reliable.
"With the emergence of online publishing, opportunities to maximize transparency of scientific research have grown considerably. However, these possibilities are still only marginally used. We argue for the implementation of (1) peer-reviewed peer review, (2) transparent editorial hierarchies, and (3) online data publication. First, peer-reviewed peer review entails a community-wide review system in which reviews are published online and rated by peers. This ensures accountability of reviewers, thereby increasing academic quality of reviews. Second, reviewers who write many highly regarded reviews may move to higher editorial positions. Third, online publication of data ensures the possibility of independent verification of inferential claims in published papers. This counters statistical errors and overly positive reporting of statistical results. We illustrate the benefits of these strategies by discussing an example in which the classical publication system has gone awry, namely controversial IQ research. We argue that this case would have likely been avoided using more transparent publication practices. We argue that the proposed system leads to better reviews, meritocratic editorial hierarchies, and a higher degree of replicability of statistical analyses."
The practices recommended in Wicherts's article would have made an incident like the earlier retraction from Nature reported here much less likely to be necessary in the first place.
Jelte M. Wicherts, Rogier A. Kievit, Marjan Bakker and Denny Borsboom. Letting the daylight in: reviewing the reviewers and other ways to maximize transparency in science. Front. Comput. Neurosci., 03 April 2012 doi: 10.3389/fncom.2012.00020
http://www.frontiersin.org/Computational_Neuroscience/10.338...
on how to make the peer-review process in scientific publishing more reliable.
"With the emergence of online publishing, opportunities to maximize transparency of scientific research have grown considerably. However, these possibilities are still only marginally used. We argue for the implementation of (1) peer-reviewed peer review, (2) transparent editorial hierarchies, and (3) online data publication. First, peer-reviewed peer review entails a community-wide review system in which reviews are published online and rated by peers. This ensures accountability of reviewers, thereby increasing academic quality of reviews. Second, reviewers who write many highly regarded reviews may move to higher editorial positions. Third, online publication of data ensures the possibility of independent verification of inferential claims in published papers. This counters statistical errors and overly positive reporting of statistical results. We illustrate the benefits of these strategies by discussing an example in which the classical publication system has gone awry, namely controversial IQ research. We argue that this case would have likely been avoided using more transparent publication practices. We argue that the proposed system leads to better reviews, meritocratic editorial hierarchies, and a higher degree of replicability of statistical analyses."
The practices recommended in Wicherts's article would have made an incident like the earlier retraction from Nature reported here much less likely to be necessary in the first place.
Absolutely shameful. Nature editors should step down and issue a public apology immediately.
Er... it's not like this just happened. The retraction being discussed was in 1998.
I'd tend to agree if we had some 3rd party verification of the inability to reproduce the results of the initial study. Right now it's somebody vs somebody else.
I agree that some caution seems advisable.
What I would really like to see in such cases is not so much other people simply trying to reproduce the results, but an investigation into why one group got different results. Certainly it could be for a scientifically uninteresting reason -- ranging from outright fraud, to sloppy experimental technique causing contamination, to arithmetic errors in analyzing the results -- but it could also be for a scientifically interesting reason: perhaps they used cells from a different supplier that differ in some relevant property that nobody had previously noticed, for example.
In short, there might be some interesting discovery to be had, buried underneath the discrepancy. I hate to see us overlook that possibility in our haste to get the "right" answer.
What I would really like to see in such cases is not so much other people simply trying to reproduce the results, but an investigation into why one group got different results. Certainly it could be for a scientifically uninteresting reason -- ranging from outright fraud, to sloppy experimental technique causing contamination, to arithmetic errors in analyzing the results -- but it could also be for a scientifically interesting reason: perhaps they used cells from a different supplier that differ in some relevant property that nobody had previously noticed, for example.
In short, there might be some interesting discovery to be had, buried underneath the discrepancy. I hate to see us overlook that possibility in our haste to get the "right" answer.
Did I read correctly, that the authors of the original paper were allowed to review the rebuttal article, and even worse, their single negative review caused the paper to not be published?
Isn't there some concept of recusal, conflict of interest in allowing a researcher whose results are being challenged to review the challenge, and veto its acceptance? That's absurd on its face.
Isn't there some concept of recusal, conflict of interest in allowing a researcher whose results are being challenged to review the challenge, and veto its acceptance? That's absurd on its face.
Like virtually every other collection of people, the biomedical sciences have their own drama and politics.
It is unfortunate that these occasionally impede the progress of science, and downstream, the eventual treatment of patients. Even so, much good work is done and will continue to be done into the future, despite politics.
It is unfortunate that these occasionally impede the progress of science, and downstream, the eventual treatment of patients. Even so, much good work is done and will continue to be done into the future, despite politics.
These journals like to say they need the exclusivity, prestige and money to support their editorial process. Yet, given the exclusivity, money and prestige, they apparently don't do a great job.
On the other hand, is there a clear alternative? Is there a way to organize the review process such that the power of the editor is reduced, yet the quality is maintained? I'm not sure.
It seems like this would require non-anonymous open review and a some kind of rating/comment system for papers. It's possible that what ends up happening is that it creates more work for the reader in the same way that it takes effort to tease out a products true worth on Amazon. That might not be a bad thing at all.
On the other hand, is there a clear alternative? Is there a way to organize the review process such that the power of the editor is reduced, yet the quality is maintained? I'm not sure.
It seems like this would require non-anonymous open review and a some kind of rating/comment system for papers. It's possible that what ends up happening is that it creates more work for the reader in the same way that it takes effort to tease out a products true worth on Amazon. That might not be a bad thing at all.
The alternative is open post-publication peer review: any scientist that reads a paper in her/his field of expertise may act as a reviewer and publish a rating or review of that paper.
The end result does not create more work for the reader, as something like the average of ratings received by a paper gives in many cases more information than just the name of the journal where it has been published. For example, this Nature paper currently has a rating of 2% on Epistemio, although it has been published in the prestigious Nature: http://www.epistemio.com/p/Jr3gS9VW .
The journal name is a too weak information as compared to individual ratings of papers, as the distribution of the quality of papers from a given journal is very skewed.
See also the papers in http://www.frontiersin.org/Computational_Neuroscience/resear...
The end result does not create more work for the reader, as something like the average of ratings received by a paper gives in many cases more information than just the name of the journal where it has been published. For example, this Nature paper currently has a rating of 2% on Epistemio, although it has been published in the prestigious Nature: http://www.epistemio.com/p/Jr3gS9VW .
The journal name is a too weak information as compared to individual ratings of papers, as the distribution of the quality of papers from a given journal is very skewed.
See also the papers in http://www.frontiersin.org/Computational_Neuroscience/resear...
This will never happen without some incentive for scientist to review papers.
The literature is dauntingly massive, and there is clearly a demand from scientists for an instant method of identifying excellent and important work. Currently, journal name fills this role (however imperfectly).
Anonymous review exists for a reason and doesn't seem to have much to do with the first issue you raise: the owners of top journals collect large economic and intellectual rents from their monopoly ownership of the top journal name. I actually don't think the anonymity of the review has much to do with the issue raised by the OP, and I'd suggest putting it aside.
So you want some sort of rating system for papers. In the long term, this is usually citation count, but in the short term it probably has to be judged by a trusted human being. Right now, the journal editors fill this role not so much because they are particularly smart, but because their monopoly on the name ensures that all the best work will be sent to them.
Note, however, that just asking the smart scientists what they're reading is not enough. Reviewing articles, and coordinating multiple reviews, is hard work. Right now, the editors get compensated in cash (derived mostly on their name monopoly) and the scientists get compensated with the warm fuzzy idea that they helped the community, and maybe with some back scratching on their own paper from the editor in the future.
A potential solution needs to figure out which humans are going to be assessing papers for extremely important (which needn't be the same thing as the peer review), how they will be compensated, where the money will come from if they are compensated in cash, how to deal with the perverse effects of cash incentives, and how to prevent any small group of humans from controlling the process. This is not an easy problem.
Anonymous review exists for a reason and doesn't seem to have much to do with the first issue you raise: the owners of top journals collect large economic and intellectual rents from their monopoly ownership of the top journal name. I actually don't think the anonymity of the review has much to do with the issue raised by the OP, and I'd suggest putting it aside.
So you want some sort of rating system for papers. In the long term, this is usually citation count, but in the short term it probably has to be judged by a trusted human being. Right now, the journal editors fill this role not so much because they are particularly smart, but because their monopoly on the name ensures that all the best work will be sent to them.
Note, however, that just asking the smart scientists what they're reading is not enough. Reviewing articles, and coordinating multiple reviews, is hard work. Right now, the editors get compensated in cash (derived mostly on their name monopoly) and the scientists get compensated with the warm fuzzy idea that they helped the community, and maybe with some back scratching on their own paper from the editor in the future.
A potential solution needs to figure out which humans are going to be assessing papers for extremely important (which needn't be the same thing as the peer review), how they will be compensated, where the money will come from if they are compensated in cash, how to deal with the perverse effects of cash incentives, and how to prevent any small group of humans from controlling the process. This is not an easy problem.
While it is dramatic, these kinds of issues are often used by people who want to deny any scientific subject. They can claim that all research is worthless.
Doesn't that make it even more important that research journals should be transparent and follow their own policies? If top journals were willing to admit mistakes promptly it would be much harder to claim that valid alternative views are being suppressed for political (or other) reasons.
Yes.
I'm not sure I get your point. Are you saying that we shouldn't be discussing problems of scientific methodology, since it strengthens the people who claim there are problems with scientific methodology?
No, I did not mean that. But blowing it all out of proportion is not good either. It was just a reminder that such uses happen. People don't have a balanced exposure to science.
Yes, there is a tendency to believe that the most recent thing published represents "the last word" on the subject. A balanced exposure to science would include an historical view of the contingency of scientific knowledge. But this could only be acquired by reviewing all the published literature.
Deniers like that publish in publications with a lot less integrity and peer review than the journal Nature, so they shouldn't use incidents like this as a basis for their delusional beliefs.
[deleted]
I don't "deny any scientific subject", nor do I think any research is worthless, but over the last few years as these sorts of stories have come to light more, I have become convinced that publication and peer review as currently manifested are worthless.
> it showed that expression of CD95L on Sertoli cells in allogeneic mismatched testes tissue transplanted under the kidney capsule was able to induce apoptosis of invading cytotoxic T cells
I imagine the way reading this made me feel is similar to how an average person feels reading about complex computer science issues.
I imagine the way reading this made me feel is similar to how an average person feels reading about complex computer science issues.
Basically it's just saying that if you take a particular genetic sequence and introduce it into foreign cells, you can put them into a body where you'd expect an immune response (t cells), but in fact you actually see those cells die instead (apoptosis).
It's too bad it didn't turn out to be true as it would have been an incredible advancement in transplantation.
It's too bad it didn't turn out to be true as it would have been an incredible advancement in transplantation.
Yes, exactly. It's a pretty straightforward statement -- if you know what the words mean, and why it's significant. If you don't have that background knowledge, it's completely opaque.
[deleted]
These high profile journals serve the purpose of publicizing discoveries that are thought to have far-reaching interest in science. However, at the same time, they have the strictest limitations on publications, meaning that the methods section, the foundation of science, are rarely described in any detail. So they have some of the worst practices for publication while supposedly chronicling the most important discoveries.
So while Nature/Science papers make the news and make people's careers, it's important to downweight the certainty of the finding until there is independent verification.