Retraction of Roomtemperature superconductivity in a carbonaceous sulfur hydride(nature.com)
nature.com
Retraction of Roomtemperature superconductivity in a carbonaceous sulfur hydride
https://www.nature.com/articles/s41586-022-05294-9
14 comments
> It seems to be that, in a collaboration between three graduate students, some background data which was foundational to the interpretation of other findings may have been fabricated.
This is a gross misinterpretation. The authors were forthcoming that they used a model of the background conditions instead of direct measurements, as is standard practice for this type of experiment, and they stand by their results.
This is a gross misinterpretation. The authors were forthcoming that they used a model of the background conditions instead of direct measurements, as is standard practice for this type of experiment, and they stand by their results.
They were not at all forthcoming. In the original paper they stated:
" The background signal, determined from a
non-superconducting C–S–H sample at 108 GPa, has been subtracted from the
data."
Then after questions were raised, they published https://arxiv.org/abs/2201.11883v1, where they say:
"We note here that we did not use the measured voltage values of 108
GPa as the background."
If the second statement is true then, at best, the first statement is highly misleading. Furthermore, they have not been at all forthcoming about how exactly that background subtraction was performed - despite repeated requests for clarification. I challenge anyone to understand the background subtraction methodology that they describe in the second paragraph of page 2 in the article linked above.
In my mind, what is at question here is not only the validity of the background subtraction, but the validity of the raw data itself. If the raw data is valid, then why are they unable to show how to go from raw data to published data?
In my mind, what is at question here is not only the validity of the background subtraction, but the validity of the raw data itself. If the raw data is valid, then why are they unable to show how to go from raw data to published data?
Their explanation of the background subtraction methodology seems pretty clear to me, and I don't see how their statement in the first paper could be construed as misleading nonetheless deceitful.
They are able to show how they went from the raw data to the published. Unless you are claiming that the raw data itself is completely made up (in which case why not just make up data that gives the result they want with a different background subtraction method?) then I don't see how the validity of the raw data is in question.
They are able to show how they went from the raw data to the published. Unless you are claiming that the raw data itself is completely made up (in which case why not just make up data that gives the result they want with a different background subtraction method?) then I don't see how the validity of the raw data is in question.
Here is the text in question:
"We selected the background after carefully investigating the temperature dependence of the non-superconducting CSH sample at 108 GPa, the closest pressure prior to the superconducting transition. We note here that we did not use the measured voltage values of 108 GPa as the background. We use the temperature dependence of the measured voltage above and below the Tc of each pressure measurement and scale to determine a user defined background (Fig. 2a). The scaling is such that one achieves an approximately zero signal above the transition temperature; the subtracted background isolates the signal due to the sample."
I challenge you to actually repeat what they did using that description. It is not a complete description.
And no, they did not "show how they went from the raw data to the published". Just because they said they did, doesn't mean they did.
The raw data is in question because it's impossible to understand how subtracting two noisy data sets would produce data that is a combination of a spline and digitized data.
"We selected the background after carefully investigating the temperature dependence of the non-superconducting CSH sample at 108 GPa, the closest pressure prior to the superconducting transition. We note here that we did not use the measured voltage values of 108 GPa as the background. We use the temperature dependence of the measured voltage above and below the Tc of each pressure measurement and scale to determine a user defined background (Fig. 2a). The scaling is such that one achieves an approximately zero signal above the transition temperature; the subtracted background isolates the signal due to the sample."
I challenge you to actually repeat what they did using that description. It is not a complete description.
And no, they did not "show how they went from the raw data to the published". Just because they said they did, doesn't mean they did.
The raw data is in question because it's impossible to understand how subtracting two noisy data sets would produce data that is a combination of a spline and digitized data.
Disappointed but not surprised to see graduate students thrown under the bus like that. They're an easy target for blame: powerless and also by the time the retraction rolls around, not there anymore. Very easy to blame someone who can't hit back.
Just to clarify, the only person suggesting the graduate students were responsible is the OP. Neither the retraction notice nor the Science news article (https://www.science.org/content/article/something-seriously-...) point to the graduate students.
In fact the Science article points to a more senior member of the team as also being a co-author on a recently retracted article that was retracted for reasons related to the same kind of data.
Moreover, check out the quote from Eremets at the end of the Science article regarding principal investigator Dias' more recent, even more groundbreaking claims: "How is this possible? Everything he touches turns to gold."
In fact the Science article points to a more senior member of the team as also being a co-author on a recently retracted article that was retracted for reasons related to the same kind of data.
Moreover, check out the quote from Eremets at the end of the Science article regarding principal investigator Dias' more recent, even more groundbreaking claims: "How is this possible? Everything he touches turns to gold."
To be clear, in either circumstance i'm not throwing the graduate students under the bus. I'm speaking about the broader structural problems which lead to this sort of pressure-induced-error(?). Fabrication in this case can be as far as invented, to applying the wrong background separation, but it fits/looks good, and no more rigor is applied for expedience.
Other than more carefully noticing, “hmm this data looks like a polynomial curve, maybe it’s fabricated?” I’m not sure what you propose the professor do instead.
The professor is not going to duplicate all of the experimental measurements done by the students and it seems totally natural to me to trust the student is not committing fraud, until there is evidence otherwise.
The professor is not going to duplicate all of the experimental measurements done by the students and it seems totally natural to me to trust the student is not committing fraud, until there is evidence otherwise.
Duplication of work prior to publication is standard in many experimental fields, like synthetic chemistry. In some cases the cost of an experiment is high enough to be a problem, but conductivity measurements are not really that exotic.
Duplicating everything is unnecessary. Duplicating the thing which is foundational to a groundbreaking discovery is necessary. Beyond fraud there could be just an honest mistake or some fluke confounding factor.
Maybe there should be a "reproducability risk" metric, calculated with the transitive closure of all published results that reference a paper that has yet to be reproduced. This could help researchers calibrate how much they should trust a result and indicate when a foundational paper really needs to be reproduced.
Taking the professor at the word that it's those graduate students' fault (who very conveniently can't defend themselves and haven't been at the University since 2020) and not the fault of the professor is a bit strange. It might after all be their fault, but it could have easily been some manipulation done or demanded by the PI who wanted a publication.
What I'm trying to say is, I'm not willing to take the professor at their word here when their word is so extremely convenient for them.
What I'm trying to say is, I'm not willing to take the professor at their word here when their word is so extremely convenient for them.
Not unsolvable - just difficult. Systematic problems driven by social incentives are very difficult to solve. But they can be solved. Especially if the institutions in question get behind solving them.
In the case of science, a lot of the bad incentives are created by a combination of the institutions (university administrations and granting agencies) and by the scientists themselves. It'll take both working together to change the incentives and solve the problems. But it can be done.
In the case of science, a lot of the bad incentives are created by a combination of the institutions (university administrations and granting agencies) and by the scientists themselves. It'll take both working together to change the incentives and solve the problems. But it can be done.
Fabricating data is fraud. The reputation of the university is harmed. The public is harmed by potentially providing grants for follow up research. Other researchers are harmed by wasting their time. It seems like one potential fix would be to prosecute people who perpetrate such fraud.
Who should you prosecute in this case? The PI? The students?
Prosecuting the PI for fraud might be possible. I'm not able to google for cases where that has been done, but I'll bet it's occurred before. I'm doubtful that the student could be prosecuted.
One basis for a fraud claim would be the statement that accompanies most grant applications, generally of the form, "X is the PI of the proposed work, and has the responsibility to ensure the quality of the investigation and scientific results..." -- and the PI signs their name, and takes the money. (And so does a university official.)
One example: https://research.wustl.edu/about/roles-responsibilities/prin...
One basis for a fraud claim would be the statement that accompanies most grant applications, generally of the form, "X is the PI of the proposed work, and has the responsibility to ensure the quality of the investigation and scientific results..." -- and the PI signs their name, and takes the money. (And so does a university official.)
One example: https://research.wustl.edu/about/roles-responsibilities/prin...
[deleted]
If you haven't worked in science, please keep in mind that the literature has never been seen as holy writ. Wrong stuff gets published all the time. You have to read critically.
Not excusing sloppy work or whatever it was. Just pointing out to any laypeople out there that science has never had (and never required) a perfect professional literature. Far from it.
Not excusing sloppy work or whatever it was. Just pointing out to any laypeople out there that science has never had (and never required) a perfect professional literature. Far from it.
My copy of the CRC Handbook of Chemistry and Physics (basically an encyclopedia of scientific facts. Things like melting points, conductivity, etc.) has an opening from the current editor explaining that when he took over as editor he tried to track down the sources of a lot of the data. Unfortunately a lot of the sources for the data wasn’t listed, missing, didn’t seem to exist, or existed but was incorrect. So he undertook a program of re-doing a lot of the studies and finding new, accurate sources. Definitely gave me sort of cyberpunk vibes of “this data just showed up one day and nobody knows where it came from.”
Edit: I’ve also run across cases where data is just incorrect and you don’t know how. E.g. a chemical ordered form a supplier is listed as having a solubility of X you the most you can ever get to dissolve is Y and X is something like 13.7 times higher than Y so it’s not like a simple misplaced decimal.
Edit: I’ve also run across cases where data is just incorrect and you don’t know how. E.g. a chemical ordered form a supplier is listed as having a solubility of X you the most you can ever get to dissolve is Y and X is something like 13.7 times higher than Y so it’s not like a simple misplaced decimal.
Not just that, but it would be impossible to achieve. One study should never be taken as truth - but rather a single data point among many. It's only when we have many data points that give us a trend line that we can begin to say we know something.
Right, any casual reader who wants good information should look for meta analyses. You should almost never draw conclusions from a single study.
Retractions are a healthy part of the process -- they shouldn't be frowned upon to the extent that they are. Me and my team just submitted a paper outlining a system we're using at scite to detect retractions more quickly from the titles themselves (instead of manual curation, or relying on publisher updates, which can be spotty / inconsistent). Happy to share details if anyone is interested, since that data will be open for free.
email: ashish AT scite.ai
email: ashish AT scite.ai
That was a bit expected, and the reason given by Nature are not the whole story.
Some of the same authors also claimed to have found metallic hydrogen [0], which is also highly controversial since the proof hold mainly on pictures taken using a Iphone camera.
[0] https://www.science.org/doi/full/10.1126/science.aal1579
[0] https://www.science.org/doi/full/10.1126/science.aal1579
This is why it’s best to adopt a skeptical attitude about published science at least until after a reasonable time period has passed for general scrutiny and retraction but maybe always. This is especially important if the subject is exciting, surprising, relates to some political controversy, or anything else that tends to provoke motivated reasoning.
Honestly, we too often forget that science works in the aggregate. There are far, far too many things that can go wrong in any individual study - from biases, to mistakes, to outright fraud, to just bad luck and weird statistical flukes. It's only after we've had several replications that we can really say we think something might be true, and only after many replications that we can say we "know" something.
This aspect of science seems to have been lost both in the public's imagination and - all too often - in the institutions own understanding and approach.
This aspect of science seems to have been lost both in the public's imagination and - all too often - in the institutions own understanding and approach.
This short article in Science [1] give more information and summarize the whole story.
[1] https://www.science.org/content/article/something-seriously-...
[1] https://www.science.org/content/article/something-seriously-...
Not sure how Nature lets through a paper like this without mandating the raw data is included and a detailed protocol to reproduce the results.
I don't think I've ever seen a paper that included much raw data. It's possible this may be a good suggestion, but it's not the way things are currently done.
Many journals (including a lot of the nature ones) are requiring data and code on publication now
Part of the problem is that as far as I can tell it’s on the reviewers to flag noncompliance, and a lot of times groups won’t actually publish code and datasets before the paper is accepted and assigned a DOI.
So really it’s a culture change around the whole publishing process that’s needed, IMO
Part of the problem is that as far as I can tell it’s on the reviewers to flag noncompliance, and a lot of times groups won’t actually publish code and datasets before the paper is accepted and assigned a DOI.
So really it’s a culture change around the whole publishing process that’s needed, IMO
Maybe not typically, but room temp superconductivity is a nobel prize level claim, and it seems that the experiment is not fully specified without the raw data.
The scientific community generally does a pretty good job of shooting down big claims like this that get published (see: Pons and Fleischmann), so what's the huge benefit? On top of this, if it's to have any value, the reviewers of the article would need to recreate whatever code was used to process the raw data OR audit the existing code to make sure it made sense, both of which are pretty heavy duty tasks. Reviewers don't get paid, so this is a good way to ensure no one wants to review articles for your journal.
If there is substantial interest in the paper after it has been published, it would be a lot easier to verify with the raw data than without. Nothing about the peer review process necessarily needs change beyond immediately rejecting all papers that do not include the minimum amount of data to validate their claims (even if said data is simply stored and not immediately reviewed).
Yesterday (before this came up on HN) I came across in my daily paper chase a follow up paper from the lead author Ranga. Looks like the same fossil superconductor, but they manage to add another sensational detail: the superconductor is composed of dissociated molecules(?!!!). Truly shades of another Hendrik Schon[0].
https://pubs.rsc.org/en/content/articlelanding/2022/cc/d2cc0...
So far, it has not been retracted.
One wonders whether anybody at the journals ever glances at the maxim "If it's too good to be true, have someone check the data."
[0]https://en.m.wikipedia.org/wiki/Sch%C3%B6n_scandal
Footnote added: the toogoodtobetrue bit in the original paper was that the transitions were a bit too sharp for a real superconductor. That was the main thing that got this started.
https://pubs.rsc.org/en/content/articlelanding/2022/cc/d2cc0...
So far, it has not been retracted.
One wonders whether anybody at the journals ever glances at the maxim "If it's too good to be true, have someone check the data."
[0]https://en.m.wikipedia.org/wiki/Sch%C3%B6n_scandal
Footnote added: the toogoodtobetrue bit in the original paper was that the transitions were a bit too sharp for a real superconductor. That was the main thing that got this started.
Reminds of Jan Hendrik Shrön
https://www.youtube.com/watch?v=nfDoml-Db64
https://en.wikipedia.org/wiki/Sch%C3%B6n_scandal
https://www.youtube.com/watch?v=nfDoml-Db64
https://en.wikipedia.org/wiki/Sch%C3%B6n_scandal
Due to the typically discrete nature of the phenomnenon in question, chemistry and molecular-scale physics seems to the worst aspects of the reproducibility crisis in science more broadly. It’s just harder to hide either mistakes or deceptions. The experiments can typically be setup by another single lab hoping to build on the work, rather than requiring the collaboration of countless others.
It’s not a terrible outcome that a paper was published then retracted merely 2 years later as opposed to 1-2 decades later.
It’s not a terrible outcome that a paper was published then retracted merely 2 years later as opposed to 1-2 decades later.
That’s a shame; I hope they’re allowed to come back in a revised article.
…or perhaps Nature felt it was too early for a discovery of this magnitude to be published ;-)
…or perhaps Nature felt it was too early for a discovery of this magnitude to be published ;-)
I’m sure the experiment could be repeated with correct data processing. The authors probably won’t do it, unless they think it will provide a positive result.
That sort of thinking is surely a great loss to scientific progress? You need a few goes by different labs to show failure, so you know that if you were going to try that technique you need to think again, surely?
Sometimes one should probably even do experiments that seem self-evidently futile.
Sometimes one should probably even do experiments that seem self-evidently futile.
[deleted]
I'm not sure why then they couldn't just apply a more standard background subtraction method and show that the result still holds.
Note that this approach was "room temperature" but under very (very) high pressure.
It's not that they did the background subtraction in a non-standard way, it's that they didn't have a measurement of the background to subtract (it being too hard to measure directly), so they instead estimated what the background would look like and subtracted that. Some of their peers are claiming their estimate is bad, but the researchers are sticking by it.
In retracting the work, the editors at Nature were in some way certain "why" they couldn't apply a more standard method.
So they've set up a whole new company on the basis of refuted experimental result. I imagine it will have the ghost of Martin Fleischmann on the board.
> So they've set up a whole new company on the basis of refuted experimental result
Nothing new about that.
Nothing new about that.
Every author of a manuscript should have full faith and knowledge of the results and presented data, but in a massive piece of work such as this, how can you? Imagine for a second, one graduate student doesn't want to face the wrath of an Assistant Professor one day, on purpose or not mishandles data. A year or two later it turns out you have not only wasted your time, but also the time of 500+ citing paper authors, numerous grant proposals, etc. Alongside the Alzheimer's retraction this year, the thirst for impact factor and the problems this causes seems to be yet another unsolvable problem in academia.
[1] https://labsites.rochester.edu/dias/