CRU climate scientists did not withold data(news.bbc.co.uk)
news.bbc.co.uk
CRU climate scientists did not withold data
http://news.bbc.co.uk/1/hi/science_and_environment/10538198.stm
7 comments
You obviously didn't read the next paragraph:
"Writing computer code to process the data "took less than two days and produced results similar to other independent analyses. No information from CRU was needed to do this"."
Personally I can understand them not wanting to "fully comply", given the number of FoI requests they would've received and their tone. Have you seen the vitriol over at Climate Audit?
edit: It gets even better a bit further down: "Asked whether it would be reasonable to conclude that anyone claiming instrumental records were unavailable or vital code missing was incompetent, another panel member, Professor Peter Clarke from Edinburgh University, said: "It's very clear that anyone who'd be competent enough to analyse the data would know where to find it. It's also clear that anyone competent could perform their own analysis without let or hindrance.""
"Writing computer code to process the data "took less than two days and produced results similar to other independent analyses. No information from CRU was needed to do this"."
Personally I can understand them not wanting to "fully comply", given the number of FoI requests they would've received and their tone. Have you seen the vitriol over at Climate Audit?
edit: It gets even better a bit further down: "Asked whether it would be reasonable to conclude that anyone claiming instrumental records were unavailable or vital code missing was incompetent, another panel member, Professor Peter Clarke from Edinburgh University, said: "It's very clear that anyone who'd be competent enough to analyse the data would know where to find it. It's also clear that anyone competent could perform their own analysis without let or hindrance.""
No, you obviously didn't read the paragraph you quoted. It suggests that in addition to withholding data, they also withheld source code. The fact that maybe you could recreate the code or download data from elsewhere does not change the fact that the CRU did withhold both source code and data.
As for the "number of FoI requests they would've received and their tone", that sounds like something apache can handle:
As for the "number of FoI requests they would've received and their tone", that sounds like something apache can handle:
GET /public_data/hockey_stick.tar.gz HTTP/1.1
Host: www.cru.uea.ac.uk/
Tone: HIGHLY VITRIOLICIf the point is to independently verify the results, you should write your own code to grind it and compare it to CRU's results.
anthonyb already quoted the relevant section: Writing computer code to process the data "took less than two days and produced results similar to other independent analyses. No information from CRU was needed to do this".
If a qualified researcher can reproduce the results in two days, I'd suggest that anyone who can't is not sufficiently qualified to be evaluating the research's efficacy in the first place.
anthonyb already quoted the relevant section: Writing computer code to process the data "took less than two days and produced results similar to other independent analyses. No information from CRU was needed to do this".
If a qualified researcher can reproduce the results in two days, I'd suggest that anyone who can't is not sufficiently qualified to be evaluating the research's efficacy in the first place.
I disagree that source code release should not be required, but that's a side issue.
The main point I'm making here is this: the title of this article (and the report) is just not true. The CRU folks withhold source code and data, and the report even admits this.
Not only did they not respond simply because of the "number" or "tone" of requests, but they actively avoided releasing data. From one of their emails: "And don't leave stuff lying around on ftp sites - you never know who is trawling them [...] If they ever hear there is a Freedom of Information Act now in the UK, I think I'll delete the file rather than send to anyone."
The main point I'm making here is this: the title of this article (and the report) is just not true. The CRU folks withhold source code and data, and the report even admits this.
Not only did they not respond simply because of the "number" or "tone" of requests, but they actively avoided releasing data. From one of their emails: "And don't leave stuff lying around on ftp sites - you never know who is trawling them [...] If they ever hear there is a Freedom of Information Act now in the UK, I think I'll delete the file rather than send to anyone."
If the data you need to reproduce their results is publicly available, then it isn't withheld, is it?
The purpose of the three separate inquiries was to separate fact from out-of-context quotes like the ones you're still trying to leverage.
I think the results are clear. The researchers should have been more open, but they didn't do anything unethical and anyone qualified could have reproduced their results using widely publicly available data.
The purpose of the three separate inquiries was to separate fact from out-of-context quotes like the ones you're still trying to leverage.
I think the results are clear. The researchers should have been more open, but they didn't do anything unethical and anyone qualified could have reproduced their results using widely publicly available data.
If I don't give you data, but maybe you can scrape a bunch of websites to get it, then yes, I withheld it. Hence, the title is misleading. The title didn't claim assorted weather stations did not withhold data, it claims the CRU did not. The text of the report admits they did.
As for my "out-of-context" quote, here is the context: http://junkscience.com/FOIA/mail/1107454306.txt . Why don't you explain to me how this was not a conspiracy to withhold data?
As for my "out-of-context" quote, here is the context: http://junkscience.com/FOIA/mail/1107454306.txt . Why don't you explain to me how this was not a conspiracy to withhold data?
> As for my "out-of-context" quote, here is the context: http://junkscience.com/FOIA/mail/1107454306.txt . Why don't you explain to me how this was not a conspiracy to withhold data?
The data they used is publicly available and the inquiry board noted that the results were easily reproduced by qualified individuals.
The fact that CRU didn't want to deal with "the contrarians" by holding their (unqualified) hands through the process is not a surprise.
What more do you want? The inquiry board said CRU should be more open. They should. But anyone qualified could reproduce their results with the public data. Hardly a damning conclusion.
The data they used is publicly available and the inquiry board noted that the results were easily reproduced by qualified individuals.
The fact that CRU didn't want to deal with "the contrarians" by holding their (unqualified) hands through the process is not a surprise.
What more do you want? The inquiry board said CRU should be more open. They should. But anyone qualified could reproduce their results with the public data. Hardly a damning conclusion.
this is the dumbest argument ever.
you lose credibility by interjecting small jabs and opinions (ie, "unqualified") and irrelevant commentary (ie, the relevance of the surprise-factor is...?)
i personally think the whole issue is silly because laymen are not familiar with the typical processes in academia.
but my opinion aside, regardless of its convenience for the scientists involved and regardless of its irrelevance to the validity of the scientific argument, the intention is clear from the above linked emails to hide data from the "contrarians".
case closed.
you lose credibility by interjecting small jabs and opinions (ie, "unqualified") and irrelevant commentary (ie, the relevance of the surprise-factor is...?)
i personally think the whole issue is silly because laymen are not familiar with the typical processes in academia.
but my opinion aside, regardless of its convenience for the scientists involved and regardless of its irrelevance to the validity of the scientific argument, the intention is clear from the above linked emails to hide data from the "contrarians".
case closed.
> this is the dumbest argument ever.
I'd say this argument is dumb-er.
> you lose credibility by interjecting small jabs and opinions (ie, "unqualified")
Your comment is amusing given your 'dumbest argument ever' intro, but regardless, that wasn't a jab (or even my own opinion):
"Asked whether it would be reasonable to conclude that anyone claiming instrumental records were unavailable or vital code missing was incompetent, another panel member, Professor Peter Clarke from Edinburgh University, said: "It's very clear that anyone who'd be competent enough to analyse the data would know where to find it. It's also clear that anyone competent could perform their own analysis without let or hindrance.""
I'd say this argument is dumb-er.
> you lose credibility by interjecting small jabs and opinions (ie, "unqualified")
Your comment is amusing given your 'dumbest argument ever' intro, but regardless, that wasn't a jab (or even my own opinion):
"Asked whether it would be reasonable to conclude that anyone claiming instrumental records were unavailable or vital code missing was incompetent, another panel member, Professor Peter Clarke from Edinburgh University, said: "It's very clear that anyone who'd be competent enough to analyse the data would know where to find it. It's also clear that anyone competent could perform their own analysis without let or hindrance.""
The issue is that most of the people attacking them have already made their mind up and are looking for any opening that they can find. They don't care about the source code or data, and if they were freely available, it would've been something else (eg. hundreds of "climate skeptics" asking stupid questions, or calling CRU over and over). It's like arguing with proponents of Intelligent Design - the more data you give them, the more they can take out of context.
[deleted]
The number and "tone" of the requests is irrelevant.
If they want to ignore or not "fully comply" with FOI requests - they shouldn't take public funds.
If they want to ignore or not "fully comply" with FOI requests - they shouldn't take public funds.
So, if each climate scientist had to field 5000 FOI requests per year, that would be no problem, right? Are you saying it would serve the public interests to spend money paying scientists to research while also requiring them to spend 99% of their time responding to FOI requests?
It seems to me that we all agree that the number and tone of requests is relevant. I think that everyone here will accept that once the number of requests prevents scientists from doing any research, than we have a serious problem. So the question is not "is the number or tone relevant?" but "what fraction of scientist time should we expect researchers to spend responding to FOI requests before we consider them unduly burdensome?"; given this question, tone seems very relevant. If researchers are responding to questions from professionals who have a good understanding of the fundamentals and are scientifically literate, then we should expect them to be more willing to respond to inquiries. But if they're responding to scientifically illiterate people who don't understand the basics of climate modelling, then I don't think we should expect them spend any time at all.
I mean, the FOI process can be abused. I think it is important that we differentiate between using the FOI process as a way of harassing researchers and serious questions from people who understand enough to actually deal with the data. In that sense, requiring people to write their own code (which is better for finding errors anyway) serves as a way to tax the harassers without unduly burdening real scientists. Real scientists will probably have planned on writing their own code anyway, but people abusing the FOI process for harassment probably don't have the capability to do so.
In my own neighborhood, I've watched crazy citizens groups use demands for data as a way to harass experts. They go to town meetings and yell and scream about data. When experts give them the answers they need (which costs time and money), they start poring through the data looking for inconsistencies. Inevitably they find something that looks suspicious if you're completely ignorant of math and science and then they pounce on it at the next town meeting. "Gotcha!" they say. But they're ignorant people who don't understand basic math and they inevitably turn out to be completely wrong. Most of the time, the truth is exactly opposite what they've claimed. Meanwhile, an expert has to explain in great detail why exactly they're wrong. The whole thing is just a huge waste of time and taxpayer money. It doesn't cost them anything to demand data and when any expert fails to give them everything (including the kitchen sink) because they're worried that these ignorant people will misunderstand things, they only scream all the louder.
Public policy is hard. Even very simple questions like "will this development make traffic on this side street worse" is actually fairly tough to answer. The vast majority of people don't have anything close to the skillset needed to intelligently review an analysis of these questions. But many think they do.
It seems to me that we all agree that the number and tone of requests is relevant. I think that everyone here will accept that once the number of requests prevents scientists from doing any research, than we have a serious problem. So the question is not "is the number or tone relevant?" but "what fraction of scientist time should we expect researchers to spend responding to FOI requests before we consider them unduly burdensome?"; given this question, tone seems very relevant. If researchers are responding to questions from professionals who have a good understanding of the fundamentals and are scientifically literate, then we should expect them to be more willing to respond to inquiries. But if they're responding to scientifically illiterate people who don't understand the basics of climate modelling, then I don't think we should expect them spend any time at all.
I mean, the FOI process can be abused. I think it is important that we differentiate between using the FOI process as a way of harassing researchers and serious questions from people who understand enough to actually deal with the data. In that sense, requiring people to write their own code (which is better for finding errors anyway) serves as a way to tax the harassers without unduly burdening real scientists. Real scientists will probably have planned on writing their own code anyway, but people abusing the FOI process for harassment probably don't have the capability to do so.
In my own neighborhood, I've watched crazy citizens groups use demands for data as a way to harass experts. They go to town meetings and yell and scream about data. When experts give them the answers they need (which costs time and money), they start poring through the data looking for inconsistencies. Inevitably they find something that looks suspicious if you're completely ignorant of math and science and then they pounce on it at the next town meeting. "Gotcha!" they say. But they're ignorant people who don't understand basic math and they inevitably turn out to be completely wrong. Most of the time, the truth is exactly opposite what they've claimed. Meanwhile, an expert has to explain in great detail why exactly they're wrong. The whole thing is just a huge waste of time and taxpayer money. It doesn't cost them anything to demand data and when any expert fails to give them everything (including the kitchen sink) because they're worried that these ignorant people will misunderstand things, they only scream all the louder.
Public policy is hard. Even very simple questions like "will this development make traffic on this side street worse" is actually fairly tough to answer. The vast majority of people don't have anything close to the skillset needed to intelligently review an analysis of these questions. But many think they do.
I believe apache is capable of handling 5000 requests per year with very little human intervention.
The CRU folks actively avoided going that route. They avoided putting data on a public ftp site because they didn't want people they disliked to download it.
http://junkscience.com/FOIA/mail/1107454306.txt
The CRU folks actively avoided going that route. They avoided putting data on a public ftp site because they didn't want people they disliked to download it.
http://junkscience.com/FOIA/mail/1107454306.txt
Apache will not answer questions about the data or code. What do you think happens when FOI requests can't make sense of the data/code researchers give them? Either the researchers have to answer questions OR they will be accused of deliberately obfuscating their results. It is a no win situation.
Of course it's a win. They can say "look, it's all there, not our fault you can't understand it." They could even behave like scientists and answer reasonable questions about their work [1].
Answering Frequently Asked Questions with little human interaction is actually a solved problem.
Your arguments might carry some weight if they were flooded with an unreasonable number of FOI requests. Or if they were flooded with a large number of unreasonable and not-repeated questions. None of that actually happened. All that happened is that a few critics asked for the data and they conspired to hide it.
[1] Last month a guy from Princeton emailed me asking for help using code he downloaded from my website. Instead of conspiring to hide it, I emailed him a more recent version of the code told him what to tweak. I'm not even a professional scientist anymore.
Answering Frequently Asked Questions with little human interaction is actually a solved problem.
Your arguments might carry some weight if they were flooded with an unreasonable number of FOI requests. Or if they were flooded with a large number of unreasonable and not-repeated questions. None of that actually happened. All that happened is that a few critics asked for the data and they conspired to hide it.
[1] Last month a guy from Princeton emailed me asking for help using code he downloaded from my website. Instead of conspiring to hide it, I emailed him a more recent version of the code told him what to tweak. I'm not even a professional scientist anymore.
they conspired to hide it
This is a flat out lie. The data is publicly available.
This is a flat out lie. The data is publicly available.
The fact that data is publicly available from other sources does not change the fact that the CRU conspired to hide their data set.
The conspiracy is proven conclusively by the hacked emails. The email linked here is part of the conspiracy: http://junkscience.com/FOIA/mail/1107454306.txt
Seriously, what more would you want in the way of proof? Videotape of Mann/Jones writing the email? A brain scan showing they weren't joking? I'm beginning to have doubts that you are arguing in good faith.
The conspiracy is proven conclusively by the hacked emails. The email linked here is part of the conspiracy: http://junkscience.com/FOIA/mail/1107454306.txt
Seriously, what more would you want in the way of proof? Videotape of Mann/Jones writing the email? A brain scan showing they weren't joking? I'm beginning to have doubts that you are arguing in good faith.
Quote from Phil Jones: “We have 25 years or so invested in the work. Why should I make the data available to you, when your aim is to try and find something wrong with it?”
Maybe "conspired to hide it" is too nuanced.
Maybe "conspired to hide it" is too nuanced.
In theory, yes. In practice, people are only human, and tend to not react well when faced with a vitriolic "tone".
The police may be required to protect and serve, but try berating a police officer and see how well he serves you when your house gets robbed. He shouldn't treat you any differently, but he probably will.
The police may be required to protect and serve, but try berating a police officer and see how well he serves you when your house gets robbed. He shouldn't treat you any differently, but he probably will.
If you're a university researcher in the UK it's impossible not to take public funds. All universities (bar one) are publically funded.
Then don't be a university researcher.
In fact, if you don't want to reveal your top secret source code/data, don't publish in science journals. It's utterly ridiculous that journals still exist which don't require source/data to be released.
In fact, if you don't want to reveal your top secret source code/data, don't publish in science journals. It's utterly ridiculous that journals still exist which don't require source/data to be released.
The data was available.
If you want to independently verify it, the best way is to write your own code and see if it spits out the same/similar results.
If you want to independently verify it, the best way is to write your own code and see if it spits out the same/similar results.
"All universities (bar one) are publically funded."
That's a problem isn't it. Maybe the UK should fix that.
That's a problem isn't it. Maybe the UK should fix that.
If only that logic worked on the DHS...
Indeed. And I did exactly as this report suggests and carried out my own independent analysis, regardless of whether CRU did or didn't make their data/software available.
http://sluggish.homelinux.net/index.php?title=Tempgraph
The station data and temperature data were already out there in the public domain.
http://sluggish.homelinux.net/index.php?title=Tempgraph
The station data and temperature data were already out there in the public domain.
Is it safe to assume from your resulting graph that "warming" in England has been fairly steady from 1750 and the mean temperature actually rose more dramatically from 1750-1850 than in modern times?
Not really. Because of the high variance it's difficult to make claims of that sort. You have to bear in mind that prior to the late 19th century Stevenson screens were not in use, and thermometers themselves were less accurate. Also it's pretty clear that there are systematic biases in the data, meaning that WMO stations on average move closer to the equator over time. I havn't checked whether this bias exists for the UK data, but it does for the overall data set.
You mean like how research 2000 hasn't complied for kos? Its not science if its not open and independently verifiable. Raw data doesn't a computer model make.
In reality it matters little what these self-examinationary enquiries come up with, because the climate scientists have lost the PR war with this one, and that's the crucial court of opinion. A few committed internet posters will point to the results and say 'but they were exonerated' - when in reality the enquiries all endorse the opinion they acted poorly.
We've gone past the high water mark of human-caused climate change now - perhaps in history the runup to Copenhagen will look like the high point for the cause. Governments the world over are turning their back as an issue. There'll always be a committed band of supporters and promoters but once governments see it as a dead issue then everyone else will move on.
The good news in all this is that as the smoke clears from the FUD surrounding climate change, it will allow alternative energy sources to be evaluated on their eficiency merits, rather than how well they link in to some government-invented paper-swapping scheme. Removing the demonisation of C02 will allow people to concentrate on the real environmental costs of any energy source and decide if the cost is worth it or not.
We've gone past the high water mark of human-caused climate change now - perhaps in history the runup to Copenhagen will look like the high point for the cause. Governments the world over are turning their back as an issue. There'll always be a committed band of supporters and promoters but once governments see it as a dead issue then everyone else will move on.
The good news in all this is that as the smoke clears from the FUD surrounding climate change, it will allow alternative energy sources to be evaluated on their eficiency merits, rather than how well they link in to some government-invented paper-swapping scheme. Removing the demonisation of C02 will allow people to concentrate on the real environmental costs of any energy source and decide if the cost is worth it or not.
From the original report [1]:
"It was not our purpose to examine, nor did we seek evidence on, the science produced by CRU. It will be for the Scientific Appraisal Panel to look in detail into all the evidence to determine whether or not the consensus view remains valid."
[1] http://www.desmogblog.com/sites/beta.desmogblog.com/files/ph...
"It was not our purpose to examine, nor did we seek evidence on, the science produced by CRU. It will be for the Scientific Appraisal Panel to look in detail into all the evidence to determine whether or not the consensus view remains valid."
[1] http://www.desmogblog.com/sites/beta.desmogblog.com/files/ph...
But the report does ask for better handling of the blogosphere, plus opening up data and source code: http://blog.jgc.org/2010/07/cce-review-report.html
Good post.
The Guardian has an interesting article from this past Sunday about scientists being more open.
http://www.guardian.co.uk/environment/2010/jul/04/climatecha...
I, for one, welcome this openness, and hope that it leads to the removal of fundamentalists from both sides from the debate. No doubt there is a big problem, and we won't solve it when people are merely shouting at each other.
The Guardian has an interesting article from this past Sunday about scientists being more open.
http://www.guardian.co.uk/environment/2010/jul/04/climatecha...
I, for one, welcome this openness, and hope that it leads to the removal of fundamentalists from both sides from the debate. No doubt there is a big problem, and we won't solve it when people are merely shouting at each other.
The ill effects of burning fossil fuels are myriad and well-documented, from smog, to oil spills, to coal mine disasters, to smog, to ocean acidification.
I'm generally fairly hostile towards the fundamentalists on the other side of the debate becaause it's generally presented as if AGW was the lynchpin without which fossil fuels are a great fuel source.
And further, it's generally presented as if the 'global warming' part were somehow scientifically dubious, when all that is dubious is the 'anthropogenic' part.
So I may be a bit of a fundamentalist, but there are a lot of fundamental reasons fossil fuels are not okay. Nuclear, wind, and solar are where it's at.
I'm generally fairly hostile towards the fundamentalists on the other side of the debate becaause it's generally presented as if AGW was the lynchpin without which fossil fuels are a great fuel source.
And further, it's generally presented as if the 'global warming' part were somehow scientifically dubious, when all that is dubious is the 'anthropogenic' part.
So I may be a bit of a fundamentalist, but there are a lot of fundamental reasons fossil fuels are not okay. Nuclear, wind, and solar are where it's at.
This looks like a whitewash to me. It wasn't just about whether they withheld data or even source code. One of their tactics, revealed by the leaked emails, was to publicly berate the scientists who opposed AGW to publish their arguments in peer reviewed journals. In private they worked to get journal editors fired, ignore journals that published opposite views, discredited and hampered the careers of opponents, etc. More importantly, this whole approach of focusing on one little fact and using that to "prove" there was no conspiracy is like focusing on the glove that would not fit in the OJ trial. If you read the leaked emails the pattern seems pretty clear -- these guys were not engaging in science but politics.
Large amounts of government funding for scientific research create an equally large incentive to produce results which are politically useful (or at least, not politically inconvenient). In the case of AGW, the easy "solution" is massive expansion of government regulation in all spheres of life, as well as creation of new markets for government carbon indulgences, an excellent mechanism for enriching politicians and selected corporate cronies.
Any contrarian viewpoints are excluded both explicitly and implicitly, because ideas that resemble what's already been published in journals are those most likely to get funding. The vast majority of scientists aren't gonna jump off the gravy train and potentially sacrifice their tenure.
Any contrarian viewpoints are excluded both explicitly and implicitly, because ideas that resemble what's already been published in journals are those most likely to get funding. The vast majority of scientists aren't gonna jump off the gravy train and potentially sacrifice their tenure.
[deleted]
The report claims that the CRU didn't withhold data simply because "any independent researcher can download station data directly from primary sources and undertake their own temperature trend analysis". So basically, the CRU withheld their own data/source code, but they didn't cover up other people's data. Huh?
Even the report admits that "there has been a consistent pattern of failing to display the proper degree of openness". This sounds like an attempt at whitewashing, basically giving reporters a justification for ignoring the whole mess.