Explaining to BBC Newsnight some of the problems with the CRU code(news.bbc.co.uk)
news.bbc.co.uk
Explaining to BBC Newsnight some of the problems with the CRU code
http://news.bbc.co.uk/1/hi/programmes/newsnight/8395514.stm
31 comments
All software used in science should be open source, because if it isn't, the science isn't repeatable.
Not true. For instance, if a report is originally typed in a closed-source, commercial text editor, that has no impact on the repeatability of the science. Similarly, Mathematica is (to my knowledge) a closed-source program, but calculations done by it can still be independently verified (so long as those calculations are made public).
You're on the right track, but need to narrow down the wording some.
You're on the right track, but need to narrow down the wording some.
> You're on the right track, but need to narrow down the wording some.
You're right. I was actually thinking that as I typed it.
How about this: when scientists take raw input data and use complex processes to convert it (by complex I mean more than you could do with a pocket calculator in a minute), then the programs doing the conversion must be open source, so that others may repeat the work done in processing.
I'm not familiar with Mathematica; how complex is the work that it does? And how easy is it to verify that its results are correct?
You're right. I was actually thinking that as I typed it.
How about this: when scientists take raw input data and use complex processes to convert it (by complex I mean more than you could do with a pocket calculator in a minute), then the programs doing the conversion must be open source, so that others may repeat the work done in processing.
I'm not familiar with Mathematica; how complex is the work that it does? And how easy is it to verify that its results are correct?
Great presentation John. You made your points very clearly, and it seems very helpful for the non-experts watching.
The only thing they got wrong was where I said I was 'shocked'. I'm not shocked, I'm surprised, and disappointed by the quality of the code.
No, I'm not shocked either. Mathematicians and scientists have no formal training in programming (at least in the UK). They're expected just to pick it up and get on with it.
They don't generally put in the necessary hours like the "self taught programmer" does, because they're generally not interested in programming itself. It's just a calculational annoyance that gets in the way.
They don't generally put in the necessary hours like the "self taught programmer" does, because they're generally not interested in programming itself. It's just a calculational annoyance that gets in the way.
> Mathematicians and scientists have no formal training in programming (at least in the UK). They're expected just to pick it up and get on with it.
That's an unreasonable generalisation.
For one thing, many university courses in science and mathematics do seem to incorporate basic programming training these days, either as part of the course, or via other facilities that the university makes available to those taking such courses. Any programming skills acquired may or may not contribute directly to the grade of degree awarded, but that is a different issue.
For another thing, I can attest that some scientists do make an effort, because I have personally been paid by PhD students to tutor them in the appropriate programming techniques to implement their models soundly.
That's an unreasonable generalisation.
For one thing, many university courses in science and mathematics do seem to incorporate basic programming training these days, either as part of the course, or via other facilities that the university makes available to those taking such courses. Any programming skills acquired may or may not contribute directly to the grade of degree awarded, but that is a different issue.
For another thing, I can attest that some scientists do make an effort, because I have personally been paid by PhD students to tutor them in the appropriate programming techniques to implement their models soundly.
I'm talking from my experience of attending the most highly regarded mathematics undergraduate programme in the UK. There was basic programming training, but the emphasis was on the "basic". There was absolutely no emphasis placed on writing maintainable code. We (in fact "they": I didn't attend the classes) were simply taught some technical details of C. In fact I think the number of hours of teaching time was single figures.
Now I can't speak for the engineers at the same university. As far as I know they were given excellent training in C++. But simply, the mathematicians did not take programming seriously, so I'm not surprised of the low quality of the implemented models.
Now I can't speak for the engineers at the same university. As far as I know they were given excellent training in C++. But simply, the mathematicians did not take programming seriously, so I'm not surprised of the low quality of the implemented models.
> I'm talking from my experience of attending the most highly regarded mathematics undergraduate programme in the UK.
That's a bold claim. Which undergraduate programme would that be, and by whom is it the most highly regarded in the UK?
That's a bold claim. Which undergraduate programme would that be, and by whom is it the most highly regarded in the UK?
I don't know which university the parent is referring to, but I can confirm that this was also the situation at the University of York, a top-10 UK university, at least at the time. (90%+ were people who hadn't quite made it into the top 2 - hence nicknames of "University of Dork" and "University of Oxford/Cambridge Rejects")
The problem existed not only mathematics, where I ended up giving extra programming classes to friends because the teaching was practically non-existant. Also physics along with the computer simulation and theoretical branches of physics, which I happened to be studying. In fact, we were only taught rudimentary Fortran (!) syntax, and laughably, OpenMP and MPI later on. Nobody who made it through the course alive did so without having either learned to program earlier or investing a large amount of time in self-study and then "winging it" - after all, the final project depended on being able to deliver a simulation of whatever physical system was being investigated.
I started programming when I was 10, so I had no trouble; together with a colleague who had a similar programming history as I, we introduced the rest of our year to basic good programming practices and workflows (split your functionality into stand-alone functions and files; use version control; avoid global variables; the merits of code reviews; etc.). A lot of people had already switched to experimental physics before we started doing this. Obviously, expecting people to understand concurrent programming at that level is pretty dangerous. In my experience, this sort of attitude breeds a view that computer programs are magical things, and once they compile and don't crash or hang, you can trust their results. This exact attitude is what I'm reminded of when I read about this CRU incident.
I don't think any of the staff ever took the issue seriously, though. Even if they weren't prepared to teach it, they should have told everyone to teach themselves over the holidays and provide exercises or whatever. At no point did anyone point out that programming was a difficult craft, and that mastery of it would be expected.
For context to those unfamiliar with the majority of UK science degree programmes: they are typically not as modular or flexible as in other countries. If you failed a module, you had one chance to re-sit the exam a few weeks later. There was no option of re-taking the module including lectures during the next term/semester; each term had a pre-assigned set of modules, there was no flexibility in when they could be taken, so you couldn't defer your concurrency project until you'd taken some more programming classes or so.
The problem existed not only mathematics, where I ended up giving extra programming classes to friends because the teaching was practically non-existant. Also physics along with the computer simulation and theoretical branches of physics, which I happened to be studying. In fact, we were only taught rudimentary Fortran (!) syntax, and laughably, OpenMP and MPI later on. Nobody who made it through the course alive did so without having either learned to program earlier or investing a large amount of time in self-study and then "winging it" - after all, the final project depended on being able to deliver a simulation of whatever physical system was being investigated.
I started programming when I was 10, so I had no trouble; together with a colleague who had a similar programming history as I, we introduced the rest of our year to basic good programming practices and workflows (split your functionality into stand-alone functions and files; use version control; avoid global variables; the merits of code reviews; etc.). A lot of people had already switched to experimental physics before we started doing this. Obviously, expecting people to understand concurrent programming at that level is pretty dangerous. In my experience, this sort of attitude breeds a view that computer programs are magical things, and once they compile and don't crash or hang, you can trust their results. This exact attitude is what I'm reminded of when I read about this CRU incident.
I don't think any of the staff ever took the issue seriously, though. Even if they weren't prepared to teach it, they should have told everyone to teach themselves over the holidays and provide exercises or whatever. At no point did anyone point out that programming was a difficult craft, and that mastery of it would be expected.
For context to those unfamiliar with the majority of UK science degree programmes: they are typically not as modular or flexible as in other countries. If you failed a module, you had one chance to re-sit the exam a few weeks later. There was no option of re-taking the module including lectures during the next term/semester; each term had a pre-assigned set of modules, there was no flexibility in when they could be taken, so you couldn't defer your concurrency project until you'd taken some more programming classes or so.
split your functionality into stand-alone functions and files; use version control; avoid global variables; the merits of code reviews
Yes, these are exactly the kinds of things I'm talking about that I have not seen taught in undergraduate mathematics programmes, but they are vital to writing good quality scientific code.
At no point did anyone point out that programming was a difficult craft, and that mastery of it would be expected
Not only difficult, but even experienced programmers introduce serious bugs by accident. Often they find them because they do extensive testing! This is not done to such a degree for scientific code.
Yes, these are exactly the kinds of things I'm talking about that I have not seen taught in undergraduate mathematics programmes, but they are vital to writing good quality scientific code.
At no point did anyone point out that programming was a difficult craft, and that mastery of it would be expected
Not only difficult, but even experienced programmers introduce serious bugs by accident. Often they find them because they do extensive testing! This is not done to such a degree for scientific code.
During my maths BA I took:
Foundation of computer science
Data structures and algorithms (SML)
Software engineering (Java)
Operating systems (C)
Numerical algorithms (C++)
as well as yearly catam projects: http://www.maths.cam.ac.uk/undergrad/catam/
There were also optional classes in bioinformatics and computational statistics and a number of lecturers encouraged the use of programming in solving weekly problem sheets.
I would estimate at least a third of my year-group were fairly competent programmers come graduation.
Foundation of computer science
Data structures and algorithms (SML)
Software engineering (Java)
Operating systems (C)
Numerical algorithms (C++)
as well as yearly catam projects: http://www.maths.cam.ac.uk/undergrad/catam/
There were also optional classes in bioinformatics and computational statistics and a number of lecturers encouraged the use of programming in solving weekly problem sheets.
I would estimate at least a third of my year-group were fairly competent programmers come graduation.
I don't know when you took it, but that's not possible today unless you do the Mathematics with Computer Science option, which most Maths undergraduates do not, and certainly not the ones who are more into the applied side of things and are likely to end up in climate research.
So you meant Cambridge as "the most highly regarded mathematics undergraduate programme in the UK"? I was afraid of that. Cambridge's mathematics research may be world class, but their teaching of undergraduates is very poor.
It helps if you're an undergraduate at one of the more mathematically inclined colleges, of which Trinity is traditionally the strongest. Even then, there is only so much that small group supervisions or example classes can make up for if the basic syllabus and presentation in lectures is poor.
Sadly, both the Mathematics with Computer Science tripos option and the Diploma in Computer Science conversion course have been dropped recently, removing two genuinely useful choices. As far as I can tell, this leaves only the well-intentioned but usually poorly realised CATAM projects as the main source of computational mathematics background for students, and the criticisms of insufficient training in real programming skills are entirely justified there. There is a new MPhil course running from this year in the Computer Lab, but this is more like Part III CompSci than the old Diploma.
(Some of these comments are based on my own experiences studying at Cambridge in the '90s, though my general view is more coloured by the many graduates from Cambridge and elsewhere with whom I have worked since that time.)
It helps if you're an undergraduate at one of the more mathematically inclined colleges, of which Trinity is traditionally the strongest. Even then, there is only so much that small group supervisions or example classes can make up for if the basic syllabus and presentation in lectures is poor.
Sadly, both the Mathematics with Computer Science tripos option and the Diploma in Computer Science conversion course have been dropped recently, removing two genuinely useful choices. As far as I can tell, this leaves only the well-intentioned but usually poorly realised CATAM projects as the main source of computational mathematics background for students, and the criticisms of insufficient training in real programming skills are entirely justified there. There is a new MPhil course running from this year in the Computer Lab, but this is more like Part III CompSci than the old Diploma.
(Some of these comments are based on my own experiences studying at Cambridge in the '90s, though my general view is more coloured by the many graduates from Cambridge and elsewhere with whom I have worked since that time.)
That's unfortunate. Given the nature of modern applied maths it seems like that should be the preferred route.
Quite right. Understandably the applied maths department wants to give a rigorous theoretical training, but rigorous programming training ought to be given at some point before research starts.
Beyond training in programming, scientists creating production code need training in software engineering. The folks over at Software Carpentry (http://software-carpentry.org/) seem to have the right idea.
Just as important, like other forms of scientific experimentation, work based on computer analysis needs to be independently reproducable. A big part of that is to make the source code and data available to other researchers (ideally, to the public at large) and well-documented.
It seems like the CRU people failed in both areas.
Just as important, like other forms of scientific experimentation, work based on computer analysis needs to be independently reproducable. A big part of that is to make the source code and data available to other researchers (ideally, to the public at large) and well-documented.
It seems like the CRU people failed in both areas.
And this poor code leads to temperature data sets that could be flawed - data sets that form the foundation under the climate models and the conclusions of the IPCC reports. These reports in turn are used by many countries to base policy on. Are governments basing policy on flawed science?
I'd still prefer the (UK) government to be basing policy on flawed science than no science ( http://news.bbc.co.uk/1/hi/8334774.stm ).
The unveilings of the CRU materials now also caused Al Gore to cancel his talk at Copenhagen. http://www.examiner.com/x-11224-Baltimore-Weather-Examiner~y...
Al Gore's talk has indeed been cancelled. His spokesperson says the cancellation is because of "all the events going on with the summit", whatever exactly that means. It could be that the real reason is the CRU kerfuffle, but so far as I can tell that's just speculation.
Most of the linked article other than the first two sentences (1) is bullshit and (2) has nothing to do with the cancellation of Gore's slideshow and book signing.
Most of the linked article other than the first two sentences (1) is bullshit and (2) has nothing to do with the cancellation of Gore's slideshow and book signing.
An inconvenient cancellation.
Prompted by a convenient truth.
This has to be the stupidest critic I have seen yet on this whole thing. In particular, complaining that the code is not commercial quality is laughable.
Most of the scientific code is awful, in climate science and elsewhere, because the constraints are totally different than commercial code. Maintainability is not so much a concern, and it is only a tool to obtain a result, not an end in itself. People don't care about the code quality most of the time.
There is another concern, which is that you rarely know what you exactly want when programming for research. You don't always have the time to design reasonable API, because the requirements keep changing - and keep in mind that you cannot spend too much time on this because you have to write papers, do some actual research, etc...
Even in my field, which is engineering and where people are expected to be more "programming methods-aware", most of the code is awful. My own code for research is awfully bad compared to what I release as open source for scientific programming: when I release something, I easily spend several times as much work on the thing, because of documentation issues, etc... Doing this for all my code is simply not possible.
If you want to dismiss climate science because of code quality, be ready to throw away the majority of science, from biology to IA through EE. Just look at lapack code: the code is awful, using goto to avoid structured programming, etc... (e.g. http://www.netlib.org/lapack/double/dsgesv.f). And it is used by 99 % of code which does linear algebra, including commercial software.
Most of the scientific code is awful, in climate science and elsewhere, because the constraints are totally different than commercial code. Maintainability is not so much a concern, and it is only a tool to obtain a result, not an end in itself. People don't care about the code quality most of the time.
There is another concern, which is that you rarely know what you exactly want when programming for research. You don't always have the time to design reasonable API, because the requirements keep changing - and keep in mind that you cannot spend too much time on this because you have to write papers, do some actual research, etc...
Even in my field, which is engineering and where people are expected to be more "programming methods-aware", most of the code is awful. My own code for research is awfully bad compared to what I release as open source for scientific programming: when I release something, I easily spend several times as much work on the thing, because of documentation issues, etc... Doing this for all my code is simply not possible.
If you want to dismiss climate science because of code quality, be ready to throw away the majority of science, from biology to IA through EE. Just look at lapack code: the code is awful, using goto to avoid structured programming, etc... (e.g. http://www.netlib.org/lapack/double/dsgesv.f). And it is used by 99 % of code which does linear algebra, including commercial software.
That's a pretty cavalier attitude to have when so much depends on the results (and I mean research programming in general, not just the CRU case). Maintainability may not be a concern, but verifiability certainly should be.
Verifiability in science is not a matter of having one program checked by different people. The reality is much more complex than that. This so called climate-gate says more about the perception of research than climate science (although it has certainly became a PR-nightmare for climate science). I think it is just another case of people have an idealistic, out of reality knowledge of how science works. Verifiability is less a concern than most people meant it to be for research. In an ideal world, it should be, but it isn't how science has been done - I would argue it has never been the case.
Scientists are actually very conservative for most parts exactly for this reason: in practice, it is just not possible to rely on consistent repeatability, etc... When you have an established, widely shared view on a topic, it takes great leaps to break it. Most research is not done to break the consensus, but to reenforce it. That's why today, if you want to show that climate is not caused by human activity, you will need to present data which are not as good as the ones one has now, but which are much, much better. So it is most likely true that if you want grants, etc... the easy way, going along the consensus is the right path.
That may not be how people think science should be done, but that's how it is done in practice in my experience. I think that for most researchers, all this affair sounds like a storm in a teapot. The behavior may look shady at times, the experiment is not always great, but that's exactly why for a new consensus to be created, you need to have better proof than the existing consensus. And yes, politics come into play, yes a lot of it is mean and some of it is not ethical. But this does not invalidate science in general - it invalidates how people think science works.
Scientists are actually very conservative for most parts exactly for this reason: in practice, it is just not possible to rely on consistent repeatability, etc... When you have an established, widely shared view on a topic, it takes great leaps to break it. Most research is not done to break the consensus, but to reenforce it. That's why today, if you want to show that climate is not caused by human activity, you will need to present data which are not as good as the ones one has now, but which are much, much better. So it is most likely true that if you want grants, etc... the easy way, going along the consensus is the right path.
That may not be how people think science should be done, but that's how it is done in practice in my experience. I think that for most researchers, all this affair sounds like a storm in a teapot. The behavior may look shady at times, the experiment is not always great, but that's exactly why for a new consensus to be created, you need to have better proof than the existing consensus. And yes, politics come into play, yes a lot of it is mean and some of it is not ethical. But this does not invalidate science in general - it invalidates how people think science works.
Concerning the practical issue of repeatability, that's why most fields require a new fact to be independently repeated by several people. At last year Scientific Python Conference, A. Martelli made an interesting point about verifiable software: the best way to know for sure if your software gives the right answer is to have several, different implementation done by different people. He gave the example of the space shuttle, which has several independent systems with a vote in case they don't agree (wikipedia has some information on it:http://en.wikipedia.org/wiki/Space_Shuttle#Flight_systems).
Is this code available anywhere? Is there an effort to embrace and improve this codebase?
A subject like global warming is potentially of such high importance that it shouldn't be left to cowboy programmers to process the data. I could have said "amateur" programmers, but there are many amateurs out there who produce very high quality code.