The unsung heroes of scientific software(nature.com)
nature.com
The unsung heroes of scientific software
http://www.nature.com/news/the-unsung-heroes-of-scientific-software-1.19100
9 comments
>Failure to distribute the data and code not only greatly devalues the contributions of many scientists, it makes replication far more difficult, and opens the door for outright fabrication.
Reasons why labs don't produce code:
(1) Fear of being scooped; suppose you put in tons of man-hours into a custom population genetics association study for malaria in Africa, you don't want your competitor lab to sequence a bunch samples in SE Asia and run your code as-is and publish a paper when you have the sequencers to do the same.
(2) Fear of competitors not being able to replicate stochastic ML results; some machine-learning and artificial neural network papers applied to Biology are stochastic. Past authors have been accused of "cherry-picking" the most "optimistic" runs that show positive results, e.g., (https://liorpachter.wordpress.com/2014/02/11/the-network-non...)
(3) Focus on science, not tool production; most labs' focus is to produce publications, not open-source software. User base for scientific software is very niche, unlike Web MVC frameworks; making the payoff calculus of packaging and supporting external users not so great. Furthermore, most labs have custom databases making integrating external software, externalizing internal software difficult.
Reasons why labs don't produce code:
(1) Fear of being scooped; suppose you put in tons of man-hours into a custom population genetics association study for malaria in Africa, you don't want your competitor lab to sequence a bunch samples in SE Asia and run your code as-is and publish a paper when you have the sequencers to do the same.
(2) Fear of competitors not being able to replicate stochastic ML results; some machine-learning and artificial neural network papers applied to Biology are stochastic. Past authors have been accused of "cherry-picking" the most "optimistic" runs that show positive results, e.g., (https://liorpachter.wordpress.com/2014/02/11/the-network-non...)
(3) Focus on science, not tool production; most labs' focus is to produce publications, not open-source software. User base for scientific software is very niche, unlike Web MVC frameworks; making the payoff calculus of packaging and supporting external users not so great. Furthermore, most labs have custom databases making integrating external software, externalizing internal software difficult.
(1) You publish the code at the time of publication of the paper, not before.
(2) This is a reason to publish the code, not a reason not to publish it.
(3) You don't necessarily have to make package it neatly like production software, but some effort is necessary.
We just wouldn't accept these kind of excuses for other descriptions of methodology, even though they might apply. There's no reason that code should be an exception.
We just wouldn't accept these kind of excuses for other descriptions of methodology, even though they might apply. There's no reason that code should be an exception.
For 1, I read that as thinking you could get a second paper out of running the same code on another dataset, and not wanting to be scooped on the second paper.
Just running the same code on another dataset is not research, it's production. Without minimal extra scientific efforts, such a work, while valuable, certainly does not deserve a scientific article in a peer review journal.
I understand the thinking but this should sound as non-sensical as
> We didn't release our first, second or third papers because we didn't want to be scooped on the fourth and if we publish then someone might build on our work!
> We didn't release our first, second or third papers because we didn't want to be scooped on the fourth and if we publish then someone might build on our work!
We didn't release our first, second or third papers because we didn't want to be scooped on the fourth and if we publish then someone might build on our work!
Which happens all the time. If you are working on something big you'll sometimes hold back on publishing parts of your research until you've completed the big thing for just that reason. Then once you've completed the big thing you publish all four papers either together or in quick succession.
Which happens all the time. If you are working on something big you'll sometimes hold back on publishing parts of your research until you've completed the big thing for just that reason. Then once you've completed the big thing you publish all four papers either together or in quick succession.
>Fear of competitors not being able to replicate stochastic ML results; some machine-learning and artificial neural network papers applied to Biology are stochastic. Past authors have been accused of "cherry-picking" the most "optimistic" runs that show positive results
That sounds like an ML phrasing for "fear of non-reproducibility", which is basically a fear of their research being bad.
That sounds like an ML phrasing for "fear of non-reproducibility", which is basically a fear of their research being bad.
Just to follow up on your last point, I have found that folks are often happy to send code, but code written in research labs is generally horrendous. I don't think that the "publish or die" culture and a culture of releasing all relevant code is compatible. It takes time to refactor and produce clean code - time that can be spent chasing the next publication.
Then requiring the code to be published and runnable would have a salutary effect on the whole circus.
Not really... I don't see how that would be anything more than adding one more task to a million other tasks. It doesn't change incentives.
I don't see the need for clean code, I just want the code they used. I've repeatedly tried to find the code used to generate various visualisations (which were the entire point of the paper) and then found
1. "It'll be released soon" on a years old page 2. 404'ing personal pages on university sites 3. Finally some code
But then the code didn't, and couldn't, generate the images they have. So now I'm not actually sure if the algorithm works as they say.
This was also one of the more successful results I've had in trying to find research code outside of ML. I'm sure other disciplines are also good but ML is mostly what I've had to find.
There's a part of the experiment which virtually anyone could replicate for a material cost of near 0 and check in detail if they wanted to and yet that seems to be the most poorly explained and shared part of it. You wouldn't get away with just mentioning the protocol you used in the lab without any references or explanations, would you?
1. "It'll be released soon" on a years old page 2. 404'ing personal pages on university sites 3. Finally some code
But then the code didn't, and couldn't, generate the images they have. So now I'm not actually sure if the algorithm works as they say.
This was also one of the more successful results I've had in trying to find research code outside of ML. I'm sure other disciplines are also good but ML is mostly what I've had to find.
There's a part of the experiment which virtually anyone could replicate for a material cost of near 0 and check in detail if they wanted to and yet that seems to be the most poorly explained and shared part of it. You wouldn't get away with just mentioning the protocol you used in the lab without any references or explanations, would you?
For academic labs you can add 'perceived future commercial value' to that list. In my experience these groups are often run by people with little or no commercial experience and lack the skills to appreciate the difference between a prototype and a commercial solution. Coupled with the pressure from Universities to 'commercialize all the things' this can inhibit attempts to distribute the code freely.
>Fear of being scooped;
This prevents a lot of good research from coming out. We should not just rely on publish or perish. Ifa lab makes a net contribution, they should still be rewarded with grants even if the idea gets "scooped".
This prevents a lot of good research from coming out. We should not just rely on publish or perish. Ifa lab makes a net contribution, they should still be rewarded with grants even if the idea gets "scooped".
>replicate stochastic ML results
(1) use an off the shelf prng
(2) provide the data
(3) provide the seed
(1) use an off the shelf prng
(2) provide the data
(3) provide the seed
Could you explain what you mean by "open code"? What is the philosophical underpinning?
That is, it seems that most of the argument for "open code" in the scientific review sense implies that 1) the software should be available at no cost, and 2) there is no need to support commercialization of said software. These principles are different than the four freedoms of "free software", and I have difficulties in reconciling the differences.
For a concrete example, I am self-employed. I sell scientific software. All of my customers receive it under the BSD license, after they pay me a good chunk of money. Thus, I sell free software for scientific research.
The FSF says "Selling a copy of a free program is legitimate, and we encourage it. (Quoting https://www.gnu.org/philosophy/selling.html .) But most of the time when I see people say "open code" or something similar, they want the right to view, use, test, and modify the source code at no cost.
If I publish a peer-reviewed paper about the software, should I be required to distribute the software to readers for no cost? Or may I set a fee of, say, $25,000 to get access to the source code under a free license? (I'm well aware of the loophole where I could publish that something is "open", but require a payment of $1 billion. My question is, what is a reasonable and fair price to charge?)
On the flip side, if I am required to publish my software under a free/open source license and not charge for access to the code, then that means any reader can take my source code and commercialize it, or even simply give it away. Commercialization is part of the four freedoms of free software, but it ends up reducing my market size and knocking other parts of the four freedoms. I won't have as much money to continue my self-funded development and research.
The principles are important because they help set guidelines for other questions. Can I require that people register their use of the software before getting access to a no-cost copy? Can I wait 6 months to respond to those requests? How long am I required to host the software, or will the journal manage all of that? May I include non-free license terms, like a requirement to cite X if someone uses the software? Does minimized/obsfucated code suffice? And many more details that have been resolved in the context of F/OSS software but have not, I believe, been resolve in the context of what's needed for peer reviewed publications.
That is, it seems that most of the argument for "open code" in the scientific review sense implies that 1) the software should be available at no cost, and 2) there is no need to support commercialization of said software. These principles are different than the four freedoms of "free software", and I have difficulties in reconciling the differences.
For a concrete example, I am self-employed. I sell scientific software. All of my customers receive it under the BSD license, after they pay me a good chunk of money. Thus, I sell free software for scientific research.
The FSF says "Selling a copy of a free program is legitimate, and we encourage it. (Quoting https://www.gnu.org/philosophy/selling.html .) But most of the time when I see people say "open code" or something similar, they want the right to view, use, test, and modify the source code at no cost.
If I publish a peer-reviewed paper about the software, should I be required to distribute the software to readers for no cost? Or may I set a fee of, say, $25,000 to get access to the source code under a free license? (I'm well aware of the loophole where I could publish that something is "open", but require a payment of $1 billion. My question is, what is a reasonable and fair price to charge?)
On the flip side, if I am required to publish my software under a free/open source license and not charge for access to the code, then that means any reader can take my source code and commercialize it, or even simply give it away. Commercialization is part of the four freedoms of free software, but it ends up reducing my market size and knocking other parts of the four freedoms. I won't have as much money to continue my self-funded development and research.
The principles are important because they help set guidelines for other questions. Can I require that people register their use of the software before getting access to a no-cost copy? Can I wait 6 months to respond to those requests? How long am I required to host the software, or will the journal manage all of that? May I include non-free license terms, like a requirement to cite X if someone uses the software? Does minimized/obsfucated code suffice? And many more details that have been resolved in the context of F/OSS software but have not, I believe, been resolve in the context of what's needed for peer reviewed publications.
(I'm not a scientist)
There are two issues I see with closed source research software.
It seems to me like someone outside the research group should review all code run for a paper as part of peer review. If there are glaring off-by-one errors, race conditions, etc., then why should we trust the results produced by the software? There are a lot of things that should get caught in peer review that would only be red flags if someone experienced was able to look at the program source.
Also, I've always been under the impression that science should be reproducible. If no one can replicate your results, then how can we trust your findings? While this is technically possible without access to source code, closing source code off from the community really makes reproducing results harder.
I'm not sure that it would be fair for journals to require open source availability for publishing, but if they don't then we need a creative solution because these are real problems.
There are two issues I see with closed source research software.
It seems to me like someone outside the research group should review all code run for a paper as part of peer review. If there are glaring off-by-one errors, race conditions, etc., then why should we trust the results produced by the software? There are a lot of things that should get caught in peer review that would only be red flags if someone experienced was able to look at the program source.
Also, I've always been under the impression that science should be reproducible. If no one can replicate your results, then how can we trust your findings? While this is technically possible without access to source code, closing source code off from the community really makes reproducing results harder.
I'm not sure that it would be fair for journals to require open source availability for publishing, but if they don't then we need a creative solution because these are real problems.
The situations you pose have little to do with being open source. It's easily possible with non-open source software.
Consider the X-PLOR program for crystallography refinement, where an academic/ research license was a few hundred dollars. That came with source code, plus the right to distribute patches and other modifications to anyone else who had an X-PLOR license, but not to everyone else.
Consider the NAUTY program for graph isomorphism, from http://users.cecs.anu.edu.au/~bdm/nauty/ . The license says "Permission is hereby given for use and/or distribution with the exception of sale for profit or application with nontrivial military significance."
Consider the many programs available for free and unrestricted download from university web sites which are "for academic use only", some of which require users to cite a given paper. (Eg, http://www.maths.lth.se/matematiklth/personal/sminchis/code/... is one I easily found with a web search which is available in source code and is "free of charge for non-commercial research and education purposes".)
These are neither open source nor free, so are they "closed source research software"? If "closed source" means "not open source" - which is the usual view - then yes, the above programs are all closed source.
Yet all of them are available for peer review, reproducibility studies, etc. that you want.
While on the other hand, to get what you want requires principles different than what the Free Software Foundation considers to be one of the essential freedoms in programming - the freedom to sell software. So at the very least, "peer-verifiable" software, for lack of a better term, is not compatible with "free software."
I personally think it's unwise to even use the terms "open source" and "closed source" in this discussion because of the confusion it adds. However, as most researchers come from academic or government labs with non-commercial funding sources, I can see why self-funded, for-profit scientific software may be overlooked.
Consider the X-PLOR program for crystallography refinement, where an academic/ research license was a few hundred dollars. That came with source code, plus the right to distribute patches and other modifications to anyone else who had an X-PLOR license, but not to everyone else.
Consider the NAUTY program for graph isomorphism, from http://users.cecs.anu.edu.au/~bdm/nauty/ . The license says "Permission is hereby given for use and/or distribution with the exception of sale for profit or application with nontrivial military significance."
Consider the many programs available for free and unrestricted download from university web sites which are "for academic use only", some of which require users to cite a given paper. (Eg, http://www.maths.lth.se/matematiklth/personal/sminchis/code/... is one I easily found with a web search which is available in source code and is "free of charge for non-commercial research and education purposes".)
These are neither open source nor free, so are they "closed source research software"? If "closed source" means "not open source" - which is the usual view - then yes, the above programs are all closed source.
Yet all of them are available for peer review, reproducibility studies, etc. that you want.
While on the other hand, to get what you want requires principles different than what the Free Software Foundation considers to be one of the essential freedoms in programming - the freedom to sell software. So at the very least, "peer-verifiable" software, for lack of a better term, is not compatible with "free software."
I personally think it's unwise to even use the terms "open source" and "closed source" in this discussion because of the confusion it adds. However, as most researchers come from academic or government labs with non-commercial funding sources, I can see why self-funded, for-profit scientific software may be overlooked.
You'll note that I'm not advocating for FOSS (and the licensing nightmares that can entail), just available source code for peer and community review.
Perhaps I'm being too loose in my terminology, but in this context I used "closed source" to mean "source code unavailable to the community." In my view the attached license(s) matter a lot less than making it so that the science can be reviewed, reproduced, and improved upon.
If the source code can be reviewed for free but not redistributed, reused or commercialized for free, then I don't see why that would hinder endeavors to review and reproduce the research. However having to go through byzantine and expensive processes to procure source code would be an impediment to those in academia who don't have funding to pay for licenses to source code just to review a paper, for example. Maybe I'm not reading closely enough, but that sounds a lot like what you're advocating in your original comment.
Perhaps I'm being too loose in my terminology, but in this context I used "closed source" to mean "source code unavailable to the community." In my view the attached license(s) matter a lot less than making it so that the science can be reviewed, reproduced, and improved upon.
If the source code can be reviewed for free but not redistributed, reused or commercialized for free, then I don't see why that would hinder endeavors to review and reproduce the research. However having to go through byzantine and expensive processes to procure source code would be an impediment to those in academia who don't have funding to pay for licenses to source code just to review a paper, for example. Maybe I'm not reading closely enough, but that sounds a lot like what you're advocating in your original comment.
I do understand the desire for "available source code for peer and community review".
What I don't understand is how to set up practical guidelines.
For example, consider "reviewed for free but not redistributed". If I review the software, and find an error, what do I do? Should I publish a paper which demonstrates the difference between the original and corrected versions? If so, I need to include the fixed code, and perhaps also the original. But that's a redistribution.
"would be an impediment to those in academia who don't have funding to pay for licenses to source code"
As a minor point which is big in my mind - most academic groups have more funding to pay for licenses than I, a self-funded, for-profit researcher, have.
"but that sounds a lot like what you're advocating in your original comment"
I mentioned that, to explain the view of most people who want access the source code. I was not advocating it.
My question was, is this requirement important enough that all of the source code must be made available at no cost? If so, it's in opposition to the FSF's four freedoms, which encourages people to sell free software, so there must be some other philosophical underpinning to justify the no-cost argument.
What is that philosophy? It can't simply be "to verify" because there are some problems, like factoring RSA-360:
And what counts as a "byzantine and expensive processes"? The process of reproducing one of the CERN papers, especially if I need to make my own accelerator, is non-trivial and expensive. Some molecular dynamics simulation software only runs on custom-made ASIC hardware, or uses $100K+ of CPU time. A software cost of $20K is only a small part of the overall cost in that case.
What I don't understand is how to set up practical guidelines.
For example, consider "reviewed for free but not redistributed". If I review the software, and find an error, what do I do? Should I publish a paper which demonstrates the difference between the original and corrected versions? If so, I need to include the fixed code, and perhaps also the original. But that's a redistribution.
"would be an impediment to those in academia who don't have funding to pay for licenses to source code"
As a minor point which is big in my mind - most academic groups have more funding to pay for licenses than I, a self-funded, for-profit researcher, have.
"but that sounds a lot like what you're advocating in your original comment"
I mentioned that, to explain the view of most people who want access the source code. I was not advocating it.
My question was, is this requirement important enough that all of the source code must be made available at no cost? If so, it's in opposition to the FSF's four freedoms, which encourages people to sell free software, so there must be some other philosophical underpinning to justify the no-cost argument.
What is that philosophy? It can't simply be "to verify" because there are some problems, like factoring RSA-360:
2186820202343172631466406372285792654649158564828384065217121866374227745448
7764963889680817334211643637752157994969516984539482486678141304751672197524
0052350576247238785129338002757406892629970748212734663781952170745916609168
9358372359962787832802257421757011302526265184263565623426823456522539874717
61591019113926725623095606566457918240614767013806590649
And what counts as a "byzantine and expensive processes"? The process of reproducing one of the CERN papers, especially if I need to make my own accelerator, is non-trivial and expensive. Some molecular dynamics simulation software only runs on custom-made ASIC hardware, or uses $100K+ of CPU time. A software cost of $20K is only a small part of the overall cost in that case.
Peer review is nearly impossible for the interesting bits of code. You have the same challenges as peer review of the paper itself: there aren't many reviewers with the prerequisite technical knowledge. The majority of the time, the qualified scientific reviewers and software reviewers will not be the same people. Viz understanding CFD analytically does not mean you understand numerical algorithm implementation, or are an expert in numerical error analysis. Hell, look at the frequent evidence all over the net that the vast majority of developers don't understand ieee 754.
Who pays the developers? If you aren't paying the developers much, what exactly are they going to catch in the 2-4 hours they may have to look at it?
I wish I had a good solution; I think about the best we can do is community-pooled development efforts ala openfoam, bioconductor, numpy, sklearn.
Who pays the developers? If you aren't paying the developers much, what exactly are they going to catch in the 2-4 hours they may have to look at it?
I wish I had a good solution; I think about the best we can do is community-pooled development efforts ala openfoam, bioconductor, numpy, sklearn.
Reproducibility in science doesn't mean "easy" reproducibility of experiments. Some experiments in science are famously known to be a one-off. Example: the first test of general relativity was during a solar eclipse, with pictures that were subject to poor lightning conditions. Of course, the reason why we know it was right is that many other tests have been made since then, even though the original one can never be replicated. Same reasoning goes for other areas. A scientific experiment with software doesn't need to be replicated with the same software to be relevant (although it would be nice to do be able to do that). As long as other people can come up with their own implementation and arrive to similar results, the researcher achieved his/her goal.
> Reproducibility in science doesn't mean "easy" reproducibility of experiments.
Where it is easy to make it easy to replicate, it should be made easy. Releasing source code, even in unsanitized form, allows for much easier inspection and replication.
The research world is also much different today than a century ago. Today the field is crowded with researchers who must publish or perish. The speed of papers, many of which are in fact nonsense, is churning out is unprecedented. As a result, a paper that can be verified but is hard to verified is often never properly verified at all.
Where it is easy to make it easy to replicate, it should be made easy. Releasing source code, even in unsanitized form, allows for much easier inspection and replication.
The research world is also much different today than a century ago. Today the field is crowded with researchers who must publish or perish. The speed of papers, many of which are in fact nonsense, is churning out is unprecedented. As a result, a paper that can be verified but is hard to verified is often never properly verified at all.
Reproducibility in science doesn't mean "easy" reproducibility of experiments.
No, but making it easy as reasonably possible or at least not going out of your way to make it hard would be nice.
No, but making it easy as reasonably possible or at least not going out of your way to make it hard would be nice.
Someone who owns the same tools should be able to replicate the results of the paper without rewriting its code. Whether these tools (e.g. Matlab or your software) come at a cost is a secondary concern as long as it is not prohibitively high.
What I dislike the most are computational articles that give no indication whatsoever about the employed tools and programming languages.
What I dislike the most are computational articles that give no indication whatsoever about the employed tools and programming languages.
A problem is, what does it mean to "own the same tools"?
If I develop my own tools, which no one else has, and I never distribute them, then it's the trivial edge case that everyone who owns those tools (me!) can replicate the results.
If I sell the tools under a BSD license for $1,000,000 then it might still be trivial for those who pay me that sum to reproduce the results. But those who argue for source code access usually want the source code available for $0 or a pittance compared to the development costs.
You agree that it should not be "prohibitively high". How do we turn that into something actionable? If $1M is too high, then what about $100K?
Does the requirement extend to providing documentation? Even if such documentation doesn't already exist? For one job, I fixed a few bugs in software that was commented in Russian, and I speak no Russian. Is this too high of a barrier to entry? And if so, should all code be commented in English?
If I develop my own tools, which no one else has, and I never distribute them, then it's the trivial edge case that everyone who owns those tools (me!) can replicate the results.
If I sell the tools under a BSD license for $1,000,000 then it might still be trivial for those who pay me that sum to reproduce the results. But those who argue for source code access usually want the source code available for $0 or a pittance compared to the development costs.
You agree that it should not be "prohibitively high". How do we turn that into something actionable? If $1M is too high, then what about $100K?
Does the requirement extend to providing documentation? Even if such documentation doesn't already exist? For one job, I fixed a few bugs in software that was commented in Russian, and I speak no Russian. Is this too high of a barrier to entry? And if so, should all code be commented in English?
I could not agree with you more. It is a disconcerting trend to see high impact papers using an "in house program" or "a modified version of a previously described tool", without providing any details about software versions, analysis steps and any custom filtering applied to the data. There could be bugs, unintentionally (or intentionally) introduced biases, or other things such as outlier removal that are not explained in the manuscripts methods.
This is also very true for statistical analysis, many papers do not provide R scripts for their analysis and only a brief overview of the analysis.
This is also very true for statistical analysis, many papers do not provide R scripts for their analysis and only a brief overview of the analysis.
> If computation is a necessary element of the research, it ought to be a necessary element of publication as well.
Correct. Imagine reading an article in a journal that said, "We were able to prove that Theorem 3.1 is true. The proof is omitted because we want to use similar techniques to prove theorems in the future, and by not publishing any details of our proof, we will have an advantage over rival researchers. That will allow us to recover the significant investment of time that we put into doing this proof."
Correct. Imagine reading an article in a journal that said, "We were able to prove that Theorem 3.1 is true. The proof is omitted because we want to use similar techniques to prove theorems in the future, and by not publishing any details of our proof, we will have an advantage over rival researchers. That will allow us to recover the significant investment of time that we put into doing this proof."
I can easily imagine such an article, because it happens all the damn time.
"Using the argument of [1] and [2] it can be shown, with brief step x, it can be shown that this theorem holds in xxxx" and then moving onto the conclusions, is exactly the same as "Using the software developed in [3] to solve Eq. 1, we show that our result is statistically significant."
If the claim is probable, it will and does get through review.
"Using the argument of [1] and [2] it can be shown, with brief step x, it can be shown that this theorem holds in xxxx" and then moving onto the conclusions, is exactly the same as "Using the software developed in [3] to solve Eq. 1, we show that our result is statistically significant."
If the claim is probable, it will and does get through review.
I lean towards more open code and data, but could one argue that if the code uses some fundamentally new, non-trivial methodology, a replication of the research should write new code implementing that method from it's description. Otherwise, if the original code is re-used, a bug in the code may very well cause the replication of an erroneous result.
There are two reasons to look at the code. The first is to look for bugs. Obviously that is easier if you have the code. Testing is much easier than writing from scratch. The second is to see how robust the results are to changes in assumptions and other aspects of the methodology. When you are talking about a big computational problem, it is not realistic to expect others to do a full, independent replication. And if you can't reproduce the original results, you need access to the original code anyway, to find the source of the differences.
In fact this is the main reason why I don't subscribe to the idea of research papers including code (as a requirement). Independent replication should rely on independent implementation. Unless the software is too generic, the original researcher should have the advantage of using the implemented software for new discoveries, as much as a traditional scientist has the advantage of using his own lab for additional research.
Be able to reimplement the code < That's an interesting perspective. However I think the vast majority of simulation work never gets that sort of scrutiny, no matter if the code is open source
Personally though I think we should give up on papers and everything should be published on an ipython notebook style page where others can play with data and code.
Personally though I think we should give up on papers and everything should be published on an ipython notebook style page where others can play with data and code.
Conversely, if the original code isn't available, the community may never know why they can't replicate the results. This could especially be a problem if the original researcher is more known and respected than those attempting replications (i.e. devolving into he said she said).
Sure thing, the best case is where the original code & data is available, and a cross check replication does an independent reimplementation. There's a little bit of risk that peeking at the original code might lead one down a wrong path, but overall I think it's still better that the original code is provided. For example, if someone isn't looking to replicate and might be instead trying to extending the original work, using the original code could save a lot of time (and could help find original mistakes too).
I think the article is talking about a bigger issue, which is expanding our idea of a "research contribution" to include software. The current attitude essentially considers software to be equipment for getting to a scientific result.
I think that if we started thinking about research software as a research contribution in itself, it would be a good way to accomplish what you talk about, e.g. by making software (and data) something that is published and cited, rather than attached as a supplement to every publication.
I think that if we started thinking about research software as a research contribution in itself, it would be a good way to accomplish what you talk about, e.g. by making software (and data) something that is published and cited, rather than attached as a supplement to every publication.
It would have the side effect of improving and propagating best practices. Most scientists are self taught programers, and it shows. The data models and codebases used by most labs are . . . amateurish.
I really like what this group is doing:
http://software-carpentry.org/
I took their instructor training last summer but I haven't had a chance to run a workshop yet. I think it's a great idea, especially given my recent exposure to researcher-written codebases.
http://software-carpentry.org/
I took their instructor training last summer but I haven't had a chance to run a workshop yet. I think it's a great idea, especially given my recent exposure to researcher-written codebases.
Seriously, I consider myself lucky when I get code that's even been commented from labmates...It took me about a year to get 50% of people using github
One ex-colleague of mine wouldn't even indent her Perl code properly. I showed here there was even a button in eclipse to do it for her.
(Using eclipse seems problematic when there is a text editor that doesn't involve any cognitive overhead to many bioinformaticians. The step through debugger did seem to appeal to this one though).
(Using eclipse seems problematic when there is a text editor that doesn't involve any cognitive overhead to many bioinformaticians. The step through debugger did seem to appeal to this one though).
There's progress on this front in computer science conferences in software-focused disciplines:
http://www.artifact-eval.org/ https://plus.google.com/+JanVitek/posts/19BH96G8rrw
The artifact evaluation workflow is a reasonable starting point for what you suggest.
http://www.artifact-eval.org/ https://plus.google.com/+JanVitek/posts/19BH96G8rrw
The artifact evaluation workflow is a reasonable starting point for what you suggest.
I concur. Once I wrote a similar answer http://academia.stackexchange.com/questions/23237/why-are-pa...
In short: benefits for scientists (career, prestige) are misaligned with benefits for science and society (reproducibility, progress, openness).
In short: benefits for scientists (career, prestige) are misaligned with benefits for science and society (reproducibility, progress, openness).
Especially if the work was funded by public money.
What should be done in the case of when a researcher creates a private company as a result of publicly-funded research, but the research isn't fully released or is obfuscated?
What should be done in the case of when a researcher creates a private company as a result of publicly-funded research, but the research isn't fully released or is obfuscated?
I've been developing research software for +10 years and I experience this all the time.
As one example, over a decade ago I wrote a biological simulation program called CompuCell3D it was the successor to CompuCell, a 2D simulator of cells as cellular automata. Since then many papers have been written based on research utilising this software. Not only have I not been credited in any of these publications but my name has been removed from the software, the website never mentions my original contribution and the current maintainers of said software are not responding to my emails.
Granted this code has changed a lot over the years but there are still large parts of the code which are still verbatim from my original software. The researchers in this project see it as totally irrelevant that I wrote the original code because they value code creation MUCH less than their research.
This particular case is especially egregious but it is a good anecdotal example of the problem. Sometimes code is not as significant or as breakthrough as raw research. It really matters what kind of coding you are talking about. You don't necessarily credit the construction crew when dedicating a new building but you do credit the architect. Coders are under valued in today's research environment.
As one example, over a decade ago I wrote a biological simulation program called CompuCell3D it was the successor to CompuCell, a 2D simulator of cells as cellular automata. Since then many papers have been written based on research utilising this software. Not only have I not been credited in any of these publications but my name has been removed from the software, the website never mentions my original contribution and the current maintainers of said software are not responding to my emails.
Granted this code has changed a lot over the years but there are still large parts of the code which are still verbatim from my original software. The researchers in this project see it as totally irrelevant that I wrote the original code because they value code creation MUCH less than their research.
This particular case is especially egregious but it is a good anecdotal example of the problem. Sometimes code is not as significant or as breakthrough as raw research. It really matters what kind of coding you are talking about. You don't necessarily credit the construction crew when dedicating a new building but you do credit the architect. Coders are under valued in today's research environment.
The paper for citing CompuCell3D is Multi-Scale Modeling of Tissues Using CompuCell3D – M. Swat, Gilberto L. Thomas, Julio M. Belmonte, A. Shirinifard, D.Hmeljak, J. A. Glazier, Computational Methods in Cell Biology, Methods in Cell Biology 110: 325-366 (2012)
I can't see that article online without paying, but I am certain it cites CompuCell, a Multi-Model Framework For Simulation of Morphogenesis – J. A. Izaguirre, R. Chaturvedi, C. Huang, T. Cickovski, J. Coffland, G. Thomas, G. Forgacs, M. Alber, G. Hentschel, S. A. Newman, and J. A. Glazier, Bioinformatics 20: 1129-1137 (2004)
that original article includes your name on the author list.
So your statement "I have not been credited in any of these publications" isn't really correct; your contribution is credited indirectly through citations. I don't see any problem with this; I doubt papers citing CompuCell3D have any text like "We acknowledge the efforts of <so and so>".
I can't see that article online without paying, but I am certain it cites CompuCell, a Multi-Model Framework For Simulation of Morphogenesis – J. A. Izaguirre, R. Chaturvedi, C. Huang, T. Cickovski, J. Coffland, G. Thomas, G. Forgacs, M. Alber, G. Hentschel, S. A. Newman, and J. A. Glazier, Bioinformatics 20: 1129-1137 (2004)
that original article includes your name on the author list.
So your statement "I have not been credited in any of these publications" isn't really correct; your contribution is credited indirectly through citations. I don't see any problem with this; I doubt papers citing CompuCell3D have any text like "We acknowledge the efforts of <so and so>".
I stand corrected I was credited in the original work. My main beef was that my name was dropped from the code itself and the website where it is hosted failed to mention me until recently.
You seem to be listed on the people page: http://www.compucell3d.org/People
You are right. They did add me. Perhaps they did get my emails but I didn't get the answer. I should have rechecked this before posting. Of course I did have to ask to be mentioned which shouldn't have been necessary.
> Coders are under valued in today's research environment.
Exactly because it is a research environment. Coders are valued in a tech company environment because that's where they're the stars. In any other organization, such as a transportation company or a government bureau, a coder is just an assistant to the main tasks, and there is no reason it shouldn't be different.
Exactly because it is a research environment. Coders are valued in a tech company environment because that's where they're the stars. In any other organization, such as a transportation company or a government bureau, a coder is just an assistant to the main tasks, and there is no reason it shouldn't be different.
(scientist w/ 15+ years experience in software dev in bioinformatics, with nice results, see http://www.ncbi.nlm.nih.gov/pubmed/?term=grosdidier+aur%C3%A... )
I think a decent software development capability is one among the many skills a scientist must be able to be comfortable with in order to provide society with valuable research results.
Professional developers are given a task, or a goal, and are focused on the right way to do it. Some do wonders, but they aren't paid to go beyond the goal they are assigned.
When you're a scientist, the software is not a goal, but only a tool that will be subjected to further iterative refinement.
No one else but you knows about the proper efficiency / flexibility / level of abstraction you need. There's a strong need for scientists combining both scientific and development skills.
Btw, I'm looking for a new position ^^
I think a decent software development capability is one among the many skills a scientist must be able to be comfortable with in order to provide society with valuable research results.
Professional developers are given a task, or a goal, and are focused on the right way to do it. Some do wonders, but they aren't paid to go beyond the goal they are assigned.
When you're a scientist, the software is not a goal, but only a tool that will be subjected to further iterative refinement.
No one else but you knows about the proper efficiency / flexibility / level of abstraction you need. There's a strong need for scientists combining both scientific and development skills.
Btw, I'm looking for a new position ^^
And in scientific computing the senior researchers are not paid very much so the supporting developers/tecnhnicians are badly paid.
When I was a Research Assistant/Experimental Officer at a world leading Rnd organization I was paid about 1/3rd of what other jobs with similar entry requirements did.
When I was a Research Assistant/Experimental Officer at a world leading Rnd organization I was paid about 1/3rd of what other jobs with similar entry requirements did.
Yep. I write simulation and data analysis software for a large company. My names appeared on ~5 papers simply because of the fact that I wrote the software used for the research.
You could have easily added a BSD derivative license where one would be required to cite your paper or just your authorship if they've used the software.
I've noticed this being a good way to maximize credit.
I've noticed this being a good way to maximize credit.
This is both BSD- and GPL-incompatible, which has all sorts of potential ramifications. Particularly the GPL-incompatibility prevents you from including GPL-licensed code.
Although both BSD and GPL already require attribution, so you could say that it's redundant.
Although both BSD and GPL already require attribution, so you could say that it's redundant.
Your argument fails miserably for research in computational biology.
One problem is citation limits for many journals (which say that you can only cite 25 or 50 papers or whatever). As a computational biologist myself, I would prefer to cite every package that I use because I understand that the authors need this to get grants for further development, etc.
But what happens when the more senior authors (who tend to be experimentalists) get the manuscript, they want to add a bunch of experimental citations and they see citations of computational methods as irrelevant and able to be sacrificed if there are too many citations.
But what happens when the more senior authors (who tend to be experimentalists) get the manuscript, they want to add a bunch of experimental citations and they see citations of computational methods as irrelevant and able to be sacrificed if there are too many citations.
What role did you have when you wrote the code? If, for example, you were a full time employee, grad student, or postdoc at a university, the copyright belongs to the university (not true for undergrads!)
The website isn't required to list your contribution. That would be a courtesy.
Also, if you have a problem with this where you think it's important to change the website or get another form of credit, your lawyer should be talking to your ex-employers lawyer. That should have been clear when the current maintainers failed to respond to your email.
The website isn't required to list your contribution. That would be a courtesy.
Also, if you have a problem with this where you think it's important to change the website or get another form of credit, your lawyer should be talking to your ex-employers lawyer. That should have been clear when the current maintainers failed to respond to your email.
Very neat, but they're only indexing CRAN and PyPI right now. In my short time doing scientific computing (staff member at a university right now), I've seen a LOT of C and C++, but those don't have canonical package repositories. I think that finding all of those codebases (even if just limited to GitHub) and indexing those would be very interesting and would be necessary to have representation from a few fields.
Yes, agreed...we've had lots of requests to add other languages.
And as you say, the growing popularity of GitHub gives us all kinds of cool data even when there's no central package manager for the language. In fact, we're mining imports of every Python and R project on GitHub right now to build out the dependency network beyond the (much much smaller) CRAN and PyPi networks.
The idea with Depsy has been to launch quickly with two languages, so people could see what it looks like, then iterate and add more as we get feedback. So we'll count your comment as +1 for C and C++ :)
And as you say, the growing popularity of GitHub gives us all kinds of cool data even when there's no central package manager for the language. In fact, we're mining imports of every Python and R project on GitHub right now to build out the dependency network beyond the (much much smaller) CRAN and PyPi networks.
The idea with Depsy has been to launch quickly with two languages, so people could see what it looks like, then iterate and add more as we get feedback. So we'll count your comment as +1 for C and C++ :)
I strongly oppose "just limiting to GitHub". If this is to be used as some sort of merit factor, its inclusion can't be contingent on using a particular centralized repo. For a start, there are several such providers. Larger projects might well prefer self-hosting, too.
I would oppose it too in the long run, but it would be the easiest low-hanging fruit for getting the ball rolling on including a wider variety of research software, and I was mostly making a suggestion for next steps in developing their (pretty cool) tool.
You have to start somewhere, and if you get GitHub working then it would be (hopefully) much easier to then include BitBucket, GitLab, etc. Indexing self-hosted repos would be pretty tricky, I imagine, if only because you'd then need to maintain a list of all of the servers to clone from.
Further, there's also the challenge of including software from all of the research groups who don't even appear to use version control, or if they do it's locked away on a private server or service. How does one attribute authorship rights to those people without source history?
Anyways, this is a long ramble now, but I'm mostly trying to illustrate that it's difficult to do what Depsy does with software when it's written in languages without a canonical package repository. That doesn't mean they shouldn't try to expand their reach just because "GitHub isn't enough."
You have to start somewhere, and if you get GitHub working then it would be (hopefully) much easier to then include BitBucket, GitLab, etc. Indexing self-hosted repos would be pretty tricky, I imagine, if only because you'd then need to maintain a list of all of the servers to clone from.
Further, there's also the challenge of including software from all of the research groups who don't even appear to use version control, or if they do it's locked away on a private server or service. How does one attribute authorship rights to those people without source history?
Anyways, this is a long ramble now, but I'm mostly trying to illustrate that it's difficult to do what Depsy does with software when it's written in languages without a canonical package repository. That doesn't mean they shouldn't try to expand their reach just because "GitHub isn't enough."
I was kind of figuring that people would register their own software, rather than trying to somehow find them. It doesn't seem much harder accessing different repos on different servers than on just github. You still have to keep an url.
Of course, if it's not a public server then you can't do anything, but then again in that case the authors can't really complain they don't get any credit for what they do.
Of course, if it's not a public server then you can't do anything, but then again in that case the authors can't really complain they don't get any credit for what they do.
They also seem to be missing Bioconductor, which is where a huge percentage of (maybe even most?) R biology packages are located. Allowing submissions and somehow indexing the relevant repositories in GitHub would be huge.
Yep, we chose CRAN for the MVP, since it's the main R package host. But adding other sources is the plan, and Bioconductor is at the top of that list for R.
When I think about unsung software heroes I think of Rob Scharein[1]. He is the creator or KnotPlot[2] which has enabled topological research to flourish. He recieves regular acknowledgement among topology researchers, and he's an awesome guy. Do yourself a favor and play with some knots.
[1] www.hypnagogic.net/rob/ [2] www.knotplot.com
[1] www.hypnagogic.net/rob/ [2] www.knotplot.com
I fully agree with the principle that experiments should be reproducible. In systems research conferences (SOSP, Eurosys, OSDI) very few of the systems publish their algorithms. The worst culprits are the companies, such as Google, Facebook, and Microsoft, who don't even publish all of their algorithms or systems properties.
Google have published significant papers that leave out the key algorithms on how parts of their systems work (e.g., In Borg how does the scheduler synchronize the cluster state with the replicas? In Spanner, what are the properties of the underlying storage system, Collosus?).
I would love to see all distributed systems papers be able to reproduce everything by just downloading a file that allows the paper's results to be reproduced. We can do it with the help of either virtualization or containers and software configuration frameworks (to parameterize the experiments). We can specify the hardware and network programmatically, we can install the software automatically, we can parameterize system software (with Chef or Puppet attributes), and run the experiments reproducibly. Without huge investment and cloud computing or Docker, we could specify systems that have reproducible hardware/network/software.
The first attempt I've seen at this is www.karamel.io that allows you to design reproducible experiments by using JClouds to spawn VMs and setup the virtualized hardware, then orhcestrating Chef to install software that can be parameterized. I hope to see more of this line of platform gain adoption in our community. It would be a boon for research.
Lots of people have proposed and implemented VM-based reproducibility environments.
1) it's actually harder to get reproducible computational experiments than they expected. For example, you can run same VM on a different processor and get different results, which makes bitwise reproduction hard, and statistical tests for nonbitwise equality are harder
2) developing and maintaining the VMs and the environment takes a fair amount of effort from skilled people
3) the resulting improvements to science don't seem to exceed the cost thresholds implied by #1 and #2, and nobody's volunteering their time.
My conclusion: good idea, but probably not critically necessary.
1) it's actually harder to get reproducible computational experiments than they expected. For example, you can run same VM on a different processor and get different results, which makes bitwise reproduction hard, and statistical tests for nonbitwise equality are harder
2) developing and maintaining the VMs and the environment takes a fair amount of effort from skilled people
3) the resulting improvements to science don't seem to exceed the cost thresholds implied by #1 and #2, and nobody's volunteering their time.
My conclusion: good idea, but probably not critically necessary.
That's good feedback. Although I'm considering mostly distributed systems, where full reproducability is not possible.
For single-threaded, deterministic programs (discrete event simulations) where performance is not a measurement point, you can do it with VMs.
The nicest thing would be to be able to parameterize experiments. Not just re-run them, but change the parameters for different runs.
Soooo... should people get recognition, or not?
Re: https://news.ycombinator.com/item?id=10838166
Re: https://news.ycombinator.com/item?id=10838166
My unsung software hero is Dr. Jeffrey Lewis Fox (deceased in 1999 at age 51), Associate Professor, Department of Pharmaceutics and Pharmaceutical Chemistry, University of Utah http://pharmacy.utah.edu/pharmaceutics/news/1999.html#fox He wrote MINSQ (now called Scientist), the only piece of software I have ever fallen in love with. He sold the first version for $10 (if I recall correctly) in 1990. It was so user-friendly that I was using it within 10 mins to solve a system of differential equations that were part of my PhD research. (Unfortunately, I don't find the more recent versions, "updated" by others, anywhere near as user-friendly.) He called his company, at the University of Utah, MicroMath. I assume he used the algorithms from "Numerical Recipes" to write MINSQ - I know that was the alternative facing me till his excellent software appeared. I really think the U of Utah should have kept the rights to the software and maintained it in his honour.
I will use this opportunity to mention Maxima (https://en.wikipedia.org/wiki/Maxima_%28software%29) that was developed by one of the best math professors that I have had the privilege to learn from: Bill Schelter. It was forked from Macsyma and released under the GPL and still appears to be actively maintained.
What about generic tools, such as webbrowsers? The web was invented at CERN by a researcher, so I guess webbrowsers must have some value to the scientific process :)
[deleted]
Failure to distribute the data and code not only greatly devalues the contributions of many scientists, it makes replication far more difficult, and opens the door for outright fabrication.
If computation is a necessary element of the research, it ought to be a necessary element of publication as well.