Science Isn’t Broken(fivethirtyeight.com)
fivethirtyeight.com
Science Isn’t Broken
http://fivethirtyeight.com/features/science-isnt-broken/
11 comments
I agree with what your saying. However, I feel publish or perish has infected many fields resulting in a lot of crap instead of more focused high quality work. Few people can risk spending 10 years on something and have nothing to show for it.
It's a negative feedback loop where quality is hard for people outside the field to judge. So, it's just race to see who can lower quality more while still being published.
It's a negative feedback loop where quality is hard for people outside the field to judge. So, it's just race to see who can lower quality more while still being published.
Well, not precisely.
I don't know how it is outside astronomy (hello parent parent), but inside astronomy it's relatively well known that there are four journals that are likely to be scrutinously peer reviewed and which _will_ reject you and which _will_ force you to do further analyses. It is also extremely well known that some journals advertise "In print in two months or less!" Inside the field (so to the people who will give you the next contract), it's obvious where you publish.
In some cases, it _is_ very valuable to put the extra months of effort in to lay down a rock solid piece. For example, in CAS, you can make an extra 10,000 RMB for publishing in a low impact Indian or Chinese journal. You can make upwards of 100,000 RMB (a full year's salary) for publishing in one of the "big four" journals. And up to 3 years salary for getting a Nature or Science publication. This is money being invested by the Chinese government to intentionally combat the plague of falsified Chinese research. I don't know of any other country or system that does this. It is also remarkably easy to get tenure in China, which allows a lot of long-term type focus. But this is a tiny bubble for China as it attempts to brain drain the rest of the world and is not likely to continue for long.
It's a big problem that the media doesn't understand which are which. For example when everyone was raving about the "faster than light" neutrinos a few years ago the first thing _every single person_ in my building said was "The Italians set their clock wrong didn't they?" Half a joke, but dead serious that no one should even consider this until it showed up in a publication.
Another big problem is that only basing your research on verifying others research is suicide for an academic career sing you're intentionally doing _low impact_ science. The only people willing to fund that are people with agendas, ie. BP, Exxon, Coca Cola etc.
I don't know how it is outside astronomy (hello parent parent), but inside astronomy it's relatively well known that there are four journals that are likely to be scrutinously peer reviewed and which _will_ reject you and which _will_ force you to do further analyses. It is also extremely well known that some journals advertise "In print in two months or less!" Inside the field (so to the people who will give you the next contract), it's obvious where you publish.
In some cases, it _is_ very valuable to put the extra months of effort in to lay down a rock solid piece. For example, in CAS, you can make an extra 10,000 RMB for publishing in a low impact Indian or Chinese journal. You can make upwards of 100,000 RMB (a full year's salary) for publishing in one of the "big four" journals. And up to 3 years salary for getting a Nature or Science publication. This is money being invested by the Chinese government to intentionally combat the plague of falsified Chinese research. I don't know of any other country or system that does this. It is also remarkably easy to get tenure in China, which allows a lot of long-term type focus. But this is a tiny bubble for China as it attempts to brain drain the rest of the world and is not likely to continue for long.
It's a big problem that the media doesn't understand which are which. For example when everyone was raving about the "faster than light" neutrinos a few years ago the first thing _every single person_ in my building said was "The Italians set their clock wrong didn't they?" Half a joke, but dead serious that no one should even consider this until it showed up in a publication.
Another big problem is that only basing your research on verifying others research is suicide for an academic career sing you're intentionally doing _low impact_ science. The only people willing to fund that are people with agendas, ie. BP, Exxon, Coca Cola etc.
> Few people can risk spending 10 years on something and have nothing to show for it.
This is a common sentiment, and fortunately for scientists I don't think it's a very accurate description of the real problems we face. I really don't think that there are many good reasons to spend 10 years on one thing when there is a substantial risk of failure. Scientists spend 10 years on problems all the time! But it's either part time, or has measurable metrics for progress along the way that can be published. I mean, you try to go to the moon before Pluto, right? Science (as an institution) is really a lot more like Tetris than people think, except some of the blocks (publications) get moved or destroyed later. But if the blocks are smaller, they're much easier to fit together and build upon, and fault tolerant and so forth, so if they blow up it's less of a big deal. Smaller, more well-defined and robust studies are generally better.
To change analogies to paths instead of blocks, which is a better description of the decision making a scientist goes through (rather than building the edifice of human knowledge), it's a lot more like finding your way through a new city than choosing two paths in a snowstorm that will not encounter anything for 10 years' walk. At every fork, you have to decide where to go and there will be some reason you have made that choice. After some smaller period, you can if you choose report where you've gotten and why. Other people care! They will be happy to read about it, as long as your logic is pretty good, especially if you make nice observations along the way. You can publish this, and probably no one will steal it from you. Maybe someone will... but if you do it right, do it incrementally and let people know what you're up to, you'll at least have your batshit ideas published as unreviewed abstracts and folks will know who made the first progress.
This is a common sentiment, and fortunately for scientists I don't think it's a very accurate description of the real problems we face. I really don't think that there are many good reasons to spend 10 years on one thing when there is a substantial risk of failure. Scientists spend 10 years on problems all the time! But it's either part time, or has measurable metrics for progress along the way that can be published. I mean, you try to go to the moon before Pluto, right? Science (as an institution) is really a lot more like Tetris than people think, except some of the blocks (publications) get moved or destroyed later. But if the blocks are smaller, they're much easier to fit together and build upon, and fault tolerant and so forth, so if they blow up it's less of a big deal. Smaller, more well-defined and robust studies are generally better.
To change analogies to paths instead of blocks, which is a better description of the decision making a scientist goes through (rather than building the edifice of human knowledge), it's a lot more like finding your way through a new city than choosing two paths in a snowstorm that will not encounter anything for 10 years' walk. At every fork, you have to decide where to go and there will be some reason you have made that choice. After some smaller period, you can if you choose report where you've gotten and why. Other people care! They will be happy to read about it, as long as your logic is pretty good, especially if you make nice observations along the way. You can publish this, and probably no one will steal it from you. Maybe someone will... but if you do it right, do it incrementally and let people know what you're up to, you'll at least have your batshit ideas published as unreviewed abstracts and folks will know who made the first progress.
Let's say you want to study long term impacts of taking Viagra which is clearly going to take some time. Sure, you could look at old data, but that always has issues. So, you get funding, start some research, and want to publish something when your done.
Option A: Pick just one thing to look at say cardiovascular health.
Option B: Collect a lot of data, and then look for trends.
Option B has much stronger risks of false positives simply because your looking at more factors, but those false positives seem much more interesting which helps you get published. Worse you can't use any of that data for verification, so you now need to run another 10 year study. Upside, you got published, downside, your results are almost meaningless. Bigger upside, you then get to publish again in 10 years.
Well what about option C use approach B but publish sooner. Now your not only risking false positives, but also extrapolating from more limited data.
Hmm, guess what say nutritionists are going to pick... "Chocolate good/bad for you!"
Option A: Pick just one thing to look at say cardiovascular health.
Option B: Collect a lot of data, and then look for trends.
Option B has much stronger risks of false positives simply because your looking at more factors, but those false positives seem much more interesting which helps you get published. Worse you can't use any of that data for verification, so you now need to run another 10 year study. Upside, you got published, downside, your results are almost meaningless. Bigger upside, you then get to publish again in 10 years.
Well what about option C use approach B but publish sooner. Now your not only risking false positives, but also extrapolating from more limited data.
Hmm, guess what say nutritionists are going to pick... "Chocolate good/bad for you!"
IMHO, the process is misrepresented >and< the process is broken. I think the article does a good job of explaining many (but not all) of the reasons I think it is broken.
I feel like every description of the problem of p-values I read is missing the real issue. And that's that people are using the p-value incorrectly.
A fairly standard definition of the p-value (from wikipedia) says: "In statistics, the p-value is a function of the observed sample results (a statistic) that is used for testing a statistical hypothesis. Before the test is performed, a threshold value is chosen, called the significance level of the test, traditionally 5% or 1% and denoted as α."
What this description is missing though is the crucial importance of the fact that the threshold value is chosen before doing the analysis. And moreover, that the entire analysis plan has been chosen before doing the analysis. Because what the p-value is really telling you is the probability that on repeating the experiment (and its accompanying analysis!) you would see a result as or more extreme than what you observed.
If your experiment comprises "try all of the combinations of variables to see what gives me the best answer", the p-value you compute would need to be some very fancy test that took that into account... and you would see your analysis as having much less statistical power.
For a simple example, look at a statistically rigorous method for dealing with multiple hypothesis testing when you plan it in advance: https://en.wikipedia.org/wiki/Bonferroni_correction.
Of course p-hacking is bad. The problem isn't frequentist statistics or p-values though, its scientists not understanding the statistics that they use. If you want to use a p-value to help make a decision about a hypothesis, you have to commit to your analysis plan in advance.
edit: Furthermore, p-values were designed to deal with experimental data. If you're doing an observational study, perhaps you should use statistical tools designed for that purpose.
To sum up: when you have people who have no idea what they're doing do statistics, they will do it badly.
A fairly standard definition of the p-value (from wikipedia) says: "In statistics, the p-value is a function of the observed sample results (a statistic) that is used for testing a statistical hypothesis. Before the test is performed, a threshold value is chosen, called the significance level of the test, traditionally 5% or 1% and denoted as α."
What this description is missing though is the crucial importance of the fact that the threshold value is chosen before doing the analysis. And moreover, that the entire analysis plan has been chosen before doing the analysis. Because what the p-value is really telling you is the probability that on repeating the experiment (and its accompanying analysis!) you would see a result as or more extreme than what you observed.
If your experiment comprises "try all of the combinations of variables to see what gives me the best answer", the p-value you compute would need to be some very fancy test that took that into account... and you would see your analysis as having much less statistical power.
For a simple example, look at a statistically rigorous method for dealing with multiple hypothesis testing when you plan it in advance: https://en.wikipedia.org/wiki/Bonferroni_correction.
Of course p-hacking is bad. The problem isn't frequentist statistics or p-values though, its scientists not understanding the statistics that they use. If you want to use a p-value to help make a decision about a hypothesis, you have to commit to your analysis plan in advance.
edit: Furthermore, p-values were designed to deal with experimental data. If you're doing an observational study, perhaps you should use statistical tools designed for that purpose.
To sum up: when you have people who have no idea what they're doing do statistics, they will do it badly.
> To sum up: when you have people who have no idea what they're doing do statistics, they will do it badly.
Or rather, when you have people who have a good idea what they're doing, they will do a good job of getting the results they were looking for. And maybe that is why statistics are so popular in political debates.
Or rather, when you have people who have a good idea what they're doing, they will do a good job of getting the results they were looking for. And maybe that is why statistics are so popular in political debates.
In the realm of science, I think it is far more likely that researchers don't realize why what they're doing isn't a good idea. "Never attribute to malice that which is adequately attributed to stupidity." Not that I think researchers are stupid. It's just very hard and very complicated.
I don't think that's a fair characterization of the issue. p-hacking is done almost exclusively by people who completely understand the definition and interpretation of p-values. However these people are also under a lot of pressure to produce positive results (and sometimes, results in a specific direction) and this biases their thinking.
The problem is not that people are not aware of the issue of testing multiple hypotheses. The problem is that (1) it's hard to say exactly what your hypothesis is before you've even looked at the data, and (2) it's hard to determine if people are choosing parameters for p-hacking or simply making choices based on their best judgement.
>Furthermore, p-values were designed to deal with experimental data. If you're doing an observational study, perhaps you should use statistical tools designed for that purpose.
This is simply wrong. p-values are equally relevant in both cases. E.g. I can use p-values to reject the hypothesis that consuming saturated fats is uncorrelated with weight gain, amongst the general population. It sounds like you are reaching beyond your actual expertise in statistics.
The problem is not that people are not aware of the issue of testing multiple hypotheses. The problem is that (1) it's hard to say exactly what your hypothesis is before you've even looked at the data, and (2) it's hard to determine if people are choosing parameters for p-hacking or simply making choices based on their best judgement.
>Furthermore, p-values were designed to deal with experimental data. If you're doing an observational study, perhaps you should use statistical tools designed for that purpose.
This is simply wrong. p-values are equally relevant in both cases. E.g. I can use p-values to reject the hypothesis that consuming saturated fats is uncorrelated with weight gain, amongst the general population. It sounds like you are reaching beyond your actual expertise in statistics.
If I understand you correctly, you're saying that p-values are useful when used once to evaluate preselected hypotheses with preselected thresholds. This is a common defense of p-values.
Here's what doesn't add up for me: this implies that for many configurations, p-values are invalid. But what was it about preselecting the experimental conditions that really makes them any different? What makes initially chosen hypotheses of higher quality than iteratively discovered, "p-hacked" hypotheses?
The need for careful selection of a limited set of variable combinations in advance seems symptomatic that the test being employed is not robust. I'm not convinced that even limited application of p-value significance testing is actually valid.
Here's what doesn't add up for me: this implies that for many configurations, p-values are invalid. But what was it about preselecting the experimental conditions that really makes them any different? What makes initially chosen hypotheses of higher quality than iteratively discovered, "p-hacked" hypotheses?
The need for careful selection of a limited set of variable combinations in advance seems symptomatic that the test being employed is not robust. I'm not convinced that even limited application of p-value significance testing is actually valid.
One way to think about it is that the p-value is trying to account for sampling and measurement error.
Imagine you are a scientist and want to find out whether the hypothesis that men are on average taller than women is true. If you could just exactly measure and take the average of the entire male and female populations you wouldn't need a hypothesis test.
Since you can't, you can do an experiment where you take a random sample of men and women. Now, you can do the average in the same way, but you need something to help figure out whether to trust the results. That's where the p-value comes in. The reason you need to be careful to select hypotheses in advance is because in order for statistics to help you account for error you need the noise in the data to be uncorrelated with the result you are trying to assess---which it won't be if you chose the result because it was the one that looked best after you account for the noise.
Imagine you are a scientist and want to find out whether the hypothesis that men are on average taller than women is true. If you could just exactly measure and take the average of the entire male and female populations you wouldn't need a hypothesis test.
Since you can't, you can do an experiment where you take a random sample of men and women. Now, you can do the average in the same way, but you need something to help figure out whether to trust the results. That's where the p-value comes in. The reason you need to be careful to select hypotheses in advance is because in order for statistics to help you account for error you need the noise in the data to be uncorrelated with the result you are trying to assess---which it won't be if you chose the result because it was the one that looked best after you account for the noise.
A cynical view: the job of a p-value is to make social scientists and psychologists look like they have great publication records. I sometimes think this should be set as a compulsory first exam question in any large-cohort p-value-using discipline:
Bob has a class of 100 undergraduate students. He tells them to go and run one experiment each as part of their final year class. Assuming all the data in all the experiments is completely random, what is the probability that at least one student will nonetheless find a "statistically significant effect" (p < 0.05) that can then be written up and get published.
rough answer: 99.4%
p < 0.05 seems to be chosen as that's a barrier that is not too harsh on the individual researcher (at very high significance levels it would be hard for anyone to run a powerful enough study in order to get published and keep their research job), but p-values are essentially defeated as a filter for genuine effects by the sheer number of experiments being run around the world.
In some senses, the answer here is simply to stop brandishing science as a social cudgel ("X is science; how dare you not believe it" or more often "how dare you employ/fund someone who does not believe it, they must be sacked") which puts it on a pedestal of being canonical truth all the time.
It is only the fact that science has recently been used more and more as a political stick to beat opponents with (and no, not just in climate and evolution, but right down to things like the best way to teach reading, or whether schools should be regulated or independent) that has meant that people think "science is broken" whenever it turns out to have got a wrong result. No, it's just expected to get it wrong fairly often. And in practice the most common "self-correction" process in science isn't a repeated experiment and published retraction, but just academics reading the paper, thinking "this one's garbage", and not basing their research from it.
Publication is not strictly a test of truth -- just a test of methodology and analysis. The test of truth always occurs in the mind of the reader.
Bob has a class of 100 undergraduate students. He tells them to go and run one experiment each as part of their final year class. Assuming all the data in all the experiments is completely random, what is the probability that at least one student will nonetheless find a "statistically significant effect" (p < 0.05) that can then be written up and get published.
rough answer: 99.4%
p < 0.05 seems to be chosen as that's a barrier that is not too harsh on the individual researcher (at very high significance levels it would be hard for anyone to run a powerful enough study in order to get published and keep their research job), but p-values are essentially defeated as a filter for genuine effects by the sheer number of experiments being run around the world.
In some senses, the answer here is simply to stop brandishing science as a social cudgel ("X is science; how dare you not believe it" or more often "how dare you employ/fund someone who does not believe it, they must be sacked") which puts it on a pedestal of being canonical truth all the time.
It is only the fact that science has recently been used more and more as a political stick to beat opponents with (and no, not just in climate and evolution, but right down to things like the best way to teach reading, or whether schools should be regulated or independent) that has meant that people think "science is broken" whenever it turns out to have got a wrong result. No, it's just expected to get it wrong fairly often. And in practice the most common "self-correction" process in science isn't a repeated experiment and published retraction, but just academics reading the paper, thinking "this one's garbage", and not basing their research from it.
Publication is not strictly a test of truth -- just a test of methodology and analysis. The test of truth always occurs in the mind of the reader.
> Because what the p-value is really telling you is the probability that on repeating the experiment you would see a result as or more extreme than what you observed.
That's incorrect. If you perform the exact same experiment twice, the chances are exactly 50% that the first result is more extreme than the second one (neglecting equal outcomes).
This illustrates nicely how hard it is to properly explain the p-value. Things would be more intuitive if people would report confidence intervals. I prefer "x is with a likelihood of 95% between a and b" a thousand times over "we measured x=c and reject the null hypothesis with 95% probability".
That's incorrect. If you perform the exact same experiment twice, the chances are exactly 50% that the first result is more extreme than the second one (neglecting equal outcomes).
This illustrates nicely how hard it is to properly explain the p-value. Things would be more intuitive if people would report confidence intervals. I prefer "x is with a likelihood of 95% between a and b" a thousand times over "we measured x=c and reject the null hypothesis with 95% probability".
> the chances are exactly 50% that the first result is more extreme than the second one
The chance is 50% if you condition only on the information you have before doing either experiment. But once you've done the first experiment, the chance of a more extreme result given what happened the first time may be much more or much less.
(Extreme example: Your experiment consists of rolling ten ordinary 6-sided dice. The null hypothesis is that they're fair dice, fairly rolled, in which case you expect a total not too different from 35 pips. All the dice come up 6. It is not now true that if you run the experiment again, you're as likely to get a more extreme result as you are to get a less extreme one!)
> confidence intervals [...] "x is with a likelihood of 95% between a and b"
But that isn't what a confidence interval means! A 95% confidence interval [a,b] means "If we ran the experiment lots of times, using the same method of computing the interval [a,b] each time, then in 95% of runs (in the long run) the true value would be in the interval [a,b] obtained on that run".
(What you described is what Bayesians call a "credible interval". Of course that interval depends on your prior.)
The chance is 50% if you condition only on the information you have before doing either experiment. But once you've done the first experiment, the chance of a more extreme result given what happened the first time may be much more or much less.
(Extreme example: Your experiment consists of rolling ten ordinary 6-sided dice. The null hypothesis is that they're fair dice, fairly rolled, in which case you expect a total not too different from 35 pips. All the dice come up 6. It is not now true that if you run the experiment again, you're as likely to get a more extreme result as you are to get a less extreme one!)
> confidence intervals [...] "x is with a likelihood of 95% between a and b"
But that isn't what a confidence interval means! A 95% confidence interval [a,b] means "If we ran the experiment lots of times, using the same method of computing the interval [a,b] each time, then in 95% of runs (in the long run) the true value would be in the interval [a,b] obtained on that run".
(What you described is what Bayesians call a "credible interval". Of course that interval depends on your prior.)
> in 95% of runs (in the long run) the true value would be in the interval [a,b] obtained on that run
So 95% of the runs produce an interval that includes the true value. Doesn't that imply that when doing one run, there is a 95% chance of the resulting interval including the true value? I.e. the probability is 95% that the true value is in the interval?
So 95% of the runs produce an interval that includes the true value. Doesn't that imply that when doing one run, there is a 95% chance of the resulting interval including the true value? I.e. the probability is 95% that the true value is in the interval?
Not really. For example, following a procedure which behaves well in the long run and provides 95% coverage you could get a confidence interval that contains only impossible values. Consider also the following example: you want to estimate x from two random numbers taken uniformly from the interval[x-1,x+1]. The interval defined by the two random numbers is a 50% confidence interval (half of the times x will be in the middle of the two). However, for some specific realizations (when the distance between the numbers is grater than one) you will be able to say with certainty that the interval contains x. See also https://github.com/richarddmorey/ConfidenceIntervalsFallacy
Thanks, you are right. I thought I understood confidence intervals, but apparently I did not.
kgwgk's explained why the answer is no, but let me offer a slightly different perspective.
The situation is similar to the one we discussed earlier where you do the same experiment twice. Before you do the experiment, your estimate of Pr(true value is in confidence interval) is 95%. But after you do the experiment, you have extra information: you know how the experiment turned out. And this may lead you to a different estimate of Pr(true value is in the confidence interval).
The situation is similar to the one we discussed earlier where you do the same experiment twice. Before you do the experiment, your estimate of Pr(true value is in confidence interval) is 95%. But after you do the experiment, you have extra information: you know how the experiment turned out. And this may lead you to a different estimate of Pr(true value is in the confidence interval).
[deleted]
No, my description is correct, though I left out "assuming the null hypothesis is true."
The equal probability you are describing is for the p-value itself being more extreme, not the result.
The equal probability you are describing is for the p-value itself being more extreme, not the result.
Yes, adding the clause with the null hypothesis corrects your statement.
The problem with p-values summed up in a comic, one of my all-time favourites, in fact:
https://xkcd.com/882/
https://xkcd.com/882/
This one illustrates it even better: https://xkcd.com/1132/
I think it'd be cool if there was a rogue batman scientist on the loose, self or publicly funded, who'd go around cleaning up the streets by replicating and forcing retractions.
Why batman? Because he had to fight cops just as often as baddies. Not that the current system is necessarily corrupt, but that our current system is almost as blind when it comes to recognition of good science. Our current system is a bit, well, dumb.
Why batman? Because he had to fight cops just as often as baddies. Not that the current system is necessarily corrupt, but that our current system is almost as blind when it comes to recognition of good science. Our current system is a bit, well, dumb.
Retraction Watch is close.
You'd need a rogue in each specialty - I'm also guessing that replicating studies doesn't lead to tenure or papers in Nature.
Much narrower focus than dluan's proposal (climate science only); pretty much entirely one-sided (criticizes work supporting the consensus view only, kinda like if Batman only fought the cops). The guy who runs it isn't a scientist (though you could argue that he's a statistician, which might be about as good for dluan's purposes).
So I don't think this really fits dluan's description very well.
So I don't think this really fits dluan's description very well.
From the article:
> some people have begun to ask: “Is science broken?” > I’ve spent many months asking dozens of scientists this question, and the answer I’ve found is a resounding no.
News at 11: cognitive dissonance is a thing.
> some people have begun to ask: “Is science broken?” > I’ve spent many months asking dozens of scientists this question, and the answer I’ve found is a resounding no.
News at 11: cognitive dissonance is a thing.
Science's old and new fruits benefit the lives of almost everyone on the planet every day and revolutionary new technologies, therapies and knowledge are created faster than ever before. It's accelerating, not slowing down and certainly not stopping. So it's not very broken.
Things can certainly improve, and they usually do, perhaps a bit gradually. Take a long view. Science is funded by the public and not the church these days for example. But 'broken'? Not really.
Edit: and non-scientists are very quick to think they know the problems and offer solutions. Working scientists are actively discussing the problems and trying things out all the time. These are not the dumbest people you'll meet. The current model is a bit like democracy - it's the worst system apart from all the other things we've ever tried.
To be clear: I'm not an apologist and I'm not sweeping troubles under the rug. It's just that it is complicated, we're working on it, and it is currently much too productive to be called broken. We have to be careful not to break it.
Things can certainly improve, and they usually do, perhaps a bit gradually. Take a long view. Science is funded by the public and not the church these days for example. But 'broken'? Not really.
Edit: and non-scientists are very quick to think they know the problems and offer solutions. Working scientists are actively discussing the problems and trying things out all the time. These are not the dumbest people you'll meet. The current model is a bit like democracy - it's the worst system apart from all the other things we've ever tried.
To be clear: I'm not an apologist and I'm not sweeping troubles under the rug. It's just that it is complicated, we're working on it, and it is currently much too productive to be called broken. We have to be careful not to break it.
We don't expect insiders to objectively improve most social systems (politics, business, education, defense, etc). Why should we expect scientists to fix science (as opposed to just pushing the system to serve the insiders)?
So the question is how is it broken, and what will the consequences of the proposed changes be?
Are scientific papers meant to be textbook quality results, with absolutely no errors? Or are they meant to speed the pace of discovery of true facts? Because I would argue that they are meant to be the latter, that they are effective at that, and that trying to make them more like the former would slow the pace of discovery.
Are scientific papers meant to be textbook quality results, with absolutely no errors? Or are they meant to speed the pace of discovery of true facts? Because I would argue that they are meant to be the latter, that they are effective at that, and that trying to make them more like the former would slow the pace of discovery.
We do expect insiders to improve those systems. Why would outsiders know what needs done?
We would like them to, but it's reasonable not to expect them to.
Am I the only one who thinks we seem to be spending a whole lot of money on research and getting mostly infinitesimal deltas in terms of results?
I don't think you're alone in thinking that, but the question is why do you think that and how do you justify your position? Is it because you get all your science news from the internet? Because if so I would argue that you're not getting much science news. Science is not well reported by the media, so you don't hear about much that happens.
I hope so. The world is a very different place than it was 20 years ago, with enormous progress in technology and medicine, which are the applied arm of science.
I think there needs to be a classification system for the "rigor" of studies. A low rigor rating doesn't mean bad science; it just means that the topic is very complex and difficult to study.
For instance, studies that compute the gyromagnetic ratio of the electron (to 10 decimal places no less) and then compare that against the experimentally obtained value would be classified as "high rigor". Studies that assess whether watermelon is linked to heart disease would be "low rigor".
The question now is how to come up with a rigorous classification system...
For instance, studies that compute the gyromagnetic ratio of the electron (to 10 decimal places no less) and then compare that against the experimentally obtained value would be classified as "high rigor". Studies that assess whether watermelon is linked to heart disease would be "low rigor".
The question now is how to come up with a rigorous classification system...
Rigor is a loaded word, but otherwise I love the proposal. Maybe call it confidence instead, or something?
I'd love this to extend to pursuits outside of science too, ranging from "mathematical result verified with multiple proof assistants" to "this thing I made up and argued persuasively". Unfortunately, most of my arguments for why Haskell is the best language would fall dangerously close to that end of the spectrum :P.
I'd love this to extend to pursuits outside of science too, ranging from "mathematical result verified with multiple proof assistants" to "this thing I made up and argued persuasively". Unfortunately, most of my arguments for why Haskell is the best language would fall dangerously close to that end of the spectrum :P.
> For instance, studies that compute the gyromagnetic ratio of the electron (to 10 decimal places no less) and then compare that against the experimentally obtained value would be classified as "high rigor".
You mean like, say, measuring the elementary charge of an electron to high precision? ;) Richard Feynman has something to say about the rigor of those experiments...
> We have learned a lot from experience about how to handle some of the ways we fool ourselves. One example: Millikan measured the charge on an electron by an experiment with falling oil drops, and got an answer which we now know not to be quite right. It's a little bit off because he had the incorrect value for the viscosity of air. It's interesting to look at the history of measurements of the charge of an electron, after Millikan. If you plot them as a function of time, you find that one is a little bit bigger than Millikan's, and the next one's a little bit bigger than that, and the next one's a little bit bigger than that, until finally they settle down to a number which is higher.
> Why didn't they discover the new number was higher right away? It's a thing that scientists are ashamed of—this history—because it's apparent that people did things like this: When they got a number that was too high above Millikan's, they thought something must be wrong—and they would look for and find a reason why something might be wrong. When they got a number close to Millikan's value they didn't look so hard. And so they eliminated the numbers that were too far off, and did other things like that...
[from "Surely You're Joking, Mr. Feynman!" via Wikipedia–I know this isn't exactly what you were talking about in your comment, but I thought it makes for an interesting discussion regardless]
Interesting paper on this here: http://arxiv.org/pdf/physics/0508199v1.pdf and also another example here with the Hubble constant here: http://www.pnas.org/content/101/1/8/F2.expansion.html
Even in Mullikan's case, the paper I linked seems to suggest he might have intentionally omitted data that didn't fit his narrative (I only know what I've read in that paper, so I wouldn't like to make any stronger claims than that). Whether he was 'right' to do that is an interesting question. Clearly the oil drop experiment was ground-breaking, and the results hugely important, but omitting inconvenient results is clearly a huge ethical violation. However, his intuition was correct, and he was observing a real effect, and the measurements he kept were good. So who can say? Science seems to be about asking the right questions as much as finding the right answers. Sometimes good science is just to get smart people talking about the "right" things, and the correct answers come later.
Perhaps peripheral, but another favourite example of mine is the Lennard-Jones potential for modelling e.g. noble gases. It has an attractive term and a repulsive term. The attractive term is distance to the power 6 and based on good physical reasoning; the repulsive term is distance to the power 12 and entirely arbitrary, just because it was easier to calculate since you'd already calculated the power 6 term. So, in a sense, it was completely made up. But it was still a useful tool because it helped people start asking the right questions, and exploring these systems, even if it was, in a sense, wrong.
You mean like, say, measuring the elementary charge of an electron to high precision? ;) Richard Feynman has something to say about the rigor of those experiments...
> We have learned a lot from experience about how to handle some of the ways we fool ourselves. One example: Millikan measured the charge on an electron by an experiment with falling oil drops, and got an answer which we now know not to be quite right. It's a little bit off because he had the incorrect value for the viscosity of air. It's interesting to look at the history of measurements of the charge of an electron, after Millikan. If you plot them as a function of time, you find that one is a little bit bigger than Millikan's, and the next one's a little bit bigger than that, and the next one's a little bit bigger than that, until finally they settle down to a number which is higher.
> Why didn't they discover the new number was higher right away? It's a thing that scientists are ashamed of—this history—because it's apparent that people did things like this: When they got a number that was too high above Millikan's, they thought something must be wrong—and they would look for and find a reason why something might be wrong. When they got a number close to Millikan's value they didn't look so hard. And so they eliminated the numbers that were too far off, and did other things like that...
[from "Surely You're Joking, Mr. Feynman!" via Wikipedia–I know this isn't exactly what you were talking about in your comment, but I thought it makes for an interesting discussion regardless]
Interesting paper on this here: http://arxiv.org/pdf/physics/0508199v1.pdf and also another example here with the Hubble constant here: http://www.pnas.org/content/101/1/8/F2.expansion.html
Even in Mullikan's case, the paper I linked seems to suggest he might have intentionally omitted data that didn't fit his narrative (I only know what I've read in that paper, so I wouldn't like to make any stronger claims than that). Whether he was 'right' to do that is an interesting question. Clearly the oil drop experiment was ground-breaking, and the results hugely important, but omitting inconvenient results is clearly a huge ethical violation. However, his intuition was correct, and he was observing a real effect, and the measurements he kept were good. So who can say? Science seems to be about asking the right questions as much as finding the right answers. Sometimes good science is just to get smart people talking about the "right" things, and the correct answers come later.
Perhaps peripheral, but another favourite example of mine is the Lennard-Jones potential for modelling e.g. noble gases. It has an attractive term and a repulsive term. The attractive term is distance to the power 6 and based on good physical reasoning; the repulsive term is distance to the power 12 and entirely arbitrary, just because it was easier to calculate since you'd already calculated the power 6 term. So, in a sense, it was completely made up. But it was still a useful tool because it helped people start asking the right questions, and exploring these systems, even if it was, in a sense, wrong.
That made a pretty convincing argument that it's broken, followed by thoroughly unconvincing, hand-wavey, claim to victory because "we eventually muddle through."
I like this summary of the evidence that most published research is false[1]:
-"One of the hottest topics in science has two main conclusions:
- [Ioannidis'] "Paper: Why most published research findings are false.
-"I do think that the reviewed papers are important contributions because they draw attention to real concerns about the modern scientific process. Namely
"The Many Labs results suggest that the hype about the failures of science are, at the very least, premature. I think an equally important idea is that science has pretty much always worked with some number of false positive and irreplicable studies. This was beautifully described by Jared Horvath in this blog post from the Economist. I think the take home message is that regardless of the rate of false discoveries, the scientific process has led to amazing and life-altering discoveries."
[1] http://simplystatistics.org/2013/12/16/a-summary-of-the-evid...
-"One of the hottest topics in science has two main conclusions:
Most published research is false
There is a reproducibility crisis in science
- [Ioannidis'] "Paper: Why most published research findings are false.
Main idea: People use hypothesis testing to determine if specific scientific discoveries are significant. This significance calculation is used as a screening mechanism in the scientific literature. Under assumptions about the way people perform these tests and report them it is possible to construct a universe where most published findings are false positive results.
Important drawback: The paper contains no real data, it is purely based on conjecture and simulation."
-"I do think that the reviewed papers are important contributions because they draw attention to real concerns about the modern scientific process. Namely
We need more statistical literacy
We need more computational literacy
We need to require code be published
We need mechanisms of peer review that deal with code
We need a culture that doesn't use reproducibility as a weapon
We need increased transparency in review and evaluation of papers"
"The Many Labs results suggest that the hype about the failures of science are, at the very least, premature. I think an equally important idea is that science has pretty much always worked with some number of false positive and irreplicable studies. This was beautifully described by Jared Horvath in this blog post from the Economist. I think the take home message is that regardless of the rate of false discoveries, the scientific process has led to amazing and life-altering discoveries."
[1] http://simplystatistics.org/2013/12/16/a-summary-of-the-evid...
Science and modern capitalistic thinking do not work well together.
Most people in business view any sort of rigorous research as a "cost" center. And this thinking has diffused into government too.
On the other hand marketing and advertising produces the most value for dollar. Why ?
In a market those activities generate new demand - something that a capitastic system needs in order to survive.
Why spend 20 years doing research while we could spend 1 year aggressively expanding our market. Even a 1% growth would he worth it given a 0% growth if the money was spent doing research.
I like that rather than villify a single person Nate silver points the finger at the system. This type of thinking is where we should all be heading towards.
Most people in business view any sort of rigorous research as a "cost" center. And this thinking has diffused into government too.
On the other hand marketing and advertising produces the most value for dollar. Why ?
In a market those activities generate new demand - something that a capitastic system needs in order to survive.
Why spend 20 years doing research while we could spend 1 year aggressively expanding our market. Even a 1% growth would he worth it given a 0% growth if the money was spent doing research.
I like that rather than villify a single person Nate silver points the finger at the system. This type of thinking is where we should all be heading towards.
It is worrisome that you were here downvoted for this view; as it is one that I believe (as a working scientist) is one of the larger issues that has been left out of this discussion. For example, public universities used to be fairly unconstrained in their support of grant research. Now most, if not all, have an office devoted to the commercialization (patent protection) of their own publicly supported research; quite often subsidized by private interests. So if a researcher wanted to study the effects of cell-phone usage on the developing brain, (which didn't involve me, but rather a colleague), then suddenly several large companies were going to "lamentably" pull out of supporting the athletic department with its "amateur sports" endeavors. And the list of subtle control of "public" research continues. Science and commercialism can mix productively, but it should never be done covertly. Unfortunately corporate thinking is nearly always covert.
On the contrary, only free markets generate enough surplus capital for it to be invested in long term research. Planned and managed economies don't generate enough production to even feed and supply their existing populations well.
It's a false dichotomy to say that you can only choose between 20 year research and 1 year marketing. Plenty of private money goes into research with long term payoff. There are private companies engaged in fusion research, because the payoff is huge if it is achieved. Even the public money that goes into publicly funded research comes from private earnings in the first place. So science and free markets are absolutely suited to each other. That's before we even start discussing corruption of objectives in totalitarian states, based on leaders whims (ie, Nazi science research, Lysenko etc)
It's a false dichotomy to say that you can only choose between 20 year research and 1 year marketing. Plenty of private money goes into research with long term payoff. There are private companies engaged in fusion research, because the payoff is huge if it is achieved. Even the public money that goes into publicly funded research comes from private earnings in the first place. So science and free markets are absolutely suited to each other. That's before we even start discussing corruption of objectives in totalitarian states, based on leaders whims (ie, Nazi science research, Lysenko etc)
That is not true and the problem is that private capital usually expects certain results, or it cuts funding.
"Planned and managed economies don't generate enough production to even feed and supply their existing populations well."
Sorry to tell you but capitalist society as it exists is planned and managed economy. Large corporations are command economies.
You should see what science has discovered about the brain:
https://www.youtube.com/watch?v=PYmi0DLzBdQ
Sorry to tell you but capitalist society as it exists is planned and managed economy. Large corporations are command economies.
You should see what science has discovered about the brain:
https://www.youtube.com/watch?v=PYmi0DLzBdQ
This article is a good summary of the issues of the (mis)use of statistics in science. Having done some research myself, in most cases the statistics don't scream a particular message to you, and it's really hard to understand the data. Even without any external pressures, statistical tools like p-values have limits. If you run 10 different models (all a priori reasonable) and 8 of them seem to say roughly the same thing, does that mean your result is correct?
Science might not be broken for the reasons mentioned, but I find the article overlooks the greatest evidence of all which is the article itself.
Objectively deconstructing the practice of p-hacking is science at work and a scientific work. So say, if we devised a way to measure p-value based research for it's p-hack-ability, we would have a way to validate what that research is saying (or not saying), as well as test the integrity of the research and its researchers.
Objectively deconstructing the practice of p-hacking is science at work and a scientific work. So say, if we devised a way to measure p-value based research for it's p-hack-ability, we would have a way to validate what that research is saying (or not saying), as well as test the integrity of the research and its researchers.
I saw a great talk about this @ SIAM data mining by Dave Madigan, for http://omop.org/
Basically, they took a TON of demographic research in the health sciences, explored all possible hyperparameter tunings, and found that they could get p-values of 0.05 in either direction for most of the papers depending on choice of data sources and many other types of hyperparameters.
Basically, they took a TON of demographic research in the health sciences, explored all possible hyperparameter tunings, and found that they could get p-values of 0.05 in either direction for most of the papers depending on choice of data sources and many other types of hyperparameters.
On the other hand, science writes want a headline "Something baffles scientists." Well, the scientists are probably baffled because it contradicts their intuition, which is a strong indicator that something is not really working. This something may be a honest mistake by the authors, a poorly understood experimental effect, outright fraud or it may be something worth reporting. The entire process of science writing is geared towards the papers which are most likely to be wrong.