Paul Meier - Statistican who saved millions of lives(significancemagazine.org)
significancemagazine.org
Paul Meier - Statistican who saved millions of lives
http://www.significancemagazine.org/details/webexclusive/1315465/Paul-Meier---Statistican-who-saved-millions-of-lives.html
3 comments
This is one of those moments for me where you kinda assumed a technique was used in the past, but the truth is it had to be invented. Then when you look at the past with this knowledge, you think, "My god, how barbaric!". Sort of how McCoy (a la Star Trek) acts when he goes back in time.
Not all problems with trials have been ironed out. Only as of 2004, have some medical journals been requiring pre-registration of clinical trials. Previously, drug manufacturers could pretend null results didn't exist.
http://en.wikipedia.org/wiki/Publication_bias
http://en.wikipedia.org/wiki/Publication_bias
First, I think this is a closed view of the issue. Second, I think publication bias has more effect on basic research and translational research (read: early animal pharmacology studies) than drug development.
Conduct of a clinical trial for the purposes of establishing the safety and efficacy of a drug product intended for a specific clinical indication has the goal of registration (NDA, BLA, PMA, 510(k)) of the drug so that the drug can be marketed and sold. Publications are great, and are a component of that, but not the primary goal.
To conduct a clinical study, the company must present all preclinical (FDA calls it nonclinical) pharmacology and toxicology on the drug product to determine whether the product is safe for testing in human trials. This is done via an investigational new drug application (IND, big damn documents that take months to prepare). But I digress...
To conduct a clinical study, the protocol has to be submitted and FDA given time to comment (and they will, often and without much room to argue). That means the patient population, randomization scheme, endpoints, power analysis, etc. are vetted by FDA pretty thoroughly. Next, the same process occurs for the study at local or national (central) institutional review boards (IRBs). They get really nitpicky (especially local university IRBs for numerous reasons). Their primary goal is to ensure the safety of patients, but they won't hesitate to comment on the design of the study in most cases. And (not sure exactly when it became a requirement) sponsors have to register their clinical trial with a national registry (which you can search on clinicaltrials.gov). The results of those studies MUST be submitted to the FDA under the IND, to the IRB, and updated in the registry, or development stops and the IND cancelled.
So unless I misunderstand your comment, the publication practices of journals affects the validity and thoroughness of clinical trial data. This is not the case. Publication bias may alter the available literature, but drugs are NOT registered on publications and effected by "pre-registration." Drugs are developed and brought to market through a seriously controlled and monitored process, not at the whim of medical journals. Now, preclinical data, basic science, "academic studies" not moving a drug toward registration, etc. are affected by publication bias. It is a problem, and pre-registration is a complex issue but beneficial.
One final note: Writing this comment made me think that an alternative form of pre-registration would be really interesting. But not to medical journals, but an open registry for "coming soon research science." Think of a registry of "here's what we want to do" for researchers in basic science, all putting up ideas (ones they are willing to do so openly) for preclinical or clinical studies on the site, indexed by keyword. This would allow researchers to look online and not see what things HAVE been done, but see what things are GOING to be done. This could help researchers identify colleagues (communicate via social features), collaborate on the research, and pool their funding efforts to get multi-institutional grants (usually higher dollar and currently a little less competitive and thus easier to get). Put your research field and interest, and studies you would do "if you had the money" and then the registry would send you recommendations of people to connect with to try and move those research areas forward. If this does not currently exist (which it does not, to my knowledge), it would be amazing to have. It may also be an easier task to implement as a new feature of an existing research social network or community. "Enhancing serendipity in research..." Interesting. Thoughts?
Conduct of a clinical trial for the purposes of establishing the safety and efficacy of a drug product intended for a specific clinical indication has the goal of registration (NDA, BLA, PMA, 510(k)) of the drug so that the drug can be marketed and sold. Publications are great, and are a component of that, but not the primary goal.
To conduct a clinical study, the company must present all preclinical (FDA calls it nonclinical) pharmacology and toxicology on the drug product to determine whether the product is safe for testing in human trials. This is done via an investigational new drug application (IND, big damn documents that take months to prepare). But I digress...
To conduct a clinical study, the protocol has to be submitted and FDA given time to comment (and they will, often and without much room to argue). That means the patient population, randomization scheme, endpoints, power analysis, etc. are vetted by FDA pretty thoroughly. Next, the same process occurs for the study at local or national (central) institutional review boards (IRBs). They get really nitpicky (especially local university IRBs for numerous reasons). Their primary goal is to ensure the safety of patients, but they won't hesitate to comment on the design of the study in most cases. And (not sure exactly when it became a requirement) sponsors have to register their clinical trial with a national registry (which you can search on clinicaltrials.gov). The results of those studies MUST be submitted to the FDA under the IND, to the IRB, and updated in the registry, or development stops and the IND cancelled.
So unless I misunderstand your comment, the publication practices of journals affects the validity and thoroughness of clinical trial data. This is not the case. Publication bias may alter the available literature, but drugs are NOT registered on publications and effected by "pre-registration." Drugs are developed and brought to market through a seriously controlled and monitored process, not at the whim of medical journals. Now, preclinical data, basic science, "academic studies" not moving a drug toward registration, etc. are affected by publication bias. It is a problem, and pre-registration is a complex issue but beneficial.
One final note: Writing this comment made me think that an alternative form of pre-registration would be really interesting. But not to medical journals, but an open registry for "coming soon research science." Think of a registry of "here's what we want to do" for researchers in basic science, all putting up ideas (ones they are willing to do so openly) for preclinical or clinical studies on the site, indexed by keyword. This would allow researchers to look online and not see what things HAVE been done, but see what things are GOING to be done. This could help researchers identify colleagues (communicate via social features), collaborate on the research, and pool their funding efforts to get multi-institutional grants (usually higher dollar and currently a little less competitive and thus easier to get). Put your research field and interest, and studies you would do "if you had the money" and then the registry would send you recommendations of people to connect with to try and move those research areas forward. If this does not currently exist (which it does not, to my knowledge), it would be amazing to have. It may also be an easier task to implement as a new feature of an existing research social network or community. "Enhancing serendipity in research..." Interesting. Thoughts?
There have been corporations who suppressed evidence that their drugs didn't work right. I'd describe that as publication bias, too, if not simply fraud...
I agree that there are long-term safety and efficacy studies that are fraudulently suppressed by some larger players in industry. It is an insidious problem that has horrible effects on patients. However, it is separate from my argument.
My point was that the registration of drugs is not effected by publication bias. The "suppressed evidence" typically comes in the form of post-marketing studies in significant patient populations, some of which aren't even randomized or placebo controlled (some are). NDA-directed studies are tightly controlled and reported. Then the drug is registered, and the FDA says, "we'd still like to see long-term safety data on this to see what things look like over time. We'll let you put this drug out there, but keep sending us more data."
I think the problem therein is related to the regs on post-marketing clinical studies. Companies have too much latitude to conduct post-marketing studies and then decide what to send FDA (or to publish), thus introducing the publication bias. FDA should have more control over Phase IV studies. The Prescription Drug User Fee Act (PDUFA, or pud-oof-a) was established to charge companies over $1million to register a drug, funneling money back into FDA so they could acquire more resources to review and grand NDAs. So the review times for NDAs went down, FDA got more efficient, and industry was happier. However, the act did not increase Phase IV controls as stringently as they should have, and companies have a lot of latitude on "additional clinical data" from studies conducted after the drug hits the market. That's why you see articles like the following:
GSK Suppresses Paxil Suicide Data: http://goo.gl/LCM94
AstraZeneca Suppresses Data on Seroquel: http://goo.gl/408fJ
Pfizer Suppresses Data on Edronax: http://goo.gl/fxLbW
So yes, there is publication bias and possibly fraud at work here, but it lies, in my view, early in the process (preclinical, basic science) and very late in the process (Phase IV studies). The registration process is pretty well controlled.
Note: My expertise is in preclinical pharmacology, toxicology, and NDA-directed studies (Phase I-III) not post-marketing studies (Phase IV), so if anyone has more experience here please speak up.
My point was that the registration of drugs is not effected by publication bias. The "suppressed evidence" typically comes in the form of post-marketing studies in significant patient populations, some of which aren't even randomized or placebo controlled (some are). NDA-directed studies are tightly controlled and reported. Then the drug is registered, and the FDA says, "we'd still like to see long-term safety data on this to see what things look like over time. We'll let you put this drug out there, but keep sending us more data."
I think the problem therein is related to the regs on post-marketing clinical studies. Companies have too much latitude to conduct post-marketing studies and then decide what to send FDA (or to publish), thus introducing the publication bias. FDA should have more control over Phase IV studies. The Prescription Drug User Fee Act (PDUFA, or pud-oof-a) was established to charge companies over $1million to register a drug, funneling money back into FDA so they could acquire more resources to review and grand NDAs. So the review times for NDAs went down, FDA got more efficient, and industry was happier. However, the act did not increase Phase IV controls as stringently as they should have, and companies have a lot of latitude on "additional clinical data" from studies conducted after the drug hits the market. That's why you see articles like the following:
GSK Suppresses Paxil Suicide Data: http://goo.gl/LCM94
AstraZeneca Suppresses Data on Seroquel: http://goo.gl/408fJ
Pfizer Suppresses Data on Edronax: http://goo.gl/fxLbW
So yes, there is publication bias and possibly fraud at work here, but it lies, in my view, early in the process (preclinical, basic science) and very late in the process (Phase IV studies). The registration process is pretty well controlled.
Note: My expertise is in preclinical pharmacology, toxicology, and NDA-directed studies (Phase I-III) not post-marketing studies (Phase IV), so if anyone has more experience here please speak up.
I recall recently reading about a clinical trial which was aborted recently on moral grounds - the treatment being tested was so effective they couldn't justify continuing to administer placebos. (Annoyingly, I can't find either the article or the name of the drug at the moment)
Since you seem to have a good understanding of the process, what would happen in this case? I'd guess something along the lines of "We've already established statistical significance, we can proceed", but wouldn't that potentially weaken the study of things like side-effects or long-term use?
Do testing protocols have provisions for this sort of thing?
Since you seem to have a good understanding of the process, what would happen in this case? I'd guess something along the lines of "We've already established statistical significance, we can proceed", but wouldn't that potentially weaken the study of things like side-effects or long-term use?
Do testing protocols have provisions for this sort of thing?
Hopefully obvious, but some of this stuff also applies to A/B testing.
Yet in the 1950s the usual technique was to give a new treatment to the patients whom it was thought would most benefit from it. It frequently happened that those were the patients whose chances of recovery were the best in any case, under the existing treatments as well as the new one. The result was all too often that new treatments were thought to be better than old ones but in fact were not.
Yet in the 1950s the usual technique was to give a new treatment to the patients whom it was thought would most benefit from it. It frequently happened that those were the patients whose chances of recovery were the best in any case, under the existing treatments as well as the new one. The result was all too often that new treatments were thought to be better than old ones but in fact were not.
> ‘When I said “randomize” in breast cancer trials I was looked at with amazement by my clinical colleagues’ said Meier in an interview in 2004. ‘ “Randomize? We know this treatment is better than that one” they said. I said “Not really…” ’