Why Are Clinical Trials So Complicated?(blogs.sciencemag.org)
blogs.sciencemag.org
Why Are Clinical Trials So Complicated?
https://blogs.sciencemag.org/pipeline/archives/2020/05/01/why-are-clinical-trials-so-complicated
63 comments
On gaming: it's very interesting that, even within all of the byzantine rules and regulations and guidances how much gray area and leeway there can be. From a drug/device company perspective, which people in the FDA assigned as reviewers / contact points for you and how well you are able to articulate & convince them of your goals can play a huge role in whether you have to do X type of trial or adhere to Y rules. This is especially true at the boundaries (new types of testing, devices, technologies) that don't quite exactly fit the more well-trod paths. Not saying this in a judgmental way -- it's just the nature of dealing with extremely complex problems while also making sure people aren't harmed unnecessarily and that at the end when you say drug/device/tech A improves condition B we actually are sure it is true.
And then there's the more malicious "gaming", fraud! Clinical research is obviously heavily incentivized toward positive trial outcomes, but for each site the act of just enrolling patients is incentivized, so you need to be damn sure the patient data (and the patients themselves!) in your study are real. And being sure is not easy or cheap.
For fun, here's the FDA's list of people who have been blacklisted from performing clinical research: https://www.fda.gov/inspections-compliance-enforcement-and-c...
Their associated "Volume Page" lists (in excruciating bureaucratese) their "crimes", which may be unrelated illegal narcotics peddling (oops) or falsifying patient data.
And then there's the more malicious "gaming", fraud! Clinical research is obviously heavily incentivized toward positive trial outcomes, but for each site the act of just enrolling patients is incentivized, so you need to be damn sure the patient data (and the patients themselves!) in your study are real. And being sure is not easy or cheap.
For fun, here's the FDA's list of people who have been blacklisted from performing clinical research: https://www.fda.gov/inspections-compliance-enforcement-and-c...
Their associated "Volume Page" lists (in excruciating bureaucratese) their "crimes", which may be unrelated illegal narcotics peddling (oops) or falsifying patient data.
I wouldn't really consider "articulate & convince them of your goals" as really gaming. There are areas that are new for everyone, and you are correct, if you don't come up with your approach, the FDA will come up with one for you.
What I've appreciated about the FDA is their scientific focus. The FDA might say "You need to do X", but if you can come back and convince them with a scientific argument that you don't have to, they will listen to you!
What I've appreciated about the FDA is their scientific focus. The FDA might say "You need to do X", but if you can come back and convince them with a scientific argument that you don't have to, they will listen to you!
The gaming has really made it harder to work with the Agency because you can't just call up or stop by and ask a question any more -- your interactions are more structured and formal.
I agree that the shift to biomarker endpoints has made the process a bit more aggressive (in probably a good way) but has also lead to all sorts of opportunities for gaming you way to pointless success...but then you have something in the market with something you can talk about!
I agree that the shift to biomarker endpoints has made the process a bit more aggressive (in probably a good way) but has also lead to all sorts of opportunities for gaming you way to pointless success...but then you have something in the market with something you can talk about!
I've worked on the drug side of the business and I agree with your assessment. I've found the FDA to be quite reasonable, and also very science driven. Most of the regulations are there because without them, problems have arisen in the past.
And as a consumer I'm very happy that if I get prescribed a drug it's been thoroughly evaluated for safety and efficacy.
And as a consumer I'm very happy that if I get prescribed a drug it's been thoroughly evaluated for safety and efficacy.
There is an issue with the framing there. From something like the dawn of written recording through to 1950, a very large number of advances were very dangerous for the people involved. Entire cities have been built and powered major advances in the state of the art despite being built in a fashion that would be illegal under a modern building code. Medical science is riddled with people doing crazy things to themselves and others. Biology has some shockers in the closet. Grand infrastructure building projects have always had a body count.
I would very much struggle to pick on a point in time up to now where a society picking 'do things safely' has developed a real advantage over a society that went with 'learn quickly'. The learn quickly people make truly horrific mistakes, but then have an opportunity to learn from them and settle into much more sophisticated equilibrium where they achieve better future results in perpetuity.
Government regulation makes things safer by banning unsafe practices. We don't see the damage this does because stuff that doesn't happen never gets seen. However, it is worth holding on to the idea that it just isn't the government's job to keep someone safe when the person involved doesn't care. People should be allowed to buy and treat themselves with potentially unsafe or ineffective drugs if they understand the risks and have read up on cases like, eg, Thalidomide. There will be benefits from learning from the mistakes and since it isn't possible to live a risk-free life the government shouldn't be trying to enforce it. A strategy of education and information is great. A strategy of prohibition and red tape is actually big-picture risky in itself with a a counterfactual point of view where things are experimented with widely.
I would very much struggle to pick on a point in time up to now where a society picking 'do things safely' has developed a real advantage over a society that went with 'learn quickly'. The learn quickly people make truly horrific mistakes, but then have an opportunity to learn from them and settle into much more sophisticated equilibrium where they achieve better future results in perpetuity.
Government regulation makes things safer by banning unsafe practices. We don't see the damage this does because stuff that doesn't happen never gets seen. However, it is worth holding on to the idea that it just isn't the government's job to keep someone safe when the person involved doesn't care. People should be allowed to buy and treat themselves with potentially unsafe or ineffective drugs if they understand the risks and have read up on cases like, eg, Thalidomide. There will be benefits from learning from the mistakes and since it isn't possible to live a risk-free life the government shouldn't be trying to enforce it. A strategy of education and information is great. A strategy of prohibition and red tape is actually big-picture risky in itself with a a counterfactual point of view where things are experimented with widely.
What you are describing is beyond unethical. How do you educate and provide information about the drug without an understanding whether it works or is safe?
Thalidomide didn’t have a sticker that said “will cause birth defects in your unborn child”.
Medical science has produced untold numbers of what would be classified as miracles 70 years ago. I can name a half dozen people I’m personally acquainted with who live today because of effective, safe treatments.
As the disgusting affair around COVID has demonstrated, hucksters and idiots will happily push unsafe, ineffective treatments for a variety of self-serving reasons. Or if that’s too controversial, how many needlessly suffered from application of junk cures like bleeding or mercury?
Thalidomide didn’t have a sticker that said “will cause birth defects in your unborn child”.
Medical science has produced untold numbers of what would be classified as miracles 70 years ago. I can name a half dozen people I’m personally acquainted with who live today because of effective, safe treatments.
As the disgusting affair around COVID has demonstrated, hucksters and idiots will happily push unsafe, ineffective treatments for a variety of self-serving reasons. Or if that’s too controversial, how many needlessly suffered from application of junk cures like bleeding or mercury?
> How do you educate and provide information about the drug without an understanding whether it works or is safe?
How do you do it for the people in a clinical trial? Somebody has to be the first.
> Medical science has produced untold numbers of what would be classified as miracles 70 years ago. I can name a half dozen people I’m personally acquainted with who live today because of effective, safe treatments.
The argument is not that no progress is made using more conservative methods, it's that there is a level of "safety" where the additional progress not made would have saved more lives than would be lost to somewhat less conservative methods.
> As the disgusting affair around COVID has demonstrated, hucksters and idiots will happily push unsafe, ineffective treatments for a variety of self-serving reasons.
Meanwhile the bureaucratic red tape around initial testing allowed the virus to spread as far as it did in the early days, dramatically increasing the scope of the problem and costing many lives.
How do you do it for the people in a clinical trial? Somebody has to be the first.
> Medical science has produced untold numbers of what would be classified as miracles 70 years ago. I can name a half dozen people I’m personally acquainted with who live today because of effective, safe treatments.
The argument is not that no progress is made using more conservative methods, it's that there is a level of "safety" where the additional progress not made would have saved more lives than would be lost to somewhat less conservative methods.
> As the disgusting affair around COVID has demonstrated, hucksters and idiots will happily push unsafe, ineffective treatments for a variety of self-serving reasons.
Meanwhile the bureaucratic red tape around initial testing allowed the virus to spread as far as it did in the early days, dramatically increasing the scope of the problem and costing many lives.
Clinical trials have informed consent and medical monitoring.
I don’t understand how someone could argue that establishing that a drug works and will not harm you is “red tape”. I won’t comment on COVID testing because the facts are not established and the depths of incompetence and criminality associated with the topic are yet to be discovered.
I don’t understand how someone could argue that establishing that a drug works and will not harm you is “red tape”. I won’t comment on COVID testing because the facts are not established and the depths of incompetence and criminality associated with the topic are yet to be discovered.
> Clinical trials have informed consent and medical monitoring.
How is "informed consent" (to the extent that that is even possible) not possible outside the context of clinical trials?
And "medical monitoring" doesn't mean it won't kill you, it only means if you die they write it down.
> I don’t understand how someone could argue that establishing that a drug works and will not harm you is “red tape”.
It prevents you from using a drug that could work based on no evidence that it doesn't. That is a much more aggressively authoritarian stance than prohibiting something which is actually known to not work or be harmful.
It also prevents you from using things that are known to work, and are used in other countries, but have no one to pay for the US clinical trials because no one holds a patent on it, e.g. because it's a part of a plant.
How is "informed consent" (to the extent that that is even possible) not possible outside the context of clinical trials?
And "medical monitoring" doesn't mean it won't kill you, it only means if you die they write it down.
> I don’t understand how someone could argue that establishing that a drug works and will not harm you is “red tape”.
It prevents you from using a drug that could work based on no evidence that it doesn't. That is a much more aggressively authoritarian stance than prohibiting something which is actually known to not work or be harmful.
It also prevents you from using things that are known to work, and are used in other countries, but have no one to pay for the US clinical trials because no one holds a patent on it, e.g. because it's a part of a plant.
Like i said, there are a host of ethical issues.
We live in a world where Alex Jones exists and people will use religion and hocus pocus vitamins/supplements to peddle fake medicine.
Those charlatans live in a regulatory loophole that allows dietary supplements to be marketed in fraudulent ways. So it’s pretty obvious to figure out what would go wrong with medicine if you care to look.
Most of the issues you are worried about are more about the business model of medicine.
We live in a world where Alex Jones exists and people will use religion and hocus pocus vitamins/supplements to peddle fake medicine.
Those charlatans live in a regulatory loophole that allows dietary supplements to be marketed in fraudulent ways. So it’s pretty obvious to figure out what would go wrong with medicine if you care to look.
Most of the issues you are worried about are more about the business model of medicine.
> And "medical monitoring" doesn't mean it won't kill you, it only means if you die they write it down.
I think perhaps you've never run a clinical trial. What's involved is far beyond what you are describing.
I think perhaps you've never run a clinical trial. What's involved is far beyond what you are describing.
You are taking an untested experimental drug. It could kill you, or cause paralysis or brain damage or sterility or cancer or organ failure and require you to get a transplant and spend the rest of your life on immunosuppressants.
Monitoring can't save you when the damage is already done by the time the evidence of damage is discovered.
Monitoring can't save you when the damage is already done by the time the evidence of damage is discovered.
Let me flip it around the other way: because many people were not casually poisoned due to laws about drug safety, aerosolized lead and lead paint chips, food safety, seat belts etc there are more people around today to develop new things. This is the Julian Simon argument — and he was a famous libertarian!
> Meanwhile the bureaucratic red tape around initial testing allowed the virus to spread
I see this lazy argument all the time. A test that has not been characterized and qualified produces unknown results (what percentage are false positive? False negative? Does that improve if you tweak the protocol?). If your test produces unknown results they are simply noise, and increasing the rate at which you produce noise cannot improve your judgement.
> Meanwhile the bureaucratic red tape around initial testing allowed the virus to spread
I see this lazy argument all the time. A test that has not been characterized and qualified produces unknown results (what percentage are false positive? False negative? Does that improve if you tweak the protocol?). If your test produces unknown results they are simply noise, and increasing the rate at which you produce noise cannot improve your judgement.
> Let me flip it around the other way: because many people were not casually poisoned due to laws about drug safety, aerosolized lead and lead paint chips, food safety, seat belts etc there are more people around today to develop new things.
That doesn't flip it around at all, it's still the same problem. There is a trade off between costing lives by being too reckless and costing lives by being too conservative. Your argument in fact exacerbates the problem of being too conservative -- of being too far away from the optimal trade off in either direction -- because it applies just as much to the lives lost to slower progress.
And banning lead or similar is a completely different situation because the data is in on lead. It's not prohibited because we don't know if it's bad for you or not, it's prohibited because we do know that it's very bad for you with a high degree of confidence. It's obvious what to do with things that are well known, the question is what to do with things that are not.
> A test that has not been characterized and qualified produces unknown results
Known and unknown are not binary states. You can have a new test which is more likely to work than not based on all currently available evidence and you have to choose whether to use it during the period when better data is being gathered. The best answer is not automatically no.
That is different than the problem for approvals in general, but still the same. In the normal case you have to choose whether less rigorous use is allowed in early stages which provides more data prior to making the decision of whether to even do the more expensive more rigorous trials, thereby making the rigorous trials that are actually performed more likely to have positive results and increasing the number of effective treatments on the market.
That doesn't flip it around at all, it's still the same problem. There is a trade off between costing lives by being too reckless and costing lives by being too conservative. Your argument in fact exacerbates the problem of being too conservative -- of being too far away from the optimal trade off in either direction -- because it applies just as much to the lives lost to slower progress.
And banning lead or similar is a completely different situation because the data is in on lead. It's not prohibited because we don't know if it's bad for you or not, it's prohibited because we do know that it's very bad for you with a high degree of confidence. It's obvious what to do with things that are well known, the question is what to do with things that are not.
> A test that has not been characterized and qualified produces unknown results
Known and unknown are not binary states. You can have a new test which is more likely to work than not based on all currently available evidence and you have to choose whether to use it during the period when better data is being gathered. The best answer is not automatically no.
That is different than the problem for approvals in general, but still the same. In the normal case you have to choose whether less rigorous use is allowed in early stages which provides more data prior to making the decision of whether to even do the more expensive more rigorous trials, thereby making the rigorous trials that are actually performed more likely to have positive results and increasing the number of effective treatments on the market.
Are you arguing the FDA was not involved in the U.S.'s delayed testing roll out?
Because everyone else I've read in the medical industry made the argument they (and the CDC) were a large cause of why our testing was behind other countries. (this includes a previous director of the FDA)
Because everyone else I've read in the medical industry made the argument they (and the CDC) were a large cause of why our testing was behind other countries. (this includes a previous director of the FDA)
I think you need to show evidence that
1.the speed of innovation was indeed higher previously, and
2. that it was higher because of less regulation/safety what are the innovations that would not have been discovered today? you're just asserting this without anything to back it up.
At least 1. Contradicts most what I have read, which is that our speed of innovation/knowledge acquisition is accelerating
At least 1. Contradicts most what I have read, which is that our speed of innovation/knowledge acquisition is accelerating
The saying in our industry is, "clinical trials are the most complicated thing in the world".
EDIT: adding details:
You have to mobilize large interdisciplinary teams to come up with products that are inherently very hard to invent, and which, after all that, can easily fail a trail and never see light of day, or can then still harm or kill people even after approved, all while operating in an understandbly extremely regulated and complicated environment.
One example:
One of the many piles of documents that you need to submit to a national health authority _just so they consider your product_ (before you can release to a single "user"!), is thorough documentation of how you _plan_ to manufacture the product, if it were approved.
Just these text files can weigh _gigabytes_. Think how much effort it would take just to write something coherent that is so long! And of course, you have to get other teams to run a bunch of experiments to inform what to write.
And this is even before you are actually producing the product for release.
Oh, and maybe your product won't actually be approved, so all that goes to waste.
And that's just the beginning.
Because all of that excludes _any_ mention of the real deal, which is whether the product actually works or is safe. All of that requires its own experiments and documentation.
And all that assumes that decades of prior research actually led to some useful prototype where you can start.
And all that excludes any mention of ethical considerations like who pays for the research, what is a just price for a cancer drug - heck, whether you should even be trying to make a profit off of lifesaving drugs.
If getting to a point like selling your app called Whats App for $18 billion (or whatever it was) would require a walk down the street, then getting a drug to market is like leaving the earth, flying past the furthest planets, and then trying to reach another galaxy.
EDIT: adding details:
You have to mobilize large interdisciplinary teams to come up with products that are inherently very hard to invent, and which, after all that, can easily fail a trail and never see light of day, or can then still harm or kill people even after approved, all while operating in an understandbly extremely regulated and complicated environment.
One example:
One of the many piles of documents that you need to submit to a national health authority _just so they consider your product_ (before you can release to a single "user"!), is thorough documentation of how you _plan_ to manufacture the product, if it were approved.
Just these text files can weigh _gigabytes_. Think how much effort it would take just to write something coherent that is so long! And of course, you have to get other teams to run a bunch of experiments to inform what to write.
And this is even before you are actually producing the product for release.
Oh, and maybe your product won't actually be approved, so all that goes to waste.
And that's just the beginning.
Because all of that excludes _any_ mention of the real deal, which is whether the product actually works or is safe. All of that requires its own experiments and documentation.
And all that assumes that decades of prior research actually led to some useful prototype where you can start.
And all that excludes any mention of ethical considerations like who pays for the research, what is a just price for a cancer drug - heck, whether you should even be trying to make a profit off of lifesaving drugs.
If getting to a point like selling your app called Whats App for $18 billion (or whatever it was) would require a walk down the street, then getting a drug to market is like leaving the earth, flying past the furthest planets, and then trying to reach another galaxy.
Is a WhatsApp-like exit really billions of times more probable than bringing a new drug to market? I am somewhat doubtful..
There’s certainly a large degree of chance involved in bringing a new drug to market but I read the closing comment of antipaul’s post to mean that it is the amount of human labour and other resources required to have that chance of success is magnitudes greater than that involved in building a tech startup.
Much of this is regulatory burden, not inherently scientific problems.
Most of those regulations are written in the blood of patients.
In addition, I suspect that you have never debugged a delicate physics experiment.
A single experiment could fill a dozen notebooks with records of what I tried and used. There were records of serial numbers, calibration sequences, colors of wires, etc. I could go on and on.
Why?
Because when something didn't work, I needed to have "ground truth" as to what was working so I could focus on the (hopefully contained) surface of uncertainty. And this was for a physics experiment where the outcome is KNOWN.
If you don't actually know the outcome, everything gets worse as you don't have a definite signal between "Hypothesis is wrong" vs "Experimental setup is broken". Now, multiply this by thousands of steps for a drug. Brrrrrrrrrr.
To top it off, the physical world isn't like software, you often can't go backwards. In the physical world, if you didn't record everything on the forward pass, you may have to go back to start. :(
In addition, I suspect that you have never debugged a delicate physics experiment.
A single experiment could fill a dozen notebooks with records of what I tried and used. There were records of serial numbers, calibration sequences, colors of wires, etc. I could go on and on.
Why?
Because when something didn't work, I needed to have "ground truth" as to what was working so I could focus on the (hopefully contained) surface of uncertainty. And this was for a physics experiment where the outcome is KNOWN.
If you don't actually know the outcome, everything gets worse as you don't have a definite signal between "Hypothesis is wrong" vs "Experimental setup is broken". Now, multiply this by thousands of steps for a drug. Brrrrrrrrrr.
To top it off, the physical world isn't like software, you often can't go backwards. In the physical world, if you didn't record everything on the forward pass, you may have to go back to start. :(
The tough part of these regulations is that patient blood is spilled on both sides of the tradeoff.
A: If you allow a drug that ends up causing more harm than benefit
B: If you ban a drug that would have caused more benefit than harm.
So making these regulations is about finding the balance between A and B.
The problem is that a type A error causes a huge scandal. Those who took the drug and died are known by name and there is terrible publicity.
But a type B error only causes people who would have died before to still die. Nothing new has happened, and there is no outcry.
So the pressure on FDA to err on the side of never having any type A deaths, while allowing possibly millions of type B deaths is enormous.
A: If you allow a drug that ends up causing more harm than benefit
B: If you ban a drug that would have caused more benefit than harm.
So making these regulations is about finding the balance between A and B.
The problem is that a type A error causes a huge scandal. Those who took the drug and died are known by name and there is terrible publicity.
But a type B error only causes people who would have died before to still die. Nothing new has happened, and there is no outcry.
So the pressure on FDA to err on the side of never having any type A deaths, while allowing possibly millions of type B deaths is enormous.
I'm noticing that you didn't really address the meat of the parent comment, about the requisite complexity in documenting even a straightforward physics experiment.
Yeah, I didn't see how that, and especially my personal experience of it, had any relevance to the issue of government drug regulation.
So I tried to move on to the substance.
It's of course true that science can be very hard. I'm... not really connecting the dots?
So I tried to move on to the substance.
It's of course true that science can be very hard. I'm... not really connecting the dots?
> I'm... not really connecting the dots?
No, you're not. And that's why you're missing the point that the details ARE the substance.
The voluminous FDA regulations surrounding drug approval are about tracing EVERYTHING about that drug so that when (not if) something goes wrong, you can figure out what went wrong.
Theoretical Example: 5 people just went blind from eyedrops of a drug that has never exhibited that problem before.
Should the drug be pulled from shelves?
If we work backward through the pipeline records and find that the 5 people all share the same batch of eyedrops, yes. If they don't share the same batch, no, but then we probably need to call the CDC as we have a new disease.
But, what's a batch in common? There is the crate that got delivered to the pharmacy--did the drug get too hot and melt some of the conatiner into it? There is the batch that got delivered to the distributor--did a rat or something urinate on the box? There is the batch that got delivered to the sterilizing company (probably gamma radiation for eyedrops)--did they not give it enough dose or did they have a metal obstruction from multiple loads in the facility.
I can keep going further and further back. Was the water in the process supposed to be deionized or sterilized in some way? Did somebody mix up two similar compounds on the line? Did you clean your reactor with the wrong compound? Some drugs are made by cells and your reactor can "drift" over time and you may have to purge it and start over when the drift gets too far.
The regulations demand that you record all of that up front before you can even BEGIN the process of getting a drug approved for use.
(The cleaning the reactor one is not theoretical: https://blogs.sciencemag.org/pipeline/archives/2007/07/09/no... )
In addition, I would like to add that the FDA are actually really helpful once you demonstrate that you are not a clown. However, they get a LOT of clowns on a daily basis and basically have to assume that you are too until you demonstrate otherwise.
No, you're not. And that's why you're missing the point that the details ARE the substance.
The voluminous FDA regulations surrounding drug approval are about tracing EVERYTHING about that drug so that when (not if) something goes wrong, you can figure out what went wrong.
Theoretical Example: 5 people just went blind from eyedrops of a drug that has never exhibited that problem before.
Should the drug be pulled from shelves?
If we work backward through the pipeline records and find that the 5 people all share the same batch of eyedrops, yes. If they don't share the same batch, no, but then we probably need to call the CDC as we have a new disease.
But, what's a batch in common? There is the crate that got delivered to the pharmacy--did the drug get too hot and melt some of the conatiner into it? There is the batch that got delivered to the distributor--did a rat or something urinate on the box? There is the batch that got delivered to the sterilizing company (probably gamma radiation for eyedrops)--did they not give it enough dose or did they have a metal obstruction from multiple loads in the facility.
I can keep going further and further back. Was the water in the process supposed to be deionized or sterilized in some way? Did somebody mix up two similar compounds on the line? Did you clean your reactor with the wrong compound? Some drugs are made by cells and your reactor can "drift" over time and you may have to purge it and start over when the drift gets too far.
The regulations demand that you record all of that up front before you can even BEGIN the process of getting a drug approved for use.
(The cleaning the reactor one is not theoretical: https://blogs.sciencemag.org/pipeline/archives/2007/07/09/no... )
In addition, I would like to add that the FDA are actually really helpful once you demonstrate that you are not a clown. However, they get a LOT of clowns on a daily basis and basically have to assume that you are too until you demonstrate otherwise.
> But, what's a batch in common? There is the crate that got delivered to the pharmacy--did the drug get too hot and melt some of the conatiner into it? There is the batch that got delivered to the distributor--did a rat or something urinate on the box? There is the batch that got delivered to the sterilizing company (probably gamma radiation for eyedrops)--did they not give it enough dose or did they have a metal obstruction from multiple loads in the facility.
> I can keep going further and further back. Was the water in the process supposed to be deionized or sterilized in some way? Did somebody mix up two similar compounds on the line? Did you clean your reactor with the wrong compound? Some drugs are made by cells and your reactor can "drift" over time and you may have to purge it and start over when the drift gets too far.
These days we could track every single step in this product supply chain far more effectively with a crypto-based, blockchain approach. It would be a lot more efficient.
> I can keep going further and further back. Was the water in the process supposed to be deionized or sterilized in some way? Did somebody mix up two similar compounds on the line? Did you clean your reactor with the wrong compound? Some drugs are made by cells and your reactor can "drift" over time and you may have to purge it and start over when the drift gets too far.
These days we could track every single step in this product supply chain far more effectively with a crypto-based, blockchain approach. It would be a lot more efficient.
The arrogance in the software space (I'm making an assumption here based on your block chain comment) is often mind blowing. Somehow people who created some software (or even more common an IT startup) think they got everything figured out.
I've just seen it recently with a diary company where a friend was working. Company was build from the ground up by an elderly couple, they sold to an investment conglomerate when they wanted to retire. These guys brought in a new CEO, CFO and upper management team all from the tech startup world. So these guys come say, we must change this, that make everything more agile, streamline... When the production manager kwho had been there for >15years) and my friend former logistics and deputy GM said this doesn't work like this, we need to take things more slowly, do one change at the time, they both were let go, who likes a naysayer. Well 1 year later the company has lost half its value, the investors fired the management team and brought in some people with dairy experience, it's still 50/50 if the company will survive.
The point of this is, the people doing this stuff aren't stupid, there are often good reasons why things are done in a certain way, so stop and listen before being a smug knowitall.
I've just seen it recently with a diary company where a friend was working. Company was build from the ground up by an elderly couple, they sold to an investment conglomerate when they wanted to retire. These guys brought in a new CEO, CFO and upper management team all from the tech startup world. So these guys come say, we must change this, that make everything more agile, streamline... When the production manager kwho had been there for >15years) and my friend former logistics and deputy GM said this doesn't work like this, we need to take things more slowly, do one change at the time, they both were let go, who likes a naysayer. Well 1 year later the company has lost half its value, the investors fired the management team and brought in some people with dairy experience, it's still 50/50 if the company will survive.
The point of this is, the people doing this stuff aren't stupid, there are often good reasons why things are done in a certain way, so stop and listen before being a smug knowitall.
You're missing the point, too.
Tracking isn't the hard part.
Documenting "what to track and what it should be" UP FRONT is the really hard part.
The goal is to set things up so that when your system scans a QR code for that barrel of "reactor cleaning fluid" and it scans as methanol instead of ethanol the system starts screaming at you.
You can do all the steps by hand with a paper register. Nobody cares what your recording substrate is. However, they do care that it gets recorded in such a way that it can be verified.
And I would say "Someone has a problem and thinks "Oh, I'll use blockchain." Now he has 2 problems."
Tracking isn't the hard part.
Documenting "what to track and what it should be" UP FRONT is the really hard part.
The goal is to set things up so that when your system scans a QR code for that barrel of "reactor cleaning fluid" and it scans as methanol instead of ethanol the system starts screaming at you.
You can do all the steps by hand with a paper register. Nobody cares what your recording substrate is. However, they do care that it gets recorded in such a way that it can be verified.
And I would say "Someone has a problem and thinks "Oh, I'll use blockchain." Now he has 2 problems."
> These days we could track every single step in this product supply chain far more effectively with a crypto-based, blockchain approach.
You can solve precisely none of the problems with a blockchain approach.
A blockchain might be used to implement a tracing system, but the blockchain is "just" a storage and query layer, not the interesting and hard part of the problem that needs to be solved.
You can solve precisely none of the problems with a blockchain approach.
A blockchain might be used to implement a tracing system, but the blockchain is "just" a storage and query layer, not the interesting and hard part of the problem that needs to be solved.
The regulatory burden is so intense because the science is so incredibly hard to get right.
And it’s simultaneously easy to convince yourself you got it right when you didn’t.
And it’s simultaneously easy to convince yourself you got it right when you didn’t.
Why do I have a hard-time believing this?
So many really bad drugs that are popularized and thrown with black-box warnings and such after the fact.
I think the integrity of that system has been subverted.
So many really bad drugs that are popularized and thrown with black-box warnings and such after the fact.
I think the integrity of that system has been subverted.
The key to taking a drug through trial is having enough cash for the trial and for your company to survive the time it takes to complete the trial successfully. Remember that said success is not guaranteed and you might have to write off years of work and start from scratch.
This is why drugs are so expensive in the West.
This is why drugs are so expensive in the West.
Drugs in the west are also expensive because successful drug companies pay their employees a lot of money (a cost that affects profit) but are still extremely profitable (by carefully figuring out what each regional market can afford to pay to maximize their revenues). These companies know that westerners will spend significant fractions of their income for medicine.
(I also agree that having large amounts of cash and runway is necessary for a successful trial).
(I also agree that having large amounts of cash and runway is necessary for a successful trial).
And to add, a clinical trial can be thought of as just one step to establish evidence for efficacy and safety.
Because if you really want to prove that it works and is safe, you have to give it to a lot of people, all of whom have different health conditions. To establish that, you would ideally give it to everyone with the disease of interest. But you can't test a drug on the entire world, so.... you only find so much in a clinical trial.
Because if you really want to prove that it works and is safe, you have to give it to a lot of people, all of whom have different health conditions. To establish that, you would ideally give it to everyone with the disease of interest. But you can't test a drug on the entire world, so.... you only find so much in a clinical trial.
It's known that about 90% of new drug candidates fail somewhere along the clinical trial process. While a successful drug by itself might cost a few 100 millions of dollars, the failures means that the amortized cost is about $1 billion.
This is why compounds change hands so many times before they are successfully commercialized. Running a phase three study costs much more than a phase two. Pharmaceutical companies partner to pool the risk and share the cost.
Subverted... In opposition to the IT startup scene, you mean?
Right now, the cost of R&D in the drug market is such that even companies worth billions recede before the task. The classic argument regarding that is too much vs. too little regulation. Unfortunately, no one can decide the truly necessary amount of consumer protection, so you end up with a balance between no innovation vs. Thalidomide.
Right now, the cost of R&D in the drug market is such that even companies worth billions recede before the task. The classic argument regarding that is too much vs. too little regulation. Unfortunately, no one can decide the truly necessary amount of consumer protection, so you end up with a balance between no innovation vs. Thalidomide.
It is more than there currently is, that is for sure. Statins, fluoroquinolones, h2 receptor antagonists. These drugs all given out to millions of people and all have terrible problems to the point that now quinolones are last resort, and acid blockers Recalled, even though theyve been prescribed for years. Way too much collateral damage.
Interned at the FDA. They take their stuff seriously.
Clinical trials have 3 phases and it all have to be explained via statistic (design of experiment) and reproducible. They have standards for everything to enter the US drug market.
Lots of biostatistics stuff too.
Even when the drug enter the market, pharma companies stop testing the drugs. It is up to the FDA to keep track of side effects and toxicity that were not caught through the phases. (example: ambien sleeping drug)
The people at FDA NCTR are wonderful, highly recommend interning there for a summer.
They have some ML stuff. I've seen a project where they attempt to figure out what bug is in the food. One of the bug was midly toxic and they wanted a image classifier to figure it out via picture of a bug part. You can only have a certain ratio of bug parts in food.
Clinical trials have 3 phases and it all have to be explained via statistic (design of experiment) and reproducible. They have standards for everything to enter the US drug market.
Lots of biostatistics stuff too.
Even when the drug enter the market, pharma companies stop testing the drugs. It is up to the FDA to keep track of side effects and toxicity that were not caught through the phases. (example: ambien sleeping drug)
The people at FDA NCTR are wonderful, highly recommend interning there for a summer.
They have some ML stuff. I've seen a project where they attempt to figure out what bug is in the food. One of the bug was midly toxic and they wanted a image classifier to figure it out via picture of a bug part. You can only have a certain ratio of bug parts in food.
Clinical trials are difficult. But when you consider they are a method to reliably generate new knowledge about complex systems (ie. humans) it doesn’t seem so bad. Actually they are remarkably efficient knowledge generators, compared to other areas of science, and have the advantage that a non-result is still useful up to a certain point.
For someone who never looked into the complexities of clinical trials, I thought that was a detailed and informative answer. It seems the key point to take away is, “The amount of noise in human clinical data can defy belief if you haven’t seen it in person”. :)
I’ve been working in this space for a while now, and this still blows my mind. Patient-reported data is noisy, doctor’s notes infamously confusing and inconsistent in formatting. Even fields you can pull from the health system’s EHR directly will be in disparate formats across different hospitals, or even different departments.
It would be easy to say it all should be standardized (and it should, for the most part), but workflow considerations keep standards from being adopted. What a data standard for oncology data would look like is very different from what one developed for ophthalmology. Not to mention UX - if you’re a doctor seeing 20 patients a day, an extra 15 minutes of data entry per patient could bring that number down to 15 patients or less. UX and data standards really does have a direct link to patient outcomes in healthcare!
It would be easy to say it all should be standardized (and it should, for the most part), but workflow considerations keep standards from being adopted. What a data standard for oncology data would look like is very different from what one developed for ophthalmology. Not to mention UX - if you’re a doctor seeing 20 patients a day, an extra 15 minutes of data entry per patient could bring that number down to 15 patients or less. UX and data standards really does have a direct link to patient outcomes in healthcare!
Indeed, we had to recently work with 500k patients' appointments. Database table Appointment had a date and "actually met" flag - sounds simple enough, right? Nope - doctors were "meeting" dead people, different departments had different rules about whether phone calls were included and what did the "actually met" flag actually mean.
A truth worth repeating!
> Once in a while, they’ll even say “Stop right now – your drug is so damn good that it’s unethical not to give it to everyone in the trial”, but years go by when that does not happen, anywhere in the industry.
We've had such a case a short time ago (months?) in the news. Does anyone remember what that was about or even has a link?
We've had such a case a short time ago (months?) in the news. Does anyone remember what that was about or even has a link?
You may be thinking of Tagrisso: https://www.statnews.com/2020/04/10/astrazeneca-tagrisso-stu...
A kind of side-note, but for similar reasons, most interventional trials in the oncology space aren’t double-blinded for that reason. If you have cancer and sign up for a trial, you’ve likely run out of standard treatment options, so being given a placebo in a potentially life-saving therapy is not easy to justify. The different arms of the trial tend to vary in inclusion/exclusion criteria and doses, but only very rarely is there a placebo involved.
Yup. I had Hodgkins many years ago. We've been treating Hodgkins for many decades with considerable success (~95% of young men with Hodgkins eventually die of something else and that may have improved further). And so it would be unethical to perform a placebo comparison because we're confident from anecdotal evidence (some patients refuse treatment) that it'll kill you.
I was considered for (but ultimately did not qualify IIRC) a trial of a new Hodgkins treatment. If I had qualified and opted in all that would have meant is basically they toss a coin, heads I get the new treatment, tails I get the existing "Gold standard".
As well as not using a placebo because it's a death sentence, they also can't truly blind the trial because the new regime is different drugs, with different side effects and a different calendar. The staff must know which they are giving you and even if they didn't just tell you it's not hard to look up online what the Gold Standard is and tell if you aren't getting it.
So all they have left it's a two arm trial in which they've tried to match the arms up, and then compare the results.
In my case the eventual trial results show the new regime was slightly worse, not better as hoped. Slightly more (statistically insignificant but not good news) deaths and worse side effects. Unfortunate, but there was only one way to find out.
I was considered for (but ultimately did not qualify IIRC) a trial of a new Hodgkins treatment. If I had qualified and opted in all that would have meant is basically they toss a coin, heads I get the new treatment, tails I get the existing "Gold standard".
As well as not using a placebo because it's a death sentence, they also can't truly blind the trial because the new regime is different drugs, with different side effects and a different calendar. The staff must know which they are giving you and even if they didn't just tell you it's not hard to look up online what the Gold Standard is and tell if you aren't getting it.
So all they have left it's a two arm trial in which they've tried to match the arms up, and then compare the results.
In my case the eventual trial results show the new regime was slightly worse, not better as hoped. Slightly more (statistically insignificant but not good news) deaths and worse side effects. Unfortunate, but there was only one way to find out.
Just because it's not reported in the Times, doesn't mean it doesn't happen.
It actually can happen quite frequently. Trial protocols can pre-specify early stopping criteria: if we see drug A is better than drug B by a factor of X at time T, then we can declare superiority with given statistical confidence.
It actually can happen quite frequently. Trial protocols can pre-specify early stopping criteria: if we see drug A is better than drug B by a factor of X at time T, then we can declare superiority with given statistical confidence.
> The amount of noise in human clinical data can defy belief if you haven’t seen it in person
I'm not surprised that patient compliance is a problem, but it still seems like an expensive burden to find patients physically near the clinical trials so they can come in to take the medication. It just seems like a logistical nightmare that software could help with.
Could you build an app that requires patients to record themselves taking their medication under whatever conditions they're supposed to? You might have to manually audit all of the recordings, but the number of people that still fail to comply might be offset but how many more patients you can include in the study. There might be a lot of other things you could verify remotely using wearable tech, e.g. heart rate now that Apple's EKG monitoring is FDA approved.
Great read, I'd like to see more "Chesterton's Fence" arguments for other highly regulated industries.
I'm not surprised that patient compliance is a problem, but it still seems like an expensive burden to find patients physically near the clinical trials so they can come in to take the medication. It just seems like a logistical nightmare that software could help with.
Could you build an app that requires patients to record themselves taking their medication under whatever conditions they're supposed to? You might have to manually audit all of the recordings, but the number of people that still fail to comply might be offset but how many more patients you can include in the study. There might be a lot of other things you could verify remotely using wearable tech, e.g. heart rate now that Apple's EKG monitoring is FDA approved.
Great read, I'd like to see more "Chesterton's Fence" arguments for other highly regulated industries.
"Could you build an app that requires patients to record themselves taking their medication under whatever conditions they're supposed to?"
Yes. I work for a company (Signant Health) which does clinical trial platform as a service (among other things) and our platform has this feature that the trials can have steps which require the users to video certain things.
And we are hiring https://signanthealth.com/careers/ :)
Yes. I work for a company (Signant Health) which does clinical trial platform as a service (among other things) and our platform has this feature that the trials can have steps which require the users to video certain things.
And we are hiring https://signanthealth.com/careers/ :)
There are apps in use now that monitor drug use for psychiatric patients by trying to detect that a drug was ingested using a sensor inside the pill that can be detected with a patch placed on the abdomen. The patch is connected to the phone and reports back to doctors.
This itself had to be approved and has a number of ways it can be prone to errors and blow a trial by misreporting if it were used for one.
Generally, especially under the current circumstances for at-risk patients, there might be a push to do more electronic reporting for clinical trials. However, the requirement to have patients near a trial center isn't that bad, you don't have to be right next to it. But when patients drop out of trials it is very expensive to the company so reducing travel problems is important in addition to maintaining consistency.
This itself had to be approved and has a number of ways it can be prone to errors and blow a trial by misreporting if it were used for one.
Generally, especially under the current circumstances for at-risk patients, there might be a push to do more electronic reporting for clinical trials. However, the requirement to have patients near a trial center isn't that bad, you don't have to be right next to it. But when patients drop out of trials it is very expensive to the company so reducing travel problems is important in addition to maintaining consistency.
> There are apps in use now that monitor drug use for psychiatric patients by trying to detect that a drug was ingested using a sensor inside the pill that can be detected with a patch placed on the abdomen. The patch is connected to the phone and reports back to doctors.
Do you have any more information about this please?
Do you have any more information about this please?
Yes, take a look at the following. It sends various data to a doctor and in addition patients can report their own data for monitoring purposes.
https://www.abilifymycite.com/
https://www.abilifymycite.com/
Excellent, thank you!
>> There are apps in use now that monitor drug use for psychiatric patients by trying to detect that a drug was ingested using a sensor inside the pill that can be detected with a patch placed on the abdomen
Wouldn't that be easily fooled by just pressing the pill against your abdomen for a while, and then throwing it away? Or sticking it with a piece of duct tape?
Seems like this solutions kind of assumes that all pshychiatric patients are stupid.
Wouldn't that be easily fooled by just pressing the pill against your abdomen for a while, and then throwing it away? Or sticking it with a piece of duct tape?
Seems like this solutions kind of assumes that all pshychiatric patients are stupid.
That particular way of gaming it won't work. The sensor inside the pill is a tiny logic circuit that transmits a signal throughout the patient's body. The current comes from basically building a battery out of the patient's stomach with the polar molecules in hydrochloric acid. The current contains some data and the worn patch detects the tiny sine wave.
It would be difficult to game this but it is still possible. The app itself could be broken for example.
I think the FDA approval for this device assumes the patient actually takes the medicine so no guarantees are made if the patient doesn't.
To tie this to my point, adding in electronic reporting and measuring opens up a number of possible ways of getting false data during a trial so I think it would be difficult to manage that. For example, in oncology trials, monitoring for adverse events and cancer progression might involve blood draws for analysis and maybe a monthly progression scan. These types of things are much easier controlled at a central location. Although not all trials are the same and electronic monitoring and reporting seems to have a lot of potential for drug usage and trials.
It would be difficult to game this but it is still possible. The app itself could be broken for example.
I think the FDA approval for this device assumes the patient actually takes the medicine so no guarantees are made if the patient doesn't.
To tie this to my point, adding in electronic reporting and measuring opens up a number of possible ways of getting false data during a trial so I think it would be difficult to manage that. For example, in oncology trials, monitoring for adverse events and cancer progression might involve blood draws for analysis and maybe a monthly progression scan. These types of things are much easier controlled at a central location. Although not all trials are the same and electronic monitoring and reporting seems to have a lot of potential for drug usage and trials.
Many trials do NOT require patients to come in to take the medication, but there are of course some which are harder to take at home (whether intravenous or intrathecal or whatever).
But compliance is certainly hard to assess and so important. Your app thought reminds me of "Video Directly Observed Therapy" which is commonly used for things like tuberculosis medications (due to painful combo of lengthy duration of treatment, disproportionately affecting less well-off / well-educated people who are more challenged in compliance, and public health risk if inadequately treated).
Several companies have also worked on pills with an component that detects when it's been ingested and beams a signal to a (external) device, thus "confirming" ingestion.
Although of course, in a way, compliance is part and parcel of your drug's overall effectiveness. Not to dive into efficacy vs effectiveness and the larger considerations... but what if your drug "works" but nobody can be compliant? Of the top of my head, a few antibiotics are like, dosed 6 times a day, so I'd rather prescribe most patients a different one that's only taken twice a day, even if the 6 times a day antibiotic is technically "better" for their infection.
But compliance is certainly hard to assess and so important. Your app thought reminds me of "Video Directly Observed Therapy" which is commonly used for things like tuberculosis medications (due to painful combo of lengthy duration of treatment, disproportionately affecting less well-off / well-educated people who are more challenged in compliance, and public health risk if inadequately treated).
Several companies have also worked on pills with an component that detects when it's been ingested and beams a signal to a (external) device, thus "confirming" ingestion.
Although of course, in a way, compliance is part and parcel of your drug's overall effectiveness. Not to dive into efficacy vs effectiveness and the larger considerations... but what if your drug "works" but nobody can be compliant? Of the top of my head, a few antibiotics are like, dosed 6 times a day, so I'd rather prescribe most patients a different one that's only taken twice a day, even if the 6 times a day antibiotic is technically "better" for their infection.
Good luck trying to get them to use an app if you can't trust them to take a medication.
If you like this post, check out the other posts on his blog too. It's fantastic content.
Because the supposed drug/therapy is so marginally effective that it need to be muddled up with tons of data (noise) and committees.
Often I think it's just bureaucracy, and incompetence. Even if drug/therapy is massively effective, I'm sure the positive results would be lost in the bureaucracy.
Often I think it's just bureaucracy, and incompetence. Even if drug/therapy is massively effective, I'm sure the positive results would be lost in the bureaucracy.
I also don't think people should harsh on the FDA because of their strictness. There's a lot of gaming attempted in dealing with them which makes them harder to work with, but in my experience they've been pretty decent to deal with. Of course we had a policy of no attempting to game the system.