A lot of this post relies on the recent open ai result they call GDPval (link below). They note some limitations (lack of iteration in the tasks and others) which are key complaints and possibly fundamental limitations of current models.
But more interesting is the 50% win rate stat that represents expert human performance in the paper.
That seems absurdly low, most employees don’t have a 50% success rate on self contained tasks that take ~1 day of work. That means at least one of a few things could be true:
1. The tasks aren’t defined in a way that makes real world sense
2. The tasks require iteration, which wasn’t tested, for real world success (as many tasks do)
I think while interesting and a very worthy research avenue, this paper is only the first in a still early area of understanding how AI will affect with the real world, and it’s hard to project well from this one paper.
I think what’s missing is what the software allows. It could be BMW/Merc etc are way more conservative on what the allow the system to do and when they force the driver to take over. In certain contexts Merc is actually willing to assert and stand by a higher level of autonomy than any other manufacturer: (https://www.motortrend.com/news/mercedes-benz-drive-pilot-le...). Taking that at face value it’s possible they can do it and choose not to because they don’t want the liability. Whatever systems are in regular cars are then either borked or deliberately have less hardware.
Tesla is uniquely risk tolerant for better or worse. You also don’t hear about people getting into accidents in a BMW on self driving because they don’t make the same claims and have tons of safeguards.
> It’s only marginally less useful to actual biology than full on X-ray structures anyway.
I'm not sure what you're implying here. Are you saying both types of structures are useful, but not as useful as the hype suggests, or that an X-Ray Crystal (XRC) and low confidence structures are both very useful with the XRC being marginally more so?
An XRC structure is great, but it's a very (very) long way from getting me to a drug. Observe the long history of fully crystalized proteins still lacking a good drug. Or this piece on the general failure of purely structure guided efforts in drug discovery for COVID (https://www.science.org/content/blog-post/virtual-screening-...). I think this tech will certainly be helpful, but for most problems I don't see it being better than a slightly-more-than-marginal gain in our ability to find medicines.
Edit: To clarify, if the current state of the field is "given a well understood structure, I often still can't find a good medicine without doing a ton of screening experiments" then it's hard to see how much this helps us. I can also see several ways in which a less than accurate structure could be very misleading.
FWIW I can see a few ways in which it could be very useful for hypothesis generation too, but we're still talking pretty early stage basic science work with lots of caveats.
Such a database would be hugely helpful across chemistry. Right now it’s extremely expensive to access databases like Reaxys or Scifinder, and they’re not usually programmatically searchable at scale. Some databases do exist based on the patent literature (https://depth-first.com/articles/2019/01/28/the-nextmove-pat...) but they’re not as well curated or complete. A pubchem like database for reactions would be really awesome.
As a fellow perfectionist who has started a company, one thing that has helped me is realizing that most decisions are a lot more reversible than they appear. Even in the legal and financial domain, most things that you might obsess over are fixable if you make a mistake, and decent lawyers will tell you which ones you really have to avoid. Sometimes it'll cost you money and time, but the biggest cost is avoiding making decisions.
Always remember, no decision is a decision. Usually that's the worst choice because almost any decision, even a wrong one, at least moves the ball in some direction, allowing you to gather more information. The only guarantee in this game is that stasis will kill you, so bias towards action. When in doubt, try to evaluate "most probable bad outcome" which is different from "worst possible outcome".
There's a lot that can be learned with building-block based experiments. If you do a building block based experiment then train a model, then predict new compounds, the models do generalize meaningfully outside the original set of building blocks into other sets of building blocks (including variations on different ways of linking the building blocks). Granted that's not the "fully novel scaffold" test, however it suggests that there should be some positive predictive value on novel scaffolds.
We've done work in this area and will be publishing some results later in the year.
This is true. Getting datasets with the necessary quality and scale for molecular ML is hard and uncommon. Experimental design is also a huge value add, especially given the enormous search space (estimates suggest there are more possible drug-like structures than there are stars in the universe). The challenge is figuring out how to do computational work in a tight marriage with the lab work to support and rapidly explore the hypotheses generated by the computational predictions. Getting compute and lab to mesh productively is hard. Teams and projects have to be designed to do so from the start to derive maximum benefit.
Also shameless plug: I started a company to do just that, anchored to generating custom million-to-billion point datasets and using ML to interpret and design new experiments at scale.
Not a chiphead, but saw this in the article that might be a reason ARM is better for this kind of thing:
"The theory goes that arm64’s fixed instruction length and relatively simple instructions make implementing extremely wide decoding and execution far more practical for Apple, compared with what Intel and AMD have to do in order to decode x86-64’s variable length, often complex compound instructions."
Not sure it's true, not an expert. But it doesn't sound wrong!
Most of the methane comes from cattle belching, so it's basically impossible to harvest at scale. The lagoons of manure also do, and that can be harvested though it's hard.
The catalyst is the part I'm most curious about. Carbon capture is a hard problem at scale. Once you have methanol, I'm not terribly surprised that you can convert it to other things enzymatically. Hard to evaluate without full text of paper though.
I'd agree if it works as advertised. Level 5 or very close to it is the key. No system has shown that, much less Tesla's. In the meantime, doing a live experiment with 3,000+ pound machines moving at 30+ mph seems like a bad idea.
I drive a lot, thanks. If you can prove me level-5 or even very good level 4 autonomous driving, and that a computational driver makes radically fewer fatal mistakes than a human, then I'm with you. In other words, if you can satisfy a good regulatory regime like the say, airplanes or drugs, then great.
I'm not sure what you're arguing in terms of acceptable risk. Biotech is incredibly regulated, specifically because the risks are so high, effectively there is very little acceptable risk. In biotech, a patient dying due to your drug is a Big Problem that will at best cause you to put a disclaimer on the package (see Black Box Warning) and at worst immediately end your drug's prospects. We can argue about trade-offs (if you've got terminal cancer, maybe a rare heart event is a worthwhile risk, probably less so if you have a rash), but this is exactly the way it should be.
Self driving cars are a nice luxury, especially in city driving, not something that radically improves our world. You get to read your phone instead of paying attention, and the trade-off is someone might get killed. It's like treating a rash with a drug that could give you a heart attack. That's a far cry from, "with this technology something that took days and $$$$ now takes hours and $" as was the case with all the older examples you listed.
If self driving cars were more like airplanes, I'd have a little more faith. Tesla's marketing BS doesn't inspire me with lots of faith.
It’s pretty clearly the current best way to do deep learning on molecules in chemistry. Transformers do really well in predicting reaction outcomes but GCNNs are still better when it comes to “is this a good molecule according to this label” type questions. That said, we have added attention layers to GCNNs so I think the lines get blurry.
We already do all of this in drug discovery research before putting things into animals. We try computer modeling (very inaccurate but a good pre filter) then cell culture, and only after that do we use animals. Not only is it ethically the right thing, it’s the economical thing. Animal studies are crazy expensive (each mouse costs a couple dollars per day just to keep alive and you need a lot of them) compared to cell culture and modeling.
The problem is that in your scenario, many many more humans would be harmed. While animal models are flawed, there are no cell culture tools that even come close to recapitulating the system wide effects you get in a whole organism. Without animal tests, all the drugs that hurt a mouse (and there are very many that fail in mouse tests for this reason) would instead do it to humans.
You can ban animal research, but if you do, know that virtually none of the medicines you take would exist. It really is that stark.
Even if the amyloid hypothesis is true (and most neuroscientists I know think it's not), this is a terrible decision.
1.) We don't know when to give the drug. Maybe giving it even earlier would help, but these trials don't tell us that. Answer: Run a new trial.
2.) We don't know how much drug to give. The drug was approved on the (bad, weak) evidence that in one high dose arm of one of 2 trials, there might have been an effect. Is that the right dose? Who knows! In the other trial, the high dose may have actually been worse. You can't titrate dosage in Alzheimers like you do in cancer where you can just watch how the tumor is shrinking. Answer: Run a new trial.
Giving this drug is not without downsides. You will have side-effects, including serious ones such as potentially brain swelling. Some people may be seriously injured or killed as a result of taking this drug. You have to make sure that the benefits outweigh that downside, and the trials show us a very dubious, weak effect.
The FDA should have said, "Good work, maybe there's an effect with this dosage, go run a new trial with the revised protocol." That's the right call.
Instead, they added, "Oh and you can sell the drug in the meantime and you don't have to tell us for 9 years."
How are you going to recruit patients for the trial? "We could give you the drug, but might give you a placebo." How many patients sign up for that instead of saying, "OR I could go out and buy the drug (which you claim totally works, and the FDA agrees!) independently."?
What if the trial fails in 9 years after you have tons of anecdotal reports (remember placebo shows an effect in past trials)? Now you have to take if off the market, imagine the loss of credibility that will entail for the FDA.
How about other drugs that reduced amyloid but showed no cognitive effect? (Eli Lilly's Solazenumab among many, many others) Should they get approved now too?
This is a mess for everyone and benefits Biogen. Everyone else loses, even the well-meaning patient advocates who created the political pressure for this decision.
That linked post is fundamentally wrong because it rests on an assumption that has essentially zero human evidence: amyloid causes Alzheimer’s. There have been several drugs that very efficiently reduce amyloid, strictly zero (including this one) have ever shown any benefit patient health even when running long trials (the Biogen trials started in 2015 and were halted for futility). There’s reason to believe the amyloid hypothesis is flawed, meaning that approving a drug that reduce amyloid is not going to help anyone, and will likely hurt people (through side effects).
If competing experts are the question, note that 3 actual experts have resigned from what are coveted positions in protest. Nearly every part of the pharma industry (including the press, investors, other companies) who doesn’t stand to profit (I.e. not Biogen) has been up in arms saying this is an awful decision using words such as “horrifying”. There is no expert disagreement.
People can try to ret-con this by saying it’s like HIV, but note that viral load is a pretty good marker for disease morbidity in most viral infections. Amyloid is nothing like that as a validated marker for disease burden.
Just got a postcard from my optometrist the other day saying it’s time for my annual prescription check. My eyes are no different, but my glasses are scratched, so I need to spend an hour or two to get this fixed instead of just reordering glasses online. Can’t wait to use a service like this instead, much faster and easier, good luck!!
My biggest worry with this would be the low level of off target edits and the number of recombination events that yielded an unwelcome product. Looks like those were very low, but with an N of 4, hard to know long term. The reason being that when you screw around with DNA you can get cancer. This has been an issue in a variety of cases with gene therapy, though is clearly getting much better. This is really cool though, exciting times!
But more interesting is the 50% win rate stat that represents expert human performance in the paper.
That seems absurdly low, most employees don’t have a 50% success rate on self contained tasks that take ~1 day of work. That means at least one of a few things could be true:
1. The tasks aren’t defined in a way that makes real world sense
2. The tasks require iteration, which wasn’t tested, for real world success (as many tasks do)
I think while interesting and a very worthy research avenue, this paper is only the first in a still early area of understanding how AI will affect with the real world, and it’s hard to project well from this one paper.
https://cdn.openai.com/pdf/d5eb7428-c4e9-4a33-bd86-86dd4bcf1...