> The AlphaFold3 analysis (the AI contribution) literally accounts for a few panels in a supplementary figure - it didn't even help guide their choice of small molecule inhibitors since those were already known.
(Disclaimer: I'm the author of a competing approach)
Gnina is Vina with its results re-scored by a NN, so the exact same concerns apply.
I'm very optimistic about AI, for the record. It's just that in this particular case, the comparison was flawed. It's the old regurgitation vs generalization confusion: We need a method that generalizes to completely novel drug candidates, but the evaluation was done on a dataset that tends to be repetitive.
> According to Stack Overflow developer survey [0] Rust is at 12.5%, ... So definitely not niche.
The annual survey is very popular in the Rust community. Its results are often used for advocacy. Participation by Rust developers is very high. So what you have is a classic case of a selection bias.
In their approach, the LLM generates inputs (images to be transformed) and solutions (Python programs that do the image transformations). The output images are created by applying the programs to the inputs.
So there's a constraint on the synthetic data here that keeps it honest -- the Python interpreter.
In the o3 announcement video, the president of ARC Prize said they'd be partnering with OpenAI to develop the next benchmark.
> mechanical turking a training set, fine tuning their model
You don't need mechanical turking here. You can use an LLM to generate a lot more data that's similar to the official training data, and then you can train on that. It sounds like "pulling yourself up by your bootstraps", but isn't. An approach to do this has been published, and it seems to be scaling very well with the amount of such generated training data (They won the 1st paper award)
> That's fine if you want to measure sample efficiency, but ARC-AGI is supposed to measure progress towards AGI.
On the Measure of Intelligence defines intelligence as skill-acquisition efficiency, I believe, where efficiency is with respect to whatever is the limiting factor. For each ARC task, the primary limiting factor is the number of samples in it. And the skill here is your ability to convert inputs into the correct outputs. In other words, in this context, intelligence is sample-efficiency, as I see it.
In https://arcprize.org/play?task=79369cc6 , the yellow 3x3 square shows you the pink pattern to look for, while allowing rotations (and ignoring the fact that the pattern may be next to other patterns)
Compute is limited during inference, and this naturally limits brute-force program search.
But this doesn't prevent one from creating a huge ARC-like dataset ahead of time, like BARC did (but bigger), and training a correspondingly huge NN on it.
Placing a limit on the submission size could foil this kind of brute-force approach though. I wonder if you are considering this for 2025?
Q: The ARC Prize blog mentions that you plan to make ARC harder for machines and easier for humans. I'm curious if it will be adapted to resist scaling the training dataset (Like what BARC did -- see my other comment here)? As it stands today, I feel like the easiest approach to solving it would be BARC x10 or so, rather than algorithmic inventions.
In fairness, their approach is non-trivial. Simply asking GPT-4o to fantasize more examples wouldn't have worked very well. Instead, they have it fantasize inputs and programs, and then run the programs on the inputs to compute the outputs.
I think it's a great contribution (although I'm surprised they didn't try making an even bigger dataset -- perhaps they ran out of time or funding)
(Disclaimer: I'm the author of a competing approach)
Searching for new small-molecule inhibitors requires going through millions of novel compounds. But AlphaFold3 was evaluated on a dataset that tends to be repetitive: https://olegtrott.substack.com/p/are-alphafolds-new-results-...