> The prompt for the Jacobian conjecture was obviously not random. the search space is too big to just try all the combinations of 3 variable polynomials.
Maybe the prompt contained a part like this: "the search space is too big to just try all the combinations of 3 variable polynomials, so be clever about it". Or maybe this part was omitted from the prompt, because modern LLMs are smart enough to figure this out without us having to mention it.
> You just know nothing about math and are happy to parrot bullshit AI salesmen are selling you.
Not the parent poster here. I do know things about math. I wrote a few papers related to the unit distance problem (https://arxiv.org/abs/2311.10069, https://arxiv.org/abs/2406.15317) and spent quite some time trying to solve it. I had no chance of coming up with the proof that the spicy autocomplete came up with. Dumb benchmark, sure.
My use case is wildly different from CSS, HTML etc. I use numerical algorithms to solve problems in pure mathematics. AI models are now better than me and my colleagues at writing code, and we are pretty good in the first place. The catch is that we do not ask AI to write whole applications, we ask it to implement building blocks like "For finite X subset R^d, find all pairs (Y, Z), Y subset X, Z subset X such that Y and Z are congruent".
I think the nuanced take on Joel's rant is this: it was good advice for 26 years. It became slightly less good advice a few months ago. This is a good time to warn overenthuastic people that it’s still good advice in 2026, and to start a discussion about which of its assumptions remain to be true in 2027 and later.
Basically a boid simulation where a swarm of birds can collectively solve MNIST. The goal is not some new SOTA architecture, it is to find the right trade-off where the system already exhibits complex emergent behavior while the swarming rules are still simple.
It is currently abandoned due to a serious lack of free time (*), but I would consider collaborating with anyone willing to put in some effort.
100% frontpage-worthy! Frankly I was already bored with all those pelicans, and a bit worried that the labs are overfitting on pelicans specifically. This test clearly demonstrates that they are not.
That's very cool, but it's not an apples to apples comparison. The reasoning model learned how to do long multiplication. (Either from the internet, or from generated examples of long multiplication that were used to sharpen its reasoning skills. In principle, it might have invented it on its own during RL, but no, I don't think so.)
In this paper, the task is to learn how to multiply, strictly from AxB=C examples, with 4-digit numbers. Their vanilla transformer can't learn it, but the one with (their variant of) chain-of-thought can. These are transformers that have never encountered written text, and are too small to understand any of it anyway.
If being probabilistic prevented learning deterministic functions, transformers couldn’t learn addition either. But they can, so that can't be the reason.
My uninformed and perhaps overly charitable interpretation: he warned them they were going to be steamrolled, they built their product anyway, and now OpenAI is buying them because (1) OpenAI doesn't want the negative publicity of steamrolling them all, and (2) OpenAI has the money and is a bit too lazy to build a clone.
You dismiss parent's example test because it's in the training data. I assume you also dismiss the Sally-Ann test, for the same reason. Could you please suggest a brand new test not in the training data?
FWIW, I tried to confuse 4o using the now-standard trick of changing the test to make it pattern-match and overthink it. It wasn't confused at all:
When you build a new model, there is a spectrum of how you use the old model: 1. taking the weights, 2. training on the logits, 3. training on model output, 4. training from scratch. We don't know how much advantage #3 gives. It might be the case that with enough output from the old model, it is almost as useful as taking the weights.
Normally it would answer with a number and an explanation. This one just asks it to skip the explanation so that string comparison can be used to evaluate it.
https://aaronlou.com/jacobian_counterexample_prompt.pdf
Obviously it is not random, but it's very generic. No mention of search space or how to reduce it.