In my experience, the model's performance in silly tasks like these is usually (not always) correlated with its performance in other areas except tool use/agent stuff.
You can just use the official Claude Code, OpenAI Codex, and Gemini extensions on VS Code. You get diffs just like in Cursor now. The performance of these models can vary wildly depending on the agent harness they're on.
The official tools won't necessarily give you the best performance, but they're a safer bet for now. This is merely anecdotal as I haven't bothered to check rigorously, but I and others online have found that GPT-5-Codex is worse in Cursor than in the official CLI/extension/web UI.
Did you read the article? All it basically says is that OpenAI faced struggles this past year -- specifically with GPT-5 aka Orion. And now they have o3, and other labs have made huge strides. So, yes, show me AI progress is slowing down!
Not really. Throwing a bunch of unfiltered garbage at the pretraining dataset, throwing in RLHF of questionable quality during post-training, and other current hacks - none of that was expected to last forever. There is so much low-hanging fruit that OpenAI left untouched and I'm sure they're still experimenting with the best pre-training and post-training setups.
One thing researchers are seeing is resistance to post-training alignment in larger models, but that's almost the opposite of a wall, they're figuring it out as well.
> Now someone has to have a new idea
OpenAI already has a few, namely the o* series in which they discovered a way to bake Chain of Thought into the model via RL. Now we have reasoning models that destroy benchmarks that they previously couldn't touch.
Anthropic has a post-training technique, RLAIF, which supplants RLHF,and it works amazingly well. Combined with countless other tricks we don't know about in their training pipeline, they've managed to squeeze so much performance out of Sonnet 3.5 for general tasks.
Gemini is showing a lot of promise with their new Flash 2.0 and Flash 2.0-Thinking models. They're the first models to beat Sonnet at many benchmarks since April. The new Gemini Pro (or Ultra? whatever they call it now) is probably coming out in January.
> The current level of LLM would be far more useful if someone could get a conservative confidence metric out of the internals of the model. This technology desperately needs to output "Don't know" or "Not sure about this, but ..." when appropriate.
You would probably enjoy this talk [0], it's by an independent researcher who IIRC is a former employee of Deepmind or some other lab. They're exploring this exact idea. It's actually not hard to tell when a model is "confused" (just look at the probability distribution of likely tokens), the challenge is in steering the model to either get back to the right track or give up and say "you know what, idk"
>Then there is the matter of actually defining general intelligence. It may also be the definition of consciousness, or at least require it. But currently, there is no mutually agreed upon definition of "general intelligence".
Here lies the problem. We should have a rule that any time we discuss AGI, we preface with the arbitrary definition that we choose to operate on. Otherwise, these discussions will inevitably devolve into people talking past each other, because everyone has a different default definition of AGI, even within the SF AI scene.
If you ask Yann LeCun, he'll say that no LLM system is even close to being generally intelligent, and that the best LLMs are still dumber than a cat.
If you ask Sam Altman, he'll say that AGI = an AI system that can perform any task as well as the average human or better.
If you ask Dario Amodei, he'll say that he doesn't like that term, mostly because by his original definition AGI is already here, since AGI = AI that is meant to do any general task, as opposed to specialized AI (e.g. AlphaGo).
> it is absurd to expect something to keep increasing forever just because it did increase for a short duration previously
The problem isn't that it stopped increasing. It's that it's steadily decreasing now. I made a comment about this recently with sources: https://news.ycombinator.com/item?id=41844739
Not convinced by the blog author's theories, but it's a real problem worth discussing.
I wish trends like the Flynn effect (rise in IQ in most of the world throughout the 20th century, including China [0], Japan and Korea [1]) had more concrete answers by now. AFAIK we still don't have any answers beyond conjectures. To me, having a better understanding of this is vital, because IQ is currently in decline in many developed countries, even when you account for immigration and demographic/race changes. Something in our environment is causing a reverse Flynn effect [2] [3], we don't know what it is, and not enough people seem to care.
> LLMs do not generate new content, they just shuffle old content together in new ways
You can say the same thing about most technical books. They're quite often little more than a more digestible summary of what you get in docs and reference manuals, with some toy examples.
Not for programming it's not. It's confusing, but o1-preview is currently pretty broken for many tasks, or in the words of Sam Altman [0], "deeply flawed". o1-mini is the recommended model [1] for programming exercises and is superior to o1-preview in OpenAI's programming benchmarks [2].
Personally, I'm sticking with Claude Sonnet 3.5 until more people figure out how to use these new models effectively. OpenAI employees said on launch day that traditional prompting techniques might not work with this one so we'll just have to experiment.
Sincerity? I don't know. He seems to really believe in AGI and the singularity and all that, but everything else seems to be lie after lie. This article [0] on the New York Magazine paints him as an incredibly insencere manipulator, which lines up with accusations made by Helen Toner and others. Things he repeatedly says (not caring about money, caring a lot about safety) are often at odds with his actions.