And reeks of the same sort of reallocation fallacy that makes people think the rich making too much makes them poor.
If we really just say 4xd everyone’s salary in America. Prices are gonna rapidly rise in everything.
Things don’t get materially better unless we build the material things we need. We need to come up with a way to build more houses. Lower the cost of healthcare. Not just increase everyone’s money supply.
These are true and it does make theses fields the first to fall, but also the hype comes form the fact that it can escape these conditions as well:
- generalize to non verifiable domains (https://arxiv.org/abs/2507.17746)[This is on going work but has had steady progress in many angles of attack)
- Visual Reasoning is alive and well in video diffusion and image models(see recent works around using diffusion model priors as world model features for physics reasoning)
- Prior art exists for anything humans can do online?(Is this one even a hold back?)
Signing this sounds like a good way to get fired. Executive in corporations gets to make the decisions. Employment is at will, if you don’t like it you get to leave otherwise you’re not fulfilling your contract
No, look a Composoer 2, it stands out starkly on its own in the pareto frontier on low cast and fast models.
Composer 2.5 was a huge leap with minimal compute from xAI.
They can compete with OpenAI and anthropic with xAI scale compute. They have a top notch model team and incredible training data and huge enterprise costumer contracts.
It’s a bit misleading to say nothing special, as they are doing more than just increasing parameter count. Progress has been steady in all the sub components of training from data filtering and weighting to sparse attention, optimizers to up and down the stack various efficiency in training computing.
They’re using more compute, a bigger model and tons of training quality improvements to get more out of an equivalent model.
This has been my thought for a long time. I think all that matters from attention is that there is crosswise comparison going on.
You need some amount of parallel compute and some amount of global comparison.
And the rest is basically a ways to parameters and scale.
(This is in theory, in practice you can get a lot of small % stability and efficiency improvements that really compound in algorithmic details of model architecture)
Confidently yes. OpenAI for sure has been training larger models internally and distilling.
Pre-training scaling laws all support larger models being more cost effeceint to train then smaller models. And distillation is comparably cheap. So you can get the most juice by training the biggest model you can and distilling it.
There is endless returns to frontier intelligence, just because most people can't make use of it doesn't mean someone can't make a ton of money off of it.
Most software engineers will just need cheap tokens.
But things like physics and drug discovery have no foreseeable upper bound.
There is endless returns to frontier intelligence, just because most people can't make use of it doesn't mean someone can't make a ton of money off of it.
Most software engineers will just need cheap tokens.
But things like physics and drug discovery have no forseeable upper bound.
There is step changes that actually merit this though. And a zero day machine IS one of those. It went from 4% zero day success rate to 85% on firefox.
Multimodal models
Language models for art