The only thing the $1T model needs to do is find some algorithmic speedup which allows it to be trained at $100B. I'm not saying that's easy or that it will happen but I just don't see why not.
> Unlimited intelligence doesn't mean unlimited resources or instantaneous implementation.
Of course you're right. At the end of the day you need to deal with the bedrock which is the laws of physics. I could be wrong but I struggle to believe we are close to the edge of what is possible in getting the most out of our limited resources or time.
Without atomic physics, uranium would just be another shiny rock in the ground. Sand is just what covers beaches. With enough time and intelligence we've made the shiny rock power cities and persuaded the sand to solve long-standing mathematical conjectures.
> LLMs show no sign of improving intelligence and instead providers are going down the ‘agentic’ rabbit hole.
I'm not sure where you're getting this. I don't work at Anthropic but Fable (Mythos) seems demonstrably smarter than Opus for pretty much any definition of smarter and they claim that Opus was used heavily in Mythos development (yeah I know take this with a massive pinch of salt).
Either way if the models are indeed helping development, even on the engineering, you can iterate on models faster and even if they're not contributing to core research yet you still have a baby exponential by improving the engineering.
> I agree with your sentiment (about the noise), however I think this over simplifies it a bit. We may get AI that is super-human at frontier research and dramatically accelerates the pace, and still have to wait decades before it disrupts the job market (or maybe never displaces all work).
I don't see why that's the case when you have super-human researchers on tap. There are indeed physical (supply chain-y) issues to deal with but isn't the whole point that:
1. Super-human at AI research + scaling to millions of instances will probably result in super-intelligence in everything which is not AI research. (a subset of which is white-collar work)
2. Use that super-intelligence to solve any supply-chain issues you might be facing.
> And we may just find that the human mind is way more capable than we thought and even with accelerating research it's just a harder problem than anyone expected, even algorithmically.
I hope so but whenever I do, I feel like I'm coping hard and not dealing with the facts.
I'm not saying we're there yet - I'm saying the trend lines are clear.
> More likely (without careful vetting by the folks aggregating these models) is that the quality will go down as more and more AI-generated output gets subsumed into these models.
This assumes that there aren't algorithmic breakthroughs which reduce training/inference costs by several OOMs.
How much do these models need to do before people throw their hands in the air and say, ok this is happening. The Erdos unit distance problem, which as far as I understand was approached by multiple competent mathematicians was solved by a frontier model. Sure people argue there was no novelty there (I cannot comment as a non-mathematician) but it feels like they can draw lines laterally from deep knowledge in different fields (in this case combinatorics and algebraic number theory I believe) and solve problems.
Now if you have millions of instances running in parallel, all "probabilistic", working on frontier AI research I really don't see the blocker (and believe me I wish I did).
This is all noise. The leaders of these companies are flip-flopping to whatever sounds best for their current agenda - hiring, fundraising, pre-IPO, etc.
The only thing that matters is if LLMs with sufficient scaling can become frontier AI researchers kicking off the exponential. Everything else is transient noise.
Getting a company to IPO is not about getting lucky once - it's making hundreds of good decisions daily. From hiring to product direction to fundraising.
Like or hate the guy you gotta give credit where it's due. (Same for Musk, etc.)
For 99% of people I don't see the usecase (except for privacy but that ship sailed a decade ago for the aforementioned 99%). If the argument is inference offline - the modern computing experience is basically all done through the browser anyway so I don't buy it.
GPUs for video games where you need low latency makes sense. Nvidia GeForce Now works but not for any serious gaming. But when it comes to LLMs at least, the 100ms latency between you and the Gemini API or whichever provider you use is negligible compared to the inference time.
I agree in general it isn't. But in this case Musk claimed that was the point of open-sourcing the algorithm. Transparency on what they are or are not suppressing.
Close to the end of the paper they casually drop that they solved all the millennium problems including the Riemann Hypothesis and have Lean proofs for all of them.
I don't know enough physics or mathematics to be able to tell whether any of this holds or if it's o3 generated engagement bait.
On further inspection the author seems to have a bored ape as their X profile pic so I'm going with engagement bait. Pretty high quality bait though.
I haven't played around with hypervisors much but the whole point of k8s is not just isolation but all the primitives the control plane gives you which you don't need to implement. Things like StatefulSet, ReplicaSet, Volumes, HorizontalPodAutoscaler, Service, DNS, ConfigMaps, Secrets, Accounts, Roles, Permissions etc.
Also the container runtime which is containerd by default I believe can be switched out for micro vms like Firecracker (never done this though - not sure how painful it is).
This is really interesting and a thorough write up. Thanks to the author for sharing their work.
Whenever I read about super low-level optimisation, my immediate feeling is that of gratitude to the author for spending so much time shaving off nanoseconds which the entire SE community gets to enjoy.
I wonder how much time humanity has collectively saved simply as a result of how all these optimisations stack up.
Honestly sets as trees isn't original. While I was learning about ZFC I came across some lectures[0] by Richard Borcherds which was the seed of insipiration for this project.