Looks like Fusion calls a bunch of models and then uses an LLM to synthesize the results, and pass to another model for final output.
Fugu looks like it's doing something different? Using an LLM earlier on in the flow as an orchestrator to decide which other LLMs to call. More coordinator than simply synthesizing results, and more "agentic".
It's interesting because it's all exposed behind a single OpenAI compatible endpoint (Responses API?) and so then presumably someone could use this for one of their single agents. Now you have agent-of-agents, nested in some sense. The token usage increases accordingly!
The jury found that Ding stole trade secrets relating to the hardware infrastructure and software platforms that allow Google’s supercomputing data center to train and serve large AI models. The trade secrets contained detailed information about the architecture and functionality of Google’s custom Tensor Processing Unit chips and systems and Google’s Graphics Processing Unit systems, the software that allows the chips to communicate and execute tasks, and the software that orchestrates thousands of chips into a supercomputer capable of training and executing cutting-edge AI workloads. The trade secrets also pertained to Google’s custom-designed SmartNIC, a type of network interface card used to facilitate high speed communication within Google’s AI supercomputers and cloud networking products.
Neat experiment that gives a mechanistic interpretation of temperature. I liked the reference to the "anomalous" tokens being near the centroid, and thus having very little "meaning" to the LLM.
Great article and nice explanation. I believe this describes “Algorithm R” in this paper from Vitter, who was probably the first to describe it: https://www.cs.umd.edu/~samir/498/vitter.pdf
When subtlety proves too constraining, competitors may escalate to overt cyberattacks, targeting datacenter chip-cooling systems or nearby power plants in a way that directly—if visibly—disrupts development. Should these measures falter, some leaders may contemplate kinetic attacks on datacenters, arguing that allowing one actor to risk dominating or destroying the world are graver dangers, though kinetic attacks are likely unnecessary. Finally, under dire circumstances, states may resort to broader hostilities by climbing up existing escalation ladders or threatening non-AI assets. We refer to attacks against rival AI projects as "maiming attacks."
A lot of my teenage years were spent building and playing with PCs and a lot of the knowledge and interest came from reading each and every issue of boot and maximum pc
Jumping into an unknown codebase (which may be a library you depend on) and being able to quickly investigate, debug, and root cause an issue is an extremely invaluable skill in my experience
Acting as if this isn’t useful in the real world won’t help. The real world is messy, documentation is often missing or unreliable, and the person/team who wrote the original code might not be around anymore.
Yeah, ops comment makes it seem like they are building racks of RTX 4090s, when this isn’t remotely true. Tensor Core performance is far different on the data center class devices vs consumer ones.
This is probably using their excess capacity, but not necessarily that their GPUs are idle.
For LLMs/large models the huge cost is memory ops to load each layer weights during the forward pass. This is why doing inference at batch size 1 is extremely wasteful: you pay all the mem ops cost and don’t use enough compute FLOPs to justify.
You want a high enough batch size so that compute:mem ops is close to the ratio for that GPU. This is usually done by batching together multiple user requests.
At times of low usage there is excess capacity because batch size is below this “optimal” ratio. So they can slot in these “relaxed SLA” requests for little marginal increase resource usage on their end.
Basically have a queue of these requests that you use to “top off” your batch size when you can.
Edit: also you may not be able to get optimal batch size depending on when the requests arrive, eg you don’t want to wait forever to fill up a batch. So again having a queue of outstanding/delayed requests to serve allows for smoothing things out and increasing compute utilization
A simplified explanation, which I think I heard from Karpathy, is that transformer models only do computation when they generate (decode) a token. So generating more tokens (using CoT) gives the model more time to “think”.
Looks like Fusion calls a bunch of models and then uses an LLM to synthesize the results, and pass to another model for final output.
Fugu looks like it's doing something different? Using an LLM earlier on in the flow as an orchestrator to decide which other LLMs to call. More coordinator than simply synthesizing results, and more "agentic".
It's interesting because it's all exposed behind a single OpenAI compatible endpoint (Responses API?) and so then presumably someone could use this for one of their single agents. Now you have agent-of-agents, nested in some sense. The token usage increases accordingly!