Heat is mostly driven by leakage current and gate capacitance.
The big issue today is leakage currents. They typical account for around 30%-50% of total chip thermal budget, and they get increasingly difficult to control with smaller devices and lower voltages. They're also get worse with increased temperature(!).
The stacked devices here aren't the worst for leakage currents, but they're not fantastic either. Look at the 2nd graph in section 5: You'll see that the current never drops to zero over the range of gate-source voltages (for V_DS=0.7V). The minimum point is the best-case leakage current, and you can see it's well above zero! (The units on the vertical axis of the graph are unknown btw: The label reads as "current drain-source, arbitrary units")
If you have two threads on different cores that write to the same cacheline, the CPU has to enforce write ordering. The way it does this is for one of the cores to acquire a write lock on the cacheline. The actual implementation of this is via some variant of the MESI protocol that results in the cacheline on one core going to an Exclusive state, while copies of the cacheline on every other core becomes Invalid.
MESI takes a non-trivial amount of time to run. It's usually mediated at the L3 cache, which is frequently clocked slower than the main core, so it's a non-trivial number of cycles for a MESI state transition to happen (think at least ~40 cycles).
Compare that with a cacheline that's already Exclusive on a core: Writing to this cacheline is a pure L1 cache write at ~1 cycle, no MESI involved, no L3 cache involved.
Note that the typical userspace space mutex (some memory location that's modified by compare-exchange) is implicitly relying on MESI whenever two cores race for the lock, or whenever a lock is release on one core and then taken on another, etc etc.
So if your userspace coherency can be handled via some sharding that avoids needing MESI to run, it can go much faster than relying on the CPU's "internal mutex" aka MESI.
To directly address "is it possible that a hardware implementation of an algorithm could be slower than its software variant": The point is that with RSEQ, you can write a _different_ algorithm that's faster than an implementation relying on hardware mediated MESI. Instead of having memory that's accessed for a bunch of different cores, there's a chunk of memory per core, and each core almost always writes only to it's allocated chunk of memory.
Is there really that big a different in TFLOPS between the GB100 and GB202 chips?
The GB100 has fewer SMs than the GB202, so I'm confused about where the 10x performance would be coming from?
There's basically a difference in philosophy. GPU chips have a bunch of cores, each of which is semi-capable, whereas TPU chips have (effectively) one enormous core.
So GPUs have ~120 small systolic arrays, one per SM (aka, a tensorcore), plus passable off-chip bandwidth (aka 16 lines of PCI).
Where has TPUs have one honking big systolic array, plus large amounts of off-chip bandwidth.
This roughly translates to GPUs being better if you're doing a bunch of different small-ish things in parallel, but TPUs are better if you're doing lots of large matrix multiplies.
The hand-waving explanation: The slower you're going, the easier (cheaper) it is to change direction. And for eliptical orbits, the outer-most part of the orbit is where you're going slow.
So to make a drastic change in direction (aka, a very different orbit):
1. First burn to move far away from the center of the orbit so that you're going very slow.
2. Then burn to make a large change in direction (orbit).
3. Then wait until you cross your desired final orbit, and burn again to close it.
The tradeoff is that these types of orbit change are very slow (because you want to be going very slow for the middle burn, which means you take ages to get there).
Not quite: It's taking advantage of (1+a)(1+b) = 1 + a + b + ab.
And where a and b are both small-ish, ab is really small and can just be ignored.
So it turns the (1+a)(1+b) into 1+a+b. Which is definitely not the same! But it turns out, machine guessing apparently doesn't care much about the difference.
I had the same thought: Just eye-balling the graphs, the result of the subtraction looks very close to just reducing the temperature.
They're effectively doing softmax with a fixed temperature, but it's unclear that this work is going to do better than just learning a per-head temperature parameter.
The other way to think about this is that it looks like a hacked-up kinda-sorta gated attention. If that's the case, then doing softmax(alphaq_1k_1^T - log_sigmoid(betaq_2k_2^T)) might be better? (where alpha,beta are learned temperatures).
Like most things, it's more complex than that, and as a result it can be either faster or slower than 'median(RTT to each DC in quorum)'.
It's a delicate balance based on the locations that rows are being read and written. In the case where a row being repeatedly written from only one location and not being read from different location, the writes can be significantly faster than would be naively expected.
> Gradient accumulation doesn't work with batch norms so you really need that memory.
Last I looked, very few SOTA models are trained with batch normalization. Most of the LLMs use layer norms which can be accumulated?
(precisely because of the need to avoid the memory blowup).
Note also that batch normalization can be done in a memory efficient way: It just requires aggregating the batch statistics outside the gradient aggregation.
It's hard to convey just how ridiculously complex and expensive 5nm is compared with 90nm.
5nm is a multi-billion dollar fab, absurdly high running costs, and very expensive wafers.
As a data point: 5nm lithography equipment is > $100M/unit, needs megawatts of power to run, and they handle about 60-80 wafers per hour.
90nm lithography equipment is ~ $1M/unit, needs kilowatts of power to run, and they handle about 60-80 wafers per hour.
So a wafer scanned on 5nm equipment that could have been scanned on 9nm equipment is carrying a very large capital and operational cost burden that's pointless.
The problem is that predicting a pixel requires knowing what the pixels around it looks like. But if we start with lots of noise, then the neighboring pixels are all just noise and have no signal.
You could also think of this as: We start with a terrible signal to noise ratio. So we need to average over very large areas to get any reasonable signal. But as we increase the signal, we can average over a smaller area to get the same signal-to-ratio.
In the beginning, we're averaging over large areas, so all the fine detail is lost. We just get 'might be a dog? maybe??'. What the network is doing is saying "if this a dog, there should be a head somewhere over here. So let me make it more like a head". Which improves the signal to noise ratio a bit.
After a few more steps, the signal is strong enough that we can get sufficient signal from smaller areas, so it starts saying 'head of a dog' in places. So the network will then start doing "Well, if this is a dog's head, there should be some eyes. Maybe two, but probably not three. And they'll be kinda somewhere around here".
Why do it this way?
Doing it this ways means the network doesn't need to learn "Here are all the ways dogs can look". Instead, it can learn a factored representation: A dog has a head and a body. The network only needs to learn a very fuzzy representation at this level. Then a head has some eyes and maybe a nose. Again, it only needs to learn a very fuzzy representation and (very) rough relative locations.
So it only when it get right down into fine detail that it actually needs to learn pixel perfect representation. But this is _way_ easier, because in small areas images have surprisingly very low entropy.
The 'text-to-image' bit is a just a twist on the basic idea. At the start when the network is going "dog? or it might be a horse?", we fiddle with the probabilities a bit so that the network starts out convinced there's a dog in there somewhere. At which point it starts making the most likely places look a little more like a dog.
Minimum is around 1 kWh/m^3 for sea-water levels of salt concentration.
(It varies a fair bit depending on salinity, the actual salts involved, the temperature etc etc).
For the first case, you can't pick a number lower than any existing number. So if 4 is already picked, then only 5 or 6 can be picked, but if we pick 6, then there's no third number available to pick, so there only 1 possibility (picking 5). So it terminates.
The worst case of (1,2,3) is respectively 4, 3 and 2, resulting in log_2 24 = 4.32 bits.
The first choice isn't picking from 6, it's picking from 4. (because if 5 or 6 are chosen, then there's not enough following numbers to be able to choose 2 more numbers). That's the piece about '(n-k+1) choices for the first number'. 5 or 6 are not possible choices given the ordering requirement.
The big issue today is leakage currents. They typical account for around 30%-50% of total chip thermal budget, and they get increasingly difficult to control with smaller devices and lower voltages. They're also get worse with increased temperature(!).
The stacked devices here aren't the worst for leakage currents, but they're not fantastic either. Look at the 2nd graph in section 5: You'll see that the current never drops to zero over the range of gate-source voltages (for V_DS=0.7V). The minimum point is the best-case leakage current, and you can see it's well above zero! (The units on the vertical axis of the graph are unknown btw: The label reads as "current drain-source, arbitrary units")