Ten Lessons from Three Generations Shaped Google’s TPUv4i [pdf](gwern.net)
gwern.net
Ten Lessons from Three Generations Shaped Google’s TPUv4i [pdf]
https://www.gwern.net/docs/ai/2021-jouppi.pdf
4 comments
Parts of this document seem like they came from meeting notes copied to latex and not really re-edited...
Explain it like I'm five? :(
The conclusions section of the paper is a good summary:
"In the process, we learned ten lessons about DSAs and DNNs in general and about DNN DSAs specifically that shaped the design of TPUv4i:
1. Logic improves more quickly than wires and SRAM ⇒ TPUv4i has 4 MXUs per core vs 2 for TPUv3 and 1 for TPUv1/v2.
2. Leverage existing compiler optimizations ⇒ TPUv4i evolved from TPUv3 instead of being a brand new ISA.
3. Design for perf/TCO instead of perf/CapEx ⇒ TDP is low, CMEM/HBM are fast, and the die is not big.
4. Backwards ML compatibility enables rapid deployment of trained DNNs ⇒TPUv4i supports bf16 and avoids arithmetic problems by looking like TPUv3 from the XLA compiler’s perspective.
5. Inference DSAs need air cooling for global scale ⇒ Its design and 1.0 GHz clock lowers its TDP to 175W.
6. Some inference apps need floating point arithmetic ⇒ It supports bf16 and int8, so quantization is optional.
7. Production inference normally needs multi-tenancy ⇒ TPUv4i’s HBM capacity can support multiple tenants.
8. DNNs grow ~1.5x annually in memory and compute ⇒ To support DNN growth, TPUv4i has 4 MXUs, fast onand off-chip memory, and ICI to link 4 adjacent TPUs.
9. DNN workloads evolve with DNN breakthroughs ⇒ Its programmability and software stack help pace DNNs.
10. The inference SLO is P99 latency, not batch size ⇒ Backwards ML compatible training tailors DNNs to TPUv4i, yielding batch sizes of 8–128 that raise throughput and meet SLOs. Applications do not restrict batch size."
"In the process, we learned ten lessons about DSAs and DNNs in general and about DNN DSAs specifically that shaped the design of TPUv4i:
1. Logic improves more quickly than wires and SRAM ⇒ TPUv4i has 4 MXUs per core vs 2 for TPUv3 and 1 for TPUv1/v2.
2. Leverage existing compiler optimizations ⇒ TPUv4i evolved from TPUv3 instead of being a brand new ISA.
3. Design for perf/TCO instead of perf/CapEx ⇒ TDP is low, CMEM/HBM are fast, and the die is not big.
4. Backwards ML compatibility enables rapid deployment of trained DNNs ⇒TPUv4i supports bf16 and avoids arithmetic problems by looking like TPUv3 from the XLA compiler’s perspective.
5. Inference DSAs need air cooling for global scale ⇒ Its design and 1.0 GHz clock lowers its TDP to 175W.
6. Some inference apps need floating point arithmetic ⇒ It supports bf16 and int8, so quantization is optional.
7. Production inference normally needs multi-tenancy ⇒ TPUv4i’s HBM capacity can support multiple tenants.
8. DNNs grow ~1.5x annually in memory and compute ⇒ To support DNN growth, TPUv4i has 4 MXUs, fast onand off-chip memory, and ICI to link 4 adjacent TPUs.
9. DNN workloads evolve with DNN breakthroughs ⇒ Its programmability and software stack help pace DNNs.
10. The inference SLO is P99 latency, not batch size ⇒ Backwards ML compatible training tailors DNNs to TPUv4i, yielding batch sizes of 8–128 that raise throughput and meet SLOs. Applications do not restrict batch size."
>>> 8. DNNs grow ~1.5x annually in memory and compute
Wow! That's a massive growth rate for ML. TPUv3 was already faster than A100 in MLPerf. But this suggests a real breakthrough is needed to keep pace with future requirements. Each MXU already handles 16k ops per tick. And with the additional constraint of optimizing per watt rather than dollar, its quite the challange ;)
Wow! That's a massive growth rate for ML. TPUv3 was already faster than A100 in MLPerf. But this suggests a real breakthrough is needed to keep pace with future requirements. Each MXU already handles 16k ops per tick. And with the additional constraint of optimizing per watt rather than dollar, its quite the challange ;)
> But this suggests a real breakthrough is needed to keep pace with future requirements.
Not necessarily. Look at papers like the lottery ticket hypothesis - big ML models may be doing better simply because gradient descent just isn't doing a good enough job. Better optimizers would go a long way than just throwing compute at the the problem. Even if you can, it's impractical to use something like GPT-3 all the time.
Not necessarily. Look at papers like the lottery ticket hypothesis - big ML models may be doing better simply because gradient descent just isn't doing a good enough job. Better optimizers would go a long way than just throwing compute at the the problem. Even if you can, it's impractical to use something like GPT-3 all the time.
Power is a problem and is scaling far less than the node scaling. Raw single threaded performance basically isn't scaling at all.
Going from 45nm to 7nm we only gained about a factor of two in terms of power consumption. Most of that is due to the fact that interconnect and SRAM aren't scaling.
The upshot is that it is worth doing almost infinite amounts of calculation in order to avoid moving anything from point A to point B in a compute unit. If you must use memory, DRAM is death so put big amounts of SRAM on chip and make the buses as wide as possible.
Going from 45nm to 7nm we only gained about a factor of two in terms of power consumption. Most of that is due to the fact that interconnect and SRAM aren't scaling.
The upshot is that it is worth doing almost infinite amounts of calculation in order to avoid moving anything from point A to point B in a compute unit. If you must use memory, DRAM is death so put big amounts of SRAM on chip and make the buses as wide as possible.
[deleted]
How does George Kurian get listed as an author on these TPU papers? I'm seriously confused.
He's a Software Engineer on the TPU team. Are you confusing him for Thomas Kurian, GCloud SVP?
Note: I work for Google, but speak for myself.
Note: I work for Google, but speak for myself.
To add to the confusion, NetApp CEO George Kurian is Thomas' twin brother: https://archive.is/jxtu6
Thank you! That's where I was confused, I worked at NetApp :(
The same way professors get named on papers where their students did all the work?
different guy
Fun fact: this pdf was paywalled, but thanks to some generous members of my Twitter, it was able to be freed. I look forward to a day when all such work is available to anyone who wants it.
Normally I’d have something else to add, but I feel down lately, and it’s nice to see that I’m able to help with one or two things that matter.
I was mostly surprised that the paywall model still persisted in 2021.
Normally I’d have something else to add, but I feel down lately, and it’s nice to see that I’m able to help with one or two things that matter.
I was mostly surprised that the paywall model still persisted in 2021.
To be clear visiting the posted link to "gwern.net" does not ask for pay to view the pdf.
Yeah, sorry for the confusion. I asked gwern to mirror the liberated pdf, since his /docs/ folder will probably be around a lot longer than me. https://www.gwern.net/docs/index There's a lot of neat stuff in there.
By the way, if any of you happen to run into paywalled pdfs, try posting the paywall link to https://reddit.com/r/scholar. It's surprising how often (and how quickly) that gets you the pdf.
By the way, if any of you happen to run into paywalled pdfs, try posting the paywall link to https://reddit.com/r/scholar. It's surprising how often (and how quickly) that gets you the pdf.
[deleted]