Large-Scale Matrix Factorization on TPUs(ai.googleblog.com)
ai.googleblog.com
Large-Scale Matrix Factorization on TPUs
https://ai.googleblog.com/2022/04/large-scale-matrix-factorization-on-tpus.html
4 comments
I feel the same way. But beyond these concerns, I must admit I also feel a sense of sadness due to the inevitable rise of application-specific hardware. When I hear about these super-fast specialised chips, all I can think about is whether I'd have enjoyed sitting in my bedroom as a teenager hacking on them. And the answer is probably no. They can do very specific things, but they don't have the kind of freedom of programming I enjoy.
Maybe I just need to look at them differently, or maybe my perception of their capabilities is just wrong. I do like GPUs, for example. While they are ornery and limited in many ways, they are still flexible enough that you can do enjoyable general-purpose programming on them.
Maybe I just need to look at them differently, or maybe my perception of their capabilities is just wrong. I do like GPUs, for example. While they are ornery and limited in many ways, they are still flexible enough that you can do enjoyable general-purpose programming on them.
A type-writer or a piano is an application specific piece of hardware, yet people can still do an amazing variety of things with them.
In the last few decades we got really lucky that general purpose CPUs kept getting faster and faster. So many of us could get away with ignoring application specific computer hardware.
Btw, as far as I can tell, TPUs are really just a variant of GPUs with an emphasis on lower precision?
Have a look also at the kind of special purpose hardware people make to mine crypto-currencies. I think the challenge for your teenage hacker is not so much to make them do interesting things, but to design them? (Assuming the tools and simulators become cheap enough?) FPGAs seem like they are cheap enough for a teenager?
In the last few decades we got really lucky that general purpose CPUs kept getting faster and faster. So many of us could get away with ignoring application specific computer hardware.
Btw, as far as I can tell, TPUs are really just a variant of GPUs with an emphasis on lower precision?
Have a look also at the kind of special purpose hardware people make to mine crypto-currencies. I think the challenge for your teenage hacker is not so much to make them do interesting things, but to design them? (Assuming the tools and simulators become cheap enough?) FPGAs seem like they are cheap enough for a teenager?
> Btw, as far as I can tell, TPUs are really just a variant of GPUs with an emphasis on lower precision?
GPUs and TPUs have substantially different architectures, it’s not accurate to consider a TPU an a variant of a GPU. See https://dl.acm.org/doi/pdf/10.1145/3360307, particularly section “Contrasting GPU and TPU Architectures”
looking only at the matrix multiply units (and ignoring the hardware multithreading in GPUs that was deliberately not part of the TPU architecture):
TPUv3: 2 cores, 2 128x128 matrix multiply units per core
V100: 80 cores, 8 4x4 matrix multiply units per core
GPUs and TPUs have substantially different architectures, it’s not accurate to consider a TPU an a variant of a GPU. See https://dl.acm.org/doi/pdf/10.1145/3360307, particularly section “Contrasting GPU and TPU Architectures”
looking only at the matrix multiply units (and ignoring the hardware multithreading in GPUs that was deliberately not part of the TPU architecture):
TPUv3: 2 cores, 2 128x128 matrix multiply units per core
V100: 80 cores, 8 4x4 matrix multiply units per core
Thanks for the background information!
It's funny you say that GPUs are general purpose.
It's almost like a universal law that generic stuff is helpful only to some extent, after which you need specialization. That's why we do PhDs I suppose. And that's why you have cardiologists.
It's almost like a universal law that generic stuff is helpful only to some extent, after which you need specialization. That's why we do PhDs I suppose. And that's why you have cardiologists.
You can still enjoy hacking with a consumer FPGA card. For ML one major problem would be memory (for a FPGA that costs the same as a GPU), there's no free lunch. But there are applications where FPGAs of the same price work better.
The sooner specialized hardware comes out the better in my books. The sooner companies deliver it, the sooner patents will inevitably expire and we can get some open source implementations. The golden age of ASICs for everyone is a decade or two away yet. But the sooner that day comes, the better for those of us who want the ownership of it. I'd love to integrate an embedded GPU for an onboard AI into my hobby electronics projects with a low power overhead. We're just not quite there yet.
They do occassionally sell TPU-enabled systems. My backorder from Mouser is still sitting there for several months but some day I too may have a TPU to hack on.
that is a totally different product which is only called TPU because the team who makes it wants to sound cool
Could you elaborate? I guess the person you're replying to is talking about the Coral stuff such as the "USB Accelerator" et al, dubbed "Edge TPUs";
I too am after one of those for my home-security camera system but I've never really looked at what they _are_.
I too am after one of those for my home-security camera system but I've never really looked at what they _are_.
Google TPUs are inspiring everyone. Dieter Fox confirmed at Stanford HAI, NVidia's investing big in cloud accelerators too ;)
http://nvdla.org/primer.html
http://nvdla.org/primer.html
Looking at the paper was pretty disappointing. "Comparison of training time per epoch (in seconds), of plausible alternatives of linear system solver on TPU" lists CG, Cholesky, LU, and QR. It sounds like this method is a viable alternative to LU or QR factorizations from a cursory read?
No, and what started tipping me off is that those methods are not comparable because CG and Cholesky are only for positive semi-definite matrices, so while it's not mentioned anywhere else, the fact that this method works with these factorization techniques means the matrices that they are looking at fall into a very small subset of special matrices. Even then, the subset is even smaller because CG works sufficiently well here without preconditioning, so the method is for not too ill-conditioned positive semi-definite matrices? And for such papers in matrix factorization, there's always some kind of discussion about numerical stability: what pivoting strategies are used, how the pivoting decreases the numerical error growth rate, with empirical studies based on the conditioning of the matrix. Because none of this is done, and in fact the exact opposite is done by emphasizing the BFloat16 precision, gives me no indication that you can have eigenvalues 1e-15 apart and have this work.
Using low precision arithmetic doesn't necessarily mean you cannot get sufficiently good results, there's a whole body of work in mixed precision algorithms. I'd point to Nick Higham's fantastic work on mixed precision matrix logarithms as a very nice example and read: https://www.maths.manchester.ac.uk/~higham/talks/essam18.pdf https://epubs.siam.org/doi/10.1137/17M1129866. But this paper discusses nothing about precision or error, shrugs it off, and doesn't even empirically measure the error in the 365mil case?
"Large-Scale Factorization of Well-Conditioned Positive Semi-Definite Matrices with TPUs" would be a much clearer title then. All of this doesn't mean it's not useful, it just means it's useful for a very specific class of problems and presenting it as a viable alternative to LU or QR factorization is pretty... out there.
No, and what started tipping me off is that those methods are not comparable because CG and Cholesky are only for positive semi-definite matrices, so while it's not mentioned anywhere else, the fact that this method works with these factorization techniques means the matrices that they are looking at fall into a very small subset of special matrices. Even then, the subset is even smaller because CG works sufficiently well here without preconditioning, so the method is for not too ill-conditioned positive semi-definite matrices? And for such papers in matrix factorization, there's always some kind of discussion about numerical stability: what pivoting strategies are used, how the pivoting decreases the numerical error growth rate, with empirical studies based on the conditioning of the matrix. Because none of this is done, and in fact the exact opposite is done by emphasizing the BFloat16 precision, gives me no indication that you can have eigenvalues 1e-15 apart and have this work.
Using low precision arithmetic doesn't necessarily mean you cannot get sufficiently good results, there's a whole body of work in mixed precision algorithms. I'd point to Nick Higham's fantastic work on mixed precision matrix logarithms as a very nice example and read: https://www.maths.manchester.ac.uk/~higham/talks/essam18.pdf https://epubs.siam.org/doi/10.1137/17M1129866. But this paper discusses nothing about precision or error, shrugs it off, and doesn't even empirically measure the error in the 365mil case?
"Large-Scale Factorization of Well-Conditioned Positive Semi-Definite Matrices with TPUs" would be a much clearer title then. All of this doesn't mean it's not useful, it just means it's useful for a very specific class of problems and presenting it as a viable alternative to LU or QR factorization is pretty... out there.
> presenting it as a viable alternative to LU or QR factorization is pretty... out there.
They aren’t presenting it as that. I think you’re just reading it with the wrong lens. “Matrix factorization” in machine learning isn’t the same problem as “matrix factorization” in numerical linear algebra.
In the linear algebra world, you’re trying to essentially keep the matrix, but get it in a different form. In the ML world, you’re trying to learn to predict something.
In general, your loss function isn’t even over all the cells of the matrix (only minimize prediction error over entries where user i actually watched movie j). So the closest numerical linear algebra problem would be factoring a matrix where 99% of the entries are NaN.
They aren’t presenting it as that. I think you’re just reading it with the wrong lens. “Matrix factorization” in machine learning isn’t the same problem as “matrix factorization” in numerical linear algebra.
In the linear algebra world, you’re trying to essentially keep the matrix, but get it in a different form. In the ML world, you’re trying to learn to predict something.
In general, your loss function isn’t even over all the cells of the matrix (only minimize prediction error over entries where user i actually watched movie j). So the closest numerical linear algebra problem would be factoring a matrix where 99% of the entries are NaN.
If solving systems of linear equations is a stronger priority they'd produce hardware with more precision. Unfortunately this is just an offshoot from a machine-learning driven project. But at least the timings could tell you how much value people can gain by implementing their own hardware solution.
google's entire quest in machine learning has simply recapitulated the previous development of high performance computing. that is, ML is just the same kind of problems the physicists were doing on supercomputers, and google is slowly getting to the point of recognizing their ML problems are just supercomputer problems. Expect larger floats in TPU hardware eventually.
TPUs are for DL, not matrix factorization, even if you might want to try that out since you have TPUs sitting around. The trend in DL has for a decade now been to ever lower precision (and many expect binary at some point and a shift away completely to spiking); why are you convinced that it will not just stop but completely reverse and go all the way to like FP64 or FP128 or something?
Because I used to work at Google on TPUs with all the teams (from ads to Physics), and have about 30 years experience in high performance computing.
Neural nets and DL are just one of the reasons to use TPUs; in the future, they might not even be the preeminent approach. DL is just one part of the much larger world of HPC, it's been very effective, but I expect that as things evolve people will rediscover the value of deep precision.
https://arxiv.org/abs/2009.06489
Neural nets and DL are just one of the reasons to use TPUs; in the future, they might not even be the preeminent approach. DL is just one part of the much larger world of HPC, it's been very effective, but I expect that as things evolve people will rediscover the value of deep precision.
https://arxiv.org/abs/2009.06489
If anyone is curious, https://www.gwern.net/docs/ai/2021-jouppi.pdf#page=5 provides a breakdown of TPU workload by type.
I disagree with Hooker's hardware lottery thesis for the simple reason that if it was true, she would be able to point to examples of DL-competitive methods which use the same amount of FLOPS to get much superior results (because everyone is so badly neglecting them due to the 'lottery'), or at least show better scaling curves so that they will at some point surpass DL, but at much worse wallclock due to whatever hardware specializations favor DL and penalize those alternatives. This is how DL operated: they ran on CPUs, very slowly compared to GPUs, but they did run, enabling eventual exploitation of GPUs. And that is how they could start a virtuous circle: the success of DL, because it's the right thing, starting from hardware not even remotely designed for DL (like 2010-era GPUs were not) and designed to favor other tasks (classic GPGU stuff) has pulled hardware in its wake. So, what are the DL-like things running on CPU or contemporary GPUs? The only actual example Hooker gives is capsule networks - which I thought were doomed when they were unveiled, a poor attempt at stuff soft attention did better already, and have not impressed anyone in the 5 years since, particularly as larger NNs (whether CNN, Transformer, or MLP) continue to deliver what capsnets promised. Nor have any more compelling examples arisen in the years since. With no examples or comparisons, it boils down to nothing much.
I disagree with Hooker's hardware lottery thesis for the simple reason that if it was true, she would be able to point to examples of DL-competitive methods which use the same amount of FLOPS to get much superior results (because everyone is so badly neglecting them due to the 'lottery'), or at least show better scaling curves so that they will at some point surpass DL, but at much worse wallclock due to whatever hardware specializations favor DL and penalize those alternatives. This is how DL operated: they ran on CPUs, very slowly compared to GPUs, but they did run, enabling eventual exploitation of GPUs. And that is how they could start a virtuous circle: the success of DL, because it's the right thing, starting from hardware not even remotely designed for DL (like 2010-era GPUs were not) and designed to favor other tasks (classic GPGU stuff) has pulled hardware in its wake. So, what are the DL-like things running on CPU or contemporary GPUs? The only actual example Hooker gives is capsule networks - which I thought were doomed when they were unveiled, a poor attempt at stuff soft attention did better already, and have not impressed anyone in the 5 years since, particularly as larger NNs (whether CNN, Transformer, or MLP) continue to deliver what capsnets promised. Nor have any more compelling examples arisen in the years since. With no examples or comparisons, it boils down to nothing much.
Norm worked just down the hall and he consulted me about physics codes that scientists were running on TPUs. Next generation of TPUs will include higher precision accelerated operations. Note that many scientists tried the "DL will solve physics problems" approach only to find out that DL mostly just did slightly better interpolation than existing systems and happened to run faster mainly because there was a larger pool of experts tuning the engine.
I appreciate your passion but it's clear that outside the world of DL there is a lot of HPC that wants TPUs with higher precision. I don't particularly agree with Sara either, as I have always just moved to whatever resource was most available (IE, finding a better lottery).
I appreciate your passion but it's clear that outside the world of DL there is a lot of HPC that wants TPUs with higher precision. I don't particularly agree with Sara either, as I have always just moved to whatever resource was most available (IE, finding a better lottery).
This special case is a common recommendation situation, for a matrix X of (unit) user-vectors (whose entries are, eg., preferences for products/1)... then X^TX compares all users with all other users.
How quickly do they arbitrarily-nearly approximate matrix factorization of huge matrices, though?
https://keg.cs.tsinghua.edu.cn/jietang/publications/WSDM18-Q...
https://www.microsoft.com/en-us/research/publication/netsmf-...
https://keg.cs.tsinghua.edu.cn/jietang/publications/WSDM18-Q...
https://www.microsoft.com/en-us/research/publication/netsmf-...
I can't resist checking out the details of anything that Google might describe using the phrase "Large-Scale".
As much as I'm interested in the nuts and bolts of their solutions, I'd also be very interested in the problems that become solvable with quick factorization of matrices with billions of rows.
As much as I'm interested in the nuts and bolts of their solutions, I'd also be very interested in the problems that become solvable with quick factorization of matrices with billions of rows.
"Join our vendor-locked implementation" is all I hear. "Trust us by investing into writing software for hardware you can never own".