Fourier analysis may help to quickly train more accurate neural networks(spectrum.ieee.org)
spectrum.ieee.org
Fourier analysis may help to quickly train more accurate neural networks
https://spectrum.ieee.org/black-box-ai
7 comments
Ahhh. Good old Fourier analysis on the change in model parameters: "low-pass filters that screen out background noise, high-pass filters that help analyze background signals, and Gabor filters that are often used in image processing. The Fourier analysis of the kernels revealed the neural network’s parameters were behaving like a combination of low-pass, high-pass, and Gabor filters."
I can't imagine the architects of the model didn't design the layers and kernels without some idea in mind of what sorts of operations they would be wanting to simulate right? Do people just throw random layers and activation functions together until something works? I'm only on the periphery of dl but those I know that work in it either have a general idea of what equuations they would want to use to do it without dl, or have a biological inspiration in mind. This doesn't seem like a biologically inspired sort of domain.
> Do people just throw random layers and activation functions together until something works?
In a lot of cases, yes. You can start with a reasonable baseline architectural guess, like “convolution should be good for vision” or “attention should be good for language”, but after that it's a lot of guess and check.
In a lot of cases, yes. You can start with a reasonable baseline architectural guess, like “convolution should be good for vision” or “attention should be good for language”, but after that it's a lot of guess and check.
This may be the case in certain shops or disciplines, but I can assure you, if you have the right insights, you can use statistical and ML principles to achieve better accuracy with orders of magnitude less compute.
I consider it a tragedy that we throw huge hogs of models into datacenters and let them churn without giving much thought to improving performance. The climate impacts relative to the real benefits are measurable and depressing for a field so focused on innovation. Usually the 100x model built by a team of "data scientists" over a month is no better than a 1x model built by a couple SMEs over a couple weeks.
I consider it a tragedy that we throw huge hogs of models into datacenters and let them churn without giving much thought to improving performance. The climate impacts relative to the real benefits are measurable and depressing for a field so focused on innovation. Usually the 100x model built by a team of "data scientists" over a month is no better than a 1x model built by a couple SMEs over a couple weeks.
Just because there are a lot of under-qualified bootcamp data scientists out there who think Pytorch can solve all their problems, that doesn't give you the right to sneer at the cutting edge of ML research on the grounds that you've built some good models in Excel in the past.
There are plenty of times when deep learning is the wrong solution, and probably most businesses out there should not ever expect to need any kind of highly compute-intensive model training. But there are also plenty cases where neural networks are the right solution as well (even if not "deep learning" as such).
Maybe I'll be convinced if you share some specific examples. Otherwise I'll assume that your comment is just arrogant derision.
There are plenty of times when deep learning is the wrong solution, and probably most businesses out there should not ever expect to need any kind of highly compute-intensive model training. But there are also plenty cases where neural networks are the right solution as well (even if not "deep learning" as such).
Maybe I'll be convinced if you share some specific examples. Otherwise I'll assume that your comment is just arrogant derision.
I have a lot to say on this and related topics, but I'll try to keep it on topic to this thread.
> sneer at the cutting edge of ML research
I was reacting specifically to mis-specifications of what "good" means in these contexts, e.g., "attention should be good for language" from GGP comment.
The major advances of these models du jour are not the outputs, but rather the infrastructure that it takes to run them. I salute all those involved in implementation. I hope (and see!) these innovations spill into things that help people and generate real value, for instance, all the things businesses use accurate climate forecasts for.
My issue mostly revolves around the discrepancy between value-created-per-FLOPS and cost-per-FLOPS. If we run huge models for what serves for us the same purpose as the TV walls in Fahrenheit 451 do for their owners, what are we adding? If we are constantly running pixel-level segmentation networks on the cameras of automobiles that are already being driven by humans, how is that helping?
Stable Diffusion is a real innovation in optimization techniques. This is the only part of recent news that can be considered "cutting edge ML research", as it is an important step in unifying optimization theory with that of non-equilibrium statistical mechanics. The parallels have been noticed there for decades, and the cutting edge of ML research today is unifying them by cribbing aspects of each into the other.
The ideas (models) themselves are quite naive. The perceptual space of these models is essentially one-dimensional. This, in the broad strokes, is a terrible model for reality, and so these casual discussions of "intelligence" seem mis-informed, not to mention under-determined. It seems like a lot of mis-informed people speak quite confidently about the things they don't know that they don't know, especially on a site where people who value their autodidactive skills tend to congregate.
Inductive bias is the primary term of art that is missing from these discussions. The models du jour do not have any architectural components which represent language per se, only flat representations thereof. The reason LLMs work so well is because of the Transformers inside; multi-head attention is a clever trick to vastly expand the possible data processing techniques as information passes and is distilled through the network into the outputs, but ultimately it is just a fancy way of doing a technique we could call "quasi-random projections" under the domain of linear algebra.
The notion of intelligence we entertain in these discussions is hopelessly anthropocentric, and, furthermore, incredibly arrogant. The recent achievements say much more about how easy it is to simulate human-comprehensible language and images than it does about how we are birthing new intelligences. I believe the hype and deployment of these tools is doing more to hurt society in the long term than it is to help it, and I welcome more candid discussions about how simple the rules that make us "intelligent" might be. That seems to me, during my studies, to be the real question.
> sneer at the cutting edge of ML research
I was reacting specifically to mis-specifications of what "good" means in these contexts, e.g., "attention should be good for language" from GGP comment.
The major advances of these models du jour are not the outputs, but rather the infrastructure that it takes to run them. I salute all those involved in implementation. I hope (and see!) these innovations spill into things that help people and generate real value, for instance, all the things businesses use accurate climate forecasts for.
My issue mostly revolves around the discrepancy between value-created-per-FLOPS and cost-per-FLOPS. If we run huge models for what serves for us the same purpose as the TV walls in Fahrenheit 451 do for their owners, what are we adding? If we are constantly running pixel-level segmentation networks on the cameras of automobiles that are already being driven by humans, how is that helping?
Stable Diffusion is a real innovation in optimization techniques. This is the only part of recent news that can be considered "cutting edge ML research", as it is an important step in unifying optimization theory with that of non-equilibrium statistical mechanics. The parallels have been noticed there for decades, and the cutting edge of ML research today is unifying them by cribbing aspects of each into the other.
The ideas (models) themselves are quite naive. The perceptual space of these models is essentially one-dimensional. This, in the broad strokes, is a terrible model for reality, and so these casual discussions of "intelligence" seem mis-informed, not to mention under-determined. It seems like a lot of mis-informed people speak quite confidently about the things they don't know that they don't know, especially on a site where people who value their autodidactive skills tend to congregate.
Inductive bias is the primary term of art that is missing from these discussions. The models du jour do not have any architectural components which represent language per se, only flat representations thereof. The reason LLMs work so well is because of the Transformers inside; multi-head attention is a clever trick to vastly expand the possible data processing techniques as information passes and is distilled through the network into the outputs, but ultimately it is just a fancy way of doing a technique we could call "quasi-random projections" under the domain of linear algebra.
The notion of intelligence we entertain in these discussions is hopelessly anthropocentric, and, furthermore, incredibly arrogant. The recent achievements say much more about how easy it is to simulate human-comprehensible language and images than it does about how we are birthing new intelligences. I believe the hype and deployment of these tools is doing more to hurt society in the long term than it is to help it, and I welcome more candid discussions about how simple the rules that make us "intelligent" might be. That seems to me, during my studies, to be the real question.
It kind of sounds like you're saying that it should be possible to build a better GPT-3 by incorporating more linguistics domain knowledge into the model somehow, beyond stacking word embeddings and transformers together. If so... then okay, show me.
The biggest innovation of deep learning is that you can take a relatively general architecture and build surprisingly sophisticated models by training that general architecture on a ton of data, that absolutely wipe the floor with the older style of models.
AlphaGo/Zero is maybe an exception here: it's not "just" a model, it's a whole system based around a model, which is specifically designed to perform well on certain tasks, and succeeds where a more naive system would probably fail. You could argue that ChatGPT itself is an early kind of advancement along those lines, going beyond just cranking predictions from a model to a more holistic task-specific system. But it's still mostly "just" a huge model trained on a huge pile of data, and I'm not going to argue strongly that it isn't. What I am arguing is that people aren't as naive as you think, and it's not for lack of trying that LLMs don't incorporate detailed knowledge about linguistics, psychology, et al.
You suggest that transformers shouldn't be considered "good" for text, because they don't include enough inductive bias. Don't they? They're really clever things in my opinion, and I think they actually represent quite a lot of inductive bias compared to what came before, and have a lot of intuitive appeal in their respective intended task domains. I hardly think it's fair or correct to consider the transformer nothing more than a linear algebra trick.
I suppose you're arguing that they aren't "good" solely because they produce state-of-the-art results and beat literally everything else we have. I think your argument is that they only work as well as they do because they need a tremendous amount of computing power and data to get useful results out of them. That might be true, but that's literally why people are trying to make them faster and less costly to train!
Don't forget that the other big innovation of deep learning is transfer learning and fine tuning. A big LLM like GPT-3 only needs to be trained fully every once in a while, while it can be reused and adapted to a huge variety of tasks relatively efficiently and cheaply. The whole point of the article at the top of this thread is making big models cheaper and faster to train. If you're bothered by the cost- and compute-efficiency of these things, what could be better than making the training process more efficient?
Moreover, it's kind of weird to insinuate that the field doesn't care about making these things faster or cheaper to run, when immediately after any new model is announced, a flood of projects follows trying to shrink the model, improve inference speed, etc.
Finally, who said anything about intelligence? I didn't, and nobody else did either. That's a completely unrelated topic as far as I'm concerned, and I leave that one to the philosophers to debate.
The biggest innovation of deep learning is that you can take a relatively general architecture and build surprisingly sophisticated models by training that general architecture on a ton of data, that absolutely wipe the floor with the older style of models.
AlphaGo/Zero is maybe an exception here: it's not "just" a model, it's a whole system based around a model, which is specifically designed to perform well on certain tasks, and succeeds where a more naive system would probably fail. You could argue that ChatGPT itself is an early kind of advancement along those lines, going beyond just cranking predictions from a model to a more holistic task-specific system. But it's still mostly "just" a huge model trained on a huge pile of data, and I'm not going to argue strongly that it isn't. What I am arguing is that people aren't as naive as you think, and it's not for lack of trying that LLMs don't incorporate detailed knowledge about linguistics, psychology, et al.
You suggest that transformers shouldn't be considered "good" for text, because they don't include enough inductive bias. Don't they? They're really clever things in my opinion, and I think they actually represent quite a lot of inductive bias compared to what came before, and have a lot of intuitive appeal in their respective intended task domains. I hardly think it's fair or correct to consider the transformer nothing more than a linear algebra trick.
I suppose you're arguing that they aren't "good" solely because they produce state-of-the-art results and beat literally everything else we have. I think your argument is that they only work as well as they do because they need a tremendous amount of computing power and data to get useful results out of them. That might be true, but that's literally why people are trying to make them faster and less costly to train!
Don't forget that the other big innovation of deep learning is transfer learning and fine tuning. A big LLM like GPT-3 only needs to be trained fully every once in a while, while it can be reused and adapted to a huge variety of tasks relatively efficiently and cheaply. The whole point of the article at the top of this thread is making big models cheaper and faster to train. If you're bothered by the cost- and compute-efficiency of these things, what could be better than making the training process more efficient?
Moreover, it's kind of weird to insinuate that the field doesn't care about making these things faster or cheaper to run, when immediately after any new model is announced, a flood of projects follows trying to shrink the model, improve inference speed, etc.
Finally, who said anything about intelligence? I didn't, and nobody else did either. That's a completely unrelated topic as far as I'm concerned, and I leave that one to the philosophers to debate.
Convolutions are learned by the neural network. In general, how layers are composed is more of an educated trial and error process than of an intentional design producing expected results.
[deleted]
It already is.. can’t find the paper, but some image related deep learning model used sin(x) type of inputs to their model. This was a core improvement.
Not unlike how you can linear regress on x1,x1^2,x1^3,x2,x2^2. Encoding prior knowledge about input data as additional inputs helps with machine learning. 1980 called it wants its accomplishments back.
Not unlike how you can linear regress on x1,x1^2,x1^3,x2,x2^2. Encoding prior knowledge about input data as additional inputs helps with machine learning. 1980 called it wants its accomplishments back.
Sin(x)-style encoding is called positional encoding and it’s used in neural radiance fields (NeRFs) and also in transformer-based models.
Could also be Sirens[1] which was all about sin as activation, while sinusoidal positional encoding are (iirc) just from the original transformer paper.
[1]: https://arxiv.org/abs/2006.09661
[1]: https://arxiv.org/abs/2006.09661
I think most transformer models use sin and cos for positional encoding. They were recommended in the original transformer paper, Attention is All You Need.
But in that case it's not quite Fourier analysis because those functions aren't applied to the input; they are just constants that are added to the input because transformers are permutation invariant (without some positional encoding).
Was there a model that applied sin and cos to the input?
But in that case it's not quite Fourier analysis because those functions aren't applied to the input; they are just constants that are added to the input because transformers are permutation invariant (without some positional encoding).
Was there a model that applied sin and cos to the input?
In a recent-ish paper, there was an analysis of positional encodings using Neural Tangent Kernels which showed positional encodings are very similar to Fourier transformations of the inputs:
Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains, https://arxiv.org/abs/2006.10739
Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains, https://arxiv.org/abs/2006.10739
Fourier analysis generalizes to harmonic analysis, where more general results can be obtained on locally compact topological groups. I wonder if there's some generalization of these ML observations to this more general setting.
Something to ponder…
The world was all model-based analysis, and then neural networks came along. Model-based analysis-which requires a thorough understanding of the model mathematics-went out the window. Now, we let the network develop its own model, but it’s based on the data that is used to train it. But what if that data is biased? What if the data doesn’t contain some necessary “truth?”
I feel that I t’s like driving a car looking out of the rear window to see where you have been. This can’t be good.
The world was all model-based analysis, and then neural networks came along. Model-based analysis-which requires a thorough understanding of the model mathematics-went out the window. Now, we let the network develop its own model, but it’s based on the data that is used to train it. But what if that data is biased? What if the data doesn’t contain some necessary “truth?”
I feel that I t’s like driving a car looking out of the rear window to see where you have been. This can’t be good.
It's not news in ML (machine learning) that whenever you do have a reasonable understanding/model of the underlying task and an algorithm for solving it, you should use that and not try to apply ML to learn a model from data. If an acceptable non-ML solution exists, then usually nobody reasonable would suggest using ML (unless they just want to pad their resume) because the known algorithm will usually do the same thing with much less computation and more reliability and interpretability.
However, there are many, many useful real world tasks where we haven't been able to do that (e.g. almost all of image analysis, image generation, a significant part of of text analysis and generation tasks), so for these tasks the (valid) criticism doesn't really matter, as we don't have any reasonable alternatives - if let the network develop its own model, we get a somewhat working solution (with all the risks you mention), and if we don't, then we best we can make is unusably bad.
If all the windows in your car are painted opaque black except the rear window, then driving the car by looking out of the rear window to see where you have been can be as good as it gets.
However, there are many, many useful real world tasks where we haven't been able to do that (e.g. almost all of image analysis, image generation, a significant part of of text analysis and generation tasks), so for these tasks the (valid) criticism doesn't really matter, as we don't have any reasonable alternatives - if let the network develop its own model, we get a somewhat working solution (with all the risks you mention), and if we don't, then we best we can make is unusably bad.
If all the windows in your car are painted opaque black except the rear window, then driving the car by looking out of the rear window to see where you have been can be as good as it gets.
It's also not a binary choice between "ML" or "hand-crafted model" either.
The most successful applications of ML in my own career have always been judiciously building small task-specific models that fit inside a broader system. And there is also usually quite a lot of consideration being given to the model input data as well.
Similarly, I have never seen a "just throw a model at it" project succeed in a finite amount of time.
The most successful applications of ML in my own career have always been judiciously building small task-specific models that fit inside a broader system. And there is also usually quite a lot of consideration being given to the model input data as well.
Similarly, I have never seen a "just throw a model at it" project succeed in a finite amount of time.
However, for the sake of argument, it would probably be better to wait until you can see out the front, otherwise you’re pretty likely to cause an accident. (My personal opinion is that of yours to, that the good from what we have now outweighs the potential bad).
The classic approach is a human generating a function from test input data to test output data.
The ML approach is a machine generating a function from test input data to test output data.
Both approaches will be compromised by biased data.
You could argue that you don't really need the data in the classic approach, if the human knows what he's doing. But that's kind of like writing code without writing tests. I'm not arguing for or against tests here, but perhaps you'd just be shifting the source of the bias from the data to the human doing the coding?
> What if the data doesn’t contain some necessary “truth?”
This is my gut reaction to a lot of data-driven-crap. "We increased our customer acquisition by-" No you didn't. "Customers are 15% more likely to-". Nope.
> I feel that it's like driving a car looking out of the rear window
I don't think this an apt metaphor here for ML in general, but I think it could apply to other HNer's utopian vision of ML, e.g. once GPT is "good enough", we'll be able to throw away the real world data and use GPT to train more models.
The ML approach is a machine generating a function from test input data to test output data.
Both approaches will be compromised by biased data.
You could argue that you don't really need the data in the classic approach, if the human knows what he's doing. But that's kind of like writing code without writing tests. I'm not arguing for or against tests here, but perhaps you'd just be shifting the source of the bias from the data to the human doing the coding?
> What if the data doesn’t contain some necessary “truth?”
This is my gut reaction to a lot of data-driven-crap. "We increased our customer acquisition by-" No you didn't. "Customers are 15% more likely to-". Nope.
> I feel that it's like driving a car looking out of the rear window
I don't think this an apt metaphor here for ML in general, but I think it could apply to other HNer's utopian vision of ML, e.g. once GPT is "good enough", we'll be able to throw away the real world data and use GPT to train more models.
What a terrible article. This could be potentially interesting in my research but no citation to any original study, just vague descriptions of findings.
The last link in the article links to the paper [0] the article talks about, I think.
[0] https://academic.oup.com/pnasnexus/advance-article/doi/10.10...
[0] https://academic.oup.com/pnasnexus/advance-article/doi/10.10...
“You’re a nice young man, but it’s Fourier analysis all the way down.” - DSP engineer to ML engineer (probably)
Each time I talk about any problem with our signal theory expert he says: "we could use a Kalman filter" (in jest). That's like saying "hey we could use a matrix".
Richard Hamming has a story where he was shocked/appalled that some folks wanted to include him on a paper after he suggested using Linear Regression in an obvious application of linear regression. The upshot is, if it were obvious it would have been done already. So many times people have done everything they can to avoid using a matrix decomposition or a proper filter.
A lot of expertise is really:
1. Knowing when to use the right tool, and
2. Being in the right place at the right time to offer the insight.
What’s obvious to the expert is frequently intractable to the outsider. In many cases, I wouldn’t call it so much “avoidance” as “not knowing the solution exists at all”.
1. Knowing when to use the right tool, and
2. Being in the right place at the right time to offer the insight.
What’s obvious to the expert is frequently intractable to the outsider. In many cases, I wouldn’t call it so much “avoidance” as “not knowing the solution exists at all”.
I mean yes and no? Kalman filters fuses information about known system behaviour along with observations in an optimal predictor (for certain types of noise). It uses a matrix in the same way that almost every operation on a computer uses a matrix.
Edit: I missed the quip about the Kalman comment being in jest, my reply probably wasn’t necessary but I’ll leave it up.
Edit: I missed the quip about the Kalman comment being in jest, my reply probably wasn’t necessary but I’ll leave it up.
In my area, every controls engineer says "we can just use a PID".
99% of the time it seems they're not wrong, despite control engineering being a pretty large area of research.
99% of the time it seems they're not wrong, despite control engineering being a pretty large area of research.
And a simple PID controller built with op-amps looks an awful lot like a simple neural network.
http://www.ecircuitcenter.com/Circuits/op_pid/op_pid.htm
https://www.nutsvolts.com/magazine/article/the_perceptron_ci...
http://www.ecircuitcenter.com/Circuits/op_pid/op_pid.htm
https://www.nutsvolts.com/magazine/article/the_perceptron_ci...
Isn't gradient descent basically PID over parameters?
And tricks like momentum basically a low-pass filter integrated in the PID loop? It's quite weird how not that many concepts from analog electronics domain have gotten carried over to ML.
I'm nominally familiar with PID loops but am not a control engineer. What other tools would you commonly bring to bear when a PID is not applicable?
DFTs and op-amps, kid.
maybe some s-plane control theory if you’re feeling fancy.
maybe some s-plane control theory if you’re feeling fancy.
> engineer
The next level up is obvious
The next level up is obvious
With todays networks and training corpus sizes and compute resources available, it's generally better to let the network learn the best transform rather than try to guess what it needs and do it manually first.. unless you have analyzed the problem thoroughly (like I guess they did in this paper). You have several dozens of layers and if you need to "waste" a few low-level layers to learn some transform, the increased training time might offset the man-hours and risk of your meat-based feature-designers to throw something sub-optimal in front of the network..