All-optical machine learning using diffractive deep neural networks(science.sciencemag.org)
science.sciencemag.org
All-optical machine learning using diffractive deep neural networks
http://science.sciencemag.org/content/early/2018/07/25/science.aat8084
11 comments
I asked this on Twitter, but maybe folks here can answer better: how important is nonlinearity for deep neural networks? This method's output seems to be a linear function of its (complex) input. Does that put important bounds on performance?
https://mobile.twitter.com/AndrewGYork/status/10228414045888...
Echoing the other respondents–if you don't have a nonlinearity, your whole network is just a sequence of linear transforms, which (multiplied out) is the same as a single linear transform. Meaning that removing the nonlinearities gives you (effectively) a one-layer network.
Mind boggled that this article comes up now. Been working on similar tech recently, and the question of non-linearality arose right away. The discussed conclusion was "impossible". Yet I was able to design a crude NAND gate. So there has to be non-linearality it the quantum nature of diffraction and interference.
Linear functions/transforms have two important properties
But yeah, in NN nonlinearities are very important, otherwise they would be simplifiable to a single transformation
f(x) + f(y) = f(x+y)
k*f(x) = f(x*k)
But yeah, in NN nonlinearities are very important, otherwise they would be simplifiable to a single transformation
Though the other comments are correct, I want to point out that you can get some nontrivial behavior with only linear functions. For example, low-rank matrix factorization is kind of like a neural network
f(x) = U * V * x,
where U is an n by k matrix and V is a k by m matrix, where k is much smaller than n and m. Basically, we are constraining the set of allowed linear transformations, which is a form of regularization. Convolutional layers in neural networks similarly restrict the allowed linear transformations.
Nevertheless, the power of linear neural networks is far less than that off nonlinear networks.
f(x) = U * V * x,
where U is an n by k matrix and V is a k by m matrix, where k is much smaller than n and m. Basically, we are constraining the set of allowed linear transformations, which is a form of regularization. Convolutional layers in neural networks similarly restrict the allowed linear transformations.
Nevertheless, the power of linear neural networks is far less than that off nonlinear networks.
Without a non-linearity, it's just a linear function. So the performance won't be that great.
Nonlinearity is very important and is the only reason why neural nets can approximate arbitrary functions. You can’t do that with linear transformations alone. Though from briefly skimming the paper they do seem to achieve similar effects through phase modulation. Otherwise even MNIST would be out of the question.
There is an arxiv version of the paper at https://arxiv.org/abs/1804.08711.
This is really important if it can be scaled up. Imagine being able to replace an onsite heavy compute core pulling several hundred watts (like in the trunk of a Waymo car) with a passive unpowered optical element. Pretty amazing.
I'd just like to point out that back in the 60s there was this physical neural network called the Perceptron. Yeah, like the ones you learn about. However, it was a patchpanel machine (the wires were the neurons) with weights that were implemented as potentiometer knowbs attached to little servos. It could do backprop (IIRC it was a single layer) and basic image recognition.
Some days I feel like neural network hardware is the new laser: at one point, nobody thought it could exist, but once one was made, new designs started to fall out of the woodwork. Like gravitational lenses, there are actually "galactic laser foundries" that generate lasers purely out of stellar physics.
Some days I feel like neural network hardware is the new laser: at one point, nobody thought it could exist, but once one was made, new designs started to fall out of the woodwork. Like gravitational lenses, there are actually "galactic laser foundries" that generate lasers purely out of stellar physics.
The title doesn't do the article justice. This is a physical, 3D-printed neural network, in form of plates that pass/reflect light. You shine your input at one end, get results at the other.
I'm very impressed. Not sure if this has any chance of being more efficient than traditional NNs implemented in silicon, but I can imagine some fun applications. For instance, with some optics in front, I think it could be used as a passive classifier of what's in front of the detector - you could set up an array of photodetectors in the back, that operate a low-power device only when appropriate pattern is detected.
I'm very impressed. Not sure if this has any chance of being more efficient than traditional NNs implemented in silicon, but I can imagine some fun applications. For instance, with some optics in front, I think it could be used as a passive classifier of what's in front of the detector - you could set up an array of photodetectors in the back, that operate a low-power device only when appropriate pattern is detected.
Can someone help me Grok how the 3D printed Neural Networks back propagate? As I am trying to go through the paper they describe it as a pure optical approach, but what adjusts the refraction elements during the learning process?
from the paper:
"[...] learnable network parameter that is iteratively adjusted during the training process of the diffractive network, using an error back-propagation method. After this numerical training phase implemented in a computer, the D^2NN design is fixed and the transmission/reflection coefficients of the neurons of all the layers are determined. This D^2NN design, once physically fabricated using e.g., 3D-printing, 3lithography, etc., can then perform, at the speed of light propagation, the specific task that it is trained for, using only optical diffraction and passive optical components/layers, creating an efficient and fast way of implementing machine learning tasks."
"[...] learnable network parameter that is iteratively adjusted during the training process of the diffractive network, using an error back-propagation method. After this numerical training phase implemented in a computer, the D^2NN design is fixed and the transmission/reflection coefficients of the neurons of all the layers are determined. This D^2NN design, once physically fabricated using e.g., 3D-printing, 3lithography, etc., can then perform, at the speed of light propagation, the specific task that it is trained for, using only optical diffraction and passive optical components/layers, creating an efficient and fast way of implementing machine learning tasks."
So the optical component is only the end result model of the NN? It isn't learning using the optics?
Only indirectly. the physical device only does feedforward so they had to train it using tensorflow on a conventional device.
Check out this new article:
https://www.osa.org/en-us/about_osa/newsroom/news_releases/2...
They implemented a back propagation algorithm using just optical.
https://www.osa.org/en-us/about_osa/newsroom/news_releases/2...
They implemented a back propagation algorithm using just optical.
Which is unfortunate, since learning is the compute intensive part
Can't access the article, but the abstract reads like this is inference only. If that's so, this still could be useful. For instance, you could train a network somewhere else, then implement the learned function optically.
You can almost hear the truckloads of grant money driving to their office.
This has no nonlinearities, this isn't a neural network, it's a linear classifier...
related: "Researchers Move Closer to Completely Optical Artificial Neural Network" https://news.ycombinator.com/item?id=17730775
[deleted]
[deleted]