Neural network chip built using memristors(arstechnica.com)
arstechnica.com
Neural network chip built using memristors
http://arstechnica.com/science/2015/05/neural-network-chip-built-using-memristors/
13 comments
I find it disturbing how everyone seems relatively unfazed by DARPA's intended use of this technology.
People have, I think, internalized the logic of arms races. Someone's going to have [scary technology X], so why shouldn't we have it first/best? (Sure, there are answers to that, but none that I think most people would stick to when push comes to shove).
How exactly is a memristor not a state device?
And about journalistic coverage... you seem to be knowledgeable about these programs, so there's an opportunity for you :)
And about journalistic coverage... you seem to be knowledgeable about these programs, so there's an opportunity for you :)
A lot of it is shrouded in secrecy I'm afraid, unless you're doing the research. I'm also very interested in memristors, I think it's a quantum leap forward for computing, in many respects. But there's very little information one can get out there. Would love to know where I can find out more.
I suggest you start with the Nature paper from the guys at HP[1]. Then move on to Leon Chua's original and later papers.
[1]: http://www.nature.com/nature/journal/v453/n7191/full/nature0...
[1]: http://www.nature.com/nature/journal/v453/n7191/full/nature0...
The top comment mentions DARPA research, but for those who are actually interested in the theory behind the memristor and its potential applications, here [1] is an excellent resource by the man who discovered the first physical memristor.
Furthermore, there is a huge amount of open access journal articles and wikipedia material on memristors. DARPA is nowhere close to the cutting edge, SK Hynix is already tooling ReRAM fabs for mass production.
1. https://www.youtube.com/watch?v=bKGhvKyjgLY
Furthermore, there is a huge amount of open access journal articles and wikipedia material on memristors. DARPA is nowhere close to the cutting edge, SK Hynix is already tooling ReRAM fabs for mass production.
1. https://www.youtube.com/watch?v=bKGhvKyjgLY
You know, I think it's pretty funny (with a heaping helping of schadenfreude) that the first graphic in the first link shows some completely arbitrary "DoD Sensing Requirements" compared to actual processor capability. How exactly does one bridge the gap between our current processor capabilities and fucking omniscience? I guess DARPA will try to find out.
They intend to bridge the gap by forfeiting the assumption that transistors and discrete logic need be the atomic units of computation. They give up discretization and call for exotic modes of computation that can achieve reliable, but non binary, behaviors. Chaining these new units together, the idea is to build 'inference modules' - small clusters capable of performing statistical induction and aduction - these modules than able to be chained to perform higher level logic like, for example, detecting shapes and features in input video for tracking.
So, Skynet, basically?
I'm very excited about this! Is there any possibility for me to work in a company working on similar stuff? That would make a dream come true for me! I live in Germany and would love to write my masters thesis a related topic!
This is so amazing! Btw. I have found an HP Invent sign in my town, but the security guard didn't answer any questions about it. The only thing he said was that I won't find any address or telephone number for it. That made me curious, because HP is working on a memristor based Computer, but I doubt that they produce it in Germany.
This is so amazing! Btw. I have found an HP Invent sign in my town, but the security guard didn't answer any questions about it. The only thing he said was that I won't find any address or telephone number for it. That made me curious, because HP is working on a memristor based Computer, but I doubt that they produce it in Germany.
Are you an EE? What exactly would you like to work on?
I've previous EE experience, but I study CS. I'm open for project suggestions and would love to do research in self-assembly for mass fabrication and study/develop AI Models.
Well, the obvious project suggestion is to read about challenges of building a larger crossbar, then work on overcoming those challenges. Literature list is provided in the paper.
This type of work all about mass fabrication, but has nothing to do with AI models. Which direction you want to go?
This type of work all about mass fabrication, but has nothing to do with AI models. Which direction you want to go?
I think given my background I'm a better match for AI, than for mass fabrication. That's what I'd really enjoy working on.
Link to original paper: http://rdcu.be/cJ0x
THANK YOU! Also, Dec 1 2014, this has been out a while then
More recent papers at https://scholar.google.com/scholar?&q=leon+chua+memristor
A neural network chip semi-conductor startup: http://brainchipinc.com/technology/
They "backdoor listed" on to an Australian mining company, share price went from 1 cent to 27 cents: https://www.google.com/finance?cid=11163357
Valued at $57m.
They "backdoor listed" on to an Australian mining company, share price went from 1 cent to 27 cents: https://www.google.com/finance?cid=11163357
Valued at $57m.
Man, why are all the silicon people so fixated on spiking nets? Maybe in 20 years once we figure out how the brain works they'll be great but if you built a convnet chip instead then it could be smashing records in real problems like speech recognition, translation, image identification, etc, today. Where are the convnet chip startups?
Partly because they want to do fundamental research, not R&D. This means they don't necessarily have a well specified set of problems where they'd need neural networks, but they want to understand biological neural nets. One approach in doing so is trying to stay as close to real neural features as possible.
Actually the reason (at least in this paper) is that spiking nets are much more power efficient.
First of all, what makes you think a hardware convnet will perform better than a software convnet?
Second, you can implement a convnet with a spiking circuit: http://link.springer.com/article/10.1007%2Fs11263-014-0788-3
Second, you can implement a convnet with a spiking circuit: http://link.springer.com/article/10.1007%2Fs11263-014-0788-3
That paper does not implement training, only testing. Low-power testing is good to have e.g. for mobile applications, but it doesn't increase the capabilities of our algorithms. What we need to advance the field is faster training of larger nets. That's where the really interesting applications will be discovered.
A convnet-optimized chip could clearly be much faster and more power efficient than a convnet running on a CPU or GPU. The move from CPU to GPU brought a 10x speedup already, but GPUs are hardly ideal for running convnets. For one thing, they have tons of graphics-specific hardware that's useless for convnets and could just be deleted in a convnet chip. For another, GPUs are much more flexible than necessary for convnets. The main operation you need to perform is convolution and you could make fixed-function convolution units that would be much more power and area efficient than generalized GPU shader cores. For yet another thing, there's no reason to believe that 32-bit IEEE 754 floating point is the best power/precision tradeoff for convnets. I'm willing to bet that you could go much lower. You could even experiment with approximate arithmetic; 0.5 ULP precision is probably not necessary.
A convnet-optimized chip could clearly be much faster and more power efficient than a convnet running on a CPU or GPU. The move from CPU to GPU brought a 10x speedup already, but GPUs are hardly ideal for running convnets. For one thing, they have tons of graphics-specific hardware that's useless for convnets and could just be deleted in a convnet chip. For another, GPUs are much more flexible than necessary for convnets. The main operation you need to perform is convolution and you could make fixed-function convolution units that would be much more power and area efficient than generalized GPU shader cores. For yet another thing, there's no reason to believe that 32-bit IEEE 754 floating point is the best power/precision tradeoff for convnets. I'm willing to bet that you could go much lower. You could even experiment with approximate arithmetic; 0.5 ULP precision is probably not necessary.
Building a hardware convnet is only beneficial when you figured out the exact parameters of the network. Everything is hardwired. Therefore, it's useless if you want to experiment with lots of different parameters, tricks, or architectures to "advance the field".
Moreover, building such a chip is an expensive and long process, and given how fast GPUs are improving, it's not clear that by the time you build it, it will still be competitive.
Finally, if you want to speed up training, try to figure out a better algorithm. For example, humans can learn from very few training examples. Current neural networks need many thousands, or even millions. There's a potential million-fold speed up in training time right here - and to find it, you need the flexibility of the software.
Moreover, building such a chip is an expensive and long process, and given how fast GPUs are improving, it's not clear that by the time you build it, it will still be competitive.
Finally, if you want to speed up training, try to figure out a better algorithm. For example, humans can learn from very few training examples. Current neural networks need many thousands, or even millions. There's a potential million-fold speed up in training time right here - and to find it, you need the flexibility of the software.
Memristor crossbars are exciting even outside of neural network applications: it can be used as a very dense, non-volatile memory. If this design can be scaled up, it could potentially replace both flash memory storage, and RAM.
These things are really not well respected for what they can do. IBM introduced their TrueNorth tech late last year. Those used a 'neural' design for the chips to overcome the von Newman barriers for computer design. Along the way, the TrueNorth chips also reduced power consumption by a LOT. However, those designs are still digital. With a memristor in the TrueNorth set-up you can have a similar system for input, processing, storage, and output in an analog system.
I feel that I need to emphasize that. The memristor is the component that will easily allow for analog logic to occur at digital speeds and with digital logic type systems (very grossly speaking). What these little guys can do is under-sold.
I feel that I need to emphasize that. The memristor is the component that will easily allow for analog logic to occur at digital speeds and with digital logic type systems (very grossly speaking). What these little guys can do is under-sold.
There have been dozens of analog neuromorphic chips built in the last 30 years. Latest ones use floating gate transistors for synapses.
Memristors, in theory, are better devices that flash memory (faster, lower power, more dense), however that's just in theory. In practice, they are very hard to scale. This crossbar is 12x12. No one knows how to build anything much larger than that, on a mass scale.
I have nothing to say but that this is a cool application of memristors. My graduate research is in cognitive computing and the thought of using circuitry to represent the synaptic weights as opposed to hardware/software based adders and multipliers is pretty awesome.
The general theory of memristors, meminductors, memcapacitors of any order (first order memristor is the genuine one invented in 1971 by Professor Leon Chua, however he generalized recently his discover to second, third order memristor, etc.)is published in a paper I wrote with him on september 2014 in International Journal of Bifurcation and Chaos. One can download it freely from the site
https://www.researchgate.net/publication/261676241_MEMFRACTA...
This looks cool but I'm somewhat skeptical. I would be more interested in seeing what problem the system solves better or decently (say even MNIST) rather than how it was built using memristors.
There is a lesson from IBM trying to mimic a rat's brain -- that is you try to solve a problem rather than just burn power.
There is a lesson from IBM trying to mimic a rat's brain -- that is you try to solve a problem rather than just burn power.
>>>I would be more interested in seeing what problem the system solves better
Better than what?
Better than what?
Better than previous state of the art on a standard machine learning dataset. Check some leaderboards here: http://rodrigob.github.io/are_we_there_yet/build/#datasets
What are you talking about? They build a 12x12 crossbar. The best you can do with it is to implement a single layer perceptron to classify 3x3 pixel patterns. Once they figure out how to scale it up, they will implement a larger network.
The technology sounds very promising but if the goal is to simulate the brain, the ANN models we have today are inadequate. Current evidence suggests that it needs to incorporate dendritic dynamics and , soon, molecular computation.
Is the goal simulation or functional equivalence?
For example, to simulate a horse, is it necessary to create legs, or is it okay to build a road and use wheels?
For example, to simulate a horse, is it necessary to create legs, or is it okay to build a road and use wheels?
possibly, we don't know yet, but ANNs are not isomorphic to real neurons. If we are talking about a bottom-up approach to intelligence, we have to opt for realistic simulation. If, on the other hand we knew what intelligence is, we can simulate it any way we like.
Memristers are supposed to be the main memory of HP's future computing project called The Machine. It is supposed to be as fast as register memory and compact as flash.
[deleted]
Lemme know when I can order a few on cut tape from DigiKey.
I know it's not what you're asking, but for the record you can buy memristor-memory-based microcontrollers in single quantities from Mouser today.
The reason I mention it is that memristors are being accused of being vaporware when really they're in production already.
The reason I mention it is that memristors are being accused of being vaporware when really they're in production already.
LINK! You just can't say that and not provide a link to the catalog for us!
"Even on a 30 nm process, it would be possible to place 25 million cells in a square centimeter, with 10,000 synapses on each cell. And all that would dissipate about a Watt."
Wow - seems like a lot.
Human brain by comparison (sourced by google): - 12 watts - 100 billion neurons - 1000 trillion connections
Computing with memsisters is going to be very interesting.
Wow - seems like a lot.
Human brain by comparison (sourced by google): - 12 watts - 100 billion neurons - 1000 trillion connections
Computing with memsisters is going to be very interesting.
The human brain has between 100 and 500 trillion synapses and consumes a lowly 12 watts. (In contrast, a 12.6 megawatt supercomputer, in 2013, took 40 minutes to simulate one second of biological brain activity.)
The article cites 250 billion synapses per watt.
For the same 12 watts as a human brain eats up, a set of memristors could simulate three trillion synapses. A cat, in comparison, has 10 trillion. To get 100 trillion synapses, multiply those 12 watts by 33.3 to get 400 watts (the draw of nearly seven 60-watt incandescent bulbs).
Since one watt bags us 250 billion synapses and 400 watts is equivalent to a memristor-based human brain, then 400 cm^2 is the area needed to emulate the meekest of human minds.
That's nearly the same area as half of a medium Domino's pizza.
Certainly is... food for thought.
The article cites 250 billion synapses per watt.
For the same 12 watts as a human brain eats up, a set of memristors could simulate three trillion synapses. A cat, in comparison, has 10 trillion. To get 100 trillion synapses, multiply those 12 watts by 33.3 to get 400 watts (the draw of nearly seven 60-watt incandescent bulbs).
Since one watt bags us 250 billion synapses and 400 watts is equivalent to a memristor-based human brain, then 400 cm^2 is the area needed to emulate the meekest of human minds.
That's nearly the same area as half of a medium Domino's pizza.
Certainly is... food for thought.
"In contrast, a 12.6 megawatt supercomputer, in 2013, took 40 minutes to simulate one second of biological brain activity."
Just as a note: we are not sure whether that large computer really simulated brain activity or not. The tricky thing in brain research is that we have practically zero* idea about what matters and what can be omitted from the simulation. (For example, the glia cells seem to be important -- until recently, we have disregarded their role.)
So at this point even we had an infinitely big computer we could not simulate the brain properly because we don't know what exactly to simulate.
*zero means that there's much more we don't know than what we know.
Just as a note: we are not sure whether that large computer really simulated brain activity or not. The tricky thing in brain research is that we have practically zero* idea about what matters and what can be omitted from the simulation. (For example, the glia cells seem to be important -- until recently, we have disregarded their role.)
So at this point even we had an infinitely big computer we could not simulate the brain properly because we don't know what exactly to simulate.
*zero means that there's much more we don't know than what we know.
As I never tire of quipping, if you can't get it high on neurotransmitters, it isn't a neural network.
"In contrast, a 12.6 megawatt supercomputer, in 2013, took 40 minutes to simulate one second of biological brain activity."
Not really a fair comparison. That's like simulating one processor with another and complaining that 1 second of activity took 40 minutes to simulate. If we can implement the NN directly, rather than simulate it, we should expect a much smaller performance gap.
If your calculations are correct, that doesn't sound so bad. If we could implement a brain-sized (by number of neurons) neural network in the area as half of a medium Domino's pizza, that seems like a damn good achievement to me!
Not really a fair comparison. That's like simulating one processor with another and complaining that 1 second of activity took 40 minutes to simulate. If we can implement the NN directly, rather than simulate it, we should expect a much smaller performance gap.
If your calculations are correct, that doesn't sound so bad. If we could implement a brain-sized (by number of neurons) neural network in the area as half of a medium Domino's pizza, that seems like a damn good achievement to me!
Remember though that the brain seems to be a rather modular structure.
While the total area is quite big, we could probably get a lot of mileage out of much smaller subsystems taking care of only, say, visual pattern recognition, or speech analysis, in much smaller dies.
Someone should read Catherine Malibou and Francisco Varela :-)
Neuroscience is already bored with not only subsystems, the holistic nature of the brain, but also the notion of the brain as the body's sole processor of cognition.
Neuroscience is already bored with not only subsystems, the holistic nature of the brain, but also the notion of the brain as the body's sole processor of cognition.
Brain synapse response time is very slow. Depending on type it is in order of milliseconds.
I guess that's part of why such a large part of our thinking is subconscious parallel processing: as a workaround for this technical limitation.
Does that take into account computation speed per synapse/memristor?
The thought of a drone with the intelligence of a cat is a scary thought...
The numbers are interesting, though. 400 square centimetres sounds to me to be in the ballpark of a human brain (accounting for several layers).
The numbers are interesting, though. 400 square centimetres sounds to me to be in the ballpark of a human brain (accounting for several layers).
> The thought of a drone with the intelligence of a cat is a scary thought...
And I immediately imagined drones going around the landing gears of bigger planes demanding their attention...
And I immediately imagined drones going around the landing gears of bigger planes demanding their attention...
put that on a 20cm x 30cm surface (laptop) ... and you have 25m x 600 = 15bn cells, 150 trillion synapses using 600W.
If they can ever fabricate such a thing ... those neural networks are going to compute some scary stuff!
If they can ever fabricate such a thing ... those neural networks are going to compute some scary stuff!
> If they can ever fabricate such a thing ... those neural networks are going to compute some scary stuff!
Do we have to pay extra for Austrian accent?
Do we have to pay extra for Austrian accent?
[deleted]
The idea, and this can be seen on DARPA's slides (http://www.darpa.mil/workarea/downloadasset.aspx?id=21474857...), is to get computation that is several orders of magnitude higher for their specialized sets of problems than what can theoretically be reached by traditional computing models even if Moore's law continues.
DARPA would like to first apply this technology to ARGUS drone systems (https://www.youtube.com/watch?v=QGxNyaXfJsA) and related technology because streaming video can't be done to the ground, tracking and decision making must be done on board - yet traditional processing platforms can only track a few orders of magnitude fewer targets that what the military would like.
In a more advanced phase, if memristor or coupled oscillator (etc) approaches to building inference models become possible, then programs written in DARPA's other initiative (Probablistic Programming) could be programmed into these exotic solid state devices to compute in a way more analogous to today's generic computation. And indeed, eventually the adoption of Probablistic Programming will train programmers to write code for quantum computers - while more complicated, replacing Probablistic Programming's PDFs with probability amplitudes almost get one there.
I hope to see more journalistic coverage of some of the other exotic devices.