Google acquires neural network startup that may help it hone speech recognition(engadget.com)
engadget.com
Google acquires neural network startup that may help it hone speech recognition
http://www.engadget.com/2013/03/13/google-acquires-neural-network-startup-dnnresearch/
5 comments
So actually if you read Geoff Hinton's paper on how they won the ImageNet challenge, they use 2 GPUs. Whether this qualifies as "custom hardware" or not, it's debatable, but it's certainly striking that you can obtain performance that is about as good as the 16k cores result. The results and the models are not directly comparable, but still.
One limitation of using GPUs is the fact that they have comparatively little on-board memory (max 6GB for now) and shuffling data in/out of it is expensive (because of communication costs).
Now there's also the idea that you can maybe obtain something interesting from a learning perspective when training on 16k cores. Because you have to essentially train a bunch of different models, that have different parameters and see potentially different subsets of your training set, ultimately you perform some sort of model averaging (since training is doing it for you). In essence, by doing distributed gradient descent you are potentially training a clever ensemble of networks, which arguably could give better generalization.
Can you have an ensemble of GPUs? Now that's an interesting question too :)
One limitation of using GPUs is the fact that they have comparatively little on-board memory (max 6GB for now) and shuffling data in/out of it is expensive (because of communication costs).
Now there's also the idea that you can maybe obtain something interesting from a learning perspective when training on 16k cores. Because you have to essentially train a bunch of different models, that have different parameters and see potentially different subsets of your training set, ultimately you perform some sort of model averaging (since training is doing it for you). In essence, by doing distributed gradient descent you are potentially training a clever ensemble of networks, which arguably could give better generalization.
Can you have an ensemble of GPUs? Now that's an interesting question too :)
I was thinking about ASIC? Here is a bitcoin mining solution: http://www.butterflylabs.com/
Check this out, if you're curious about this line of thought: http://yann.lecun.com/exdb/publis/pdf/farabet-fpl-09.pdf
The ImageNet winning model is a essentially glorified ConvNet :)
The ImageNet winning model is a essentially glorified ConvNet :)
I'd argue we hit the tipping point more than ten years ago. Finance and a lot of tech (e.g. google) had already been transformed by machine learning at that point. At my last job we were working with some pretty complex neural network models that were first developed in the early to mid 90s.
AI research died off in the 1960s and 1970s because the results didn't even get close to matching the expectations. Machine learning's current commercial success ensures that won't happen in the foreseeable future. That said, it's worth tempering our enthusiasm. To put it perspective, deep networks are extremely far from recreating general, human-level intelligence. There's a long way to go, and I'm not convinced the path we're on is the 'correct' one to reaching the kind of general AI people see in the movies.
AI research died off in the 1960s and 1970s because the results didn't even get close to matching the expectations. Machine learning's current commercial success ensures that won't happen in the foreseeable future. That said, it's worth tempering our enthusiasm. To put it perspective, deep networks are extremely far from recreating general, human-level intelligence. There's a long way to go, and I'm not convinced the path we're on is the 'correct' one to reaching the kind of general AI people see in the movies.
I was under the impression that "Turing Test-passing general sapience" was actually a pretty niche field in AI/ML these days. Most everyone seems to have found it cheaper to build a special-purpose machine learning engine/algorithm for their particular problem than to wait for a speaking, reasoning human-equivalent computer.
True general artificial intelligence would be capable of actually solving problems. Which means it could write code, engineer things, maybe even improve itself, etc. Most machine learning is limited to mindlessly exploring a search space which makes it good mostly at pattern recognition. But they are pretty terrible at solving difficult problems or writing code and things like that. Including special purpose algorithms.
We have the learning part down (somewhat), but the intelligence part not so much.
We have the learning part down (somewhat), but the intelligence part not so much.
It's my understanding that deep learning networks have to be tuned almost perfectly to perform well. Not my area ef expertise, but I'm not convinced just yet.
It's true that Deep Belief Networks take a lot of work compared to, say, Random Forests, but they make good Neural Networks much easier.
Before Deep Learning, Neural Networks required a lot of hand tuning. Without that kind of tuning, Neural Networks with more than 1 or 2 hidden layers tended to perform very badly, because they would get stuck in plateaus or poor local optima.
In contrast, (one facet of) Deep Learning focuses on using unsupervised learning to better initialize NNs, which in turn allows for much higher levels of accuracy. In essence, Deep Learning helps pick the NN parameters automatically. So it's actually decreasing the amount of hand tuning needed.
Before Deep Learning, Neural Networks required a lot of hand tuning. Without that kind of tuning, Neural Networks with more than 1 or 2 hidden layers tended to perform very badly, because they would get stuck in plateaus or poor local optima.
In contrast, (one facet of) Deep Learning focuses on using unsupervised learning to better initialize NNs, which in turn allows for much higher levels of accuracy. In essence, Deep Learning helps pick the NN parameters automatically. So it's actually decreasing the amount of hand tuning needed.
In fact almost the opposite! Unsupervised pre-training learns the statistical invariants in the inputs and these become the features.
This is in contrast to hand engineering features for your particular domain which, unsurprisingly, doesn't generalise well.
This is in contrast to hand engineering features for your particular domain which, unsurprisingly, doesn't generalise well.
Basically not true (anymore). If you take the model that is described by Hinton et al and apply it to some vision task for which you generated a lot of data, it will invariably work better than the competition, almost out of the box (yeah you have to try a few learning rates, but that basically the case for any ML algorithm). It's pretty amazing actually.
I took Geoff Hinton's Intro to Neural Networks course a few years back. After class he would have optional Q&A sessions where he would answer anything about anything. They were amazing. I think we even asked him once about his views on Margaret Thatcher.
After we covered deep belief nets, RBMs, and autoencoders, I asked him if anyone was using deep belief nets in the real world. He said "no, but these results are too impressive to deny. Within a few years, they will be."
Looks like he was right. Congrats, Geoff.
After we covered deep belief nets, RBMs, and autoencoders, I asked him if anyone was using deep belief nets in the real world. He said "no, but these results are too impressive to deny. Within a few years, they will be."
Looks like he was right. Congrats, Geoff.
Hopefully he will be able to continue teaching the Neural Networks course on Coursera
https://www.coursera.org/course/neuralnets
I took Geoff Hinton's Coursera NN class last year - excellent.
I am glad that he is remaining part time at his university.
I share other people's opinions on the importance of deep learning. At first, when it was introduced in class, I thought that it was a nice hack - but it is much better than that.
I was on a DARPA NN advisory panel in the late 1980s and did a fairly successful commercial NN product. Lots of advancements in the fields in the last 20 years!
I am glad that he is remaining part time at his university.
I share other people's opinions on the importance of deep learning. At first, when it was introduced in class, I thought that it was a nice hack - but it is much better than that.
I was on a DARPA NN advisory panel in the late 1980s and did a fairly successful commercial NN product. Lots of advancements in the fields in the last 20 years!
Congrats to Prof. Hinton, Alex, and Ilya, I have worked some with these techniques and they are so powerful. I'm sure being part of the Google team will give them the access to the processing power and the raw data to make some really cool stuff.
How far this could go is hard to say right now but things like this [2] (10 million images, 1 week training, 16,000 cores) are bringing enormous processing power to bear for the first time on these kind of problems.
Is anyone able to comment on what moving to custom hardware will mean for performance on these kind of tasks?
[1] http://www.cs.toronto.edu/~hinton/absps/imagenet.pdf
[2] http://static.googleusercontent.com/external_content/untrust...