A Guide to Deep Learning and Neural Networks(serokell.io)
serokell.io
A Guide to Deep Learning and Neural Networks
https://serokell.io/blog/deep-learning-and-neural-network-guide
5 comments
In my experience no amount of reading or watching videos can really give you a sense of how deep learning works, what it’s good and bad at, and how it both does and doesn’t feel like magic. The only thing that works is coding something up in Torch or similar and trying it out.
I mean .. I disagree?
It's certainly simple to give "a sense" of how deep learning works -
1) translate real-world states and behaviors into data structures (typically multi-dimensional arrays);
2) use (relatively straightforward) math to find patterns in the large matrices;
3) assume those patterns are representative of the real-world states and behaviors it translated;
4) produces a model which "works in reverse" i.e. uses the patterns it found to generate real-world states and behaviors, particularly states and behaviors it has never translated or seen before.
And therefore what it's good at - math, finding patterns - are simple to get a sense of, what it's bad at - assuming the patterns it's found are representative - is simple to understand, and why it feels like magic - the speed at which it can acquire a skill, that it's trained solely through observation - and why it doesn't feel like magic - it's just math and pattern-matching, things we teach in preschool.
Every new breakthrough in AI/DL research is either "more math", "same math but faster", "more patterns", "better patterns" ... Always good to read and see how very smart people solved problems, but the underlying principles are really simple.
What's way harder to wrap your head around is the distributed compute and data pipelines and hardware architectures required to support the scales at which these simple mathematical pattern-matching algorithms need to operate to be useful.
It's certainly simple to give "a sense" of how deep learning works -
1) translate real-world states and behaviors into data structures (typically multi-dimensional arrays);
2) use (relatively straightforward) math to find patterns in the large matrices;
3) assume those patterns are representative of the real-world states and behaviors it translated;
4) produces a model which "works in reverse" i.e. uses the patterns it found to generate real-world states and behaviors, particularly states and behaviors it has never translated or seen before.
And therefore what it's good at - math, finding patterns - are simple to get a sense of, what it's bad at - assuming the patterns it's found are representative - is simple to understand, and why it feels like magic - the speed at which it can acquire a skill, that it's trained solely through observation - and why it doesn't feel like magic - it's just math and pattern-matching, things we teach in preschool.
Every new breakthrough in AI/DL research is either "more math", "same math but faster", "more patterns", "better patterns" ... Always good to read and see how very smart people solved problems, but the underlying principles are really simple.
What's way harder to wrap your head around is the distributed compute and data pipelines and hardware architectures required to support the scales at which these simple mathematical pattern-matching algorithms need to operate to be useful.
I have a similar experience. For the last 10 months I've had the opportunity to use Keras at work. While reading deep learning articles prior to that gave me an idea of what I needed to know, it was no substitute for experience. In a lot of ways it parallels learning monads. It is something that has to be experienced. As an aside we're now at peak VAE tutorials in the same way we hit peak monad tutorials a few years back.
If one found this article interesting and are interested in getting some hands-on learning/training on the topic I'd recommend this free open online course provided by a collaboration of Nordic Universities:
https://www.elementsofai.com/
Some might be to easy for you.. then just skim through it, but if Deep Learning is new to you there's some more in-depth interactive exercises in it.
Some might be to easy for you.. then just skim through it, but if Deep Learning is new to you there's some more in-depth interactive exercises in it.
Not sure why we need another one of these, it does not add anything compared to every other tutorial there's around, it only makes things more confusing with imprecision and errors, in particular:
1. "Machine learning attempts to extract new knowledge from a large set of pre-processed data loaded into the system. Programmers need to formulate the rules for the machine, and it learns based on them" - plain wrong, machine learning _learns_ the rules, that's the whole point.
2. "The main architectures of deep learning are: [...] recurrent neural networks, [...] and recursive neural networks" - recursive NNs include recurrent NNs, and almost nobody uses the term "recursive".
3. Comparing DL to ML makes no sense. It's like comparing apples to fruits. Not to mention that almost every point is questionable in some way or the other: low quality datasets give low quality predictors, regardless, ANNs can be even more sensitive to raw data compared to some other ML algorithms, and not all ML models that are not ANNs are interpretable.
4. "Deep learning doesn’t rely on human expertise as much as traditional machine learning" - of course it does, simply because DL is _not_ the right thing to do in _most_ cases.
5. "in many cases, deep learning cannot be substituted." - see previous point.
6. "An artificial neural network is heavily inspired by the structure of a human brain" - the connection is tenuous at best.
7. "Every neural network consists of [...], and functions" - activations is the right term.
8. "Neurons only operate numbers in the range [0,1] or [-1,1]." - wrong.
9. "There are a lot of activation functions, we will consider the most common ones: linear, sigmoid and hyperbolic tangent" - linear activations are used only in very specific situations. And the relu family has been the most commonly used activation for years.
And I am only half-way.
I am really sorry to be like this but this is very low quality material.
1. "Machine learning attempts to extract new knowledge from a large set of pre-processed data loaded into the system. Programmers need to formulate the rules for the machine, and it learns based on them" - plain wrong, machine learning _learns_ the rules, that's the whole point.
2. "The main architectures of deep learning are: [...] recurrent neural networks, [...] and recursive neural networks" - recursive NNs include recurrent NNs, and almost nobody uses the term "recursive".
3. Comparing DL to ML makes no sense. It's like comparing apples to fruits. Not to mention that almost every point is questionable in some way or the other: low quality datasets give low quality predictors, regardless, ANNs can be even more sensitive to raw data compared to some other ML algorithms, and not all ML models that are not ANNs are interpretable.
4. "Deep learning doesn’t rely on human expertise as much as traditional machine learning" - of course it does, simply because DL is _not_ the right thing to do in _most_ cases.
5. "in many cases, deep learning cannot be substituted." - see previous point.
6. "An artificial neural network is heavily inspired by the structure of a human brain" - the connection is tenuous at best.
7. "Every neural network consists of [...], and functions" - activations is the right term.
8. "Neurons only operate numbers in the range [0,1] or [-1,1]." - wrong.
9. "There are a lot of activation functions, we will consider the most common ones: linear, sigmoid and hyperbolic tangent" - linear activations are used only in very specific situations. And the relu family has been the most commonly used activation for years.
And I am only half-way.
I am really sorry to be like this but this is very low quality material.
The fact this post differentiates between Machine Learning and Deep Learning (Deep Learning is a subset of Machine Learning) is rather silly. Also early on it states "Usually, deep learning is unsupervised or semi-supervised.", which is false? I would not upvote, seems pretty questionable...
Appropriate terminology aside, deep learning at one point at least clearly represented neutral networks as a "one size fits all" algorithmic data structure because it could (more) easily scale, be distributed, etc. and could produce results unsupervised in a way that other ML algos could not.
Now with all the discussion on ethics and bias in AI, not sure if unsupervised black box architectures really have a place in the discussion ...
Now with all the discussion on ethics and bias in AI, not sure if unsupervised black box architectures really have a place in the discussion ...
Distinctions like this remind me what a professor once told me when he was introducing the module on nanostructures: "This used to just be called Chemistry, but we realised we could get more funding calling it this way".