How Claude Shannon helped kick-start machine learning(spectrum.ieee.org)
spectrum.ieee.org
How Claude Shannon helped kick-start machine learning
https://spectrum.ieee.org/claude-shannon-information-theory
33 comments
He was also a jazz musician (the clarinet), a somewhat accomplished juggler, a devoted unicycle enthusiast, and left behind a basement full of contraptions he was building in various states of finish - like the electronic mouse navigating a maze, a chess playing machine, and all other kinds of curiosities. His papers are coherent and still relevant to this day and follow the birth of each of these fields like information theory and artificial intelligence. Who knows what else he might have been working on at Bell labs that we may not be privy too.
Not explicitly mentioned in TFA is that Shannon was the first to apply the Minimax algorithm [1] for computer chess. The Minimax algorithm, later streamlined to include alpha-beta pruning, has been a key component in AI game playing machines ever since.
[1]. https://en.wikipedia.org/wiki/Minimax [2]. https://stanford.edu/~cpiech/cs221/apps/deepBlue.html
I've posted this before, a compiled list of the machines and gadgets Claude Shannon built to experiment with simple AI ideas to play games [3]. Apologies for the repeat:
[3]. https://boardgamegeek.com/geeklist/143233/claude-shannon-man...
[1]. https://en.wikipedia.org/wiki/Minimax [2]. https://stanford.edu/~cpiech/cs221/apps/deepBlue.html
I've posted this before, a compiled list of the machines and gadgets Claude Shannon built to experiment with simple AI ideas to play games [3]. Apologies for the repeat:
[3]. https://boardgamegeek.com/geeklist/143233/claude-shannon-man...
Interesting episode from Tim Harford's Cautionary Tales podcast about Claude Shannon and his various other interests:
https://www.pushkin.fm/episode/fritterin-away-genius/
https://www.pushkin.fm/episode/fritterin-away-genius/
"His papers are coherent"
High praise for one of the greatest minds of his generation
High praise for one of the greatest minds of his generation
It actually is, most papers written today are cut and paste jobs with multiple people writing one section. The quality and focus from one paragraph to another can change. I think the lack of computers helped a bit as well for the quality of earlier papers. There used to be a lot of effort needed to publish a single paper since computers couldn't do typesetting.
Aside from that he has what comes off as a very clear mind. If you haven't I would highly recommend reading his papers. Coherent is a good adjective to describe his work.
Aside from that he has what comes off as a very clear mind. If you haven't I would highly recommend reading his papers. Coherent is a good adjective to describe his work.
They are easy to understand even for average people.
Damn, I was happier not knowing about the juggling and unicycle stuff.
A film "The Bit Player" featuring "contemporary interviews, archival film, animation and dialogue from interviews with Shannon" was released in 2019. [0][1].
[0] https://www.youtube.com/watch?v=D7WmMSAxq8s
[1] https://www.amazon.com/Bit-Player-John-Hutton/dp/B08D2N2MPH/...
[0] https://www.youtube.com/watch?v=D7WmMSAxq8s
[1] https://www.amazon.com/Bit-Player-John-Hutton/dp/B08D2N2MPH/...
Robert Price's 1982 interview ("oral history") with Shannon:
https://ethw.org/Oral-History:Claude_E._Shannon
https://ethw.org/Oral-History:Claude_E._Shannon
Thanks.
Unfortunately I am prime member but apparently I cannot watch this outside US
Unfortunately I am prime member but apparently I cannot watch this outside US
In my mind Claude Shannon is the greatest mind in modern history (move over Einstein!). He’s vastly under-appreciated outside of CS circles.
Information theory continues to be the one thing that sets my brain on fire to this day. Nothing else in science/engineering/math is as self-reinforcing in its concepts to me. Perhaps this is simply because I deal with the consequences of information theory every single day when I have to make decisions like "should this be a 32 or 64 bit integer" or "Should I compress individual items or batches of those items"
You don't even have to be able to remember the equations to get this stuff right many times. There is definitely an intuition that you can develop where you look at a test file's size and instantly realize it could not contain the x264 stream of data you were expecting it to.
You don't even have to be able to remember the equations to get this stuff right many times. There is definitely an intuition that you can develop where you look at a test file's size and instantly realize it could not contain the x264 stream of data you were expecting it to.
As a physicst I associate him primarily with Shannon-Nyquist, but his name has that Gaussian quality of cropping up everywhere.
Ironically, here you've just coined yet another thing named after Gauss!
Maybe von Neumann belongs in the same category.
Maybe von Neumann belongs in the same category.
Shannon-Nyquist alone is quite the achievement for anybody.
He's up there, but von neumann still beats him out IMO.
He was clearly a smart person.
I do wonder though, if the reason we won’t see his likes again is because he was truly a one-off, or that his particular environment enabled him to show his excellence.
I’ve been reading “The Idea Factory”, about Bell Labs, recently.
I do wonder though, if the reason we won’t see his likes again is because he was truly a one-off, or that his particular environment enabled him to show his excellence.
I’ve been reading “The Idea Factory”, about Bell Labs, recently.
Environment definitely plays a big role. It’s a balance of environment and individual capabilities. When assessing intellectual contributions, I think people tend to underestimate environmental and structural factors (were you working at a place like bell labs, did you go to an elite school, etc.) and overestimate the “innate” abilities or gifts of the individual.
If you look at intellectual history, almost every genius worked in an environment in which they were surrounded by equally brilliant minds, or had ample correspondence with other thinkers of their time (of course there are exceptions).
If you look at intellectual history, almost every genius worked in an environment in which they were surrounded by equally brilliant minds, or had ample correspondence with other thinkers of their time (of course there are exceptions).
If we're talking about Machine Learning in particular Turing deserves a mention for his 1948 paper The Unorganized Machine. It's insane to me how this anticipated neural nets and genetic algorithms at a time when computers barely existed.
https://weightagnostic.github.io/papers/turing1948.pdf
https://weightagnostic.github.io/papers/turing1948.pdf
[deleted]
algorithms are just a condensed logical representation of information. that's why i associate him w the field
You have any questions to me?
It is amazing how much of what we perceive as new had actually deep roots in past thinking.
It is amazing how much of what we perceive as new had actually deep roots in past thinking.
What's it feel like to be dead?
The reports of my death are greatly exaggerated.
Jon vom Neumann also comes to mind when thinking about brilliant people from the past century.
There's a new bio of von Neumann "The Man from the Future: The Visionary Life of John von Neumann" coming out in February in the US. [0] If you're impatient you can order a copy from Amazon's UK site where it's been available since October.
[0] https://www.amazon.com/Man-Future-Visionary-Life-Neumann/dp/...
[0] https://www.amazon.com/Man-Future-Visionary-Life-Neumann/dp/...
> Nobel Laureate Hans Bethe could not comprehend von Neumann's incredible intellect: "I have sometimes wondered whether a brain like von Neumann's does not indicate a species superior to that of man."
There are plenty of smart humans, but John von Neumann was something else entirely. Like the author of the article concludes about Shannon, I wonder if we will ever see another "human" mind like that again.
There are plenty of smart humans, but John von Neumann was something else entirely. Like the author of the article concludes about Shannon, I wonder if we will ever see another "human" mind like that again.
>There are two types of genius. Ordinary geniuses do great things, but they leave you room to believe that you could do the same if only you worked hard enough. Then there are magicians, and you can have no idea how they do it. Feynman was a magician.
—Hans Bethe
Seems like Bethe hung out with some elevated company.
Seems like Bethe hung out with some elevated company.
Backyard BBQ’s and cocktail parties must have been phenomenal. Just to be a fly on the wall. Wow.
I was chatting about him with a friend and pointing out all the things in the room influenced by him. Of course he led a pretty normal life, so no one is going to make a "Beautiful Mind" type movie about him. Other than playing his record player too loud in his office and having wild martini happy hours with his wife and friends, I can't find anything controversial about him.
Driving around while reading the newspaper and getting into frequent crashes, and advocating for the immediate nuclear annihilation of Russia, are at least 2 other controversial pastimes.