Mike Jordan at Berkeley sent me his list on what people should learn for ML. The list is definitely on the more rigorous side (ie aimed at more researchers than practitioners), but going through these books (along with the requisite programming experience) is a useful, if not painful, exercise.
I personally think that everyone in machine learning should be
(completely) familiar with essentially all of the material in the
following intermediate-level statistics book:
1.)
Casella, G. and Berger, R.L. (2001).
"Statistical Inference"
Duxbury Press.
For a slightly more advanced book that's quite clear on mathematical
techniques, the following book is quite good:
2.)
Ferguson, T. (1996).
"A Course in Large Sample Theory"
Chapman & Hall/CRC.
You'll need to learn something about asymptotics at some point, and
a good starting place is:
3.)
Lehmann, E. (2004).
"Elements of Large-Sample Theory"
Springer.
Those are all frequentist books. You should also read something
Bayesian:
4.)
Gelman, A. et al. (2003).
"Bayesian Data Analysis"
Chapman & Hall/CRC.
and you should start to read about Bayesian computation:
5.)
Robert, C. and Casella, G. (2005).
"Monte Carlo Statistical Methods"
Springer.
On the probability front, a good intermediate text is:
6.)
Grimmett, G. and Stirzaker, D. (2001).
"Probability and Random Processes"
Oxford.
At a more advanced level, a very good text is the following:
7.)
Pollard, D. (2001).
"A User's Guide to Measure Theoretic Probability"
Cambridge.
The standard advanced textbook is
Durrett, R. (2005).
"Probability: Theory and Examples"
Duxbury.
Machine learning research also reposes on optimization theory.
A good starting book on linear optimization that will prepare
you for convex optimization:
8.)
Bertsimas, D. and Tsitsiklis, J. (1997).
"Introduction to Linear Optimization"
Athena.
And then you can graduate to:
9.)
Boyd, S. and Vandenberghe, L. (2004).
"Convex Optimization"
Cambridge.
Getting a full understanding of algorithmic linear algebra is
also important. At some point you should feel familiar with
most of the material in
10.)
Golub, G., and Van Loan, C. (1996).
"Matrix Computations"
Johns Hopkins.
It's good to know some information theory. The classic is:
11.)
Cover, T. and Thomas, J.
"Elements of Information Theory"
Wiley.
Finally, if you want to start to learn some more abstract math,
you might want to start to learn some functional analysis (if you
haven't already). Functional analysis is essentially linear algebra
in infinite dimensions, and it's necessary for kernel methods, for
nonparametric Bayesian methods, and for various other topics.
Here's a book that I find very readable:
12.)
Kreyszig, E. (1989).
"Introductory Functional Analysis with Applications"
Wiley.
I've been pretty good friends with one of the band members of VW since college, and I don't think the comments stem from a "too-cool-for-school" mindset.
I remember one of my nerd friends and I talked about our approach for a poker bot to this member of VW, and he was extremely interested in the idea, even though he didn't have a technical background.
The thing that really bothers him, though, is "inauthenticity", and having a mainstream music band member play a web entrepreneur would probably elicit jeers.
(Sorry for the incoherent rant that starts now) I've built high-end loudspeakers for about 9 years, and I'm of two minds when reading this article. While it's nice for a mainstreamish outlet to describe the experience of listening to a high-end audio system, there exists so much dis/misinformation in this article that it makes my skin crawl. So I will attempt to explain what I think high-end audio is (or should be) all about without all the journalistic BS (both from the gizmodo and the stereophile guy).
The purpose of a high-end audio system is to create music that is indistinguishable from live sound. This is a really hard thing to do and my guess is at best, we are only 20% of the way there. But, given that most consumer-level stuff is probably at 0.42%, listening to a good high-end audio system is generally nothing short of incredible. The thing that really gets people when they first listen to the "good stuff" is that on a good high-end audio system, there will be sounds that don't seem to be coming from the speakers themselves, but from somewhere else in the room. The good stuff really is better, but you really can spend a ton of money and get shitty products (to be honest, I've listened to the $65k MAXX3's featured in the article and its more expensive sibling, the $125k X1s, and I thought they were decent speakers, but certainly not worth more than $8k).
The problem with high-end audio, especially from a business perspective is two-fold. 1) How do you make better speakers than your competitors if the best you can do is 20% and you don't even know how to get to that 80%? and 2) How are you supposed to differentiate yourself from your competitors when the end goal is that every single speaker is supposed to sound exactly the same?
Enter the magazines Stereophile and The Absolute Sound. These journalists with "golden ears" will purportedly tell us mere mortals what differences there are between speakers. The ones who do the best (like the Wilson Audio's featured in the article) will also get the most money.
So, every manufacturer will push to get the best reviews from these magazines, and will even go to a sort of soft corruption to garner a strong review. During the review process, an equipment manufacturer will send his/her component to the reviewer for a few months so the reviewer can demo it in his/her own demo room. In an ideal world the reviewer will review the product accurately and send it back. What actually happens is that the "demo" unit will become available to the reviewer indefinitely without a need to return the product. The reviewer, however, already has a component in his own listening room, so why does he want another one? Well, because the review can sell it and make some extra money on used equipment (30% of a 10k CD player is still a lot of money). If the review pans the product, though, then the resale price will be far less than if the reviewer praises the product. So voila, instant good review. (If you read stereophile for a few months, you will see every other issue that the reviewer has found the best component he has ever listened to).
This sort of practice seems to have been going on for at least a couple decades. Instead of trying to solve the insanely difficult problem of making speakers sound better, high end audio manufacturers spend their money on branding and shenanigans with reviewers. Their equipment (more for speakers than say CD players) will always be better than consumer grade stuff because no one has worked on the problem of making a better speaker and consumer grade speakers are made to be manufactured as cheaply as possible. The high end audio industry is a real mess because everyone forgot why there are here in the first place.
Btw, it would be interesting if the less financially strapped members started designing speakers (you're all smart enough to do as good a job as Wilson Audio). The drivers for these speakers are available to people like us, because driver manufacturers do not do enough volume to stop selling to the DIY community. For instance, the drivers for the MAXXs break down roughly as follows: ~$150/ea for the Focal tweeters, ~$175/ea for Scanspeak midranges, ~$350/ea for the smaller Focal woofer, and ~$500 for the bigger one. While that ~$3000 for the drivers alone, that still a lot less than $65k (and besides, you could actually do better with a lot less money).
Having done my undergrad at an Ivy League institution, I can say without a doubt that everything mentioned in the article is absolutely true. I would even go so far as to say that most of my peers were for the most part semi-competent (on good days), driven individuals with either the skill and nurturing to game the system. The result is a fairly disappointing group of people who care more about a Wall St./Big Consulting job, but honestly, admissions officials care little to change the system, so this is the result. That said, however, many of the undergrads I have TAed for at Berkeley have the same mindset, with the exception that the Wall St. job is now a job at Microsoft, Google, Yahoo, etc.
To a certain extent, we shouldn't care about Ivy League institutions, because in the end, they are just a name, right? What is sad, though, is that there are brilliant professors at these universities having to teach these pathetic students, and not those who actually care and have some skill in a particular subject. What is sad is that parents, employers and institutions themselves proclaim that these students are the best and the brightest and we believe them. What is sad is that the resources of the Ivy League institutions are widening compared to strong public/private universities, but the "elite" institutions are not giving these resources to those who can best take advantage of them (I think what should have been more carefully scrutinized was Harvard President Drew Faust's comments a few months ago that Ivy League institutions will have the best scientific research because they have the most money/best profs, and that other institutions with fewer resources should work on "smaller problems" - the most disgusting thing I have heard in a long time). In short, the whole system is powerful, yet completely broken.
If, by some luck of lottery (at least in my case), you are someone who really cares in this "elite" environment, once you get past the disappointment of not relating to your peers, you have a great opportunity to do research for professors who are not only brilliant, but also very helpful since you are one of very few who care. I just hope that the many others like me who did not win the lottery get the same chances I do. I know at Berkeley, the grad students are given tremendous opportunities with respect to research (with probably only slightly more money concerns than places like Stanford/MIT), but it's very tough for an undergrad to get the same opportunities. I know I wouldn't have survived as an ugrad here, but I'm not sure if other institutions have similar environments.