Agreed! While these results are very promising, there's still a lot to explore in this space.
In addition to the "prompt consistency" and "thought-control" ideas mentioned in the post, I'm definitely curious how the performance is on more complex structured data (things like codegen).
It's too bad that reality doesn't really care if you have clever accounting tricks to show that "things aren't really as bad as they look!"
The only number that matters at all is global green house gas emissions. We need to get that to zero, and it has continued to go up.
Imagine you have a drinking problem and your doctor tells you you need to stop drinking alcohol or you will suffer liver failure. You can argue all day about what percentage of your drinks are alcoholic. "Doctor, I drink 3 cans of coke and only 12 G&Ts now, not 1 can of coke and 24 beers like I used to, I went from 4% of my drinks being non-alcoholic to 20%!" If your daily total alcohol consumed keep rising you have a problem.
That's the situation we're in with co2 emissions. There are many clever ways to make it look like everything is fine, but reality doesn't care. If our global CO2 emissions continue to rise we have absolutely no hope.
What if the police asked for the names of everyone who discussed using cannabis in their email exchanges? Would you really argue that in states where cannabis use was illegal that it would be perfectly okay to do this, collect just the names of people who had used mentioned cannabis in email and then visit their homes to question them further about this?
I'm pretty surprised how many people on HN seem to find this acceptable when this is clearly a form of dragnet surveillance.
For a more rigorous, scientific and yet still thoroughly digestible talk I strongly recommend watching Dr. Richard Alley's talk "The Biggest Control Knob: Carbon Dioxide in Earth's Climate History" [0]
As a Bayesian I particularly enjoy Alley's running theme that: while there certainly could be alternative explanations for what is happening, we simply cannot find anything that explains the data better than CO2.
I see many skeptics pointing out tiny holes in the main AGW hypotheses, but the real Bayesian test is "how much better does one hypothesis explain the observed data than then other?". When you put all the pieces of the atmospheric CO2 argument together it seem to explain what we're observing dramatically better than a thousand "...but what about?" that don't fit together into a coherent counter hypothesis.
As an example: Suppose I come home and see my front window broken, my door open and my laptop missing. I assume I have been robbed based on this evidence. You could say "but couldn't the window have been broken by some kid throwing rocks?", "maybe you left the door unlocked and the wind blew it open", "are you absolutely sure you didn't leave your laptop at work?"
While individually each of these counter hypotheses may explain a single event just as well, together they don't work:
P(window broke, door open, laptop missing | robbed) x P(robbed) >> P(window broke, door open, laptop missing | neighbor threw rock, left door unlocked and left laptop at work) x P(neighbor threw rock & left door unlocked & left laptop at work).
At a high level, compilers just translate one programming language to another. A key part of this translation is the Abstract Syntax Tree (AST), which represents the programming language transformed into a tree of computation independent of the syntax of the language. Once you have the AST, you can then step through it and translate the tree to anther language like java bytecode, ASM, CIL, etc.
When compiling the AST sits in the middle of the whole process. The first part deals with parsing your language into the AST the second part deals with transforming the AST to the target language (including possible optimizations of the tree).
Lisp is effectively the raw AST, which is where its power comes from. The use of parentheses is the cleanest way to directly represent a raw tree that you can interact with. This means that you can use Lisp to cut the language design process in half from either direction:
On the one hand, if you are worried about parsing your language, then you can simply transform it into lisp, which is virtually identical to creating the AST, and then you can use a lisp interpreter/compiler for the second half.
On the other hand, if you're interested in writing the interpreter/compiler for a language and don't want to stress about parsing, you can write it for lisp and not have to worry about parsing a complex language. If you follow the Norvig code you can extend that example to a language devoid of parenthesis by writing the parser for it.
Even if you want to do both it's not a terrible idea to prototype both halves using Lisp and then perform the minimal work pull out the Lisp code and replace it with some other implementation of the AST.
Part II of Skiena's wonderful "The Algorithm Design Manual" is basically a 300 page, high-level overview of a wide range of advanced data structures/algorithms, practical insight into when to use them, and advice on which implementations to use.
Even if you already have and are happy with CLRS, Skiena's book is a great addition to your library.
If you really want them to write "well-tested and modular algorithmic code" then you should teach them how to write packages in RStudio. Then require everything submitted to be in RMarkdown.
I mentor learning data scientists and my advice is always to start using RMarkdown as soon they're remotely comfortable with RStudio. Not only does it avoid issues with an easily polluted global namespace, but more importantly encourages literate programming from the early stages. In stats/data science literate programming is vital to having any idea what you were working on a few months ago. It also makes writing reports much, much easier.
RStudio makes it pretty easy to put together R packages, and the package structure for R does a great job of enforcing proper documentation and testing. Sourcing R files should primarily be used to quickly play around with ideas, or for exploratory data analysis that doesn't fit well inside an RMarkdown document. Any code you intended on reusing between projects should end up in a local package.
I do think it's a problem that R has no intermediate method of organizing code like simple modules in Python. But this means if you're serious about writing clean R, you just have to bite the bullet and teach students to write packages.
I worked at BBN (Bolt, Beranek and Newman) from 2006-2008 and was fortunate enough to have a one-on-one phone call with Leo while I worked there. At that time he was in his early 90s, and was still amazingly sharp and full of energy. Even in a brief phone call his curiosity and kindness left a long lasting impact.
BBN was (and presumably still is, though now part of Raytheon) an amazing company to work for even 50 years after its heyday as one of the original contractors on the ARPANET project. I remember getting to meet Ray Tomlinson (who sadly passed this March) and a wide range of others who were instrumental in the early days of the internet. Seeing what was happening in CS research at the time was pivotal for me changing my career towards computer science.
One of my favorite anecdotes about the early years of the company: As mentioned in another comment, BBN started as an acoustics firm. While JCR Licklider was there briefly in the early 60s he got the company to purchase a computer. This was expensive and somewhat out of the scope of the company, when asked why purchase such the thing the response was “this company is full of smart people, they’ll figure something out”. A few years later Licklider was a PM at ARPA in charge of the ARPANET project which BBN would soon become the lead contractor on.
Leo definitely lived to a ripe old age, but he will be missed nonetheless.
This statement is a misunderstanding of the Bayesian approach:
"Take the proposition that the Earth goes round the Sun. It either does or it doesn’t, so it’s hard to see how we could pick a probability for this statement."
Bayes Factor, the Bayesian alternative to a NHST, is quite a bit different than simply creating the Bayesian equivalent of a t-test. Bayes factor asks "How many times better is my Hypothesis at explaining the data than an alternate Hypothesis". So the Bayesian statement would first pit one model of the Earth's orbit against another. The Bayesian statement of the question of the Earth's orbit would be:
"How much more likely is the astronomical data we've observed to have happened given that the Earth revolves around the sun than it is if the sun revolved around the Earth."
For a more concrete example let's suppose that we have a coin. I think the coin has only heads and you think it is a fair coin, with a 50/50 chance of getting heads or tails. We observe three heads in a row. My hypothesis says that the probability of getting 3 heads in a row given a trick coin is 1. Your hypothesis says that the probability of getting 3 heads in a row given a fair coin is 0.5 x 0.5 x 0.5 = 0.125. My hypothesis explains the data 1/0.125 = 8 times better than your hypothesis. Now suppose the next flip is a tail. The probability of HHHT in my model is 0 and yours is 0.5 x 0.5 x 0.5 x 0.5 = 0.0625. You're hypothesis explains the model infinitely better than mine!
Now we can say that our new Hypothesis is that the coin is fair. Suppose another friend comes along and claims that they thought the coin had a 75% chance of getting heads and only a 25% chance of tails. We flip the coin 5 more times and get HHTTH. Your hypothesis says 0.5^5 = 0.03125, and the friend's says 0.75^3 x 0.25^2 = 0.0263... You're hypothesis explains the data only 1.2 times better than theirs. Clearly, we need more data to feel really confident in one hypothesis over the other.
If you want an even longer example, I wrote a post awhile back about "Bayesian Reasoning in the Twilight Zone" that goes into more detail (including priors)[0]
For anyone interested, I noticed that Dyalog made their proprietary implementation of APL free for non-commercial use just recently [0]. I suspect this change might have been in reaction to an uptick in traffic to their site from this other APL HN post[1] a bit more than a month ago (there's some good discussion their as well).
The best answer to this is the success of JavaScript. JavaScript historically has been a deeply flawed language. However, it was able to become one of the most widely used languages precisely because, despite all its warts, it implements the core foundations from the lambda calculus correctly (lambda functions, first class functions and closures). This allowed JS programmers enormous power to overcome these warts. Here are two examples:
JavaScript historically has no way to do namespacing. In most languages this would be a deal breaker. But because JavaScript has lambda functions, closures and first class functions a solution could be crafted from scratch! Immediately-Invoked Function Expressions (IIFE) were one of the most powerful early techniques to create scopes on the fly in JavaScript. Without these tools from the lambda calculus you would normally have to rely on language level changes to allow these problems to be fixed.
The other early challenge of JavaScript was the need for asynchronous callbacks. Often these require passing around data that you might not have access to when a function is written. Lambda functions allow programmers to quickly write ad hoc logic. First class functions allow programmers to pass around this logic. And most importantly closures allow you to create functions on the fly based on data that is not known until an asynchronous callback is applied. Again without the core ideas of the lambda calculus in place this type of power would require significant language-level design changes.
10 years ago JavaScript was a hideous language, with many major issues. But because in this mess was contained the power of the lambda calculus the language was able to be salvaged and extended to a wide range of uses.
The Neural Network Playground is great for understanding this[0]!
The default example is classification of a circle of one class surrounded by a donut of another. There are two features x_1 and x_2 (this is the "raw data").
One solution to this problem is to use a single layer and a single neuron but engineer features manually. These manually engineered features are x_1*x_2, x_1^2,x_2^2, sin(x_1) and sin(x_2). Here's a link to this model (long url)[1].
This model performs very well at learning to classify the data just by combining these manual features with a single neuron. The problem is a human needs to figure out these features. Try removing some and observe the different performance given different manual features. You'll see how important it is to engineer the correct ones.
Alternatively you can have 2 layers of 4 neurons [2]. In nearly the exact number of iterations this network also learns to classify the data correctly. This is because the non-linear interactions between neurons are actually transforming the inputs the appropriate ways. That is to say the networks is learning to engineer the features itself. Try removing layers/nodes and you'll find that a simpler network will have a harder and harder time at this.
I recommend playing around with the various tradeoff between manually engineered features and network complexity. The interesting thing you will observe is that in some cases the manual features are much faster to learn a simplier model than the network. The big issues comes up when we can't simply "see" the problem in 2d so we have no idea what features may and may not be useful.
Even worse there are people (utter frauds of course) who confuse MAD and MAD! In all seriousness, there is much confusion between Median Absolute Deviation and Mean Absolute Deviation out there. Ironically the MAD in this article is still not a robust measure of variation in data as it will break for the many distributions that have undefined/infinite mean (Cauchy and Levy as examples).
Even then many summary statistics rely on a well-defined PDF which is also not true for many real life cases. I think most data scientists out there are very familiar with quantiles, which are often more useful as all random variables have a CDF (and the quantile is just the inverse CDF).
I quite enjoy Taleb's writing (I tend to find his ego a bit amusing) but I think even he is guilty of Jaynes' "Mind Projection Fallacy"[0] in regards searching for more meaning than exists in Fat-tailed distributions. When we model our data with infinite/undefined mean and variance distributions we're just saying "I don't know". No amount of cleverness with summary statistics, or understanding of pathological distributions will create information where there is none.
The overall point being: there are many, many ways of viewing statistics and it's pretty trivial to find a perspective that allows you to call someone a "fraud". Sure there are actual frauds in data science, but one of the biggest strengths in this trend is bringing quantitative people from a wide range of backgrounds to gain refreshing insights. It is much more useful to encourage cross-discipline exploration than to simply say "you don't belong here".
A single layer autoencoder with n nodes is equivalent to doing PCA and taking the first n principal components. If you're familiar with PCA in natural language processing, which is called Latent Semantic Analysis (or Indexing), projecting high dimensional data on a lower dimensional surface can actually improve your features. This is because similar words will project onto the same Principal component allowing you to model some semantic information.
Autoencoders with more than 1 layer are more interesting because you end up doing what is essentially non-linear PCA by projecting your data onto a curved manifold. This famous paper, "Reducing the Dimensionality of Data with Neural Networks" [0], by Hinton shows the improvement in how linearly separable documents become once multi-layer autoencoders are used.
The old argument was that unsupervised pretraining helps get proper weights faster, but this has largely been disproven. However, I do believe AEs assist in semi-supervised learning because they project the initial data into a more useful space. As you can seen in the article I linked the projected data are much more linearly separable.
And as a practical evidence: I used a 5 layer AE in the kaggle black box competition [1] to eventually outrank of team of Hinton's grad students. The problem had a larger unsupervised data set with a small number of labels. Using the autoencoders before the MLP ended up nearly doubling our team's score.
A very useful visual to add next to this is "The Most Common Job In Every State"[0] map from NPR's Planet Money. As soon as I was aware of the research in autonomous driving I was pretty confident trucking would be the first industry hit. Until I saw that map I had no sense of how major of a disruption this could be.
Geoffrey Hinton's "Neural Networks for Machine Learning" on Coursera [0] is a pretty excellent course to cover the basics. There's a lot in the course, but if you just skim over the videos you'll get a pretty good "big picture" view of what's out there. As with any quantitive topic, it's best to take a first pass where you just glance at the math and come back later to really focus on the missing pieces
An important thing to realize is that much of deep learning is decades old neural networks research that has for one reason or another become more viable recently.
If you really want to dive into the technical details there's really no better book than Hull's "Options, Futures, and Other Derivatives". It's extremely well written and if you have a basic understanding of calculus and probability the math isn't too difficult. The only catch is that it is a very expensive book, but if you buy a used copy a few editions back it is more affordable. Also don't worry about the "derivatives" subject matter, if you want to understand derivatives you naturally have to understand the underlying instrument. If you just read the first 100 or so pages (covering Futures pricing) you'll have a pretty good sense of the basics of thinking about financial markets.
I really recommend this book even if you're not interested in Finance as a general guide for thinking about stochastic processes in a practical manner. Nearly all "basic" business/web metrics can be understood best if you understand how to correctly model financial instruments. Personally, I think the basics of quantitative finance are just as relevant to Data Science as machine learning is.
Here's an interview question I always ask that has worked pretty well:
"If you could wave a wand and instantly change one thing about this company/job/team, what would it be?"
This is similar to "what is wrong" but frames it in a positive light, so people are more open and creative.
If the answer is anything about people "I wish communication was better", "It would help if more people were on board for this project", "A change in management wouldn't hurt, haha j/k" etc. That's a red flag.
If it's about non-people "I wish we didn't have so much legacy code", "I would love it if we could get our testing setup better", "There are no good places to get coffee around" that's a good sign that aren't major people problems.
If they can't think of one, that's a real cause for concern!
This is one of my favorite questions in general because what people wish for tells a lot in many ways about the major problems, but without people begin guarded. They're fantasizing not venting.
As far as I can tell Wall Street has always had a penchant for pedigree in addition to skills. Large companies are often the same. People used to talk about all software engineers needing CS degrees in the future. Again in Wall Street and large companies you'll see this, but there are still plenty of well paid, talented software developers with no formal CS training.
In my experience hiring and chatting with other people hiring data scientists, there's the same trouble as there is with software engineers. No matter how many people have the training there's still a dearth of applicants that are truly talented and can actually do things. PhDs fleeing academia for a promise of easy employment and money are a huge bulk of new data scientists that I've seen and most of them have a very hard time taking deep knowledge and applying it to solve real-world problems.
At least in tech I think the future of Data Science lies in the perpetually small group of people that will have a proven track record of coming into companies and actually solving problems, just as it has been in software development.
"There are only two kinds of languages: the ones people complain about and the ones nobody uses."
-- Bjarne Stroustrup
I've seen many programming languages become popular (or fail to) over the years and this quote has always seemed to hold true.
With Scala there was the first wave of people working in it that claimed that it was a grand panacea for all the problems in software (as is always the case with new languages).
However, it wasn't until people started to really claim that the language was certainly doomed that it clearly was a success.
In general I have found that Stroustrups quote is, counter-intuitively, a good way to determine whether the next hot new language will really stick or not. Furthermore, I have to admit that even some of my favorite programming languages fail this test and honestly these languages are extremely unlikely to ever achieve mainstream success.
And if you really think about it, it's not so counter-intuitive. Programming languages don't show their real limitations until you are very deep in a large complicated project. The frustrations of the beginner are never the same as the frustrations of an expert and only an expert can really feel that a language is "doomed". This sense of "doom" is often just the realization of the once language X zealot who now sees that this new language is not a true panacea. But this moment of disillusionment is also the moment an idealized programming language has proven itself a practical one. The more people that feel this loss of faith in their favorite new language, the more people are building large, practical, real-world software projects with it.
In addition to the "prompt consistency" and "thought-control" ideas mentioned in the post, I'm definitely curious how the performance is on more complex structured data (things like codegen).