The Importance of Sequential Testing(austinrochford.com)
austinrochford.com
The Importance of Sequential Testing
http://www.austinrochford.com/posts/2014-01-01-intro-to-sequential-testing.html
2 comments
Haha, yes, I am the author. I do have a very formal style, an unfortunate product of too many years in grad school.
The SPRT is a nice toy test, but only useful for point hypotheses. There are generalizations to more realistic composite hypotheses, though.
The SPRT is a nice toy test, but only useful for point hypotheses. There are generalizations to more realistic composite hypotheses, though.
It would be interesting to me if you covered the generalizations. This field seems to have evolved entirely separately from the banditry I'm accustomed to.
I'll put a follow up on some other sequential tests into my list of blog-posts-to-do and let you know when it happens.
A heads up to the author: with narrower browser window sizes (e.g. less than ~1024px), the left side menu overlaps the text.
Just noticed that on my iPad, will look into it.
Good stuff, and just one of the many reasons that most A/B testing practice is flawed. I haven't seen the SPRT used much in practice. I'm not sure why. A quick Googling indicates non-parametric sequential tests have been developed. What about the case with more than two alternatives?
My field is bandit algorithms, which are a generalisation of hypothesis tests. Here there are algorithms like Empirical Bernstein Stopping (http://icml2008.cs.helsinki.fi/papers/523.pdf) which achieve the same early stopping from a different approach. If this is of interest sign up to the course I'm running (http://noelwelsh.com/data/2013/08/16/free-bandit-course-draf...) -- I plan to cover this algorithm soon.
BTW, you have a very formal writing style. I think if you're trying for a more general audience you should loosen up a bit. This ain't a journal paper. :-)