We're pretty sure this is the only analysis of its kind at the moment. You have to have a pretty large number of experiments to be able to do this analysis. Hopefully we see more soon (co-author speaking)
I would suggest joining an early stage start up and getting involved with anything remotely to do with data science at every opportunity.
I joined a small company as an analyst with no programming experience and minimal statistical knowledge. I was a graduate but not in a relevant subject and just taught myself the relevant skills on the side. It was a lot of work but not a waste of time.
The programming side of the job can be learnt fairly quickly but the maths and stats side takes longer. I don't think you can really succeed in data science without both. Saying that, you certainly don't have to have a degree to be able to use that knowledge. I did just do a statistics degree though, and it has made the job a lot more pleasurable.
I've always enjoyed the Jabberwocky by Lewis Carroll. It is full of emotive gibberish that creates wonderful imagery and feeling out of meaningless sounds.
Which is all poetry really is in the end!
’Twas brillig, and the slithy toves
Did gyre and gimble in the wabe:
All mimsy were the borogoves,
And the mome raths outgrabe.
“Beware the Jabberwock, my son!
The jaws that bite, the claws that catch!
Beware the Jubjub bird, and shun
The frumious Bandersnatch!”
He took his vorpal sword in hand;
Long time the manxome foe he sought—
So rested he by the Tumtum tree
And stood awhile in thought.
And, as in uffish thought he stood,
The Jabberwock, with eyes of flame,
Came whiffling through the tulgey wood,
And burbled as it came!
One, two! One, two! And through and through
The vorpal blade went snicker-snack!
He left it dead, and with its head
He went galumphing back.
“And hast thou slain the Jabberwock?
Come to my arms, my beamish boy!
O frabjous day! Callooh! Callay!”
He chortled in his joy.
’Twas brillig, and the slithy toves
Did gyre and gimble in the wabe:
All mimsy were the borogoves,
And the mome raths outgrabe.
Fantastic to see Optimizely changing their stats model. The more education that is done on web experimentation the better as there certainly still is a lot of snake oil being sold out there!
Their chosen technique is one way of solving the problem of communicating statistics to non-technical audiences however the interpretation of the results may suffer here. I can imagine that this technique will lead to overestimations of the effect size in situations where the threshold is reached early in an experiment as it will reward extreme values observed when the experiment is under powered.