[untitled]

3 comments

mrdmnd12 anni fa

I've been working on this problem as well - the idea is to come up with a "dynamic decision" boundary that allows you to have a "virtual p-value" for different points in time that is equivalent to the 5% p-value at the end of a standard experiment. For example, maybe you stop any test with a p-value less than 0.05 after 30 days, but you require a p-value less than 0.01 for 10 < d <= 30 and less than 0.001 for 1 < d <= 10.

Here are some sources on how to go about doing that:

http://people.bath.ac.uk/mascj/papers/cj_bwt_kuwait.pdf

http://people.bath.ac.uk/mascj/talks_2013/cj_york_may2013.pd...

pneumatics12 anni fa

This is an important lesson, but I was hoping for an implementation of stopping rules. If you're looking to actually stop your N-armed bandit trials properly, look at the S+ package SeqTrial. It implements the stopping rule framework described in KittelsonEmerson99 (http://www.ncbi.nlm.nih.gov/pubmed/11315020).

raviparikh12 anni fa

Thanks! Stopping rules is probably something I'll look at in a later post.

Also, building in stopping rules is something Optimizely should consider.