Ask HN: What is your one tip about A/B testing you would share?
Wondering what everyone would tell someone new to a/b testing for website and conversion rate optimization. Could be anything from expectations to approach, etc. Thanks!
9 comments
That most A/B tests don't prove what they aim to prove. People just add a new version of the site, then wait until 100 people have seen it, measure a small improvement and migrate to the new version.
However if you then run the test a second time, but make the "old" site the alternative, you might find it too sees a small measure of improvement and thus you could bounce back and forth between A and B until the end of time.
I am not a statistician so I won't try to give you advice on what a statistically significant result is. However many /many/ articles have been written on that topic, and many products have been designed which you can slot your data into to see if the 'B' option is legitimately better.
However if you then run the test a second time, but make the "old" site the alternative, you might find it too sees a small measure of improvement and thus you could bounce back and forth between A and B until the end of time.
I am not a statistician so I won't try to give you advice on what a statistically significant result is. However many /many/ articles have been written on that topic, and many products have been designed which you can slot your data into to see if the 'B' option is legitimately better.
Learn what "statistical power" (http://en.wikipedia.org/wiki/Statistical_power) means, and then understand that the fact n% of users prefer one option doesn't necessarily make it a good idea. Statistical significance is far more important than the actual result. This is a really good paper about it: http://www.qubitproducts.com/sites/default/files/pdf/most_wi...
Before you get too far, try an A/A test. By this, I mean let the A and B choices be identical. You would certainly expect the outcomes to be equal. Right?
I have seen statistically significant differences in outcomes in A/A testing.
A/B testing has value but being sure to A/A test may temper your expectations and/or point at problems in your setup before you get too far.
I have seen statistically significant differences in outcomes in A/A testing.
A/B testing has value but being sure to A/A test may temper your expectations and/or point at problems in your setup before you get too far.
http://nginx.org/en/docs/http/ngx_http_upstream_module.html#... - to make clients 'stick' to upstreams, if you're using nginx.
If you radically change a feature then regular users will first start playing with it. Simply because it's new. In that case your test needs be longer or exclude regular users.
Ensure that your testing things that are actually going to impact your conversion rate or goals. You don't have to test everything - just the things that matter.
Don't A/B test your credibility. Not everything should be (or can be) subjected to hypothesis testing.
Choose and test one independent variable at a time. Then, check your results for statistical significance.
Kittens will always improve conversion rates.