Ask HN: How to split traffic in A/B testing?
I've been reading up a lot on A/B testing lately on HN. Since sequential A/B testing is not recommended due to external factors, I'm just curious as to how someone is suppose to split web traffic (say 50/50 without using hardware to balance).
2 comments
The way A/Bingo handles this is to assign every user a unique ID, and for each test take the MD5 hash of their ID concatenated with the test name. That gives you some big ol' integer. Modulo the number of choices gives you the number of the choice to show them. This is extremely fast, durable, and has very little state, which are nice things to have in an A/B testing algorithm.
So that means, that all you need is their UID, and for each subsequent page hit, for a particular test, the test choice to show them is deterministic. But, where does the even distribution of test choices come from? The hash or UID or the combo of the two?
MD5 hashes mod a small number can be assumed to be randomly distributed. Hypothetically assuming I had evidence to the contrary, I would shortly be very famous in some circles. Practical results bears the expectation out.
Which programming language do you use?
Maybe someone has already built a framework for you to use?
I personally used ABingo for Ruby on Rails (http://www.bingocardcreator.com/abingo), Visual Website Optimizer (http://visualwebsiteoptimizer.com) and Google Website Optimizer (http://www.google.com/websiteoptimizer) for any technology.
The advantage of running a technology specific tool is mainly because of performance. Most technology-agnostic tools execute a javascript on your page to be able to execute the A/B testing logic where ABingo for example is executed on the server side (no additional http request)
What generic tools give you though is the ability to change your tests without changing your code.
Maybe someone has already built a framework for you to use?
I personally used ABingo for Ruby on Rails (http://www.bingocardcreator.com/abingo), Visual Website Optimizer (http://visualwebsiteoptimizer.com) and Google Website Optimizer (http://www.google.com/websiteoptimizer) for any technology.
The advantage of running a technology specific tool is mainly because of performance. Most technology-agnostic tools execute a javascript on your page to be able to execute the A/B testing logic where ABingo for example is executed on the server side (no additional http request)
What generic tools give you though is the ability to change your tests without changing your code.