Google's Chief Economist Hal Varian: Computer Mediated Transactions [pdf](sims.berkeley.edu)
sims.berkeley.edu
Google's Chief Economist Hal Varian: Computer Mediated Transactions [pdf]
http://www.sims.berkeley.edu/~hal/Papers/2010/cmt.pdf
4 comments
For me the most interesting data point was this paragraph on page 10:
In 2008, Google ran 6,000 experiments involving web search which resulted in 450–500 changes in the system. Some of these were experiments with the user interface, some were basic changes to the algorithm (Hoff (2009)). The ads team at Google ran a similar number of experiments, tweaking everything from the background color of the ads, to the spacing between the ads and search results, to the underlying ranking algorithm.
That's over a dozen A/B tests per day. For each product line.
In 2008, Google ran 6,000 experiments involving web search which resulted in 450–500 changes in the system. Some of these were experiments with the user interface, some were basic changes to the algorithm (Hoff (2009)). The ads team at Google ran a similar number of experiments, tweaking everything from the background color of the ads, to the spacing between the ads and search results, to the underlying ranking algorithm.
That's over a dozen A/B tests per day. For each product line.
Greg Linden's take on the same:
http://glinden.blogspot.com/2010/02/personalization-and-diff...
Prof. Varian lectured on this topic at the School of Information at UC Berkeley in October, 2009. The audio is here -
http://www.ischool.berkeley.edu/newsandevents/events/dls2009...
http://www.ischool.berkeley.edu/newsandevents/events/dls2009...
Heck, you can do all of the above in your spare time.