Scalable A/B Experiments at Pinterest(engineering.pinterest.com)
engineering.pinterest.com
Scalable A/B Experiments at Pinterest
http://engineering.pinterest.com/post/95378137929/scalable-a-b-experiments-at-pinterest
2 comments
"We have hundreds of A/B experiments running at anytime" which means you have a very big attribution problem.
I work at Pinterest, and have some involvement in how we learn from experiments.
In each experiment, we assign users randomly based on a hash of experiment name and userid. So, we assume (and have tested) that the groups of one experiment have a fairly uniform mix of users from the groups of any other experiment. This means the primary effect of being in a particular experiment group G should be much larger than the effect of being in G and some other experiment's group. So, we usually analyze the effects of an experiment independent of any other experiments. Millions of users helps here.
I think this is how trials normally work: assume that random assignment into groups will spread all other factors equally across the groups.
There are experiments which depend on each other (change the same UI, one assumes a precondition of another), and that situation is complicated. We try to avoid it when we can, and negotiate carefully when we find ourselves in it.
You might also be referring to the fact that looking at hundreds of things will find "statistically significant" effects simply at random. As my advisor said, "95% confidence means wrong 1 in 20 times." That is a risk to be managed. We always have to ask things like: is this a reasonable outcome for this experiment? Do we have other corroborating evidence? Is the effect consistent? Do we have reason to disbelieve the result? Some fraction of our experiments are run incorrectly or have a broken implementation; we detect some, likely not all.
Also: is the risk higher to run an experiment that may be misinterpreted, or not to experiment at all and just release things? We like to learn from data if we think we can. There is art in the tradeoffs.
In each experiment, we assign users randomly based on a hash of experiment name and userid. So, we assume (and have tested) that the groups of one experiment have a fairly uniform mix of users from the groups of any other experiment. This means the primary effect of being in a particular experiment group G should be much larger than the effect of being in G and some other experiment's group. So, we usually analyze the effects of an experiment independent of any other experiments. Millions of users helps here.
I think this is how trials normally work: assume that random assignment into groups will spread all other factors equally across the groups.
There are experiments which depend on each other (change the same UI, one assumes a precondition of another), and that situation is complicated. We try to avoid it when we can, and negotiate carefully when we find ourselves in it.
You might also be referring to the fact that looking at hundreds of things will find "statistically significant" effects simply at random. As my advisor said, "95% confidence means wrong 1 in 20 times." That is a risk to be managed. We always have to ask things like: is this a reasonable outcome for this experiment? Do we have other corroborating evidence? Is the effect consistent? Do we have reason to disbelieve the result? Some fraction of our experiments are run incorrectly or have a broken implementation; we detect some, likely not all.
Also: is the risk higher to run an experiment that may be misinterpreted, or not to experiment at all and just release things? We like to learn from data if we think we can. There is art in the tradeoffs.
Did you guys look into Bayesian Bandits? They seem much more tradeoff-able.
Using Bayesian methods to learn about our experimental effects is very interesting to me. For example http://www.evanmiller.org/bayesian-ab-testing.html, which may be the type of bandit you refer to.
In the best case, it might be more robust to low-data outliers that cause random blips that look significant, as well as addressing the problem of looking at an experiment over and over (which frequentists are uncomfortable with: more chances to succeed).
However, it is non-trivial to understand how this works for us in practice. Examples: we want overall results as well as days-in results (to look for novelty effects); we would have to choose priors with consideration, because they have a huge impact on most results; etc.
This takes time and effort, and there are a lot of other things competing for those resources.
In the best case, it might be more robust to low-data outliers that cause random blips that look significant, as well as addressing the problem of looking at an experiment over and over (which frequentists are uncomfortable with: more chances to succeed).
However, it is non-trivial to understand how this works for us in practice. Examples: we want overall results as well as days-in results (to look for novelty effects); we would have to choose priors with consideration, because they have a huge impact on most results; etc.
This takes time and effort, and there are a lot of other things competing for those resources.
I work at Pinterest on Growth, so I use our AB framework every day. Most of our experiments have nothing to do with each other – two of our current experiments are "add a button promoting group boards, compare to control" and "test different designs for our typeahead, compare against each other". Pinterest is a pretty big website, so it's unlikely that our experiments affect each other in ways that'd make our tests invalid.
This is an unfortunate over simplification (for both responses). Argument towards invalid data is stating an assumption- "Most of our experiments have nothing to do with each other" < that isn't a scientific method. Its simply an opinion that you are taking no effort to gain clean data - meaning the truth.
If a scenarios exist that can invalidate your findings, then you use every scientific method to avoid them not enable.
"Pinterest is a pretty big website, so it's unlikely that our experiments affect each other in ways that'd make our tests invalid." << This is the very reason why you will have an attribution problem.
If a scenarios exist that can invalidate your findings, then you use every scientific method to avoid them not enable.
"Pinterest is a pretty big website, so it's unlikely that our experiments affect each other in ways that'd make our tests invalid." << This is the very reason why you will have an attribution problem.
Attribution issues aside, there's two scarier potential issues I see, actually I see these with all posts on "here's our company's cool new A/B testing framework", that really scare me:
1. You're running a ton of tests, yet I see no mention of how you're adjusting your tests to account for multiple testing. The more tests you run the higher the chance you have of getting a false positive. Couple this with the majority of things you test probably won't be significantly better, and your chance of encountering a false positive is much much higher than you might think. You're running hundreds of tests and using a p-value of 0.05, but your chances of a false positive in your tests is much much much higher than 5%. Beware multiple tests and especially beware of the base rate fallacy.
2. Your post has no mention of statistical power or the size of the effect you're looking to detect. That makes me think you might not have considered this. If you don't know the effect size you're looking for or your statistical power, your A/B test results can't be trusted -- as you have no idea what your chances are of actually detecting a beneficial result if it exists.
1. You're running a ton of tests, yet I see no mention of how you're adjusting your tests to account for multiple testing. The more tests you run the higher the chance you have of getting a false positive. Couple this with the majority of things you test probably won't be significantly better, and your chance of encountering a false positive is much much higher than you might think. You're running hundreds of tests and using a p-value of 0.05, but your chances of a false positive in your tests is much much much higher than 5%. Beware multiple tests and especially beware of the base rate fallacy.
2. Your post has no mention of statistical power or the size of the effect you're looking to detect. That makes me think you might not have considered this. If you don't know the effect size you're looking for or your statistical power, your A/B test results can't be trusted -- as you have no idea what your chances are of actually detecting a beneficial result if it exists.
These are both good points. I think it's worth calling out here that this post is really about the infrastructure to perform experiments, not necessarily the means of analysis (although the ui you see in the screenshots performs that analysis).
In terms of these issues, we handle them in a few ways. In particular, experiment review catches many of these issues. Think of it like code review for your experiments.
In order to run an experiment, we require you to have an "experiment helper" sign off on your change. This involves reviewing your group sizes, verifying that you have the statistical power you need to test the magnitude of change your hypothesis expects, verifying you interact with the framework correctly etc.. Training to become an experiment helper is generally not very easy, and involves a combination of shadowing existing reviewers, performing enough reviews across the stack, and taking a test to verify you understand potential errors (the test itself being composed of many experiments where we have made mistakes).
Changes to experiments (increasing group sizes, terminating an experiment, modifying an experiment etc.) all require this review.
In terms of these issues, we handle them in a few ways. In particular, experiment review catches many of these issues. Think of it like code review for your experiments.
In order to run an experiment, we require you to have an "experiment helper" sign off on your change. This involves reviewing your group sizes, verifying that you have the statistical power you need to test the magnitude of change your hypothesis expects, verifying you interact with the framework correctly etc.. Training to become an experiment helper is generally not very easy, and involves a combination of shadowing existing reviewers, performing enough reviews across the stack, and taking a test to verify you understand potential errors (the test itself being composed of many experiments where we have made mistakes).
Changes to experiments (increasing group sizes, terminating an experiment, modifying an experiment etc.) all require this review.
1. Beware multiple tests: agreed. We could apply corrections (e.g., Bonferroni-like), but don't underestimate the logistical complication of this. Before a month starts, we don't even know how many experiments will be run (although we could try to predict). A different way to address this problem: use other information. Do the results make sense (good)? Is there corroboratory evidence (good)? Are there crazy outliers or things that smell funny (bad)? Of course the experimenter often has a bias (it worked!). Can they convince others? etc.
2. To compute statistical power, you need an estimate of effect size. For many experiments, we don't know; we could run pilots, but in the naive case that doubles the number of experiments to run, and time to wait. By default, we recommend a sample size that will detect a certain effect size (but not smaller ones). That is, we decide we are not interested in small effects, because it makes our life simpler.
2. To compute statistical power, you need an estimate of effect size. For many experiments, we don't know; we could run pilots, but in the naive case that doubles the number of experiments to run, and time to wait. By default, we recommend a sample size that will detect a certain effect size (but not smaller ones). That is, we decide we are not interested in small effects, because it makes our life simpler.
1) it is true that the more tests you run the higher chance you see one FP, however the tests are not rejected/accepted as a whole. each test will show it's own results.
2) the size effect did take into considerations, there's also r scripts to analyze the results. the impact is trivial
2) the size effect did take into considerations, there's also r scripts to analyze the results. the impact is trivial
+1
Not necessarily.
Tests and metrics might be page-specific. For instance, "How long did Experiment User stay on Page X vs. How long did Control User stay on Page X?" That wouldn't cause any difficulty, assuming at most one experiment per page.
Even if more than one experiment is running per page, or if metrics are more complex (e.g. "Does showing a more prominent image result in more repins of that image over the next seven days?") there are methods that can help isolate the effects of an individual experiment.
Tests and metrics might be page-specific. For instance, "How long did Experiment User stay on Page X vs. How long did Control User stay on Page X?" That wouldn't cause any difficulty, assuming at most one experiment per page.
Even if more than one experiment is running per page, or if metrics are more complex (e.g. "Does showing a more prominent image result in more repins of that image over the next seven days?") there are methods that can help isolate the effects of an individual experiment.
Agreed. The question I'm interested in is, how do we solve that attribution problem?
"There is no persistence layer involved for experiment group allocation, so that we could minimize the latency/load on our production services. We left all of the complicated metric computations to offline data processing." Does that mean that the experiment to which the user is allocated is stored through a log in memory instead of being stored on disk/on disk and in memory (through logs)? If so, are there frequent trade-offs between memory/cpu usage in running experiments?
Putting ("triggering") someone into an experiment group is most often some function calls: do they meet the criteria (e.g., country, ..), do they fall in the right bucket. The fact that they've been triggered is logged to Kafka, an open-source messaging system whose logs we push (eventually) into Hive.