Triple Your Google AdWords CTR Overnight by Doing Nothing(seobook.com)
seobook.com
Triple Your Google AdWords CTR Overnight by Doing Nothing
http://www.seobook.com/archives/002026.shtml
This is a very curious phenomenon. Same everything, yet vastly different click through rate. Have any of you tried this?
9 comments
From 3 years ago but it certainly still holds true. As mentioned in the comments, many PPC'ers use http://www.splittester.com to help determine if there's enough data to determine a winner.
SEOBook recently opened another site focused primarily on PPC at PPCBlog.com and this was a recent discussion in the forums.
SEOBook recently opened another site focused primarily on PPC at PPCBlog.com and this was a recent discussion in the forums.
It's worth noting that splittester.com calculates 95% confidence that those two (identical) ads will have different long term response rates.
Which should be a pretty big wake up call to anyone not doing the math. 1 in 20 events happen all the time, yet naively reading the results makes it look like one is almost certainly better than the other.
Agreed, this is such a common problem. Another really common manifestation is when you A/B test while looking at multiple variables. If you observe the results of 20 variables, you should expect one of them to have 95% confidence!
Sure because 95% leaves 5% of room to cherry pick aberrations.
Pretty kludgey way of adding statistical significance.
I would expect anyone on HN would be capable of running the numbers and generating the actual stats.
If not, it would be a good learning exercise.
I would expect anyone on HN would be capable of running the numbers and generating the actual stats.
If not, it would be a good learning exercise.
Kludgey is right; and it won't always be accurate: numbers might happen to agree on a too-small sample; or diverge on a large-enough sample.
But I think it's a very simple, concrete and credible demonstration to users. And if they observe such an expt over time, over a few trials, they'll quickly develop an intuition for when the sample size is large enough. And this is more valuable than an abstract calculation. (Of course, ideal to have both.)
Unfortunately, the empirical approach doesn't take into account the effect of how near the probability is to 50% vs the extremes 0% or 100%; nor of the variance of the population. However, in practice, these are probably similar enough over all ads and populations for the effect to be negligible - especially if you give yourself a margin of safety.
But I think it's a very simple, concrete and credible demonstration to users. And if they observe such an expt over time, over a few trials, they'll quickly develop an intuition for when the sample size is large enough. And this is more valuable than an abstract calculation. (Of course, ideal to have both.)
Unfortunately, the empirical approach doesn't take into account the effect of how near the probability is to 50% vs the extremes 0% or 100%; nor of the variance of the population. However, in practice, these are probably similar enough over all ads and populations for the effect to be negligible - especially if you give yourself a margin of safety.
I hacked together a gtest calculator in ruby if anyone is interested. I keep meaning to clean it up, but I haven't had time.
http://github.com/zachaysan/Split-Test-with-the-G-Test
http://github.com/zachaysan/Split-Test-with-the-G-Test
Can't hurt to try this approach, as it takes approximately 10 seconds to set it up. Will give it a shot, and report back if the results are interesting...