How Much Optimization Is The Right Amount?(training.kalzumeus.com)
training.kalzumeus.com
How Much Optimization Is The Right Amount?
https://training.kalzumeus.com/newsletters/archive/premature_optimization_postmature_optimization
10 comments
That's true, but there's still a valid reason to test things on small traffic: you don't necessarily need to have statistical certainty on every decision. A lot of the time, it's sufficient to know that when comparing two dramatically different options, you can't quickly and easily distinguish between them. That's a strong hint that you should put your future efforts elsewhere. It makes no sense to dedicate lots of effort to a project if the impact is statistically indistinguishable from noise.
Moreover, the changes that matter early on are the 10x changes, and they'll often be subjectively visible to you, even if the math shrugs and tells you that it isn't confident. So while it probably isn't worth testing button copy or images on 100 visitors a day, it might be worth testing complete redesigns, or entirely different value propositions.
Moreover, the changes that matter early on are the 10x changes, and they'll often be subjectively visible to you, even if the math shrugs and tells you that it isn't confident. So while it probably isn't worth testing button copy or images on 100 visitors a day, it might be worth testing complete redesigns, or entirely different value propositions.
10x changes are hard to come by, even at an early stage. That's like going from 3% to 30% conversion. If your site is mostly broken and one change takes you to 30% conversion, then you probably didn't need an a-b test to figure it out.
More realistically, when you're an early stage startup, is that 10x means going from 0.1% to 1% conversion. That kind of change can be achieved by changing pricing models, changing product descriptions, changing the purchase flow, etc. And you'll notice it -- 1% means getting one buyer about every other day on 50 uniques; 0.1% means getting a buyer every three weeks. You won't need statistical validation to tell you that things have changed.
But even "smaller" changes -- 10%, 20%, 30% -- are usually noticeable, even with a tiny stream of traffic. The point is, the kinds of winners that you need at an early stage are usually big enough to feel. Rarely do you get to the win in single-percentage increments.
But even "smaller" changes -- 10%, 20%, 30% -- are usually noticeable, even with a tiny stream of traffic. The point is, the kinds of winners that you need at an early stage are usually big enough to feel. Rarely do you get to the win in single-percentage increments.
As a point of context, I'm guessing this blog post was based a little bit on the "20 days as a growth hacker" submission (and Patrick's subsequent comment): https://news.ycombinator.com/item?id=6883357
That did make me tie a string around my finger, but there's no particular line in the article that you should read as me referring to him.
The editorial calendar basically dictates that I have to talk about A/B testing for the time being (product launch and all), and I generally prefer to say new things rather than saying old things. I've written an awful lot about A/B testing from different angles, but have rarely said "Actually, hold up, don't do it."
I previously had, if anything, a little too much "All I have is a HammerFactory so all problems must implement INail or they will be recast until they do" going on due to my own business history. I've talked to many people who have that to varying degrees over the years.
So "Hmm, as I come to be a little more mature in my understanding of this, there's actually some times where I wouldn't reach for scalable, metrics-driven, quantitative conversion optimization as the first solution" seemed like a pretty good skeleton of a piece.
The editorial calendar basically dictates that I have to talk about A/B testing for the time being (product launch and all), and I generally prefer to say new things rather than saying old things. I've written an awful lot about A/B testing from different angles, but have rarely said "Actually, hold up, don't do it."
I previously had, if anything, a little too much "All I have is a HammerFactory so all problems must implement INail or they will be recast until they do" going on due to my own business history. I've talked to many people who have that to varying degrees over the years.
So "Hmm, as I come to be a little more mature in my understanding of this, there's actually some times where I wouldn't reach for scalable, metrics-driven, quantitative conversion optimization as the first solution" seemed like a pretty good skeleton of a piece.
Finally Patrick seriously addressed the topic of when A/B testing is NOT needed.
Good article. I especially liked the section on how to get your first user for a more serious business app and how you should treat them. That fleshed out a lot of what I was already thinking about how to go about launching the app that I'm working on.
Simple: optimize the amount of optimization, and you're good to go!
Lots of good points there, one of my favorites is the Collison Close: “Open up your laptop. I'll get you an account and code the integration right now.”
Title reads: How to optimize your optimization.
Now what I'd really like to know is, how much optimization of my optimization is the right amount?
Jokes asides, good read!
Now what I'd really like to know is, how much optimization of my optimization is the right amount?
Jokes asides, good read!
Premature optimization of your optimization is the root of the root of all evil.
You're doing it all wrong! You're only optimizing for the local maxima of your optimization's optimization!
As always your stuff is great to read and very insightful! Thanks again Patrick!
relevant xkcd: http://xkcd.com/1205/
Everybody else: There's another reason to "not optimize yet" in many cases: If your traffic doesn't support A/B testing. Taking the example of 100 visitors/day of whom 2.5/day sign up for a free trial -- you're going to have anywhere from 0 to 5 signups/day on a regular basis. If could increase your yield from 2.5% to 3.5%, you should be thrilled -- but an A/B test would take over a month to distinguish that from a random fluctuation.