1) Agreed that the smaller the effect, the more statistical power (usually from a larger sample size) you need to detect them. But to assume that all changes have tiny effects, and therefore not detectable and a waste of time, is a flawed assumption.
Once upon a time we published over 100 a/b tests here: https://www.goodui.org/evidence/ and clearly the relative effects vary (not all single changes have always a small effect).
More so, the effects of a/b tests can be further increased by grouping multiple higher confidence ideas together into a single variation.
2) Short term gains may (or may not) lead to long term disengagement. Measuring micro (shallow) and macro (deeper) metrics would be the right way to answer this.
Facebook is known to run countless online experiments – most of which are kept secret. Luckily for us we have the Internet Archive that allows us to pry into these possible design optimizations.
Sure. But the idea is to do a quick ice breaker project that can later turn into a larger one down the road. Hence in that case a $500 - $2000 structured project makes sense, I think.
The question I recently have asked myself was this: on these structured or productized pages, should we push the customer to a low friction gradual engagement lead generation action, or close the sale with a payment? I'm now trying out the former. Thoughts?
Once upon a time we published over 100 a/b tests here: https://www.goodui.org/evidence/ and clearly the relative effects vary (not all single changes have always a small effect).
More so, the effects of a/b tests can be further increased by grouping multiple higher confidence ideas together into a single variation.
2) Short term gains may (or may not) lead to long term disengagement. Measuring micro (shallow) and macro (deeper) metrics would be the right way to answer this.