I often have the situation where a design discussion could easily be settled by A/B testing, only there aren't enough events to get statistically valid results in a reasonable amount of time.
My initial reaction is that this is a sign that we should either focus on getting more traffic or that if we can't measure a difference we shouldn't care either way. However this doesn't really wash in low volume / high value businesses, where a single extra sale means $10-20K.
So how do you handle it? I'd like to avoid reverting to long discussions where everyone is speculating about user behavior with no way of validating the outcome.
My initial reaction is that this is a sign that we should either focus on getting more traffic or that if we can't measure a difference we shouldn't care either way. However this doesn't really wash in low volume / high value businesses, where a single extra sale means $10-20K.
So how do you handle it? I'd like to avoid reverting to long discussions where everyone is speculating about user behavior with no way of validating the outcome.