Good point. Trials work well in some cases, not in others. For example, for products focused on startups, trials can be very effective. For products purchased by people other than those that use them, trials simply aren't used. So...knowing if purchasers = users is important.
Hi, I'm one of the creators of http://www.abtests.com. The issue of statistical significance has come up over and over, so I'll try to explain our view of it.
We ask people to input their raw data...both trials and conversions. If they do this honestly (anybody can fake data about anything) then in our view the results speak for themselves. We've had folks upload data that was obviously not statistically significant, and we've had people write blog posts denouncing those results. We've also had folks upload test data that was statistically significant and people say they're learning a lot.
So we've had both solid and suspect data uploaded to the site with good discussion around it. This is exactly what we hoped for...I think in the future as more tests get uploaded the wheat will be separated from the chaff, so to speak, and those tests with significant data will get lots more attention than those that don't. In fact, we're already seeing this in the traffic logs.
And, as several folks have mentioned, many tools do the hard stats math for you, telling you when your data is statistically significant. This helps people know when they can be confident in sharing their data with others.
Hey folks, thanks for the feedback. The reason why I titled the piece "slow erosion" is that this will take years...I know how valuable Google is for fact-based searches. But slowly, social search and in-context advertising will gain, not because Google will get worse, but because people will be starting from a different place. They'll start on Twitter or Facebook.
The key insight is that people trust others for certain types of information such as recommendations. A single recommendation from a friend is much more powerful than a list of restaurants Google will give you. "Hey, I know you and I know you will like this restaurant". You could argue that Google will eventually know a lot about us (they do already), but the fact is that social interaction trumps reference material in a lot of cases.
So this will probably take years, and many of the arguments against are that right now Google is better. No denying that...but have you seen the ads on Facebook lately? They're pretty stinky, but twice as good as they were even months ago.