So since Optimizely basically lets the test run until it "reaches statistical significance" with their Chance to Beat Baseline number, would you say they are doing it wrong?
I've often run tests that seem to be in the 90%+ chance to beat baseline, but the graphs just didn't look like they were finished or had enough data, so I'd let it run a bit longer. Sure enough, big changes in percent lift, and also big changes in confidence dropping down below 90% (sometimes MUCH lower), and occasionally coming back up.
Not the parent, but I managed and eventually helped lead the paid search group at one of the top search agencies. We had clients with 7 figure budgets all the way up to 9 figure budgets (or essentially unlimited as long as we were within certain parameters).
It's a bit scary and heart attack inducing if something goes majorly wrong (like going dark for a day because someone forgot to check the flight dates in the account after you took it over from the previous agency). But like the OP said, it just becomes numbers.
The cool thing is you not only get much better support and escalation paths at Google, but you can test things at a very large scale and get answers VERY quickly. And little things that might not make a huge difference on smaller accounts, suddenly equate to tens or hundreds of thousands of dollars difference in total revenue. But again, all things are relative, so %'s might not change much.
There's a big one missing from this list that is somewhat recent.
Selling the placement of audience segmentation/retargeting cookies on a publisher's site to a data aggregator like Bluekai (typically on a CPM bases) who then sells that aggregated data to DSPs/DMPs for use in display advertising.
If you want to learn about a whole different layer of the ad tech industry, just start perusing some of the questions on Quora for Bluekai (http://www.quora.com/BlueKai) and then do some digging on the companies in the Data Aggregator and Data Supplier buckets of the LUMAscape for display advertising (http://www.lumapartners.com/lumascapes/display-ad-tech-lumas...).
While I love your site, I have to respectfully disagree with much of what you are saying here.
First, I agree on one main point--your site is not a "startup" or even really a "company" at this point. That said, you still very much have a product, and you are actively monetizing it.
But then let's get into this point by point...
You say that your ads are "non-targeted" but that is not at all accurate. Just because you aren't doing anything actively to target it, doesn't mean Google isn't. Google's AdSense network has a TON of targeting on the back-end that is determining what ads to serve. This can be based on the profile of the user, retargeting lists they may be a part of, the category of site Google has you categorized as (possibly health?) etc. Your ads are VERY targeted. That is actually part of the beauty of the AdSense product.
The second point I really take issue with is that you don't seem to think your site falls into the "if you're not paying for it, you're the product" bucket. Or at least that's how I parsed what you wrote there (apologies if I was off on that).
People aren't paying for your product, and so you are making money by serving them ads. By the nature of it, any ad-supported site falls into that bucket. You require more user impressions/eye balls and that is what you are "selling" to advertisers/Google. This is not inherently evil, and I don't think one needs to prey on their users to leverage advertising. That doesn't mean the saying doesn't apply to them though.
Again, I like what you're doing and think the single banner is a tactful approach (although you'd probably make a lot more by recommending sleep/health-related products and using affiliate links). But those two things you said just aren't accurate so I wanted to clarify on those a bit.
Is it really that big of an issue to let some site accurately track the source of their referral traffic? That's all UTM codes do by the way--it isn't anything nefarious.
I always check them out with lots of hope, but then am dismayed to find that they are usually entirely filled with technical roles and almost completely devoid of any marketing/product/business roles.
This is likely in part due to the HN audience composition, but my hunch is that its also partly because people are making assumptions about the HN audience composition.
I'd love to see one post for technical hiring and one post for business hiring. I think calling it out as its own post would actually encourage people to post those other jobs that they might normally think are not appropriate for the HN audience.
Not sure if you saw it, but one of his concerns is the time debt that essentially binds him to the will of his investors if he were to take their money.
IMHO this is a very valid concern for a "lifestyle" business, if that is indeed what he intends it to be.
Methodology here: https://help.optimizely.com/hc/en-us/articles/200039895-Chan...
I've often run tests that seem to be in the 90%+ chance to beat baseline, but the graphs just didn't look like they were finished or had enough data, so I'd let it run a bit longer. Sure enough, big changes in percent lift, and also big changes in confidence dropping down below 90% (sometimes MUCH lower), and occasionally coming back up.