Looking at just the script tag (eg: mbox.js/cdn.optimizely) and taking it as signal doesn't provide accurate results especially in case of providers who provide the option of self-hosting.
I will send you an email shortly with the methods that i used and let me know what you think.
Having a robust A/B, MVT testing infrastructure that lets you test on page components is the most important thing but it is only a small part of the equation. I don't have a comparison chart of Optimizely vs T&T features but there are some common feature sets and every provider lacks one or the other. Having used Optimizely, i would say Optimizely has about 30-40% of features that other enterprise level products have (personal guess not based on metrics/facts!). Some areas where the smaller products lack features are
- Personalization, segmentation and targeting (I am not talking about basic geo-targeting, repeat users, technographics etc). The features in this category can go way deep.
- Industry specific features (Having a one product fits all approach will be hard to sell in many industries. Eg: There is a huge difference in how retail and publishing uses a testing product)
- Reports and analytics (every product provides charts and graphs but this is one area where you can keep building and you will still be lacking some thing or the other)
- Upsell features (recommendation engines, mobile, social)
Many of these additional features are hard/time consuming to build and will not make sense for a good percentage of users but having those will help the product to score in the enterprise land. After a certain point when you have all the basic features, identifying what to build is probably one of the hardest tasks.
I guess you are thinking Visual Web Optimizer. No, I am not a part of VWO and also not an employee of a company that is a direct competitor to Optimizely. Adding to what user 'litek' mentioned, there are at least a dozen well known competitors in the A/B, MVT space.
I am a regular contributor on HN but I work in this industry and so i am using a throwaway account.
Dan,
1.3 billion 'visitors' is a really vague representation. Is it user sessions or page views or number of unique visitors measured in one of the experiment/control groups?
Test & Target mbox hell as mentioned in the post is very very true. Learning to use it within one hour is a joke. The technology they use is really stale and hasn't changed a lot in the recent years. Many of the players who started after 2008/09 have far superior implementations, thanks to lot of advancements in browser technologies which T&T failed to capitalize.
I am very happy for Optimizely's success and these two posts will bring them a good amount of well deserved PR. Test & target and in fact a good number of other providers are way ahead of Optimizely in this game. The number of sessions/experiments is one good metric but it is definitely not the best one to measure the leader in the race.
* What is optimziely's ARR?
At 2800 customers paying $250/month [1], it should be around $8.5M. Lets be generous and make it $10M. A handful of T&T's high profile clients will be paying Adobe that amount.
* How many IR100, IR500 [2], top 100 travel, publishing, finance sites are in Optimizely's client list?
Builtwith's data is not very accurate. We have worked with the guy from builtwith to compute some lists and their methodologies are not very great. They just look for signals in the script tags and it throws lot of false positives. Out of my own interest, i have spent hours writing parsers and scripts to compile the list and Optimizely is not really in that picture.
You can have thousands of clients paying you hundreds of dollars or have a few hundred clients paying 5/6 figure amounts. Optimizely has been after the former while the other players including T&T have been targeting the latter. I guess Optimizely is now trying to expand their enterprise client list since that is where the big money is but it takes a lot more to compete in that arena with long sales cycles and vast requirements.
* How many user sessions are being served?
Requests is probably the worst measure. T&T mentioned it because they can show their artificially inflated metric which is a result of their terrible implementation. IMO, user sessions is a better measure. 1.3 billion user sessions (in 2 years)[3] is still no where close to what the other providers serve. Number of tests currently running is also a good measure.
Optimizely is a great product and they have made it super easy to run A/B tests but false proclamation claiming you are number one when you are not even close is not very nice.
[1] Optimizely probably has a good number of clients in that 2800+ who pay $10k-20K a month. The highest openly priced plan is $360. I guess it wouldn't be wrong to assume that 50% of the customers will be in one of the two lower plans ($17 or $71). Optimizely's current ARR would be somewhere between $5M to $10M (My guess!)
I will send you an email shortly with the methods that i used and let me know what you think.