The Most Radical A/B Test I've Ever Done(kalzumeus.com)
kalzumeus.com
The Most Radical A/B Test I've Ever Done
http://www.kalzumeus.com/2010/06/04/the-most-radical-ab-test-ive-ever-done/
9 comments
Many teachers return each September to a computer that has been wiped clean.
I hear that. Frequently, in September, from several dozen teachers asking me "How do I reinstall Bingo Card Creator on my computer which was just wiped clean? Also, do I have to pay?"
I think focusing on the web app will mitigate some of the anxiety that teachers experience in that area.
It totally solved that problem, where the feature I made in the downloadable version to do the same thing has been a catastrophic flop. (Twenty people have ever used it, and I still get emails asking to explain how to copy files onto a floppy so that they can be read on the machine at school. And then emails the next day saying that the computer doesn't know what a .bcf file is. And then emails the next day saying that they installed Bingo Card Creator and opened the file but it says it is the free trial which is odd because they know they purchased the program and they never had this issue at home [i.e. where they followed directions and put in their Registration Key].)
I hear that. Frequently, in September, from several dozen teachers asking me "How do I reinstall Bingo Card Creator on my computer which was just wiped clean? Also, do I have to pay?"
I think focusing on the web app will mitigate some of the anxiety that teachers experience in that area.
It totally solved that problem, where the feature I made in the downloadable version to do the same thing has been a catastrophic flop. (Twenty people have ever used it, and I still get emails asking to explain how to copy files onto a floppy so that they can be read on the machine at school. And then emails the next day saying that the computer doesn't know what a .bcf file is. And then emails the next day saying that they installed Bingo Card Creator and opened the file but it says it is the free trial which is odd because they know they purchased the program and they never had this issue at home [i.e. where they followed directions and put in their Registration Key].)
Teachers here in the UK have it really bad; a lot of the IT infrastructure is farmed out to BT or other completely useless corporates. Either what happens is they can't cope and so are dropped after a few months or they keep having to replace all the poorly configured laptops.
My Mum has been through about 8 new laptops in the last 14 months...
(the best one is when the laptop leads all went missing [never found out who did it] and their response was... ...to replace the laptops. rolls eyes)
And lets not even start to talk about the restrictions imposed on these laptops.... (I mean; I can understand it for the kids, but the teachers?)
My Mum has been through about 8 new laptops in the last 14 months...
(the best one is when the laptop leads all went missing [never found out who did it] and their response was... ...to replace the laptops. rolls eyes)
And lets not even start to talk about the restrictions imposed on these laptops.... (I mean; I can understand it for the kids, but the teachers?)
As someone who works in information security at a very large district, I will say this is a brilliant decision! =)
The last thing I need is school personnel downloading Java apps. Not that yours is malicious - but once someone learns how to install/run something, it becomes exponentially more difficult to teach them to identify what is malicious and what is not.
+1 for syncing between work/school. Often the computers at schools are inferior to ones that teachers have a home, or at least a different platform.
The last thing I need is school personnel downloading Java apps. Not that yours is malicious - but once someone learns how to install/run something, it becomes exponentially more difficult to teach them to identify what is malicious and what is not.
+1 for syncing between work/school. Often the computers at schools are inferior to ones that teachers have a home, or at least a different platform.
My first thoughts are that removing the downloadable version will work out well by getting rid of the "Tyranny of Choice" and thus helping with conversions.
Has anyone else tried this approach?
Has anyone else tried this approach?
And I thought the post _had_ the results of this test :)
Anyway, interesting test and do keep us posted about the results.
Anyway, interesting test and do keep us posted about the results.
"How do I copy the files to my niece’s computer? By the way it is a Mac and I use a Yahoo. Try using the web application."
This gem from the article really illustrates the added difficulty of supporting a desktop application vs a web application. For this particular request what OS she is running doesn't matter, but for many other requests it could be very important to know. Given the technical sophistication of this user, just figuring out what environment the application is running in could be an arduous process.
This gem from the article really illustrates the added difficulty of supporting a desktop application vs a web application. For this particular request what OS she is running doesn't matter, but for many other requests it could be very important to know. Given the technical sophistication of this user, just figuring out what environment the application is running in could be an arduous process.
Very interesting.
My initial thought (unencumbered by any actual knowledge) is that the folks who download the client version (instead of sign up for the web version) don't do so for any particular reason, so I don't think that removing the option should actually hurt.
Sounds like a good fit for Bayesian probability, using the old numbers as priors.
My initial thought (unencumbered by any actual knowledge) is that the folks who download the client version (instead of sign up for the web version) don't do so for any particular reason, so I don't think that removing the option should actually hurt.
Sounds like a good fit for Bayesian probability, using the old numbers as priors.
Downloaded app can be used offline.
But the question is: is that really important to the majority of the users who download it? Or are they acting blindly?
Now, it could be that there is a contingent of bingo-loving teachers who absolutely prefer to have the software locally installed, but I'm suspecting (without any good evidence, I grant) that this is not really a key decision-factor for most of Patrick's customers who choose to download.
Now, it could be that there is a contingent of bingo-loving teachers who absolutely prefer to have the software locally installed, but I'm suspecting (without any good evidence, I grant) that this is not really a key decision-factor for most of Patrick's customers who choose to download.
I believed for years that there was a group of customers who would pay for downloadable versions when they wouldn't pay for web versions. In fact, I put AdWords ads on many free web-based competitors, and those are my best performing ads anywhere.
But I've come to believe, after listening to customers and seeing the behavior in tests where I emphasize or deemphasize the options, that this doesn't say that users want to use downloadable applications. It was just that users want easy to use applications that don't suck. Next to the free web apps in 2006, BCC downloadable was easy to use and didn't suck. Next to BCC downloadable, BCC online is easy to use and doesn't suck.
But I've come to believe, after listening to customers and seeing the behavior in tests where I emphasize or deemphasize the options, that this doesn't say that users want to use downloadable applications. It was just that users want easy to use applications that don't suck. Next to the free web apps in 2006, BCC downloadable was easy to use and didn't suck. Next to BCC downloadable, BCC online is easy to use and doesn't suck.
I'm not paying to use the web version, but if it's written in HTML5 it can leverage the cache-manifest (or Gears, either way) and work offline as well.
Do you know one way or another that it isn't?
Do you know one way or another that it isn't?
Unless HTML5 has some previously undisclosed way of running ImageMagick and Prawn, I'm going to go out on a limb and say that this will not work offline that well.
Isn't it odd to announce the A/B test when you've just started it ?
I mean, I don't think HN readers are in the bingo cards market, but still, it feels like he might be influencing the results by blogging about this beforehand.
I mean, I don't think HN readers are in the bingo cards market, but still, it feels like he might be influencing the results by blogging about this beforehand.
If I woke up in the morning and had a shocking lack of worries, between two software products and client work I could probably find something real to worry about without having to manufacture worries out of the ether.
"Several dozen HN users will go to the website, see which variation they get, clear their cookies until they get the right one, then pay me $30... and then cackle madly" is not a real worry.
"Several dozen HN users will go to the website, see which variation they get, clear their cookies until they get the right one, then pay me $30... and then cackle madly" is not a real worry.
Again, I don't think blogging about this will influence your A/B test, because your (future) customers and people reading your blog don't overlap very much (or at all).
But, I do think publishing the A/B test does influence the outcome if they do overlap.
You are talking about the $30 that you could get but miss the point: you might get some extra money but in the process corrupt the validity of your A/B test and make a bad decision afterwards, based on that result. Well, not that it matters anyhow, as I'm pretty sure you have actually decided already to stop doing the Java app.
I would like to have somebody that knows more about sociology and statistical surveys explain if what I am saying makes sense or not.
But, I do think publishing the A/B test does influence the outcome if they do overlap.
You are talking about the $30 that you could get but miss the point: you might get some extra money but in the process corrupt the validity of your A/B test and make a bad decision afterwards, based on that result. Well, not that it matters anyhow, as I'm pretty sure you have actually decided already to stop doing the Java app.
I would like to have somebody that knows more about sociology and statistical surveys explain if what I am saying makes sense or not.
The point of the A/B test is that, once devised, it is purely statistics and the software (from here on in) will tell patio11 which of the two options to adopt.
[deleted]
I'd be surprised if teachers didn't check out your blog before buying. Teachers "get" blogs, hell they LOVE blogs. They even get their students to blog (to build writing skills using peer interaction). Seems like the ones who are confident enough to go out and use your software may also be the ones who would check out the blog. Just a guess on my part, but at least keep an eye out for this.
The chances of his userbase reading his tech blog and being impacted by something like this seems pretty small to me.
Somewhat agree with you. I recently blogged about why you shouldn't publicize that you are doing an A/B test http://visualwebsiteoptimizer.com/split-testing-blog/3-contr...
Though as you say, the target market and HN crowd isn't exactly overlapping but there may be curious folks who can skew the results or the difference of performance between variations.
Though as you say, the target market and HN crowd isn't exactly overlapping but there may be curious folks who can skew the results or the difference of performance between variations.
In what way would this influence things? Sure, it might draw some additional uninterested visitors, but that will just depress conversion rates all around, not exclusively for A or B. The results remain valid.
Assuming no actual possible customer reads this blog, this doesn't influences the number of new customers, but it does changes the overall numbers.
But, if people know they are being A/B tested, this allows them to make a decision about what they think they prefer.
Sometimes what people think and are willing to vote (say, by complaining they don't like their option, or by buying a product to support it) isn't what they would have actually picked.
But, if people know they are being A/B tested, this allows them to make a decision about what they think they prefer.
Sometimes what people think and are willing to vote (say, by complaining they don't like their option, or by buying a product to support it) isn't what they would have actually picked.
Yes but that changes the real difference between conversion rates of A and B. Even though trend won't get affected but actual numbers will. Not that serious but given that this problem is easily avoidable one may want to not announce the test.
I love this article. And his article on why he's done making desktop software is even better:
http://www.kalzumeus.com/2009/09/05/desktop-aps-versus-web-a...
http://www.kalzumeus.com/2009/09/05/desktop-aps-versus-web-a...
I've just searched google for:
Bingo cards Generator
Printable bingo cards
Create bingo cards
Bingo Creator
In all of them you are not one of the top 3 results. It might suggest that a better approach would be to work on your SEO than to increase your ad money.
Bingo cards Generator
Printable bingo cards
Create bingo cards
Bingo Creator
In all of them you are not one of the top 3 results. It might suggest that a better approach would be to work on your SEO than to increase your ad money.
Enough of this uninteresting Bingo Card Creating program. How do these posts keep getting pushed up to the first page?
I just wish I had enough rep to down vote your comment :P His posts and complete transparency about growing his business epitomize the type of community contribution that brings me back to this site time and time again.
printing bingo cards is a business any 12 year old with a computer could set up. Couldn't you do this on Print Shop and an Apple IIe? Most tech stuff posted on HN is the cutting edge of web apps. And if it wasn't, I wouldn't read this site multiple times per day. Sure, his writing skills and analysis are decent/good, but the content he writes about is laughable. He should spend some of his blogging energy on developing a more interesting topic/business to blog about.
I will feed the troll:
He should spend some of his blogging energy on developing a more interesting topic/business to blog about.
Patio11 is not your dancing monkey.
Sure, his writing skills and analysis are decent/good, but the content he writes about is laughable.
Laughing all the way to the bank.
Most tech stuff posted on HN is the cutting edge of web apps. And if it wasn't, I wouldn't read this site multiple times per day.
Well, can't say we'd miss you if you stopped contributing, but I'd have to take issue with the first part. At this moment, the front page has 30 entries of course. I can't believe I'm doing this, but they are about Flash, SproutRobot, Falcon 9, bcvi, HTML5, ideas, diamonds, deflate, BigPipe, Falcon9 (again), Titan, YourWorldOfText, fivethirtyeight.com, being wrong, Toxoplasma, Soros, Readability, Google IO, Posterous, Steve Jobs, Wakemate, Amelie(), the FTC, patio11's a/b test, chrome dev tools, twitter employees leaving, Apple and HTML5, Bing cashback, a world cup calendar and John Gruber. I'll leave it to you to determine if "most tech stuff on HN is the cutting edge of web apps." I tend to think that most business stuff on HN is there to help educate someone to run their business, generally technical in nature.
Finally, there's printing bingo cards is a business any 12 year old with a computer could set up.
Why haven't you, then?
He should spend some of his blogging energy on developing a more interesting topic/business to blog about.
Patio11 is not your dancing monkey.
Sure, his writing skills and analysis are decent/good, but the content he writes about is laughable.
Laughing all the way to the bank.
Most tech stuff posted on HN is the cutting edge of web apps. And if it wasn't, I wouldn't read this site multiple times per day.
Well, can't say we'd miss you if you stopped contributing, but I'd have to take issue with the first part. At this moment, the front page has 30 entries of course. I can't believe I'm doing this, but they are about Flash, SproutRobot, Falcon 9, bcvi, HTML5, ideas, diamonds, deflate, BigPipe, Falcon9 (again), Titan, YourWorldOfText, fivethirtyeight.com, being wrong, Toxoplasma, Soros, Readability, Google IO, Posterous, Steve Jobs, Wakemate, Amelie(), the FTC, patio11's a/b test, chrome dev tools, twitter employees leaving, Apple and HTML5, Bing cashback, a world cup calendar and John Gruber. I'll leave it to you to determine if "most tech stuff on HN is the cutting edge of web apps." I tend to think that most business stuff on HN is there to help educate someone to run their business, generally technical in nature.
Finally, there's printing bingo cards is a business any 12 year old with a computer could set up.
Why haven't you, then?
"printing bingo cards is a business any 12 year old with a computer could set up."
LOL. Comments like this show that you'd be better off reading and learning more about what constitutes a business on those (x - 1) re-visits to this site each day - here we have one of the (relatively) few successful businesses that we get to read in-depth about (i.e., actually making money) and you're complaining about the lack of WizzBang Widget 14.9 or TooToot2000 tech fad coverage.
LOL. Comments like this show that you'd be better off reading and learning more about what constitutes a business on those (x - 1) re-visits to this site each day - here we have one of the (relatively) few successful businesses that we get to read in-depth about (i.e., actually making money) and you're complaining about the lack of WizzBang Widget 14.9 or TooToot2000 tech fad coverage.
Desktop applications are increasingly difficult to manage on a school computer. Many teachers return each September to a computer that has been wiped clean.
At the same time, the internet is becoming more and more accessible in schools. Teachers will be able to use the web application no matter which room they are in or which computers they have access to.
Another challenge for non-tech savvy teachers is syncing work between the school and home. I think focusing on the web app will mitigate some of the anxiety that teachers experience in that area.