Ask HN: Small sample size user testing fallacy
3 comments
Depends on the range of Android versions you support and what your app does. I'm not an Android dev but I'm fairly certain if you all you want to do is display some text it'll probably work fine across most versions of Android. If you are using some bleeding edge API you may have some issues.
The "lets just release it" strategy isn't terrible. It's a trade off between your dev time of finding bugs pre-release and having users find them and what the impact is. If you take money you should make sure that process work on any version of your app that supports it. If the payment screen has some weird visual bug but functionally it works fine then it's probably okay for users to find.
Cover your core cases on all versions you support, things outside of that can probably be passed on to users safely.
The "lets just release it" strategy isn't terrible. It's a trade off between your dev time of finding bugs pre-release and having users find them and what the impact is. If you take money you should make sure that process work on any version of your app that supports it. If the payment screen has some weird visual bug but functionally it works fine then it's probably okay for users to find.
Cover your core cases on all versions you support, things outside of that can probably be passed on to users safely.
Depends what kind of testing you’re talking about. Tech / does it work testing should be pretty easy to accomplish regardless of users/phones available. UX testing is a lot harder and probably better to do a combo of “just release” with recruiting testers.
If your code fails on 100% of devices, testing with 1 device will catch it.
If your code fails with 99% of devices, testing with 5 different devices will almost certainly catch it.
Etc.
Which is to say, you won't catch obscure bugs, but you'll catch wide-spread bugs. And wide-spread bugs are exactly the ones you want to catch before general release.
If your code fails with 99% of devices, testing with 5 different devices will almost certainly catch it.
Etc.
Which is to say, you won't catch obscure bugs, but you'll catch wide-spread bugs. And wide-spread bugs are exactly the ones you want to catch before general release.
My friend's argument goes like this: If we cannot test our code with big enough representative sample of Android phones, then let's just release the code and collect usage statistics from real users without user testing it at all.
Basically since testing our code with few phones does not prove that our code will work on 99% of the devices so it is as good as not testing it at all.
I think there must be a logical fallacy in this argument, what is it?
I can think of a counter argument that goes: let's assume we get 99% confidence that our code runs on every phone by testing it with 30 devices:
(1/x)^30 = 1 - 99% then x = 1.166
confidence(N) = 1 - (1/1.166)^N
Now with N = 5, I am already more than 50% confident that the code works on every device. But then the question is where does my assumption of confidence(30) = 99% come from in the first place.