Ask HN: How do you decide which bugs to ship?
6 comments
How popular is the part of the product that is impacted? Do analytics show that it’s not that bad, and that the bug might go unnoticed? Is the bug resulting in an utterly negative experience, or is it something that the user can work around? Might the bug actually be already part of their workflow? Is fixing the bug going to raise more questions than letting it be? And a nice lump of gut feel on top of this.
Basically anything that corrupts data, fails to apply auth controls, or total site outage inducers don’t ship, but almost everything else is fair game (I’m sure I’m forgetting something but you get a taste for these things over time).
When your release cycle is like 10 mins, there’s no real reason to fret over most problems. Obviously do try to write bug free code, but beyond what I listed it’s hardly a fireable offense to push a bug.
The old adage about bugs in prod being 10x more expensive is not true.
When your release cycle is like 10 mins, there’s no real reason to fret over most problems. Obviously do try to write bug free code, but beyond what I listed it’s hardly a fireable offense to push a bug.
The old adage about bugs in prod being 10x more expensive is not true.
The one org I work for that bothers to make any informed decisions about this (the other just lets the Product Owner determine and its arbitrary) looks at customer ARR.
If the bug impacts people who bring in a lot of money, it gets fixed. If the bug impacts some small business on the nearly free plan, that bug is never getting addressed. A business that only provides $2000 a year in revenue is business you cut loose at the first sign of friction.
If the bug impacts people who bring in a lot of money, it gets fixed. If the bug impacts some small business on the nearly free plan, that bug is never getting addressed. A business that only provides $2000 a year in revenue is business you cut loose at the first sign of friction.
Criteria: how easy it is to fix, are there any workarounds available, how disruptive it is to core use cases, is there any immediate business impact, are we planning to work on that part of the system any time soon, is it a new bug or something that's been there for a while, could it corrupt user data, does it cause a compliance issue.
None. Don’t tolerate bugs.
Unless we're talking medical, aerospace, weapon system, financial infrastructure code etc, from my experience this is not achievable.
OPs question implies knowingly shipping defects, or knowingly tolerating known defects. Bugs are not some force of nature, they are created by programmers who don’t test the results of their programming prior to shipping. They persist by programmers choosing not to fix them.
For practical purposes it is achievable to ship working software. Plenty of people do it.
I’ve used Safari, Notes, Reminders, and Calendar daily for years and have never encountered a bug. Same with my time tracking and invoice software. Maybe there are bugs but I have been able to go years without encountering them. I have also never knowingly shipped a bug for my service https://sumi.news, and have fixed every bug brought to my attention. There are zero bugs in my bug tracker, and when they are reported they are fixed.
If you’re discovering more bugs than you can fix, and you always have a long list of bugs, that implies something wrong with how you design, implement, and test software, and an engineering team that is incapable (for whatever reason) of producing working software.
Tolerating defects is little more than apathy and a lack of quality standards and craftsmanship.
For practical purposes it is achievable to ship working software. Plenty of people do it.
I’ve used Safari, Notes, Reminders, and Calendar daily for years and have never encountered a bug. Same with my time tracking and invoice software. Maybe there are bugs but I have been able to go years without encountering them. I have also never knowingly shipped a bug for my service https://sumi.news, and have fixed every bug brought to my attention. There are zero bugs in my bug tracker, and when they are reported they are fixed.
If you’re discovering more bugs than you can fix, and you always have a long list of bugs, that implies something wrong with how you design, implement, and test software, and an engineering team that is incapable (for whatever reason) of producing working software.
Tolerating defects is little more than apathy and a lack of quality standards and craftsmanship.
1. impact in $ terms (to keep things simple, I'm implying a mapping between loss of reputation, loss of opportunity, legal troubles etc. to money): do we lose more money by leaving or removing the bug? Removing a bug might mean disabling a feature which generates income. Removing a bug might mean disabling a feature for the downtime of which we pay contractual penalties. Removing a bug might mean extra effort.
2. workarounds: maybe there is an affordable workaround, eg. running a script periodically which corrects wrong entries in a database or showing users a message how to work around the issue themselves or explaining an odd application behaviour.
3. synergies: maybe there is a good reason to ship a bug because we learn something about the system. Eg. I once took over, rather hastily, a tech lead job where my predecessor left on very short notice. He had built a proprietary, undocumented deployment automation system to work around organisational slowness. When I had to deploy a new version of an application (which was a complex process as it involved several teams and needed to be planned sufficiently ahead of time), I knew deployment would fail, but I chose to do it nonetheless in order to learn more about the deployment system through observing how it behaved during a deployment.