100% test coverage(brandur.org)
brandur.org
100% test coverage
https://brandur.org/fragments/100-percent-coverage
15 comments
I started my career off in the chip design world, where 100% line, branch, FSM transition, functional group, and toggle (i.e. individual bits in a bus switching 0 -> 1 _and_ 1 -> 0) was "table stakes". There are a lot of comments in here saying that achieving 100% coverage would be expensive - and they're right. The majority of headcount in modern chip design houses is taken up by verification engineers, whose sole job is to architect, implement and maintain a minimal test corpus that achieves that high bar. The cost of failure is simply too high to omit this step.
It was unsettling to me after moving to a SWE job where coverage was kind of... not as important.
It was unsettling to me after moving to a SWE job where coverage was kind of... not as important.
I can think of certain types of software that are so critical that they must be 100% thoroughly verified to be logically correct, e.g. software in surgical machines, airplanes, or high-frequency trading. The vast majority of applications, though, would survive with less coverage, though I still think that the sweet spot is somewhere between 70-80%.
I've seen bad codebases rightfully elevated to greatness because of their test suite.
The confidence it adds allows new people to make changes much faster.
The confidence it adds allows new people to make changes much faster.
I've seen great teams grind to a halt because of edicts.
The elevation of process over results is demoralizing to builders.
The elevation of process over results is demoralizing to builders.
Sure, but I've seen enough premadonnas walk away from a tyre fire that fed their ego to build that I don't take their egos too seriously. Stifling process? No, unwavering diligence? Yes.
Sure, but I've seen enough prima donna middle managers who fed their ego by enforcing process to know the pragmatism of builders is less harmful.
I'll take a pragmatist who understands when 94% or 60% is good enough test coverage over the manager who demands 100%. I'm positive I know who has the bigger ego there.
Perhaps you're reading your own experience into this, but I pretty clearly set up a scenario, and yes, I do think there's a lot of ego in anyone demanding 100% coverage, especially if they aren't writing the code.
I'll take a pragmatist who understands when 94% or 60% is good enough test coverage over the manager who demands 100%. I'm positive I know who has the bigger ego there.
Perhaps you're reading your own experience into this, but I pretty clearly set up a scenario, and yes, I do think there's a lot of ego in anyone demanding 100% coverage, especially if they aren't writing the code.
Definitely, I'm happy with anything over 50%.
So, you disagreed with the article, agreed with me, but decided to leave a pithy comment giving the impression I was arguing for the kind of dogmatism the article advocates? Cool. Probably don't do that.
But builder morale is not a product that generates revenue
Neither does process. Product velocity to zero for the sake of a management metric is a good way to kill revenue while getting some managers promoted.
Confidence is only sexy when it’s deserved.
People who jump straight to 100% coverage often cheat, and now we have to manually inspect all of the tests to see if they are actual tests. Sometimes life is easier when you are seeing that there are spots that need more tests because coverage is only 85% in a file.
People who jump straight to 100% coverage often cheat, and now we have to manually inspect all of the tests to see if they are actual tests. Sometimes life is easier when you are seeing that there are spots that need more tests because coverage is only 85% in a file.
Those people are shown out.
Unless you have a lot of resources to do in-depth code review of tests, a good automated mutation testing pipeline, or a dedicated testing team, this kind of behavior can take a very long time to root out (especially if they are more or less competent otherwise).
Nice work if you can get it.
There are at least 3 grades of fudging the numbers that I've encountered. Blatant is only one of them. "Wishful thinking" has many more practitioners.
There are at least 3 grades of fudging the numbers that I've encountered. Blatant is only one of them. "Wishful thinking" has many more practitioners.
%100 test coverage is absolutely essential for dependencies that are used in too many places and can affect too many places if something goes wrong. but it is necessary to decide whether it is really necessary to approach all addictions in this way. what I'm talking about is of course valid for a large codebase. And yes, 100% test coverage does not eliminate the possibility of error :D
> But 100% coverage definitely helps, especially in languages like Ruby where an untaken branch can contain a serious error that’s accepted by the interpreter, but will take down production if hit.
And if you're in a language providing more constraints, the benefits can quickly turn into a cargo cult. This is certainly the mindset of Ruby folks, and such a mindset left a native client team with 0 system devs and all web dev converts who refused to engage with our actual issues and focused only on dogma within 2 years of hiring a manager with that background. Color me skeptical of this applying to all projects.
And if you're in a language providing more constraints, the benefits can quickly turn into a cargo cult. This is certainly the mindset of Ruby folks, and such a mindset left a native client team with 0 system devs and all web dev converts who refused to engage with our actual issues and focused only on dogma within 2 years of hiring a manager with that background. Color me skeptical of this applying to all projects.
I wouldn't worry too much about getting to 100%... With tests the spectrum can be ALWAYS extended to the right side. E.g. you could argue that you should also do mutation testing because 100% coverage doesn't mean tests are sensitive enough to changes in code branches, etc... Once you realize this you never really arrive at 100%
That said having coverage target is more often positive than not having one. Coverage target kind of sends "we care about test suite" message
That said having coverage target is more often positive than not having one. Coverage target kind of sends "we care about test suite" message
I mean, this is the kind of thing I agree with, and would even say I'd trust a project with 90% coverage but no surviving test mutants vs. 100% and lots of mutants.
But in reality I'm usually looking at piles of like... 60-70%ish? Thousands of mutants. Maybe a round-trip fuzz or property test if I'm lucky.
So I keep aiming for 100% and hoping anyone else follows...
But in reality I'm usually looking at piles of like... 60-70%ish? Thousands of mutants. Maybe a round-trip fuzz or property test if I'm lucky.
So I keep aiming for 100% and hoping anyone else follows...
These are also more expensive kinds of tests to setup and run so it's often not feasible, maybe in extreme cases. In a few Rails codebases I tried to mutation test something critical only to see that getting it to work would take more time than worth the effort
How do you get 100% branch coverage in a dynamic language?
Every method dispatch or property access is a branch to an arbitrary piece of code, or a source of an arbitrary error to handle, depending on what object it happens to end up dealing with.
Every method dispatch or property access is a branch to an arbitrary piece of code, or a source of an arbitrary error to handle, depending on what object it happens to end up dealing with.
How do we test for 100% branch coverage on python that covers list comprehensions?
The tradeoff between the cost of many tests and the benefits of them is why I usually use a lower coverage requirement. Something like 80% or sometimes 60% seems to give most of the benefits with less of the cost.
You probably can get to 100% branch test coverage while being able to tie all tests to business requirements. It'll be expensive, though.
I don’t get the 100% branch coverage requirement. If the code is off of ‘master’ why would you have to go and fix it?
Branch coverage means something different. If you have a function with 2 if statements, there are 4 branches. 100% code coverage could be 1 test and would run the code before + after the if statements, and enter each if statement block. 100% branch coverage would test this code 4 times: once without entering either if statement, once only entering the first, once only entering the second, and once entering both.
Branch coverage refers to branch instructions (in the underlying machine code), not a branch in whatever source control system you're using.
I think he's referring to branching in the code (if-then-else blocks).
I was confused also but decided they meant if/else branching
Conditional branches, not repository branches.
> We run one of our projects not only with 100% test coverage, but 100% branch coverage. It’s deeply unpleasant.
well, in my experience 100% line and branch coverage is not enough.
Countless times I see people write tests "to pass", but nothing to make them fail. I speak about null-pointers, invalid dates, strings with garbage, out of bounds indices etc.
Countless times I see people write tests "to pass", but nothing to make them fail. I speak about null-pointers, invalid dates, strings with garbage, out of bounds indices etc.
How do you manage breaking changes from changes in third party APIs and libraries then?
Woof
Woof
quickcheck is eating popcorn with 3d glasses on.
Also, I don't think I need to tell anyone here but "100% branch coverage" still doesn't mean "100% branch combination coverage", which would quickly make your test code base grow to the size of the universe anyway.