Ask HN: What is the cost of unit test coverage in complex C++ projects
3 comments
I think your grammar is excellent and would never have guessed that English is not your native language.
Code coverage is a good metric to get people to start writing tests, but not a good metric to focus on. People may feel encouraged when their tests increase coverage from say, say 20% to 25%, so they write more tests. Though increasing coverage is not the end goal. For example, in Java you have setters and getters and writing tests for such methods just to increase coverage is useless busywork.
Instead, the goal is for tests to prevent bugs from sneaking into the software. Since many bugs only manifest themselves in strange special cases, tests that prevent such bugs must trigger those special cases. Suppose the website crashes if two users who share the same ip refreshes the page at a special stage in the signup flow. A test to prevent that bug must setup the same circumstances and emulate the same user interactions. Before the bug is fixed the test will fail, afterwards it will succeed. Such tests are incredibly useful but also takes lots of effort to write.
Code coverage is a good metric to get people to start writing tests, but not a good metric to focus on. People may feel encouraged when their tests increase coverage from say, say 20% to 25%, so they write more tests. Though increasing coverage is not the end goal. For example, in Java you have setters and getters and writing tests for such methods just to increase coverage is useless busywork.
Instead, the goal is for tests to prevent bugs from sneaking into the software. Since many bugs only manifest themselves in strange special cases, tests that prevent such bugs must trigger those special cases. Suppose the website crashes if two users who share the same ip refreshes the page at a special stage in the signup flow. A test to prevent that bug must setup the same circumstances and emulate the same user interactions. Before the bug is fixed the test will fail, afterwards it will succeed. Such tests are incredibly useful but also takes lots of effort to write.
> Some people involved in the project are talking about "100% unit test coverage" as a target.
100% targets ignore that software engineering is different from programming: costs in engineering hours, resource invested, maintenance burden etc. all should lower the target.
Also think hard if unit test is suitable for your program. Most compilers aren't unit tested. Distributed system unit testing requires an absurd mock infrastructure.
> The project include some fault-tolerant distributed servers, proxies and various API proxies/converters
Make the unit test target ~20% and focus rest of the time you would've spent on writing unit tests to making the system debuggable: all the best distributed systems are extremely debuggable, since there are no unit tests that will help you once there's an RPC (such things subvert any and all invariants you programmed)
- https://rbcs-us.com/documents/Why-Most-Unit-Testing-is-Waste... also p. fun read
100% targets ignore that software engineering is different from programming: costs in engineering hours, resource invested, maintenance burden etc. all should lower the target.
Also think hard if unit test is suitable for your program. Most compilers aren't unit tested. Distributed system unit testing requires an absurd mock infrastructure.
> The project include some fault-tolerant distributed servers, proxies and various API proxies/converters
Make the unit test target ~20% and focus rest of the time you would've spent on writing unit tests to making the system debuggable: all the best distributed systems are extremely debuggable, since there are no unit tests that will help you once there's an RPC (such things subvert any and all invariants you programmed)
- https://rbcs-us.com/documents/Why-Most-Unit-Testing-is-Waste... also p. fun read
Thank you! Very interesting article.
When I see the word fault-tolerant, C++ is not the best choice.
Ada or, to a lesser extent, Rust remains at the forefront of fault-tolerant.
The rule of thumb for test suite is 4x the code-line being tested, way more test unit code-lines if in C++.
Ada or, to a lesser extent, Rust remains at the forefront of fault-tolerant.
The rule of thumb for test suite is 4x the code-line being tested, way more test unit code-lines if in C++.
Articles and discussions on the net about the right balance for code coverage in unit tests are plentiful. But I have not found much relevant material about the actual cost of targeting say 50%, 70%, 90% or 99% unit test coverage. And it looks to me that most of the numbers regards interactive web pages and mobile apps. Not the more complex technology beneath. I'm not opposing testing - I think testing is great. I just want a realistic idea about the actual cost of various code coverage percentages in projects of some complexity.
My experience from smaller open source projects in modern C++ that I've written from scratch is that "proper" unit testing adds about 60 - 100% coding overhead (designing and writing the tests, and also being mindful about testing while designing and writing the actual code). I usually only test the public methods on an interface, unless there are some special reasons to test or access private methods. I try to avoid mocking. The aim of the unit test is to validate that the algorithm in a public method works as documented. Since some of these methods are interpreters with potentially indefinite variations in input, I'm testing for valid and invalid input, but only with a few workloads to test specific paths and error conditions in the code. The tests will typically catch logical regressions, but not security vulnerabilities or performance regressions. I have not used code coverage tools, but my gut feeling is that I typically cover < 70% of my code in my small open source projects with unit tests.
All projects of this nature are unique, and increasingly asynchronous and parallel algorithms does, in my experience (may be I'm doing something wrong), increase unit testing complexity and the time needed to design the code and write the tests. In my opinion, it's not acceptable to introduce performance regressions in hot paths in the code to facilitate testing (for example by using virtual classes in stead of templates).
So, with this language; C++ 17 (no, re-writing millions of lines of C++ code in Rust or Ruby is not an option right now) and this complexity (parallel and asynchronous algorithms in a fault-tolerant distributed system with massive disk/network IO and capacity for billions of low latency external requests per day) - what could the cost-curve for unit test coverage / coding time look like, from 50% to 99% code coverage?
Some people involved in the project are talking about "100% unit test coverage" as a target.
Note that I only focus on unit testing here. That's because the other forms of testing in this project are done by different teams, and don't have a direct impact on the time it takes to get a feature from a specification document to code complete. Also, please excuse my grammar ;) English is not my native language.