Mutation testing based on LLVM (2018) [pdf](lowlevelbits.org)
lowlevelbits.org
Mutation testing based on LLVM (2018) [pdf]
https://lowlevelbits.org/pdfs/Mull_Mutation_2018.pdf
2 comments
For a great explanation of why you would want to use mutation testing, see https://pitest.org/ (a similar tool but for Java). In short - it is to gauge the quality of your test suite, because ideally any mutation of your app should cause a test failure.
Well, not any mutation. Deleting an assertion should not cause any problems. Replacing bubblesort by quicksort shouldn't either. Etc.
This is a valid concern, for sure. In fact, these are called Equivalent Mutants. We observed it several times, here are some examples. You flip the if/then branches and the code essentially does the same, switching from a fast to slow implementation of the same algorithm. Another example is mutating the code in a C++ destructor, which is not easy to test.
With that being said, even despite the equivalent mutants, you can get a pretty good insight. Here is an example of finding from a proprietory real-time OS used in the space industry: https://gist.github.com/AlexDenisov/b5d2e23457b88813b5ab9d5d... (this is a part of an email which I could safely post online).
With that being said, even despite the equivalent mutants, you can get a pretty good insight. Here is an example of finding from a proprietory real-time OS used in the space industry: https://gist.github.com/AlexDenisov/b5d2e23457b88813b5ab9d5d... (this is a part of an email which I could safely post online).
Hmm, I’m thinking that removing an assertion would potentially allow your program to continue into an undefined state that the assertion was guarding.
You’d also want to watch out for the particular implementation of sorting algos... if you swap out a stable quicksort for a nonstable bubblesort, your benchmark tests might still pass, but you could wind up with different orderings of “equivalent” items in your list before/after that could cause subtle issues.
If you have a test that truly still passes regardless of any mutation, I’d have to wonder if it should exist at all.
You’d also want to watch out for the particular implementation of sorting algos... if you swap out a stable quicksort for a nonstable bubblesort, your benchmark tests might still pass, but you could wind up with different orderings of “equivalent” items in your list before/after that could cause subtle issues.
If you have a test that truly still passes regardless of any mutation, I’d have to wonder if it should exist at all.
Is mutation testing different than mutation-based fuzz testing? If so, how?
Mutation testing mutates the program (by making changes in the code, like flipping a condition in an if statement), and then checks whether the test suite catches it. That is, it tests the tests.
Mutation-based fuzz testing mutates the input to the program (often by starting from know-valid data and applying a mutation to the data). So it tests the program, much like unit-tests.
Mutation-based fuzz testing mutates the input to the program (often by starting from know-valid data and applying a mutation to the data). So it tests the program, much like unit-tests.