Greatest: C unit testing library in one file(github.com)
github.com
Greatest: C unit testing library in one file
https://github.com/silentbicycle/greatest
16 comments
I (the author) have been doing a lot of embedded development lately, and in those cases you really can't assume very much beyond core ANSI C features. That's also why greatest doesn't do any dynamic allocation, at all. There's a relatively small struct for the current testing progress, and everything else can easily run out of ROM.
My primary goal with greatest is to assume as little as possible, and avoid imposing any additional constraints on projects just because people want to test them. I also want the implementation to be very transparent -- it's just a 600-ish line header file, carefully documented. It's a major pet peeve of mine when I have to work around features that try to automatically be helpful, but have edge cases that break down in opaque ways. (Build systems tend to be particularly bad in that way.)
Also, I'm beyond shocked that the name wasn't already taken. ;)
One of my upcoming tasks is making stdio optional in greatest, because printf doesn't do you a whole lot of good if you're running it on an Arduino or something, but in those contexts automated testing is especially valuable.
My primary goal with greatest is to assume as little as possible, and avoid imposing any additional constraints on projects just because people want to test them. I also want the implementation to be very transparent -- it's just a 600-ish line header file, carefully documented. It's a major pet peeve of mine when I have to work around features that try to automatically be helpful, but have edge cases that break down in opaque ways. (Build systems tend to be particularly bad in that way.)
Also, I'm beyond shocked that the name wasn't already taken. ;)
One of my upcoming tasks is making stdio optional in greatest, because printf doesn't do you a whole lot of good if you're running it on an Arduino or something, but in those contexts automated testing is especially valuable.
Cool, I really like this! I wrote a popular C function faking framework in a single header file for exactly the same reasons: all bets are off in embedded targets.
https://github.com/meekrosoft/fff
The test C test framework in the project is 4 lines long, but does require the standard library. https://github.com/meekrosoft/fff/blob/master/test/c_test_fr...
The test C test framework in the project is 4 lines long, but does require the standard library. https://github.com/meekrosoft/fff/blob/master/test/c_test_fr...
fff looks cool, too. Thanks! I'll check it out. :)
I've tended to use CMock (https://github.com/ThrowTheSwitch/CMock) for mocking, though it depends on some Ruby-based tooling. The last time I looked into replacing CMock I wound up hip-deep in the binutils. (It was fun when I stopped getting google results from StackOverflow and started getting them from Phrack, though. ;) )
I've tended to use CMock (https://github.com/ThrowTheSwitch/CMock) for mocking, though it depends on some Ruby-based tooling. The last time I looked into replacing CMock I wound up hip-deep in the binutils. (It was fun when I stopped getting google results from StackOverflow and started getting them from Phrack, though. ;) )
Did you try cmocka? http://cmocka.org/
It was the most featureful that I found when I was looking last year.
It was the most featureful that I found when I was looking last year.
I have not, but I'll check it out, thanks.
Author of greatest, here.
I wrote a blog post about it a couple months ago: http://spin.atomicobject.com/2013/07/31/greatest-c-testing-e...
I wrote a blog post about it a couple months ago: http://spin.atomicobject.com/2013/07/31/greatest-c-testing-e...
Nice unit test framework you have written, exactly what I have been looking for my C projects. I, too, dabble in embedded environments and can't rely on big libraries.
One question/suggestion, though. In the end of all the samples you need GREATEST_MAIN_DEFS/main()/GREATEST_MAIN_BEGIN/RUN_SUITE(xxx)/GREATEST_MAIN_END. Could this be a simple oneliner like GREATEST_MAIN(xxx)?
I do understand that this is probably required for embedded systems where you might not have a regular main(), but I would just like to write GREATEST_MAIN(xxx). Does this make sense to you? Are there use cases where it makes sense to do something else in main()?
Anyway, good stuff!
One question/suggestion, though. In the end of all the samples you need GREATEST_MAIN_DEFS/main()/GREATEST_MAIN_BEGIN/RUN_SUITE(xxx)/GREATEST_MAIN_END. Could this be a simple oneliner like GREATEST_MAIN(xxx)?
I do understand that this is probably required for embedded systems where you might not have a regular main(), but I would just like to write GREATEST_MAIN(xxx). Does this make sense to you? Are there use cases where it makes sense to do something else in main()?
Anyway, good stuff!
Thanks!
I use GREATEST_MAIN_BEGIN and GREATEST_MAIN_END so that people can put other things in main, around them. Also, there are often several RUN_SUITE calls, and I can't depend on vararg macros portably without also depending on C99. While there are some ways to work around that, they don't deliver enough value to justify the added complication.
I use GREATEST_MAIN_BEGIN and GREATEST_MAIN_END so that people can put other things in main, around them. Also, there are often several RUN_SUITE calls, and I can't depend on vararg macros portably without also depending on C99. While there are some ways to work around that, they don't deliver enough value to justify the added complication.
Yeah, I just realized that you can't really recurse with macros without doing nasty repetition, so it is not possible to do GREATEST_MAIN(suite1, suite2, suite3, etc), not even using C99 variadic macros.
Had it been possible, it would definitely make sense to add this oneliner utility. Since it would end up being very limited, it probably doesn't.
Had it been possible, it would definitely make sense to add this oneliner utility. Since it would end up being very limited, it probably doesn't.
I think it could work like this:
SUITE_LIST(suites) {
suite1,
suite2,
// ...
};
GREATEST_MAIN(suites);
C unit testing framework in 2 defs:
http://www.jera.com/techinfo/jtns/jtn002.html
http://www.jera.com/techinfo/jtns/jtn002.html
I wrote greatest because minunit doesn't provide any of the infrastructure that nearly every test runner grows eventually, in some ad hoc way. Halting after the first error is just not good enough after projects grow beyond a certain size. Also, being able to run just specific suite(s) or test(s) by name patterns from the command line really speeds up one's workflow.
I still wanted something as straightforward to use as minunit, though. Not knocking it. I'd rather use minunit than something that imposed its own architectural opinions on my projects, or consumed resources already in short supply. Also, I make a lot of small C utilities and libraries, and having to define a "project" just to have tests adds a bit too much friction.
(I wrote a bit more about that here: http://spin.atomicobject.com/2013/07/31/greatest-c-testing-e... )
I still wanted something as straightforward to use as minunit, though. Not knocking it. I'd rather use minunit than something that imposed its own architectural opinions on my projects, or consumed resources already in short supply. Also, I make a lot of small C utilities and libraries, and having to define a "project" just to have tests adds a bit too much friction.
(I wrote a bit more about that here: http://spin.atomicobject.com/2013/07/31/greatest-c-testing-e... )
Agreed. I've skimmed the source and it looks like it would be much easier to integrate into a build system as well.
I find Minunit helpful because its easy to memorize and dump into a new project for quick testing. Its brevity also seems to help introduce c devs to unit testing.
I find Minunit helpful because its easy to memorize and dump into a new project for quick testing. Its brevity also seems to help introduce c devs to unit testing.
Minunit has grown significantly since the initial 2-def version: https://github.com/siu/minunit .
One disadvantage of the macro-based approach - and it looks like greatest suffers from the same problem - which isn't actually that unusual - is the difficulty of setting a breakpoint on the code that fails a test. This comes in handy when a test fails and you'd like to investigate.
And due to C's limitations, you will actually have to do this quite often, on a wide range of failures, because the failure messages don't seem to print any details about the values being compared. So you'll need to hit the debugger to find out anything out.
And due to C's limitations, you will actually have to do this quite often, on a wide range of failures, because the failure messages don't seem to print any details about the values being compared. So you'll need to hit the debugger to find out anything out.
I'm been thinking about ways to get around this. I'm using macros for __FILE__ and __LINE__, but it does interfere with breakpoints.
One of my open issues is to add an ASSERT_EQ_MEMCMP function that does print out the difference, probably as a hexdump. Similarly, greatest could have something like 'ASSERT_EQ_NUM' for the cases where printing the value is meaningful, and an ASSERT_EQ_STRUCT that takes cmp and print functions. I've been thinking about it for a while, because I want to get it right before I change the API.
One of my open issues is to add an ASSERT_EQ_MEMCMP function that does print out the difference, probably as a hexdump. Similarly, greatest could have something like 'ASSERT_EQ_NUM' for the cases where printing the value is meaningful, and an ASSERT_EQ_STRUCT that takes cmp and print functions. I've been thinking about it for a while, because I want to get it right before I change the API.
Joe Walnes' tinytest is just a 113 line header file: https://github.com/joewalnes/tinytest
Greatest is 591 lines .h and 133 lines .c.
You can see tinytest in action here: https://github.com/cucumber/bool/blob/master/c/test/test.c
Greatest is 591 lines .h and 133 lines .c.
You can see tinytest in action here: https://github.com/cucumber/bool/blob/master/c/test/test.c
They've got a lot in common. tinytest looks a bit simpler, but also doesn't do any allocation and looks reasonably portable. Cool.
The single biggest difference is that greatest has more control over about which test(s) it runs. tinytest just bails at the first failure, whereas greatest can run everything and report, or individual test(s) or groups of tests whose names match a substring. That adds some to the implementation, but I've found it particularly valuable once a program has several modules and hundreds of tests.
Also, greatest supports parametric testing, which means that adding fuzzing/randomized testing is pretty straightforward (though usually project-specific).
tinytest requires C99, greatest doesn't. That shouldn't matter, but unfortunately sometimes it does. Embedded compilers for proprietary architectures can be lacking, in particular.
greatest also doesn't assume ANSI escape sequences are okay. Sometimes dumping out a bunch of [1;31m stuff is annoying. That's a matter of taste, though, and it'd be easy to make it optional.
The single biggest difference is that greatest has more control over about which test(s) it runs. tinytest just bails at the first failure, whereas greatest can run everything and report, or individual test(s) or groups of tests whose names match a substring. That adds some to the implementation, but I've found it particularly valuable once a program has several modules and hundreds of tests.
Also, greatest supports parametric testing, which means that adding fuzzing/randomized testing is pretty straightforward (though usually project-specific).
tinytest requires C99, greatest doesn't. That shouldn't matter, but unfortunately sometimes it does. Embedded compilers for proprietary architectures can be lacking, in particular.
greatest also doesn't assume ANSI escape sequences are okay. Sometimes dumping out a bunch of [1;31m stuff is annoying. That's a matter of taste, though, and it'd be easy to make it optional.
Also, a minor nitpick: the 591 line header is the whole thing. The 133-line .c file is only an example.
Macros are surprisingly useful in the unit testing context. Whether or not it's considered good practice, I like to embed my tests within the header files that they're associated with. With macros and gcc's -D flag in order to generate production and testing builds separately.
I'm new to the world of C, is this typical?
I'm new to the world of C, is this typical?
It's fairly typical. The standard "assert" macro behaves exactly that way, it does not generate code if the code is compiled with NDEBUG defined.
Of course you shouldn't abuse it but in this instance it's perfectly fine IMHO.
This reminds me that in addition to that the linux kernel has a pretty cool feature for debugging called "dynamic debugging" where you can enable and disable "printk" (the kernel's version of printf) messages dynamically from the shell through the debugfs:
http://lwn.net/Articles/434833/
Of course you shouldn't abuse it but in this instance it's perfectly fine IMHO.
This reminds me that in addition to that the linux kernel has a pretty cool feature for debugging called "dynamic debugging" where you can enable and disable "printk" (the kernel's version of printf) messages dynamically from the shell through the debugfs:
http://lwn.net/Articles/434833/
I've seen people do that before, as well as putting the tests in the .c file, guarded by "#if TEST_MODULE_NAME", and exporting a (module_name)_run_tests(void) function.
My preference is to have separate test_module_name.c files. Failing that, I would include tests in the module's .c file rather than the header, so that the header can be relatively short and focused on documenting the module's interface, rather than its implementation details.
greatest doesn't force any specific style there, by design.
My preference is to have separate test_module_name.c files. Failing that, I would include tests in the module's .c file rather than the header, so that the header can be relatively short and focused on documenting the module's interface, rather than its implementation details.
greatest doesn't force any specific style there, by design.
I'm using Seatest[0] for a hobby project I'm writing. I was looking for simple xUnit-style assertions. Works pretty well out of the box, but I did modify it to make it ANSI C compatible and change the output. (I want colored terminal output, darn it! Green if OK, red if not!)
[0] https://code.google.com/p/seatest/
[0] https://code.google.com/p/seatest/
I've never used a unit test framework for C ever.
I usually write a test exerciser for the library/command and a "known good output" file then diff the output and the known good.
will take a look at this for certain.
I usually write a test exerciser for the library/command and a "known good output" file then diff the output and the known good.
will take a look at this for certain.
I used to use FCTX for unit testing, which was just a header file. I don't know if it used nonstandard compiler features under the hood. Unfortunately, while it's mirrored, it doesn't look like the main development website is around any more.
Github mirror: https://github.com/imb/fctx
Github mirror: https://github.com/imb/fctx
+1 on FCTX, even in its dormant state. A sample: http://agentzlerich.blogspot.com/2011/01/c-header-only-unit-...
In the same flavour: https://github.com/orangeduck/ptest
In our team we relay on Unity for our unit tests. We do embedded C and try to practice TDD as much as possible. The author claims it depends on Ruby and such to work. I wanted to point out this is not the case, and that actually we are using it 'clean' with no trouble.
Unity alone doesn't, but ceedling and CMock do. I was referring to them as a group, because I usually see them used together, but you're right.
(I work with most of the maintainers of those tools, and have some minor commits myself.)
(I work with most of the maintainers of those tools, and have some minor commits myself.)
Yet another very simple single-file in 65 lines of code here: https://github.com/tkrajina/yactest
CppUTest isn't bad for C projects, even if the test framework uses C++.
http://cpputest.github.io/
edit - grammar
edit - grammar
Sorry I don't want to hijack the authors effort, I am just mentioning it for anybody considering additional testing frameworks.
Question to professional C programmers: Do you write unit tests?
At my current job I've never met C++, let alone C programmer who writes unit tests. Is it common?
At my current job I've never met C++, let alone C programmer who writes unit tests. Is it common?
In my experience, unit testing is more common in embedded C development, because the code is held to much higher standard of accountability.
Sloppy embedded code can kill people, or at the very least lead to incredibly expensive hardware recalls. With the exception of poorly thought-out privacy policies, it's unlikely that errors in a web app can cause that level of problems.
It's also much easier to deploy an update to a web app than a Mars rover or a car's brake controllers.
Sloppy embedded code can kill people, or at the very least lead to incredibly expensive hardware recalls. With the exception of poorly thought-out privacy policies, it's unlikely that errors in a web app can cause that level of problems.
It's also much easier to deploy an update to a web app than a Mars rover or a car's brake controllers.
I see few unit tests, but lots and lots of integration tests, and it looks like greatest could be used for that too.
will you be adding it to CCAN (the C code archive, in a similar spirit to Perl's CPAN)?
see http://ccodearchive.net/
see http://ccodearchive.net/
this is a bloated minunit :)
just the short of thing a QA dev would do to keep his job.
i kid. kid. It is nice, but overkill in my humble opinion. Failing on the first error for unittest is a feature. because you are not supposed to even commit it if it is failing, so no need to fancy graphics and reports. It either passed, or not.
just the short of thing a QA dev would do to keep his job.
i kid. kid. It is nice, but overkill in my humble opinion. Failing on the first error for unittest is a feature. because you are not supposed to even commit it if it is failing, so no need to fancy graphics and reports. It either passed, or not.
> Failing on the first error for unittest is a feature. because you are not supposed to even commit it if it is failing, so no need to fancy graphics and reports.
A test suite may run for a long time and bailing out at first failure is a very stupid thing to do. It may be nice to have it as an option but not the only way of doing it.
Even a humble test suite might take 5 minutes to run, going to grab some coffee and coming back to see the first test fail would be a bad thing. A big test suite might take hours on a CI server, having it fail on the first test would be a major waste of time.
And with a modern distributed version control, there's no reason why you shouldn't commit code (not to the master branch!) that doesn't pass tests. To share it with your team mates or push the code to be tested on a CI server, for example.
A test suite may run for a long time and bailing out at first failure is a very stupid thing to do. It may be nice to have it as an option but not the only way of doing it.
Even a humble test suite might take 5 minutes to run, going to grab some coffee and coming back to see the first test fail would be a bad thing. A big test suite might take hours on a CI server, having it fail on the first test would be a major waste of time.
And with a modern distributed version control, there's no reason why you shouldn't commit code (not to the master branch!) that doesn't pass tests. To share it with your team mates or push the code to be tested on a CI server, for example.
if you unit tests take that long, break it up.
or you are probably doing functional testing in your units.
or you are probably doing functional testing in your units.
Is it though? If I made changes that broke something surely I would want to know everything that broke rather than just the first test in an arbitrarily ordered set of tests!
Though I like your style, very simple, and simple is good. The state is binary, tests are passing or they are not. Unfortunately in this particular reality the added complexity of knowing all broken tests can prove very useful.
Though I like your style, very simple, and simple is good. The state is binary, tests are passing or they are not. Unfortunately in this particular reality the added complexity of knowing all broken tests can prove very useful.
Knowing that a change broke a specific set of tests tells you a lot more than knowing it broke the first one of them that happened to run.
If I'm using fuzzing to run thousands of tests, and all the ones whose seed led to the same specific bit pattern are failing, that's particularly useful information. I found a particularly subtle bug in heatshrink (https://github.com/atomicobject/heatshrink) that way, but a single failing test would have looked like noise.
If I'm using fuzzing to run thousands of tests, and all the ones whose seed led to the same specific bit pattern are failing, that's particularly useful information. I found a particularly subtle bug in heatshrink (https://github.com/atomicobject/heatshrink) that way, but a single failing test would have looked like noise.
I'd like at least the option to keep on working, making changes, and at the end fix all the broken tests. Sometimes it's easier to develop this way than to stop everytime and fix a broken test, usually when you change the architecture / big code changes.
Failing on the first error is not a feature, because the error that happen to be first may not give enough information on where it went wrong, but a different test may.
Also consider that in embedded software the hardware changes much more. You can't just "not commit something that fails". Something might work on hardware v1 but fail on v2 (that uses the cheaper version of the microcontroller, whatever).
While that may not work in some environments that don't support these extensions, I'm having trouble imagining ones that support C stdio but do not support linker sets.