Unit Testing in C++ and Objective-C just got easier(levelofindirection.com)
levelofindirection.com
Unit Testing in C++ and Objective-C just got easier
http://www.levelofindirection.com/journal/2010/12/28/unit-testing-in-c-and-objective-c-just-got-easier.html
18 comments
Google's gtest (and gmock) is a wonderful piece of C++ code. It needs very small amount boilerplate, less than most junit based frameworks (which is amazing, given this C++ that we are talking about) and great mocking abilities (for C++). I don't see how this CATCH can match it anytime soon. Headers-only approach is interesting but can be bad for compilation performance, which is very very important when projects grow in size.
I hear you on the compilation time. I've not yet found it to be a problem but I haven't used it at large scales yet. I'm considering support for a lib based alternative version for larger scale use (I believe boost.test has the same approach).
As for gtest, I believe I already a decent amount of the same ground. In same cases I do things different ways, so it's difficult to compare directly. In others it may be simply a matter of "coming soon".
I don't see anything in gtest that is not already supported or readily achievable in CATCH.
What I'm not planning on supporting directly is mocking support - but there is no reason why gmock couldn't be used here.
As for gtest, I believe I already a decent amount of the same ground. In same cases I do things different ways, so it's difficult to compare directly. In others it may be simply a matter of "coming soon".
I don't see anything in gtest that is not already supported or readily achievable in CATCH.
What I'm not planning on supporting directly is mocking support - but there is no reason why gmock couldn't be used here.
The headers are included only in test code which is usually in separate compilation units, so the compile times shouldn't be a problem
Arguably compilation time of your test code is even more important! You don't want to be kept waiting for feedback in a TDD cycle. I agree that compilation time can be a limiting factor of the header-only approach (actually link time may be the killer). But it's generally easier to give a header-only library support for libs than the other way around. I plan to keep header-only as the default way of using CATCH.
If your project is template intensive you're going to have long compilation time.
Our work around is parallel compilation.
Our work around is parallel compilation.
What impresses me is this:
The assertion macro is REQUIRE( expression ) [...] Don't worry - lhs and rhs are captured anyway - more on this later.
Phil wrote a bit about this technique -- capturing the two sides of an assertion -- here: http://www.levelofindirection.com/journal/2010/5/21/the-ulti...
The assertion macro is REQUIRE( expression ) [...] Don't worry - lhs and rhs are captured anyway - more on this later.
Phil wrote a bit about this technique -- capturing the two sides of an assertion -- here: http://www.levelofindirection.com/journal/2010/5/21/the-ulti...
Very interesting trick. Basically when you write something like
While very convenient it is kind of easy to "break". For example
CHECK(a == b);
#define CAPTURE(expr) SomeStrangeObject() ->* expr
SomeStrangeObject() ->* a == b
(SomeStrangeObject() ->* a) == b
While very convenient it is kind of easy to "break". For example
int
CHECK( 1 );
CHECK( (one == 1) );
my_test.cpp(6): 1 failed for: 2 == 1
my_test.cpp(7): (one == 1) failed for: 0
CHECK ( one + 1 == 2 );
my_test.cpp:10: error: no match for ‘operator+’ in ‘Catch::ResultBuilder(((const char*)"one + 1 == 2"), 0, ((const std::string&)(& std::basic_string<char, std::char_traits<char>, std::allocator<char> >(((const char*)"my_test.cpp"), ((const std::allocator<char>&)((const std::allocator<char>*)(& std::allocator<char>())))))), 10u, ((const std::string&)(& std::basic_string<char, std::char_traits<char>, std::allocator<char> >(((const char*)"CHECK"), ((const std::allocator<char>&)((const std::allocator<char>*)(& std::allocator<char>()))))))).Catch::ResultBuilder::operator->* [with T = int](((const int&)((const int*)(& one)))) + 1’
CHECK( 1 == 1 && 1);
my_test.cpp:9: error: no match for ‘operator&&’ in ‘((Catch::ResultBuilder*)Catch::ResultBuilder(((const char*)"1 == 1 && 1"), 0, ((const std::string&)(& std::basic_string<char, std::char_traits<char>, std::allocator<char> >(((const char*)"my_test.cpp"), ((const std::allocator<char>&)((const std::allocator<char>*)(& std::allocator<char>())))))), 9u, ((const std::string&)(& std::basic_string<char, std::char_traits<char>, std::allocator<char> >(((const char*)"CHECK"), ((const std::allocator<char>&)((const std::allocator<char>*)(& std::allocator<char>()))))))).Catch::ResultBuilder::operator->* [with T = int](((const int&)((const int*)(&1)))))->Catch::ResultBuilder::operator== [with RhsT = int](((const int&)((const int*)(&1)))) && (one == 1)’
my_test.cpp:9: note: candidates are: operator&&(bool, bool) <built-in>
You make a good point and I have put in some workarounds for this already. I'll probably put more in over time.
The trade-off is that I didn't want to over-use expression templates so I have kept it to one level. But my aim is to at least allow arbitrary expressions to compile and correctly report failures. It should also detect when the expression cannot be decomposed and report this in the output so you can rewrite the expression if you do hit it. I have done this for some operators already.
I want to say mine are nitpickings anyway, the library is very nice and I share your dislike for complex unit test libraries. I had to use the Boost Testing framework recently and that was a pain.
Don't know if it is relevant, but I think you can also work around the parentheses problem using some preprocessor magic. Boost Preprocessor does something similar in its handling of lists, but I haven't been using it for a very long time and I don't remember how it did the trick.
Don't know if it is relevant, but I think you can also work around the parentheses problem using some preprocessor magic. Boost Preprocessor does something similar in its handling of lists, but I haven't been using it for a very long time and I don't remember how it did the trick.
I really like INFO(); that's a nice feature I've not seen anywhere else.
What are the differences between a C++ and Objective-C test framework? Is there anything ObjC-specific about CATCH?
Some of the other design choices here I've used, and later abandoned, in my own testing libraries:
> The assertion macro is REQUIRE( expression ), rather than the, now traditional, REQUIRE_EQUALS( lhs, rhs ), or similar. Don't worry - lhs and rhs are captured anyway - more on this later.
Using C++ operators for assertions is very limiting; some interesting assertions, like floating-point near-equality, have no operator and thus must be tested separately. I like to use separate functions to build "assertion results", so you can get code equivalent to:
> We didn't name the function. We named the test case. This frees us from couching our names in legal C++ identifiers. We also provide a longer form description that serves as an active comment
It can be nice to have a description of the test case, but free-form names like you have are a major pain to write out on the command line if you're running a particular set of tests during development. If you want a description, allow it in addition to the test name.
What are the differences between a C++ and Objective-C test framework? Is there anything ObjC-specific about CATCH?
Some of the other design choices here I've used, and later abandoned, in my own testing libraries:
> The assertion macro is REQUIRE( expression ), rather than the, now traditional, REQUIRE_EQUALS( lhs, rhs ), or similar. Don't worry - lhs and rhs are captured anyway - more on this later.
Using C++ operators for assertions is very limiting; some interesting assertions, like floating-point near-equality, have no operator and thus must be tested separately. I like to use separate functions to build "assertion results", so you can get code equivalent to:
REQUIRE(equal(lhs, rhs));
REQUIRE(equal_within(lhs, rhs, 0.001));
REQUIRE(not_null(lhs));
> We didn't name the function. We named the test case. This frees us from couching our names in legal C++ identifiers. We also provide a longer form description that serves as an active comment
It can be nice to have a description of the test case, but free-form names like you have are a major pain to write out on the command line if you're running a particular set of tests during development. If you want a description, allow it in addition to the test name.
Floating point comparisons are achieved through the Approx helper class. E.g:
The style you suggest is almost supported out of the box, and I've been considering fully supporting it - especially for container oriented tests.
As to the test naming - the idea is that you provide a short, hierarchical name, and a longer description (so the description is in addition to the test name):
TEST_CASE( "example/test1", "Such and such should do so and so" )
To run a test you need only specify the name (example/test1) - or a part of the name (e.g. example/*) (allowing tests to be easily grouped into suites with no extra work).
REQUIRE( pi == Approx( 3.14159265358 ) );
The style you suggest is almost supported out of the box, and I've been considering fully supporting it - especially for container oriented tests.
As to the test naming - the idea is that you provide a short, hierarchical name, and a longer description (so the description is in addition to the test name):
TEST_CASE( "example/test1", "Such and such should do so and so" )
To run a test you need only specify the name (example/test1) - or a part of the name (e.g. example/*) (allowing tests to be easily grouped into suites with no extra work).
Also interesting to read, a previous blog entry from May 21, where he explains the birth of this test framework:
http://www.levelofindirection.com/journal/2010/5/21/the-ulti...
> Most require you to build libraries. This can be a turn off to anyone who wants to get up and running quickly - especially if you just want to try something out. This is especially true of exploratory TDD coding.
This is absolutely true, and especially painful/noticeable if you target multiple platforms. To work around this problem, I've started using the stand-alone version of the Boost Unit Test framework:
This is absolutely true, and especially painful/noticeable if you target multiple platforms. To work around this problem, I've started using the stand-alone version of the Boost Unit Test framework:
#include <boost/test/included/unit_test.hpp>
Reminds me of py.test http://pytest.org/assert.html#assert-with-the-assert-stateme..., which is a rather good thing :)
[deleted]
Interesting, but I want a demo of how mocks and fakes are handled.
Mocks and fakes are not directly supported, however if you are rolling your own CATCH allows test code to be called back by your application code.
Although I haven't tried it yet there should be no problem using CATCH with a third-party mocking framework.