Why I don't mock(blog.metaobject.com)
blog.metaobject.com
Why I don't mock
http://blog.metaobject.com/2014/05/why-i-don-mock.html
23 comments
I spent more time building my mocks instead of building my tests. Sometimes a file might have 90% mocks and 10% tests. Or maybe that's just my mocking framework...
The idea is that if you end up with code like this, it shows you have too many collaborators and should refactor them away.
Exactly.
An other reason could be it's a main entry point for code execution, a class that only purpose is the glue others.
In that case, having a lot of mocks is a code smell pointing that you should probably test this on the integration level rather than on the unit one (and thus, discard mocks and use "real" everything).
(but beware not to be too much self indulgent on what is a gluing class, most of the time, they are not and it's indeed a god class problem).
An other reason could be it's a main entry point for code execution, a class that only purpose is the glue others.
In that case, having a lot of mocks is a code smell pointing that you should probably test this on the integration level rather than on the unit one (and thus, discard mocks and use "real" everything).
(but beware not to be too much self indulgent on what is a gluing class, most of the time, they are not and it's indeed a god class problem).
It could just be that the module of code under test doesn't do enough. If you need to mock a service, database, other service, error-handler, etc ... but the core logic is just verifying that <some condition> is met, then this is going to be the case.
Personally I like to avoid mocking for cases like this, since integration tests can usually suffice.
As you said, though, it could also be the framework. If you're in C#-land, Moq is pretty terse with the syntax so the actual mocking code isn't that big.
Personally I like to avoid mocking for cases like this, since integration tests can usually suffice.
As you said, though, it could also be the framework. If you're in C#-land, Moq is pretty terse with the syntax so the actual mocking code isn't that big.
In many cases, isolated testing does not require test doubles. There are alternatives to mocks: as long as the system under test is not manipulating its dependencies, you can get the state you need from the external resources and pass that into your classes.
A lot of unit tests are merely a poor man's design-by-contract / static analysis.
I generally agree. I think Uncle Bob has recently realized all his advocating without public explanation causes a lot of confusion. He's been blogging quite a bit this past month, two posts of which are related to mocking:
http://blog.8thlight.com/uncle-bob/2014/05/10/WhenToMock.htm...
http://blog.8thlight.com/uncle-bob/2014/05/14/TheLittleMocke...
http://blog.8thlight.com/uncle-bob/2014/05/10/WhenToMock.htm...
http://blog.8thlight.com/uncle-bob/2014/05/14/TheLittleMocke...
I really don't get the debate. If you want to test state vs. behavior you still need to orchestrate and verify that state. You're still "mocking" because you're (presumably) not using a production system on which to test and you're using predictable, idempotent tests and test data. The only difference is you're not using a mocking "framework". Seems to me all you're doing is inheriting a lot of external dependencies that have little or nothing to do with your test. I don't see mocking as behavior vs. state, but rather just a way to remove dependencies that lead to brittle tests.
What am I missing?
What am I missing?
> you still need to orchestrate and verify that state.
Only if you limit yourself to testing using unit tests. Two of the strongest testing tools care not one bit about internal state.
Combinatorial testing tests all possible combinations of input pairs, and looks for the system to fail.
Fuzz testing creates a fairly large volume of noise as input, and looks for the system to fail.
Integration testing should be in your toolbox right alongside unit testing, and will exercise objects which are not appropriate for unit tests.
> inheriting a lot of external dependencies that have little or nothing to do with your test.
I've found that this kind of thinking leads to development practices which rely on unit tests & production as their sole methods of testing. If a component is part of your production system, it should also be part of your test system.
Only if you limit yourself to testing using unit tests. Two of the strongest testing tools care not one bit about internal state.
Combinatorial testing tests all possible combinations of input pairs, and looks for the system to fail.
Fuzz testing creates a fairly large volume of noise as input, and looks for the system to fail.
Integration testing should be in your toolbox right alongside unit testing, and will exercise objects which are not appropriate for unit tests.
> inheriting a lot of external dependencies that have little or nothing to do with your test.
I've found that this kind of thinking leads to development practices which rely on unit tests & production as their sole methods of testing. If a component is part of your production system, it should also be part of your test system.
I found the article disappointing. Instead of giving reasonable arguments against mocking, the author offers an example where mocking was used wrong and therefore propagates that it shouldn't be used at all.
The problem is not mocking. I found it to be absolutely helpful if used right (and right often means sparsely). The problem is this "all or nothing" mentality that many programmers have. Either they mock the hell out of the codebase or they don't use it at all.
I don't use mock objects, unless it's message based business logic, often located at the topmost layer of abstraction.
The problem is not mocking. I found it to be absolutely helpful if used right (and right often means sparsely). The problem is this "all or nothing" mentality that many programmers have. Either they mock the hell out of the codebase or they don't use it at all.
I don't use mock objects, unless it's message based business logic, often located at the topmost layer of abstraction.
So I do a lot of systems programming type development.
In these cases, whether you're PXE booting a server or talking to 500 different APIs that all range between the latest database or cloud API, flashing firmware on a RAID controller (shudder), whatever, there's a point of diminishing returns. There's also most of the bugs that you won't be able to simulate, because they come from external factors.
The magic lies in automated integration tests.
QA automation is AMAZING stuff. Set up real servers and real test environments when you can. Test real database upgrades.
Mock testing can have it's place, but often you find yourself making internal API contracts more rigid than you want, which can limit refactoring.
In these cases, whether you're PXE booting a server or talking to 500 different APIs that all range between the latest database or cloud API, flashing firmware on a RAID controller (shudder), whatever, there's a point of diminishing returns. There's also most of the bugs that you won't be able to simulate, because they come from external factors.
The magic lies in automated integration tests.
QA automation is AMAZING stuff. Set up real servers and real test environments when you can. Test real database upgrades.
Mock testing can have it's place, but often you find yourself making internal API contracts more rigid than you want, which can limit refactoring.
How do you test your system's response to error conditions? Do you have hooks in your integrated system to force errors (throw exceptions, violate contracts, etc.)?
In some cases, yes.
You can do things like set up scenarios you know won't work, such as connecting to machines that don't exist or sending the wrong parameters.
The main thing is if the external inputs can't be made to "violate a contract", the internals aren't the problem.
Ultimately, it depends on what you are building. Simple CRUD web applications are different from systems development type applications in many cases. Many of these classes can be more easily mocked out.
Often the breaking change you'll see is an upstream API or SaaS endpoint changing in ways you didn't expect (or the difference between a Linux utility on the latest Ubuntu, an older RHEL, and Solaris), so it's often better to concentrate on testing the real thing.
And these tests are expensive, so an automated test matrix that deploys a VM fleet with lots of permutations is usually necessary.
You can do things like set up scenarios you know won't work, such as connecting to machines that don't exist or sending the wrong parameters.
The main thing is if the external inputs can't be made to "violate a contract", the internals aren't the problem.
Ultimately, it depends on what you are building. Simple CRUD web applications are different from systems development type applications in many cases. Many of these classes can be more easily mocked out.
Often the breaking change you'll see is an upstream API or SaaS endpoint changing in ways you didn't expect (or the difference between a Linux utility on the latest Ubuntu, an older RHEL, and Solaris), so it's often better to concentrate on testing the real thing.
And these tests are expensive, so an automated test matrix that deploys a VM fleet with lots of permutations is usually necessary.
Another magical thing (that I'm yet to try myself) is automated testing against live system. This ensures that the system actually works in production, not just on staging or test environment set up specifically for QA. I was initially against it, but reading about it recently[0] made me reconsider my opinion for some reason.
[0] Meditations by Sean Cassidy, http://blog.seancassidy.me/meditations.html
[0] Meditations by Sean Cassidy, http://blog.seancassidy.me/meditations.html
I too do systemsish programming in C++. All the places I've ever done this have "unit" tests that link in almost all of the executable and inject test data through the normal frontend. There's very little mocking; sometimes there will be a setFooForUnitTests() function to short-circuit some normal configuration path.
A big danger with mocks is just shoving the problems under the rug rather than fixing them. Mocks represent assumptions, typically untested assumptions. There is generally nothing enforcing a mock's behaviors to be in accordance with whatever it is mocking, so that leaves it as a free parameter which can easily be incorrect if someone makes an error (which obviously can and does happen).
This isn’t a danger, it’s a benefit: your tests of one unit (whatever you consider a “unit” to be) are isolated from the behaviour of the rest of the system. For example, if you break one part of your implementation, you get one failing unit test, not a thousand.
Checking these assumptions is what integration and acceptance tests are for.
Checking these assumptions is what integration and acceptance tests are for.
As always theres a middle ground between the extremes of MOCK-ALL-TEH-THINGS or mock nothing!!
Since reading Sandi Metz's very fantastic "Practical Object-oriented design in ruby" it clear to me that mocking is great for checking that objects other the one under test receive messages sent by the methods being tested (makes tests easier to setup and write usually). If I find that I'm doing too much mocking, then its usually because the method I'm using is doing way too much. FWIW.
Since reading Sandi Metz's very fantastic "Practical Object-oriented design in ruby" it clear to me that mocking is great for checking that objects other the one under test receive messages sent by the methods being tested (makes tests easier to setup and write usually). If I find that I'm doing too much mocking, then its usually because the method I'm using is doing way too much. FWIW.
I really want to see people's complex projects and example code. There have been way too much opinion on this topic lately without any concrete evidence to show the strengths/weaknesses of using or not using mocks.
I can only give you an anecdote as our product is closed source. Mocks went like this, in a bell curve fashion:
1. Mocks are great, we'll use them everywhere.
2. Mocks are preventing our tests from being refactored.
3. Mocks are preventing our code form being refactored.
4. Mocks are preventing progress on the platform.
5. Code is being modified and tests and mocks are being removed to keep productivity up.
6. There are no mocks and few tests.
7. Tests increase again as defects are found. No mocks required.
This is over the space of 8 years and our problem was the amount of coupling they caused.
1. Mocks are great, we'll use them everywhere.
2. Mocks are preventing our tests from being refactored.
3. Mocks are preventing our code form being refactored.
4. Mocks are preventing progress on the platform.
5. Code is being modified and tests and mocks are being removed to keep productivity up.
6. There are no mocks and few tests.
7. Tests increase again as defects are found. No mocks required.
This is over the space of 8 years and our problem was the amount of coupling they caused.
I have had the exact opposite experience. People weren't using mocks therefore when you changed something at a low level, it caused tests all over the system to break that didn't particularly care about that behavior. It also makes test data set up much more complicated for both unit and integration tests.
I think there might be subtle differences is how we are mocking objects and how we organize our code that might make mocking have different outcomes. That's why I would like to see actual examples that can be deeply analyzed. I suspect that it's in fact the way the code is written and what the underlying platform allows that makes one more suitable than the other rather than mocks being fundamentally a poor choice.
I think there might be subtle differences is how we are mocking objects and how we organize our code that might make mocking have different outcomes. That's why I would like to see actual examples that can be deeply analyzed. I suspect that it's in fact the way the code is written and what the underlying platform allows that makes one more suitable than the other rather than mocks being fundamentally a poor choice.
Mocks don't cause coupling. Programmers do.
Cute, but even as terse as this comment is its sentiment is incorrect.
Mocks, almost by their very nature, encourage coupling. Ask yourself the question "What tends to get mocked? " Answer: Objects which are difficult and/or heavy to instantiate, but which are outside of your direct control and cannot be changed, e.g.: database connections, network sockets, etc.
So when you want to write tests around this you are almost forced to mock them out. So you instantiate those classes (as mocks) in your test class, so that the code under test. There are now two references to that class: one in the test, and one in the code being tested.
This is coupling. QED.
Mocks, almost by their very nature, encourage coupling. Ask yourself the question "What tends to get mocked? " Answer: Objects which are difficult and/or heavy to instantiate, but which are outside of your direct control and cannot be changed, e.g.: database connections, network sockets, etc.
So when you want to write tests around this you are almost forced to mock them out. So you instantiate those classes (as mocks) in your test class, so that the code under test. There are now two references to that class: one in the test, and one in the code being tested.
This is coupling. QED.
> Ask yourself the question "What tends to get mocked?"
Answer: every collaborator of the object under test, so that only the tested object is actually having its implementation exercised. All other objects that it needs to talk to — database connections, network sockets, or whatever — are replaced with mocks, and the test subject’s interactions with those mocks are verified, so that a) we have confidence that the tested object is interacting with its collaborators in the expected way, and b) the test itself doesn’t depend at all on the actual behaviour of those collaborators’ implementations, which behaviour should be explicitly exercised elsewhere.
That’s the opposite of coupling. I don’t understand where your answer comes from or what it means.
Answer: every collaborator of the object under test, so that only the tested object is actually having its implementation exercised. All other objects that it needs to talk to — database connections, network sockets, or whatever — are replaced with mocks, and the test subject’s interactions with those mocks are verified, so that a) we have confidence that the tested object is interacting with its collaborators in the expected way, and b) the test itself doesn’t depend at all on the actual behaviour of those collaborators’ implementations, which behaviour should be explicitly exercised elsewhere.
That’s the opposite of coupling. I don’t understand where your answer comes from or what it means.
I view this as definitely enforcing coupling.
The issue is that you should want to make it very easy to change the API signature of internal components.
Mock objects can tend to reinforce internal coding choices when things that are not a public interface, or a service boundary, are mocked out.
If you're mocking at this level, there is a potential for refactoring of intra-class API contracts to become much harder to change.
Thus, it would be better IMHO to mock out only meaningful service boundaries, and concentrate testing at the public contracts.
You can still get very strong coverage, but you're just thinking about a different level of inputs.
The issue is that you should want to make it very easy to change the API signature of internal components.
Mock objects can tend to reinforce internal coding choices when things that are not a public interface, or a service boundary, are mocked out.
If you're mocking at this level, there is a potential for refactoring of intra-class API contracts to become much harder to change.
Thus, it would be better IMHO to mock out only meaningful service boundaries, and concentrate testing at the public contracts.
You can still get very strong coverage, but you're just thinking about a different level of inputs.
The point more accurately is that coupling to the interface used to communicate with your collaborator I.e. If you were to refactor the way that the test object communicates with it's collaborator (without changing it's external behaviour) you are now forced to update the mocks. If the collaborator is only used within your test object and nowhere else in your system the communication between them doesn't need to be pinned down, the test should just test both of them at the same time. This is the difference between a white box test (where your mocking has to expose implementation details) and blackbox testing (where you can refactor without breaking tests).
Actually, my answer to "What tends to get mocked?" is quite different from yours. My answer is "Everything that is not what is currently being tested." So if I'm testing method Foo.bar, all of the method calls that leave Foo.bar are mocked. If Foo.bar calls A.b(), A will be mocked as its functionality is not currently under test. The only real exception is if A.b() is a trivial implementation and I don't already have A mocked for some other reason. If A.b() is a setter/getter, for example, I might not mock it.
Mocking "everything that is not what is currently being tested" is tautological.
Mocking and stunning in the manner you describe makes tests extremely dependent upon the current implementation. In fact, it inverts the relationship your tests should have with your development process.
Tests should allow you to refactor without breaking tests, and tests should break when APIs you call change in incompatible ways. These are the two primary long-term benefits that tests offer. Stubbing and mocking all of their internals causes tests to break when you change internal implementation (library Foo was switched to library Bar, but Foo is mocked and has a different API than Bar). It also causes tests not to break when the Foo API is incompatibly changed by another developer.
Mocking and stunning in the manner you describe makes tests extremely dependent upon the current implementation. In fact, it inverts the relationship your tests should have with your development process.
Tests should allow you to refactor without breaking tests, and tests should break when APIs you call change in incompatible ways. These are the two primary long-term benefits that tests offer. Stubbing and mocking all of their internals causes tests to break when you change internal implementation (library Foo was switched to library Bar, but Foo is mocked and has a different API than Bar). It also causes tests not to break when the Foo API is incompatibly changed by another developer.
Tests are generally for making sure your code appropriately handles cases where Foo behaves a certain way. That could be returning a certain value or throwing an exception, whatever types of branches you might have. Foo's implementation doesn't matter, therefore it gets mocked. If Bar doesn't, for example, throw the same exceptions, you've changed your method behavior and it makes sense to also update your tests/mocks.
Do you always hit the real database and network when you test? Because what you describe would also rule out stubs and fakes.
Assume that when you have N consumers of an interface then your coupling is N. Mocks consume an interface as does your codebase so your coupling is at least N=2 whereas your codebase alone was N=1. By using mocks, programmers cause coupling to increase so programmers should stop using mocks to decrease coupling.
This is technically true but I don't think it's a good argument against mocking or test doubles.
By this reasoning, the test suite itself causes coupling to increase. For any component C, you have at least N=2: when it's used in production code, and when it's tested. If you don't test C, you reduce N to 1. This doesn't sound like a good argument against testing, though.
By this reasoning, the test suite itself causes coupling to increase. For any component C, you have at least N=2: when it's used in production code, and when it's tested. If you don't test C, you reduce N to 1. This doesn't sound like a good argument against testing, though.
> Mocks consume an interface as does your codebase so your coupling is at least N=2 whereas your codebase alone was N=1.
I really hope that you're not being serious here.
I really hope that you're not being serious here.
100%. Its called efferent coupling.
It's answer which is technically 100% right; and practically 100% missing the point. So it seemed facetious.
The Openstack project is a good example of a complex project with complicated testing. Merges are gated on successful test runs of a handful of different tests (unit, integration, etc)
An example of complicated mocking can be seen here:
https://github.com/openstack/nova/blob/master/nova/tests/vir...
In particular, SpawnTestCase.
Disclaimer: While I didn't write this example, I am responsible for some of the other mess
An example of complicated mocking can be seen here:
https://github.com/openstack/nova/blob/master/nova/tests/vir...
In particular, SpawnTestCase.
Disclaimer: While I didn't write this example, I am responsible for some of the other mess
I don't use mocks much, except with regard to the "now" method. There's some code that relies on the current time, I can use mock to mock many values of 'current time' instead of being forced to only test the case where the current time is when I run the unit test. This is especially important when dealing with code involving time zone and day light savings.
My experience with time is that whenever you build a system that is time-dependent in any meaningful way, you want to have "synthetic time" that you control, which you only feed with real time in a single, controllable place.
Otherwise you get nasty effects, such as time moving on while your system is processing etc.
Otherwise you get nasty effects, such as time moving on while your system is processing etc.
I'm guessing you mean like:
def has_expired(now=None):
#.. implementation
def has_expired():
#.. implementation
def get_request_handler(request):
#...code
now = timezone.now()
if has_expired(now=now):
raise Error403.
If you are only changing the value of what an API returns, that is not a mock. It is a stub.
I agree. But another option is to make all the methods that would call now() take an extra parameter of the current time. Talk about making your dependencies explicit...
But now you've...
a) changed your interface in a way that invites use cases you never intended (or wanted) to support.
b) added branching code anywhere that requires now()
a) changed your interface in a way that invites use cases you never intended (or wanted) to support.
b) added branching code anywhere that requires now()
I think OOP is exposing serious flaws in how humans write software and solve problems. I'm not sure, but it seems there is an impedance mismatch in how we write code, and how we'd like to test code, but there is a reluctance or outright defiance toward the idea of writing testable code.
The FP community doesn't seem to have a problem so much with testing simply because they mostly write pure functions that are data in -> data out and that is the easiest thing in the world to test.
It occurs to me that most developers aren't writing testable OO code, and thus we have a never-ending argument about the most clever way to test (or not test) code that is not easily tested in isolation.
If we never get around to writing testable code, we aren't going to have systems that are easy to test. If we don't have easy to test systems, testing is going to suck and people won't do it.
I don't know that FP is the answer, but just asking people to write better OO systems isn't working. The problem certainly is more fundamental than simply TDD or mocking or whatever language or tool we are using.
The problem is in how we look at solving problems in code and how that relates to system maintenance over time. It's a people problem, not a tooling one.
The FP community doesn't seem to have a problem so much with testing simply because they mostly write pure functions that are data in -> data out and that is the easiest thing in the world to test.
It occurs to me that most developers aren't writing testable OO code, and thus we have a never-ending argument about the most clever way to test (or not test) code that is not easily tested in isolation.
If we never get around to writing testable code, we aren't going to have systems that are easy to test. If we don't have easy to test systems, testing is going to suck and people won't do it.
I don't know that FP is the answer, but just asking people to write better OO systems isn't working. The problem certainly is more fundamental than simply TDD or mocking or whatever language or tool we are using.
The problem is in how we look at solving problems in code and how that relates to system maintenance over time. It's a people problem, not a tooling one.
I don't think there is enough of an FP community doing large, heavily tested systems to allow accurate generalisations like that. I can think of plenty "We used FP, and everything went better than expected!" articles, but you probably don't hear from the people who tried, but came across problems.
I agree, and I don't know that FP is the solution.
I just know that the OO design approach isn't working super well with the people who are using it, and we now have a couple decades where design hasn't progressed significantly on the average project.
So, how to we change the human behavior that is leading to less than ideal outcomes?
I just know that the OO design approach isn't working super well with the people who are using it, and we now have a couple decades where design hasn't progressed significantly on the average project.
So, how to we change the human behavior that is leading to less than ideal outcomes?
While I think I get where the OP is going, this seems just too easy to claim without any real-life code. So let's try this then. The last time I used mocks: we're interfacing with motor control hardware.
In code we have interfaces for axis configuration, axis movement and so on. All implemented for different controllers. On top of that there's a layer defining specific movements. Say you want to go from point a to b while avoiding c, there's a piece of code that calculates a route (so it needs axes configurations for max velocity/range/...) and another piece that applies this route to a set of axes (so it needs axes to drive).
Apart from dragging around a bunch of motors everywhere I code, how am I supposed to properly test those two pieces without any mocks? Telling me the design is all wrong to start with is fine, but then you also have to come up with a better one of course :). Telling me that route calculation shouldn't depend directly on axis configuration but should use it's own configuration type and I should then add a conversion between the two is also no solution as it a) introduces an extra layer and b) still requires a mock to test the conversion, d'uh :)
In code we have interfaces for axis configuration, axis movement and so on. All implemented for different controllers. On top of that there's a layer defining specific movements. Say you want to go from point a to b while avoiding c, there's a piece of code that calculates a route (so it needs axes configurations for max velocity/range/...) and another piece that applies this route to a set of axes (so it needs axes to drive).
Apart from dragging around a bunch of motors everywhere I code, how am I supposed to properly test those two pieces without any mocks? Telling me the design is all wrong to start with is fine, but then you also have to come up with a better one of course :). Telling me that route calculation shouldn't depend directly on axis configuration but should use it's own configuration type and I should then add a conversion between the two is also no solution as it a) introduces an extra layer and b) still requires a mock to test the conversion, d'uh :)
There's a time and place for various tools. Mocking isn't all good, it's not all bad. Sometimes it's a good tool for accomplishing an immediate goal. Sometimes it's a crutch that encourages poor design. Keep an open mind and make evidence-based decisions. Only Siths deal in absolutes.
"Do or do not. There is no try". Absolutely.
I don't understand how you can be 100% against mocking. When I want to visualize testing, I often find it useful to think about how it's applied in the real world. For example, when building a car, the parts are tested in isolation (for the most part). Individual parts are tested - size, shape, stress, etc. Groups of parts are also tested in isolation. For example, the engine is actually connected to a mock drivetrain, mock computer, mock user inputs, etc. I see that as useful, and I think the same goes for software. It can simply be too costly in terms of time, bandwidth, etc to test against real resources.
Good points... but what happens when you walk into a legacy code base with few tests that wasn't designed to be tested and is tightly coupled to a component such as a database?
I dislike the "that's not supposed to throw an exception" developer statement, if something is not supposed to throw an exception under normal operating you can almost guarantee that it will throw exceptions in the demo. So one of the few cases where I use lots of mock is in Exception Handling.
Some exceptions are almost impossible to generate consistently in tests, or can't be simulated easily next to test that are supposed to pass when that exception state doesn't occur - think API failures, databases being yanked mid-query, etc.
Some exceptions are almost impossible to generate consistently in tests, or can't be simulated easily next to test that are supposed to pass when that exception state doesn't occur - think API failures, databases being yanked mid-query, etc.
I had a random idea the other day to create a program that monitors/logs system level calls and lets you replay them for a program. This would effectively let you use your program as normal, log any actual pieces that use external resources, and then automatically use the saved results as part of your test framework by overriding those system calls at the appropriate time.
Is anyone aware of a solution like this? I certainly don't have time to create it but I would definitely find it useful.
Is anyone aware of a solution like this? I certainly don't have time to create it but I would definitely find it useful.
We have something like this at the company I work for. We have a "simulation layer" which wraps ALL third party code. This lets us simulate the failure of file system calls, network drivers... And we use it to write our tests.
I did this for a gdb wrapper I wrote. It logs all the stdin/stdout between itself and gdb and between itself and the user. You can then take that log file and replay it to simulate both the user input and the gdb output.
Although not quite the same, VCR for Ruby does this for network requests.
IIRC, SoapUI does something similar.
TFA says "For example, when building SportStats v2 at the BBC we thought we needed a database for persistence. But we didn't build it in until we needed it, and we didn't mock it out either. We waited until the code told us that we now needed a database. It never did"
So they seem to be saying "if you write code upfront before you need it, you might not need it, therefor mocking is bad"
IMHO, that is not a valid conclusion to draw.
So they seem to be saying "if you write code upfront before you need it, you might not need it, therefor mocking is bad"
IMHO, that is not a valid conclusion to draw.
I agree with you, that seems to be what the author is saying, and his conclusion doesn't follow. Also, I'm not sure if he really means mocking or any kind of test doubles.
I thought everyone agreed that writing code upfront before you need it is always a bad idea, especially specific implementations like a database.
I thought everyone agreed that writing code upfront before you need it is always a bad idea, especially specific implementations like a database.
So, a lot of ideas are thrown here without any real use cases.People say "mocks are bad,whitebox testing is bad".But how can I unit test anything without mocks?
let's take javascript.I cant generate mocks from my code,i have to write them.I have this data access object that takes a connnection.
the obvious thing here is to spy on a hypothetic connection.query, fake a db call and return a predictable result.I cant do anything without a mock.
let's take javascript.I cant generate mocks from my code,i have to write them.I have this data access object that takes a connnection.
var dao=new Dao(connection);
dao.getById(1,function(err,res){ console.log(res) });
the obvious thing here is to spy on a hypothetic connection.query, fake a db call and return a predictable result.I cant do anything without a mock.
Mocks shouldn't be the first resort; they're there as a way to allow you to test code that you can't improve right now. I wouldn't want to use them on new code, but as a stepping-stone for testing and improving a large existing codebase they have their place.
That’s not what they’re for! Mocks allow you to design the interface of a new object by implementing that object’s collaborators before you implement the object itself. This is a very high-fidelity source of information about what the interface should look like; anything else is just (educated?) guessing.
That may not be what they're "for" (wtf does that even mean?) but they're a damn good use of mocks. You can't fix everything right now.
> That may not be what they're "for" (wtf does that even mean?)
It means that “[mocks are] a way to allow you to test code that you can't improve right now” and “[mocks are] a stepping-stone for testing and improving a large existing codebase” is a mischaracterisation of the role and benefits of mocks.
If using them like that is useful for you then that’s great, but it’s misleading to say that that’s why “they’re there”.
Steve Freeman & Nat Pryce’s book [0], for example, is an excellent illustration of what mocks are for.
[0] http://www.growing-object-oriented-software.com/
It means that “[mocks are] a way to allow you to test code that you can't improve right now” and “[mocks are] a stepping-stone for testing and improving a large existing codebase” is a mischaracterisation of the role and benefits of mocks.
If using them like that is useful for you then that’s great, but it’s misleading to say that that’s why “they’re there”.
Steve Freeman & Nat Pryce’s book [0], for example, is an excellent illustration of what mocks are for.
[0] http://www.growing-object-oriented-software.com/
A mock is overkill for that case; a stub or a dummy implementation (e.g. in-memory datastore) suffices when the interface with the collaborators is small and well-defined, as it should be when making a new design. The case where you need a full-on mock object is, IME, only when the collaborator is not decoupled enough from the primary object.
I don’t understand what this means.
The difference between a stub and a mock is about the object under test, not the collaborators that are being replaced with doubles. We use a mock when we want to test the object’s interactions with its collaborators; we use a stub when we don’t care about the interactions, only about the values that result from them.
What does that decision have to do with whether the interface is small and well-defined?
The difference between a stub and a mock is about the object under test, not the collaborators that are being replaced with doubles. We use a mock when we want to test the object’s interactions with its collaborators; we use a stub when we don’t care about the interactions, only about the values that result from them.
What does that decision have to do with whether the interface is small and well-defined?
I'm new to mocking (to serious unit testing, really), is mocking functions of 3rd-party libraries and/or web API's recommended? It sounds like a good idea, especially for functions that need to hit the network for a web API, but maybe I'm missing something?
I would say it's generally a bad idea to mock 3rd party libraries. It's code that's not under your control and testing with a mock which may not match the actual behaviour could cause you trouble. In the case of network requests, something like VCR[1] helps a lot.
[1] https://github.com/vcr/vcr
[1] https://github.com/vcr/vcr
I would disagree. If you want to unit-test your code that depends on those 3rd party libraries, then your choices are to:
1) Use the real thing. This can be slower and a lot less reliable.
2) Capture some typical output of the 3rd party api, and make a mock that emits it.
Which technique is useful? Both are. But the second one gives you fast-running fine-grained repeatable unit tests so therefore is your first line of testing.
So you discover in an integration test (#1) that the live api has a behaviour that the mock doesn't, maybe an occasional malformed response due to an error at the other end. Great, capture it and add it to the mock. Now you can reproduce it at will and write fast tests on how you handle it.
1) Use the real thing. This can be slower and a lot less reliable.
2) Capture some typical output of the 3rd party api, and make a mock that emits it.
Which technique is useful? Both are. But the second one gives you fast-running fine-grained repeatable unit tests so therefore is your first line of testing.
So you discover in an integration test (#1) that the live api has a behaviour that the mock doesn't, maybe an occasional malformed response due to an error at the other end. Great, capture it and add it to the mock. Now you can reproduce it at will and write fast tests on how you handle it.
I was thinking more of the case where the 3rd party library does no network communication. If it still takes a long time to execute, that may be a concern. But if the output is complex, then mocking it may be more trouble than it's worth.
For network requests to an API, your option #2 is exactly what I was suggesting.
For network requests to an API, your option #2 is exactly what I was suggesting.
I'm not familiar with VCR, but it looks like it's a little similar to https://docs.angularjs.org/api/ngMock/service/$httpBackend
There's a lot of value in coarse-grained mocks like this for testing the system; I also find value in finer grained mocks for testing smaller components in isolation. But neither of these techniques is the only valid one.
And neither of them forces you to write things upfront that you don't need later; which is what the original article seems to imply.
There's a lot of value in coarse-grained mocks like this for testing the system; I also find value in finer grained mocks for testing smaller components in isolation. But neither of these techniques is the only valid one.
And neither of them forces you to write things upfront that you don't need later; which is what the original article seems to imply.
VCR seems similar to $httpBackend. The main difference is that it automatically records the HTTP response from the target server the first time tests are executed. Then it allows you to reuse that response in future test runs to save having to manually mock responses.
Anyway, sounds like we're more or less agreeing :)
Anyway, sounds like we're more or less agreeing :)
3) Define a Gateway interface (e.g. DataStoreGateway) that presents exactly the API you want to consume; write one implementation of it that speaks to your third-party API (e.g. ODBCDataStoreGateway), and another that doesn't do much (e.g. MemoryDataStoreGateway).
Note that a trivial gateway-implementation, like MemoryDataStoreGateway, isn't a mock! It's a fully-functional component that works perfectly well from its consumers' perspectives. But, unlike the version that consumes a third-party component, it doesn't have nearly any failure modes that would confuse your tests. It just does what it does, simply, and lets your tests test what you're trying to test.
And note that when you stop mocking, and limit yourself to switching out fully-fledged gateway implementations, "dependency injection" stops being this huge pain with Factories and heavily-parameterized initializers. Instead, you can just make your objects discover their collaborators through a service registry. Your unit tests just register the trivial implementations in the service registry on setup().
---
† Don't get a bad taste in your mouth thinking about Spring here. A simpler, healthier example is Erlang's "global" module (http://erlang.org/doc/man/global.html).
Note that a trivial gateway-implementation, like MemoryDataStoreGateway, isn't a mock! It's a fully-functional component that works perfectly well from its consumers' perspectives. But, unlike the version that consumes a third-party component, it doesn't have nearly any failure modes that would confuse your tests. It just does what it does, simply, and lets your tests test what you're trying to test.
And note that when you stop mocking, and limit yourself to switching out fully-fledged gateway implementations, "dependency injection" stops being this huge pain with Factories and heavily-parameterized initializers. Instead, you can just make your objects discover their collaborators through a service registry. Your unit tests just register the trivial implementations in the service registry on setup().
---
† Don't get a bad taste in your mouth thinking about Spring here. A simpler, healthier example is Erlang's "global" module (http://erlang.org/doc/man/global.html).
> But, unlike the version that consumes a third-party component, it doesn't have nearly any failure modes that would confuse your tests.
That's all fine until you need to test handling the third-party component's failure modes.
> Note that a trivial gateway-implementation, like MemoryDataStoreGateway, isn't a mock!
Probably not. Though it depends at which point you insert the "trivial" components. If you're using a trivial component to unit test a collaborating class that isn't at the edges of the system, then that's a mock and it's helpful. It's not the only helpful technique or even the usual one, but "mocks are never helpful, always avoid them" is silly thing to say.
A mock store for one unit test is far more lightweight than a MemoryDataStoreGateway, and allows an easy way to simulate server errors, so sometimes it is a better thing to code.
> This is what "dependency injection" is really about, actually: not inserting mock-objects ... but rather allowing you to plug in ... versions of your collaborators.
It can be about both. They are not exclusive.
> Instead, you can just make your objects discover their collaborators through a service registry
You may be happy with every object reaching out to a global singleton. But I'm going to stay well away. Thanks but no thanks.
That's all fine until you need to test handling the third-party component's failure modes.
> Note that a trivial gateway-implementation, like MemoryDataStoreGateway, isn't a mock!
Probably not. Though it depends at which point you insert the "trivial" components. If you're using a trivial component to unit test a collaborating class that isn't at the edges of the system, then that's a mock and it's helpful. It's not the only helpful technique or even the usual one, but "mocks are never helpful, always avoid them" is silly thing to say.
A mock store for one unit test is far more lightweight than a MemoryDataStoreGateway, and allows an easy way to simulate server errors, so sometimes it is a better thing to code.
> This is what "dependency injection" is really about, actually: not inserting mock-objects ... but rather allowing you to plug in ... versions of your collaborators.
It can be about both. They are not exclusive.
> Instead, you can just make your objects discover their collaborators through a service registry
You may be happy with every object reaching out to a global singleton. But I'm going to stay well away. Thanks but no thanks.
"Global" is a matter of perspective. You can pass your objects a service registry to initialize themselves from, if you like. You can have as many service registries as you like.
The difference, then, from passing them their pre-initialized collaborators directly, is that if they need to create new collaborators on their own, they can ask the service registry what the current implementation they should be instantiating is. (It's basically a change in perspective from having objects that spawn specialized, short-lived objects, to having long-lived objects that can deal with changing conditions around them. Think about, say, Docker containers using links vs. etcd discovery. It's nice to not have to bring down your webserver just because your DB moved to a new IP address.)
Also, if you were thinking about my Erlang example specifically when you said "global singleton object" -- Erlang, despite the name, doesn't have one global registry, either. Instead, Erlang's process registry is per-node -- and nodes are the boundary where you deploy a release. In fact, ignoring the pragmatic OS-threading and networking implications, an Erlang node is almost exactly a "switchboard" on which services are plugged into one another, and an Erlang release is almost exactly a declaration of the current connections on such a switchboard. Need a place where your service is registered to X instead of Y? Bring up a node where that's true.)
The difference, then, from passing them their pre-initialized collaborators directly, is that if they need to create new collaborators on their own, they can ask the service registry what the current implementation they should be instantiating is. (It's basically a change in perspective from having objects that spawn specialized, short-lived objects, to having long-lived objects that can deal with changing conditions around them. Think about, say, Docker containers using links vs. etcd discovery. It's nice to not have to bring down your webserver just because your DB moved to a new IP address.)
Also, if you were thinking about my Erlang example specifically when you said "global singleton object" -- Erlang, despite the name, doesn't have one global registry, either. Instead, Erlang's process registry is per-node -- and nodes are the boundary where you deploy a release. In fact, ignoring the pragmatic OS-threading and networking implications, an Erlang node is almost exactly a "switchboard" on which services are plugged into one another, and an Erlang release is almost exactly a declaration of the current connections on such a switchboard. Need a place where your service is registered to X instead of Y? Bring up a node where that's true.)
When you're testing the third-party component's failure modes, the tests you're writing are functional tests belonging to the nontrivial gateway implementation (e.g. ODBCDataStoreGateway).
The important thing about the gateway interface is that it's an error boundary: you don't need to think about "server errors" when dealing with a DataStoreGateway; DataStoreGateways don't have server errors. The ODBC object the ODBCDataStoreGateway holds might hand it a server error, but it won't propagate that back to you. It'll just put out a plain† DataStoreGateway::InsertError or somesuch.
† The exception should probably hold a copy of the implementation-exception that triggered it, but this is just for the sake of debugging the implementation. The client should never unwrap the internal exception.
The important thing about the gateway interface is that it's an error boundary: you don't need to think about "server errors" when dealing with a DataStoreGateway; DataStoreGateways don't have server errors. The ODBC object the ODBCDataStoreGateway holds might hand it a server error, but it won't propagate that back to you. It'll just put out a plain† DataStoreGateway::InsertError or somesuch.
† The exception should probably hold a copy of the implementation-exception that triggered it, but this is just for the sake of debugging the implementation. The client should never unwrap the internal exception.
> you don't need to think about "server errors" when dealing with a DataStoreGateway; DataStoreGateways don't have server errors.
I'm not sure why you even make that distinction. More than once I've dealt with servers that occasionally time out and fail; (say once a week or so). I wish to test what I show on my UI under those conditions, and what part of the text and status codes returned from the remote server that represent (or whatever) its failure I will choose to display.
Doing so without using a mock seems pointlessly obtuse and roundabout when I can make a mock to insert whatever error response I want at any interesting point in my code.
I'm not sure why you even make that distinction. More than once I've dealt with servers that occasionally time out and fail; (say once a week or so). I wish to test what I show on my UI under those conditions, and what part of the text and status codes returned from the remote server that represent (or whatever) its failure I will choose to display.
Doing so without using a mock seems pointlessly obtuse and roundabout when I can make a mock to insert whatever error response I want at any interesting point in my code.
Split that problem into two problems:
1. Your collaborator cannot be reached.
2. Your client has a network connectivity problem.
I've never seen a client who needs to know #1. All they care about is that there was a DataStoreGateway::TransientInsertError, and that TransientInsertErrors mean retrying and notifying the UI that their work hasn't been committed yet.
#2 isn't a problem the DataStoreGateway should be responsible for, because it will affect almost everything in the system in one way or another, and you don't want to have to write that code repeatedly. It's more likely a kind of Alarm--the kind of thing that gets pushed out from wherever it is to a global alarm handler, which then does something like putting the UI into a different root view-controller state, e.g. darkening it and covering it with a modal saying "lost connection."
1. Your collaborator cannot be reached.
2. Your client has a network connectivity problem.
I've never seen a client who needs to know #1. All they care about is that there was a DataStoreGateway::TransientInsertError, and that TransientInsertErrors mean retrying and notifying the UI that their work hasn't been committed yet.
#2 isn't a problem the DataStoreGateway should be responsible for, because it will affect almost everything in the system in one way or another, and you don't want to have to write that code repeatedly. It's more likely a kind of Alarm--the kind of thing that gets pushed out from wherever it is to a global alarm handler, which then does something like putting the UI into a different root view-controller state, e.g. darkening it and covering it with a modal saying "lost connection."
Actually no, it was not either of those cases, it was what I said. The APIS that I'm working with aren't that much like a DataStoreGateway.
Your design points may be valid, but they don't really talk about mocks; why bend over backwards to avoid testing using a mock?
Your design points may be valid, but they don't really talk about mocks; why bend over backwards to avoid testing using a mock?
For something like a web API I think most people would support mocking. If it's just a library it's a lot less clear-cut; my approach would be to use the real thing until it gets to be a problem (because it's making your tests slow, or because it doesn't expose the right interface, or because it's too hard to get it into the state that you want to test). Certainly use mocks when you need to, but the rest of the time there are arguments for both sides and you probably need to figure out what you prefer yourself.
[deleted]
[deleted]
Was hoping to read an admonition against excessive snark and bullying in Hacker News comments. Ah well.
Mocking helps with feature isolation. If a test failed you can be sure it was because of the use case of the test.
Mocking helps when you are calling into code that you can't control. E.g, when calling into an API that is broken and sometimes returns incorrect results.
I don't write very many tests anymore. I haven't mocked an object in years. I also have a lot of bugs in my code.
Mocking helps when you are calling into code that you can't control. E.g, when calling into an API that is broken and sometimes returns incorrect results.
I don't write very many tests anymore. I haven't mocked an object in years. I also have a lot of bugs in my code.
Mocking helps with feature isolation.
By enabling isolation in the test, mocking reduces the pressure to design more isolated code.
Mocking helps when you are calling into code that you can't control.
Pervasive mocking helps isolate you from the pressure to remove (deep) dependencies on code that you can't control. Redesign your code so it only interacts with such code at the edges. At those places, create APIs that are independent of the code you can't control, and adapt that code to fit your APIs.
By enabling isolation in the test, mocking reduces the pressure to design more isolated code.
Mocking helps when you are calling into code that you can't control.
Pervasive mocking helps isolate you from the pressure to remove (deep) dependencies on code that you can't control. Redesign your code so it only interacts with such code at the edges. At those places, create APIs that are independent of the code you can't control, and adapt that code to fit your APIs.
> By enabling isolation in the test, mocking reduces the pressure to design more isolated code.
Having just written some tests (with mocks) today, I have read your comment several times and failed to extract any meaning whatsoever from it. It seems like the kind of comment that's counter-intuitive because it's just nonsense. Mocking reduces isolation by enabling isolation? Um.
> Redesign your code so it only interacts with such code at the edges.
Yes, and those edges can then be mocked when you want to test the rest of the code. Apis, interfaces etc. are a good fit to mocking. Mocking helps isolation.
TFA's argument seems to be that they ended up not needing a database, and if they'd have "used mocking" they would have started by mocking the database, therefor mocking encourages bad design. Nope, starting with things that you don't need yet encourages bad design. It doesn't matter if you're mocking those things or not.
Having just written some tests (with mocks) today, I have read your comment several times and failed to extract any meaning whatsoever from it. It seems like the kind of comment that's counter-intuitive because it's just nonsense. Mocking reduces isolation by enabling isolation? Um.
> Redesign your code so it only interacts with such code at the edges.
Yes, and those edges can then be mocked when you want to test the rest of the code. Apis, interfaces etc. are a good fit to mocking. Mocking helps isolation.
TFA's argument seems to be that they ended up not needing a database, and if they'd have "used mocking" they would have started by mocking the database, therefor mocking encourages bad design. Nope, starting with things that you don't need yet encourages bad design. It doesn't matter if you're mocking those things or not.
> Pervasive mocking helps isolate you from the pressure to remove (deep) dependencies on code that you can't control. Redesign your code so it only interacts with such code at the edges. At those places, create APIs that are independent of the code you can't control, and adapt that code to fit your APIs.
Yes... and that process you describe can be accomplished easily using mocks or stubs (depending on what you wish to test).
Yes... and that process you describe can be accomplished easily using mocks or stubs (depending on what you wish to test).
"Redesign your code so it only interacts with such code at the edges."
yeah, and what if you don't have time to do this? sometimes mocks are appropriate because of schedules and other priorities. Much better to mock than to not write tests because mocking is not pure.
yeah, and what if you don't have time to do this? sometimes mocks are appropriate because of schedules and other priorities. Much better to mock than to not write tests because mocking is not pure.
Quit? Before it makes you quit?
If your schedules and other priorities are against fixing broken code, I'd be leery of your organization's long-term health. If you're in for the long haul--and at my current gig I certainly am--then eliminating bad code (and any code as tightly coupled as what you describe) is the only sane option. Because it'll kill you eventually.
If your schedules and other priorities are against fixing broken code, I'd be leery of your organization's long-term health. If you're in for the long haul--and at my current gig I certainly am--then eliminating bad code (and any code as tightly coupled as what you describe) is the only sane option. Because it'll kill you eventually.
[deleted]
Isolated unit testing is supposed to improve your designs. But when you mock everything, you end up with tests that know everything about your implementation (white box testing). The result is that refactoring breaks your tests. Your tests no longer document how to use the system; they are merely a mirror of its implementation.
Kent Beck's central point in his book is that TDD mitigates fear, allows refactoring, and gives you immediate feedback. Extensive mocking makes developers fearful to refactor (thus hurting design) and reduces the quality of feedback. Not to mention it makes your tests hard to refactor too.
Martin Fowler explains the difference between "classical" and "mockist" testing:
http://martinfowler.com/articles/mocksArentStubs.html#Classi...