Is TDD Dead?(martinfowler.com)
martinfowler.com
Is TDD Dead?
http://martinfowler.com/articles/is-tdd-dead/
A series of conversations between Kent Beck, David Heinemeier Hansson, and Martin Fowler on the topic of Test-Driven Development (TDD) and its impact upon software design.
45 comments
My experience with TDD is that it is very helpful from a design perspective because it gets you to think about the code before you write it. How is the API going to work? What should I name this method? As an industry we use to dedicate too much time to this (RUP, etc). Now, we don't think about it enough.
As far as actually improving code quality... not so much. For the money, I would much rather see an hour of functional test development rather than an hour of unit test development (and I put TDD in the unit test category). I think functional tests provide more value and our industry has put way too much emphasis on unit tests.
100% unit test coverage says nothing about code quality. 100% functional coverage does. Functional tests also tend to be less brittle – which is a nice bonus.
As far as actually improving code quality... not so much. For the money, I would much rather see an hour of functional test development rather than an hour of unit test development (and I put TDD in the unit test category). I think functional tests provide more value and our industry has put way too much emphasis on unit tests.
100% unit test coverage says nothing about code quality. 100% functional coverage does. Functional tests also tend to be less brittle – which is a nice bonus.
Haven't seen the supposed benefit here as well.
I tried it as an experiment. With 100% coverage, TDD everything.
Code still fails in production. That is because unit tests detract from running functional/integration tests. Database, connections, hardware all mocked and each little cog is the right size. Put the cogs together into a gearbox and it shits the bed.
The one thing TDD helped was to find exact code breakage easier. Functional tests give higher level (less precise) error, maybe "something failed in the backend" then one has to look at logs to see what happens and such. But I'll take that over a false sense of security.
So am definitely not against unit tests, I am all for having a mix of both.
Another way to look at it, unit tests are a bit like drugs. It is easy to get addicted, create 10000 tests and a little micro-burst of pleasure hormone every time you see the dot on the screen for a passing test, it just feels good. But it is important to realize that is what it is (maybe I am writing all those tests just as a therapeutic exercise...), as it might not translate immediately to a ship-able working product.
I tried it as an experiment. With 100% coverage, TDD everything.
Code still fails in production. That is because unit tests detract from running functional/integration tests. Database, connections, hardware all mocked and each little cog is the right size. Put the cogs together into a gearbox and it shits the bed.
The one thing TDD helped was to find exact code breakage easier. Functional tests give higher level (less precise) error, maybe "something failed in the backend" then one has to look at logs to see what happens and such. But I'll take that over a false sense of security.
So am definitely not against unit tests, I am all for having a mix of both.
Another way to look at it, unit tests are a bit like drugs. It is easy to get addicted, create 10000 tests and a little micro-burst of pleasure hormone every time you see the dot on the screen for a passing test, it just feels good. But it is important to realize that is what it is (maybe I am writing all those tests just as a therapeutic exercise...), as it might not translate immediately to a ship-able working product.
I test your way, but if you're not aware of it you might want to watch this talk called Integration Tests are a Scam: http://www.infoq.com/presentations/integration-tests-scam It may explain why the "gearbox shit the bed" and how to fix that problem.
If you want a talk that preaches to the/our choir, check out this one: http://vimeo.com/68375232
It's interesting how different they are. Mutually exclusive in fact. But is one wrong, both or neither?
If you want a talk that preaches to the/our choir, check out this one: http://vimeo.com/68375232
It's interesting how different they are. Mutually exclusive in fact. But is one wrong, both or neither?
> Integration Tests are a Scam: http://www.infoq.com/presentations/integration-tests-scam
Thank you, that was interesting! A lot of good points. I agree with some of them and don't agree with other parts of it.
First, let's start with this basic principle -- "customers don't care if I have 1M unit tests or 0 unit tests". They just don't. They will complain and not like my product if it keeps crashing or is showing other pathological behavior. First and foremost it is important to ship, and to ship working code as far as customers are concerned. Taken from that side, I want to ensure that when the system is setup in customer's configuration (and we do know it, in our line of work), then we want to stimulate the system as close as possible to what a customer would do. If that doesn't break and is behavior properly. We gain immediately a large degree of confidence because it is closer to "doesn't break when customer uses" use case.
Unit tests also promise that, and they would also promise that "if I have unit-tested my code well, there is a higher chance that the code will not break when it gets into customers' hands". However notice that this is further from being true than the first one.
The problem with integration tests as discussed in the talk it they take a long time to run and give imprecise results. When something is broken I might not be able to quickly zoom in on it. I can't experiment as often as I would like because it take too long to discover precise mistakes. And I agree with those, but, I still maintain it is better to have integration tests because once you get into a stable steady state and all of the sudden one commit breaks the tests it usually is pretty clear what broke, in the worst case just revert the commit or examine in detail until it works. That helps with the "I don't know what broke, something did" part. Usually that something is the one commit where tests started failing.
Yeah, it would be better to also have complete 100% unit test coverage as well and zoom in immediately on the problem. No doubt that 100% unit test coverage also costs a lot of time and money and I posit it is often un-necessary.
Another criticism, perhaps I misunderstood the presenter, it seems to me that the talk focuses more on machine-local, object oriented code. Today the systems are more and more distributed. In a pure, local, single threaded, object oriented code (say a 3D modeler or Photoshop).The behavior or objects is more predictable and it is easier to write what he calls "focused tests". Now introduce threads and well that starts to become hairy. Add in multiple machines (a distributed system) and it gets even more complicated. There is a whole slew of strange problems. Often involving consistency, timeouts, networks failure, partitions, hardware peculiarities. Un-expected input that is generated maybe because a front end service is broken and back-end write might never expect that case and crashes. Unit tests will catch basic errors but it will mask those harder to find those kind of errors.
One more criticism is psychological and it is the false sense of security. And I have seen this happen from those that advocate unit testing everything 100%, when their code breaks, they simply don't believe their code could break because it has 100% coverage and they "just ran it and it passed". Well in production it crashes. And customers don't pay for passing tests they pay for a working product. Anyway I am repeating myself at this point and this is longer rant, so, sorry about that.
Thank you, that was interesting! A lot of good points. I agree with some of them and don't agree with other parts of it.
First, let's start with this basic principle -- "customers don't care if I have 1M unit tests or 0 unit tests". They just don't. They will complain and not like my product if it keeps crashing or is showing other pathological behavior. First and foremost it is important to ship, and to ship working code as far as customers are concerned. Taken from that side, I want to ensure that when the system is setup in customer's configuration (and we do know it, in our line of work), then we want to stimulate the system as close as possible to what a customer would do. If that doesn't break and is behavior properly. We gain immediately a large degree of confidence because it is closer to "doesn't break when customer uses" use case.
Unit tests also promise that, and they would also promise that "if I have unit-tested my code well, there is a higher chance that the code will not break when it gets into customers' hands". However notice that this is further from being true than the first one.
The problem with integration tests as discussed in the talk it they take a long time to run and give imprecise results. When something is broken I might not be able to quickly zoom in on it. I can't experiment as often as I would like because it take too long to discover precise mistakes. And I agree with those, but, I still maintain it is better to have integration tests because once you get into a stable steady state and all of the sudden one commit breaks the tests it usually is pretty clear what broke, in the worst case just revert the commit or examine in detail until it works. That helps with the "I don't know what broke, something did" part. Usually that something is the one commit where tests started failing.
Yeah, it would be better to also have complete 100% unit test coverage as well and zoom in immediately on the problem. No doubt that 100% unit test coverage also costs a lot of time and money and I posit it is often un-necessary.
Another criticism, perhaps I misunderstood the presenter, it seems to me that the talk focuses more on machine-local, object oriented code. Today the systems are more and more distributed. In a pure, local, single threaded, object oriented code (say a 3D modeler or Photoshop).The behavior or objects is more predictable and it is easier to write what he calls "focused tests". Now introduce threads and well that starts to become hairy. Add in multiple machines (a distributed system) and it gets even more complicated. There is a whole slew of strange problems. Often involving consistency, timeouts, networks failure, partitions, hardware peculiarities. Un-expected input that is generated maybe because a front end service is broken and back-end write might never expect that case and crashes. Unit tests will catch basic errors but it will mask those harder to find those kind of errors.
One more criticism is psychological and it is the false sense of security. And I have seen this happen from those that advocate unit testing everything 100%, when their code breaks, they simply don't believe their code could break because it has 100% coverage and they "just ran it and it passed". Well in production it crashes. And customers don't pay for passing tests they pay for a working product. Anyway I am repeating myself at this point and this is longer rant, so, sorry about that.
> The problem with integration tests as discussed in the talk it they take a long time to run and give imprecise results.
I think you left out the main complaint: It's impossible to cover a majority of code paths.
I think you left out the main complaint: It's impossible to cover a majority of code paths.
Whenever I tried to learn TDD, it only seemed relevant for back-end testing (eg REST APIs). But is there anyone left who is strictly a back-end guy? Most programmers now bill themselves as full-stack app developers, and probably spend at least half their time working on the front-end/UI.
I hadn't heard of functional testing before. From a stackoverflow answer:
> Functional Tests are written from the user's perspective. They ensure that the system is functioning as users are expecting it to.
This makes much more sense to me than TDD. TDD is writing functions which test other functions. Are the users writing code, and passing arguments to function calls? If not, then time spent on TDD isn't adding much value for the users.
So what does it mean to write functional tests? I'd argue that writing modular React components is a form of functional testing. Devcards[1] is similar, and looks to me exactly like what would be considered functional testing. These two workflows feel much more natural than testing UIs methods like automated selenium browser interactions to generate screenshots.
1. http://rigsomelight.com/2014/06/03/devcards-taking-interacti...
I hadn't heard of functional testing before. From a stackoverflow answer:
> Functional Tests are written from the user's perspective. They ensure that the system is functioning as users are expecting it to.
This makes much more sense to me than TDD. TDD is writing functions which test other functions. Are the users writing code, and passing arguments to function calls? If not, then time spent on TDD isn't adding much value for the users.
So what does it mean to write functional tests? I'd argue that writing modular React components is a form of functional testing. Devcards[1] is similar, and looks to me exactly like what would be considered functional testing. These two workflows feel much more natural than testing UIs methods like automated selenium browser interactions to generate screenshots.
1. http://rigsomelight.com/2014/06/03/devcards-taking-interacti...
> Whenever I tried to learn TDD, it only seemed relevant for back-end testing (eg REST APIs). But is there anyone left who is strictly a back-end guy? Most programmers now bill themselves as full-stack app developers, and probably spend at least half their time working on the front-end/UI.
yes?
> So what does it mean to write functional tests? I'd argue that writing modular React components is a form of functional testing. Devcards[1] is similar, and looks to me exactly like what would be considered functional testing. These two workflows feel much more natural than testing UIs methods like automated selenium browser interactions to generate screenshots.
no.
yes?
> So what does it mean to write functional tests? I'd argue that writing modular React components is a form of functional testing. Devcards[1] is similar, and looks to me exactly like what would be considered functional testing. These two workflows feel much more natural than testing UIs methods like automated selenium browser interactions to generate screenshots.
no.
As to code quality, the closest I can come to agreeing with you is that I think some people do unit tests ritualistically, not thoughtfully, and those suites can be relatively low in value. That brittleness you describe is often a sign of that. I rarely see that from people doing actual TDD, though.
Where I part ways, though is the notion that an hour of activity X substitutes for an hour of activity Y. For any given fear you have about your code failing, you should pick the testing approach that yields the most risk reduction per unit of effort. I don't think we can externally judge the right choice for a given team and code base. As long as it's reasonably easy for people to write tests at a variety of levels (unit, integration, functional, end-to-end), and as long as they're committed to continuous improvement, I'm inclined to trust that they'll figure out the right mix for their project.
Where I part ways, though is the notion that an hour of activity X substitutes for an hour of activity Y. For any given fear you have about your code failing, you should pick the testing approach that yields the most risk reduction per unit of effort. I don't think we can externally judge the right choice for a given team and code base. As long as it's reasonably easy for people to write tests at a variety of levels (unit, integration, functional, end-to-end), and as long as they're committed to continuous improvement, I'm inclined to trust that they'll figure out the right mix for their project.
I share your perspective on this. I have also found that writing tests forces you to write testable code (goes without saying). Often times, if code is impossible to test and requires to you to mock and inject a bunch of different things just to write the test then you're design is probably no good. So, while this necessarily mean you have to do TDD, it's interesting to see that testable code and good design kind of line up.
> it's interesting to see that testable code and good design kind of line up
They can line up, but I'm not sure there's a causation. There are many different ways to write tests the wrong way. It's taken a lot of trail and error for me to get better at it (I can't even say I'm "great" yet). I'd have given up long ago if I wasn't a passionate developer. A less passionate developer would think the whole concept is BS and give up long ago if he took my path.
One of the first mistakes I made was using a mocking framework to test my implementation. The test read like, "first call this method, then call this, then call this". Needless to say it was worthless. I was just writing my code twice.
Then you can use mocks inside-out and end up writing code you don't even need to call, etc.
I'm sure there's many ways to do TDD right, but from in my personal experience, there's more ways to do it wrong.
They can line up, but I'm not sure there's a causation. There are many different ways to write tests the wrong way. It's taken a lot of trail and error for me to get better at it (I can't even say I'm "great" yet). I'd have given up long ago if I wasn't a passionate developer. A less passionate developer would think the whole concept is BS and give up long ago if he took my path.
One of the first mistakes I made was using a mocking framework to test my implementation. The test read like, "first call this method, then call this, then call this". Needless to say it was worthless. I was just writing my code twice.
Then you can use mocks inside-out and end up writing code you don't even need to call, etc.
I'm sure there's many ways to do TDD right, but from in my personal experience, there's more ways to do it wrong.
I don't know about your code quality assertion. My experience has been that it produces code right off the bat that is equivalent to code that has been through 1 or 2 rounds of functional testing. I ignore code coverage, though. It's not a good metric.
My personal experience is that any code that I write that I haven't tested has a bug in it. Usually a NPE/SegFault.
> and I put TDD in the unit test category
there's your problem. TDD with functional tests, only unit test when it's a complicated & important piece of domain/business logic.
there's your problem. TDD with functional tests, only unit test when it's a complicated & important piece of domain/business logic.
I believe that's called BDD. Behavior Driven Development. It has a separate name because TDD-ers kept making this mistake.
Functional tests + strong static functional language => win ?
Software architects arguing about methodologies without discussing the real problems and context where those methodologies are applicable - because they can't show that code. Instead they use some fuzzy general notions which don't communicate any relevant information.
I don't understand - how is this useful? Can anything be learnt from this at all? Its very clear to me that no useful conclusions can be reached from debating like this - is it just me?
The only substantial content was a comment on the gist [1] that explores a problem that looks like it came from a production system. Methodology discussions would really benefit from at least one semi-realistic example like that.
[1]: https://gist.github.com/dhh/4849a20d2ba89b34b201#comment-122...
I don't understand - how is this useful? Can anything be learnt from this at all? Its very clear to me that no useful conclusions can be reached from debating like this - is it just me?
The only substantial content was a comment on the gist [1] that explores a problem that looks like it came from a production system. Methodology discussions would really benefit from at least one semi-realistic example like that.
[1]: https://gist.github.com/dhh/4849a20d2ba89b34b201#comment-122...
Was TDD ever really alive? I have a hard time getting less technical bosses to accept automated unit testing at all, without having them just reduce it to a % coverage metric that I not only have to report, but also satisfy annual goals. For them, TDD is a potential source of new buzzwords.
TDD is one of those things that only works when developers have a high degree of control over their own work. Anecdotally, that has been hard for me to find.
I think the only way to get widespread and useful adoption is to actually construct a high-level language around it, such that the code and the test are tied together by syntax, in the same file, and coverage is enforced by the compiler.
Otherwise, you're stuck trying to enforce TDD via voluntary human cooperation. While you may have confidence in writing and refactoring your code, that just shifts the uncertainty that remains in development of a complete product onto the screening of new hires and avoidance of certain management edicts.
Essentially, you need a fully automated test cop.
TDD is one of those things that only works when developers have a high degree of control over their own work. Anecdotally, that has been hard for me to find.
I think the only way to get widespread and useful adoption is to actually construct a high-level language around it, such that the code and the test are tied together by syntax, in the same file, and coverage is enforced by the compiler.
Otherwise, you're stuck trying to enforce TDD via voluntary human cooperation. While you may have confidence in writing and refactoring your code, that just shifts the uncertainty that remains in development of a complete product onto the screening of new hires and avoidance of certain management edicts.
Essentially, you need a fully automated test cop.
For the environment you describe in first sentence, no good software methodology could live. It's just not possible (unless you ignore them / break "rules") to create good software with incompetent, meddling, pointy heads like that.
That sort of environment is depressingly common outside of the major tech hubs. My current employer just celebrated its millionth line of code on a glorified CRUD+reports application. I have been reprimanded for suggesting better development practices. Obviously, I am trying to find a better job that still pays reasonably well for a 40-hour work week.
Damn - has anyone suggested to them to perhaps pick a non-LOC based milestone to celebrate at? Or perhaps was the announcement aimed more at non-technical folks elsewhere in the company?
I used to be TDD fanatic with java junit + spring unit tests and I went for 100% test coverage .
However, in my Clojure code, I feel 100% test coverage is unnecessary. REPL does a good job of proving my code works as intended. I wonder if I am doing it wrong!
However, in my Clojure code, I feel 100% test coverage is unnecessary. REPL does a good job of proving my code works as intended. I wonder if I am doing it wrong!
A REPL does a good job of proving your code works as intended now. If you save that code in a file and run it regularly (which is a fine way to start making a test suite), then you can prove your code still works later.
That's especially important when multiple people work on a project: you have some chance of remembering what important REPL incantations prove a system works. They won't. If you haven't put your REPL magic in a test suite, then they're going to break your stuff.
That's especially important when multiple people work on a project: you have some chance of remembering what important REPL incantations prove a system works. They won't. If you haven't put your REPL magic in a test suite, then they're going to break your stuff.
The Doctest approach (https://docs.python.org/2/library/doctest.html) is kind of a way of formalizing the REPL – you still essentially have a REPL, but you have the interaction written down in such a way that you can re-run it. Often you can copy-and-paste your REPL interaction into a document and then it's a test. It's a little more work because the computer doesn't know what "close enough" is – so something that includes a timestamp, for instance, causes failures (since it always changes) unless you annotate the expected output a little.
I write tests not to convince myself my code is correct (it often is, and repl and ad-hoc testing are more than enough to make sure it does the right thing now) but to prevent myself & future others from breaking it as they maintain it in the future.
The problem with that style of testing at the repl is it proves your code worked for those tests at that time, but unless you turn that into test cases that will keep on being run then there's a good chance that you'll forget one of those test cases in the future.
Trust me, a large, old, code base like that is terrifying to work with.
Don't obsess over code coverage though, concentrate on areas of risk and scenarios, I've written code with 100% coverage that still had an exciting range of bugs, and I've seen extremely robust code that had relatively low coverage overall but had tests focusing on the really tricky areas.
Trust me, a large, old, code base like that is terrifying to work with.
Don't obsess over code coverage though, concentrate on areas of risk and scenarios, I've written code with 100% coverage that still had an exciting range of bugs, and I've seen extremely robust code that had relatively low coverage overall but had tests focusing on the really tricky areas.
Unit tests are only partly to prove your code works when you initially write it, though. They also have significant value in allowing others—or Future You—the confidence to touch nearby code without feeling scared of breaking things. This is very important in Java and similar languages; in Clojure I suspect¹ this is less important just because functional styles lend themselves better to obviously-correct code.
¹Disclaimer: I have no experience with Clojure, working primarily in Scala these days. I do aim for good unit test coverage in Scala just the same as I did when I worked in Java.
¹Disclaimer: I have no experience with Clojure, working primarily in Scala these days. I do aim for good unit test coverage in Scala just the same as I did when I worked in Java.
I defense of unit tests, if REPL is not automated then you are losing that aspect. Yes the first time your write something it works fine to manually run things in a REPL. But next time you refactor or a coworker updates your code, you want to be able to run the tests and find what has happened.
You are choosing to not unit test a dynamic language project?
In part IV DHH says that he believes the code should be the master reference over the documentation or the tests. The problem with that is there can never be a bug because current behaviour is by definition the reference behaviour.
[deleted]
I can't find the summary. This appears to be a link to the videos themselves.
You need to click the 'more...' button at the bottom of each topic to see the summary.
Wow, thanks for that....what a terrible UI presentation.
Specifying invariants via tests before or during the writing of code is a great technique.
Making your whole development approach blindly revolve around one technique is self-evidently a poor idea.
Making your whole development approach blindly revolve around one technique is self-evidently a poor idea.
Well. It's... it's good he calls it "Summary", because if he called it "Brief" it'd be very ironic, correct?
Like any experienced developer I have a lot of respect for Martin Fowler and his work, but sometimes I wonder if his best skill is saying simple things in many, many, many words.
Like any experienced developer I have a lot of respect for Martin Fowler and his work, but sometimes I wonder if his best skill is saying simple things in many, many, many words.
Well, could anybody sum this up? Because I'm definitely not going to read Part 2 since it seems Fowler didn't even bother to produce correct sentences.
Automated tests are awesome but they aren't useful in all cases and it doesn't always make sense to write them first.
INSERT STUPID ATTENTION GRABBING HEADLINE
I disagree. Betteridge's law is useful.
I think you meant to say: "Is Betteridge's law useful?"
It is a shitty generalization slandering all articles which (possibly poorly) chose a question as headline.
This is obviously an "article" where the author was legitimately interested in answering the question in the title. This is one of the cases where Betteridge's law does not apply.
Betteridge's law may be useful to remind people to watch out for poor journalism and fact checking. In that case, I propose that all articles shall henceforth use questions as titles because people should do that anyway.
This is obviously an "article" where the author was legitimately interested in answering the question in the title. This is one of the cases where Betteridge's law does not apply.
Betteridge's law may be useful to remind people to watch out for poor journalism and fact checking. In that case, I propose that all articles shall henceforth use questions as titles because people should do that anyway.
It's funny because author at the conclusions basically says that it is useful and people should use it hence the answer to question "Is TDD Dead?" is No.