Why I don’t use TDD or BDD(blog.kangasbros.fi)
blog.kangasbros.fi
Why I don’t use TDD or BDD
http://blog.kangasbros.fi/?p=30
28 comments
>>> Face the "evil" (whatever that means) side like a man.
What? It's not a battle or a problem. Sometimes you need to get something out, or you're writing a little throwaway bit of code. Sometimes that involves a tradeoff in future maintainability/pain. It shouldn't justify never testing, but I think it's a valid reason to not religiously follow TDD.
What? It's not a battle or a problem. Sometimes you need to get something out, or you're writing a little throwaway bit of code. Sometimes that involves a tradeoff in future maintainability/pain. It shouldn't justify never testing, but I think it's a valid reason to not religiously follow TDD.
But the article isn't about not religiously following TDD. It's about why "I don’t write any unit tests or specs". Testing isn't discontinuously one or the other. There's a huge, continuous spectrum in between the two extremes.
He said that integration tests cover the tests he needs and that unit tests don't add much, while being time consuming. And he never said one word about "spec".
The only real way there has ever been to make sure the computer is following your instructions, is to tell those instructions to it twice. (Non-inferential) static typing is about telling the computer the input and output types of your functions twice. Unit tests are about telling the computer the algorithm (or an equivalent, such as a direct mapping function from a subset of example inputs to pre-calculated outputs), twice. These are both just protoforms of Reliable Computing—the name for the actual software engineering that NASA and friends do, where they make the whole system twice (or more) and then run those systems in parallel to ensure they're always "voting" the same way.
But sometimes you don't need engineering; sometimes you just need a dirty hack. Hacking is basically the opposite of engineering: it espouses DSLs and macros that encode everything exactly once, it prides elegance and other aesthetic factors of code above engineering tolerances. And, of course, it's a lot more fun to do—so it gets people interested in your code, which can actually be the best thing, even more so than engineering it. Linux is a big hack, and because of that all sorts of people enjoy working with it. _why was an eminent hacker, and because of that his code (and tutorials) are still used when getting people interested in coding. None of this code is guaranteed to work, unlike code that has been engineered—but that's okay, in 99% of cases.
But sometimes you don't need engineering; sometimes you just need a dirty hack. Hacking is basically the opposite of engineering: it espouses DSLs and macros that encode everything exactly once, it prides elegance and other aesthetic factors of code above engineering tolerances. And, of course, it's a lot more fun to do—so it gets people interested in your code, which can actually be the best thing, even more so than engineering it. Linux is a big hack, and because of that all sorts of people enjoy working with it. _why was an eminent hacker, and because of that his code (and tutorials) are still used when getting people interested in coding. None of this code is guaranteed to work, unlike code that has been engineered—but that's okay, in 99% of cases.
To elaborate: you tell the computer a specification ("what to do"), you tell the computer an implementation ("how to do it"), and then you get the computer to check that the implementation conforms to the specification.
Summary: ``I don't get it.''
The parts of my code bases that are the most trouble and the hardest to work with and the parts that have the least testing have the greatest overlap.
I have some bad tests that fail sporadically, but I also have some tests that fail that indicate something is actually wrong and will show up in a production environment in a really, really bad way some day (speaking from experience).
If you're _why, I think you've got a good case for why not to write unit tests. Unfortunately, nobody is _why
The parts of my code bases that are the most trouble and the hardest to work with and the parts that have the least testing have the greatest overlap.
I have some bad tests that fail sporadically, but I also have some tests that fail that indicate something is actually wrong and will show up in a production environment in a really, really bad way some day (speaking from experience).
If you're _why, I think you've got a good case for why not to write unit tests. Unfortunately, nobody is _why
I'm ambivalent about testing. I've no fear of not writing tests for something small or scrappy. But.. when I've persevered, weeks into a project they've saved my ass more than once and I've been able to recognize situations where a bug would have previously forced me to spend hours debugging or to release broken software.
TDD isn't a one size fits all technique. But just because it takes time, doesn't always yield immediate benefits, and can leave a bad taste in the mouth at first, it doesn't mean you wouldn't ultimately benefit (even beer shares these negatives ;-)).
The great thing, though, is that there are so many ways to practice testing. If you don't want to go with full on "test first" TDD practices, even rolling out a handful of automatic integration tests could cover your most onerous painpoints (this is my strategy on old projects that have no tests - something's better than nothing). Though TDD's proponents foam at the mouth about always testing, it's not a binary option - you can wade in the shallow end and get into deeper waters over time as it begins to click.
TDD isn't a one size fits all technique. But just because it takes time, doesn't always yield immediate benefits, and can leave a bad taste in the mouth at first, it doesn't mean you wouldn't ultimately benefit (even beer shares these negatives ;-)).
The great thing, though, is that there are so many ways to practice testing. If you don't want to go with full on "test first" TDD practices, even rolling out a handful of automatic integration tests could cover your most onerous painpoints (this is my strategy on old projects that have no tests - something's better than nothing). Though TDD's proponents foam at the mouth about always testing, it's not a binary option - you can wade in the shallow end and get into deeper waters over time as it begins to click.
Here's my take on testing:
I'm the kind of guy who likes to get things working as fast as possible. When I'm excited about an app or feature idea, I can't be bothered with writing tests along the way, much less before I write my code. I get much more satisfaction out of writing code, running it in the browser, and testing it the old fashioned way. I build apps and features quickly and this usually makes me (or the client I'm building it for) very happy in the process.
I do, however, understand the value of testing. Unlike the OP I have seen real world cases where testing has been a godsend. I often feel guilty for not writing tests, even if it's just a toy app I'm developing for my own curiosity.
To balance these opposing tendencies of mine, I have taken a ATDD approach to testing, or "Adaptive Test Driven Development". That is, write tests for the most critical components of the site, such as anything dealing with money or sensitive information, and write tests for the parts of the app that get used the most, both by me when I'm developing them, and the users once it's deployed.
When I've gone to the browser for the 5th time to test a feature I'm developing, it's a pretty good sign that an automated test would be more efficient than my manual testing, so I set aside the time to write it. If I know there's a part of the app that errors will be completely detrimental to, I will write tests for that, if only for my own psyche.
To be completely honest, TDD/BDD and some of the rhetoric surrounding it seems a bit ivory-towerish - a way for the templars of hacking to distinguish themselves from the proletariat developers. If I had been told TDD is the only way I could develop a rails app when I first started doing it I probably would have given up pretty early on. So if TDD is getting in the way of your being productive, it kind of defeats the purpose of it even existing.
The important thing is to strike a balance that works for you, and keep an open mind about other developer's habits and tendencies.
I'm the kind of guy who likes to get things working as fast as possible. When I'm excited about an app or feature idea, I can't be bothered with writing tests along the way, much less before I write my code. I get much more satisfaction out of writing code, running it in the browser, and testing it the old fashioned way. I build apps and features quickly and this usually makes me (or the client I'm building it for) very happy in the process.
I do, however, understand the value of testing. Unlike the OP I have seen real world cases where testing has been a godsend. I often feel guilty for not writing tests, even if it's just a toy app I'm developing for my own curiosity.
To balance these opposing tendencies of mine, I have taken a ATDD approach to testing, or "Adaptive Test Driven Development". That is, write tests for the most critical components of the site, such as anything dealing with money or sensitive information, and write tests for the parts of the app that get used the most, both by me when I'm developing them, and the users once it's deployed.
When I've gone to the browser for the 5th time to test a feature I'm developing, it's a pretty good sign that an automated test would be more efficient than my manual testing, so I set aside the time to write it. If I know there's a part of the app that errors will be completely detrimental to, I will write tests for that, if only for my own psyche.
To be completely honest, TDD/BDD and some of the rhetoric surrounding it seems a bit ivory-towerish - a way for the templars of hacking to distinguish themselves from the proletariat developers. If I had been told TDD is the only way I could develop a rails app when I first started doing it I probably would have given up pretty early on. So if TDD is getting in the way of your being productive, it kind of defeats the purpose of it even existing.
The important thing is to strike a balance that works for you, and keep an open mind about other developer's habits and tendencies.
Use automated tests for any software where the probability of you working on it in the future is > 0.
Any testing is better than none.
Good testing is better than bad.
Good tests;
* Are relatively simple to write if the design is decent
* Make clear what is being tested
* Do not break if something unrelated changes
* Run quickly
Understand the difference between unit testing, and integration testing. Chances are, you're doing the latter, thinking you're doing the former, and wondering why it's so much work.
Lastly, know when TDD and BDD is appropriate.
For me, it is around the 25% mark; earlier, and I spend too much time fixing the tests as the design goalposts change, later, and I spend too much time fixing bugs that could have been prevented by tests.
Any testing is better than none.
Good testing is better than bad.
Good tests;
* Are relatively simple to write if the design is decent
* Make clear what is being tested
* Do not break if something unrelated changes
* Run quickly
Understand the difference between unit testing, and integration testing. Chances are, you're doing the latter, thinking you're doing the former, and wondering why it's so much work.
Lastly, know when TDD and BDD is appropriate.
For me, it is around the 25% mark; earlier, and I spend too much time fixing the tests as the design goalposts change, later, and I spend too much time fixing bugs that could have been prevented by tests.
It is very much dependent on the kind of code you write. If you write UI code then automated testing might not help you much. But for algorithmic code you can usually write short tests that test very comprehensively and catch nearly all bugs.
For example if you are writing a sorting function you write some code to generate a random array, sort it, and test whether it is sorted.
I do not see much value in the kind of testing a lot of people seem to do though, for example in a web application putting something in the database and then testing that it's there. If the test is about as complicated or more complicated than the thing it's testing that is a good sign the test is not very useful.
For example if you are writing a sorting function you write some code to generate a random array, sort it, and test whether it is sorted.
n = random int;
A = int[n];
for(i in 0..n) A[i] = random int;
sort(A);
for(i in 0..n-1) assert(A[i] <= A[i+1]);
I do not see much value in the kind of testing a lot of people seem to do though, for example in a web application putting something in the database and then testing that it's there. If the test is about as complicated or more complicated than the thing it's testing that is a good sign the test is not very useful.
The entire point of writing software is to automate processes that can be done manually, but are too time-consuming and/or hard to do so. Testing is a great example: yes, I could run through my entire application and hand-test it every time I make a change, or I could automate the process and save myself a ton of time and work.
Viewpoints like these are the reason we have a long way to go before software is truly reliable.
Viewpoints like these are the reason we have a long way to go before software is truly reliable.
The issue isn't about testing, it's about unit testing. And really more specifically about BDD/TDD.
And I must admit that I've struggled with TDD myself. And I've generally hit a couple of issues:
1) Bug rates in test code for some reason seem to be higher than that in production code. Not sure why. But the end result is a LOT more test bugs than product bugs. I've sat in meetings where people wanted to block shipping the product because of test bugs. No evidence there were any blocking issues in production code.
2) The opportunity cost of TDD style unit tests are rather high. Given the choice I think I'd rather a more in-depth design and code review.
While I think unit tests are valuable, I take a more sparse notion of them. Based on code review and design, unit test the code that you have worries about. Where you think there's a greater chance of a seemingly innocuous change breaking the code.
I've seen little value, in the projects I've been a part of, with the BDD/TDD approach. It does seem like a lot of busy work and not a lot thought put into where the value is. And there's actually pretty good industry data (and I suspect open source data as well) that its actually not hard to determine what parts of a program are going to be the buggiest in production. I'd really love to see a test approach that is density driven...
But with that said, if you're writing a 20,000 line web app, TDD/BDD is probably fine. You can generally kick out the code and get full test coverage in very short order.
And I must admit that I've struggled with TDD myself. And I've generally hit a couple of issues:
1) Bug rates in test code for some reason seem to be higher than that in production code. Not sure why. But the end result is a LOT more test bugs than product bugs. I've sat in meetings where people wanted to block shipping the product because of test bugs. No evidence there were any blocking issues in production code.
2) The opportunity cost of TDD style unit tests are rather high. Given the choice I think I'd rather a more in-depth design and code review.
While I think unit tests are valuable, I take a more sparse notion of them. Based on code review and design, unit test the code that you have worries about. Where you think there's a greater chance of a seemingly innocuous change breaking the code.
I've seen little value, in the projects I've been a part of, with the BDD/TDD approach. It does seem like a lot of busy work and not a lot thought put into where the value is. And there's actually pretty good industry data (and I suspect open source data as well) that its actually not hard to determine what parts of a program are going to be the buggiest in production. I'd really love to see a test approach that is density driven...
But with that said, if you're writing a 20,000 line web app, TDD/BDD is probably fine. You can generally kick out the code and get full test coverage in very short order.
One caveat: I come from the Ruby world, which has top-notch testing support. So maybe it's horrible in other-platform-land.
I think that it is slow going, at first. Testing is a different skill than regular coding, and at first it goes slow. But now, I spend much less time writing tests because I already know how to properly write them.
It's hard to properly evaluate the amount of time tests save you because it's an up-front investment. But I can't count the number of times I've wanted to change something fairly large in my codebase, and did so fearlessly, because then all my tests broke and I was able to quickly fix things where they were broken. In one recent project I decided to completely re-do the authentication system, and it only took me about an hour, because I already had an exact list of every place that needed attention, due to having great test coverage.
> It does seem like a lot of busy work and not a lot thought put into where the value is.
I see it the other way: if code has to be written, it should also be tested. If it's valuable enough to have spend the time to implement, it's worth making sure that it works correctly.
I think that it is slow going, at first. Testing is a different skill than regular coding, and at first it goes slow. But now, I spend much less time writing tests because I already know how to properly write them.
It's hard to properly evaluate the amount of time tests save you because it's an up-front investment. But I can't count the number of times I've wanted to change something fairly large in my codebase, and did so fearlessly, because then all my tests broke and I was able to quickly fix things where they were broken. In one recent project I decided to completely re-do the authentication system, and it only took me about an hour, because I already had an exact list of every place that needed attention, due to having great test coverage.
> It does seem like a lot of busy work and not a lot thought put into where the value is.
I see it the other way: if code has to be written, it should also be tested. If it's valuable enough to have spend the time to implement, it's worth making sure that it works correctly.
"But I can't count the number of times I've wanted to change something fairly large in my codebase, and did so fearlessly, because then all my tests broke and I was able to quickly fix things where they were broken "
To me that is a bit scary. Tests don't prove the absence of bugs. They can only prove the presence of them.
I never make big changes, even in the most tested pieces of code that have unit, integration, UI, and manual testing without some trepidation. Because the fact of the matter is that these code paths get the reputation they get because a customer comes back and says this doesn't work and we're like, "oh... that's how you use it?"
"I see it the other way: if code has to be written, it should also be tested. If it's valuable enough to have spend the time to implement, it's worth making sure that it works correctly."
I don't disagree. It should be tested. I think our disagreement is if TDD/BDD style unit testing is the best way to do so.
To me that is a bit scary. Tests don't prove the absence of bugs. They can only prove the presence of them.
I never make big changes, even in the most tested pieces of code that have unit, integration, UI, and manual testing without some trepidation. Because the fact of the matter is that these code paths get the reputation they get because a customer comes back and says this doesn't work and we're like, "oh... that's how you use it?"
"I see it the other way: if code has to be written, it should also be tested. If it's valuable enough to have spend the time to implement, it's worth making sure that it works correctly."
I don't disagree. It should be tested. I think our disagreement is if TDD/BDD style unit testing is the best way to do so.
> Tests don't prove the absence of bugs. They can only prove the presence of them.
Absolutely, and having a list of where they were (now) present was a real productivity boon.
It's true, I think we're arguing over something very small. I don't think automated testing is the end-all be-all. But I do think it's much much better than not doing it.
Absolutely, and having a list of where they were (now) present was a real productivity boon.
It's true, I think we're arguing over something very small. I don't think automated testing is the end-all be-all. But I do think it's much much better than not doing it.
I do want to be clear. That I'm not arguing against "automated testing". It's really against a style of unit test development that is all the rage: TDD/BDD. The devs I've worked who practice it, is through the use something like Cucumber to drive the development of the application with tons of unit tests.
I'm a huge advocate of automated testing. And again, just in my experience, the productivity and perceived code quality I saw coming out of the BDD/TDD side of the house didn't seem to be a step in the right direction.
I'm a huge advocate of automated testing. And again, just in my experience, the productivity and perceived code quality I saw coming out of the BDD/TDD side of the house didn't seem to be a step in the right direction.
I write unit tests so that I can refactor fearlessly. How does the author handle situations where he can either 1) put in a hack to fix a bug or handle a feature request, or 2) restructure the code to make it fit the modified problem description better? If you have unit tests you can do the latter, and feel good about what you have. If you don't have a testing infrastructure, you must do the first, because you have no confidence about how the behavior of the code might change and you have to be paranoid.
I don't understand how anyone can be fearless during refactoring. Suppose I refactor string search from the simple version (worst case time O(n*k)) to Knuth-Morris-Pratt (worst cast time O(n+k)). The API is unchanged, so this is an excellent example of a refactoring. But the KMP implementation has so many more edge cases where I might make a mistake that I wouldn't be comfortable using only the tests uses to drive the simple version.
I would be happier with code coverage-based tests, or better, branch coverage-based tests. That would help tell me which parts of my new implementations need test cases, although it wouldn't be sufficient.
Would you really be fearless for this type of refactoring? And how does following TDD practices help out? It seems to make the fallacy that refactoring is trivial and therefore doesn't need additional testing.
I would be happier with code coverage-based tests, or better, branch coverage-based tests. That would help tell me which parts of my new implementations need test cases, although it wouldn't be sufficient.
Would you really be fearless for this type of refactoring? And how does following TDD practices help out? It seems to make the fallacy that refactoring is trivial and therefore doesn't need additional testing.
On small projects where one has complete ownership and knowledge of the code, one can get by without testing. When you have a large project with tens to hundreds of developers, automated testing gives the developers the confidence and comfort to make large or deep changes with an accurate measure as to the overall stability of the product, even the areas where an individual developer may have little domain knowledge of.
Agreed. I submitted a couple of patches to Rails recently and I would have been far more reluctant to try to make fixes if I hadn't been able to verify them with a fairly extensive test suite.
People get too dogmatic about testing and particular testing methodologies but testing in general is a very powerful and useful tool when used wisely.
People get too dogmatic about testing and particular testing methodologies but testing in general is a very powerful and useful tool when used wisely.
> I just personally haven’t found any use for them in any of my projects or my clients’ projects.
I wonder how that person makes sure that what they deliver actually works. Testing by hand can only get you so far. And when the client calls a year later with some change requests, tests are a great way to know that you haven't fucked up something that used to work because you forgot a small implementation detail.
I wonder how that person makes sure that what they deliver actually works. Testing by hand can only get you so far. And when the client calls a year later with some change requests, tests are a great way to know that you haven't fucked up something that used to work because you forgot a small implementation detail.
I think that sometimes people lose sight of the fact that other people are different. You may not care for sports cars while someone else might. Doesn't make one of you superior to the other. Neither one of you can say that the other is wrong and should have made the choice you did.
It is the same thing with many programming methedologies, including unit tests. Maybe unit tests work well for large companies, and I would think they might be ideal for parts of certain types of server applications.
But I have tried the methodology, and found that it slows me down and produces lower quality code. For me, it produces a lot of false positives as the some number of unit tests become out of date each sprint, and the significantly increased overhead slows down development significantly which means less quality functionality per unit time gets developed. I've tested this theory with a real product written %100 solely by me, and found that the actual results in production where there were no unit tests and no design documents (but lots of design "notes") resulted in a shocking result: Not a single exception over a large deployment among thousands of customers and many months and only one reported bug which at the end of the day is actually working correctly but the design decision was not clearly thought out and the software does something the user doesn't expect in a corner case.
Here I see people talking about refusing to hire people who don't believe in Unit tests, and everywhere else on the planet I see people talking about Unit tests as if they are a religion, and as if nobody could write competent code without using them.
But realize this, every programmer is different, and the path to the highest quality code per unit time may not be via unit tests.
It is the same thing with many programming methedologies, including unit tests. Maybe unit tests work well for large companies, and I would think they might be ideal for parts of certain types of server applications.
But I have tried the methodology, and found that it slows me down and produces lower quality code. For me, it produces a lot of false positives as the some number of unit tests become out of date each sprint, and the significantly increased overhead slows down development significantly which means less quality functionality per unit time gets developed. I've tested this theory with a real product written %100 solely by me, and found that the actual results in production where there were no unit tests and no design documents (but lots of design "notes") resulted in a shocking result: Not a single exception over a large deployment among thousands of customers and many months and only one reported bug which at the end of the day is actually working correctly but the design decision was not clearly thought out and the software does something the user doesn't expect in a corner case.
Here I see people talking about refusing to hire people who don't believe in Unit tests, and everywhere else on the planet I see people talking about Unit tests as if they are a religion, and as if nobody could write competent code without using them.
But realize this, every programmer is different, and the path to the highest quality code per unit time may not be via unit tests.
This guy has everything backwards. I sort of hate writing tests, but still write them first most of the time. In the end, they catch so many bugs and save so much time, I would never go back to hacking code and point and click testing. The only way manual testing works is if you have an over-financed company with a huge QA department.
boring ...
I like it when people write blogs like this. One question to determine if it was bad satire or not and often times, the interview is over. Makes hiring so much easier.
>>> “Examples” of unit tests suck donkey balls every time.
So lack of good examples imply that the whole system is useless? Yes, an example of "assert True" is useless, but just download any of the thousands of open source projects that use unit tests and look at the code.
>>> They seem to be a great way to spend some extra time doing time consuming, not very hard work.
It's not that time consuming, really. And it being easy work is a good thing. A lot of benefit for easy work!
>>> You have to think about the “evil” business side too.
Running away from your problems is never a good idea. Face the "evil" (whatever that means) side like a man.
>>> Unit tests=more code
OK, there is a difference between useless code and good code. Unit tests doesn't make your system any more complicated or harder to read. In fact, it makes your system more stable and easier to read.
>>> That is mostly because I’m a lazy-ass bastard.
Ah, so we get to the bottom of the problem.