Unit testing is not (generally) useful(prehacked.com)
prehacked.com
Unit testing is not (generally) useful
http://prehacked.com/60388.html
15 comments
I agree. Unit tests are what make refactoring safe.
Adding known bugs to the unit test suite is a good way to prevent regressions.
Adding known bugs to the unit test suite is a good way to prevent regressions.
"preventing regressions" is a subset of "finding bugs", and I've found the other methods I listed to be more effective (from an ROI perspective) at doing that too.
Apparently you work at Justin.TV. I don't know how big your individual projects are, but personally, I didn't get unit-testing religion until I started working on really big stuff, close to the limits of what I'm capable of. I worked for a year on a 60,000 line program, and it started getting painful. I worked for another year on a 100,000 line program, and that's when I finally had to relent and start writing unit tests.
Here's a common scenario that made me want to write tests. I want to implement feature A, but to make it work, I'm first going to have to write library B and C. So I write library B, but there is no way to test it in the program yet, because feature A doesn't exist. Right after I've finished the code, I have a pretty good idea of where things might go wrong, and what corner cases I want to exercise. But by the time I finish library C and then get around to implementing feature A, I've forgotten that stuff. Pain points go unaddressed.
In summary, unit tests really start to show their value when you're working on a program that's too big to fit into your head all at once.
Here's a common scenario that made me want to write tests. I want to implement feature A, but to make it work, I'm first going to have to write library B and C. So I write library B, but there is no way to test it in the program yet, because feature A doesn't exist. Right after I've finished the code, I have a pretty good idea of where things might go wrong, and what corner cases I want to exercise. But by the time I finish library C and then get around to implementing feature A, I've forgotten that stuff. Pain points go unaddressed.
In summary, unit tests really start to show their value when you're working on a program that's too big to fit into your head all at once.
Couldn't you have tried to break that 100,000 line program up into ten 10,000 line programs? You might have found the additional benefit that each of the ten programs could be useful on its own. That's how I would usually attack this kind of problem, rather than throwing unit tests at it.
I think it would be preferable to do both. Have a few really solid and well contained modules that are well tested.
Testing is only a piece.
Testing is only a piece.
It seems like you're still using some form of automation to find bugs, regardless of whether these would be classified as unit tests, correct?
Sure, some of the monitoring systems I've written do some fairly detailed testing. For example, I wrote the Justin.TV chat system. There's a group of bots that periodically log in to the production chat server, send messages to each other, and email me if anything doesn't work.
That's certainly not unit testing though.
That's certainly not unit testing though.
I speak only for myself, but I would say the following: The real key is fully automated testing, no matter how it is done. The finer the grain, the better, generally. Matching an actual business case is near ideal.
Fully automated is necessary so you can run them frequently, automatically, near-continuously.
But beyond that, I think you get into religious territory, and I think people getting dogmatic about the definition of "unit test" tends to mistake definition for virtue (a common failing). If you've got automated testing, great! You win. Doesn't matter how it works.
Or, rather, it does matter, but only within your context, which nobody else is really competent to judge you on.
Fully automated is necessary so you can run them frequently, automatically, near-continuously.
But beyond that, I think you get into religious territory, and I think people getting dogmatic about the definition of "unit test" tends to mistake definition for virtue (a common failing). If you've got automated testing, great! You win. Doesn't matter how it works.
Or, rather, it does matter, but only within your context, which nobody else is really competent to judge you on.
Yeah, the only thing that matters to me is whether the pro-unit testing position has become more religious or whether the anti-unit testing position has become more religious. I'll take whatever seems serious. I liked what Joel Spolsky had to say yesterday but I disagree with the linked blog post today.
do they ever say anything interesting to eachother?
heh, no they're awfully dull - they just say things like "testing 123" and "received 'testing 123' from bot456".
Also, TDD/BDD are most useful as design practices - not as bug-finding practices. They help you think more carefully about your design and make it more modular and clear.
I agree entirely. Whenever I leave out writing unit tests, I end up with a web of messy interdependencies which ends up far too hard to extend, and will always need refactored as soon as I need to add the smallest feature.
Writing unit tests up front forces me to write the code in a way where inputs & outputs can easily be stubbed out, which in turn makes writing messy interdependencies simply too hard!
Basically, if I write with unit tests, it becomes easier to write maintainable code, and harder to write unmaintainable code.
Writing unit tests up front forces me to write the code in a way where inputs & outputs can easily be stubbed out, which in turn makes writing messy interdependencies simply too hard!
Basically, if I write with unit tests, it becomes easier to write maintainable code, and harder to write unmaintainable code.
>> "If not for my unit tests, I'd be scared to change much of anything, for fear of stirring up a whole bunch of trouble that I'll have to fix later on."
For me, if I had unit tests everywhere, I'd be scared to change much of anything, knowing that after the refactoring, and usual testing/checking things work, I'd then have to rewrite/fix unit tests.
For me, if I had unit tests everywhere, I'd be scared to change much of anything, knowing that after the refactoring, and usual testing/checking things work, I'd then have to rewrite/fix unit tests.
That's why I stopped for a while. Then I discovered BDD style testing (or context/specification, or...).
Whatever it's called, it's not quite as fine grained as a unit test and it's not as large in scope as integration testing. It's at the level of your business specs. So the only time you'll break your tests is if you've jacked up a spec.
Honestly, if my changes cause new behavior in the system but the people paying me won't care, then I don't care either.
Whatever it's called, it's not quite as fine grained as a unit test and it's not as large in scope as integration testing. It's at the level of your business specs. So the only time you'll break your tests is if you've jacked up a spec.
Honestly, if my changes cause new behavior in the system but the people paying me won't care, then I don't care either.
Unit testing can be this way, but it doesn't have to be. At a really low level, unit testing can be "testing the implementation", which may be useful while you're writing code, but isn't useful during a refactoring.
But slightly higher level tests can be written to almost never have this problem. Let's say you've written code to ensure that a CC transaction doesn't go through with an invalid number. No matter what the implementation looks like - no matter how much you refactor - this test should always work. The same is true for a wide range of useful tests. (These aren't strictly "unit" tests, but they fall into a similar category.)
My personal experience is that a good unit test suite is 99% useful when refactoring, and 1% painful.
But slightly higher level tests can be written to almost never have this problem. Let's say you've written code to ensure that a CC transaction doesn't go through with an invalid number. No matter what the implementation looks like - no matter how much you refactor - this test should always work. The same is true for a wide range of useful tests. (These aren't strictly "unit" tests, but they fall into a similar category.)
My personal experience is that a good unit test suite is 99% useful when refactoring, and 1% painful.
Judging by his analogy and depth of insight, I'm guessing he's never even tried to write unit tests on a daily basis and certainly never done TDD. It's the same old argument that everyone makes when they've never actually learned to do something properly.
Actually I wrote unit tests daily at my last company, for about two years, and in the last six months of that I gave TDD a try.
My claim isn't really that they don't work, it's that (compared to the other things I listed) they have a tiny ROI.
My claim isn't really that they don't work, it's that (compared to the other things I listed) they have a tiny ROI.
Unit tests have nothing to do with, and aren't in any way mutually exclusive to load testing and logging and user reports. Frankly, it just sounds naive to say that you replaced unit testing with logging, or load testing, or letting your users find the bugs for you.
People that do unit tests do all that stuff too!
Your lack of specs also has nothing to do with unit testing. It might dictate the difficultly in writing acceptance tests, but that's a completely different subject.
I don't feel the need to proselytize unit tests since you've already written them off, but what's the point to posting about why you don't like something only to back it up with some sort of rambling, incoherent analogy that you probably don't even believe?
People that do unit tests do all that stuff too!
Your lack of specs also has nothing to do with unit testing. It might dictate the difficultly in writing acceptance tests, but that's a completely different subject.
I don't feel the need to proselytize unit tests since you've already written them off, but what's the point to posting about why you don't like something only to back it up with some sort of rambling, incoherent analogy that you probably don't even believe?
I certainly do believe the analogy.
The point was that the ROI on unit tests is almost always very much lower than the ROI of the others. So when time gets tight (as it always is in a startup), you should generally put effort into all the other things and not into unit tests.
The point was that the ROI on unit tests is almost always very much lower than the ROI of the others. So when time gets tight (as it always is in a startup), you should generally put effort into all the other things and not into unit tests.
I'm currently writing a personal project for university, and I've made sure to test it all the way through.
When I was part-way into coding this thing, I took a day to go through all the code and write tests for anything I could. I found a few bugs, so I could at least say that it helped somehow. But the tests have helped a lot more when I added to it - it feels -good- to see a big green bar light up whenever I add something. It's -nice- to run all tests, leave it a minute, and confirm that it's not going to blow up before I have to demonstrate to anybody. Of course, finding bugs is nice, and I set up a better testing rig to make sure it doesn't fail on anything I haven't thought of, but the benefits of unit testing go further than bug detection.
When you're all crestfallen with no desire to code, finding bugs is only secondary.
When I was part-way into coding this thing, I took a day to go through all the code and write tests for anything I could. I found a few bugs, so I could at least say that it helped somehow. But the tests have helped a lot more when I added to it - it feels -good- to see a big green bar light up whenever I add something. It's -nice- to run all tests, leave it a minute, and confirm that it's not going to blow up before I have to demonstrate to anybody. Of course, finding bugs is nice, and I set up a better testing rig to make sure it doesn't fail on anything I haven't thought of, but the benefits of unit testing go further than bug detection.
When you're all crestfallen with no desire to code, finding bugs is only secondary.
It's a false sense of security though. It's like putting a big badge on your website saying it 'validates' - even if it's totally broken in real world browsers.
In my experience, the more common bugs are integration bugs, scaling bugs, weird memory usage issues, not to mention interfacing to external systems - for example some weird browser comes along and does idiotic requests you'd never expect a sane person to do.
Those things are very hard to test for, and don't really produce a good ROI as abstractbill says... Best just to log things, graph things where possible, spot weirdness, investigate and fix.
In my experience, the more common bugs are integration bugs, scaling bugs, weird memory usage issues, not to mention interfacing to external systems - for example some weird browser comes along and does idiotic requests you'd never expect a sane person to do.
Those things are very hard to test for, and don't really produce a good ROI as abstractbill says... Best just to log things, graph things where possible, spot weirdness, investigate and fix.
Everything you mentioned I count as part of TDD or BDD or what ever automated testing acronym you like.
What I suspect is that all of us have basically the same idea in mind, and what we're disagreeing over is mostly semantics and context.
Btw, all of my unit tests include memory usage validation. It's easy in c and c++, but I'm not sure how it would be done in a web development environment.
What I suspect is that all of us have basically the same idea in mind, and what we're disagreeing over is mostly semantics and context.
Btw, all of my unit tests include memory usage validation. It's easy in c and c++, but I'm not sure how it would be done in a web development environment.
If passing unit tests give you the motivation you need to work, that's great!
Personally, just making something people want is enough motivation for me ;-)
Personally, just making something people want is enough motivation for me ;-)
To me Unit Testing is more about testing the expected outcome and making sure of the status quo rather then finding bugs and testing all the extreme small edge cases. Then when an abnormal and unexpected situation occurs, writing a test that induces it and fixing the code so that all the related tests pass.
To use the doctor analogy, I would say unit testing is more about testing patient outcomes when they are put through varying courses of drugs or treatments (fixing bugs and adding new features) rather then trying to make them die before declaring them healthy. Is the patients blood pressure normal? Check. Now the patient wants to start some treatment for something else. Oh wait, I'm getting an abnormal blood pressure reading.
To use the doctor analogy, I would say unit testing is more about testing patient outcomes when they are put through varying courses of drugs or treatments (fixing bugs and adding new features) rather then trying to make them die before declaring them healthy. Is the patients blood pressure normal? Check. Now the patient wants to start some treatment for something else. Oh wait, I'm getting an abnormal blood pressure reading.
I didn't appreciate unit testing until I started doing development test-first in Smalltalk. You just sketch out the skeleton of your classes with instance variables and accessors. Then you write tests the way you want your objects to work (design your API by pretending to use it). Then run your tests, and marvel as the debugger stops on missing methods and lets you create them at that point and continue the test. It's a terrific way to get your code right the first time.
I don't know what he's using there at Justin.tv, but most languages that are more rigid (than Smalltalk or Ruby for example) are certainly not quite as nice for unit testing.
I don't know what he's using there at Justin.tv, but most languages that are more rigid (than Smalltalk or Ruby for example) are certainly not quite as nice for unit testing.
Sounds great! But in the meantime, I've put a couple of prototypes online, got a bunch of feedback, iterated, and figured out that nobody wanted the thing we designed in the first place. I've moved on to an evolution of the original idea, while you're still focused on getting the first idea to work "perfectly".
My point is that unit tests aren't cost-free, and their ROI is awful for most things.
My point is that unit tests aren't cost-free, and their ROI is awful for most things.
That's funny, cause I practice something I'll start calling "cost-free unit testing".
Step 1: If you're thinking about how a code api should act, sketch it out inside a test.
Step 2: If you're about to jump into a repl and poke at your code, open a unit test instead.
Now you've simply replaced other activites at no net cost to you. My core takeaway is that everyone tests their code somehow. You're best off if you can automate that process. Don't obsess over it (I almost never test my html/views), but try to find opportunity.
Step 1: If you're thinking about how a code api should act, sketch it out inside a test.
Step 2: If you're about to jump into a repl and poke at your code, open a unit test instead.
Now you've simply replaced other activites at no net cost to you. My core takeaway is that everyone tests their code somehow. You're best off if you can automate that process. Don't obsess over it (I almost never test my html/views), but try to find opportunity.
Unit tests don't take that long. If it takes you that much longer to write your unit tests, you must be doing something fundamentally wrong - like perhaps testing the wrong thing or focusing on the wrong kind of testing.
It is hard to make inline unit tests when the inputs and outputs are really complicated, like the request/reply data structures in a web framework.
Totally agree, that's why you need good MVC separation and you test those layers separately.
The models testing is obvious. What you're referring to there is controller testing. In Rails/Rspec, the way you do that is by mocking/stubbing the models (so you're only testing the controller logic). Then you pass in the parameters that would come in from the web, and check that the controller code (which should be fairly straightforward) behaves appropriately.
This article explains this approach quite clearly: http://www.patmaddox.com/blog/2007/9/15/easy-controller-test...
The models testing is obvious. What you're referring to there is controller testing. In Rails/Rspec, the way you do that is by mocking/stubbing the models (so you're only testing the controller logic). Then you pass in the parameters that would come in from the web, and check that the controller code (which should be fairly straightforward) behaves appropriately.
This article explains this approach quite clearly: http://www.patmaddox.com/blog/2007/9/15/easy-controller-test...
Whenever someone comments on the usefulness of unit testing (or in the extreme, testing in general), I point them to Feynman's contribution to the Challenger shuttle report:
Appendix F - Personal Observations on Reliability of Shuttle http://www.ralentz.com/old/space/feynman-report.html
A good summary is at http://duartes.org/gustavo/blog/post/richard-feynman-challen...
Appendix F - Personal Observations on Reliability of Shuttle http://www.ralentz.com/old/space/feynman-report.html
A good summary is at http://duartes.org/gustavo/blog/post/richard-feynman-challen...
How many people are writing mission critical software? In startups, hardly any.
If you rely on your software to be up and running for your livelihood (aka any web startup), it is mission critical.
I disagree completely. A failure might mean you loose a bit of ad revenue, or maybe a customer.
The consequences of a bug are usually very small. It's also very obvious if a bug is causing you pain.
The consequences of a bug are usually very small. It's also very obvious if a bug is causing you pain.
For me the primary use of unit testing is in the prototype stage. I write the tests instead of the application code to make sure everything is sane. I find lots of architectural/assumption problems this way.
After you've got real app code the tests serve more or less as regression tests to make sure I haven't broken anything major deep in the guts of my models and such.
In short, like some of the other comments here: Unit tests aren't for finding bugs, they're for helping you not write the bugs in the first place.
After you've got real app code the tests serve more or less as regression tests to make sure I haven't broken anything major deep in the guts of my models and such.
In short, like some of the other comments here: Unit tests aren't for finding bugs, they're for helping you not write the bugs in the first place.
It all depends on how your development works.
Let's look at a scenario where you are using an outside framework.
Not writing tests for your code? Framework v2.0 comes out and you want to use a great new feature in that version. Will your app work if you upgrade? Sure you'll have to test/QA your app regardless, but with tests you might have a better idea of where to look for problems.
This is what I'm facing at the moment as a single developer. The app has no test coverage so updating is a laborious process of testing everything by hand.
But getting into testing has it's own problems. The biggest I have with testing is when I read discussions about bugs within the test framework. How much time will you spend working around bugs in the testing framework?
Let's look at a scenario where you are using an outside framework.
Not writing tests for your code? Framework v2.0 comes out and you want to use a great new feature in that version. Will your app work if you upgrade? Sure you'll have to test/QA your app regardless, but with tests you might have a better idea of where to look for problems.
This is what I'm facing at the moment as a single developer. The app has no test coverage so updating is a laborious process of testing everything by hand.
But getting into testing has it's own problems. The biggest I have with testing is when I read discussions about bugs within the test framework. How much time will you spend working around bugs in the testing framework?
The article has a rather narrow perspective on unit testing, given his product is not mission critical. He's basically treating live production as a giant integration test.
Some of my perspectives:
class/function level unit testing is only needed for common library code.
component level unit testing is critical if you want to do any significant refactoring.
non trivial integration tests (especially timing sensitive multithreaded code) can find many subtle bugs that would be too late to find in production.
I've written a few tests (for a database engine) that paid themselves many times over.
Some of my perspectives:
class/function level unit testing is only needed for common library code.
component level unit testing is critical if you want to do any significant refactoring.
non trivial integration tests (especially timing sensitive multithreaded code) can find many subtle bugs that would be too late to find in production.
I've written a few tests (for a database engine) that paid themselves many times over.
The article has a rather narrow perspective on unit testing, given his product is not mission critical.
I do have a habit of assuming every other hacker is doing some kind of a startup, so this is a fair point.
I do have a habit of assuming every other hacker is doing some kind of a startup, so this is a fair point.
I'd say that it's also important once the team gets beyond a certain size; codebase and team size tend to scale together, of course, but not always. The value of tests in catching regressions in code you didn't write, didn't realize was there, and don't understand is immense. It's one kind of difficult to not introduce regressions in 100k lines of code that you were deeply involved with; it's a whole different deal to walk into 100k lines of someone else's code and try to change it without breaking anything.
Unit testing is like vaccination: it's good for me that I'm vaccinated against smallpox, but it's also really good for you that I won't be getting sick and giving it to you, and your ability to avoid vaccination (or testing) and still function is often directly correlated to how well everyone else is vaccinated (or how well they've tested their code so you know when you're breaking it).
Those don't always have to be unit tests in the strictest sense, so you could perhaps argue that a test that breaks as a result of unrelated changes isn't really a "unit" test because it wasn't properly isolated, but I really hate those sorts of language arguments. I'll call something a "unit" test if it's not an "end-to-end" test, even if I didn't mock or stub out every possible dependency. And it certainly sounds like the author of this post is arguing against unit tests in that sense rather than in the stricter sense.
Unit testing is like vaccination: it's good for me that I'm vaccinated against smallpox, but it's also really good for you that I won't be getting sick and giving it to you, and your ability to avoid vaccination (or testing) and still function is often directly correlated to how well everyone else is vaccinated (or how well they've tested their code so you know when you're breaking it).
Those don't always have to be unit tests in the strictest sense, so you could perhaps argue that a test that breaks as a result of unrelated changes isn't really a "unit" test because it wasn't properly isolated, but I really hate those sorts of language arguments. I'll call something a "unit" test if it's not an "end-to-end" test, even if I didn't mock or stub out every possible dependency. And it certainly sounds like the author of this post is arguing against unit tests in that sense rather than in the stricter sense.
It's a money make. We needed some python work done. It was a small project 42 hours 6 hours was unit testing. Unit testing increased the invoice by almost 17%. I can see why consultants like it.
I know many people who do unit testing (myself included), some of them consultants. Not a single one of them does it "to buff up the invoice". In fact, every single one of them will argue (sincerely) that good unit testing practices help reduce the time they need to deliver code and increase the quality of that code. They'd also argue that good unit testing practices make it easier to then hand over the code to someone else.
This is only valid if the code is not going to be maintained. If it is, and especially if someone other than the original developers are going to maintain it, then unit tests are worth their wait in gold.
This is another case of overgeneralizing based on limited experience. Automated unit testing may well be of limited value for building the applications that run a video sharing web site (where failures are no big deal). But now try building a complex library used as a core component of several packaged applications. The library developer doesn't really have a "program" to put through user testing. The only solution to keeping the library stable while refactoring and adding new features is to build a comprehensive suite of unit tests that exercise the library's public API.
If your unit testing isn't useful, then find a way to make it useful. Author misses the point completely.
Not unit testing is like building a car door and not check that it fits until all parts are to be assembled, and then reviewing what's wrong. Dumb.
Not unit testing is like building a car door and not check that it fits until all parts are to be assembled, and then reviewing what's wrong. Dumb.
When (or should I say if) JustinTV grows much bigger and hires a lot more programmers this whole unit tests are not useful thing will come back to bite them.
But I can see how right now it seems their time is better spent on other things.
But I can see how right now it seems their time is better spent on other things.
Out of interest, are you speaking from personal experience?
Yep, I got TDD religion when I started working for a huge company.
I did some testing, but not much when I worked for a startup.
I did some testing, but not much when I worked for a startup.
For a huge company, I'd say TDD/pair programming etc is also to ensure some productivity from bad programmers.
If one bad programmer writes the unit tests, and another bad programmer writes the code, then you might end up with a mediocre outcome, instead of a bad outcome. Same with pair programming.
Likewise, if you get a good programmer to write the unit tests, then give the programming work to a terrible programmer, given enough time, he'll write a working solution.
If one bad programmer writes the unit tests, and another bad programmer writes the code, then you might end up with a mediocre outcome, instead of a bad outcome. Same with pair programming.
Likewise, if you get a good programmer to write the unit tests, then give the programming work to a terrible programmer, given enough time, he'll write a working solution.
I'd have to say that two bad programmers are not any better then one bad programmer even with TDD.
But TDD benefits even the best programmers when they are dealing with a huge, ugly, old code base.
When on the other hand you're user testing every day or multiple times a day, like you might with a web startup, then TDD may only slow you down.
But TDD benefits even the best programmers when they are dealing with a huge, ugly, old code base.
When on the other hand you're user testing every day or multiple times a day, like you might with a web startup, then TDD may only slow you down.
When on the other hand you're user testing every day or multiple times a day, like you might with a web startup, then TDD may only slow you down.
These days I'm at a point where I can usually push 4 or 5 new releases into production every day. It's pretty awesome :-)
These days I'm at a point where I can usually push 4 or 5 new releases into production every day. It's pretty awesome :-)
Really it's like that bridge in 'Apocalypse Now'. The sergeant is telling Martin Sheen how the bridge is always bombed and they have to come in and re-build it, "just so the generals can say the bridge is open." Well, I have done a lot of unit testing (CPPUnit), and it is like that. We do unit testing so management can say the product has been unit tested. Coals to Newcastle, to be sure, but hey, it's a job!
But if you want to take the argument seriously, I would say that "finding bugs" is not the primary reason I write unit tests anyway. It's to prevent regressions, and make me feel safe that I haven't broken anything after a refactoring. If not for my unit tests, I'd be scared to change much of anything, for fear of stirring up a whole bunch of trouble that I'll have to fix later on.