So please, don’t fire your QA department just yet. Their job is still important, even if you unit test.
There is a vast difference between the gamut of possible automated tests one could write and what is known colloquially as a unit test. A number of different kinds of automated tests are written against frameworks that are built on top of unit testing frameworks. That doesn’t make them unit tests. It doesn’t make sense to call them unit tests. A square is a rectangle, but does that make every rectangle a square? An automated acceptance or integration test is subject to a completely different set of problems (in areas such as specification, maintenance, complexity) than a unit test. In fact, about the only thing they share is their lifecycle and execution model, which many times has been retrofitted into the JUnit lifecycle and execution model.
Your Unit Tests Lie to You
February 17, 2009 by Janusz Gorycki
Unit tests are a good tool for two purposes:

* they are a fast and simple verification vehicle, allowing you to test your unit’s responses to variety of external stimuli – either ones that you could think of while developing, or ones simulating conditions that led to a bug in your code (if these conditions can be reduced to a unit test, which may or may not be the case)
* they give you a cerating assurance that the code you have created for the unit will not be so easily broken in the future by somebody who is refactoring it for whatever reason (bug fixes, improvements, rearchitecting, whatever)

And that is pretty much it.

Ok, So How do They “Lie”?

Let’s try to answer the question of what purposes unit test are not useful for, or what are the dangers of relying on them, shall we?

* Unit test are not a necessarily a means for documenting your system, sorry. I have seen a fair amount of unit test that did, and an even larger pile of those that didn’t, instead increasing chaos and miscommunication. Tests will only become your system’s docs, if you create them with this specific purpose in mind. The same can be said about your actual code – it may or may not be self-documenting
* “Green” (all passing) unit tests don’t necessarily mean that your system is healthy. It is not humanly possible to test for every scenario, you can break your system in many funky ways and still make your even most stringent unit tests pass. Boy, do I have stories to tell about that one…
* Trusting your unit test mocks is wrong. Mocks are only able to superficially mimick the real world. Claiming that mocks can substitute the real environment is like claiming that a blowup doll is a good approximation of a marriage
* They are worthless if they are slow. In a non-trivially large system, unit testing can take hours (if not more) – and your automated build will take the same amount of time. This makes unit tests worthless as a vehicle for instant feedback – nobody will bother waiting half a day for the results of a build after they committed some refactored code. Yes, you can and should partition your code to smaller modules, but then your builds do not test everything – they only test your module

Don’t Unit Test Anymore… No, Really!

by Chris Wash on Feb.17, 2009, under Java, Software Engineering

I just read Your Unit Tests Lie to You by Janusz Gorycki and I was going to leave a comment there, but thought it was more appropriate to expand my comments off into their own thing. For those that haven’t read the article, its basic premise is to grab hold of the nearest “test infected” reader and shake the warm and fuzzy out of them. It paints the short sightedness of many recent “unit testing” converts as living in a dream world where unit tests should replace formal testing. It follows with many sentiments I’ve read (and written about here) for a while now. It’s not that I disagree with what is being said in the article, or its tone for that matter; most of what is being said is spot on. Unit testing is definitely not a silver bullet. If you read my blog often, you no doubt get that. The article ends:

So please, don’t fire your QA department just yet. Their job is still important, even if you unit test.

So to Janusz, the fundamental problem here is a general ignorance of the purposes behind a unit test suite. I agree 100% that’s the primary factor behind his problem. What don’t we agree on? Semantics. But semantics are important! How far do we have to go for a true zen-understanding of this issue? Not far. Indulge me –
When is A Unit Test is not a Unit Test?

Here’s my thesis: you may use a unit testing framework, but what you write are developer tests. Even if they are technically unit tests, it is against everyone’s interest to call them this. Picky, useless distinction, you say? Hear me out.

There is a vast difference between the gamut of possible automated tests one could write and what is known colloquially as a unit test. A number of different kinds of automated tests are written against frameworks that are built on top of unit testing frameworks. That doesn’t make them unit tests. It doesn’t make sense to call them unit tests. A square is a rectangle, but does that make every rectangle a square? An automated acceptance or integration test is subject to a completely different set of problems (in areas such as specification, maintenance, complexity) than a unit test. In fact, about the only thing they share is their lifecycle and execution model, which many times has been retrofitted into the JUnit lifecycle and execution model.
“Unit Testing” and Linguistic Drift

I’ve recently seen a number of different incarnations (1, 2, 3) of the argument that we should eschew use of the term “best practice” because of the implications of its linguistic drift and general propensity of people to turn off their brains when just spoon-fed answers, not having to experience deriving the solution for themselves. Similarly, the popularity of unit testing frameworks and the sheer frequency with which the term has been used have, in a sense, set the idea of developer testing back considerably. Cedric Beust makes the point that in many cases we’ve confused [developer] testing terminology with JUnit terminology, and TestNG was in part a response to that. Here we oversimplify the problems we choose to bite-off and the goals we strive toward. They’re not realistic. Is it no wonder so many people fall flat when trying to adopt “unit testing”?

While there may be some overlap with the goals of validation and verification, most in the know consider the true benefits of “unit testing” to be a totally different animal altogether. We seem surprised to find the benefits of doing developer testing have little to do with what “testers” do. This dischord causes a lot of confusion, and has sparked a lot of articles. Some draw this conclusion, appropriately, that developer testing, while it fits a rigid definition of what testing is, shares little with what a typical “tester” is responsible for (true V&V). Quite often developers are the only ones doing any automation work, including this automated “developer” testing. Developers tend to do it for all kinds of different reasons, too. We’ll use a suite of automated tests to proceed without fear of integration errors. That adds value totally independent of validation and verification practices. If our suite catches regressions before we hand a single version off to testers, that saves both developers and testers time. I could go on and on, but the term “unit testing” conveys very little of these kinds of benefits to the development lifecycle, and as it turns out, causes a great deal of confusion.
The Right Usage

When we’re talking pure unit tests, that is, black-box testing components in pure isolation (which requires isolated dependencies), not in concert, as developers we can certainly find merit in this practice. But what we’ve learned is that this idea of unit testing, while quite beneficial as a development practice, shares very few goals with “testing” as we know it from a classical definition of the word (end-to-end V&V). Perhaps a few years ago this distinction was not that true, but we now know better. We know the danger in trying to bend unit tests into something they’re not.
http://agilesoftwaredevelopment.com/blog/janusz-gorycki/your-unit-tests-are-useless
http://cwash.org/2009/02/17/dont-unit-test-anymore-no-really/

knowing how to use a unit testing framework, a very simplistic construct on the surface, to do all of these wonderful things is certainly not something that just “falls out” of developing a unit of code – it’s not something we should expect developers to just deliver each iteration as part of their deliverable like it’s “done.” Effort invested to develop and maintain test code alongside the code under test is not free.

Even if you are a master programmer, you have no idea what to test for. There, I said it. And I have a war story to back me up:
One of the best developers in our team has created a fine piece of code, unit tested the hell out of it and was proud as a baby of his accomplishment. And a fine piece it was indeed – chapeau bas! Yet – it failed “in the field”. Its problem was performance – it failed to be efficient in the face of real-life conditions. And the developer in question could’t quite believe his eyes whan somebody else on the team investigated the problemand pointed out where exactly in his code the root cause of the problem was – it was a classical case of “this one is obviosly correct, it is not necessary to test it”. So if this bloke is unable to create proper unit test suite (and his software development skillz are absolutely top-notch), nobody can.

So please, don’t fire your QA department just yet. Their job is still important, even if you unit test.