Saturday, February 25, 2012

Unit tests that give you code coverage, but don't test anything

Back in November/2011, I wrote about how high code coverage may not be enough. Code coverage, when it's actually triggered by tests that actually test something, may not be enough as it doesn't test all potential paths (like the ones in a regular expression) but they can actually mean something.

This time, it is about a way more serious issue: unit tests do not test anything at all, but guarantee code coverage.

Let me give you a very simple example:
public static int Sum(int a, int b)
    return a + b;
Very simple method. Now, let's say we write this test:
public static void TestSum()
    Program.Sum(100, 200);
I have the desired code coverage, but what are we actually testing here?

I saw many different variations of such tests, what compelled me to write this post. I believe we had enough arguments before against anyone who claimed that the code had quality because it does have some measure of code coverage.

If this is the metric management values, and nobody is actually reviewing test cases as they should, this is very likely to happen. Pretty much everybody is happy, as the tests pass and the team has a high code coverage. But this is self-deceiving on the product quality, at the least - assuming of course that this was not done intentionally.

When I look back at the different organizations I worked for, I believe this is more likely to happen when other engineers write tests than the original author. For instance, if a junior engineer is assigned to increase the code coverage, he may do it without actually testing anything. One of the reasons is that s/he may not know anything about the code - while another reason is just that this engineer is striving for the valued metric: code coverage.

Another fact that contributes to this kind of practice is that reviewers tend to pay more attention to the product code than to the tests themselves. So these things slip. I even admit that for a long time I assumed that engineers knew what they were doing when they tested and never looked that carefully.

At the end of the day, this kind of work wastes company's and engineer's time and don't accomplish anything. This goes back to my point about not deceiving oneself about quality (as written in "Be serious when providing a web service").

UPDATE: See new post "Code coverage and missed requirements".
Post a Comment