Happy Path Tests

Someone posted an article to the relevant company internal news group at which I was consulting, referring to JUnit Antipatterns and happy path testing. Another colleague replied and started off by saying, "You'll find that he only uses the term 'bad test' in the conclusion."

He continues by saying the author states that "Writing good tests is the hard part," implying that many of the easy tests may be bad tests and goes on to show illustrative examples. He could define 'bad test' as 'not(good test)' I suppose.

After referring to the section Easy Tests my colleague quotes the article, "The result is lots of passing tests that don't exercise the system, which leads to a misrepresentation of the code's health," and this is a bad outcome which need not be spelled out.

He goes on to say that he can only assume that the author was referring to tests that don't work. e.g. 1000 assert(true)'s. "That is, it's hard to say whether he's considering a 'happy path' test to be a *bad* test. You can't be serious. You did read the examples and make such a trivial assertion(asserting(1000 assert(true)'s))) as satire of some sort, right?"

Well no, actually.

The author is quite explicit and says, "Actually, happy path tests aren't an antipattern. The antipattern is when the developer stops at happy path tests."

After a couple of examples showing why "happy path tests" are necessary but are not themselves sufficient for adequate testing, the author states, "The point is, it isn't difficult to get a test to pass coincidentally," and this is a bad outcome as well.

I disagree with the assertion that "One single test of one component that tests the 'Happy Path' gives a massive, massive gain in confidence in the system. Even if it only tests 5% of the system in one ideal case it actually *proves* something about the behaviour of the system, where before you knew absolutely nothing."

Such tests prove nothing and in fact are worse than having no tests because such a view engenders a false sense of security and confidence in the system that is misplaced. One counter-example, quoted directly from the article:

One happy path test verifies that Factorial.eval(3) returns 6.

public class Factorial {
  public int eval(int _num) {
      return 6;
  }
}

How's that for a false positive? If you've never encountered test-driven development, you might object that no one would write such a simple-minded implementation. One of the practices of test-driven development (TDD) is to write the tests first, then "do the simplest thing that could possibly work" -- in this case, return 6.

In answer to my other colleague I choose to borrow from the legal profession, "If the facts are on your side, bang on the facts. If the law is on your side, bang on the law. If neither the facts nor the law is on your side, bang on the table."

Good tests demonstrate a requirement (user or discovered, user-actor or controller-actor) of the system, usually asserting on an invariant of the class. I disagree that tests are especially useful for construction and other static tests, and these are taken care of by the compiler and linker anyway, by default giving us the underhyped benefit of static-type safety.

The place for static setup is in, you guessed it, setUp(), so the test fixture is not polluted by such detail.

So what do we test in the fixture itself? Dynamic behaviour. Specifically, scenarios of unit tests, or bits of user stories, to test that the software works as expected. From the junit cookstour, "We found that the most challenging application of JUnit was testing its own behavior."

I can certainly believe it.