SetUp/TearDown methods in unit tests are apparently considered bad now

Xunit, a replacement for Nunit (the .NET unit testing library) was just announced. It's actually done by one of the original designers of Nunit, so it's worth paying attention.

Among other design considerations, it removes support for the [SetUp] and [TearDown] tags, forcing you to do all your test initialization at the top of each test, and all your test cleanup at the bottom. (This isn't actually so bad, because if you can just encapsulate it in a single function and call that function.)

I find this interesting because I've always had a bad feeling about SetUp/TearDown, but I never knew why. WvTest (our C++ unit testing framework) supports no such thing, and I never once missed it when writing tests. But I thought maybe I was just missing something.

Now that someone else has implemented the feature and learned the hard way not to use it, they can explain the actual problems: first, if your test function doesn't call the SetUp function itself, you have to go looking elsewhere to see if there is one and what it does; and second, the one-universal SetUp function tends to become a kitchen sink of all the setup you need for all your tests, so even simple tests end up running an overly complex SetUp function, creating things they don't need and masking the simplicity of the test itself. The solution is simple: just remove that feature from the test framework, and suddenly your problems are gone! And if you really want a SetUp() function, one explicit call at the top of each test isn't a lot of typing, and it's much more clear.

Interestingly, the only testrunner in Xunit is a command-line one - just like WvTest (oddly, the command line is the one that seems to be missing in recent versions of Nunit - or maybe it's just hidden away somewhere). The importance of command-line testing was strikingly obvious to me from the beginning. How can you run your tests automatically in the autobuilder if they need you to click a GUI button?

Now I just need to wait for them to adopt the last of my WvTest innovations, which I actually just stole from perl's self-test framework. Print every assertion as it runs, don't just do a report at the end. This makes it easier to see in detail what's going on, but it also means that you can build a basic "GUI testrunner" by just running the command-line testrunner and piping it to a program that counts and filters the results. WvTest has such a filter written in tcl/tk, so when I said above that "the only testrunner is a command-line one" I lied a bit) . And the GUI testrunner will work with any program that produces output in WvTest format!

(Perl, incidentally, just has a simple filter that converts the verbose test output to a series of "." characters, which is the text-mode equivalent of a completion bar. If any test fails, you can re-run it and see its output, but in the expected end-user case where all the tests pass, it just gives a pretty summary.)

In any case, it looks like Xunit will be a good choice for a unit testing framework for our .Net work, where WvTest doesn't exist. We had recently ditched Nunit because it was too bloated and complex. Xunit has less stuff - perfect!