Thursday, August 25, 2011

Minimalistic Unit Testsing in F#

So you want to do some unit testing in F#. Should you really use a framework?

The blessing and the curse of it is that there are so many frameworks out there that choosing one takes about as much time as creating one. Besides, what should a unit testing framework do anyway? There are frameworks offering real features such as randomized (QuickCheck) or exhaustive (SmallCheck) testing. On the other hand, frameworks like NUnit and xUnit do not seem to offer much - just bells and whistles such as result reporting and IDE integration. Is it worth it?

To roll our own, throw out everything but the bare bones. No reflection - all tests first class. No randomized / exhaustive testing - simple tests do not need that. Simply spend a few lines to define nice syntax, and start testing. Compile the test assembly to an executable, and run it on every build.

For example:


  1. Hi, I think it's a nice kata or something to implement some simple unit-testing yourself. But after all TDD and co. Are about speed to. If I use some unit-test framework with nice integration into VS2010 (or ReSharper), like most Frameworks out ther (with or without or ReSharper) offers, I get a LOT of improvement over running this stuff myself. For example I can only run the tests again that failed. I can debug the test that failed with one click. I can find the errors and jump to the error-site with one clicke ... IMHO this is worth the little inconvenience of downloading and installing the framework once ;)

  2. Hi toyvo, I realize that your post here is about shedding unit test frameworks, however, I can't help but point you to my own: I think you'll find Unquote unique, in that it embraces your minimalistic pursuit (it has just two primary functions, test and raises), and works with all unit testing frameworks or none (falls back on throwing a custom exception). The beauty is that you write plain, statically checked F# expression tests, and get nice step-by-step failure messages without any effort.

  3. @Carsten, of course you should use whatever works for you :)

    @Stephen, I would strongly question your design. Yes, it might be very educational when you are learning F# to see the evaluation trace. But personally all I want tests to do is test, and do that as quickly as possible, therefore I would consider throwing out any sort of reflection as a slowdown. Once again, use whatever works for you :)

  4. Hi Anton,

    I re-factored the asserts in Assert.fs module and added the following function as well as changed the printing for fail cases. This way one can double click on a failed test and get directly to the line that failed:

    let errorLocation trace =
    let mtch = System.Text.RegularExpressions.Regex.Match(trace, @"(.:\\.*\\(?!Assert)[^.]*\.fs):line (\d+)")
    sprintf "%s(%s)" mtch.Groups.[1].Value mtch.Groups.[2].Value

    | Fail reason as e->
    status <- 1
    stderr.WriteLine("{0}: {1}.{2} -- {3}", errorLocation e.StackTrace, section, name, reason)

  5. Obviously this requires one to run the test project as post-build command:


    Thanks for the tip!