In a previous post I reviewed MSTest. This time I will focus on NUnit. NUnit is an open-source testing framework that has been around for a long time – in fact I think there was a time when this was the only test framework for .NET - and is very mature and stable. From the NUnit website:
NUnit is a unit-testing framework for all .Net languages. Initially ported from JUnit, the current production release, version 2.6, is the seventh major release of this xUnit based unit testing tool for Microsoft .NET. It is written entirely in C# and has been completely redesigned to take advantage of many .NET language features, for example custom attributes and other reflection related capabilities. NUnit brings xUnit to all .NET languages.
Integration
NUnit is a separate project meaning that direct Visual Studio support is not provided. However Visual Studio 2012 will allow different frameworks apart from MSTest to be used as the primary unit testing framework - this includes TFS builds too.
In the meantime, the following steps are required:
Download from NuGet
NuGet provides two packages for NUnit:
The NUnit package contains the core of the framework while the NUnit.Runners contains the various test runners needed to run the test assemblies.
Adding these packages to a Visual Studio project is very simple as NuGet will automatically download the latest versions and insert the correct project references required.
One initial gotcha though is that all the test runners by default assume the .NET 2.0 CLR is the correct runtime environment to use, so if you are using the .NET 4 CLR you will have to update all the *.config files to use the correct framework version; simply a matter of commenting out one line in each *.config file, though there are several to modify.
Project Items and Snippets
Unlike MSTest which provides project items and snippets with the IDE, NUnit does not provide any by default. However these items are not difficult to create yourself if required.
Standalone
Although some initial setup is required one possible benefit is that NUnit is a standalone framework - it can be run anywhere without requiring installation, simply by copying the correct files.
Team Build
TFS 2012 will be able to use the same Unit Test plugin model that Visual Studio 2012 uses meaning in future it will be a lot easier to integrate NUnit into the Team Build process.
Until then though it is possible to use NUnit within Team Build but only via a custom build activity and translating the NUnit XML output into MSTest results. This webpage explains how it is possible to do it, though the process looks quite longwinded to me.
Writing Tests
Tests are written like this in NUnit:
1: using System;
2: using System.Collections.Generic;
3: using System.Linq;
4: using System.Text;
5: using NUnit.Framework;
6: 7: namespace SampleCode.NUnit
8: {9: // Denotes a Test Class for NUnit to find
10: [TestFixture]11: public class CalculatorTests
12: {13: // Denotes a Test Method for NUnit to run
14: [Test]15: public void Add_AddOneAndTwo_ReturnsThree()
16: { 17: var result = Calculator.Add(1, 2); 18: 19: // Assertions can be written with a fluent API to make them more descriptive
20: Assert.That(result, Is.EqualTo(3)); 21: } 22: 23: } 24: }There are a much wider variety of assertions provided by NUnit compared to MSTest. There is also a fluent/constraint API provided which allows a more descriptive style of writing if you find that useful.
Data Driven Tests
NUnit supports data-driven tests by writing parameterised test methods and decorating them with special attributes like so:
1: [Test] 2: [TestCase(1, 2, 3)] 3: [TestCase(5, 10, 15)] 4: [TestCase(30, 2, 32)]5: public void Add_AddDataValues_ReturnsExpectedResult(int first, int second, int expected)
6: { 7: var actualResult = Calculator.Add(first, second); 8: 9: Assert.That(actualResult, Is.EqualTo(expected)); 10: }Unlike MSTest, NUnit sees this as three separate tests and will display it as such in the test runners.
It is also possible to write a data-driven test by defining a data source using the [TestCaseSource] attribute which defines a property/method which returns an enumerable collection of object values to pass in as parameters.
The only thing that NUnit does not provide out-of-the-box is the ability to store data values and load from a file/external source, only code values can be used. However, using the [TestCaseSource] attribute utility methods can be created by yourself to do that.
Running Tests
Until Visual Studio 2012 comes out NUnit tests cannot be run directly via the IDE but there are a number of other options available.
The console runner is the most basic test runner available and works from the command line.
The GUI runner is a standalone application with it's own user interface. Tests are represented in a tree view along with their results and any failures are reported on the right-hand side. Notice also that data-driven tests are split further down into more tests to run.
A nice feature of the GUI runner is that it is able to detect changes in the test assembly when it is rebuilt. When a change is detected, the GUI runner can be configured to reload the assembly and automatically run the last set of tests to allow for a continuous testing strategy.
It is also possible to run NUnit via an MSBuild task, e.g. as an after-build step. This is done by simply executing the console runner using the <Exec> task, similar to this:
1: <Target Name="AfterBuild">
2: <Exec Command="..\packages\NUnit.Runners.2.6.1\tools\nunit-console.exe $(TargetPath)" />
3: </Target>
Build output then appears as follows:
------ Build started: Project: SampleCode, Configuration: Debug Any CPU ------
SampleCode -> C:\Experiments\UnitTestAnalysis\SampleCode\bin\Debug\SampleCode.dll
------ Build started: Project: SampleCode.NUnit, Configuration: Debug Any CPU ------
SampleCode.NUnit -> C:\Experiments\UnitTestAnalysis\SampleCode.NUnit\bin\Debug\SampleCode.NUnit.dll
NUnit-Console version 2.6.1.12217
Copyright (C) 2002-2012 Charlie Poole.
Copyright (C) 2002-2004 James W. Newkirk, Michael C. Two, Alexei A. Vorontsov.
Copyright (C) 2000-2002 Philip Craig.
All Rights Reserved.
Runtime Environment -
OS Version: Microsoft Windows NT 6.1.7601 Service Pack 1
CLR Version: 4.0.30319.269 ( Net 4.0 )
ProcessModel: Default DomainUsage: Single
Execution Runtime: net-4.0
....
Tests run: 4, Errors: 0, Failures: 0, Inconclusive: 0, Time: 0.103 seconds
Not run: 0, Invalid: 0, Ignored: 0, Skipped: 0
========== Build: 2 succeeded or up-to-date, 0 failed, 0 skipped ==========Performance
Performance of running tests seems to be faster than MSTest, even with a significant number of tests to execute.
Reports
Apart from the output represented by various test runners, an XML report can be produced by either the console or GUI runner. Once in an XML format, this can then be transformed into another format, e.g. a HTML file to make it human readable or a *.trx (MSTest) output file so that Visual Studio can understand it.
Unfortunately no sample XSL stylesheets are provided as standard by the NUnit package so this work has to be carried out by yourself or by searching on the Internet for a suitable one.
Extensibility
There are some extensibility points within NUnit but a lot of the most common features are provided as standard so there is very little else to require from it.
My Opinion
In my opinion NUnit is a lot better than MSTest. NUnit is very stable, fast, well supported and documented and supports many of the use case scenarios that you would typically use day to day. Although MSTest beats it in terms of IDE integration even without the new Unit Test Adapter it is still very easy to setup and use; the fact that the GUI runner automatically reloads assembly changes means you can have an almost-continuous test setup.
And for a while I was perfectly happy with using NUnit, until I found out about xUnit.net…
More next time!
No comments:
Post a Comment