Running and publishing NUnit test results into TFS 2010

Always keen that our adopted environment shouldn't dictate tooling it was important that we could achieve a good level integration between with testing tools and Team Foundation Server; however as TFS 2010 moved to using web services and XAML build workflow it can look rather challenging.

NOTE
The snippets below are based around NUnit testing tool but it could be employed just as easily for others that are able to output their results XML in the same format, xUnit being a fine example.

Looking at integration we identified two essential requirements;

NUnit-Console test runner executes as part of the build workflow

Test results are published back into TFS 2010 and statistics

Runner

Executing nunit-console.exe using "InvokeProcess" is pretty straight forward, you will find many examples on the Internet although I used Peter Gafder's as my base to work from. Here is a slightly modified version:

    <mtbwa:InvokeProcess Arguments="[String.Format("""{0}"" /xml:""{1}"" /nologo /nodots", item, testResultXmlPath)]" DisplayName="Execute NUnit" FileName="C:\Program Files (x86)\NUnit 2.5.10\bin\net-2.0\nunit-console.exe" sad:VirtualizedContainerService.HintSize="256,403">
      <mtbwa:InvokeProcess.ErrorDataReceived>
        <ActivityAction x:TypeArguments="x:String">
          <ActivityAction.Argument>
            <DelegateInArgument x:TypeArguments="x:String" Name="errOutput" />
          </ActivityAction.Argument>
          <mtbwa:WriteBuildError sad:VirtualizedContainerService.HintSize="222,22" Message="[errOutput]" />
        </ActivityAction>
      </mtbwa:InvokeProcess.ErrorDataReceived>
      <mtbwa:InvokeProcess.OutputDataReceived>
        <ActivityAction x:TypeArguments="x:String">
          <ActivityAction.Argument>
            <DelegateInArgument x:TypeArguments="x:String" Name="stdOutput" />
          </ActivityAction.Argument>
          <Sequence sad:VirtualizedContainerService.HintSize="222,235">
            <sad:WorkflowViewStateService.ViewState>
              <scg:Dictionary x:TypeArguments="x:String, x:Object">
                <x:Boolean x:Key="IsExpanded">True</x:Boolean>
                <x:Boolean x:Key="IsPinned">True</x:Boolean>
              </scg:Dictionary>
            </sad:WorkflowViewStateService.ViewState>
            <mtbwa:WriteBuildMessage sad:VirtualizedContainerService.HintSize="200,22" Importance="[Microsoft.TeamFoundation.Build.Client.BuildMessageImportance.High]" Message="[	  stdOutput]" mva:VisualBasic.Settings="Assembly references and imported namespaces serialized as XML namespaces" />

            <!-- if test output contains errors or failures -->

            <If Condition="[(Not stdOutput.Contains("Failures: 0") And stdOutput.Contains("Failures:")) Or (Not stdOutput.Contains("Errors: 0") And stdOutput.Contains("Errors:"))]" DisplayName="If there were failed tests" sad:VirtualizedContainerService.HintSize="464,203">
              <If.Then>
                <Sequence sad:VirtualizedContainerService.HintSize="264,270">
                  <sad:WorkflowViewStateService.ViewState>
                    <scg:Dictionary x:TypeArguments="x:String, x:Object">
                      <x:Boolean x:Key="IsExpanded">True</x:Boolean>
                    </scg:Dictionary>
                  </sad:WorkflowViewStateService.ViewState>

                  <!-- put build test status in failed state -->

                  <Assign sad:VirtualizedContainerService.HintSize="242,57">
                    <Assign.To>
                      <OutArgument x:TypeArguments="mtbc:BuildPhaseStatus">[BuildDetail.TestStatus]</OutArgument>
                    </Assign.To>
                    <Assign.Value>
                      <InArgument x:TypeArguments="mtbc:BuildPhaseStatus">[Microsoft.TeamFoundation.Build.Client.BuildPhaseStatus.Failed]</InArgument>
                    </Assign.Value>
                  </Assign>

                  <!-- if treatTestFailureAsBuildFailure create build error, else create build warning -->

                  <If Condition="[treatTestFailureAsBuildFailure]" sad:VirtualizedContainerService.HintSize="242,49">
                    <If.Then>
                      <mtbwa:WriteBuildError sad:VirtualizedContainerService.HintSize="219,100" Message="[assemblyName & " -> " & stdOutput]" />
                    </If.Then>
                    <If.Else>
                      <mtbwa:WriteBuildWarning sad:VirtualizedContainerService.HintSize="220,100" Message="[assemblyName & " -> " & stdOutput]" />
                    </If.Else>
                  </If>
                </Sequence>
              </If.Then>
            </If>
          </Sequence>
        </ActivityAction>
      </mtbwa:InvokeProcess.OutputDataReceived>
    </mtbwa:InvokeProcess>

When executing the build workflow NUnit-Console.exe will now execute and the XML output placed into the build logs directory, the build status is also updated with a success/partial-success state. Our problem here is that not only do we not know how many were executed, how many passed/failed but importantly the associated stack trace of failing tests.

how many passed/failed

Publishing

TFS 2010 uses a proprietary results format (no suprise there then) with a ".trx" extension to publish results so we need to transpose the NUnit results before sending back through web services.

NUnitTfs

I'd been aware of NUnitTFS (NUnit Team Build) for some time but hadn't realised from the CodePlex welcome page their was a version for TFS 2010 that included publishing back through web-services; be sure to click on 'Downloads' and look for the latest release of version 2.0 (currently alpha).

There are some gotcha's with NUnitTFS, namely that you ensure you have an active build configuration:

TFS Build Properties

.. and that if targetting a Team Project inside a collection, you update the URLs in "NUnitTfs.exe.config" to include the collection name;

    <?xml version="1.0" encoding="utf-8" ?>
    <configuration>
    	<system.serviceModel>

    		<bindings>
    			(removed for brevity)
    		</bindings>

    		<client>
    			<!-- TFS 2010 services. -->
    			<endpoint address="http://tfs.acme.co.uk:8080/tfs/MyCollection/TestManagement/v1.0/TestResults.asmx"
    			 binding="basicHttpBinding" bindingConfiguration="TestResultsServiceSoap"
    			 contract="Tfs2010.TestResultsServiceV1.TestResultsServiceSoap" name="TestResultsServiceSoap" />
    			<endpoint address="http://tfs.acme.co.uk:8080/tfs/MyCollection/Services/v3.0/IdentityManagementService.asmx"
    			 binding="basicHttpBinding" bindingConfiguration="IdentityManagementWebServiceSoap"
    			 contract="Tfs2010.IdentityManagementServiceV3.IdentityManagementWebServiceSoap"
    			 name="IdentityManagementWebServiceSoap" />
    			<endpoint address="http://tfs.acme.co.uk:8080/tfs/MyCollection/Build/v3.0/BuildService.asmx"
    			 binding="basicHttpBinding" bindingConfiguration="BuildWebServiceSoap"
    			 contract="Tfs2010.BuildServiceV3.BuildWebServiceSoap" name="BuildWebServiceSoap" />
    		</client>
    	</system.serviceModel>
    </configuration>

Workflow

Thanks to Karsten Strøbæk's blog post discussing using NUnitTFS for xUnit I had the final piece to the puzzle; all I needed to do now was to piece the two workflows together, add some configuration variables for paths, and a couple of minor workflow tweaks to ensure smooth operation.

Build Configuration

XAML

    <mtbwa:FindMatchingFiles DisplayName="Find Test Assemblies --> testAssemblies" sap:VirtualizedContainerService.HintSize="464,22" MatchPattern="[testResultXmlPath]" Result="[testResults]" />
    <If Condition="[testResults.Count() > 0]" DisplayName="If Test Results Found">
    <If.Then>
      <Sequence>
        <mtbwa:InvokeProcess Arguments="[String.Format("-n {0} -t {1} -p ""{2}"" -f {3} -b ""{4}"" -v 2010", testResultXmlPath, BuildDetail.TeamProject, BuildSettings.PlatformConfigurations(0).Platform, BuildSettings.PlatformConfigurations(0).Configuration, BuildDetail.BuildNumber)]" DisplayName="Publish NUnit results" FileName="[TFSPublish]" sap:VirtualizedContainerService.HintSize="234,198" WorkingDirectory="[outputDirectory]">
          <mtbwa:InvokeProcess.ErrorDataReceived>
            <ActivityAction x:TypeArguments="x:String">
              <ActivityAction.Argument>
                <DelegateInArgument x:TypeArguments="x:String" Name="errOutput" />
              </ActivityAction.Argument>
              <mtbwa:WriteBuildError DisplayName="Write NUnit Publish Failure" sap:VirtualizedContainerService.HintSize="200,22" Message="[errOutput]" />
            </ActivityAction>
          </mtbwa:InvokeProcess.ErrorDataReceived>
          <mtbwa:InvokeProcess.OutputDataReceived>
            <ActivityAction x:TypeArguments="x:String">
              <ActivityAction.Argument>
                <DelegateInArgument x:TypeArguments="x:String" Name="stdOutput" />
              </ActivityAction.Argument>
              <mtbwa:WriteBuildMessage DisplayName="Write NUnit results publish output" sap:VirtualizedContainerService.HintSize="200,22" Message="[stdOutput]" mva:VisualBasic.Settings="Assembly references and imported namespaces serialized as XML namespaces" />
            </ActivityAction>
          </mtbwa:InvokeProcess.OutputDataReceived>
        </mtbwa:InvokeProcess>
      </Sequence>
    </If.Then>
    </If>

Final solution

You can find the completed XAML build workflow in a Gist here; I don't pretend to be any kind of workflow expert so feedback and improvements very welcome :-)