Add an option to include exception messages in Microsoft.Testing.Extensions.JUnitReport failure bodies for GitLab compatibility.

[picolino]

Summary

Add an option to Microsoft.Testing.Extensions.JUnitReport that includes the exception or assertion message in the text content of the JUnit <failure> or <error> element, rather than storing it only in the message attribute.

This is needed for proper failure diagnostics in GitLab test reports.

Background and Motivation

I am using Microsoft Testing Platform together with Microsoft.Testing.Extensions.JUnitReport to publish .NET test results to GitLab CI.

When a test fails, the generated JUnit report contains the exception or assertion message in the message attribute of the <failure> element, while the element body contains only the stack trace.

For example:

<failure
    message="Expected the operation not to throw, but it threw System.InvalidOperationException: The expected condition was not met."
    type="Xunit.Sdk.XunitException">
  at FluentAssertions.Execution.LateBoundTestFramework.Throw(String message)
  at FluentAssertions.Execution.TestFrameworkProvider.Throw(String message)
  at FluentAssertions.Execution.DefaultAssertionStrategy.HandleFailure(String message)
  ...
</failure>

GitLab displays the text content of <failure> as the test's System output, but it does not display the message attribute in the test summary UI.

As a result, the GitLab test report shows only the stack trace:

at FluentAssertions.Execution.LateBoundTestFramework.Throw(String message)
at FluentAssertions.Execution.TestFrameworkProvider.Throw(String message)
at FluentAssertions.Execution.DefaultAssertionStrategy.HandleFailure(String message)
...

The most important diagnostic information - the actual exception or assertion message - is missing.

This is especially problematic for assertion libraries such as FluentAssertions, where the stack trace is often generic and the assertion message contains the actual reason for the failure.

Other JUnit reporters address this compatibility issue by supporting a verbose failure-body mode that writes both the message and the stack trace into the <failure> element body, like such: https://github.com/spekt/testlogger/blob/master/src/JUnit.Xml.Package/README.md#failurebodyformat

Proposed Feature

Add a configuration option to Microsoft.Testing.Extensions.JUnitReport that controls the content written into the body of <failure> and <error> elements.

For example:

--report-junit-failure-body-format

Possible values could be:

• StackTrace — current behavior;
• Message;
• MessageAndStackTrace.

Alternatively, the option could be named similarly to existing JUnit reporters:

--report-junit-failure-body-format verbose

With the proposed verbose behavior, the generated XML would look like this:

<failure
    message="Expected the operation not to throw, but it threw System.InvalidOperationException: The expected condition was not met."
    type="Xunit.Sdk.XunitException">
Expected the operation not to throw, but it threw System.InvalidOperationException: The expected condition was not met.

at FluentAssertions.Execution.LateBoundTestFramework.Throw(String message)
at FluentAssertions.Execution.TestFrameworkProvider.Throw(String message)
at FluentAssertions.Execution.DefaultAssertionStrategy.HandleFailure(String message)
...
</failure>

The message attribute should remain for compatibility with JUnit consumers that read it directly.

The same value should also be included in the element body so that systems such as GitLab can display meaningful failure details.

Alternative Designs

Always include the message in the element body

Instead of introducing a new option, the reporter could always write both the exception message and the stack trace into the <failure> or <error> body.

This would provide the best compatibility with GitLab and similar JUnit consumers, but it would change the current output and could duplicate the message in tools that already display the message attribute.

GitLab-specific post-processing

A CI pipeline can modify the generated XML before publishing it and prepend the message attribute to the <failure> body.

For example:

<failure message="Assertion message">
Assertion message

Stack trace...
</failure>

This works as a workaround, but every project using Microsoft Testing Platform with GitLab must maintain its own XML transformation script.

It also makes the pipeline more complex and introduces an additional processing step that should ideally be handled by the report generator.

Generate TRX and convert it to JUnit

Another option is to generate a TRX report and convert it to a GitLab-compatible JUnit report using a separate tool.

This adds another dependency and report conversion stage, and it defeats the purpose of using the official JUnit report extension directly.

Rely on GitLab to display the message attribute

GitLab could potentially change its JUnit parser or test report UI to display the message attribute.

However, including the message in the failure body is already a commonly supported compatibility approach and would improve interoperability with other JUnit consumers as well, not only GitLab.

[Evangelink]

Thanks for the detailed writeup — the problem is real and the analysis is accurate.

We'd like to go with your "Always include the message in the element body" alternative rather than adding an option, and make it the only behavior. Reasoning:

This is a fidelity gap, not a stylistic choice. Canonical JUnit XML bodies come from Java's Throwable.printStackTrace(), whose first line is com.example.AssertionError: expected:<1> but was:<2> — the message is inherently part of the body. .NET's Exception.StackTrace omits that header line (only ToString() includes it), which is exactly why our output looks message-less next to a genuine Surefire report. GitLab and CircleCI aren't being quirky here; they're reading the body assuming Java semantics. Including the message moves us closer to the de-facto format, so it's not something we want to hide behind an opt-in flag.

Both schemas allow it. Neither the Ant/windyroad JUnit.xsd nor Surefire's surefire-test-report.xsd constrains the <failure>/<error> body — it's free-form text (the Ant schema's "e.g., a stack trace" is illustrative, not normative). So the change stays schema-valid.

On duplication: the message will appear in both the message attribute and the body. That's precisely the shape every Maven/Surefire report already has, so consumers that render both have handled it for years. We'd rather match the ecosystem than optimize away a cosmetic repeat.

Timing: Microsoft.Testing.Extensions.JUnitReport hasn't shipped a stable version yet (only 1.0.0-alpha.* on NuGet), so there's no back-compat cost to changing the default now. A new CLI option, by contrast, is a permanent contract — and we'd only be offering users a way to opt into less diagnostic information. If a concrete need for StackTrace-only bodies shows up later, adding the option then is non-breaking, whereas removing one wouldn't be.

The planned output, mirroring printStackTrace() (with graceful degradation when the type or stack trace is missing):

<failure message="Expected the operation not to throw, but it threw System.InvalidOperationException: The expected condition was not met."
         type="Xunit.Sdk.XunitException">
Xunit.Sdk.XunitException: Expected the operation not to throw, but it threw System.InvalidOperationException: The expected condition was not met.
   at FluentAssertions.Execution.LateBoundTestFramework.Throw(String message)
   at FluentAssertions.Execution.TestFrameworkProvider.Throw(String message)
   ...
</failure>

The message and type attributes stay exactly as they are today, so consumers reading them directly are unaffected.

One deliberate difference from spekt's FailureBodyFormat=Verbose: we do not plan to fold standard output into the failure body. It's already emitted as <system-out> per test case, and duplicating it would bloat reports for the noisier suites.

Does that cover your GitLab scenario? If you're relying on anything beyond the message being present in the body, let us know before we implement.

[picolino]

@Evangelink thanks for the fastest reply and the implementation in dotnet repos I have ever faced 😄

Seems like solution you describe completely cover my scenario with Gitlab, thank you again, waiting for next package version to test it