AzureDevOpsReporter: skip Assert frames by type name instead of filename (fixes #6925) by Evangelink · Pull Request #8284 · microsoft/testfx

Evangelink · 2026-05-16T10:43:52Z

Fixes #6925. Closes #8278.

Problem

AzureDevOpsReporter.GetErrorText walks a failing test's stack frames and skips the frame if the file path ends with Assert.cs, so the reporter annotates the user's call site rather than the framework's assertion implementation:

https://github.com/microsoft/testfx/blob/main/src/Platform/Microsoft.Testing.Extensions.AzureDevOpsReport/AzureDevOpsReporter.cs#L215-L223

But Assert / CollectionAssert / StringAssert are split into partial-class files (Assert.AreEqual.cs, Assert.IComparable.cs, Assert.IsInstanceOfType.cs, CollectionAssert.Equality.cs, etc.) whose file names do not end with Assert.cs. The filter misses every one of them, so for cases like the Assert.IsInstanceOfType failure in #6925 the reporter incorrectly annotates the framework file instead of the user's test.

The filter also has the opposite problem: a user file innocently named MyAssert.cs would be wrongly skipped.

Fix

Switch the skip decision from the file name to the fully-qualified type name captured in the stack frame's code field (already parsed by StackTraceHelper.GetFrameRegex and previously unused). A frame is skipped iff its code starts with one of:

Microsoft.VisualStudio.TestTools.UnitTesting.Assert.
Microsoft.VisualStudio.TestTools.UnitTesting.CollectionAssert.
Microsoft.VisualStudio.TestTools.UnitTesting.StringAssert.

This naturally covers every partial-class file (current and future), is robust against file renames/splits, and removes the MyAssert.cs false positive.

A complementary follow-up — adding [StackTraceHidden] on the same APIs once RFC 012 lands — is tracked separately in #8277 and would obviate the reporter heuristic entirely on .NET 6+. This PR is still needed for .NET Framework.

Tests

Added 5 tests to AzureDevOpsTests:

SkipsMSTestAssertImplementationFrameInPartialClassFile — regression test for Assert.IComparable not compatible with AzDoReport extension #6925: synthetic stack with a frame in Assert.IComparable.cs followed by a user frame; asserts the user frame is reported.
SkipsMSTestCollectionAssertImplementationFrameInPartialClassFile — same for CollectionAssert.Equality.cs.
SkipsMSTestStringAssertImplementationFrame — same for StringAssert.cs.
DoesNotSkipUserFrameWhoseFileNameEndsWithAssertCs — verifies the previous false positive on user files named *Assert.cs is gone (mocked IFileSystem).
DoesNotSkipUserFrameWhoseTypeNameStartsWithAssert — verifies a user type in a non-MSTest namespace named Assert is not mistaken for the framework's Assert.

The two pre-existing tests' expected line numbers were updated because the using Moq; directive shifted the throw statements down.

All 7 tests pass on net9.0, net8.0, and net472. Full Microsoft.Testing.Extensions.UnitTests suite (124 tests) passes.

Copilot

Pull request overview

This PR fixes Azure DevOps reporting so MSTest assertion implementation frames are skipped by fully-qualified frame code/type prefix rather than by source filename.

Changes:

Adds MSTest assertion type-prefix filtering in AzureDevOpsReporter.
Adds regression tests for partial assertion files and user files/types that should not be skipped.
Updates existing expected line numbers after new imports.

Show a summary per file

File	Description
`src/Platform/Microsoft.Testing.Extensions.AzureDevOpsReport/AzureDevOpsReporter.cs`	Replaces filename-based Assert frame skipping with code-prefix-based assertion implementation detection.
`test/UnitTests/Microsoft.Testing.Extensions.UnitTests/AzureDevOpsTests.cs`	Adds synthetic stack trace tests covering MSTest assertion frame skipping and false-positive avoidance.

Copilot's findings

Files reviewed: 2/2 changed files
Comments generated: 1

Evangelink

Review Summary

This is a clean, well-scoped fix for #6925. All 21 review dimensions assessed; no blocking issues found.

Dimension	Status
1. Algorithmic Correctness	✅ Root cause fixed (type prefix vs. filename). `GetStackFrameLocation` guarantees `code` is non-null/non-empty before `IsAssertionImplementationFrame` is called.
2. Threading & Concurrency	✅ N/A — no shared mutable state introduced. `AssertionImplementationCodePrefixes` is `static readonly` (immutable array).
3. Security & IPC	✅ N/A
4. Public API / Binary Compat	✅ All new code is `private static` — no public surface added.
5. Performance	✅ `foreach` over 3-element static array; no allocations on the hot path.
6. Cross-TFM	✅ `string.StartsWith(StringComparison.Ordinal)` available on all targets.
7. IDisposable	✅ N/A
8. Defensive Coding	✅ `code` is validated non-empty by `GetStackFrameLocation` before use.
9. Localization	✅ Log messages are trace-only diagnostics, not user-facing; no `.resx` required.
10. Test Isolation	✅ New tests use synthetic stack traces; no shared state.
11. Assertion Quality	✅ `BannedSymbols.txt` bans `AwesomeAssertions`; tests correctly use MSTest `Assert.AreEqual`.
12. Flakiness	✅ `GetCurrentLocation` uses `[CallerFilePath]/[CallerLineNumber]` — deterministic at compile time.
13–21. Remaining dimensions	✅ N/A for this change.

Observations (non-blocking):

The GetCurrentLocation() helper captures line 65 as userLine, then immediately uses it in the synthetic stack trace at line 72 and in the expected assertion output at line 80. The test is self-consistent, but note that inserting any line between GetCurrentLocation() and the stackTrace construction would silently shift the captured line without breaking the test (because both references move together). This is an accepted pattern for this kind of test, just worth knowing.
The PR description mentions a follow-up [StackTraceHidden] initiative (#8277) that would make this heuristic unnecessary on .NET 6+. This PR remains correct and necessary for .NET Framework.

LGTM — the fix is correct, tested, and well-explained.

Generated by Expert Code Review (on open) for issue #8284 · ● 4.6M

Evangelink · 2026-05-16T11:57:09Z

@copilot address review comments and fix the broken test below:

test/IntegrationTests/MSTest.Acceptance.IntegrationTests/TelemetryTests.cs#L132

test/IntegrationTests/MSTest.Acceptance.IntegrationTests/TelemetryTests.cs(132,1): error : [VSTest_RunTests_Succeeds ("net462")] [net11.0] Assert.AreEqual failed. Expected:<0>. Actual:<1>. 'expected' expression: '0', 'actual' expression: 'testResult.ExitCode'. dotnet test failed:
Telemetry is: Disabled
Determining projects to restore...
Restored D:\a_work\1\s\artifacts\tmp\Debug\testsuite\gKgYA\TelemetryTests\vstest\TelemetryVSTestProject.csproj (in 484 ms).
TelemetryVSTestProject -> D:\a_work\1\s\artifacts\tmp\Debug\testsuite\gKgYA\TelemetryTests\vstest\bin\Release\net462\TelemetryVSTestProject.dll
D:\a_work\1\s.dotnet\sdk\11.0.100-preview.5.26227.104\Microsoft.Common.CurrentVersion.targets(5742,5): error MSB3491: Could not write lines to file "obj\Release\net462\TelemetryVSTestProject.csproj.FileListAbsolute.txt". Cannot create a file when that file already exists. [D:\a_work\1\s\artifacts\tmp\Debug\testsuite\gKgYA\TelemetryTests\vstest\TelemetryVSTestProject.csproj::TargetFramework=net462]

Copilot · 2026-05-16T12:05:41Z

@copilot address review comments and fix the broken test below:

test/IntegrationTests/MSTest.Acceptance.IntegrationTests/TelemetryTests.cs#L132
...

Addressed in e1d0f4f: the regression stack frame now uses Assert.IsLessThan from Assert.IComparable.cs, and TelemetryTests is marked [DoNotParallelize] so the shared generated VSTest asset is no longer built concurrently.

Copilot

Copilot's findings

Files reviewed: 3/3 changed files
Comments generated: 1

Evangelink · 2026-05-17T19:11:48Z

@copilot resolve the merge conflicts in this pull request

Copilot · 2026-05-17T19:18:21Z

@copilot resolve the merge conflicts in this pull request

Addressed in d936685.

Copilot

Copilot's findings

Files reviewed: 5/5 changed files
Comments generated: 3

        CheckParameterNotNull(collection, "Assert.AreAllDistinct", "collection");
        CheckParameterNotNull(comparer, "Assert.AreAllDistinct", "comparer");
-        AreAllDistinctImpl(collection, comparer, comparerTypeName: comparer.GetType().ToString(), message, collectionExpression);
+        AreAllDistinctImpl(collection, comparer, comparerTypeName: comparer.GetType().Name, message, collectionExpression);


+        Assert.IsTrue(
+            testHostResult.ExitCode is (int)ExitCode.Success or (int)ExitCode.TestHostProcessExitedNonGracefully,
+            $"Expected hang dump template scenarios to exit with {(int)ExitCode.Success} or {(int)ExitCode.TestHostProcessExitedNonGracefully}, but got {testHostResult.ExitCode}.{Environment.NewLine}{testHostResult}");


+            ApartmentState? targetApartmentState = MSTestSettings.RunConfigurationSettings.ExecutionApartmentState ??
+                (Thread.CurrentThread.GetApartmentState() == ApartmentState.STA ? ApartmentState.STA : null);

-            entryPointThread.SetApartmentState(ApartmentState.STA);
-            entryPointThread.Start();
-            return tcs.Task;
+            if (targetApartmentState is not null)
+            {
+                return ExecuteWithCustomApartmentStateAsync(taskGetter, targetApartmentState.Value);


Fix #6925. The previous heuristic skipped stack frames whose file path ended with 'Assert.cs', but the MSTest Assert class is split into partial-class files (Assert.AreEqual.cs, Assert.IComparable.cs, ...) whose names do not match. As a result, the reporter annotated the framework implementation instead of the user's call site. Switch the skip rule to a fully-qualified type prefix check on the 'code' capture from the stack frame regex. This covers every partial-class file (current and future) and removes the false-positive that wrongly skipped user files named *Assert.cs. Closes #8278. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Evangelink <11340282+Evangelink@users.noreply.github.com>

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…PR check annotation title The AzDO reporter built the failure message as a single line: [TestName] [tfm] Assert.AreEqual failed. Expected:<X>. Actual:<Y>. ... Both Azure DevOps and GitHub PR checks (via the dotnet problem matcher) treat the first line of the message as the bold annotation title. With the previous format the title was dominated by the assertion text and the test name was lost between brackets, making failing tests harder to identify at a glance. Split the message so the test name sits on its own line (becoming the annotation title) and the assertion message follows on the next line: HangDump_TemplateFileName_CreateDump ("net10.0") [net11.0] Assert.AreEqual failed. Expected:<X>. Actual:<Y>. ... Also avoid the noisy duplicate TFM (`MyTest (net9.0) [net9.0]`) when MTP has already appended the TFM to the display name in multi-TFM mode. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI review requested due to automatic review settings May 16, 2026 10:43

Copilot started reviewing on behalf of Evangelink May 16, 2026 10:45 View session

Copilot AI reviewed May 16, 2026

View reviewed changes

Comment thread test/UnitTests/Microsoft.Testing.Extensions.UnitTests/AzureDevOpsTests.cs Outdated

Evangelink commented May 16, 2026

View reviewed changes

Copilot started work on behalf of Evangelink May 16, 2026 11:57 View session

Copilot finished work on behalf of Evangelink May 16, 2026 12:06

Copilot AI review requested due to automatic review settings May 16, 2026 14:26

Copilot started reviewing on behalf of Evangelink May 16, 2026 14:27 View session

Copilot AI reviewed May 16, 2026

View reviewed changes

Comment thread src/Platform/Microsoft.Testing.Extensions.AzureDevOpsReport/AzureDevOpsReporter.cs Outdated

Evangelink mentioned this pull request May 16, 2026

Fix TelemetryTests flake from parallel dotnet test races #8292

Merged

Copilot started work on behalf of Evangelink May 17, 2026 10:17 View session

Copilot finished work on behalf of Evangelink May 17, 2026 10:24

Copilot started work on behalf of Evangelink May 17, 2026 19:11 View session

Copilot AI review requested due to automatic review settings May 17, 2026 19:17

Evangelink removed the request for review from Copilot May 17, 2026 19:17

Copilot finished work on behalf of Evangelink May 17, 2026 19:18

Copilot AI review requested due to automatic review settings May 18, 2026 08:52

Copilot started reviewing on behalf of Evangelink May 18, 2026 08:53 View session

Copilot AI reviewed May 18, 2026

View reviewed changes

Evangelink and others added 5 commits May 18, 2026 15:15

Fix telemetry test race and refine assert regression

1413ff3

Co-authored-by: Evangelink <11340282+Evangelink@users.noreply.github.com>

Handle Assert.That Azure DevOps frames

f685886

Co-authored-by: Evangelink <11340282+Evangelink@users.noreply.github.com>

Fix inherited CI regressions on PR 8284

11d3e8f

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Evangelink force-pushed the dev/amauryleve/fix-6925-azdo-assert-skip branch from 9fac123 to 706b1c6 Compare May 18, 2026 13:23

Evangelink merged commit d1d69ca into main May 18, 2026
17 checks passed

Evangelink deleted the dev/amauryleve/fix-6925-azdo-assert-skip branch May 18, 2026 15:00

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

AzureDevOpsReporter: skip Assert frames by type name instead of filename (fixes #6925)#8284

AzureDevOpsReporter: skip Assert frames by type name instead of filename (fixes #6925)#8284
Evangelink merged 5 commits into
mainfrom
dev/amauryleve/fix-6925-azdo-assert-skip

Evangelink commented May 16, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Evangelink left a comment

Uh oh!

Evangelink commented May 16, 2026

Uh oh!

Copilot AI commented May 16, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Evangelink commented May 17, 2026

Uh oh!

Copilot AI commented May 17, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

Evangelink commented May 16, 2026

Problem

Fix

Tests

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Copilot's findings

Uh oh!

Uh oh!

Evangelink left a comment

Choose a reason for hiding this comment

Review Summary

Uh oh!

Evangelink commented May 16, 2026

Uh oh!

Copilot AI commented May 16, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot's findings

Uh oh!

Uh oh!

Evangelink commented May 17, 2026

Uh oh!

Copilot AI commented May 17, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot's findings

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants