Brainstorm on ideas for more AzDO extensions in MTP

[Youssef1313]

Summary

The NuGet package Microsoft.TeamFoundationServer.Client contains Microsoft.VisualStudio.Services.TestResults.WebApi.dll which contains TestResultsHttpClient (it's public API). This class contains all the important pieces for publishing test results. There may be other classes that we could make a good use of.

Proposed Feature

This is still "brainstorming" phase, nothing concrete yet. But:

• We could provide an extension to publish test results as tests are running, without the need for TRX, and without needing to wait to the end of the test run.
• We could query history on test failures, and maybe publish the test in a special way to indicate it's flaky, and how many times it failed over a month, etc.

Side note: See Roslyn for how to do the authentication. It's simply just reading SYSTEM_ACCESSTOKEN and SYSTEM_COLLECTIONURI env variables and use them to create VssConnection.

We could also see if there is any test plan related stuff in this package and whether we could use this package to support this in MTP.

[Evangelink]

After reading through the existing Microsoft.Testing.Extensions.AzureDevOpsReport (which today only emits ##vso[task.logissue] lines) and the adjacent open issues (#5979, #4477, #5362, #5754, #6843), here is a concrete brainstorm of 5 directions, plus an architectural question at the end I'd like input on.

1. Live test-result publishing (the core ask from this issue)

What the user gets today is a couple of console lines per failure. We never populate the AzDO Tests tab — that still requires generating a TRX and adding a downstream PublishTestResults@2 task.

Concretely:

• Auth like RunTests/TestHistoryManager in Roslyn: read SYSTEM_ACCESSTOKEN + SYSTEM_COLLECTIONURI (+ SYSTEM_TEAMPROJECT, BUILD_BUILDID) and build the client.
• On session start (ITestSessionLifetimeHandler): create a test run (State=InProgress, name = <assembly> (<tfm>) on <agent>).
• On every TestNodeUpdateMessage: batch and post results as tests finish, instead of waiting for the whole run — this is the real value: long pipelines can see passing/failing tests minutes earlier, and parallel jobs no longer race on TRX uploads.
• On session end: Completed. On crash / hang-dump: Aborted, so runs aren't stuck InProgress forever.
• Multi-TFM / multi-assembly: write the runId to a small file under TestResultsDirectory so sibling processes join the same AzDO run rather than creating N parallel ones — single consolidated entry in the Tests tab.
• CLI: --publish-azdo-test-results (zero-arity), --publish-azdo-run-name, optional --publish-azdo-release-uri for release pipelines.

2. Flaky-test history annotations and quarantine awareness

The issue explicitly mentions querying history; today the reporter has no notion of it, so every failure looks equally bad.

• New opt-in --report-azdo-flaky-history <days>. At startup, query history grouped by automatedTestName for the window.
• A new failure (0 failures in window) → keep as error, tag [regression].
• A passing test that historically fails ≥X% → emit warning "passed now, failed 4/20 in last 14d".
• A known-flaky failure → optionally demote to warning so PR gates aren't blocked by known noise (off by default).
• --report-azdo-quarantine-file <path> listing FQN patterns the team has officially quarantined; emit a build tag quarantined-test-skipped via ##vso[build.addbuildtag] so build queries can filter them.
• This data is the same the terminal reporter would need for show summary of retried and flaky tests #5362, so the two features can share one historical-lookup component.

3. Auto-attach artifacts (dumps, screenshots, logs, code coverage)

HangDump, CrashDump, CodeCoverage, TRX all drop files into TestResultsDirectory. Nothing teaches AzDO to attach them in the right place — the most common "MTP feels native in AzDO" gap I see.

a) Pipeline-level (no auth needed) — extend the existing extension:

• After session end, for each artifact under TestResultsDirectory, emit ##vso[artifact.upload containerfolder=TestResults;artifactname=…] so dumps/coverage become first-class build artifacts without a PublishBuildArtifacts@1 task.
• Emit ##vso[build.addbuildtag]has-crashdump / has-hangdump when those extensions fired.

b) Result-level (depends on #1) — when the live publisher is on:

• For a failed/timeout/aborted node, find the matching TestNodeFileArtifact and upload it as a test-result attachment so it shows under the test in the Tests tab.
• Auto-attach stdout/stderr as ConsoleLog.
• Attach *.cobertura.xml / *.coverage as CodeCoverage on the run, so AzDO's coverage tab populates without PublishCodeCoverageResults@2.

4. Rich job summary + per-test timeline progress (zero-dep, high-ROI)

Pure logging commands, no SDK dependency:

• New option --report-azdo-summary[=path]. On session end, render a summary (total/pass/fail/skip, slowest 10, top 5 failing classes, link to the live run from Initial commit! 🎉 #1, list of quarantined tests skipped from Porting latest changes. #2) and emit ##vso[task.uploadsummary]<path> so it shows in the build page Extensions tab.
• On format — and answering your HTML question directly: task.uploadsummary accepts both .md and .html (sanitized). I'd default to markdown because the summary tab is narrow, the content is small, and markdown round-trips nicely into other places (PR comments, GitHub Actions step summaries, etc.). But when the new HTML reporter from HTML logger porting #5754 lands, we should add --report-azdo-summary-format=html|md|auto (default auto) where auto reuses the HTML report's own output if that extension is loaded, and falls back to a self-contained markdown otherwise. That way users who already opted into the HTML reporter get the prettier view "for free" with no duplicate rendering work.
• Per-test timeline progress for long runs:
  • On session start: ##vso[task.logdetail id=<guid>;name=<assembly>(<tfm>);type=Build;state=InProgress].
  • As TestNodeUpdateMessages flow in: ##vso[task.setprogress value=<pct>] (we have total from discovery and completed count). Today an MTP test step looks frozen for 40 minutes in the AzDO log UI; this fixes that.
  • On end: ##vso[task.logdetail …;result=Succeeded/Failed;state=Completed].
• Make the stack-frame filter user-configurable (today only Assert.cs is hard-coded at line 215 of AzureDevOpsReporter): --report-azdo-stackframe-filter <regex> (repeatable) so teams can hide their own assertion helpers / retry harnesses. This recurs in Should output device log issues with Azure syntax when running on Azure? #5979 discussions too.

5. Azure Test Plans integration (test case association + plan/suite execution)

This is exactly the long-standing #4477 ask. The AzDO REST API exposes everything we need; VSTest used to wire it up.

• Publishing side (built on Initial commit! 🎉 #1): when [TestProperty("TestCase", "12345")] or [WorkItem(12345)] is present, populate TestCase.Id / TestCaseReferenceId on the result so the outcome appears in the Test Plans dashboard. A small [AzureDevOpsTestCase(12345)] sugar attribute in MSTest would avoid the magic-string trap.
• Discovery / selection side: --azdo-test-plan <id> and --azdo-test-suite <id>. At startup, resolve the suite's automatedTestNames and register an ITestExecutionFilterFactory to filter the run. This is the "run a test plan from AzDO without VSTest" scenario.
• Outcome write-back: entries we couldn't resolve get outcome=NotExecuted instead of silently being skipped (today's VSTest behaviour).

---

One package or many? + how to avoid dep conflicts

You're right that splitting hurts discoverability. My initial reflex was "put the heavy REST stuff in a separate package" so the lightweight AzureDevOpsReport stays zero-dep, but I want to push back on my own instinct here:

Option A — Microsoft.TeamFoundationServer.Client dep, split package. That SDK pulls Newtonsoft.Json, several Microsoft.VisualStudio.Services.* assemblies, and System.Net.Http abstractions. It is a real conflict risk if a test project is itself testing code that consumes those same packages at a different version. Splitting helps but only partially — once you reference the package, the binding redirects bite anyway.

Option B — single package, talk REST directly via HttpClient. The endpoints we need are small and stable:

• POST /{project}/_apis/test/runs?api-version=7.1
• POST /{project}/_apis/test/runs/{runId}/results?api-version=7.1
• POST /{project}/_apis/test/runs/{runId}/attachments?api-version=7.1
• PATCH /{project}/_apis/test/runs/{runId}?api-version=7.1
• GET /{project}/_apis/test/runs?…&minLastUpdatedDate=… for history
• GET /{project}/_apis/testplan/Plans/{planId}/Suites/{suiteId}/TestCase for test plans

Auth is just Authorization: Bearer $(System.AccessToken). No SDK, no Newtonsoft, no transitive surface — System.Text.Json is enough and we already use it elsewhere in MTP. This is the same approach the GitHub Actions runner and most CI integrations take, and it dodges the dep-conflict problem entirely.

I'd lean Option B: one package (Microsoft.Testing.Extensions.AzureDevOpsReport), feature-flagged by CLI options, with internal modules for "stdout logging commands" (today) vs "REST client" (new). Discoverability stays trivial — users only need to remember one package name — and the only growth is in the option surface.

If we ever genuinely need the typed SDK (e.g. for Test Plans complex types), my fallback would be: single package + an isolated AssemblyLoadContext on netcoreapp to load the SDK assemblies side-by-side without leaking into the user's resolution, with a small REST-only shim on netstandard2.0/netfx where ALC isolation isn't viable. But I don't think we'll need that for the scenarios above.

Suggested rollout

1. (Bug Fix #258333 : Tests running multiple times in case we have multiple test projects #4) Summary + progress + configurable stack filter — pure logging, lands in the existing extension, no new dep, no new package.
2. (#3a) Pipeline-level artifact upload — same.
3. (Initial commit! 🎉 #1) Live publishing via REST (no SDK dep).
4. (#3b) Result-level attachments on top of Initial commit! 🎉 #1.
5. (Porting latest changes. #2) Flaky-history annotations (reuses Initial commit! 🎉 #1's REST client).
6. (Changes to move to Om 11.0 nuget pacakge. #5) Test Plans support (largest scope).

Happy to turn any one of these into a focused design issue / PR — say which.

[Youssef1313]

What the user gets today is a couple of console lines per failure. We never populate the AzDO Tests tab — that still requires generating a TRX and adding a downstream PublishTestResults@2 task.

That's today's behavior, yes. But generally, populating AzDO Tests tab doesn't require generating a TRX or using PublishTestResults, if we implemented the stuff programmatically.