[RFC] Agent/LLM-efficient test output for Microsoft.Testing.Platform

[Evangelink]

Summary

Open an umbrella discussion to track LLM- and agent-efficiency improvements for Microsoft.Testing.Platform (MTP) and MSTest output.

LLMs and coding agents (Copilot, Claude Code, Cursor, Codex, Aider, …) are increasingly the primary readers of dotnet test output. Each token of noise we emit translates into context-window pressure, slower agent loops, and more LLM cost. The repo already has scaffolding for this (LLMEnvironmentDetector, LLM-aware --show-stdout/--show-stderr defaults via #8771, --list-tests json via #8280, --no-ansi/--no-progress flags via #7647/#7649), but several large wins are still missing.

This RFC-style issue collects the candidate improvements and links to focused sub-issues so they can be discussed, prioritized, and shipped individually.

Motivation

A typical agent flow today looks like:

1. Agent runs dotnet test (or dotnet run --project ... for MTP).
2. The test host prints a banner, runtime info, a live progress region, and per-test outcome lines (~all in color).
3. On failure, each failing test prints message + expected/actual + stack trace including ~10 MTP/MSTest plumbing frames.
4. The agent must reread the whole transcript to figure out which tests failed and where.

Symptoms an agent sees today, all of which cost tokens:

• ANSI color escape codes are emitted in non-LLM CI runs unless NO_COLOR-equivalent flags are set. The de-facto industry standard env var NO_COLOR is not honored by name.
• Stack traces include the full MTP plumbing chain (Microsoft.Testing.Platform.Hosts.*, Microsoft.Testing.Platform.MSTest.*, Microsoft.VisualStudio.TestPlatform.MSTestAdapter.*, …). User code is only a handful of frames; the rest is noise.
• There is no compact "what failed and where" footer, so the agent has to re-parse the whole log.
• There is no first-class machine-readable test-results output (TRX is XML and verbose; --list-tests json only covers discovery; HTML report is not parseable). Each failure block is paragraph-style text.
• --show-stdout / --show-stderr help text still claims default is All even though it flips to Failed in LLM mode (tracked at Update --show-stdout / --show-stderr help text to reflect LLM-aware default #8772, still open).
• dotnet test/MSBuild wrapping can hide MTP's failure detail behind TestingPlatformShowTestsFailure=false (see No failing test details with MTP and dotnet test support with xunit v3 #5188).

Detailed proposals

Each item below should be filed as a separate issue (linked when filed) so it can be designed, sized, and shipped independently.

Output: noise reduction

• Honor NO_COLOR env var natively (industry standard, see https://no-color.org). Treat any non-empty NO_COLOR like --no-ansi. --ansi on|off|auto should still take precedence.
  → Honor NO_COLOR environment variable in MTP terminal output #8825

• Filter MTP/MSTest internal stack frames in the terminal failure output. Hide frames whose declaring type lives under Microsoft.Testing.Platform.* (non-public namespaces) or Microsoft.VisualStudio.TestPlatform.MSTestAdapter.*. Keep user-code frames. Apply at the rendering layer (FormatStackTrace/AppendStackFrame in TerminalTestReporter.Formatting.cs) so raw exception data flowing to TRX/HTML/AzDO reports is untouched. Default-on in LLM mode, opt-in flag (--full-stack-trace) for humans who want everything.
  → Filter MTP/MSTest internal stack frames from terminal failure output #8826 — closed as not planned (decided against: internal MTP/MSTest frames are kept for debuggability)

• Deduplicate identical failures across tests. Group failures whose (exception type + first N stack frames + first line of message) are identical; print details once, reference subsequent failures by count + list of test ids. Reduces N×output bloat when a shared helper breaks.
  → sub-issue: to be filed

• Crash/hang dump output volume — verify Excessive output when crashdump kicks in #7045 fix scope; sometimes a dump still prints multi-screen native stacks before the test summary.

Output: machine-readable / structured

• Microsoft.Testing.Extensions.JsonReport NDJSON event stream (new extension, modeled on TrxReport/HtmlReport/AzureDevOpsReport). One JSON object per test event (started/completed) + a final summary. Stable schema documented in PACKAGE.md. Inspired by Jest's --json, pytest-json-report, and Vitest's JSON reporter.
  → Add JSON / NDJSON test report extension (Microsoft.Testing.Extensions.JsonReport) #8828

• Compact end-of-run failure index footer (sentinel-delimited block listing only Test FQN | first-line error | file:line per failure). Agents can grep that block alone instead of re-reading the full log. Default-on in LLM mode only, no CLI flag to avoid help/info-snapshot churn for the first revision.
  → Emit compact end-of-run failure index for LLM/agent consumption #8827

• OpenTelemetry sanitization — sanitize raw paths/PII/stack traces before export ([MTP Diagnostic Analysis] OpenTelemetry exports raw file paths, stdout/stderr, and stack traces without filtering #8411 already open).

Input: agent ergonomics

• Fix --show-stdout/--show-stderr help text to honestly describe the LLM-mode-aware default (Update --show-stdout / --show-stderr help text to reflect LLM-aware default #8772 already open). Needs XLF regen + help/info acceptance updates.

• Reduce default banner/preamble in LLM mode. "Microsoft (R) Testing Platform vX.Y.Z" + runtime info + per-extension banners cost tokens before any test even runs. In LLM mode, skip or compress to a single line.

• --help --output json / --info --output json for machine-readable help/info. Agents can query for the option they need without scanning ~160 lines of text.

• Stable error codes for common misconfigurations. Today, errors like "option X requires Y" are English-only — give them codes (e.g. MTP1001) so agents can match without prompt-engineering.

dotnet test / MSBuild wrapping

• Agent-friendly dotnet test defaults / documentation. TestingPlatformShowTestsFailure=false by default in Microsoft.Testing.Platform.MSBuild.targets means agents only see a pointer to a log file unless they opt in. Flip default to true when an LLM env is detected (or document the env var to flip it). Cross-link No failing test details with MTP and dotnet test support with xunit v3 #5188.

Documentation

• Public doc page: "Microsoft.Testing.Platform for AI agents." Document LLMEnvironmentDetector behavior, detected env vars, NO_COLOR, recommended flags, JSON report extension.

• AGENTS.md at repo root referencing .github/copilot-instructions.md (industry convention emerging in 2025).

Drawbacks

• Each item changes some part of the terminal transcript and risks breaking log-scraping CI consumers. Mitigations: gate behavioral changes to LLM-mode-only by default, expose opt-in/opt-out flags for humans, and keep raw report data (TRX/HTML/AzDO) untouched.
• Adding a new JSON report extension grows the package surface and the maintenance footprint.
• Stack-trace filtering can hide a frame that's actually relevant to extension authors debugging MTP itself; needs an opt-out path.

Alternatives

• Do nothing and rely on agents to learn to ignore the noise. Trades user tokens/cost.
• Wait for the .NET SDK to standardize an agent-friendly test output. Doesn't help today.

Compatibility

Not a breaking change if behavioral defaults stay LLM-mode-gated and human-mode output is unchanged.

Unresolved questions

• Should the JSON report extension be packaged separately or rolled into an existing one?
• Should NO_COLOR support also imply --no-progress, or stay color-only as the spec says?
• Should the failure-index footer be a flag (--failure-index / --no-failure-index) or env var, or strictly LLM-mode-gated for v1?

References

• src/Platform/Microsoft.Testing.Platform/Helpers/LLMEnvironmentDetector.cs
• src/Platform/Microsoft.Testing.Platform/OutputDevice/TerminalOutputDevice.cs (LLM-aware behavior switches)
• src/Platform/Microsoft.Testing.Platform/OutputDevice/Terminal/ExceptionFlattener.cs (stack trace passthrough)
• src/Platform/Microsoft.Testing.Platform/OutputDevice/Terminal/TerminalTestReporter.Formatting.cs (stack-frame rendering)
• Related: Add "no progress" and "no ANSI" environment variables #7647, Avoid ANSI and progress output when running in LLM environment #7649, [Lean Squad] feat(fv): Task 1 — expand TARGETS.md to 17 targets; add LLMEnvironmentDetector research #8084, Add --list-tests json for machine-readable test discovery output #8280, [MTP Diagnostic Analysis] OpenTelemetry exports raw file paths, stdout/stderr, and stack traces without filtering #8411, Suppress banner and default stdout/stderr to failed in LLM environments #8771, Update --show-stdout / --show-stderr help text to reflect LLM-aware default #8772, No failing test details with MTP and dotnet test support with xunit v3 #5188, Excessive output when crashdump kicks in #7045
• External: https://no-color.org, Jest --json, pytest-json-report, Vitest reporters

Filed by @Evangelink based on an analysis of the MTP/MSTest output surfaces from an agent-consumer perspective.

[Evangelink]

Adding an energy-efficiency lens to these proposals, since reducing LLM agent output volume has a measurable energy dimension.

The energy cost of noisy test output

LLM-based agents are increasingly the primary consumers of dotnet test output. Each extra token in the output:

1. Consumes GPU energy during the LLM's decode pass — token generation is the dominant cost in transformer inference.
2. Increases context window pressure, potentially forcing a re-prompt (another full-context inference) when the window fills.
3. Delays agent iteration loops — more tokens to process = more wall-clock time = more energy per CI/CD cycle.

Rough estimate: a typical failing dotnet test run with 5 failed tests currently outputs ~200 lines including ANSI codes, banners, runtime info, and full stack traces. Modern LLMs running at ~10 ms/token decode cost ~100 ms to process that block. At millions of CI runs per day across the .NET ecosystem, even a 50 % output reduction would translate to material token-processing energy savings.

Prioritised by energy impact

Ordering the RFC proposals by estimated energy reduction per agent invocation:

Proposal	Token Reduction	Energy Impact	Notes
Filter MTP/MSTest internal stack frames (#8826)	HIGH (~60–80 % of stack trace token count)	HIGH	Stack frames are long strings; filtering plumbing frames is the biggest single win
Compact failure-index footer (#8827)	MEDIUM	MEDIUM	Allows agents to skip reading the body entirely — whole-context reduction
Microsoft.Testing.Extensions.JsonReport NDJSON (#8828)	MEDIUM	MEDIUM	Machine-readable = no redundant re-parsing; compact format
Honor NO_COLOR (#8825, ✅ done)	LOW-MEDIUM	LOW-MEDIUM	ANSI escape codes add ~10–15 % to raw byte count in failure output
Reduce banner in LLM mode (✅ done)	LOW	LOW	Banner is small relative to test output

Stack-frame filtering: a note on implementation energy

The proposal to filter in FormatStackTrace/AppendStackFrame at the rendering layer (not at the exception capture layer) is the right call from an energy perspective too: it means the filtering happens once, at print time, rather than on every code path that might consume the exception (TRX writer, HTML report, AzDO reporter). No double-processing cost.

Suggestion: measure output size as a CI health metric

Once the LLM-mode changes land, it would be worth tracking raw byte count of dotnet test output for a representative test suite as a CI signal — similar to how bundle size is tracked for web apps. A sudden output-size increase flags a noisy new message before it ships. This would give the project an ongoing "output energy budget" anchor.

Green Software Foundation framing

These proposals collectively address Demand Shaping (emit less data = less downstream processing demand) and Hardware Efficiency (agent GPU inference cost per run). The compact failure-index footer in particular is a good example of shaping demand at the source — agents only need to read one small block instead of scanning the whole log.

🤖 Automated content by GitHub Copilot. Posted via a maintainer's GitHub token, so it appears under their account — the account owner did not write or approve this content personally. Generated by the Efficiency Improver workflow. · 2.3K AIC · ⌖ 40.6 AIC · ⊞ 58.6K · [◷]( · ◷)

Add this agentic workflows to your repo

To install this agentic workflow, run

gh aw add githubnext/agentics/workflows/efficiency-improver.md@main

[Evangelink]

🔄 State refresh — sub-issue reconciliation

Reviewed the linked sub-issues and the current codebase state. Summary of where each RFC item stands:

✅ Completed / merged

• Honor NO_COLOR — Honor NO_COLOR environment variable in MTP terminal output #8825 ✅ closed (completed). NO_COLOR handling is in the terminal output device.
• Fix --show-stdout / --show-stderr help text — Update --show-stdout / --show-stderr help text to reflect LLM-aware default #8772 ✅ closed (completed).
• Reduce banner/preamble in LLM mode — Suppress banner and default stdout/stderr to failed in LLM environments #8771 ✅ merged (banner suppression + Failed stdout/stderr default in LLM envs).
• Crash/hang dump output volume — Excessive output when crashdump kicks in #7045 ✅ closed (completed).
• --list-tests json (prior art) — Add --list-tests json for machine-readable test discovery output #8280 ✅ merged.

🚫 Not planned (state corrected in checklist)

• Filter MTP/MSTest internal stack frames — Filter MTP/MSTest internal stack frames from terminal failure output #8826 was closed as not planned. Updated the checklist entry to reflect that the internal frames are intentionally kept for debuggability. If this should be revisited, we'd reopen with an opt-in flag design.

🟡 Still open

• JSON / NDJSON report extension — Add JSON / NDJSON test report extension (Microsoft.Testing.Extensions.JsonReport) #8828 (open).
• Compact end-of-run failure index footer — Emit compact end-of-run failure index for LLM/agent consumption #8827 (open).
• OpenTelemetry sanitization — [MTP Diagnostic Analysis] OpenTelemetry exports raw file paths, stdout/stderr, and stack traces without filtering #8411 (open).
• Agent-friendly dotnet test / MSBuild defaults — cross-links No failing test details with MTP and dotnet test support with xunit v3 #5188 (open).

📥 Not yet filed (no sub-issue)

• Deduplicate identical failures across tests.
• --help --output json / --info --output json.
• Stable error codes for common misconfigurations.
• Public doc page "Microsoft.Testing.Platform for AI agents."
• AGENTS.md referencing .github/copilot-instructions.md — note: a root AGENTS.md exists today, but it is the gh-copilot-curate-managed skills index, not the copilot-instructions pointer this item asks for, so it remains unchecked.

Checkbox marks in the body were already accurate for everything except #8826, which is now annotated.