Skip to content

Update --show-stdout / --show-stderr help text to reflect LLM-aware default - #8797

Merged
Evangelink merged 1 commit into
mainfrom
dev/amauryleve/fix-8772
Jun 3, 2026
Merged

Update --show-stdout / --show-stderr help text to reflect LLM-aware default#8797
Evangelink merged 1 commit into
mainfrom
dev/amauryleve/fix-8772

Conversation

@Evangelink

Copy link
Copy Markdown
Member

Follow-up from #8771 / fixes #8772.

When an LLM/AI agent environment is detected, the defaults for --show-stdout and --show-stderr flip from All to Failed (see TerminalOutputDevice.GetShowOutputMode). The option descriptions in PlatformResources.resx still hard-coded Default is 'All'. though, so agents invoking --help saw misleading guidance about the very default that was changed for their benefit.

Changes

  • Updated TerminalShowStdoutOptionDescription and TerminalShowStderrOptionDescription in PlatformResources.resx to:

    Valid values are 'All', 'Failed', 'None'. Default is 'All' (or 'Failed' when an LLM/AI agent environment is detected).

  • Regenerated the 13 XLF localization files via dotnet msbuild .../Microsoft.Testing.Platform.csproj /t:UpdateXlf.

  • Updated the wildcard / regex expectations in:

    • test/IntegrationTests/Microsoft.Testing.Platform.Acceptance.IntegrationTests/HelpInfoTests.cs
    • test/IntegrationTests/Microsoft.Testing.Platform.Acceptance.IntegrationTests/HelpInfoAllExtensionsTests.cs
    • test/IntegrationTests/MSTest.Acceptance.IntegrationTests/HelpInfoTests.cs

Validation

Locally green on net8.0 / net10.0 / net462:

  • HelpInfoTests (21 tests) — passed
  • HelpInfoAllExtensionsTests (9 tests) — passed
  • MSTest HelpInfoTests (6 tests) — passed

…efault

Mention that the default flips from 'All' to 'Failed' when an LLM/AI agent environment is detected (added in #8771).

Fixes #8772

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings June 3, 2026 11:36

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Updates Microsoft.Testing.Platform’s CLI help text for --show-stdout / --show-stderr so it accurately describes the LLM/AI-aware default behavior introduced in #8771, and keeps the help/info acceptance tests and localized resources in sync.

Changes:

  • Updated the two terminal option description resource strings to document the conditional default (All, or Failed in detected LLM/AI agent environments).
  • Regenerated all 13 localized XLF files for the updated resources.
  • Updated help/info acceptance test expectations (including regex escaping where needed for the --info output).
Show a summary per file
File Description
test/IntegrationTests/MSTest.Acceptance.IntegrationTests/HelpInfoTests.cs Updates expected --help output text for --show-stdout / --show-stderr.
test/IntegrationTests/Microsoft.Testing.Platform.Acceptance.IntegrationTests/HelpInfoTests.cs Updates expected --help and --info outputs (regex-escaped) for the revised option descriptions.
test/IntegrationTests/Microsoft.Testing.Platform.Acceptance.IntegrationTests/HelpInfoAllExtensionsTests.cs Updates expected --help and --info outputs (with all extensions) for the revised option descriptions.
src/Platform/Microsoft.Testing.Platform/Resources/PlatformResources.resx Updates the canonical English resource strings for the two option descriptions.
src/Platform/Microsoft.Testing.Platform/Resources/xlf/PlatformResources.cs.xlf Regenerated localization file reflecting updated source strings (marked for review).
src/Platform/Microsoft.Testing.Platform/Resources/xlf/PlatformResources.de.xlf Regenerated localization file reflecting updated source strings (marked for review).
src/Platform/Microsoft.Testing.Platform/Resources/xlf/PlatformResources.es.xlf Regenerated localization file reflecting updated source strings (marked for review).
src/Platform/Microsoft.Testing.Platform/Resources/xlf/PlatformResources.fr.xlf Regenerated localization file reflecting updated source strings (marked for review).
src/Platform/Microsoft.Testing.Platform/Resources/xlf/PlatformResources.it.xlf Regenerated localization file reflecting updated source strings (marked for review).
src/Platform/Microsoft.Testing.Platform/Resources/xlf/PlatformResources.ja.xlf Regenerated localization file reflecting updated source strings (marked for review).
src/Platform/Microsoft.Testing.Platform/Resources/xlf/PlatformResources.ko.xlf Regenerated localization file reflecting updated source strings (marked for review).
src/Platform/Microsoft.Testing.Platform/Resources/xlf/PlatformResources.pl.xlf Regenerated localization file reflecting updated source strings (marked for review).
src/Platform/Microsoft.Testing.Platform/Resources/xlf/PlatformResources.pt-BR.xlf Regenerated localization file reflecting updated source strings (marked for review).
src/Platform/Microsoft.Testing.Platform/Resources/xlf/PlatformResources.ru.xlf Regenerated localization file reflecting updated source strings (marked for review).
src/Platform/Microsoft.Testing.Platform/Resources/xlf/PlatformResources.tr.xlf Regenerated localization file reflecting updated source strings (marked for review).
src/Platform/Microsoft.Testing.Platform/Resources/xlf/PlatformResources.zh-Hans.xlf Regenerated localization file reflecting updated source strings (marked for review).
src/Platform/Microsoft.Testing.Platform/Resources/xlf/PlatformResources.zh-Hant.xlf Regenerated localization file reflecting updated source strings (marked for review).

Copilot's findings

  • Files reviewed: 17/17 changed files
  • Comments generated: 0

@Evangelink Evangelink left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Expert Review — PR #8797: Update --show-stdout / --show-stderr help text to reflect LLM-aware default

21/21 dimensions clean — no findings.


Dimension Assessment

# Dimension Verdict
1 Algorithmic Correctness ✅ N/A — no logic changed
2 Threading & Concurrency ✅ N/A — no shared state touched
3 Security & IPC Contract Safety ✅ N/A
4 Public API & Binary Compatibility ✅ N/A — no API surface changed
5 Performance & Allocations ✅ N/A
6 Cross-TFM Compatibility ✅ N/A
7 Resource & IDisposable Management ✅ N/A
8 Defensive Coding at Boundaries ✅ N/A
9 Localization & Resources ✅ LGTM — .resx updated; all 13 XLF files regenerated via dotnet msbuild /t:UpdateXlf, not manually edited; <source> blocks updated and stale <target> state correctly demoted to needs-review-translation in each locale (cs, de, es, fr, it, ja, ko, pl, pt-BR, ru, tr, zh-Hans, zh-Hant).
10 Test Isolation ✅ N/A
11 Assertion Quality ✅ LGTM — test literals match the new string exactly; the one location that uses regex matching (HelpInfoTests.cs ~line 287) correctly escapes the parentheses as \( and \).
12 Flakiness Patterns ✅ N/A
13 Test Completeness & Coverage ✅ LGTM — all three help-info test files updated: HelpInfoTests.cs (MTP), HelpInfoAllExtensionsTests.cs (MTP), HelpInfoTests.cs (MSTest). Both the short-form help block and the verbose diagnostic block (--list-tests style) are covered.
14 Data-Driven Test Coverage ✅ N/A
15 Code Structure & Simplification ✅ N/A
16 Naming & Conventions ✅ N/A
17 Documentation Accuracy ✅ LGTM — new text accurately reflects TerminalOutputDevice.GetShowOutputMode behavior: default is All, flips to Failed when LLM/AI agent environment detected. Wording is concise and correct.
18 Analyzer & Code Fix Quality ✅ N/A
19 IPC Wire Compatibility ✅ N/A
20 Build Infrastructure & Dependencies ✅ N/A
21 Scope & PR Discipline ✅ LGTM — single concern: string update + XLF regeneration + test expectations. No mixed refactoring. References #8771 and fixes #8772.

Notes

  • The needs-review-translation state in the XLF files is the correct XLIFF lifecycle state for "source changed, translation pending re-review." No action needed beyond what the localization pipeline will handle automatically.
  • The parentheses-escaping difference between test files is intentional: files that use Contains or verbatim string matching use literal (); files that use regex matching escape to \(\). Both are correct for their respective assertion styles.

Generated by Expert Code Review (on open) for issue #8797 · sonnet46 1.8M

@Evangelink
Evangelink enabled auto-merge (squash) June 3, 2026 13:44
@Evangelink
Evangelink merged commit b8a83c6 into main Jun 3, 2026
36 checks passed
@Evangelink
Evangelink deleted the dev/amauryleve/fix-8772 branch June 3, 2026 13:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Update --show-stdout / --show-stderr help text to reflect LLM-aware default

3 participants