Show slowest tests

[Evangelink]

Summary

Allow to display slowest tests for a given assembly.

Background and Motivation

On our internal test runs we have an option to display slowest tests. I think this would be a useful feature to provide to users.

Proposed Feature

Add a --show-slowest or --show-slowest-test option.

[Evangelink]

To de defined: should this be a core feature or an extension feature?

I think that because this is not linked to any external deps/service this could be a built-in feature but we could think about having it directly as some external package it we are planning to connect it to some service (AzDO?).
@MarcoRossignoli if I am not mistaken, nothing prevents us (technically) to have this feature as core (built-in) feature and to potentially have some extensions that would enrich this feature (e.g. connection to some history service), right?

[MarcoRossignoli]

I would personally do not add "questionable" service built-in. UX falls already in that bucket but we need to show something and we're giving our taste.
I would keep as an extension.

Slowest test is also a not well defined concept. Hypothetical fixture system, init feature etc...are affecting the concept of slowest? If my custom adapter group by "group of tests with a category" and I want to measure that group? Some other esoteric framework could implement different concept regarding timing, personally I would not add in the core lib but in an extension where we ship "our" idea of slowest.

[Youssef1313]

When implementing this, we should also consider how we want dotnet test to behave.

Should dotnet test re-implement this feature in a way that considers all test projects and show the slowest test for the whole test session? Or should this be a per-testapp feature.

This question generally keeps popping up for various features.

[Evangelink]

Adding a thought from a recent design discussion (#9125, #9139, #9140) — a related angle worth capturing on this issue.

A different framing: declared expected duration

Instead of "platform calculates which tests are slowest" (relative, list-based — the original framing of this issue), what if:

Each test declares its expected duration (or expected range); the platform surfaces a warning when actual > declared.

That's a declarative twin of #9140's history-driven threshold — same emission point, different input source. Author knows this integration test should complete in under 2s; CI, IDE, and reporters all surface a warning when it doesn't.

Three architectural options

	Approach	Cost	Trade-off
a	Pure extension. Each framework ships its own attribute (e.g. [ExpectedDuration(2000)] on MSTest, equivalent on xUnit/NUnit). An extension reads it, compares, warns via the IProgressEnricher hook (#9139).	Zero MTP changes.	N frameworks → N attribute conventions → N extensions. Reporters can't act uniformly without per-framework knowledge.
b	Platform contract. MTP defines a standard test-node metadata key (expected.duration.max etc.); framework bridges populate it from whatever attribute the framework chooses; platform emits a SlowExpectedTest event when exceeded.	Public API addition + buy-in from adapter authors.	One standard surface; reporters and extensions act uniformly.
c	Hybrid. Platform exposes the metadata key + warning event; frameworks may populate it; UI/log emission is delegated to extensions/reporters.	Modest.	Probably the right balance with Marco's earlier don't put taste in core guidance on this very issue.

The "what state?" question — recommend not introducing a new TestOutcome

A slow-but-correct test passed its assertions. Marking it failed:

• Breaks the meaning of fail — green/red CI signal becomes ambiguous (people chase non-bugs).
• Maps awkwardly into TRX/JUnit XML — no PassedWithWarning state exists in those formats; they'd round to pass anyway.
• Conflicts mentally with the hard-timeout semantics frameworks already have ([Fact(Timeout = 1000)], MSTest [Timeout], NUnit [Timeout] all FAIL the test when exceeded).

Recommended principle: outcome stays Passed; the warning rides on a separate annotation channel on the result. Reporters render yellow/orange; CI extensions emit ::warning:: (GitHub) or ##vso[task.logissue type=warning] (AzDO); IDEs can surface a slow tests filter. Test still counts as passing in counts and exit codes.

If a user genuinely wants hard failure on slow tests, the existing Timeout attributes already do that.

Coexistence with existing Timeout

Two distinct concepts that can coexist on the same test:

Concept	Behaviour	Existing API
Hard timeout	Abort the test, mark Failed	[Fact(Timeout)], MSTest [Timeout], NUnit [Timeout]
Expected duration (soft)	Let the test finish, mark Passed, attach a warning	new — this issue

Example intent: [ExpectedDuration(2_000)] [Timeout(10_000)] → "I expect this in ≤2s, kill it at 10s."

Is there a point at all in having this on MTP side?

The blunt answer: only if we choose (b) or (c).

• One standard mechanism > N framework-specific ones — reporters / CI extensions don't need per-framework knowledge.
• Plugs directly into the new IProgressEnricher surface from Silence-driven progress heartbeat renderer for SimpleAnsi/NoAnsi terminal modes #9139.
• Connects with Azure DevOps history-driven slow-test threshold enricher #9140 — both the declarative threshold (this issue) and the historical-data threshold (Azure DevOps history-driven slow-test threshold enricher #9140) feed the same warning channel; consumers don't care which one fired.
• Enables uniform IDE integration (Test Explorer can show slow tests across frameworks).

If we choose (a), it's purely a framework-by-framework story and MTP should stay out of it (which is consistent with @MarcoRossignoli's earlier comment on this issue).

Open questions

1. (a), (b), or (c)? Leaning (c) personally — keeps core lean while making the data uniform.
2. Single max only, or full min/target/max range? Range adds unexpectedly fast detection (regression-the-other-way), but more API surface.
3. Per data row or per method for parameterised tests?
4. Should the same channel accept both absolute (2s) and relative (3× historical p99, from Azure DevOps history-driven slow-test threshold enricher #9140) declarations, so reporters have one thing to render?

Related design context: #9125 (umbrella for progress reporting), #9139 (IProgressEnricher hook surface this would plug into), #9140 (history-driven threshold — the dynamic twin of this declarative proposal).