Add ITestFilter / [TestFilterProvider] for programmatic test filtering (#8894)

[Evangelink]

Summary

Implements the design from #8894: a small, opt-in extension point that lets users plug in their own filtering logic in MSTest before any test-class type is loaded or any [AssemblyInitialize] / [ClassInitialize] runs. This makes it cheap to drop or skip tests programmatically — the original motivation for the issue.

What's exposed

All new public API lives in Microsoft.VisualStudio.TestTools.UnitTesting:

• [assembly: TestFilterProvider(typeof(MyFilter))] — assembly-level attribute, AllowMultiple = false (at most one per test assembly), points at an ITestFilter implementation. Same shape as the existing [AssemblyFixtureProvider].
• interface ITestFilter — a single method: TestFilterResult Filter(TestFilterContext context).
• readonly struct TestFilterResult with static TestFilterResult Run, static TestFilterResult Drop, and static TestFilterResult Skip(string reason).
• enum TestFilterAction { Run = 0, Drop = 1, Skip = 2 }.
• sealed class TestFilterContext — surfaces only what we know without loading the test type. Designed as a parameterless-ctor + public mutable properties bag (no init, no required, no positional ctor) so we can add fields in future releases without breaking source or binary callers.
  • Flat identification (always populated): FullyQualifiedName, DisplayName, MethodName, Source (test assembly file path, matches VSTest TestCase.Source).
  • Structured identification (in TestMethodIdentifier shape, populated when discovery provides them; otherwise null): Namespace, ClassName, ManagedTypeName, ManagedMethodName, MethodArity, ParameterTypeFullNames. Parsed cheaply from the managed name — no MethodInfo reflection, so the type is still not loaded.
  • Test metadata: Categories, Traits, Priority.

Example

using Microsoft.VisualStudio.TestTools.UnitTesting;

[assembly: TestFilterProvider(typeof(SkipSlowTestsOnCI))]

public sealed class SkipSlowTestsOnCI : ITestFilter
{
    private static readonly bool IsCI =
        string.Equals(Environment.GetEnvironmentVariable("CI"), "true", StringComparison.OrdinalIgnoreCase);

    public TestFilterResult Filter(TestFilterContext context)
    {
        if (!IsCI)
        {
            return TestFilterResult.Run;
        }

        // Categories / Traits / Priority are surfaced without loading the test type.
        foreach (string category in context.Categories)
        {
            if (string.Equals(category, "Slow", StringComparison.OrdinalIgnoreCase))
            {
                return TestFilterResult.Skip("Slow tests are skipped on CI.");
            }
        }

        // Structured identification — same shape as TestMethodIdentifier, also without loading the type.
        if (context.Namespace?.StartsWith("Contoso.Internal.Diagnostics", StringComparison.Ordinal) == true)
        {
            return TestFilterResult.Drop;
        }

        return TestFilterResult.Run;
    }
}

Why a single filter per assembly

The original design allowed multiple [TestFilterProvider] attributes with AND composition. We simplified to a single filter per test assembly because:

• No ordering question to standardize. With multiple filters there is no obvious "right" order across assemblies/libraries, and any choice we expose becomes public API forever.
• No dedup or cross-assembly BFS. Discovery now inspects only the test assembly itself, which is cheaper and easier to reason about.
• Composition stays in user code, where it belongs. Users who want to combine multiple strategies (e.g. a library-provided filter + a CI filter) instantiate them inside their own ITestFilter and call them in the order they choose. Library authors expose ITestFilter types; the test author opts in explicitly.
• Single decision point per test. One call, one result, no precedence rules to document.

How it works

• Discovery is in a new partial TypeCache.FilterDiscovery.cs. It inspects only the test assembly itself (no BFS over references) via a fast CustomAttributeData probe, then materializes the attribute and validates the FilterType (non-generic, not abstract/interface, implements ITestFilter, has a public parameterless ctor). The resolved filter is cached per source assembly via TypeCache.GetOrLoadTestFilter so the cost is paid at most once per run. A small TestFilterBox wrapper lets the cache distinguish "not computed yet" from "computed and no filter declared".
• Invocation happens at the very top of UnitTestRunner.RunSingleTestAsync, before _typeCache.GetTestMethodInfo. That means a Drop literally pays zero [AssemblyInitialize] / [ClassInitialize] / type-load cost — which is the whole point of the issue.
• Skip surfaces as a normal Ignored test result (same outcome as [Ignore]).
• Filter exceptions become an Error test result (UTA078) so they can't silently drop tests.
• Multiple [TestFilterProvider] on the same assembly are rejected at discovery time with UTA079 instead of silently using the first one.

Validation hardening (post code-review pass)

• TestFilterResult.Skip(reason) now throws on null / empty / whitespace reasons, matching the XML doc contract.
• ITestFilter XML docs now state the UTA078 contract on Filter exceptions and clarify that the resolved filter is cached per source assembly and reused for every test in that assembly.
• New unit tests cover TestFilterResult validation and equality plus every InstantiateTestFilter error branch (UTA074 generic, UTA075 abstract/interface, UTA076 wrong base type, UTA077 ctor failure / non-public ctor).

API redesign (second review pass)

Pivoted TestFilterContext from a 7-arg positional ctor + init properties to a parameterless ctor + plain mutable properties. Rationale:

• init is banned for new public API in this repo (see .github/copilot-instructions.md). Mutable set accessors are required.
• Forward-compatible by construction. Adding new optional properties later is a pure additive change — no ctor overloads, no source breaks, no binary breaks for callers using object-initializer syntax.
• Structured identification surfaces the same shape as MTP's TestMethodIdentifier (Namespace, ClassName, ManagedTypeName, ManagedMethodName, MethodArity, ParameterTypeFullNames), populated from the Hierarchy[Namespace/Class] slots that discovery already produces and from a cheap ManagedNameParser.ParseManagedMethodName call (no MethodInfo, no reflection, no type-load). Flat fields stay too for callers that just want a string match.
• Source renamed from AssemblyName. The underlying TestMethod.AssemblyName is actually a file PATH (matches VSTest TestCase.Source); calling it AssemblyName would mislead users.
• [Ignore] ordering is now documented on ITestFilter.Filter. The filter runs before [Ignore] is evaluated ([Ignore] requires loading the type, which the filter is specifically designed to avoid). Returning Run therefore does NOT override a later [Ignore].

Explicitly out of scope

Two reviewer-raised questions were considered and intentionally not addressed:

• Bypassing default adapter filtering (e.g. --filter, [Ignore]). NO — preserves the contracts CI scripts and IDEs rely on. ITestFilter is layered on top of, not in place of, those gates.
• Exposing already-included / already-excluded test info. NO — adapter-level --filter runs before ITestFilter, so the filter only ever sees survivors of that gate. "Already-included" would be tautological, and tracking "already-excluded" across the run would force materializing state we currently don't need.

Assembly cleanup edge case

The cleanup tail of RunSingleTestAsync previously gated end-of-assembly cleanup on testMethodInfo?.Parent.Parent.IsAssemblyInitializeExecuted == true. If the last test of an assembly is filtered out, testMethodInfo is null and that guard never fires. Fixed by introducing a runner-level _assemblyInitializeWasExecuted flag that survives across filtered-out tests and is OR'd into the guard inside the new FinishFilteredOutTestAsync.

Out of scope (deferred follow-ups)

• A MSTESTNNN analyzer to validate [TestFilterProvider] targets at compile time (public + parameterless, implements ITestFilter, etc.). Diagnostics are currently runtime-only (UTA073–UTA078).
• Acceptance tests covering Drop + ClassInit pairing and last-test-of-assembly cleanup. The new code is covered indirectly by the existing adapter unit-test suite (all green); behavioral tests are intentionally left for a follow-up so this PR stays scoped to the API surface + plumbing.
• Trace-logging the number of dropped tests so users can spot accidental filter drops.

Validation

• build.cmd -projects src\Adapter\MSTestAdapter.PlatformServices\MSTestAdapter.PlatformServices.csproj — green.
• build.cmd -projects src\TestFramework\TestFramework\TestFramework.csproj — green.
• build.cmd -test -projects test\UnitTests\MSTestAdapter.PlatformServices.UnitTests\MSTestAdapter.PlatformServices.UnitTests.csproj — all TFMs pass (net462, net48, net8.0, net9.0, net8.0-windows10.0.18362.0).
• build.cmd -test -projects test\UnitTests\TestFramework.UnitTests\TestFramework.UnitTests.csproj — all TFMs pass (net48, net8.0, net9.0, net8.0-windows10.0.18362.0).

Closes #8894 once design questions in the issue are settled.

[Copilot]

Pull request overview

Adds an opt-in, programmatic filtering extension point to MSTest via a new [assembly: TestFilterProvider] attribute and ITestFilter interface, allowing tests to be dropped or skipped before any test type is loaded or any assembly/class initialization runs (reducing upfront cost for filtered-out tests).

Changes:

• Introduces new public filtering API surface in Microsoft.VisualStudio.TestTools.UnitTesting (ITestFilter, TestFilterContext, TestFilterResult, TestFilterAction, TestFilterProviderAttribute) and registers it in PublicAPI.Unshipped.txt.
• Implements adapter-side discovery/caching of filter providers (BFS over referenced assemblies) and applies filters at the start of UnitTestRunner.RunSingleTestAsync.
• Adds localized diagnostics/resources for filter-provider discovery/instantiation/invocation failures (UTA073–UTA078).

Show a summary per file

File	Description
src/TestFramework/TestFramework/PublicAPI/PublicAPI.Unshipped.txt	Declares newly introduced public APIs for the filtering feature.
src/TestFramework/TestFramework/Filtering/ITestFilter.cs	New interface that users implement to decide Run/Drop/Skip per test.
src/TestFramework/TestFramework/Filtering/TestFilterContext.cs	New metadata-only context passed to filters (no type load).
src/TestFramework/TestFramework/Filtering/TestFilterAction.cs	New enum representing filter decision actions.
src/TestFramework/TestFramework/Filtering/TestFilterResult.cs	New result type returned by filters (Run/Drop/Skip).
src/TestFramework/TestFramework/Attributes/Lifecycle/TestFilterProviderAttribute.cs	New assembly-level attribute used to register filters.
src/Adapter/MSTestAdapter.PlatformServices/Execution/UnitTestRunner.cs	Applies filters before type-load/init and adds bookkeeping for filtered-out tests + assembly cleanup edge case.
src/Adapter/MSTestAdapter.PlatformServices/Execution/TypeCache.FilterDiscovery.cs	Discovers and instantiates registered filters (BFS over reference graph), caches per test assembly source.
src/Adapter/MSTestAdapter.PlatformServices/Resources/Resource.resx	Adds new UTA messages for filter provider diagnostics.
src/Adapter/MSTestAdapter.PlatformServices/Resources/xlf/Resource.cs.xlf	Localization entry updates for new diagnostics.
src/Adapter/MSTestAdapter.PlatformServices/Resources/xlf/Resource.de.xlf	Localization entry updates for new diagnostics.
src/Adapter/MSTestAdapter.PlatformServices/Resources/xlf/Resource.es.xlf	Localization entry updates for new diagnostics.
src/Adapter/MSTestAdapter.PlatformServices/Resources/xlf/Resource.fr.xlf	Localization entry updates for new diagnostics.
src/Adapter/MSTestAdapter.PlatformServices/Resources/xlf/Resource.it.xlf	Localization entry updates for new diagnostics.
src/Adapter/MSTestAdapter.PlatformServices/Resources/xlf/Resource.ja.xlf	Localization entry updates for new diagnostics.
src/Adapter/MSTestAdapter.PlatformServices/Resources/xlf/Resource.ko.xlf	Localization entry updates for new diagnostics.
src/Adapter/MSTestAdapter.PlatformServices/Resources/xlf/Resource.pl.xlf	Localization entry updates for new diagnostics.
src/Adapter/MSTestAdapter.PlatformServices/Resources/xlf/Resource.pt-BR.xlf	Localization entry updates for new diagnostics.
src/Adapter/MSTestAdapter.PlatformServices/Resources/xlf/Resource.ru.xlf	Localization entry updates for new diagnostics.
src/Adapter/MSTestAdapter.PlatformServices/Resources/xlf/Resource.tr.xlf	Localization entry updates for new diagnostics.
src/Adapter/MSTestAdapter.PlatformServices/Resources/xlf/Resource.zh-Hans.xlf	Localization entry updates for new diagnostics.
src/Adapter/MSTestAdapter.PlatformServices/Resources/xlf/Resource.zh-Hant.xlf	Localization entry updates for new diagnostics.

Copilot's findings

• Files reviewed: 22/22 changed files
• Comments generated: 3

[Copilot]

Copilot's findings

• Files reviewed: 24/24 changed files
• Comments generated: 3

[Copilot]

Copilot's findings

• Files reviewed: 26/26 changed files
• Comments generated: 2

[Copilot]

Copilot's findings

• Files reviewed: 26/26 changed files
• Comments generated: 1

[Copilot]

Review details

• Files reviewed: 26/26 changed files
• Comments generated: 0 new
• Review effort level: Low

[Evangelink]

🧪 Test quality grade — PR #8896

20 new test methods graded across 3 files — all earned A (90–100). The PR adds focused exception-with-error-code tests for TypeCache.InstantiateTestFilter validation branches, comprehensive equality-contract tests for TestFilterResult, and a clean edge-case method in TestExecutionManagerTests. No issues found; the test suite is exemplary.

Δ	Test	Grade	Band	Notes
new	TestExecutionManagerTests. SendTestResults_ WhenUnitTestResultsIsEmpty_ RecordsEndWithoutResult	A	90–100	Clear AAA; two assertions cover the None outcome and the empty results list.
new	TestFilterResultTests. Default_ DefaultsToRunAction	A	90–100	Comment documents the default-value contract; two assertions verify Action and SkipReason.
new	TestFilterResultTests. Drop_ HasActionDropAndNullReason	A	90–100	No issues found.
new	TestFilterResultTests. Equality_ BoxedComparisonAgainstNonResultIsFalse	A	90–100	No issues found.
new	TestFilterResultTests. Equality_ RunAndDropAreNotEqual	A	90–100	No issues found.
new	TestFilterResultTests. Equality_ TwoSkipResultsWithDifferentReasonsAreNotEqual	A	90–100	No issues found.
new	TestFilterResultTests. Equality_ TwoSkipResultsWithSameReasonAreEqual	A	90–100	Thorough equality contract: Equals, ==, !=, and GetHashCode all verified.
new	TestFilterResultTests. Run_ HasActionRunAndNullReason	A	90–100	No issues found.
new	TestFilterResultTests. Skip_ WhenReasonIsEmpty_ ThrowsArgumentException	A	90–100	Comment documents the UX rationale; parameter-name precision improves debuggability.
new	TestFilterResultTests. Skip_ WhenReasonIsNull_ ThrowsArgumentNullException	A	90–100	Exception test also checks the parameter name — precise guard-clause testing.
new	TestFilterResultTests. Skip_ WhenReasonIsWhitespace_ ThrowsArgumentException	A	90–100	No issues found.
new	TestFilterResultTests. Skip_ WithValidReason_ SetsActionAndReason	A	90–100	No issues found.
new	TypeCacheTestFilterProviderTests. InstantiateTestFilter_ WhenConstructorThrows_ ThrowsUTA077	A	90–100	Comment explains TargetInvocationException wrapping; exception + message assertion.
new	TypeCacheTestFilterProviderTests. InstantiateTestFilter_ WhenTypeDoesNotImplementITestFilter_ ThrowsUTA076	A	90–100	No issues found.
new	TypeCacheTestFilterProviderTests. InstantiateTestFilter_ WhenTypeIsAbstract_ ThrowsUTA075	A	90–100	No issues found.
new	TypeCacheTestFilterProviderTests. InstantiateTestFilter_ WhenTypeIsClosedGeneric_ ThrowsUTA074	A	90–100	No issues found.
new	TypeCacheTestFilterProviderTests. InstantiateTestFilter_ WhenTypeIsInterface_ ThrowsUTA075	A	90–100	No issues found.
new	TypeCacheTestFilterProviderTests. InstantiateTestFilter_ WhenTypeIsMissingPublicParameterlessConstructor_ ThrowsUTA077	A	90–100	No issues found.
new	TypeCacheTestFilterProviderTests. InstantiateTestFilter_ WhenTypeIsOpenGeneric_ ThrowsUTA074	A	90–100	Exception assertion with specific error-code message pattern; compact and focused.
new	TypeCacheTestFilterProviderTests. InstantiateTestFilter_ WhenTypeIsValid_ ReturnsInstance	A	90–100	NotBeNull chained with BeOfType verifies both non-null and correct concrete type.

_{This advisory comment was generated automatically. Grades are heuristic
and informational — they do not block merging. Re-run with
/grade-tests.}

🤖 Automated content by GitHub Copilot. Posted via a maintainer's GitHub token, so it appears under their account — the account owner did not write or approve this content personally. Generated by the Grade Tests on PR (on open / sync) workflow. · 333 AIC · ⌖ 25.2 AIC · ⊞ 45.7K · [◷]( · ◷)