diff --git a/docs/Changelog.md b/docs/Changelog.md index fc0c10db8a..15da56f9c6 100644 --- a/docs/Changelog.md +++ b/docs/Changelog.md @@ -10,6 +10,7 @@ See full log [of v4.3.3...v4.4.0](https://github.com/microsoft/testfx/compare/v4 ### Added +* Add `[DependsOn]` and equivalent `testconfig.json` declarations (`mstest:execution:dependencies`, with `chains` for straight sequences and `nodes` for fan-in/fan-out) to declare that a test runs after one or more other tests. Declarations form a directed acyclic graph rather than a flat order, so tests that share a prerequisite still run in parallel with each other; a test whose prerequisite does not pass is skipped (transitively) unless it sets `ProceedOnFailure`, and dependency cycles are reported before the run starts. The attribute works on both hosts; the `testconfig.json` declarations are Microsoft.Testing.Platform only. See [RFC 022](RFCs/022-Test-Dependencies.md) * Add dedicated timeout configuration for `[GlobalTestInitialize]` / `[GlobalTestCleanup]` fixtures via the `timeout:globalTestInitialize` / `timeout:globalTestCleanup` `testconfig.json` keys (RunSettings XML: `GlobalTestInitializeTimeout` / `GlobalTestCleanupTimeout`). These fall back to the per-test `testInitialize` / `testCleanup` timeouts when unset, and global fixture timeout/cancellation diagnostics now use dedicated messages ("Global test initialize/cleanup method ...") in [#9985](https://github.com/microsoft/testfx/issues/9985) ### Changed diff --git a/docs/RFCs/022-Test-Dependencies.md b/docs/RFCs/022-Test-Dependencies.md new file mode 100644 index 0000000000..d5ababe875 --- /dev/null +++ b/docs/RFCs/022-Test-Dependencies.md @@ -0,0 +1,338 @@ +# RFC 022 - Test Dependencies + +- [ ] Approved in principle +- [x] Under discussion +- [x] Implementation +- [ ] Shipped + +## Summary + +Add a `[DependsOn]` attribute to MSTest, plus equivalent **declarations in `testconfig.json`**, that let a test +declare which other tests must run before it. The declarations form a **directed acyclic graph**, not a +flat list: when several tests share a prerequisite they become runnable at the same moment and the +in-assembly parallel scheduler is free to run them concurrently. If a prerequisite does not pass, its +dependents are **skipped** rather than failed, and the skip propagates. + +This extends the parallelization model of +[RFC 004 - In-assembly Parallel Execution](004-In-Assembly-Parallel-Execution.md) and reuses the +scheduling machinery introduced by [RFC 020 - Resource Lock Attribute](020-Resource-Lock-Attribute.md). + +## Motivation + +MSTest has no way to say "run B after A". The two ordering knobs it does have are deliberately coarse: + +- `[Priority]` is **metadata only** — it is read at discovery (`TypeEnumerator.GetTestFromMethod`) and + never consulted by the executor. +- `OrderTestsByNameInClass` (3.6) sorts alphabetically inside a class, and `RandomizeTestOrder` (4.3) + exists precisely to *expose* accidental order dependencies. + +Users have asked for this repeatedly — [#25](https://github.com/microsoft/testfx/issues/25) ("Add support +for ordered tests", open since 2016), [#3162](https://github.com/microsoft/testfx/issues/3162) +("Controlling execution order of unit tests"), [#572](https://github.com/microsoft/testfx/issues/572). +The legacy answer, Visual Studio's `.orderedtest` XML, only ever worked in MSTest V1 and is a pure linear +sequence with no branching, no failure semantics, and GUIDs that make it unauthorable by hand. + +### Why this is not just an `[Order(int)]` attribute + +An integer order is a *total* order: it serializes everything it touches, so a suite that needs one +setup step before twenty tests pays for nineteen unnecessary serializations. A graph is a *partial* +order: it constrains only the pairs that genuinely depend on each other and leaves everything else free +to run in parallel. That difference is the whole value proposition, and it is what separates the +frameworks that do this well (TestNG, TUnit) from the ones that only sort (NUnit `[Order]`, xUnit's +`ITestCaseOrderer`, JUnit's `MethodOrderer`). + +It is also why the acceptance test for this feature asserts *overlap*, not just order. + +### Prior art + +| Framework | API | Failure → | Independent branches parallel | Cycles | File-based | +|---|---|---|---|---|---| +| **TUnit** | `[DependsOn(name / type / type+name)]`, `ProceedOnFailure` | **Skip** | **Yes** | DFS, up-front | No | +| **TestNG** | `@Test(dependsOnMethods, dependsOnGroups, alwaysRun)` | **Skip** | **Yes** | Yes | **Yes** (`testng.xml`) | +| **Playwright** | project `dependencies: [...]` | **Skip** | **Yes** | — | **Yes** (config) | +| **pytest** | `pytest-dependency` (`depends=[...]`) + `pytest-order` | **Skip** | No | No | No | +| **JUnit 5** | `@Order` / `MethodOrderer` only — dependencies deliberately rejected | — | No | — | properties | +| **NUnit / xUnit** | `[Order]` / `ITestCaseOrderer` — start order only | No | No | — | No | +| **MSTest (today)** | `[Priority]` (inert), `OrderTestsByNameInClass`, `RandomizeTestOrder` | — | — | — | sort options only | + +The consensus among the frameworks that actually model dependencies is unanimous on four points, and +this design follows all four: **skip, don't fail**; **provide an escape hatch**; **detect cycles +eagerly**; **keep independent branches parallel**. + +JUnit 5's refusal is a deliberate, well-argued position — test independence is a property worth +defending, and their non-obvious default ordering exists to stop people relying on order by accident. +This RFC does not dispute that for unit tests; see [Guidance](#guidance). + +## Design + +### API shape + +Namespace `Microsoft.VisualStudio.TestTools.UnitTesting`: + +```csharp +[AttributeUsage(AttributeTargets.Class | AttributeTargets.Method, AllowMultiple = true, Inherited = false)] +public sealed class DependsOnAttribute : Attribute +{ + public DependsOnAttribute(string testMethodName); // a method of the same class + public DependsOnAttribute(Type testClass); // every test of another class + public DependsOnAttribute(Type testClass, string testMethodName); + + public Type? TestClass { get; } + public string? TestMethodName { get; } + public bool ProceedOnFailure { get; set; } +} +``` + +```csharp +[TestClass] +public class CheckoutTests +{ + [TestMethod] + public void CreateCart() { } + + // Fan-out: both wait for CreateCart, then may run concurrently with each other. + [TestMethod, DependsOn(nameof(CreateCart))] public void AddItem() { } + [TestMethod, DependsOn(nameof(CreateCart))] public void ApplyCoupon() { } + + // Fan-in: waits for both. + [TestMethod, DependsOn(nameof(AddItem)), DependsOn(nameof(ApplyCoupon))] + public void PlaceOrder() { } + + // Runs even though its prerequisite failed. + [TestMethod, DependsOn(nameof(PlaceOrder), ProceedOnFailure = true)] + public void WriteAuditRecord() { } +} +``` + +### Design decisions + +**Skip, don't fail, when a prerequisite fails.** A failing prerequisite means the dependent's +precondition demonstrably did not hold, so its result carries no information about the code it tests. +Failing it would report one bug as N failures and bury the root cause. Skipping reports one failure +surrounded by clearly-labelled skips. The skip message names the test that did not pass, because a skip +with no reason is indistinguishable from a test that was filtered out. This matches TestNG, TUnit, +pytest-dependency and Playwright's serial mode. + +**`ProceedOnFailure` is per declaration, and merges conservatively.** A dependent proceeds only when +*every* edge it declares says it may; one edge asking for the ordinary skip is enough to hold it back. The +same rule applies when the *same* prerequisite is declared twice — for example at class scope with +`ProceedOnFailure = true` and again on the method with the default — so a broad opt-out can never silently +override a narrower declaration. Under-skipping runs a test whose precondition failed (a confusing spurious +failure); over-skipping only costs coverage that was already compromised. The conservative direction is +therefore to skip. + +**Not inherited across overrides.** `[ResourceLock]` and `[DoNotParallelize]` are inherited because +over-applying them is merely slower. A dependency is different: carrying one concrete test's prerequisites +onto a method that *overrides* it creates an edge nobody wrote, on a method the author rewrote. That +direction fails *dangerous*, not *slow*, so `Inherited = false`. Note this is only about override chains - +a base-declared test method still runs as a test of each derived class, and its dependency travels with +it, resolved against the derived class so that each derived class orders its own copies of the two tests. +Dropping it there would silently discard the ordering the author declared. + +**Cycles fail the tests in the cycle, and only those.** A cycle is a configuration error, so it is +reported before anything runs, as an error message naming the cycle path (`A > B > A`). The tests in the +cycle are reported as **failed** — they cannot be ordered, and silently skipping them would hide a bug in +the declarations. Everything outside the cycle still runs, so one bad declaration does not cost the whole +run. Tests downstream of the cycle are skipped by the ordinary "prerequisite did not pass" rule, with no +special case. + +**A dependency that matches no test in the run is ignored, with a warning.** This is the single most +consequential trade-off in the design. Skipping the dependent instead would be "safer" in the abstract, +but it would make `--filter`, and running one test from an IDE, useless the moment that test has a +prerequisite — a well-known complaint about `pytest-dependency`. Debuggability wins; the warning keeps a +typo from being silent. A build-time analyzer (see [Future work](#future-work)) is the right place to +catch genuinely misspelled references. + +**References are resolved at discovery.** `[DependsOn(typeof(X))]` is stored as `X.FullName`, because the +graph is rebuilt at execution time — possibly in another app domain — where the `Type` is gone. The +attribute is deliberately shaped so that `nameof` and `typeof` carry the reference, which makes a rename +a compile error rather than a silent no-op. This is the main thing TestNG's string-only +`dependsOnMethods` gets wrong. + +### Scheduling: projecting a test graph onto chunks + +MSTest does not schedule tests; it schedules **chunks** — a whole class under `ExecutionScope.ClassLevel` +(the default), a single test under `MethodLevel`. Dependencies are declared between *tests*. The test +graph is therefore projected onto chunks: + +- an edge **inside** a chunk orders the tests within that chunk (topological sort, ties broken by + declaration order so runs are reproducible); +- an edge **between** chunks gates when the dependent chunk may start. + +A chunk becomes available the moment the last chunk it waits for **completes** — completion, not success, +because whether an individual test actually runs is decided per test, right before it starts, so that an +outcome recorded moments ago on another worker is always taken into account. + +```mermaid +graph LR + Root --> BranchA + Root --> BranchB + BranchA --> Join + BranchB --> Join +``` + +`BranchA` and `BranchB` are released together and run on different workers; `Join` waits for both. + +#### Cycles that exist only in the projection + +Under `ClassLevel`, class A's test can depend on class B's while another of B's depends on one of A's. +No test depends on itself — the test graph is sound — but the *class* graph has a cycle and cannot be +scheduled at class granularity. Dropping the ordering would be worse than losing the parallelism: the +run-time gate cannot distinguish "has not run yet" from "did not pass", so unordered dependents would be +**skipped nondeterministically** — varying with worker count and thread timing — while the run still +reported success. Instead, the tests of the classes caught in the cycle are moved into the **sequential +phase**, where the topological order is honoured exactly. Only those tests lose their parallelism; +unrelated classes keep theirs, and a warning names the classes and points at `MethodLevel` to get the +parallelism back. + +This is reported as a *warning*, not an error: the declared order is still satisfied, so it is a +recoverable downgrade rather than the unschedulable configuration a real cycle represents. + +#### `[DoNotParallelize]` and the demotion rule + +Non-parallelizable tests run in a sequential phase *after* the parallel phase. A parallel test waiting on +a sequential prerequisite could therefore never observe it complete. The fix is to move the **dependent**, +transitively, into the sequential phase — not to move the prerequisite, which would break +`[DoNotParallelize]`'s own guarantee that such tests never run alongside anything else. + +### Dependency declarations in `testconfig.json` + +Some orchestration cannot, or should not, live in the test source: tests owned by another team, a chain +that reviewers want to read in one place, or an order maintained by someone who does not edit code. This +is what `testng.xml` and `playwright.config.ts` are for. MSTest already has a configuration file, so the +declarations go in it rather than in a format of their own: + +```json +{ + "mstest": { + "execution": { + "dependencies": { + "chains": [ + [ + "Contoso.Tests.SetupTests.CreateDatabase", + "Contoso.Tests.ImportTests.ImportCatalog", + "Contoso.Tests.CheckoutTests.PlaceOrder" + ] + ], + "nodes": [ + { + "test": "Contoso.Tests.ReportTests.WriteAudit", + "dependsOn": [ + "Contoso.Tests.CheckoutTests.PlaceOrder", + "Contoso.Tests.ImportTests.*" + ], + "proceedOnFailure": true + } + ] + } + } + } +} +``` + +`chains` is the flat case — each entry waits for the one before it — and `nodes` is the tree case, where +one test names several prerequisites (fan-in) and several nodes may name the same one (fan-out). A test is +referenced by `Namespace.Class.Method`, or `Namespace.Class.*` for every test of a class. + +**This is Microsoft.Testing.Platform only, by design.** `testconfig.json` is supplied only on the MTP path +— the VSTest entry points pass a null configuration (`MSTestDiscoverer.cs`: `configuration: null`) — so +configured dependencies are an MTP capability. A separate file format reachable from RunSettings was +considered and rejected: it would have meant a second syntax, a second parser and a second set of +diagnostics for the same concept, to serve a host that is being superseded. Tests running under VSTest +declare dependencies with the attribute, which works everywhere. + +Configured edges are **merged** with attribute edges; neither overrides the other, and after parsing the +two are indistinguishable to the graph. + +## Implementation + +| Concern | Location | +|---|---| +| Attribute | `src/TestFramework/TestFramework/Attributes/Lifecycle/DependsOnAttribute.cs` | +| Carried edge | `src/Adapter/MSTestAdapter.PlatformServices/ObjectModel/TestDependencyInfo.cs` | +| Read at discovery | `TypeEnumerator.GetTestFromMethod` → `UnitTestElement.Dependencies` | +| VSTest transport | `AdapterTestProperties.DependenciesProperty`, `UnitTestElementExtensions` / `TestCaseExtensions` | +| Graph, cycles, chunking | `Execution/TestDependencyGraph.cs` | +| Run-time gate | `Execution/TestDependencyCoordinator.cs` | +| Configured edges | `Execution/TestDependencyDeclaration.cs`, `MSTestSettings.Configuration.cs` | +| Scheduler | `TestExecutionManager.Parallelization.cs` (`ExecuteTestsWithDependencyGraphAsync`, `ExecuteChunksInTopologicalOrderAsync`) | + +`TestDependencyGraph.Build` returns `null` when no test in the source declares a dependency, and the +executor then takes exactly the path it takes today. This **fast path is deliberate**: the feature must +cost nothing for the overwhelming majority of runs that do not use it, and it keeps the existing +scheduling code reachable and unchanged rather than rewritten. + +The ready-queue is a semaphore over a `ConcurrentQueue` of chunk indices: one permit per queued chunk, +plus one per worker once the last chunk completes, so every worker wakes and exits instead of blocking on +a queue that will never be fed again. Because the chunks caught in a projected cycle are demoted to the +sequential phase before scheduling, the chunk graph handed to the loop is always a DAG, so some chunk +always has no unmet prerequisite and the loop cannot deadlock. The bookkeeping that releases a chunk's +dependents runs in a `finally`, so a chunk that *throws* also cannot strand the workers waiting on it. + +### Class and assembly cleanup + +`ClassCleanupManager` counts down the tests of each class, and end-of-assembly cleanup is gated on +*every* class having reached zero (`ShouldRunEndOfAssemblyCleanup => _remainingTestCountsByClass.IsEmpty`). +The countdown is built from the tests that were **selected**, so a test whose outcome this feature decides +without running it — skipped because a prerequisite did not pass, or failed because it is in a cycle — +still owes its decrement. Left unaccounted, the class never completes: its `[ClassCleanup]` is silently +lost, and with it the whole assembly's `[AssemblyCleanup]`. + +Both paths therefore call `UnitTestRunner.NotifyTestNotRunAsync`, which performs exactly the bookkeeping +a test dropped by an `ITestFilter` already does — the same situation of selected, counted, never run — so +the two features share one implementation rather than each growing their own. + +### Testing + +- `test/UnitTests/MSTestAdapter.PlatformServices.UnitTests/Execution/TestDependencyGraphTests.cs` — 24 + tests over resolution, fan-out independence, cycles (real and projection-only), demotion, the + unmatched-reference warning, `ProceedOnFailure` merging, ordering determinism, and encode/decode. +- `test/IntegrationTests/MSTest.Acceptance.IntegrationTests/TestDependencyExecutionTests.cs` — end-to-end + over net462/net8.0/net10.0. The assets record the millisecond each test body entered and left, and those + timestamps assert the *ordering* guarantee only. Overlap of independent branches is proved structurally + instead, by a bounded `CountdownEvent` that both branches must reach before either is released - a run + that serialized them never satisfies it and fails on the timeout. Inferring overlap from wall-clock + timings would be flaky on a loaded machine, so it is deliberately not done. Also covers skip propagation + with `ProceedOnFailure`, cycle reporting, and the `testconfig.json` declarations (ordering via `chains`, + plus fan-out from `nodes` where one dependent is skipped and a `proceedOnFailure` sibling still runs). + +## Guidance + +Test dependencies couple tests together and make a dependent impossible to run in isolation, which is why +they are a poor fit for unit tests, and why JUnit 5 rejects them outright. The +[2014 ISSTA study on test independence](https://homes.cs.washington.edu/~mernst/pubs/test-independence-issta2014.pdf) +found order-dependent tests in a quarter of the suites it examined, nearly all unintentionally. + +Use `[DependsOn]` for multi-step integration and end-to-end suites where re-establishing expensive state +in every test is impractical, and prefer, in order: + +1. **No dependency** — set state up in `[TestInitialize]` / `[ClassInitialize]`. +2. **A fixture** — if the state is expensive but shareable, `[AssemblyInitialize]` is cheaper than an edge. +3. **`[DependsOn]`** — when the *sequence itself* is the thing under test. + +`RandomizeTestOrder` remains the tool for finding dependencies you did not declare. + +## Future work + +- **A `MSTEST0xxx` analyzer** for references that resolve to no method, self-references, and cycles that + are statically visible. This is the natural complement to the "ignore unmatched references" decision + and would make renames safe at build time. Tracked by + [#10259](https://github.com/microsoft/testfx/issues/10259), which also covers the missing + `WellKnownTypeNames` registration, usage telemetry, and NativeAOT source-generation support. +- **Per-data-row dependencies.** Naming a data-driven test currently creates an edge to *all* of its + cases: the dependent waits for every row and is skipped if any row fails. Matching row *i* of B to row + *i* of A is the hardest open problem in this space — even TUnit's mature implementation requires manual + workarounds ([TUnit#1570](https://github.com/thomhurst/TUnit/issues/1570)) — so V1 documents the + limitation rather than half-solving it. +- **Category dependencies**, the analogue of TestNG's `dependsOnGroups`, mapping onto `[TestCategory]`. + The configuration's `Class.*` wildcard already covers the common case. +- **Cross-assembly dependencies.** The graph is per source, matching the scope of in-assembly + parallelization. + +## Unresolved questions + +- Should a projection-only cycle under `ClassLevel` fall back to scheduling *only the affected classes* at + method granularity, instead of demoting them to the sequential phase? That would keep some parallelism, + at the cost of a scope that varies per class within one run. +- Should `RandomizeTestOrder` warn when it is combined with declared dependencies, as it already does for + `OrderTestsByNameInClass`? The graph wins today, silently. diff --git a/docs/testconfig.schema.json b/docs/testconfig.schema.json index d912b180a9..ee9a240850 100644 --- a/docs/testconfig.schema.json +++ b/docs/testconfig.schema.json @@ -99,6 +99,48 @@ "type": "integer", "description": "Seed used when 'randomizeTestOrder' is true, to make the randomized order reproducible." }, + "dependencies": { + "type": "object", + "description": "Declares test dependencies outside the test source, as an alternative to the [DependsOn] attribute. A test is referenced by 'Namespace.Class.Method', or 'Namespace.Class.*' for every test of a class. Merged with attribute-declared dependencies.", + "additionalProperties": false, + "properties": { + "chains": { + "type": "array", + "description": "Ordered sequences: within each chain, every test waits for the one before it.", + "items": { + "type": "array", + "description": "A chain needs at least two tests; a shorter one declares no dependency.", + "minItems": 2, + "items": { "type": "string" } + } + }, + "nodes": { + "type": "array", + "description": "Explicit dependency nodes, allowing one test to wait for several prerequisites (fan-in) and several tests to wait for the same one (fan-out).", + "items": { + "type": "object", + "additionalProperties": false, + "required": [ "test", "dependsOn" ], + "properties": { + "test": { + "type": "string", + "description": "The test that runs after its prerequisites." + }, + "dependsOn": { + "type": "array", + "description": "The tests that must run first. At least one is required; an empty list declares no dependency.", + "minItems": 1, + "items": { "type": "string" } + }, + "proceedOnFailure": { + "type": "boolean", + "description": "When true, the test runs even if a prerequisite did not pass. Ordering is still enforced. Defaults to false, which skips the test." + } + } + } + } + } + }, "mapInconclusiveToFailed": { "type": "boolean", "description": "When true, inconclusive test outcomes are reported as failures. Defaults to false." diff --git a/src/Adapter/MSTest.TestAdapter/AdapterTestProperties.cs b/src/Adapter/MSTest.TestAdapter/AdapterTestProperties.cs index 7be46022aa..24479e1fe4 100644 --- a/src/Adapter/MSTest.TestAdapter/AdapterTestProperties.cs +++ b/src/Adapter/MSTest.TestAdapter/AdapterTestProperties.cs @@ -33,6 +33,10 @@ internal static class AdapterTestProperties // Each entry is a single-character mode prefix ('R' for Read, 'W' for ReadWrite) followed by the resource key. internal static readonly TestProperty ResourceLocksProperty = TestProperty.Register("MSTestDiscoverer.ResourceLocks", ResourceLocksLabel, typeof(string[]), TestPropertyAttributes.Hidden, typeof(TestCase)); + // Each entry is a single-character 'proceed on failure' prefix ('P' when set, 'S' otherwise) followed by + // the target class full name, a line feed, and the target method name (either may be empty). + internal static readonly TestProperty DependenciesProperty = TestProperty.Register("MSTestDiscoverer.Dependencies", DependenciesLabel, typeof(string[]), TestPropertyAttributes.Hidden, typeof(TestCase)); + internal static readonly TestProperty ExecutionIdProperty = TestProperty.Register("ExecutionId", ExecutionIdLabel, typeof(Guid), TestPropertyAttributes.Hidden, typeof(TestResult)); internal static readonly TestProperty ParentExecIdProperty = TestProperty.Register("ParentExecId", ParentExecIdLabel, typeof(Guid), TestPropertyAttributes.Hidden, typeof(TestResult)); @@ -96,6 +100,8 @@ internal static class AdapterTestProperties #endif private const string DoNotParallelizeLabel = "DoNotParallelize"; private const string ResourceLocksLabel = "ResourceLocks"; + + private const string DependenciesLabel = "Dependencies"; private const string ExecutionIdLabel = "ExecutionId"; private const string ParentExecIdLabel = "ParentExecId"; private const string WorkItemIdsLabel = "WorkItemIds"; diff --git a/src/Adapter/MSTest.TestAdapter/Extensions/TestCaseExtensions.cs b/src/Adapter/MSTest.TestAdapter/Extensions/TestCaseExtensions.cs index 282c8688d6..0b7af8b855 100644 --- a/src/Adapter/MSTest.TestAdapter/Extensions/TestCaseExtensions.cs +++ b/src/Adapter/MSTest.TestAdapter/Extensions/TestCaseExtensions.cs @@ -142,6 +142,12 @@ internal static UnitTestElement ToUnitTestElementWithUpdatedSource(this TestCase testElement.ResourceLocks = Array.ConvertAll(encodedResourceLocks, ResourceLockInfo.Decode); } + string[]? encodedDependencies = testCase.GetPropertyValue(AdapterTestProperties.DependenciesProperty, null); + if (encodedDependencies is { Length: > 0 }) + { + testElement.Dependencies = Array.ConvertAll(encodedDependencies, TestDependencyInfo.Decode); + } + return testElement; } diff --git a/src/Adapter/MSTest.TestAdapter/Extensions/UnitTestElementExtensions.cs b/src/Adapter/MSTest.TestAdapter/Extensions/UnitTestElementExtensions.cs index 5a071c8a53..80fc184dc2 100644 --- a/src/Adapter/MSTest.TestAdapter/Extensions/UnitTestElementExtensions.cs +++ b/src/Adapter/MSTest.TestAdapter/Extensions/UnitTestElementExtensions.cs @@ -122,6 +122,12 @@ internal static TestCase ToTestCase(this UnitTestElement element) testCase.SetPropertyValue(AdapterTestProperties.ResourceLocksProperty, Array.ConvertAll(resourceLocks, ResourceLockInfo.Encode)); } + // Set the declared dependencies if present, so they survive the discovery -> execution round-trip. + if (element.Dependencies is { Length: > 0 } dependencies) + { + testCase.SetPropertyValue(AdapterTestProperties.DependenciesProperty, Array.ConvertAll(dependencies, TestDependencyInfo.Encode)); + } + if (element.UnfoldingStrategy != TestDataSourceUnfoldingStrategy.Auto) { testCase.SetPropertyValue(AdapterTestProperties.UnfoldingStrategy, (int)element.UnfoldingStrategy); diff --git a/src/Adapter/MSTest.TestAdapter/InternalAPI/InternalAPI.Unshipped.txt b/src/Adapter/MSTest.TestAdapter/InternalAPI/InternalAPI.Unshipped.txt index d5cf586f19..4d171bbb50 100644 --- a/src/Adapter/MSTest.TestAdapter/InternalAPI/InternalAPI.Unshipped.txt +++ b/src/Adapter/MSTest.TestAdapter/InternalAPI/InternalAPI.Unshipped.txt @@ -5,6 +5,8 @@ abstract Microsoft.Testing.Extensions.RunSettingsConfigurationProviderBase.ReadR abstract Microsoft.Testing.Extensions.RunSettingsConfigurationProviderBase.Uid.get -> string! abstract Microsoft.Testing.Extensions.RunSettingsConfigurationProviderBase.Version.get -> string! const Microsoft.Testing.Extensions.RunSettingsCommandLineOptionsProviderBase.RunSettingsOptionName = "settings" -> string! +const Microsoft.Testing.Extensions.TestCaseFilterCommandLineOptionsProviderBase.TestCaseFilterOptionName = "filter" -> string! +const Microsoft.Testing.Extensions.TestRunParametersCommandLineOptionsProviderBase.TestRunParameterOptionName = "test-parameter" -> string! Microsoft.Testing.Extensions.RunSettingsCommandLineOptionsProviderBase Microsoft.Testing.Extensions.RunSettingsCommandLineOptionsProviderBase.RunSettingsCommandLineOptionsProviderBase(Microsoft.Testing.Platform.Extensions.IExtension! extension, Microsoft.Testing.Platform.Helpers.IFileSystem! fileSystem, string! optionDescription, string! fileDoesNotExistErrorFormat, string! fileCannotBeReadErrorFormat) -> void Microsoft.Testing.Extensions.RunSettingsConfigurationProviderBase @@ -25,13 +27,16 @@ Microsoft.Testing.Extensions.RunSettingsEnvironmentVariableProviderBase.UpdateAs Microsoft.Testing.Extensions.RunSettingsEnvironmentVariableProviderBase.ValidateTestHostEnvironmentVariablesAsync(Microsoft.Testing.Platform.Extensions.TestHostControllers.IReadOnlyEnvironmentVariables! environmentVariables) -> System.Threading.Tasks.Task! Microsoft.Testing.Extensions.RunSettingsEnvironmentVariableProviderBase.Version.get -> string! Microsoft.Testing.Extensions.RunSettingsProviderHelper -const Microsoft.Testing.Extensions.TestCaseFilterCommandLineOptionsProviderBase.TestCaseFilterOptionName = "filter" -> string! Microsoft.Testing.Extensions.TestCaseFilterCommandLineOptionsProviderBase Microsoft.Testing.Extensions.TestCaseFilterCommandLineOptionsProviderBase.TestCaseFilterCommandLineOptionsProviderBase(Microsoft.Testing.Platform.Extensions.IExtension! extension, string! optionDescription) -> void -const Microsoft.Testing.Extensions.TestRunParametersCommandLineOptionsProviderBase.TestRunParameterOptionName = "test-parameter" -> string! Microsoft.Testing.Extensions.TestRunParametersCommandLineOptionsProviderBase Microsoft.Testing.Extensions.TestRunParametersCommandLineOptionsProviderBase.TestRunParametersCommandLineOptionsProviderBase(Microsoft.Testing.Platform.Extensions.IExtension! extension, string! optionDescription, string! argumentIsNotParameterErrorFormat) -> void -override sealed Microsoft.Testing.Extensions.TestRunParametersCommandLineOptionsProviderBase.ValidateOptionArgumentsAsync(Microsoft.Testing.Platform.Extensions.CommandLine.CommandLineOption! commandOption, string![]! arguments) -> System.Threading.Tasks.Task! +Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.MSTestEngine +Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.MSTestEngine.Cancel() -> void +Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.MSTestEngine.DiscoverAsync(System.Collections.Generic.IEnumerable! sources, string? settingsXml, Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.Interface.IAdapterMessageLogger! logger, Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.Interface.IUnitTestElementSink! elementSink, Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.Interface.ITestElementFilterProvider? filterProvider, Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.Interface.IConfiguration? configuration, Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.Interface.ITestSourceHandler! testSourceHandler, bool isMTP) -> System.Threading.Tasks.Task! +Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.MSTestEngine.MSTestEngine(System.Threading.CancellationToken cancellationToken, System.Func!, System.Threading.Tasks.Task!>? telemetrySender = null, Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution.TestExecutionManager? testExecutionManager = null) -> void +Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.MSTestEngine.RunFromSourcesAsync(System.Collections.Generic.IEnumerable! sources, string? settingsXml, string? testRunDirectory, Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.Interface.IAdapterMessageLogger! logger, System.Func! recorderFactory, Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.Interface.ITestElementFilterProvider? filterProvider, Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.Interface.IConfiguration? configuration, Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.Interface.ITestSourceHandler! testSourceHandler, bool isMTP) -> System.Threading.Tasks.Task! +Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.MSTestEngine.RunFromTestElementsAsync(System.Func!>! testElementsFactory, System.Collections.Generic.IEnumerable! sourcesForInitialization, string? settingsXml, string? testRunDirectory, Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.Interface.IAdapterMessageLogger! logger, System.Func! recorderFactory, Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.Interface.ITestElementFilterProvider? filterProvider, Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.Interface.IConfiguration? configuration, Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.Interface.ITestSourceHandler! testSourceHandler) -> System.Threading.Tasks.Task! Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.TestingPlatformAdapter.IUnitTestElementFilterExpression Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.TestingPlatformAdapter.IUnitTestElementFilterExpression.MatchTestElement(System.Func! propertyValueProvider) -> bool Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.TestingPlatformAdapter.MtpTestElementFilterProvider @@ -46,16 +51,12 @@ override Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.TestingPlatformA override Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.TestingPlatformAdapter.MSTestRunSettingsConfigurationProvider.Version.get -> string! override sealed Microsoft.Testing.Extensions.RunSettingsCommandLineOptionsProviderBase.ValidateCommandLineOptionsAsync(Microsoft.Testing.Platform.CommandLine.ICommandLineOptions! commandLineOptions) -> System.Threading.Tasks.Task! override sealed Microsoft.Testing.Extensions.RunSettingsCommandLineOptionsProviderBase.ValidateOptionArgumentsAsync(Microsoft.Testing.Platform.Extensions.CommandLine.CommandLineOption! commandOption, string![]! arguments) -> System.Threading.Tasks.Task! +override sealed Microsoft.Testing.Extensions.TestRunParametersCommandLineOptionsProviderBase.ValidateOptionArgumentsAsync(Microsoft.Testing.Platform.Extensions.CommandLine.CommandLineOption! commandOption, string![]! arguments) -> System.Threading.Tasks.Task! static Microsoft.Testing.Extensions.RunSettingsProviderHelper.ApplyEnvironmentVariables(System.Xml.Linq.XDocument! runSettings, Microsoft.Testing.Platform.Extensions.TestHostControllers.IEnvironmentVariables! environmentVariables) -> void static Microsoft.Testing.Extensions.RunSettingsProviderHelper.CanReadFile(Microsoft.Testing.Platform.Helpers.IFileSystem! fileSystem, string! filePath) -> bool static Microsoft.Testing.Extensions.RunSettingsProviderHelper.FindInvalidTestParameter(string![]! arguments) -> string? static Microsoft.Testing.Extensions.RunSettingsProviderHelper.HasEnvironmentVariables(System.Xml.Linq.XDocument! runSettings) -> bool static Microsoft.Testing.Extensions.RunSettingsProviderHelper.TryLoadRunSettingsAsync(Microsoft.Testing.Platform.CommandLine.ICommandLineOptions! commandLineOptions, Microsoft.Testing.Platform.Helpers.IFileSystem! fileSystem, Microsoft.Testing.Platform.Helpers.IEnvironment! environment, string! runSettingsOptionName) -> System.Threading.Tasks.Task! static Microsoft.VisualStudio.TestTools.UnitTesting.MSTestTestNodeConverter.ToEmptyResultTestNode(Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.ObjectModel.UnitTestElement! element, bool isTrxEnabled) -> Microsoft.Testing.Platform.Extensions.Messages.TestNode! -Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.MSTestEngine -Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.MSTestEngine.Cancel() -> void -Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.MSTestEngine.DiscoverAsync(System.Collections.Generic.IEnumerable! sources, string? settingsXml, Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.Interface.IAdapterMessageLogger! logger, Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.Interface.IUnitTestElementSink! elementSink, Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.Interface.ITestElementFilterProvider? filterProvider, Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.Interface.IConfiguration? configuration, Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.Interface.ITestSourceHandler! testSourceHandler, bool isMTP) -> System.Threading.Tasks.Task! -Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.MSTestEngine.MSTestEngine(System.Threading.CancellationToken cancellationToken, System.Func!, System.Threading.Tasks.Task!>? telemetrySender = null, Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution.TestExecutionManager? testExecutionManager = null) -> void -Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.MSTestEngine.RunFromSourcesAsync(System.Collections.Generic.IEnumerable! sources, string? settingsXml, string? testRunDirectory, Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.Interface.IAdapterMessageLogger! logger, System.Func! recorderFactory, Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.Interface.ITestElementFilterProvider? filterProvider, Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.Interface.IConfiguration? configuration, Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.Interface.ITestSourceHandler! testSourceHandler, bool isMTP) -> System.Threading.Tasks.Task! -Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.MSTestEngine.RunFromTestElementsAsync(System.Func!>! testElementsFactory, System.Collections.Generic.IEnumerable! sourcesForInitialization, string? settingsXml, string? testRunDirectory, Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.Interface.IAdapterMessageLogger! logger, System.Func! recorderFactory, Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.Interface.ITestElementFilterProvider? filterProvider, Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.Interface.IConfiguration? configuration, Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.Interface.ITestSourceHandler! testSourceHandler) -> System.Threading.Tasks.Task! +static readonly Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.AdapterTestProperties.DependenciesProperty -> Microsoft.VisualStudio.TestPlatform.ObjectModel.TestProperty! static readonly Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.AdapterTestProperties.ResourceLocksProperty -> Microsoft.VisualStudio.TestPlatform.ObjectModel.TestProperty! diff --git a/src/Adapter/MSTest.TestAdapter/InternalAPI/uwp/InternalAPI.Unshipped.txt b/src/Adapter/MSTest.TestAdapter/InternalAPI/uwp/InternalAPI.Unshipped.txt index b5d6b30cc7..497815dbc8 100644 --- a/src/Adapter/MSTest.TestAdapter/InternalAPI/uwp/InternalAPI.Unshipped.txt +++ b/src/Adapter/MSTest.TestAdapter/InternalAPI/uwp/InternalAPI.Unshipped.txt @@ -5,4 +5,5 @@ Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.MSTestEngine.DiscoverAsyn Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.MSTestEngine.MSTestEngine(System.Threading.CancellationToken cancellationToken, System.Func!, System.Threading.Tasks.Task!>? telemetrySender = null, Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution.TestExecutionManager? testExecutionManager = null) -> void Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.MSTestEngine.RunFromSourcesAsync(System.Collections.Generic.IEnumerable! sources, string? settingsXml, string? testRunDirectory, Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.Interface.IAdapterMessageLogger! logger, System.Func! recorderFactory, Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.Interface.ITestElementFilterProvider? filterProvider, Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.Interface.IConfiguration? configuration, Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.Interface.ITestSourceHandler! testSourceHandler, bool isMTP) -> System.Threading.Tasks.Task! Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.MSTestEngine.RunFromTestElementsAsync(System.Func!>! testElementsFactory, System.Collections.Generic.IEnumerable! sourcesForInitialization, string? settingsXml, string? testRunDirectory, Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.Interface.IAdapterMessageLogger! logger, System.Func! recorderFactory, Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.Interface.ITestElementFilterProvider? filterProvider, Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.Interface.IConfiguration? configuration, Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.Interface.ITestSourceHandler! testSourceHandler) -> System.Threading.Tasks.Task! +static readonly Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.AdapterTestProperties.DependenciesProperty -> Microsoft.VisualStudio.TestPlatform.ObjectModel.TestProperty! static readonly Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.AdapterTestProperties.ResourceLocksProperty -> Microsoft.VisualStudio.TestPlatform.ObjectModel.TestProperty! diff --git a/src/Adapter/MSTestAdapter.PlatformServices/Discovery/TypeEnumerator.cs b/src/Adapter/MSTestAdapter.PlatformServices/Discovery/TypeEnumerator.cs index 1a43a2f8a1..8cab727c67 100644 --- a/src/Adapter/MSTestAdapter.PlatformServices/Discovery/TypeEnumerator.cs +++ b/src/Adapter/MSTestAdapter.PlatformServices/Discovery/TypeEnumerator.cs @@ -22,6 +22,8 @@ internal class TypeEnumerator private readonly ReflectHelper _reflectHelper; private List? _classResourceLocks; private bool _classResourceLocksComputed; + private List? _classDependencies; + private bool _classDependenciesComputed; /// /// Initializes a new instance of the class. @@ -158,6 +160,7 @@ internal UnitTestElement GetTestFromMethod(MethodInfo method, bool classDisables TestCategory = reflectionOperations.GetTestCategories(method, _type), DoNotParallelize = classDisablesParallelization || _reflectHelper.IsAttributeDefined(method), ResourceLocks = MergeResourceLocks(GetClassResourceLocks(), ReadResourceLocks(method)), + Dependencies = MergeDependencies(GetClassDependencies(), ReadDependencies(method), _type.FullName!, method.Name), #if !WINDOWS_UWP && !WIN_UI DeploymentItems = PlatformServiceProvider.Instance.TestDeployment.GetDeploymentItems(method, _type, warnings), #endif @@ -239,6 +242,108 @@ internal UnitTestElement GetTestFromMethod(MethodInfo method, bool classDisables return locks; } + /// + /// Reads (and caches for this type) the [DependsOn] attributes declared on the test class. + /// + private List? GetClassDependencies() + { + if (!_classDependenciesComputed) + { + _classDependencies = ReadDependencies(_type); + _classDependenciesComputed = true; + } + + return _classDependencies; + } + + /// + /// Reads the [DependsOn] attributes declared directly on + /// (a class or a method), in declaration order. Returns when none are present. + /// + private List? ReadDependencies(ICustomAttributeProvider attributeProvider) + { + List? dependencies = null; + foreach (DependsOnAttribute attribute in _reflectHelper.GetAttributes(attributeProvider)) + { + // A type reference is resolved to its CLR full name here, at discovery, because the graph is + // rebuilt at execution time - possibly in another app domain - where the Type is no longer + // available. FullName matches UnitTestElement.TestMethod.FullClassName, which TypeEnumerator + // also derives from Type.FullName. + (dependencies ??= []).Add(new TestDependencyInfo( + attribute.TestClass?.FullName, + attribute.TestMethodName, + attribute.ProceedOnFailure)); + } + + return dependencies; + } + + /// + /// Merges the class-level and method-level dependencies into a single distinct set, preserving + /// declaration order (class first). When the same prerequisite is declared more than once, the + /// ProceedOnFailure flags are merged conservatively - the edge proceeds past a failed + /// prerequisite only when every declaration of it says so - matching the rule applied across distinct + /// prerequisites in TestDependencyGraph.ResolveEdges. Returns when neither + /// declares any dependency. + /// + private static TestDependencyInfo[]? MergeDependencies(List? classDependencies, List? methodDependencies, string declaringClassFullName, string methodName) + { + if (classDependencies is null && methodDependencies is null) + { + return null; + } + + var result = new List(); + var indexByTarget = new Dictionary(StringComparer.Ordinal); + + // A class-level [DependsOn(nameof(Setup))] is expanded onto every method of the class, including + // Setup itself. The user wrote "every *other* test waits for Setup", never "Setup waits for + // itself", so that generated self-edge is dropped here. A self reference written directly on the + // method is kept, because there the user really did name the test they were annotating, and it + // surfaces as a cycle. + AddAll(classDependencies, result, indexByTarget, declaringClassFullName, methodName); + AddAll(methodDependencies, result, indexByTarget, declaringClassFullName, methodName: null); + + return result.Count == 0 ? null : [.. result]; + + static void AddAll(List? source, List target, Dictionary indexByTarget, string declaringClassFullName, string? methodName) + { + if (source is null) + { + return; + } + + foreach (TestDependencyInfo dependency in source) + { + if (methodName is not null + && string.Equals(dependency.TargetMethodName, methodName, StringComparison.Ordinal) + && (dependency.TargetClassFullName is null || string.Equals(dependency.TargetClassFullName, declaringClassFullName, StringComparison.Ordinal))) + { + continue; + } + + string key = dependency.DescribeTarget(); + if (indexByTarget.TryGetValue(key, out int existingIndex)) + { + // Conservative merge: one declaration asking for the ordinary skip is enough to hold the + // dependent back, which is the same rule ResolveEdges applies across distinct + // prerequisites. Merging the other way would let a class-level ProceedOnFailure silently + // override a method-level default and run a test whose precondition demonstrably did not + // hold; over-skipping only costs coverage that was already compromised. + if (!dependency.ProceedOnFailure && target[existingIndex].ProceedOnFailure) + { + target[existingIndex] = dependency; + } + } + else + { + indexByTarget[key] = target.Count; + target.Add(dependency); + } + } + } + } + /// /// Merges the class-level and method-level resource locks into a single distinct set (ordinal, case-sensitive), /// keeping the strongest mode per key ( wins over diff --git a/src/Adapter/MSTestAdapter.PlatformServices/Execution/TestDependencyCoordinator.cs b/src/Adapter/MSTestAdapter.PlatformServices/Execution/TestDependencyCoordinator.cs new file mode 100644 index 0000000000..8295a5da1d --- /dev/null +++ b/src/Adapter/MSTestAdapter.PlatformServices/Execution/TestDependencyCoordinator.cs @@ -0,0 +1,131 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT license. See LICENSE file in the project root for full license information. + +using Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.ObjectModel; + +namespace Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution; + +/// +/// Tracks, for one source, the outcome of every test that takes part in the dependency graph, and answers +/// the only question the executor asks of it: whether a test may run, or must be skipped because a +/// prerequisite did not pass. +/// +/// +/// Instances are shared by the parallel workers, so every access to the outcome state is synchronized. The +/// state is tiny (two bits per test) and touched twice per test, so a single lock is cheaper and easier to +/// reason about than finer-grained synchronization. +/// +internal sealed class TestDependencyCoordinator +{ + private readonly TestDependencyGraph _graph; + private readonly Dictionary _indexByTest; + private readonly bool[] _completed; + private readonly bool[] _passed; +#if NET9_0_OR_GREATER + private readonly Lock _lock = new(); +#else + private readonly object _lock = new(); +#endif + + public TestDependencyCoordinator(TestDependencyGraph graph) + { + _graph = graph; + _completed = new bool[graph.Tests.Length]; + _passed = new bool[graph.Tests.Length]; + + // The scheduler hands back the very instances the graph was built from, so reference identity is the + // correct - and cheapest - way to find a test's node. Value equality is not an option: two data rows + // of the same test method are distinct nodes but compare equal on their identifying fields. + _indexByTest = new Dictionary(graph.Tests.Length, ReferenceComparer.Instance); + for (int i = 0; i < graph.Tests.Length; i++) + { + _indexByTest[graph.Tests[i]] = i; + } + } + + /// + /// Determines whether must be skipped because one of its prerequisites did not + /// pass, and if so produces the message to report. A test that declares no dependency, or that opted + /// into ProceedOnFailure, is never skipped by this method. + /// + /// + /// A prerequisite that has not completed counts as not passed. That happens for a test dropped from the + /// run because it takes part in a cycle, and it is the conservative reading: the dependent's precondition + /// demonstrably did not hold. + /// + public bool ShouldSkip(UnitTestElement test, [NotNullWhen(true)] out string? reason) + { + reason = null; + if (!_indexByTest.TryGetValue(test, out int index)) + { + return false; + } + + int[] prerequisites = _graph.TestPrerequisites[index]; + if (prerequisites.Length == 0 || _graph.ProceedOnFailure[index]) + { + return false; + } + + lock (_lock) + { + foreach (int prerequisite in prerequisites) + { + if (!_completed[prerequisite] || !_passed[prerequisite]) + { + reason = string.Format( + CultureInfo.CurrentCulture, + Resource.DependsOnPrerequisiteNotPassed, + _graph.Tests[prerequisite].TestMethod.FullyQualifiedName); + return true; + } + } + } + + return false; + } + + /// + /// Records the outcome of a test that the scheduler ran, so that its dependents can be gated on it. + /// + public void RecordOutcome(UnitTestElement test, bool passed) + { + if (!_indexByTest.TryGetValue(test, out int index)) + { + return; + } + + lock (_lock) + { + _completed[index] = true; + _passed[index] = passed; + } + } + + /// + /// Records that a test will never run - because it takes part in a cycle, or because it was skipped - + /// so that everything downstream of it is skipped in turn. + /// + public void RecordNotRun(UnitTestElement test) + { + if (!_indexByTest.TryGetValue(test, out int index)) + { + return; + } + + lock (_lock) + { + _completed[index] = true; + _passed[index] = false; + } + } + + private sealed class ReferenceComparer : IEqualityComparer + { + public static readonly ReferenceComparer Instance = new(); + + public bool Equals(UnitTestElement? x, UnitTestElement? y) => ReferenceEquals(x, y); + + public int GetHashCode(UnitTestElement obj) => RuntimeHelpers.GetHashCode(obj); + } +} diff --git a/src/Adapter/MSTestAdapter.PlatformServices/Execution/TestDependencyDeclaration.cs b/src/Adapter/MSTestAdapter.PlatformServices/Execution/TestDependencyDeclaration.cs new file mode 100644 index 0000000000..291fbcef9e --- /dev/null +++ b/src/Adapter/MSTestAdapter.PlatformServices/Execution/TestDependencyDeclaration.cs @@ -0,0 +1,185 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT license. See LICENSE file in the project root for full license information. + +using Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.ObjectModel; +using Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.Interface; + +using MessageLevel = Microsoft.VisualStudio.TestTools.UnitTesting.MessageLevel; + +namespace Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution; + +/// +/// A dependency edge declared in testconfig.json rather than by a [DependsOn] attribute: +/// runs after . +/// +/// +/// +/// Declaring dependencies in configuration exists for the cases the attribute cannot serve - orchestrating +/// tests somebody else owns, keeping a whole pipeline visible in one reviewable place, or letting a role +/// that does not edit code define the run. It is the same idea as TestNG's testng.xml or +/// Playwright's project dependencies, expressed in the configuration file MSTest already has. +/// +/// +/// A test is referenced by Namespace.Class.Method, or by Namespace.Class.* to mean every test +/// of a class. Configured edges are merged with attribute-declared ones; neither overrides the other. +/// +/// +#if NETFRAMEWORK +[Serializable] +#endif +internal sealed class TestDependencyDeclaration +{ + public TestDependencyDeclaration(string dependent, string prerequisite, bool proceedOnFailure) + { + Dependent = dependent; + Prerequisite = prerequisite; + ProceedOnFailure = proceedOnFailure; + } + + /// Gets the reference to the test that runs second. + public string Dependent { get; } + + /// Gets the reference to the test that must run first. + public string Prerequisite { get; } + + /// Gets a value indicating whether the dependent runs even when the prerequisite does not pass. + public bool ProceedOnFailure { get; } + + /// + /// Reports the declarations whose diagnostics can only be judged against the whole run: a malformed + /// reference, and a dependent that matches no test anywhere. Kept separate from , + /// which runs once per source and therefore cannot tell "this names a test in another assembly" from + /// "this names nothing at all". + /// + public static void ReportUnmatchedDeclarations(IEnumerable declarations, IEnumerable allTests, IAdapterMessageLogger? logger) + { + if (logger is null) + { + return; + } + + UnitTestElement[] tests = [.. allTests]; + foreach (TestDependencyDeclaration declaration in declarations) + { + if (!TestReference.TryParse(declaration.Dependent, out TestReference? dependent)) + { + logger.SendMessage( + MessageLevel.Warning, + string.Format(CultureInfo.CurrentCulture, Resource.DependencyConfigurationInvalidReference, declaration.Dependent)); + continue; + } + + if (!TestReference.TryParse(declaration.Prerequisite, out TestReference? _)) + { + logger.SendMessage( + MessageLevel.Warning, + string.Format(CultureInfo.CurrentCulture, Resource.DependencyConfigurationInvalidReference, declaration.Prerequisite)); + continue; + } + + if (!tests.Any(dependent.Matches)) + { + logger.SendMessage( + MessageLevel.Warning, + string.Format(CultureInfo.CurrentCulture, Resource.DependencyConfigurationDependentNotFound, declaration.Dependent)); + } + } + } + + /// + /// Applies to the tests they name, so that from here on a configured + /// edge is indistinguishable from one declared by an attribute. + /// + /// when at least one edge was applied. + public static bool ApplyAll(IEnumerable declarations, UnitTestElement[] tests, IAdapterMessageLogger? adapterMessageLogger) + { + bool applied = false; + foreach (TestDependencyDeclaration declaration in declarations) + { + if (!TestReference.TryParse(declaration.Dependent, out TestReference? dependent)) + { + continue; + } + + if (!TestReference.TryParse(declaration.Prerequisite, out TestReference? prerequisite)) + { + continue; + } + + foreach (UnitTestElement test in tests) + { + if (!dependent.Matches(test)) + { + continue; + } + + // A wildcard dependent (Ns.Class.*) expands onto every test of the class, including the + // prerequisite itself when that is also named in the class. The user wrote "every test of + // this class waits for Setup", never "Setup waits for itself", so that generated self-edge is + // dropped - exactly as discovery drops it for a class-level [DependsOn]. Without this, Setup + // would be reported as a cycle and the whole class skipped. + // + // Only a *specific* prerequisite is suppressed this way. When the prerequisite is itself a + // whole class, dropping the edge here would silently discard the entire declaration; those + // edges go to the graph, which removes just each test's own self-edge and reports whatever + // genuine cycle remains. + if (dependent.MethodName is null && prerequisite.MethodName is not null && prerequisite.Matches(test)) + { + continue; + } + + // The prerequisite is stored as declared rather than resolved to concrete tests here: the + // graph already knows how to expand a class-wide target and how to report one that matches + // nothing, and doing it in one place keeps both sources of edges behaving identically. + var dependency = new TestDependencyInfo(prerequisite.ClassName, prerequisite.MethodName, declaration.ProceedOnFailure); + test.Dependencies = test.Dependencies is { Length: > 0 } existing + ? [.. existing, dependency] + : [dependency]; + applied = true; + } + } + + return applied; + } + + /// + /// A reference to a test, or to every test of a class, written as Namespace.Class.Method or + /// Namespace.Class.*. + /// + private sealed class TestReference + { + private TestReference(string className, string? methodName) + { + ClassName = className; + MethodName = methodName; + } + + public string ClassName { get; } + + /// Gets the method name, or when the reference covers a whole class. + public string? MethodName { get; } + + public static bool TryParse(string value, [NotNullWhen(true)] out TestReference? reference) + { + string trimmed = value.Trim(); + int lastDot = trimmed.LastIndexOf('.'); + + // Both parts are required: a bare identifier could be a class or a method, and guessing would + // silently point the edge at the wrong thing. + if (lastDot <= 0 || lastDot == trimmed.Length - 1) + { + reference = null; + return false; + } + + string className = trimmed.Substring(0, lastDot); + string methodName = trimmed.Substring(lastDot + 1); + reference = methodName == "*" ? new TestReference(className, null) : new TestReference(className, methodName); + return true; + } + + public bool Matches(UnitTestElement element) + => string.Equals(element.TestMethod.FullClassName, ClassName, StringComparison.Ordinal) + && (MethodName is null || string.Equals(element.TestMethod.Name, MethodName, StringComparison.Ordinal)); + } +} diff --git a/src/Adapter/MSTestAdapter.PlatformServices/Execution/TestDependencyGraph.cs b/src/Adapter/MSTestAdapter.PlatformServices/Execution/TestDependencyGraph.cs new file mode 100644 index 0000000000..5d99270fdd --- /dev/null +++ b/src/Adapter/MSTestAdapter.PlatformServices/Execution/TestDependencyGraph.cs @@ -0,0 +1,967 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT license. See LICENSE file in the project root for full license information. + +using Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.ObjectModel; + +using ExecutionScope = Microsoft.VisualStudio.TestTools.UnitTesting.ExecutionScope; + +namespace Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution; + +/// +/// The resolved [DependsOn] dependency graph for one test source: which test must run before which, +/// how the tests are grouped into scheduling chunks, and in what order a chunk's tests run. +/// +/// +/// +/// The graph is built once per source, before anything runs, from the dependencies carried on the +/// s (see ), which come from +/// [DependsOn] attributes and/or testconfig.json. It is a pure data structure: it neither +/// runs tests nor observes outcomes - does that at run time. +/// +/// +/// Edges are resolved between tests, but MSTest schedules chunks (a whole class under +/// , a single test under ). +/// The test-level graph is therefore projected onto chunks: an edge inside a chunk only orders the tests +/// within it, while an edge between chunks gates when the dependent chunk may start. +/// +/// +internal sealed class TestDependencyGraph +{ + private TestDependencyGraph( + UnitTestElement[] tests, + int[][] testPrerequisites, + bool[] proceedOnFailure, + UnitTestElement[][] parallelChunks, + int[][] parallelChunkPrerequisites, + UnitTestElement[] sequentialTests, + IReadOnlyList brokenTests, + IReadOnlyList errors, + IReadOnlyList warnings) + { + Tests = tests; + TestPrerequisites = testPrerequisites; + ProceedOnFailure = proceedOnFailure; + ParallelChunks = parallelChunks; + ParallelChunkPrerequisites = parallelChunkPrerequisites; + SequentialTests = sequentialTests; + BrokenTests = brokenTests; + Errors = errors; + Warnings = warnings; + } + + /// Gets every test of the source, in the order the graph indexes them. + public UnitTestElement[] Tests { get; } + + /// + /// Gets, for each test, the indices into of the tests that must run first. + /// + public int[][] TestPrerequisites { get; } + + /// + /// Gets, for each test, whether it runs even when a prerequisite did not pass. A test with several + /// prerequisites has a single value because the flag is merged per dependent when the edges are + /// resolved: it is set only when every declared edge asked to proceed. + /// + public bool[] ProceedOnFailure { get; } + + /// + /// Gets the scheduling chunks of the parallel phase. The tests inside a chunk are already ordered so + /// that intra-chunk prerequisites come first. + /// + public UnitTestElement[][] ParallelChunks { get; } + + /// + /// Gets, for each parallel chunk, the indices of the chunks that must complete before it may start. + /// + public int[][] ParallelChunkPrerequisites { get; } + + /// + /// Gets the tests of the sequential phase (those marked [DoNotParallelize], plus those pulled in + /// by the demotion rule), ordered so that prerequisites come first. + /// + public UnitTestElement[] SequentialTests { get; } + + /// + /// Gets the tests that take part in a dependency cycle, each paired with the description of the cycle + /// it is in. They cannot be ordered, so they are reported as failed instead of being run; their + /// dependents are then skipped by the ordinary "a prerequisite did not pass" rule. + /// + public IReadOnlyList BrokenTests { get; } + + /// + /// A test that cannot be scheduled because it takes part in a dependency cycle, together with the + /// description of that cycle. The message is per test rather than per run so that a suite containing two + /// unrelated cycles reports each failure against its own cycle instead of every cycle in the assembly. + /// + internal sealed class BrokenTest + { + public BrokenTest(UnitTestElement element, string cycleMessage) + { + Element = element; + CycleMessage = cycleMessage; + } + + public UnitTestElement Element { get; } + + public string CycleMessage { get; } + } + + /// + /// Gets the fatal configuration errors: the dependency cycles that make tests unschedulable. These are + /// reported as failures against the tests in the cycle, so nothing that merely degrades - such as a cycle + /// in the class-level projection, which is recovered from by running those tests sequentially - belongs + /// here; that goes to . + /// + public IReadOnlyList Errors { get; } + + /// + /// Gets the non-fatal problems to report: dependencies that match no test in this run, and cycles in the + /// class-level projection that were recovered from by demoting the affected tests to the sequential phase. + /// + public IReadOnlyList Warnings { get; } + + /// + /// Builds the graph for , or returns when no test declares + /// a dependency, in which case the caller keeps its ordinary scheduling path. The null fast path matters: + /// it keeps every run that does not use the feature on exactly the code it uses today. + /// + /// The tests of one source, after filtering. + /// The effective parallelization scope, which decides what a chunk is. + /// + /// Whether the parallel phase exists at all. When it does not, every test is placed in the sequential + /// phase, so a declared order is still honored when parallelization is turned off. + /// + public static TestDependencyGraph? Build(UnitTestElement[] tests, ExecutionScope scope, bool parallelizationEnabled) + { + bool anyDependency = false; + foreach (UnitTestElement test in tests) + { + if (test.Dependencies is { Length: > 0 }) + { + anyDependency = true; + break; + } + } + + if (!anyDependency) + { + return null; + } + + var warnings = new List(); + var errors = new List(); + + (int[][] testPrerequisites, bool[] proceedOnFailure) = ResolveEdges(tests, warnings); + + string?[] cycleMessageByTest = FindCycles(testPrerequisites, tests, errors); + bool[] isBroken = new bool[tests.Length]; + for (int i = 0; i < tests.Length; i++) + { + isBroken[i] = cycleMessageByTest[i] is not null; + } + + // A test in a cycle cannot be ordered, so it is dropped from scheduling and reported as failed, with + // the description of the cycle *it* is in. Its dependents keep the edge: at run time the prerequisite + // is recorded as "did not pass", so they are skipped with the ordinary message rather than silently + // running out of order. + var brokenTests = new List(); + for (int i = 0; i < tests.Length; i++) + { + if (cycleMessageByTest[i] is { } cycleMessage) + { + brokenTests.Add(new BrokenTest(tests[i], cycleMessage)); + } + } + + bool[] isSequential = ComputeSequentialSet(tests, testPrerequisites, isBroken, parallelizationEnabled); + + (UnitTestElement[][] parallelChunks, int[][] parallelChunkPrerequisites) = BuildChunks( + tests, + testPrerequisites, + SelectIndices(tests.Length, i => !isSequential[i] && !isBroken[i]), + scope, + warnings, + out List? projectionCycleTests); + + // A cycle that exists only in the class-level projection means the declared order is satisfiable + // between tests but not between whole classes. Rather than leave those tests unordered - which would + // make the run-time gate skip them nondeterministically, because a prerequisite that has not run yet + // is indistinguishable from one that failed - the affected tests are moved into the sequential phase, + // where the topological order can be honoured exactly. The cost is parallelism for those tests only, + // and the reported warning tells the user how to get it back. + if (projectionCycleTests is not null) + { + foreach (int index in projectionCycleTests) + { + isSequential[index] = true; + } + + // Anything that waited on a demoted test has to move too, for the same reason [DoNotParallelize] + // dependents do: the sequential phase runs after the parallel one. + PropagateSequential(tests.Length, testPrerequisites, isBroken, isSequential); + + (parallelChunks, parallelChunkPrerequisites) = BuildChunks( + tests, + testPrerequisites, + SelectIndices(tests.Length, i => !isSequential[i] && !isBroken[i]), + scope, + warnings, + out _); + } + + UnitTestElement[] sequentialTests = OrderTopologically( + SelectIndices(tests.Length, i => isSequential[i] && !isBroken[i]), + testPrerequisites, + tests); + + return new TestDependencyGraph( + tests, + testPrerequisites, + proceedOnFailure, + parallelChunks, + parallelChunkPrerequisites, + sequentialTests, + brokenTests, + errors, + warnings); + } + + /// + /// Resolves every declared dependency into edges between test indices. A dependency that names a class + /// but no method matches every test of that class; one that names no class is looked up in the + /// dependent's own class. A dependency that matches no test in this run is reported as a warning and + /// dropped, so that running a subset of the suite (for example with a filter, or a single test from an + /// IDE) stays possible. + /// + private static (int[][] Prerequisites, bool[] ProceedOnFailure) ResolveEdges(UnitTestElement[] tests, List warnings) + { + var byClass = new Dictionary>(StringComparer.Ordinal); + var byClassAndMethod = new Dictionary>(StringComparer.Ordinal); + for (int i = 0; i < tests.Length; i++) + { + TestMethod testMethod = tests[i].TestMethod; + AddToIndex(byClass, testMethod.FullClassName, i); + AddToIndex(byClassAndMethod, MakeKey(testMethod.FullClassName, testMethod.Name), i); + } + + int[][] prerequisites = new int[tests.Length][]; + bool[] proceedOnFailure = new bool[tests.Length]; + var resolved = new List(); + var seen = new HashSet(); + + for (int i = 0; i < tests.Length; i++) + { + if (tests[i].Dependencies is not { Length: > 0 } dependencies) + { + prerequisites[i] = []; + continue; + } + + resolved.Clear(); + seen.Clear(); + + // A dependent proceeds past a failed prerequisite only when every edge it declares says so: + // one edge that wants the ordinary skip is enough to hold it back. + bool allProceed = true; + + foreach (TestDependencyInfo dependency in dependencies) + { + string targetClass = dependency.TargetClassFullName ?? tests[i].TestMethod.FullClassName; + List? matches; + if (dependency.TargetMethodName is null) + { + byClass.TryGetValue(targetClass, out matches); + } + else + { + byClassAndMethod.TryGetValue(MakeKey(targetClass, dependency.TargetMethodName), out matches); + } + + int added = 0; + if (matches is not null) + { + foreach (int match in matches) + { + // A whole-class dependency naturally includes the dependent itself when it lives in + // that class; that is not a cycle the user wrote, so drop it. An explicit self + // reference by name is kept, so it surfaces as a cycle. + if (match == i && dependency.TargetMethodName is null) + { + continue; + } + + added++; + if (seen.Add(match)) + { + resolved.Add(match); + } + } + } + + if (added == 0) + { + warnings.Add(string.Format( + CultureInfo.CurrentCulture, + Resource.DependsOnTargetNotFound, + tests[i].TestMethod.FullyQualifiedName, + dependency.DescribeTarget())); + continue; + } + + allProceed &= dependency.ProceedOnFailure; + } + + prerequisites[i] = [.. resolved]; + proceedOnFailure[i] = resolved.Count > 0 && allProceed; + } + + return (prerequisites, proceedOnFailure); + + static void AddToIndex(Dictionary> index, string key, int value) + { + if (!index.TryGetValue(key, out List? list)) + { + index[key] = list = []; + } + + list.Add(value); + } + + static string MakeKey(string className, string methodName) => className + "." + methodName; + } + + /// + /// Finds every test that takes part in a dependency cycle, and describes each cycle as a path + /// (A > B > A). Cycle membership is computed with Tarjan's strongly-connected-components + /// algorithm rather than by collecting back-edge paths from a plain depth-first search: a DFS reports the + /// path it happened to close, so with overlapping cycles (A depends on B and C, B on A, C on B) it finds + /// A > B > A, marks B finished, and never notices that C is in a cycle too. Every member of a + /// component with a cycle has to be found, because each one is unschedulable. + /// + private static string?[] FindCycles(int[][] prerequisites, UnitTestElement[] tests, List errors) + { + string?[] cycleMessageByTest = new string?[prerequisites.Length]; + int[] index = new int[prerequisites.Length]; + int[] lowLink = new int[prerequisites.Length]; + bool[] onStack = new bool[prerequisites.Length]; + int[] edgeCursor = new int[prerequisites.Length]; + for (int i = 0; i < index.Length; i++) + { + index[i] = -1; + } + + var stack = new List(); + var callStack = new List(); + int nextIndex = 0; + + for (int start = 0; start < prerequisites.Length; start++) + { + if (index[start] != -1) + { + continue; + } + + // Iterative Tarjan: the recursive form would stack-overflow on a long dependency chain. + callStack.Add(start); + index[start] = lowLink[start] = nextIndex++; + edgeCursor[start] = 0; + stack.Add(start); + onStack[start] = true; + + while (callStack.Count > 0) + { + int current = callStack[callStack.Count - 1]; + if (edgeCursor[current] < prerequisites[current].Length) + { + int next = prerequisites[current][edgeCursor[current]++]; + if (index[next] == -1) + { + index[next] = lowLink[next] = nextIndex++; + edgeCursor[next] = 0; + stack.Add(next); + onStack[next] = true; + callStack.Add(next); + } + else if (onStack[next]) + { + lowLink[current] = Math.Min(lowLink[current], index[next]); + } + + continue; + } + + callStack.RemoveAt(callStack.Count - 1); + if (callStack.Count > 0) + { + int parent = callStack[callStack.Count - 1]; + lowLink[parent] = Math.Min(lowLink[parent], lowLink[current]); + } + + if (lowLink[current] != index[current]) + { + continue; + } + + // 'current' roots a strongly connected component: pop it off the stack. + var component = new List(); + int member; + do + { + member = stack[stack.Count - 1]; + stack.RemoveAt(stack.Count - 1); + onStack[member] = false; + component.Add(member); + } + while (member != current); + + // A component of one is only a cycle when the test names itself; anything larger always is. + bool isCycle = component.Count > 1 || Array.IndexOf(prerequisites[current], current) >= 0; + if (!isCycle) + { + continue; + } + + // A component is a *set*, not a path: rendering its members in any order would claim edges + // that were never declared (with A depending on B and C, B on A and C on B, "A > B > C > A" + // asserts a B -> C edge that does not exist, and tells the user to remove it). Walk real + // edges inside the component instead, so every arrow in the message is one the user wrote. + int[] cycleNodes = FindCyclePathWithin(prerequisites, component, current); + var cycle = new List(cycleNodes.Length + 1); + foreach (int node in cycleNodes) + { + cycle.Add(tests[node].TestMethod.FullyQualifiedName); + } + + cycle.Add(tests[cycleNodes[0]].TestMethod.FullyQualifiedName); + + string message = string.Format(CultureInfo.CurrentCulture, Resource.DependsOnCycle, string.Join(" > ", cycle)); + errors.Add(message); + foreach (int node in component) + { + cycleMessageByTest[node] = message; + } + } + } + + return cycleMessageByTest; + } + + /// + /// Returns a genuine closed walk through starting at , + /// following only declared prerequisite edges that stay inside the component. A breadth-first search finds + /// the shortest such walk, which keeps the reported path as small as the graph allows. The component is + /// strongly connected, so a walk back to the root always exists. + /// + private static int[] FindCyclePathWithin(int[][] prerequisites, List component, int root) + { + var inComponent = new HashSet(component); + + // Self-reference: the whole cycle is the one node. + if (Array.IndexOf(prerequisites[root], root) >= 0) + { + return [root]; + } + + var parent = new Dictionary(); + var queue = new Queue(); + queue.Enqueue(root); + + while (queue.Count > 0) + { + int current = queue.Dequeue(); + foreach (int prerequisite in prerequisites[current]) + { + if (!inComponent.Contains(prerequisite)) + { + continue; + } + + if (prerequisite == root) + { + // Walk the parents back to the root, then reverse so the path reads root-first. + var path = new List { current }; + int node = current; + while (parent.TryGetValue(node, out int previous)) + { + path.Add(previous); + node = previous; + } + + path.Reverse(); + return [.. path]; + } + + if (!parent.ContainsKey(prerequisite)) + { + parent[prerequisite] = current; + queue.Enqueue(prerequisite); + } + } + } + + // Unreachable for a strongly connected component, but never render a path we cannot justify. + return [root]; + } + + /// + /// Decides which tests run in the sequential phase. That is every [DoNotParallelize] test, plus - + /// transitively - everything that depends on one: because the sequential phase runs after the parallel + /// phase, a parallel test that waited for a sequential prerequisite would never see it complete. Moving + /// the dependent instead of moving the prerequisite keeps [DoNotParallelize]'s own guarantee + /// (that such tests never run alongside anything) intact. + /// + private static bool[] ComputeSequentialSet(UnitTestElement[] tests, int[][] prerequisites, bool[] isBroken, bool parallelizationEnabled) + { + bool[] isSequential = new bool[tests.Length]; + if (!parallelizationEnabled) + { + for (int i = 0; i < isSequential.Length; i++) + { + isSequential[i] = true; + } + + return isSequential; + } + + for (int i = 0; i < tests.Length; i++) + { + if (tests[i].DoNotParallelize && !isBroken[i]) + { + isSequential[i] = true; + } + } + + PropagateSequential(tests.Length, prerequisites, isBroken, isSequential); + return isSequential; + } + + /// + /// Extends to everything that (transitively) waits on a test already in + /// it, because the sequential phase runs after the parallel one: a parallel test waiting on a sequential + /// prerequisite would never observe it complete. + /// + private static void PropagateSequential(int testCount, int[][] prerequisites, bool[] isBroken, bool[] isSequential) + { + // dependents[p] lists the tests that wait for p, so the demotion can be pushed forward along edges. + var dependents = new List[testCount]; + for (int i = 0; i < testCount; i++) + { + foreach (int prerequisite in prerequisites[i]) + { + (dependents[prerequisite] ??= []).Add(i); + } + } + + var queue = new Queue(); + for (int i = 0; i < testCount; i++) + { + if (isSequential[i]) + { + queue.Enqueue(i); + } + } + + while (queue.Count > 0) + { + int current = queue.Dequeue(); + if (dependents[current] is not { } currentDependents) + { + continue; + } + + foreach (int dependent in currentDependents) + { + if (!isSequential[dependent] && !isBroken[dependent]) + { + isSequential[dependent] = true; + queue.Enqueue(dependent); + } + } + } + } + + /// + /// Groups the parallel-phase tests into scheduling chunks and projects the test-level edges onto them. + /// A cycle that exists only in the projection (class A waits for class B and B waits for A, although no + /// single test waits for itself) is reported through so the caller + /// can move those tests into the sequential phase, where the test-level order is honoured exactly. + /// + private static (UnitTestElement[][] Chunks, int[][] ChunkPrerequisites) BuildChunks( + UnitTestElement[] tests, + int[][] prerequisites, + List parallelIndices, + ExecutionScope scope, + List warnings, + out List? projectionCycleTests) + { + projectionCycleTests = null; + int[] chunkOfTest = new int[tests.Length]; + for (int i = 0; i < chunkOfTest.Length; i++) + { + chunkOfTest[i] = -1; + } + + var chunkMembers = new List>(); + if (scope == ExecutionScope.ClassLevel) + { + var chunkByClass = new Dictionary(StringComparer.Ordinal); + foreach (int index in parallelIndices) + { + string className = tests[index].TestMethod.FullClassName; + if (!chunkByClass.TryGetValue(className, out int chunk)) + { + chunkByClass[className] = chunk = chunkMembers.Count; + chunkMembers.Add([]); + } + + chunkOfTest[index] = chunk; + chunkMembers[chunk].Add(index); + } + } + else + { + foreach (int index in parallelIndices) + { + chunkOfTest[index] = chunkMembers.Count; + chunkMembers.Add([index]); + } + } + + var chunkPrerequisites = new HashSet[chunkMembers.Count]; + for (int i = 0; i < chunkPrerequisites.Length; i++) + { + chunkPrerequisites[i] = []; + } + + foreach (int index in parallelIndices) + { + foreach (int prerequisite in prerequisites[index]) + { + int prerequisiteChunk = chunkOfTest[prerequisite]; + + // Edges to a test that is not scheduled in the parallel phase (it is sequential, or broken) + // do not gate a chunk: the sequential phase runs later, and a broken test never runs. The + // run-time outcome check still applies to the dependent. + if (prerequisiteChunk >= 0 && prerequisiteChunk != chunkOfTest[index]) + { + chunkPrerequisites[chunkOfTest[index]].Add(prerequisiteChunk); + } + } + } + + int[][] chunkPrerequisiteArrays = new int[chunkMembers.Count][]; + for (int i = 0; i < chunkMembers.Count; i++) + { + chunkPrerequisiteArrays[i] = [.. chunkPrerequisites[i]]; + } + + if (HasChunkCycle(chunkPrerequisiteArrays, out int[]? blockedChunks, out int[]? cycleChunks)) + { + // Everything blocked has to leave the parallel phase - a chunk merely downstream of the cycle + // cannot be scheduled there either - but only the chunks genuinely in the cycle may be named as + // depending on each other, which is all the message claims. + projectionCycleTests = []; + foreach (int chunk in blockedChunks!) + { + projectionCycleTests.AddRange(chunkMembers[chunk]); + } + + var classNames = new List(); + foreach (int chunk in cycleChunks!) + { + classNames.Add(tests[chunkMembers[chunk][0]].TestMethod.FullClassName); + } + + warnings.Add(string.Format(CultureInfo.CurrentCulture, Resource.DependsOnClassLevelCycle, string.Join(", ", classNames))); + + // The caller re-chunks without these tests, so the arrays built here are discarded. Returning them + // unchanged keeps this method free of a partially-valid state. + return (BuildChunkArrays(chunkMembers, prerequisites, tests), chunkPrerequisiteArrays); + } + + return (BuildChunkArrays(chunkMembers, prerequisites, tests), chunkPrerequisiteArrays); + } + + private static UnitTestElement[][] BuildChunkArrays(List> chunkMembers, int[][] prerequisites, UnitTestElement[] tests) + { + var chunks = new UnitTestElement[chunkMembers.Count][]; + for (int i = 0; i < chunkMembers.Count; i++) + { + chunks[i] = OrderTopologically(chunkMembers[i], prerequisites, tests); + } + + return chunks; + } + + /// + /// Reports whether the chunk graph contains a cycle, and if so which chunks take part in it. + /// + /// + /// Reports whether the chunk graph contains a cycle. Kahn's algorithm cannot settle a node that is on a + /// cycle or merely downstream of one, so the two are separated: + /// is everything that cannot be scheduled (all of it has to leave the parallel phase), while + /// is the subset genuinely in a cycle, which is what the diagnostic may + /// claim depends on each other. + /// + private static bool HasChunkCycle(int[][] chunkPrerequisites, out int[]? blockedChunks, out int[]? cycleChunks) + { + int[] remaining = new int[chunkPrerequisites.Length]; + var dependents = new List[chunkPrerequisites.Length]; + for (int i = 0; i < chunkPrerequisites.Length; i++) + { + remaining[i] = chunkPrerequisites[i].Length; + foreach (int prerequisite in chunkPrerequisites[i]) + { + (dependents[prerequisite] ??= []).Add(i); + } + } + + var queue = new Queue(); + for (int i = 0; i < remaining.Length; i++) + { + if (remaining[i] == 0) + { + queue.Enqueue(i); + } + } + + int settled = 0; + while (queue.Count > 0) + { + int current = queue.Dequeue(); + settled++; + if (dependents[current] is not { } currentDependents) + { + continue; + } + + foreach (int dependent in currentDependents) + { + if (--remaining[dependent] == 0) + { + queue.Enqueue(dependent); + } + } + } + + if (settled == chunkPrerequisites.Length) + { + blockedChunks = null; + cycleChunks = null; + return false; + } + + var blocked = new List(); + for (int i = 0; i < remaining.Length; i++) + { + if (remaining[i] > 0) + { + blocked.Add(i); + } + } + + blockedChunks = [.. blocked]; + cycleChunks = FindChunksOnCycle(chunkPrerequisites, blocked); + return true; + } + + /// + /// Narrows the blocked chunks down to those actually on a cycle, using the same iterative Tarjan pass as + /// : a chunk is on a cycle exactly when its strongly connected component has more + /// than one member, or when it names itself. Peeling chunks that no other blocked chunk depends on is not + /// sufficient - a chunk sitting on a one-way path from one cycle to another has a blocked dependent, so it + /// would survive the peel and be named as mutually dependent when it is only downstream of one cycle and + /// upstream of the other. + /// + private static int[] FindChunksOnCycle(int[][] chunkPrerequisites, List blocked) + { + bool[] isOnCycle = new bool[chunkPrerequisites.Length]; + int[] index = new int[chunkPrerequisites.Length]; + int[] lowLink = new int[chunkPrerequisites.Length]; + bool[] onStack = new bool[chunkPrerequisites.Length]; + int[] edgeCursor = new int[chunkPrerequisites.Length]; + for (int i = 0; i < index.Length; i++) + { + index[i] = -1; + } + + var stack = new List(); + var callStack = new List(); + int nextIndex = 0; + + for (int start = 0; start < chunkPrerequisites.Length; start++) + { + if (index[start] != -1) + { + continue; + } + + // Iterative Tarjan: the recursive form would stack-overflow on a long chain of chunks. + callStack.Add(start); + index[start] = lowLink[start] = nextIndex++; + edgeCursor[start] = 0; + stack.Add(start); + onStack[start] = true; + + while (callStack.Count > 0) + { + int current = callStack[callStack.Count - 1]; + if (edgeCursor[current] < chunkPrerequisites[current].Length) + { + int next = chunkPrerequisites[current][edgeCursor[current]++]; + if (index[next] == -1) + { + index[next] = lowLink[next] = nextIndex++; + edgeCursor[next] = 0; + stack.Add(next); + onStack[next] = true; + callStack.Add(next); + } + else if (onStack[next]) + { + lowLink[current] = Math.Min(lowLink[current], index[next]); + } + + continue; + } + + callStack.RemoveAt(callStack.Count - 1); + if (callStack.Count > 0) + { + int parent = callStack[callStack.Count - 1]; + lowLink[parent] = Math.Min(lowLink[parent], lowLink[current]); + } + + if (lowLink[current] != index[current]) + { + continue; + } + + var component = new List(); + int member; + do + { + member = stack[stack.Count - 1]; + stack.RemoveAt(stack.Count - 1); + onStack[member] = false; + component.Add(member); + } + while (member != current); + + // A component of one is only a cycle when the chunk names itself; anything larger always is. + if (component.Count > 1 || Array.IndexOf(chunkPrerequisites[current], current) >= 0) + { + foreach (int node in component) + { + isOnCycle[node] = true; + } + } + } + } + + // Walking 'blocked' rather than the components keeps the reported order stable and guarantees the + // result is a subset of the blocked set, which is what the caller's warning claims. + var onCycle = new List(); + foreach (int chunk in blocked) + { + if (isOnCycle[chunk]) + { + onCycle.Add(chunk); + } + } + + return [.. onCycle]; + } + + /// + /// Orders so that a test comes after the prerequisites that are themselves in + /// the set, breaking ties by the original position so the result stays deterministic and as close to the + /// declared order as the constraints allow. Any node left over by a cycle is appended in its original + /// order rather than dropped, so a test can never disappear from the run. + /// + private static UnitTestElement[] OrderTopologically(List indices, int[][] prerequisites, UnitTestElement[] tests) + { + if (indices.Count <= 1) + { + return indices.Count == 0 ? [] : [tests[indices[0]]]; + } + + var position = new Dictionary(indices.Count); + for (int i = 0; i < indices.Count; i++) + { + position[indices[i]] = i; + } + + int[] remaining = new int[indices.Count]; + var dependents = new List[indices.Count]; + for (int i = 0; i < indices.Count; i++) + { + foreach (int prerequisite in prerequisites[indices[i]]) + { + if (position.TryGetValue(prerequisite, out int prerequisitePosition)) + { + remaining[i]++; + (dependents[prerequisitePosition] ??= []).Add(i); + } + } + } + + // A sorted set keyed by original position makes the tie-break deterministic: among the tests that + // are ready, the one declared first runs first. + var ready = new SortedSet(); + for (int i = 0; i < indices.Count; i++) + { + if (remaining[i] == 0) + { + ready.Add(i); + } + } + + var ordered = new List(indices.Count); + bool[] emitted = new bool[indices.Count]; + while (ready.Count > 0) + { + int current = ready.Min; + ready.Remove(current); + ordered.Add(tests[indices[current]]); + emitted[current] = true; + + if (dependents[current] is not { } currentDependents) + { + continue; + } + + foreach (int dependent in currentDependents) + { + if (--remaining[dependent] == 0) + { + ready.Add(dependent); + } + } + } + + for (int i = 0; i < indices.Count; i++) + { + if (!emitted[i]) + { + ordered.Add(tests[indices[i]]); + } + } + + return [.. ordered]; + } + + private static List SelectIndices(int count, Func predicate) + { + var result = new List(); + for (int i = 0; i < count; i++) + { + if (predicate(i)) + { + result.Add(i); + } + } + + return result; + } +} diff --git a/src/Adapter/MSTestAdapter.PlatformServices/Execution/TestExecutionManager.Parallelization.cs b/src/Adapter/MSTestAdapter.PlatformServices/Execution/TestExecutionManager.Parallelization.cs index 35839e6405..cde3da1b8e 100644 --- a/src/Adapter/MSTestAdapter.PlatformServices/Execution/TestExecutionManager.Parallelization.cs +++ b/src/Adapter/MSTestAdapter.PlatformServices/Execution/TestExecutionManager.Parallelization.cs @@ -39,6 +39,15 @@ group test by test.TestMethod.AssemblyName into testGroup Shuffle(random, testsBySource); } + // Configured declarations are applied per source below, but their diagnostics have to be judged + // against the whole run: a declaration naming a test in assembly A is legitimately absent from + // assembly B, so warning per source would report every valid declaration as unmatched once for each + // other assembly. Reported once here, against every test in the run. + if (MSTestSettings.CurrentSettings.DeclaredDependencies is { Length: > 0 } declaredDependencies) + { + TestDependencyDeclaration.ReportUnmatchedDeclarations(declaredDependencies, tests, messageLogger); + } + foreach (var group in testsBySource) { _testRunCancellationToken?.ThrowIfCancellationRequested(); @@ -144,151 +153,441 @@ private async Task ExecuteTestsInSourceAsync(IEnumerable tests, parallelScope = MSTestSettings.CurrentSettings.ParallelizationScope.Value; } - if (!MSTestSettings.CurrentSettings.DisableParallelization && sourceSettings.CanParallelizeAssembly && parallelWorkers > 0) + bool parallelizationEnabled = !MSTestSettings.CurrentSettings.DisableParallelization && sourceSettings.CanParallelizeAssembly && parallelWorkers > 0; + + // Sorting the graph's input by name is what makes name order the *tie-breaker* within the dependency + // order, rather than something the graph overrides: OrderTopologically breaks ties by position in this + // array, so tests that are simultaneously ready still run in name order. Doing it here rather than in + // the runner is deliberate - re-sorting a chunk after the topological pass would undo the ordering the + // graph just established. Ordering keys mirror the historical VSTest ManagedType/ManagedMethod + // test-case properties, which are only populated when the test method carries managed method metadata. + if (MSTestSettings.CurrentSettings.OrderTestsByNameInClass && !MSTestSettings.CurrentSettings.RandomizeTestOrder) + { + testsToRun = + [ + .. testsToRun + .OrderBy(t => t.TestMethod.HasManagedMethodAndTypeProperties ? t.TestMethod.ManagedTypeName : null, StringComparer.Ordinal) + .ThenBy(t => t.TestMethod.HasManagedMethodAndTypeProperties ? t.TestMethod.ManagedMethodName : null, StringComparer.Ordinal), + ]; + } + + // Edges declared in testconfig.json are merged onto the elements first, so that from the graph's + // point of view a configured dependency is indistinguishable from an attribute-declared one. + if (MSTestSettings.CurrentSettings.DeclaredDependencies is { Length: > 0 } declaredDependencies) + { + TestDependencyDeclaration.ApplyAll(declaredDependencies, testsToRun, adapterMessageLogger: null); + } + + // Returns null - and so leaves every run that does not use [DependsOn] on exactly the path it uses + // today - unless at least one test in this source declares a dependency. + var dependencyGraph = TestDependencyGraph.Build(testsToRun, parallelScope, parallelizationEnabled); + if (parallelizationEnabled) { // Parallelization is enabled. Let's do further classification for sets. adapterMessageLogger.SendMessage( MessageLevel.Informational, string.Format(CultureInfo.CurrentCulture, Resource.TestParallelizationBanner, source, parallelWorkers, parallelScope)); - // Create test sets for execution, we can execute them in parallel based on parallel settings - // Parallel and not parallel sets. - // Single-pass partition: enumerate testsToRun once via GroupBy (lazy GroupBy evaluated twice - // — once per FirstOrDefault — caused 2 full passes over testsToRun). - IEnumerable? parallelizableTestSet = null; - IEnumerable? nonParallelizableTestSet = null; - foreach (IGrouping group in testsToRun.GroupBy(t => t.DoNotParallelize)) + if (dependencyGraph is not null) { - if (group.Key) - { - nonParallelizableTestSet = group; - } - else - { - parallelizableTestSet = group; - } + await ExecuteTestsWithDependencyGraphAsync(dependencyGraph, adapterMessageLogger, source, sourceLevelParameters, testRunner, usesAppDomains, parallelWorkers).ConfigureAwait(false); } + else + { + await ExecuteParallelizedTestsAsync(testsToRun, adapterMessageLogger, source, sourceLevelParameters, testRunner, usesAppDomains, parallelWorkers, parallelScope).ConfigureAwait(false); + } + } + else if (dependencyGraph is not null) + { + await ExecuteTestsWithDependencyGraphAsync(dependencyGraph, adapterMessageLogger, source, sourceLevelParameters, testRunner, usesAppDomains, parallelWorkers: 1).ConfigureAwait(false); + } + else + { + await ExecuteTestsWithTestRunnerAsync(testsToRun, adapterMessageLogger, source, sourceLevelParameters, testRunner, usesAppDomains).ConfigureAwait(false); + } + + if (PlatformServiceProvider.Instance.IsGracefulStopRequested) + { + testRunner.ForceCleanup(sourceLevelParameters!, new RemotingMessageLogger(adapterMessageLogger)); + } + + if (PlatformServiceProvider.Instance.AdapterTraceLogger.IsInfoEnabled) + { + PlatformServiceProvider.Instance.AdapterTraceLogger.Info("Executed tests belonging to source {0}", source); + } + } - if (parallelizableTestSet != null) + /// + /// The ordinary parallel path: split the tests into a parallelizable and a non-parallelizable set, chunk the + /// former according to , drain the chunks with a fixed pool of workers, then + /// run the non-parallelizable set. + /// + private async Task ExecuteParallelizedTestsAsync( + UnitTestElement[] testsToRun, + IAdapterMessageLogger adapterMessageLogger, + string source, + IDictionary sourceLevelParameters, + UnitTestRunner testRunner, + bool usesAppDomains, + int parallelWorkers, + ExecutionScope parallelScope) + { + // Create test sets for execution, we can execute them in parallel based on parallel settings + // Parallel and not parallel sets. + // Single-pass partition: enumerate testsToRun once via GroupBy (lazy GroupBy evaluated twice + // — once per FirstOrDefault — caused 2 full passes over testsToRun). + IEnumerable? parallelizableTestSet = null; + IEnumerable? nonParallelizableTestSet = null; + foreach (IGrouping group in testsToRun.GroupBy(t => t.DoNotParallelize)) + { + if (group.Key) { - ConcurrentQueue>? queue = null; + nonParallelizableTestSet = group; + } + else + { + parallelizableTestSet = group; + } + } - // Chunk the sets into further groups based on parallel level - switch (parallelScope) - { - case ExecutionScope.MethodLevel: - if (_testOrderRandom is { } methodRandom) - { - IEnumerable[] methodChunks = [.. parallelizableTestSet.Select(t => (IEnumerable)[t])]; - Shuffle(methodRandom, methodChunks); - queue = new ConcurrentQueue>(methodChunks); - } - else - { - queue = new ConcurrentQueue>(parallelizableTestSet.Select(t => new[] { t })); - } + if (parallelizableTestSet != null) + { + ConcurrentQueue>? queue = null; + + // Chunk the sets into further groups based on parallel level + switch (parallelScope) + { + case ExecutionScope.MethodLevel: + if (_testOrderRandom is { } methodRandom) + { + IEnumerable[] methodChunks = [.. parallelizableTestSet.Select(t => (IEnumerable)[t])]; + Shuffle(methodRandom, methodChunks); + queue = new ConcurrentQueue>(methodChunks); + } + else + { + queue = new ConcurrentQueue>(parallelizableTestSet.Select(t => new[] { t })); + } + + break; - break; + case ExecutionScope.ClassLevel: + if (_testOrderRandom is { } classRandom) + { + IEnumerable[] classChunks = + [ + .. parallelizableTestSet + .GroupBy(t => t.TestMethod.FullClassName) + .Select(g => (IEnumerable)g.ToArray()), + ]; + Shuffle(classRandom, classChunks); + queue = new ConcurrentQueue>(classChunks); + } + else + { + queue = new ConcurrentQueue>(parallelizableTestSet.GroupBy(t => t.TestMethod.FullClassName)); + } + + break; + } - case ExecutionScope.ClassLevel: - if (_testOrderRandom is { } classRandom) + var tasks = new List(); + var resourceLockManager = new ResourceLockManager(); + + for (int i = 0; i < parallelWorkers; i++) + { + _testRunCancellationToken?.ThrowIfCancellationRequested(); + + tasks.Add(_taskFactory(async () => + { + try + { + while (!queue!.IsEmpty) { - IEnumerable[] classChunks = - [ - .. parallelizableTestSet - .GroupBy(t => t.TestMethod.FullClassName) - .Select(g => (IEnumerable)g.ToArray()), - ]; - Shuffle(classRandom, classChunks); - queue = new ConcurrentQueue>(classChunks); + _testRunCancellationToken?.ThrowIfCancellationRequested(); + + if (queue.TryDequeue(out IEnumerable? testSet)) + { + IReadOnlyList chunkLocks = ResourceLockManager.GetChunkLocks(testSet); + if (chunkLocks.Count == 0) + { + await ExecuteTestsWithTestRunnerAsync(testSet, adapterMessageLogger, source, sourceLevelParameters, testRunner, usesAppDomains).ConfigureAwait(false); + } + else + { + await resourceLockManager.ExecuteWithLocksAsync( + chunkLocks, + () => ExecuteTestsWithTestRunnerAsync(testSet, adapterMessageLogger, source, sourceLevelParameters, testRunner, usesAppDomains), + _testRunCancellationToken?.CancellationToken ?? CancellationToken.None).ConfigureAwait(false); + } + } } - else + } + catch (OperationCanceledException) when (_testRunCancellationToken?.Canceled == true) + { + // Expected when the test run is canceled. Swallow it so the worker exits gracefully. + if (PlatformServiceProvider.Instance.AdapterTraceLogger.IsInfoEnabled) { - queue = new ConcurrentQueue>(parallelizableTestSet.GroupBy(t => t.TestMethod.FullClassName)); + PlatformServiceProvider.Instance.AdapterTraceLogger.Info("Parallel test worker canceled for source {0}", source); } + } + })); + } - break; + try + { + await Task.WhenAll(tasks).ConfigureAwait(false); + } + catch (Exception ex) + { + string exceptionToString = ex.ToString(); + if (PlatformServiceProvider.Instance.AdapterTraceLogger.IsErrorEnabled) + { + PlatformServiceProvider.Instance.AdapterTraceLogger.Error("Error occurred while executing tests in parallel{0}{1}", Environment.NewLine, exceptionToString); } - var tasks = new List(); - var resourceLockManager = new ResourceLockManager(); + adapterMessageLogger.SendMessage(MessageLevel.Error, exceptionToString); + throw; + } + } - for (int i = 0; i < parallelWorkers; i++) - { - _testRunCancellationToken?.ThrowIfCancellationRequested(); + // Queue the non parallel set + if (nonParallelizableTestSet != null) + { + await ExecuteTestsWithTestRunnerAsync(nonParallelizableTestSet, adapterMessageLogger, source, sourceLevelParameters, testRunner, usesAppDomains).ConfigureAwait(false); + } + } + + /// + /// The dependency-graph path: report the graph's diagnostics, fail the tests caught in a cycle, then run + /// the parallel chunks in topological order - releasing each chunk as soon as the chunks it waits for have + /// finished, so independent branches still run concurrently - and finally the sequential phase. + /// + private async Task ExecuteTestsWithDependencyGraphAsync( + TestDependencyGraph graph, + IAdapterMessageLogger adapterMessageLogger, + string source, + IDictionary sourceLevelParameters, + UnitTestRunner testRunner, + bool usesAppDomains, + int parallelWorkers) + { + var coordinator = new TestDependencyCoordinator(graph); + + // If testRunner is in a different AppDomain, we cannot pass the message logger directly. + IAdapterMessageLogger remotingMessageLogger = usesAppDomains + ? new RemotingMessageLogger(adapterMessageLogger) + : adapterMessageLogger; + + Dictionary lifecycleContextProperties = [with(sourceLevelParameters!)]; + + foreach (string warning in graph.Warnings) + { + adapterMessageLogger.SendMessage(MessageLevel.Warning, warning); + } + + foreach (string error in graph.Errors) + { + adapterMessageLogger.SendMessage(MessageLevel.Error, error); + } + + // A test in a cycle is reported as failed - the declaration, not the test, is what is broken, and a + // silent skip would hide it. Each failure carries the description of the cycle *that test* is in, so + // an assembly with two unrelated cycles does not stamp both paths onto every failure. Recording it as + // "did not pass" also makes everything downstream skip. + foreach (TestDependencyGraph.BrokenTest brokenTest in graph.BrokenTests) + { + _testRunCancellationToken?.ThrowIfCancellationRequested(); + coordinator.RecordNotRun(brokenTest.Element); + + DateTimeOffset now = DateTimeOffset.Now; + var cycleResult = new TestTools.UnitTesting.TestResult + { + Outcome = TestTools.UnitTesting.UnitTestOutcome.Failed, + TestFailureException = new InvalidOperationException(brokenTest.CycleMessage), + }; + + await _testResultRecorder.RecordStartAsync(brokenTest.Element).ConfigureAwait(false); + + // Selected but never run, so the class-cleanup countdown still owes this test its decrement. + // See UnitTestRunner.NotifyTestNotRunAsync. + UnitTestElement brokenElement = brokenTest.Element.WithUpdatedSource(source); + Dictionary testContextProperties = GetTestContextProperties(brokenTest.Element.ExecutionContextProperties, sourceLevelParameters, brokenElement); + TestTools.UnitTesting.TestResult[] cleanupResults = usesAppDomains || Thread.CurrentThread.GetApartmentState() == ApartmentState.STA +#pragma warning disable VSTHRD103 // Call async methods when in an async method - mirrors RunSingleTest: Task cannot cross app domains, and an await would leave the STA thread. + ? testRunner.NotifyTestNotRun(brokenElement, testContextProperties, lifecycleContextProperties, remotingMessageLogger) +#pragma warning restore VSTHRD103 + : await testRunner.NotifyTestNotRunAsync(brokenElement, testContextProperties, lifecycleContextProperties, remotingMessageLogger).ConfigureAwait(false); + + await SendTestResultsAsync( + brokenTest.Element, + [cycleResult, .. cleanupResults], + now, + now, + _testResultRecorder).ConfigureAwait(false); + } + + if (graph.ParallelChunks.Length > 0) + { + await ExecuteChunksInTopologicalOrderAsync(graph, coordinator, adapterMessageLogger, source, sourceLevelParameters, testRunner, usesAppDomains, parallelWorkers).ConfigureAwait(false); + } + + if (graph.SequentialTests.Length > 0) + { + await ExecuteTestsWithTestRunnerAsync(graph.SequentialTests, adapterMessageLogger, source, sourceLevelParameters, testRunner, usesAppDomains, coordinator).ConfigureAwait(false); + } + } + + /// + /// Drains the parallel chunks with a fixed pool of workers, honoring the chunk-level dependency edges: a + /// chunk becomes available the moment the last chunk it waits for finishes, which is what lets the branches + /// of a fan-out run at the same time. Completion - not success - releases a dependent chunk; whether its + /// individual tests actually run is decided per test by . + /// + private async Task ExecuteChunksInTopologicalOrderAsync( + TestDependencyGraph graph, + TestDependencyCoordinator coordinator, + IAdapterMessageLogger adapterMessageLogger, + string source, + IDictionary sourceLevelParameters, + UnitTestRunner testRunner, + bool usesAppDomains, + int parallelWorkers) + { + int chunkCount = graph.ParallelChunks.Length; + int[] pendingPrerequisites = new int[chunkCount]; + var dependents = new List[chunkCount]; + for (int i = 0; i < chunkCount; i++) + { + pendingPrerequisites[i] = graph.ParallelChunkPrerequisites[i].Length; + foreach (int prerequisite in graph.ParallelChunkPrerequisites[i]) + { + (dependents[prerequisite] ??= []).Add(i); + } + } + + var ready = new ConcurrentQueue(); + + // One permit per queued chunk, plus - once everything is done - one per worker so that every worker + // wakes up and exits instead of blocking on a queue that will never be fed again. + var available = new SemaphoreSlim(0); + int queued = 0; + for (int i = 0; i < chunkCount; i++) + { + if (pendingPrerequisites[i] == 0) + { + ready.Enqueue(i); + queued++; + } + } + + // Release(0) throws, and a zero count would mean every chunk had an unmet prerequisite - that is, + // a cycle in the chunk graph. Build guarantees that cannot happen (chunks caught in a projection + // cycle are demoted to the sequential phase), but the guard keeps a future regression in that + // invariant from surfacing here as an argument exception instead of where it belongs. + if (queued > 0) + { + available.Release(queued); + } + + int completedChunks = 0; + int effectiveWorkers = Math.Max(1, Math.Min(parallelWorkers, chunkCount)); + var resourceLockManager = new ResourceLockManager(); + var tasks = new List(effectiveWorkers); + + for (int worker = 0; worker < effectiveWorkers; worker++) + { + _testRunCancellationToken?.ThrowIfCancellationRequested(); - tasks.Add(_taskFactory(async () => + tasks.Add(_taskFactory(async () => + { + try + { + while (true) { + _testRunCancellationToken?.ThrowIfCancellationRequested(); + await available.WaitAsync(_testRunCancellationToken?.CancellationToken ?? CancellationToken.None).ConfigureAwait(false); + + // A permit with nothing behind it is the shutdown signal released below. + if (!ready.TryDequeue(out int chunkIndex)) + { + return; + } + + UnitTestElement[] chunk = graph.ParallelChunks[chunkIndex]; try { - while (!queue!.IsEmpty) + IReadOnlyList chunkLocks = ResourceLockManager.GetChunkLocks(chunk); + if (chunkLocks.Count == 0) { - _testRunCancellationToken?.ThrowIfCancellationRequested(); - - if (queue.TryDequeue(out IEnumerable? testSet)) + await ExecuteTestsWithTestRunnerAsync(chunk, adapterMessageLogger, source, sourceLevelParameters, testRunner, usesAppDomains, coordinator).ConfigureAwait(false); + } + else + { + await resourceLockManager.ExecuteWithLocksAsync( + chunkLocks, + () => ExecuteTestsWithTestRunnerAsync(chunk, adapterMessageLogger, source, sourceLevelParameters, testRunner, usesAppDomains, coordinator), + _testRunCancellationToken?.CancellationToken ?? CancellationToken.None).ConfigureAwait(false); + } + } + finally + { + // The bookkeeping below must happen even when the chunk failed, otherwise the + // chunks waiting on it would never be released and the workers waiting for them + // would block forever - turning one faulty chunk into a hung run. Releasing on + // completion rather than success is also what the dependency semantics ask for: + // whether the dependent tests actually run is decided per test by the + // coordinator, which sees the (absent) outcomes and skips them. + if (dependents[chunkIndex] is { } chunkDependents) + { + foreach (int dependent in chunkDependents) { - IReadOnlyList chunkLocks = ResourceLockManager.GetChunkLocks(testSet); - if (chunkLocks.Count == 0) - { - await ExecuteTestsWithTestRunnerAsync(testSet, adapterMessageLogger, source, sourceLevelParameters, testRunner, usesAppDomains).ConfigureAwait(false); - } - else + if (Interlocked.Decrement(ref pendingPrerequisites[dependent]) == 0) { - await resourceLockManager.ExecuteWithLocksAsync( - chunkLocks, - () => ExecuteTestsWithTestRunnerAsync(testSet, adapterMessageLogger, source, sourceLevelParameters, testRunner, usesAppDomains), - _testRunCancellationToken?.CancellationToken ?? CancellationToken.None).ConfigureAwait(false); + ready.Enqueue(dependent); + available.Release(); } } } - } - catch (OperationCanceledException) when (_testRunCancellationToken?.Canceled == true) - { - // Expected when the test run is canceled. Swallow it so the worker exits gracefully. - if (PlatformServiceProvider.Instance.AdapterTraceLogger.IsInfoEnabled) + + // Once every chunk has been accounted for, wake every worker so they observe the + // empty queue and exit instead of waiting for a permit that will never come. + if (Interlocked.Increment(ref completedChunks) == chunkCount) { - PlatformServiceProvider.Instance.AdapterTraceLogger.Info("Parallel test worker canceled for source {0}", source); + available.Release(effectiveWorkers); } } - })); - } - - try - { - await Task.WhenAll(tasks).ConfigureAwait(false); + } } - catch (Exception ex) + catch (OperationCanceledException) when (_testRunCancellationToken?.Canceled == true) { - string exceptionToString = ex.ToString(); - if (PlatformServiceProvider.Instance.AdapterTraceLogger.IsErrorEnabled) + // Expected when the test run is canceled. Swallow it so the worker exits gracefully. + if (PlatformServiceProvider.Instance.AdapterTraceLogger.IsInfoEnabled) { - PlatformServiceProvider.Instance.AdapterTraceLogger.Error("Error occurred while executing tests in parallel{0}{1}", Environment.NewLine, exceptionToString); + PlatformServiceProvider.Instance.AdapterTraceLogger.Info("Parallel test worker canceled for source {0}", source); } - - adapterMessageLogger.SendMessage(MessageLevel.Error, exceptionToString); - throw; } - } - - // Queue the non parallel set - if (nonParallelizableTestSet != null) - { - await ExecuteTestsWithTestRunnerAsync(nonParallelizableTestSet, adapterMessageLogger, source, sourceLevelParameters, testRunner, usesAppDomains).ConfigureAwait(false); - } - } - else - { - await ExecuteTestsWithTestRunnerAsync(testsToRun, adapterMessageLogger, source, sourceLevelParameters, testRunner, usesAppDomains).ConfigureAwait(false); + })); } - if (PlatformServiceProvider.Instance.IsGracefulStopRequested) + try { - testRunner.ForceCleanup(sourceLevelParameters!, new RemotingMessageLogger(adapterMessageLogger)); + await Task.WhenAll(tasks).ConfigureAwait(false); } + catch (Exception ex) + { + string exceptionToString = ex.ToString(); + if (PlatformServiceProvider.Instance.AdapterTraceLogger.IsErrorEnabled) + { + PlatformServiceProvider.Instance.AdapterTraceLogger.Error("Error occurred while executing tests in parallel{0}{1}", Environment.NewLine, exceptionToString); + } - if (PlatformServiceProvider.Instance.AdapterTraceLogger.IsInfoEnabled) + adapterMessageLogger.SendMessage(MessageLevel.Error, exceptionToString); + throw; + } + finally { - PlatformServiceProvider.Instance.AdapterTraceLogger.Info("Executed tests belonging to source {0}", source); + available.Dispose(); } } } diff --git a/src/Adapter/MSTestAdapter.PlatformServices/Execution/TestExecutionManager.Runner.cs b/src/Adapter/MSTestAdapter.PlatformServices/Execution/TestExecutionManager.Runner.cs index dfbfe00ec8..025d7a8c82 100644 --- a/src/Adapter/MSTestAdapter.PlatformServices/Execution/TestExecutionManager.Runner.cs +++ b/src/Adapter/MSTestAdapter.PlatformServices/Execution/TestExecutionManager.Runner.cs @@ -1,4 +1,4 @@ -// Copyright (c) Microsoft Corporation. All rights reserved. +// Copyright (c) Microsoft Corporation. All rights reserved. // Licensed under the MIT license. See LICENSE file in the project root for full license information. using Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.ObjectModel; @@ -54,11 +54,16 @@ private async Task ExecuteTestsWithTestRunnerAsync( string source, IDictionary sourceLevelParameters, UnitTestRunner testRunner, - bool usesAppDomains) + bool usesAppDomains, + TestDependencyCoordinator? dependencyCoordinator = null) { // Ordering keys mirror the historical VSTest ManagedType/ManagedMethod test-case properties, which are // only populated when the test method carries managed method metadata (see UnitTestElement.ToTestCase). - IEnumerable orderedTests = MSTestSettings.CurrentSettings.OrderTestsByNameInClass && !MSTestSettings.CurrentSettings.RandomizeTestOrder + // When a dependency graph is in effect the order is already fixed by the graph's topological sort - + // which took name order as its tie-breaker, so the setting is honored there rather than bypassed - + // and re-sorting here would undo it. + IEnumerable orderedTests = dependencyCoordinator is null + && MSTestSettings.CurrentSettings.OrderTestsByNameInClass && !MSTestSettings.CurrentSettings.RandomizeTestOrder ? tests.OrderBy(t => t.TestMethod.HasManagedMethodAndTypeProperties ? t.TestMethod.ManagedTypeName : null) .ThenBy(t => t.TestMethod.HasManagedMethodAndTypeProperties ? t.TestMethod.ManagedMethodName : null) : tests; @@ -81,6 +86,24 @@ private async Task ExecuteTestsWithTestRunnerAsync( UnitTestElement unitTestElement = currentTest.WithUpdatedSource(source); + // A test whose prerequisite did not pass is reported as skipped instead of being run. This is + // decided here, right before the test starts, so that the outcome of a prerequisite that finished + // moments ago on another worker is always taken into account. + if (dependencyCoordinator is not null && dependencyCoordinator.ShouldSkip(currentTest, out string? skipReason)) + { + await ReportSkippedDependentAsync( + currentTest, + unitTestElement, + skipReason, + dependencyCoordinator, + sourceLevelParameters, + lifecycleContextProperties, + testRunner, + usesAppDomains, + remotingMessageLogger).ConfigureAwait(false); + continue; + } + // Report through the neutral recorder using the element itself; the adapter-side recorder resolves // the host test case (preserving host-injected TCM / data-collector properties) with full fidelity. await _testResultRecorder.RecordStartAsync(currentTest).ConfigureAwait(false); @@ -121,7 +144,71 @@ private async Task ExecuteTestsWithTestRunnerAsync( DateTimeOffset endTime = DateTimeOffset.Now; + dependencyCoordinator?.RecordOutcome(currentTest, AllPassed(unitTestResult)); + await SendTestResultsAsync(currentTest, unitTestResult, startTime, endTime, _testResultRecorder).ConfigureAwait(false); } } + + /// + /// Determines whether a test counts as having passed for the purpose of gating its dependents. A test + /// that produced no result at all (which the recorder reports as an error) does not qualify, and a + /// data-driven test qualifies only when every one of its rows passed. + /// + private static bool AllPassed(TestTools.UnitTesting.TestResult[] results) + { + if (results.Length == 0) + { + return false; + } + + foreach (TestTools.UnitTesting.TestResult result in results) + { + if (result.Outcome != TestTools.UnitTesting.UnitTestOutcome.Passed) + { + return false; + } + } + + return true; + } + + /// + /// Reports a test that was not run because one of its prerequisites did not pass. It is recorded as + /// skipped - not failed - so that a single root cause stays visible as one failure surrounded by clearly + /// labelled skips, and it is recorded as "did not pass" so the skip propagates to its own dependents. + /// + private async Task ReportSkippedDependentAsync( + UnitTestElement test, + UnitTestElement unitTestElement, + string reason, + TestDependencyCoordinator dependencyCoordinator, + IDictionary sourceLevelParameters, + Dictionary lifecycleContextProperties, + UnitTestRunner testRunner, + bool usesAppDomains, + IAdapterMessageLogger remotingMessageLogger) + { + dependencyCoordinator.RecordNotRun(test); + + DateTimeOffset now = DateTimeOffset.Now; + await _testResultRecorder.RecordStartAsync(test).ConfigureAwait(false); + + // The test was selected, so it is counted in the class-cleanup countdown even though it is not going + // to run. Tell the runner, or the class never completes - which loses its [ClassCleanup] and, because + // end-of-assembly cleanup waits on every class, the assembly's [AssemblyCleanup] too. + Dictionary testContextProperties = GetTestContextProperties(test.ExecutionContextProperties, sourceLevelParameters, unitTestElement); + TestTools.UnitTesting.TestResult[] cleanupResults = usesAppDomains || Thread.CurrentThread.GetApartmentState() == ApartmentState.STA +#pragma warning disable VSTHRD103 // Call async methods when in an async method - mirrors RunSingleTest: Task cannot cross app domains, and an await would leave the STA thread. + ? testRunner.NotifyTestNotRun(unitTestElement, testContextProperties, lifecycleContextProperties, remotingMessageLogger) +#pragma warning restore VSTHRD103 + : await testRunner.NotifyTestNotRunAsync(unitTestElement, testContextProperties, lifecycleContextProperties, remotingMessageLogger).ConfigureAwait(false); + + await SendTestResultsAsync( + test, + [TestTools.UnitTesting.TestResult.CreateIgnoredResult(reason), .. cleanupResults], + now, + now, + _testResultRecorder).ConfigureAwait(false); + } } diff --git a/src/Adapter/MSTestAdapter.PlatformServices/Execution/UnitTestRunner.RunSingleTest.cs b/src/Adapter/MSTestAdapter.PlatformServices/Execution/UnitTestRunner.RunSingleTest.cs index a4c4793d0f..15983e69d2 100644 --- a/src/Adapter/MSTestAdapter.PlatformServices/Execution/UnitTestRunner.RunSingleTest.cs +++ b/src/Adapter/MSTestAdapter.PlatformServices/Execution/UnitTestRunner.RunSingleTest.cs @@ -35,6 +35,62 @@ internal TestResult[] RunSingleTest( internal async Task RunSingleTestAsync(UnitTestElement unitTestElement, IDictionary testContextProperties, IAdapterMessageLogger messageLogger) => await RunSingleTestAsync(unitTestElement, testContextProperties, testContextProperties, messageLogger).ConfigureAwait(false); + // Task cannot cross app domains. + // For now, TestExecutionManager will call this sync method which is hacky. + internal TestResult[] NotifyTestNotRun( + UnitTestElement unitTestElement, + IDictionary testContextProperties, + IDictionary lifecycleContextProperties, + IAdapterMessageLogger messageLogger) + => NotifyTestNotRunAsync(unitTestElement, testContextProperties, lifecycleContextProperties, messageLogger).GetAwaiter().GetResult(); + + /// + /// Tells the runner that a selected test is not going to execute, because the scheduler decided its + /// outcome without running it - its prerequisite did not pass, or it is part of a dependency cycle. + /// + /// + /// The class-cleanup countdown is built from every selected test, so a test that never reaches + /// + /// still owes its decrement. This performs exactly the bookkeeping a filtered-out test does, which is + /// the same situation: selected, counted, never run. + /// + /// The test that will not run. + /// Properties scoped to this test. + /// Properties scoped to assembly and class lifecycle methods. + /// The message logger. + /// + /// Any results produced by cleanup that this call triggered - empty unless a [ClassCleanup] or + /// [AssemblyCleanup] ran and failed. The test's own "skipped" or "failed" result is reported by + /// the caller, which is what decided the outcome. + /// + internal async Task NotifyTestNotRunAsync( + UnitTestElement unitTestElement, + IDictionary testContextProperties, + IDictionary lifecycleContextProperties, + IAdapterMessageLogger messageLogger) + { + if (unitTestElement is null) + { + throw new ArgumentNullException(nameof(unitTestElement)); + } + + ITestContext? testContextForTestExecution = null; + try + { + testContextForTestExecution = PlatformServiceProvider.Instance.GetTestContext(unitTestElement.TestMethod, null, testContextProperties, messageLogger, UnitTestOutcome.InProgress); + return await FinishTestThatDidNotRunAsync( + unitTestElement.TestMethod, + lifecycleContextProperties, + messageLogger, + [], + testContextForTestExecution).ConfigureAwait(false); + } + finally + { + (testContextForTestExecution as IDisposable)?.Dispose(); + } + } + /// /// Runs a single test. /// @@ -82,7 +138,7 @@ internal async Task RunSingleTestAsync( TestResult[]? filterResult = ApplyTestFilter(unitTestElement); if (filterResult is not null) { - return await FinishFilteredOutTestAsync( + return await FinishTestThatDidNotRunAsync( testMethod, lifecycleContextProperties, messageLogger, @@ -118,7 +174,7 @@ internal async Task RunSingleTestAsync( // Remember that assembly initialize ran for this assembly so the end-of-assembly cleanup // guard still fires even when the last test of the assembly is filtered out (and therefore - // has no testMethodInfo of its own). See FinishFilteredOutTestAsync. + // has no testMethodInfo of its own). See FinishTestThatDidNotRunAsync. _assemblyInitializeWasExecuted |= assemblyInfo.IsAssemblyInitializeExecuted; if (assemblyInitializeResult.Outcome != UnitTestOutcome.Passed) diff --git a/src/Adapter/MSTestAdapter.PlatformServices/Execution/UnitTestRunner.TestFilter.cs b/src/Adapter/MSTestAdapter.PlatformServices/Execution/UnitTestRunner.TestFilter.cs index cd2f57de2c..584e3f422d 100644 --- a/src/Adapter/MSTestAdapter.PlatformServices/Execution/UnitTestRunner.TestFilter.cs +++ b/src/Adapter/MSTestAdapter.PlatformServices/Execution/UnitTestRunner.TestFilter.cs @@ -140,13 +140,20 @@ private static TestFilterContext CreateFilterContext(UnitTestElement element) } /// - /// Handles the bookkeeping (class-cleanup countdown, class cleanup, end-of-assembly cleanup) for a - /// test that was filtered out by a . Mirrors the tail of - /// normal test execution path. The filtered-out test never loaded its own type, but if a + /// Handles the bookkeeping (class-cleanup countdown, class cleanup, end-of-assembly cleanup) for a test + /// that was selected for the run - and so counted in the class-cleanup countdown - but never executed. + /// Mirrors the tail of normal test execution path. The test never loaded its own type, but if a /// sibling test of the same class already ran in this worker the class was initialized and still /// owes its [ClassCleanup], so it is executed here when this is the last test of the class. /// - private async Task FinishFilteredOutTestAsync( + /// + /// Shared by every "selected but not run" path: a test dropped by an , and a + /// test whose outcome the scheduler decided without running it (a prerequisite did not pass, or it is + /// part of a dependency cycle). Skipping this bookkeeping leaves the countdown permanently above zero, + /// which silently loses the class's [ClassCleanup] and - because end-of-assembly cleanup is gated + /// on every class having completed - the whole assembly's [AssemblyCleanup]. + /// + private async Task FinishTestThatDidNotRunAsync( TestMethod testMethod, IDictionary lifecycleContextProperties, IAdapterMessageLogger messageLogger, diff --git a/src/Adapter/MSTestAdapter.PlatformServices/InternalAPI/InternalAPI.Unshipped.txt b/src/Adapter/MSTestAdapter.PlatformServices/InternalAPI/InternalAPI.Unshipped.txt index 4f92ca88d0..00ff1dc5a1 100644 --- a/src/Adapter/MSTestAdapter.PlatformServices/InternalAPI/InternalAPI.Unshipped.txt +++ b/src/Adapter/MSTestAdapter.PlatformServices/InternalAPI/InternalAPI.Unshipped.txt @@ -1,8 +1,3 @@ -#nullable enable -Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution.AsyncReaderWriterLock -Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution.AsyncReaderWriterLock.AcquireReaderAsync(System.Threading.CancellationToken cancellationToken) -> System.Threading.Tasks.Task! -Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution.AsyncReaderWriterLock.AcquireWriterAsync(System.Threading.CancellationToken cancellationToken) -> System.Threading.Tasks.Task! -Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution.AsyncReaderWriterLock.AsyncReaderWriterLock() -> void *REMOVED*Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution.ConsoleErrorRouter.ConsoleErrorRouter(System.IO.TextWriter! originalConsoleErr) -> void *REMOVED*Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution.ConsoleOutRouter.ConsoleOutRouter(System.IO.TextWriter! originalConsoleOut) -> void *REMOVED*Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution.ConsoleRouter.ConsoleRouter(System.IO.TextWriter! originalConsole) -> void @@ -16,17 +11,49 @@ Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution.AsyncReaderWrit *REMOVED*Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.TestContextImplementation.WriteConsoleOut(string? value) -> void *REMOVED*Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.TestContextImplementation.WriteTrace(char value) -> void *REMOVED*Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.TestContextImplementation.WriteTrace(string? value) -> void +#nullable enable +Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution.AsyncReaderWriterLock +Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution.AsyncReaderWriterLock.AcquireReaderAsync(System.Threading.CancellationToken cancellationToken) -> System.Threading.Tasks.Task! +Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution.AsyncReaderWriterLock.AcquireWriterAsync(System.Threading.CancellationToken cancellationToken) -> System.Threading.Tasks.Task! +Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution.AsyncReaderWriterLock.AsyncReaderWriterLock() -> void Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution.ConsoleErrorRouter.ConsoleErrorRouter(System.IO.TextWriter! originalConsoleErr, System.Func! modeProvider) -> void Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution.ConsoleOutRouter.ConsoleOutRouter(System.IO.TextWriter! originalConsoleOut, System.Func! modeProvider) -> void Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution.ConsoleRouter.ConsoleRouter(System.IO.TextWriter! originalConsole, System.Func! modeProvider) -> void Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution.ResourceLockManager Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution.ResourceLockManager.ExecuteWithLocksAsync(System.Collections.Generic.IReadOnlyList! locks, System.Func! action, System.Threading.CancellationToken cancellationToken) -> System.Threading.Tasks.Task! Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution.ResourceLockManager.ResourceLockManager() -> void +Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution.TestDependencyCoordinator +Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution.TestDependencyCoordinator.RecordNotRun(Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.ObjectModel.UnitTestElement! test) -> void +Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution.TestDependencyCoordinator.RecordOutcome(Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.ObjectModel.UnitTestElement! test, bool passed) -> void +Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution.TestDependencyCoordinator.ShouldSkip(Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.ObjectModel.UnitTestElement! test, out string? reason) -> bool +Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution.TestDependencyCoordinator.TestDependencyCoordinator(Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution.TestDependencyGraph! graph) -> void +Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution.TestDependencyDeclaration +Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution.TestDependencyDeclaration.Dependent.get -> string! +Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution.TestDependencyDeclaration.Prerequisite.get -> string! +Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution.TestDependencyDeclaration.ProceedOnFailure.get -> bool +Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution.TestDependencyDeclaration.TestDependencyDeclaration(string! dependent, string! prerequisite, bool proceedOnFailure) -> void +Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution.TestDependencyGraph +Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution.TestDependencyGraph.BrokenTest +Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution.TestDependencyGraph.BrokenTest.BrokenTest(Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.ObjectModel.UnitTestElement! element, string! cycleMessage) -> void +Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution.TestDependencyGraph.BrokenTest.CycleMessage.get -> string! +Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution.TestDependencyGraph.BrokenTest.Element.get -> Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.ObjectModel.UnitTestElement! +Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution.TestDependencyGraph.BrokenTests.get -> System.Collections.Generic.IReadOnlyList! +Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution.TestDependencyGraph.Errors.get -> System.Collections.Generic.IReadOnlyList! +Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution.TestDependencyGraph.ParallelChunkPrerequisites.get -> int[]![]! +Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution.TestDependencyGraph.ParallelChunks.get -> Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.ObjectModel.UnitTestElement![]![]! +Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution.TestDependencyGraph.ProceedOnFailure.get -> bool[]! +Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution.TestDependencyGraph.SequentialTests.get -> Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.ObjectModel.UnitTestElement![]! +Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution.TestDependencyGraph.TestPrerequisites.get -> int[]![]! +Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution.TestDependencyGraph.Tests.get -> Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.ObjectModel.UnitTestElement![]! +Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution.TestDependencyGraph.Warnings.get -> System.Collections.Generic.IReadOnlyList! Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution.TraceTextWriter.TraceTextWriter(System.IO.TextWriter! console, System.Func! modeProvider) -> void +Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution.UnitTestRunner.NotifyTestNotRun(Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.ObjectModel.UnitTestElement! unitTestElement, System.Collections.Generic.IDictionary! testContextProperties, System.Collections.Generic.IDictionary! lifecycleContextProperties, Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.Interface.IAdapterMessageLogger! messageLogger) -> Microsoft.VisualStudio.TestTools.UnitTesting.TestResult![]! +Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution.UnitTestRunner.NotifyTestNotRunAsync(Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.ObjectModel.UnitTestElement! unitTestElement, System.Collections.Generic.IDictionary! testContextProperties, System.Collections.Generic.IDictionary! lifecycleContextProperties, Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.Interface.IAdapterMessageLogger! messageLogger) -> System.Threading.Tasks.Task! Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution.UnitTestRunner.RunSingleTest(Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.ObjectModel.UnitTestElement! unitTestElement, System.Collections.Generic.IDictionary! testContextProperties, System.Collections.Generic.IDictionary! lifecycleContextProperties, Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.Interface.IAdapterMessageLogger! messageLogger) -> Microsoft.VisualStudio.TestTools.UnitTesting.TestResult![]! Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution.UnitTestRunner.RunSingleTestAsync(Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.ObjectModel.UnitTestElement! unitTestElement, System.Collections.Generic.IDictionary! testContextProperties, System.Collections.Generic.IDictionary! lifecycleContextProperties, Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.Interface.IAdapterMessageLogger! messageLogger) -> System.Threading.Tasks.Task! Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Helpers.FixtureKind.GlobalTestCleanup = 7 -> Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Helpers.FixtureKind Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Helpers.FixtureKind.GlobalTestInitialize = 6 -> Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Helpers.FixtureKind +Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.MSTestSettings.DeclaredDependencies.get -> Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution.TestDependencyDeclaration![]? Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.MSTestSettings.GlobalTestCleanupTimeout.get -> int Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.MSTestSettings.GlobalTestInitializeTimeout.get -> int Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.MSTestSettings.OutputCaptureMode.get -> Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.TestOutputCaptureMode @@ -35,9 +62,17 @@ Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.ObjectModel.ResourceLockI Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.ObjectModel.ResourceLockInfo.Mode.get -> Microsoft.VisualStudio.TestTools.UnitTesting.ResourceAccessMode Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.ObjectModel.ResourceLockInfo.Resource.get -> string! Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.ObjectModel.ResourceLockInfo.ResourceLockInfo(string! resource, Microsoft.VisualStudio.TestTools.UnitTesting.ResourceAccessMode mode) -> void +Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.ObjectModel.TestDependencyInfo +Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.ObjectModel.TestDependencyInfo.DescribeTarget() -> string! +Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.ObjectModel.TestDependencyInfo.ProceedOnFailure.get -> bool +Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.ObjectModel.TestDependencyInfo.TargetClassFullName.get -> string? +Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.ObjectModel.TestDependencyInfo.TargetMethodName.get -> string? +Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.ObjectModel.TestDependencyInfo.TestDependencyInfo(string? targetClassFullName, string? targetMethodName, bool proceedOnFailure) -> void Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.ObjectModel.TestMethod.FullyQualifiedName.get -> string! Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.ObjectModel.UnitTestElement.CachedTestNodeUid.get -> System.Guid? Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.ObjectModel.UnitTestElement.CachedTestNodeUid.set -> void +Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.ObjectModel.UnitTestElement.Dependencies.get -> Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.ObjectModel.TestDependencyInfo![]? +Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.ObjectModel.UnitTestElement.Dependencies.set -> void Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.ObjectModel.UnitTestElement.ResourceLocks.get -> Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.ObjectModel.ResourceLockInfo![]? Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.ObjectModel.UnitTestElement.ResourceLocks.set -> void Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.TestOutputCaptureMode @@ -51,6 +86,9 @@ Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.TestContextIm Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.TestContextImplementation.TraceBuilder.get -> Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.TestContextImplementation.SynchronizedStringBuilder! override Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.TestContextImplementation.TestTempDirectory.get -> string? static Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution.ResourceLockManager.GetChunkLocks(System.Collections.Generic.IEnumerable! testSet) -> System.Collections.Generic.IReadOnlyList! +static Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution.TestDependencyDeclaration.ApplyAll(System.Collections.Generic.IEnumerable! declarations, Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.ObjectModel.UnitTestElement![]! tests, Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.Interface.IAdapterMessageLogger? adapterMessageLogger) -> bool +static Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution.TestDependencyDeclaration.ReportUnmatchedDeclarations(System.Collections.Generic.IEnumerable! declarations, System.Collections.Generic.IEnumerable! allTests, Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.Interface.IAdapterMessageLogger? logger) -> void +static Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution.TestDependencyGraph.Build(Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.ObjectModel.UnitTestElement![]! tests, Microsoft.VisualStudio.TestTools.UnitTesting.ExecutionScope scope, bool parallelizationEnabled) -> Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution.TestDependencyGraph? static Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution.TestMethodInfo.ParameterMetadataScanCount.get -> int static Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution.TestMethodInfo.ResetParameterMetadataCacheForTesting() -> void static Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution.TestMethodInfo.SetParameterMetadataScanCallbackForTesting(System.Action? callback) -> void @@ -59,5 +97,7 @@ static Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Helpers.RuntimeCon static Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.MSTestDiscovererHelpers.GetSettingsExceptionMessage(Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.ObjectModel.AdapterSettingsException! ex) -> string! static Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.ObjectModel.ResourceLockInfo.Decode(string! encoded) -> Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.ObjectModel.ResourceLockInfo! static Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.ObjectModel.ResourceLockInfo.Encode(Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.ObjectModel.ResourceLockInfo! info) -> string! +static Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.ObjectModel.TestDependencyInfo.Decode(string! encoded) -> Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.ObjectModel.TestDependencyInfo! +static Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.ObjectModel.TestDependencyInfo.Encode(Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.ObjectModel.TestDependencyInfo! info) -> string! static Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.TestContextImplementation.ConfigureLiveOutputWriter(System.IO.TextWriter! liveOutputWriter) -> void static Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.TestContextImplementation.SetLiveOutputWriterForTesting(System.IO.TextWriter! liveOutputWriter) -> System.IDisposable! diff --git a/src/Adapter/MSTestAdapter.PlatformServices/MSTestSettings.Configuration.cs b/src/Adapter/MSTestAdapter.PlatformServices/MSTestSettings.Configuration.cs index bb40df1723..6bfaf8b6ec 100644 --- a/src/Adapter/MSTestAdapter.PlatformServices/MSTestSettings.Configuration.cs +++ b/src/Adapter/MSTestAdapter.PlatformServices/MSTestSettings.Configuration.cs @@ -2,6 +2,7 @@ // Licensed under the MIT license. See LICENSE file in the project root for full license information. #if !WINDOWS_UWP +using Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution; using Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.ObjectModel; using Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices; @@ -171,6 +172,122 @@ private static void ParseDebuggerLaunchModeSetting(IConfiguration configuration, } } + /// + /// Reads the mstest:execution:dependencies section, which declares test dependencies outside the + /// test source as an alternative to [DependsOn]. Two shapes are supported: chains, where + /// each entry is an ordered list and every element waits for the one before it (the flat case), and + /// nodes, where an entry names one test and the prerequisites it waits for (the tree case: + /// fan-in, fan-out and per-node options). + /// + /// + /// The configuration model flattens arrays to indexed keys (...:nodes:0:test), so the indices are + /// probed until a key is absent. That is the same convention Microsoft.Extensions.Configuration uses. + /// + private static void ParseDependencySettings(IConfiguration configuration, IAdapterMessageLogger? logger, MSTestSettings settings) + { + const string root = "mstest:execution:dependencies"; + List? declarations = null; + + // chains: [ [ "A.B.First", "A.B.Second" ], ... ] - each element waits for the previous one. + // + // The configuration model flattens an array to indexed keys and stores nothing under an *empty* + // element, and IConfiguration's indexer cannot tell an absent key from one present with a null value. + // A stray empty chain would therefore look exactly like the end of the outer array and silently + // discard every chain after it - dependencies vanishing without a diagnostic, which is the failure + // mode this feature must never have. The schema forbids chains shorter than two entries (a shorter + // one declares no edge anyway); the single-index lookahead below keeps one authoring slip from + // costing the declarations that follow it. + for (int chainIndex = 0; ; chainIndex++) + { + string? firstOfChain = configuration[$"{root}:chains:{chainIndex}:0"]; + if (firstOfChain is null) + { + if (configuration[$"{root}:chains:{chainIndex + 1}:0"] is null) + { + break; + } + + logger?.SendMessage( + MessageLevel.Warning, + string.Format(CultureInfo.CurrentCulture, Resource.DependencyConfigurationEmptyChain, chainIndex)); + continue; + } + + string previous = firstOfChain; + for (int testIndex = 1; ; testIndex++) + { + if (configuration[$"{root}:chains:{chainIndex}:{testIndex}"] is not { } current) + { + break; + } + + (declarations ??= []).Add(new TestDependencyDeclaration(current, previous, proceedOnFailure: false)); + previous = current; + } + } + + // nodes: [ { "test": "...", "dependsOn": [ "..." ], "proceedOnFailure": true }, ... ] + // + // As for chains, a node missing its "test" key is indistinguishable from the end of the array through + // the indexer, so the same single-index lookahead keeps one malformed entry from silently discarding + // every node after it. + for (int nodeIndex = 0; ; nodeIndex++) + { + string? test = configuration[$"{root}:nodes:{nodeIndex}:test"]; + if (test is null) + { + if (configuration[$"{root}:nodes:{nodeIndex + 1}:test"] is null) + { + break; + } + + logger?.SendMessage( + MessageLevel.Warning, + string.Format(CultureInfo.CurrentCulture, Resource.DependencyConfigurationNodeWithoutTest, nodeIndex)); + continue; + } + + bool proceedOnFailure = false; + if (configuration[$"{root}:nodes:{nodeIndex}:proceedOnFailure"] is { } rawProceedOnFailure) + { + if (bool.TryParse(rawProceedOnFailure, out bool parsed)) + { + proceedOnFailure = parsed; + } + else + { + logger?.SendMessage( + MessageLevel.Warning, + string.Format(CultureInfo.CurrentCulture, Resource.InvalidValue, rawProceedOnFailure, $"{root}:nodes:{nodeIndex}:proceedOnFailure")); + } + } + + bool anyPrerequisite = false; + for (int prerequisiteIndex = 0; ; prerequisiteIndex++) + { + if (configuration[$"{root}:nodes:{nodeIndex}:dependsOn:{prerequisiteIndex}"] is not { } prerequisite) + { + break; + } + + anyPrerequisite = true; + (declarations ??= []).Add(new TestDependencyDeclaration(test, prerequisite, proceedOnFailure)); + } + + if (!anyPrerequisite) + { + logger?.SendMessage( + MessageLevel.Warning, + string.Format(CultureInfo.CurrentCulture, Resource.DependencyConfigurationNodeWithoutPrerequisites, test)); + } + } + + if (declarations is not null) + { + settings.DeclaredDependencies = [.. declarations]; + } + } + private static void ParseTimeoutSetting(IConfiguration configuration, string key, IAdapterMessageLogger? logger, Action setSetting) { if (configuration[$"mstest:{key}"] is not string value) @@ -233,6 +350,7 @@ internal static void SetSettingsFromConfig(IConfiguration configuration, IAdapte ParseBooleanSetting(configuration, "execution:randomizeTestOrder", logger, value => settings.RandomizeTestOrder = value); ParseSignedIntegerSetting(configuration, "execution:randomTestOrderSeed", logger, value => settings.RandomTestOrderSeed = value); + ParseDependencySettings(configuration, logger, settings); ParseCaptureTraceOutputSetting(configuration, "output:captureTrace", logger, value => settings.OutputCaptureMode = value); ParseBooleanSetting(configuration, "parallelism:enabled", logger, value => settings.DisableParallelization = !value); ParseBooleanSetting(configuration, "execution:mapInconclusiveToFailed", logger, value => settings.MapInconclusiveToFailed = value); diff --git a/src/Adapter/MSTestAdapter.PlatformServices/MSTestSettings.cs b/src/Adapter/MSTestAdapter.PlatformServices/MSTestSettings.cs index baf35f9265..1267db0518 100644 --- a/src/Adapter/MSTestAdapter.PlatformServices/MSTestSettings.cs +++ b/src/Adapter/MSTestAdapter.PlatformServices/MSTestSettings.cs @@ -1,6 +1,8 @@ // Copyright (c) Microsoft Corporation. All rights reserved. // Licensed under the MIT license. See LICENSE file in the project root for full license information. +using Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution; + using DebuggerLaunchMode = Microsoft.VisualStudio.TestTools.UnitTesting.DebuggerLaunchMode; using ExecutionScope = Microsoft.VisualStudio.TestTools.UnitTesting.ExecutionScope; @@ -180,6 +182,19 @@ public static RunConfigurationSettings RunConfigurationSettings /// internal bool OrderTestsByNameInClass { get; private set; } + /// + /// Gets the test dependencies declared in testconfig.json under + /// mstest:execution:dependencies, as an alternative to the [DependsOn] attribute, or + /// when none are declared. Configured edges are merged with attribute-declared + /// ones. + /// + /// + /// This is only populated on the Microsoft.Testing.Platform path: testconfig.json is not supplied + /// to the VSTest entry points, which pass a null configuration. Tests running under VSTest declare + /// dependencies with the attribute. + /// + internal TestDependencyDeclaration[]? DeclaredDependencies { get; private set; } + /// /// Gets a value indicating whether tests should be executed in a random order to help expose hidden dependencies between tests. /// diff --git a/src/Adapter/MSTestAdapter.PlatformServices/ObjectModel/TestDependencyInfo.cs b/src/Adapter/MSTestAdapter.PlatformServices/ObjectModel/TestDependencyInfo.cs new file mode 100644 index 0000000000..edae0bc688 --- /dev/null +++ b/src/Adapter/MSTestAdapter.PlatformServices/ObjectModel/TestDependencyInfo.cs @@ -0,0 +1,98 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT license. See LICENSE file in the project root for full license information. + +namespace Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.ObjectModel; + +/// +/// A single declared dependency edge (from a [DependsOn] attribute or from testconfig.json) +/// carried on a : which test must run first, and whether the +/// dependent still runs when that test does not pass. +/// +#if NETFRAMEWORK +[Serializable] +#endif +internal sealed class TestDependencyInfo +{ + /// + /// The separator between the class and method parts of the encoded form. A CLR type full name and a + /// method name can never contain a line feed, so it can never occur inside either field. + /// + private const char FieldSeparator = '\n'; + + public TestDependencyInfo(string? targetClassFullName, string? targetMethodName, bool proceedOnFailure) + { + // Normalize empty to null so that the encoder's "empty means absent" convention and the + // resolver's null checks agree, whatever produced the value (attribute, configuration, decoder). + TargetClassFullName = string.IsNullOrEmpty(targetClassFullName) ? null : targetClassFullName; + TargetMethodName = string.IsNullOrEmpty(targetMethodName) ? null : targetMethodName; + ProceedOnFailure = proceedOnFailure; + } + + /// + /// Gets the full name of the class declaring the prerequisite, or when the + /// prerequisite is a method of the dependent's own class. + /// + public string? TargetClassFullName { get; } + + /// + /// Gets the name of the prerequisite test method, or when the dependency + /// targets every test of . + /// + public string? TargetMethodName { get; } + + /// + /// Gets a value indicating whether the dependent runs even when the prerequisite does not pass. + /// Ordering is enforced either way. + /// + public bool ProceedOnFailure { get; } + + /// + /// Renders the target as a human-readable reference for diagnostics, for example + /// MyNamespace.MyClass.MyMethod, MyNamespace.MyClass.* or MyMethod. + /// + public string DescribeTarget() + => TargetClassFullName is null + ? TargetMethodName ?? "*" + : $"{TargetClassFullName}.{TargetMethodName ?? "*"}"; + + /// + /// Encodes a dependency as a single string for transport across the VSTest TestProperty + /// boundary: a one-character flag for (P when it is set, + /// S - skip - otherwise), then the class name, a line feed, and the method name. An absent + /// class or method is written as the empty string. + /// + /// + /// The flag is a fixed-width prefix, as for , so that a + /// well-formed payload always splits at exactly one position whatever the names contain. The + /// encoding is not self-describing; and are a matched + /// pair over a private adapter property, so only ever sees this method's + /// output. + /// + public static string Encode(TestDependencyInfo info) + => (info.ProceedOnFailure ? "P" : "S") + info.TargetClassFullName + FieldSeparator + info.TargetMethodName; + + /// + /// Decodes a dependency previously produced by . Malformed data fails closed: + /// an unrecognized flag decodes as "skip on failure", which is the conservative behavior, and a + /// payload without a separator is read as a bare method name in the dependent's own class. + /// + public static TestDependencyInfo Decode(string encoded) + { + if (encoded.Length == 0) + { + return new TestDependencyInfo(null, null, false); + } + + // Only consume the first character when it is a flag we actually wrote; otherwise treat the + // whole payload as data so that an unrecognized form still names the same test rather than a + // truncated - and therefore different - one. + bool hasFlag = encoded[0] is 'P' or 'S'; + bool proceedOnFailure = encoded[0] == 'P'; + string payload = hasFlag ? encoded.Substring(1) : encoded; + + int separator = payload.IndexOf(FieldSeparator); + return separator < 0 + ? new TestDependencyInfo(null, payload, proceedOnFailure) + : new TestDependencyInfo(payload.Substring(0, separator), payload.Substring(separator + 1), proceedOnFailure); + } +} diff --git a/src/Adapter/MSTestAdapter.PlatformServices/ObjectModel/UnitTestElement.cs b/src/Adapter/MSTestAdapter.PlatformServices/ObjectModel/UnitTestElement.cs index f38792dfff..16ab9b6e2c 100644 --- a/src/Adapter/MSTestAdapter.PlatformServices/ObjectModel/UnitTestElement.cs +++ b/src/Adapter/MSTestAdapter.PlatformServices/ObjectModel/UnitTestElement.cs @@ -64,6 +64,14 @@ public UnitTestElement(TestMethod testMethod) /// public ResourceLockInfo[]? ResourceLocks { get; set; } + /// + /// Gets or sets the dependencies declared for this test (via [DependsOn] on the method and/or its + /// class, and/or the mstest:execution:dependencies section of testconfig.json), used by the + /// scheduler to run this test after its prerequisites and to skip it when they do not pass. + /// when none are declared. + /// + public TestDependencyInfo[]? Dependencies { get; set; } + #if !WINDOWS_UWP && !WIN_UI /// /// Gets or sets the deployment items for the test method. diff --git a/src/Adapter/MSTestAdapter.PlatformServices/Resources/Resource.resx b/src/Adapter/MSTestAdapter.PlatformServices/Resources/Resource.resx index ae174a7a5d..1ee32f55a2 100644 --- a/src/Adapter/MSTestAdapter.PlatformServices/Resources/Resource.resx +++ b/src/Adapter/MSTestAdapter.PlatformServices/Resources/Resource.resx @@ -387,6 +387,41 @@ but received {4} argument(s), with types '{5}'. Test method {0} was not found. {0} is the test method name. + + The dependency node at index {0} of the 'mstest:execution:dependencies:nodes' configuration section has no 'test' value and is ignored. + {0} is the zero-based index of the node. {Locked="mstest:execution:dependencies:nodes"}{Locked="test"} + + The chain at index {0} of the 'mstest:execution:dependencies:chains' configuration section is empty and declares no dependency. Remove it: a chain needs at least two tests to express an order. + {0} is the zero-based index of the empty chain. {Locked="mstest:execution:dependencies:chains"} + + + The test reference '{0}' in the 'mstest:execution:dependencies' configuration section is not valid. Use 'Namespace.Class.Method', or 'Namespace.Class.*' for every test of a class. + {0} is the invalid reference. {Locked="mstest:execution:dependencies"} + + + The 'mstest:execution:dependencies' configuration section declares dependencies for '{0}', which matches no test in this run. They are ignored. + {0} is the declared test reference. {Locked="mstest:execution:dependencies"} + + + The 'mstest:execution:dependencies' configuration section declares a node for '{0}' with no 'dependsOn' entries, which has no effect. + {0} is the declared test reference. {Locked="mstest:execution:dependencies"}{Locked="dependsOn"} + + + Dependency cycle detected between tests: {0}. The tests in the cycle cannot be ordered and are reported as failed. Remove one of the dependency declarations to break the cycle. + {0} is the cycle rendered as a path, for example 'A > B > A'. + + + Dependencies between the following test classes cannot be scheduled with class-level parallelization because the classes depend on each other: {0}. Their tests are run sequentially, in dependency order, so the declared order is still honored. Use <Scope>MethodLevel</Scope> (or [Parallelize(Scope = ExecutionScope.MethodLevel)]) to schedule each test individually and run them in parallel again. + {0} is a comma-separated list of test class names. {Locked="MethodLevel"}{Locked="[Parallelize(Scope = ExecutionScope.MethodLevel)]"}{Locked="Scope"} + + + Test '{0}' declares a dependency on '{1}', which matches no test in this test source. Dependencies are resolved within a single source, so a target in another test assembly is not supported. The dependency is ignored. + {0} is the fully qualified name of the test declaring the dependency. {1} is the declared target. + + + Test skipped because it depends on '{0}', which did not pass. + {0} is the fully qualified name of the test that did not pass. + Test Parallelization enabled for {0} (Workers: {1}, Scope: {2}) {0} is the test source path. {1} is the worker count. {2} is the parallelization scope. {Locked="Workers"}{Locked="Scope"} diff --git a/src/Adapter/MSTestAdapter.PlatformServices/Resources/xlf/Resource.cs.xlf b/src/Adapter/MSTestAdapter.PlatformServices/Resources/xlf/Resource.cs.xlf index 2f075cd7ac..d0bf85d28d 100644 --- a/src/Adapter/MSTestAdapter.PlatformServices/Resources/xlf/Resource.cs.xlf +++ b/src/Adapter/MSTestAdapter.PlatformServices/Resources/xlf/Resource.cs.xlf @@ -101,6 +101,51 @@ byl však přijat tento počet argumentů: {4} s typy {5}. Trasování ladění: + + The 'mstest:execution:dependencies' configuration section declares dependencies for '{0}', which matches no test in this run. They are ignored. + The 'mstest:execution:dependencies' configuration section declares dependencies for '{0}', which matches no test in this run. They are ignored. + {0} is the declared test reference. {Locked="mstest:execution:dependencies"} + + + The chain at index {0} of the 'mstest:execution:dependencies:chains' configuration section is empty and declares no dependency. Remove it: a chain needs at least two tests to express an order. + The chain at index {0} of the 'mstest:execution:dependencies:chains' configuration section is empty and declares no dependency. Remove it: a chain needs at least two tests to express an order. + {0} is the zero-based index of the empty chain. {Locked="mstest:execution:dependencies:chains"} + + + The test reference '{0}' in the 'mstest:execution:dependencies' configuration section is not valid. Use 'Namespace.Class.Method', or 'Namespace.Class.*' for every test of a class. + The test reference '{0}' in the 'mstest:execution:dependencies' configuration section is not valid. Use 'Namespace.Class.Method', or 'Namespace.Class.*' for every test of a class. + {0} is the invalid reference. {Locked="mstest:execution:dependencies"} + + + The 'mstest:execution:dependencies' configuration section declares a node for '{0}' with no 'dependsOn' entries, which has no effect. + The 'mstest:execution:dependencies' configuration section declares a node for '{0}' with no 'dependsOn' entries, which has no effect. + {0} is the declared test reference. {Locked="mstest:execution:dependencies"}{Locked="dependsOn"} + + + The dependency node at index {0} of the 'mstest:execution:dependencies:nodes' configuration section has no 'test' value and is ignored. + The dependency node at index {0} of the 'mstest:execution:dependencies:nodes' configuration section has no 'test' value and is ignored. + {0} is the zero-based index of the node. {Locked="mstest:execution:dependencies:nodes"}{Locked="test"} + + + Dependencies between the following test classes cannot be scheduled with class-level parallelization because the classes depend on each other: {0}. Their tests are run sequentially, in dependency order, so the declared order is still honored. Use <Scope>MethodLevel</Scope> (or [Parallelize(Scope = ExecutionScope.MethodLevel)]) to schedule each test individually and run them in parallel again. + Dependencies between the following test classes cannot be scheduled with class-level parallelization because the classes depend on each other: {0}. Their tests are run sequentially, in dependency order, so the declared order is still honored. Use <Scope>MethodLevel</Scope> (or [Parallelize(Scope = ExecutionScope.MethodLevel)]) to schedule each test individually and run them in parallel again. + {0} is a comma-separated list of test class names. {Locked="MethodLevel"}{Locked="[Parallelize(Scope = ExecutionScope.MethodLevel)]"}{Locked="Scope"} + + + Dependency cycle detected between tests: {0}. The tests in the cycle cannot be ordered and are reported as failed. Remove one of the dependency declarations to break the cycle. + Dependency cycle detected between tests: {0}. The tests in the cycle cannot be ordered and are reported as failed. Remove one of the dependency declarations to break the cycle. + {0} is the cycle rendered as a path, for example 'A > B > A'. + + + Test skipped because it depends on '{0}', which did not pass. + Test skipped because it depends on '{0}', which did not pass. + {0} is the fully qualified name of the test that did not pass. + + + Test '{0}' declares a dependency on '{1}', which matches no test in this test source. Dependencies are resolved within a single source, so a target in another test assembly is not supported. The dependency is ignored. + Test '{0}' declares a dependency on '{1}', which matches no test in this test source. Dependencies are resolved within a single source, so a target in another test assembly is not supported. The dependency is ignored. + {0} is the fully qualified name of the test declaring the dependency. {1} is the declared target. + Test Run deployment issue: Failed to load the assembly or module '{0}'. Reason: {1} Problém s nasazením testovacího běhu: Sestavení nebo modul '{0}' se nepovedlo načíst. Důvod: {1} diff --git a/src/Adapter/MSTestAdapter.PlatformServices/Resources/xlf/Resource.de.xlf b/src/Adapter/MSTestAdapter.PlatformServices/Resources/xlf/Resource.de.xlf index 84ddbda5aa..9030025033 100644 --- a/src/Adapter/MSTestAdapter.PlatformServices/Resources/xlf/Resource.de.xlf +++ b/src/Adapter/MSTestAdapter.PlatformServices/Resources/xlf/Resource.de.xlf @@ -101,6 +101,51 @@ aber empfing {4} Argument(e) mit den Typen „{5}“. Debugablaufverfolgung: + + The 'mstest:execution:dependencies' configuration section declares dependencies for '{0}', which matches no test in this run. They are ignored. + The 'mstest:execution:dependencies' configuration section declares dependencies for '{0}', which matches no test in this run. They are ignored. + {0} is the declared test reference. {Locked="mstest:execution:dependencies"} + + + The chain at index {0} of the 'mstest:execution:dependencies:chains' configuration section is empty and declares no dependency. Remove it: a chain needs at least two tests to express an order. + The chain at index {0} of the 'mstest:execution:dependencies:chains' configuration section is empty and declares no dependency. Remove it: a chain needs at least two tests to express an order. + {0} is the zero-based index of the empty chain. {Locked="mstest:execution:dependencies:chains"} + + + The test reference '{0}' in the 'mstest:execution:dependencies' configuration section is not valid. Use 'Namespace.Class.Method', or 'Namespace.Class.*' for every test of a class. + The test reference '{0}' in the 'mstest:execution:dependencies' configuration section is not valid. Use 'Namespace.Class.Method', or 'Namespace.Class.*' for every test of a class. + {0} is the invalid reference. {Locked="mstest:execution:dependencies"} + + + The 'mstest:execution:dependencies' configuration section declares a node for '{0}' with no 'dependsOn' entries, which has no effect. + The 'mstest:execution:dependencies' configuration section declares a node for '{0}' with no 'dependsOn' entries, which has no effect. + {0} is the declared test reference. {Locked="mstest:execution:dependencies"}{Locked="dependsOn"} + + + The dependency node at index {0} of the 'mstest:execution:dependencies:nodes' configuration section has no 'test' value and is ignored. + The dependency node at index {0} of the 'mstest:execution:dependencies:nodes' configuration section has no 'test' value and is ignored. + {0} is the zero-based index of the node. {Locked="mstest:execution:dependencies:nodes"}{Locked="test"} + + + Dependencies between the following test classes cannot be scheduled with class-level parallelization because the classes depend on each other: {0}. Their tests are run sequentially, in dependency order, so the declared order is still honored. Use <Scope>MethodLevel</Scope> (or [Parallelize(Scope = ExecutionScope.MethodLevel)]) to schedule each test individually and run them in parallel again. + Dependencies between the following test classes cannot be scheduled with class-level parallelization because the classes depend on each other: {0}. Their tests are run sequentially, in dependency order, so the declared order is still honored. Use <Scope>MethodLevel</Scope> (or [Parallelize(Scope = ExecutionScope.MethodLevel)]) to schedule each test individually and run them in parallel again. + {0} is a comma-separated list of test class names. {Locked="MethodLevel"}{Locked="[Parallelize(Scope = ExecutionScope.MethodLevel)]"}{Locked="Scope"} + + + Dependency cycle detected between tests: {0}. The tests in the cycle cannot be ordered and are reported as failed. Remove one of the dependency declarations to break the cycle. + Dependency cycle detected between tests: {0}. The tests in the cycle cannot be ordered and are reported as failed. Remove one of the dependency declarations to break the cycle. + {0} is the cycle rendered as a path, for example 'A > B > A'. + + + Test skipped because it depends on '{0}', which did not pass. + Test skipped because it depends on '{0}', which did not pass. + {0} is the fully qualified name of the test that did not pass. + + + Test '{0}' declares a dependency on '{1}', which matches no test in this test source. Dependencies are resolved within a single source, so a target in another test assembly is not supported. The dependency is ignored. + Test '{0}' declares a dependency on '{1}', which matches no test in this test source. Dependencies are resolved within a single source, so a target in another test assembly is not supported. The dependency is ignored. + {0} is the fully qualified name of the test declaring the dependency. {1} is the declared target. + Test Run deployment issue: Failed to load the assembly or module '{0}'. Reason: {1} Bereitstellungsproblem beim Testlauf: Die Assembly oder das Modul „{0}“ konnte nicht geladen werden. Grund: {1} diff --git a/src/Adapter/MSTestAdapter.PlatformServices/Resources/xlf/Resource.es.xlf b/src/Adapter/MSTestAdapter.PlatformServices/Resources/xlf/Resource.es.xlf index 9af11178cb..4aaa1a5a7a 100644 --- a/src/Adapter/MSTestAdapter.PlatformServices/Resources/xlf/Resource.es.xlf +++ b/src/Adapter/MSTestAdapter.PlatformServices/Resources/xlf/Resource.es.xlf @@ -101,6 +101,51 @@ pero recibió {4} argumentos, con los tipos '{5}'. Seguimiento de depuración: + + The 'mstest:execution:dependencies' configuration section declares dependencies for '{0}', which matches no test in this run. They are ignored. + The 'mstest:execution:dependencies' configuration section declares dependencies for '{0}', which matches no test in this run. They are ignored. + {0} is the declared test reference. {Locked="mstest:execution:dependencies"} + + + The chain at index {0} of the 'mstest:execution:dependencies:chains' configuration section is empty and declares no dependency. Remove it: a chain needs at least two tests to express an order. + The chain at index {0} of the 'mstest:execution:dependencies:chains' configuration section is empty and declares no dependency. Remove it: a chain needs at least two tests to express an order. + {0} is the zero-based index of the empty chain. {Locked="mstest:execution:dependencies:chains"} + + + The test reference '{0}' in the 'mstest:execution:dependencies' configuration section is not valid. Use 'Namespace.Class.Method', or 'Namespace.Class.*' for every test of a class. + The test reference '{0}' in the 'mstest:execution:dependencies' configuration section is not valid. Use 'Namespace.Class.Method', or 'Namespace.Class.*' for every test of a class. + {0} is the invalid reference. {Locked="mstest:execution:dependencies"} + + + The 'mstest:execution:dependencies' configuration section declares a node for '{0}' with no 'dependsOn' entries, which has no effect. + The 'mstest:execution:dependencies' configuration section declares a node for '{0}' with no 'dependsOn' entries, which has no effect. + {0} is the declared test reference. {Locked="mstest:execution:dependencies"}{Locked="dependsOn"} + + + The dependency node at index {0} of the 'mstest:execution:dependencies:nodes' configuration section has no 'test' value and is ignored. + The dependency node at index {0} of the 'mstest:execution:dependencies:nodes' configuration section has no 'test' value and is ignored. + {0} is the zero-based index of the node. {Locked="mstest:execution:dependencies:nodes"}{Locked="test"} + + + Dependencies between the following test classes cannot be scheduled with class-level parallelization because the classes depend on each other: {0}. Their tests are run sequentially, in dependency order, so the declared order is still honored. Use <Scope>MethodLevel</Scope> (or [Parallelize(Scope = ExecutionScope.MethodLevel)]) to schedule each test individually and run them in parallel again. + Dependencies between the following test classes cannot be scheduled with class-level parallelization because the classes depend on each other: {0}. Their tests are run sequentially, in dependency order, so the declared order is still honored. Use <Scope>MethodLevel</Scope> (or [Parallelize(Scope = ExecutionScope.MethodLevel)]) to schedule each test individually and run them in parallel again. + {0} is a comma-separated list of test class names. {Locked="MethodLevel"}{Locked="[Parallelize(Scope = ExecutionScope.MethodLevel)]"}{Locked="Scope"} + + + Dependency cycle detected between tests: {0}. The tests in the cycle cannot be ordered and are reported as failed. Remove one of the dependency declarations to break the cycle. + Dependency cycle detected between tests: {0}. The tests in the cycle cannot be ordered and are reported as failed. Remove one of the dependency declarations to break the cycle. + {0} is the cycle rendered as a path, for example 'A > B > A'. + + + Test skipped because it depends on '{0}', which did not pass. + Test skipped because it depends on '{0}', which did not pass. + {0} is the fully qualified name of the test that did not pass. + + + Test '{0}' declares a dependency on '{1}', which matches no test in this test source. Dependencies are resolved within a single source, so a target in another test assembly is not supported. The dependency is ignored. + Test '{0}' declares a dependency on '{1}', which matches no test in this test source. Dependencies are resolved within a single source, so a target in another test assembly is not supported. The dependency is ignored. + {0} is the fully qualified name of the test declaring the dependency. {1} is the declared target. + Test Run deployment issue: Failed to load the assembly or module '{0}'. Reason: {1} Problema de implementación de serie de pruebas: no se pudo cargar el ensamblado o módulo "{0}". Motivo: {1} diff --git a/src/Adapter/MSTestAdapter.PlatformServices/Resources/xlf/Resource.fr.xlf b/src/Adapter/MSTestAdapter.PlatformServices/Resources/xlf/Resource.fr.xlf index a78dad5824..68241d24eb 100644 --- a/src/Adapter/MSTestAdapter.PlatformServices/Resources/xlf/Resource.fr.xlf +++ b/src/Adapter/MSTestAdapter.PlatformServices/Resources/xlf/Resource.fr.xlf @@ -101,6 +101,51 @@ mais a reçu {4} argument(s), avec les types « {5} ». Trace du débogage : + + The 'mstest:execution:dependencies' configuration section declares dependencies for '{0}', which matches no test in this run. They are ignored. + The 'mstest:execution:dependencies' configuration section declares dependencies for '{0}', which matches no test in this run. They are ignored. + {0} is the declared test reference. {Locked="mstest:execution:dependencies"} + + + The chain at index {0} of the 'mstest:execution:dependencies:chains' configuration section is empty and declares no dependency. Remove it: a chain needs at least two tests to express an order. + The chain at index {0} of the 'mstest:execution:dependencies:chains' configuration section is empty and declares no dependency. Remove it: a chain needs at least two tests to express an order. + {0} is the zero-based index of the empty chain. {Locked="mstest:execution:dependencies:chains"} + + + The test reference '{0}' in the 'mstest:execution:dependencies' configuration section is not valid. Use 'Namespace.Class.Method', or 'Namespace.Class.*' for every test of a class. + The test reference '{0}' in the 'mstest:execution:dependencies' configuration section is not valid. Use 'Namespace.Class.Method', or 'Namespace.Class.*' for every test of a class. + {0} is the invalid reference. {Locked="mstest:execution:dependencies"} + + + The 'mstest:execution:dependencies' configuration section declares a node for '{0}' with no 'dependsOn' entries, which has no effect. + The 'mstest:execution:dependencies' configuration section declares a node for '{0}' with no 'dependsOn' entries, which has no effect. + {0} is the declared test reference. {Locked="mstest:execution:dependencies"}{Locked="dependsOn"} + + + The dependency node at index {0} of the 'mstest:execution:dependencies:nodes' configuration section has no 'test' value and is ignored. + The dependency node at index {0} of the 'mstest:execution:dependencies:nodes' configuration section has no 'test' value and is ignored. + {0} is the zero-based index of the node. {Locked="mstest:execution:dependencies:nodes"}{Locked="test"} + + + Dependencies between the following test classes cannot be scheduled with class-level parallelization because the classes depend on each other: {0}. Their tests are run sequentially, in dependency order, so the declared order is still honored. Use <Scope>MethodLevel</Scope> (or [Parallelize(Scope = ExecutionScope.MethodLevel)]) to schedule each test individually and run them in parallel again. + Dependencies between the following test classes cannot be scheduled with class-level parallelization because the classes depend on each other: {0}. Their tests are run sequentially, in dependency order, so the declared order is still honored. Use <Scope>MethodLevel</Scope> (or [Parallelize(Scope = ExecutionScope.MethodLevel)]) to schedule each test individually and run them in parallel again. + {0} is a comma-separated list of test class names. {Locked="MethodLevel"}{Locked="[Parallelize(Scope = ExecutionScope.MethodLevel)]"}{Locked="Scope"} + + + Dependency cycle detected between tests: {0}. The tests in the cycle cannot be ordered and are reported as failed. Remove one of the dependency declarations to break the cycle. + Dependency cycle detected between tests: {0}. The tests in the cycle cannot be ordered and are reported as failed. Remove one of the dependency declarations to break the cycle. + {0} is the cycle rendered as a path, for example 'A > B > A'. + + + Test skipped because it depends on '{0}', which did not pass. + Test skipped because it depends on '{0}', which did not pass. + {0} is the fully qualified name of the test that did not pass. + + + Test '{0}' declares a dependency on '{1}', which matches no test in this test source. Dependencies are resolved within a single source, so a target in another test assembly is not supported. The dependency is ignored. + Test '{0}' declares a dependency on '{1}', which matches no test in this test source. Dependencies are resolved within a single source, so a target in another test assembly is not supported. The dependency is ignored. + {0} is the fully qualified name of the test declaring the dependency. {1} is the declared target. + Test Run deployment issue: Failed to load the assembly or module '{0}'. Reason: {1} Problème de déploiement de l’exécution de tests : impossible de charger l’assembly ou le module « {0} ». Raison : {1} diff --git a/src/Adapter/MSTestAdapter.PlatformServices/Resources/xlf/Resource.it.xlf b/src/Adapter/MSTestAdapter.PlatformServices/Resources/xlf/Resource.it.xlf index beef9b386a..811ee725fb 100644 --- a/src/Adapter/MSTestAdapter.PlatformServices/Resources/xlf/Resource.it.xlf +++ b/src/Adapter/MSTestAdapter.PlatformServices/Resources/xlf/Resource.it.xlf @@ -101,6 +101,51 @@ ma ha ricevuto {4} argomenti, con tipi '{5}'. Traccia di debug: + + The 'mstest:execution:dependencies' configuration section declares dependencies for '{0}', which matches no test in this run. They are ignored. + The 'mstest:execution:dependencies' configuration section declares dependencies for '{0}', which matches no test in this run. They are ignored. + {0} is the declared test reference. {Locked="mstest:execution:dependencies"} + + + The chain at index {0} of the 'mstest:execution:dependencies:chains' configuration section is empty and declares no dependency. Remove it: a chain needs at least two tests to express an order. + The chain at index {0} of the 'mstest:execution:dependencies:chains' configuration section is empty and declares no dependency. Remove it: a chain needs at least two tests to express an order. + {0} is the zero-based index of the empty chain. {Locked="mstest:execution:dependencies:chains"} + + + The test reference '{0}' in the 'mstest:execution:dependencies' configuration section is not valid. Use 'Namespace.Class.Method', or 'Namespace.Class.*' for every test of a class. + The test reference '{0}' in the 'mstest:execution:dependencies' configuration section is not valid. Use 'Namespace.Class.Method', or 'Namespace.Class.*' for every test of a class. + {0} is the invalid reference. {Locked="mstest:execution:dependencies"} + + + The 'mstest:execution:dependencies' configuration section declares a node for '{0}' with no 'dependsOn' entries, which has no effect. + The 'mstest:execution:dependencies' configuration section declares a node for '{0}' with no 'dependsOn' entries, which has no effect. + {0} is the declared test reference. {Locked="mstest:execution:dependencies"}{Locked="dependsOn"} + + + The dependency node at index {0} of the 'mstest:execution:dependencies:nodes' configuration section has no 'test' value and is ignored. + The dependency node at index {0} of the 'mstest:execution:dependencies:nodes' configuration section has no 'test' value and is ignored. + {0} is the zero-based index of the node. {Locked="mstest:execution:dependencies:nodes"}{Locked="test"} + + + Dependencies between the following test classes cannot be scheduled with class-level parallelization because the classes depend on each other: {0}. Their tests are run sequentially, in dependency order, so the declared order is still honored. Use <Scope>MethodLevel</Scope> (or [Parallelize(Scope = ExecutionScope.MethodLevel)]) to schedule each test individually and run them in parallel again. + Dependencies between the following test classes cannot be scheduled with class-level parallelization because the classes depend on each other: {0}. Their tests are run sequentially, in dependency order, so the declared order is still honored. Use <Scope>MethodLevel</Scope> (or [Parallelize(Scope = ExecutionScope.MethodLevel)]) to schedule each test individually and run them in parallel again. + {0} is a comma-separated list of test class names. {Locked="MethodLevel"}{Locked="[Parallelize(Scope = ExecutionScope.MethodLevel)]"}{Locked="Scope"} + + + Dependency cycle detected between tests: {0}. The tests in the cycle cannot be ordered and are reported as failed. Remove one of the dependency declarations to break the cycle. + Dependency cycle detected between tests: {0}. The tests in the cycle cannot be ordered and are reported as failed. Remove one of the dependency declarations to break the cycle. + {0} is the cycle rendered as a path, for example 'A > B > A'. + + + Test skipped because it depends on '{0}', which did not pass. + Test skipped because it depends on '{0}', which did not pass. + {0} is the fully qualified name of the test that did not pass. + + + Test '{0}' declares a dependency on '{1}', which matches no test in this test source. Dependencies are resolved within a single source, so a target in another test assembly is not supported. The dependency is ignored. + Test '{0}' declares a dependency on '{1}', which matches no test in this test source. Dependencies are resolved within a single source, so a target in another test assembly is not supported. The dependency is ignored. + {0} is the fully qualified name of the test declaring the dependency. {1} is the declared target. + Test Run deployment issue: Failed to load the assembly or module '{0}'. Reason: {1} Problema di distribuzione durante l'esecuzione dei test: non è possibile caricare l'assembly o il modulo '{0}'. Motivo: {1} diff --git a/src/Adapter/MSTestAdapter.PlatformServices/Resources/xlf/Resource.ja.xlf b/src/Adapter/MSTestAdapter.PlatformServices/Resources/xlf/Resource.ja.xlf index cd34edeb14..983fd6f425 100644 --- a/src/Adapter/MSTestAdapter.PlatformServices/Resources/xlf/Resource.ja.xlf +++ b/src/Adapter/MSTestAdapter.PlatformServices/Resources/xlf/Resource.ja.xlf @@ -102,6 +102,51 @@ but received {4} argument(s), with types '{5}'. デバッグ トレース: + + The 'mstest:execution:dependencies' configuration section declares dependencies for '{0}', which matches no test in this run. They are ignored. + The 'mstest:execution:dependencies' configuration section declares dependencies for '{0}', which matches no test in this run. They are ignored. + {0} is the declared test reference. {Locked="mstest:execution:dependencies"} + + + The chain at index {0} of the 'mstest:execution:dependencies:chains' configuration section is empty and declares no dependency. Remove it: a chain needs at least two tests to express an order. + The chain at index {0} of the 'mstest:execution:dependencies:chains' configuration section is empty and declares no dependency. Remove it: a chain needs at least two tests to express an order. + {0} is the zero-based index of the empty chain. {Locked="mstest:execution:dependencies:chains"} + + + The test reference '{0}' in the 'mstest:execution:dependencies' configuration section is not valid. Use 'Namespace.Class.Method', or 'Namespace.Class.*' for every test of a class. + The test reference '{0}' in the 'mstest:execution:dependencies' configuration section is not valid. Use 'Namespace.Class.Method', or 'Namespace.Class.*' for every test of a class. + {0} is the invalid reference. {Locked="mstest:execution:dependencies"} + + + The 'mstest:execution:dependencies' configuration section declares a node for '{0}' with no 'dependsOn' entries, which has no effect. + The 'mstest:execution:dependencies' configuration section declares a node for '{0}' with no 'dependsOn' entries, which has no effect. + {0} is the declared test reference. {Locked="mstest:execution:dependencies"}{Locked="dependsOn"} + + + The dependency node at index {0} of the 'mstest:execution:dependencies:nodes' configuration section has no 'test' value and is ignored. + The dependency node at index {0} of the 'mstest:execution:dependencies:nodes' configuration section has no 'test' value and is ignored. + {0} is the zero-based index of the node. {Locked="mstest:execution:dependencies:nodes"}{Locked="test"} + + + Dependencies between the following test classes cannot be scheduled with class-level parallelization because the classes depend on each other: {0}. Their tests are run sequentially, in dependency order, so the declared order is still honored. Use <Scope>MethodLevel</Scope> (or [Parallelize(Scope = ExecutionScope.MethodLevel)]) to schedule each test individually and run them in parallel again. + Dependencies between the following test classes cannot be scheduled with class-level parallelization because the classes depend on each other: {0}. Their tests are run sequentially, in dependency order, so the declared order is still honored. Use <Scope>MethodLevel</Scope> (or [Parallelize(Scope = ExecutionScope.MethodLevel)]) to schedule each test individually and run them in parallel again. + {0} is a comma-separated list of test class names. {Locked="MethodLevel"}{Locked="[Parallelize(Scope = ExecutionScope.MethodLevel)]"}{Locked="Scope"} + + + Dependency cycle detected between tests: {0}. The tests in the cycle cannot be ordered and are reported as failed. Remove one of the dependency declarations to break the cycle. + Dependency cycle detected between tests: {0}. The tests in the cycle cannot be ordered and are reported as failed. Remove one of the dependency declarations to break the cycle. + {0} is the cycle rendered as a path, for example 'A > B > A'. + + + Test skipped because it depends on '{0}', which did not pass. + Test skipped because it depends on '{0}', which did not pass. + {0} is the fully qualified name of the test that did not pass. + + + Test '{0}' declares a dependency on '{1}', which matches no test in this test source. Dependencies are resolved within a single source, so a target in another test assembly is not supported. The dependency is ignored. + Test '{0}' declares a dependency on '{1}', which matches no test in this test source. Dependencies are resolved within a single source, so a target in another test assembly is not supported. The dependency is ignored. + {0} is the fully qualified name of the test declaring the dependency. {1} is the declared target. + Test Run deployment issue: Failed to load the assembly or module '{0}'. Reason: {1} テスト実行の展開に関する問題: アセンブリまたはモジュール '{0}' を読み込めませんでした。理由: {1} diff --git a/src/Adapter/MSTestAdapter.PlatformServices/Resources/xlf/Resource.ko.xlf b/src/Adapter/MSTestAdapter.PlatformServices/Resources/xlf/Resource.ko.xlf index 162f6a4113..659a089a71 100644 --- a/src/Adapter/MSTestAdapter.PlatformServices/Resources/xlf/Resource.ko.xlf +++ b/src/Adapter/MSTestAdapter.PlatformServices/Resources/xlf/Resource.ko.xlf @@ -101,6 +101,51 @@ but received {4} argument(s), with types '{5}'. 디버그 추적: + + The 'mstest:execution:dependencies' configuration section declares dependencies for '{0}', which matches no test in this run. They are ignored. + The 'mstest:execution:dependencies' configuration section declares dependencies for '{0}', which matches no test in this run. They are ignored. + {0} is the declared test reference. {Locked="mstest:execution:dependencies"} + + + The chain at index {0} of the 'mstest:execution:dependencies:chains' configuration section is empty and declares no dependency. Remove it: a chain needs at least two tests to express an order. + The chain at index {0} of the 'mstest:execution:dependencies:chains' configuration section is empty and declares no dependency. Remove it: a chain needs at least two tests to express an order. + {0} is the zero-based index of the empty chain. {Locked="mstest:execution:dependencies:chains"} + + + The test reference '{0}' in the 'mstest:execution:dependencies' configuration section is not valid. Use 'Namespace.Class.Method', or 'Namespace.Class.*' for every test of a class. + The test reference '{0}' in the 'mstest:execution:dependencies' configuration section is not valid. Use 'Namespace.Class.Method', or 'Namespace.Class.*' for every test of a class. + {0} is the invalid reference. {Locked="mstest:execution:dependencies"} + + + The 'mstest:execution:dependencies' configuration section declares a node for '{0}' with no 'dependsOn' entries, which has no effect. + The 'mstest:execution:dependencies' configuration section declares a node for '{0}' with no 'dependsOn' entries, which has no effect. + {0} is the declared test reference. {Locked="mstest:execution:dependencies"}{Locked="dependsOn"} + + + The dependency node at index {0} of the 'mstest:execution:dependencies:nodes' configuration section has no 'test' value and is ignored. + The dependency node at index {0} of the 'mstest:execution:dependencies:nodes' configuration section has no 'test' value and is ignored. + {0} is the zero-based index of the node. {Locked="mstest:execution:dependencies:nodes"}{Locked="test"} + + + Dependencies between the following test classes cannot be scheduled with class-level parallelization because the classes depend on each other: {0}. Their tests are run sequentially, in dependency order, so the declared order is still honored. Use <Scope>MethodLevel</Scope> (or [Parallelize(Scope = ExecutionScope.MethodLevel)]) to schedule each test individually and run them in parallel again. + Dependencies between the following test classes cannot be scheduled with class-level parallelization because the classes depend on each other: {0}. Their tests are run sequentially, in dependency order, so the declared order is still honored. Use <Scope>MethodLevel</Scope> (or [Parallelize(Scope = ExecutionScope.MethodLevel)]) to schedule each test individually and run them in parallel again. + {0} is a comma-separated list of test class names. {Locked="MethodLevel"}{Locked="[Parallelize(Scope = ExecutionScope.MethodLevel)]"}{Locked="Scope"} + + + Dependency cycle detected between tests: {0}. The tests in the cycle cannot be ordered and are reported as failed. Remove one of the dependency declarations to break the cycle. + Dependency cycle detected between tests: {0}. The tests in the cycle cannot be ordered and are reported as failed. Remove one of the dependency declarations to break the cycle. + {0} is the cycle rendered as a path, for example 'A > B > A'. + + + Test skipped because it depends on '{0}', which did not pass. + Test skipped because it depends on '{0}', which did not pass. + {0} is the fully qualified name of the test that did not pass. + + + Test '{0}' declares a dependency on '{1}', which matches no test in this test source. Dependencies are resolved within a single source, so a target in another test assembly is not supported. The dependency is ignored. + Test '{0}' declares a dependency on '{1}', which matches no test in this test source. Dependencies are resolved within a single source, so a target in another test assembly is not supported. The dependency is ignored. + {0} is the fully qualified name of the test declaring the dependency. {1} is the declared target. + Test Run deployment issue: Failed to load the assembly or module '{0}'. Reason: {1} 테스트 실행 배포 문제: 어셈블리 또는 모듈 '{0}'을(를) 로드하지 못했습니다. 이유: {1} diff --git a/src/Adapter/MSTestAdapter.PlatformServices/Resources/xlf/Resource.pl.xlf b/src/Adapter/MSTestAdapter.PlatformServices/Resources/xlf/Resource.pl.xlf index d59346023d..a41586b4e7 100644 --- a/src/Adapter/MSTestAdapter.PlatformServices/Resources/xlf/Resource.pl.xlf +++ b/src/Adapter/MSTestAdapter.PlatformServices/Resources/xlf/Resource.pl.xlf @@ -101,6 +101,51 @@ ale odebrał argumenty {4} z typami „{5}”. Ślad debugowania: + + The 'mstest:execution:dependencies' configuration section declares dependencies for '{0}', which matches no test in this run. They are ignored. + The 'mstest:execution:dependencies' configuration section declares dependencies for '{0}', which matches no test in this run. They are ignored. + {0} is the declared test reference. {Locked="mstest:execution:dependencies"} + + + The chain at index {0} of the 'mstest:execution:dependencies:chains' configuration section is empty and declares no dependency. Remove it: a chain needs at least two tests to express an order. + The chain at index {0} of the 'mstest:execution:dependencies:chains' configuration section is empty and declares no dependency. Remove it: a chain needs at least two tests to express an order. + {0} is the zero-based index of the empty chain. {Locked="mstest:execution:dependencies:chains"} + + + The test reference '{0}' in the 'mstest:execution:dependencies' configuration section is not valid. Use 'Namespace.Class.Method', or 'Namespace.Class.*' for every test of a class. + The test reference '{0}' in the 'mstest:execution:dependencies' configuration section is not valid. Use 'Namespace.Class.Method', or 'Namespace.Class.*' for every test of a class. + {0} is the invalid reference. {Locked="mstest:execution:dependencies"} + + + The 'mstest:execution:dependencies' configuration section declares a node for '{0}' with no 'dependsOn' entries, which has no effect. + The 'mstest:execution:dependencies' configuration section declares a node for '{0}' with no 'dependsOn' entries, which has no effect. + {0} is the declared test reference. {Locked="mstest:execution:dependencies"}{Locked="dependsOn"} + + + The dependency node at index {0} of the 'mstest:execution:dependencies:nodes' configuration section has no 'test' value and is ignored. + The dependency node at index {0} of the 'mstest:execution:dependencies:nodes' configuration section has no 'test' value and is ignored. + {0} is the zero-based index of the node. {Locked="mstest:execution:dependencies:nodes"}{Locked="test"} + + + Dependencies between the following test classes cannot be scheduled with class-level parallelization because the classes depend on each other: {0}. Their tests are run sequentially, in dependency order, so the declared order is still honored. Use <Scope>MethodLevel</Scope> (or [Parallelize(Scope = ExecutionScope.MethodLevel)]) to schedule each test individually and run them in parallel again. + Dependencies between the following test classes cannot be scheduled with class-level parallelization because the classes depend on each other: {0}. Their tests are run sequentially, in dependency order, so the declared order is still honored. Use <Scope>MethodLevel</Scope> (or [Parallelize(Scope = ExecutionScope.MethodLevel)]) to schedule each test individually and run them in parallel again. + {0} is a comma-separated list of test class names. {Locked="MethodLevel"}{Locked="[Parallelize(Scope = ExecutionScope.MethodLevel)]"}{Locked="Scope"} + + + Dependency cycle detected between tests: {0}. The tests in the cycle cannot be ordered and are reported as failed. Remove one of the dependency declarations to break the cycle. + Dependency cycle detected between tests: {0}. The tests in the cycle cannot be ordered and are reported as failed. Remove one of the dependency declarations to break the cycle. + {0} is the cycle rendered as a path, for example 'A > B > A'. + + + Test skipped because it depends on '{0}', which did not pass. + Test skipped because it depends on '{0}', which did not pass. + {0} is the fully qualified name of the test that did not pass. + + + Test '{0}' declares a dependency on '{1}', which matches no test in this test source. Dependencies are resolved within a single source, so a target in another test assembly is not supported. The dependency is ignored. + Test '{0}' declares a dependency on '{1}', which matches no test in this test source. Dependencies are resolved within a single source, so a target in another test assembly is not supported. The dependency is ignored. + {0} is the fully qualified name of the test declaring the dependency. {1} is the declared target. + Test Run deployment issue: Failed to load the assembly or module '{0}'. Reason: {1} Problem z wdrożeniem przebiegu testu: nie można załadować zestawu lub modułu „{0}”. Przyczyna: {1} diff --git a/src/Adapter/MSTestAdapter.PlatformServices/Resources/xlf/Resource.pt-BR.xlf b/src/Adapter/MSTestAdapter.PlatformServices/Resources/xlf/Resource.pt-BR.xlf index 06f3f5d34a..b431eee286 100644 --- a/src/Adapter/MSTestAdapter.PlatformServices/Resources/xlf/Resource.pt-BR.xlf +++ b/src/Adapter/MSTestAdapter.PlatformServices/Resources/xlf/Resource.pt-BR.xlf @@ -101,6 +101,51 @@ mas {4} argumentos recebidos, com tipos '{5}'. Rastreamento de Depuração: + + The 'mstest:execution:dependencies' configuration section declares dependencies for '{0}', which matches no test in this run. They are ignored. + The 'mstest:execution:dependencies' configuration section declares dependencies for '{0}', which matches no test in this run. They are ignored. + {0} is the declared test reference. {Locked="mstest:execution:dependencies"} + + + The chain at index {0} of the 'mstest:execution:dependencies:chains' configuration section is empty and declares no dependency. Remove it: a chain needs at least two tests to express an order. + The chain at index {0} of the 'mstest:execution:dependencies:chains' configuration section is empty and declares no dependency. Remove it: a chain needs at least two tests to express an order. + {0} is the zero-based index of the empty chain. {Locked="mstest:execution:dependencies:chains"} + + + The test reference '{0}' in the 'mstest:execution:dependencies' configuration section is not valid. Use 'Namespace.Class.Method', or 'Namespace.Class.*' for every test of a class. + The test reference '{0}' in the 'mstest:execution:dependencies' configuration section is not valid. Use 'Namespace.Class.Method', or 'Namespace.Class.*' for every test of a class. + {0} is the invalid reference. {Locked="mstest:execution:dependencies"} + + + The 'mstest:execution:dependencies' configuration section declares a node for '{0}' with no 'dependsOn' entries, which has no effect. + The 'mstest:execution:dependencies' configuration section declares a node for '{0}' with no 'dependsOn' entries, which has no effect. + {0} is the declared test reference. {Locked="mstest:execution:dependencies"}{Locked="dependsOn"} + + + The dependency node at index {0} of the 'mstest:execution:dependencies:nodes' configuration section has no 'test' value and is ignored. + The dependency node at index {0} of the 'mstest:execution:dependencies:nodes' configuration section has no 'test' value and is ignored. + {0} is the zero-based index of the node. {Locked="mstest:execution:dependencies:nodes"}{Locked="test"} + + + Dependencies between the following test classes cannot be scheduled with class-level parallelization because the classes depend on each other: {0}. Their tests are run sequentially, in dependency order, so the declared order is still honored. Use <Scope>MethodLevel</Scope> (or [Parallelize(Scope = ExecutionScope.MethodLevel)]) to schedule each test individually and run them in parallel again. + Dependencies between the following test classes cannot be scheduled with class-level parallelization because the classes depend on each other: {0}. Their tests are run sequentially, in dependency order, so the declared order is still honored. Use <Scope>MethodLevel</Scope> (or [Parallelize(Scope = ExecutionScope.MethodLevel)]) to schedule each test individually and run them in parallel again. + {0} is a comma-separated list of test class names. {Locked="MethodLevel"}{Locked="[Parallelize(Scope = ExecutionScope.MethodLevel)]"}{Locked="Scope"} + + + Dependency cycle detected between tests: {0}. The tests in the cycle cannot be ordered and are reported as failed. Remove one of the dependency declarations to break the cycle. + Dependency cycle detected between tests: {0}. The tests in the cycle cannot be ordered and are reported as failed. Remove one of the dependency declarations to break the cycle. + {0} is the cycle rendered as a path, for example 'A > B > A'. + + + Test skipped because it depends on '{0}', which did not pass. + Test skipped because it depends on '{0}', which did not pass. + {0} is the fully qualified name of the test that did not pass. + + + Test '{0}' declares a dependency on '{1}', which matches no test in this test source. Dependencies are resolved within a single source, so a target in another test assembly is not supported. The dependency is ignored. + Test '{0}' declares a dependency on '{1}', which matches no test in this test source. Dependencies are resolved within a single source, so a target in another test assembly is not supported. The dependency is ignored. + {0} is the fully qualified name of the test declaring the dependency. {1} is the declared target. + Test Run deployment issue: Failed to load the assembly or module '{0}'. Reason: {1} Problema de implantação de Execução de Teste: falha ao carregar o assembly ou o módulo ''{0}''. Motivo: {1} diff --git a/src/Adapter/MSTestAdapter.PlatformServices/Resources/xlf/Resource.ru.xlf b/src/Adapter/MSTestAdapter.PlatformServices/Resources/xlf/Resource.ru.xlf index 445b79ab1e..a40974c981 100644 --- a/src/Adapter/MSTestAdapter.PlatformServices/Resources/xlf/Resource.ru.xlf +++ b/src/Adapter/MSTestAdapter.PlatformServices/Resources/xlf/Resource.ru.xlf @@ -101,6 +101,51 @@ but received {4} argument(s), with types '{5}'. Трассировка отладки: + + The 'mstest:execution:dependencies' configuration section declares dependencies for '{0}', which matches no test in this run. They are ignored. + The 'mstest:execution:dependencies' configuration section declares dependencies for '{0}', which matches no test in this run. They are ignored. + {0} is the declared test reference. {Locked="mstest:execution:dependencies"} + + + The chain at index {0} of the 'mstest:execution:dependencies:chains' configuration section is empty and declares no dependency. Remove it: a chain needs at least two tests to express an order. + The chain at index {0} of the 'mstest:execution:dependencies:chains' configuration section is empty and declares no dependency. Remove it: a chain needs at least two tests to express an order. + {0} is the zero-based index of the empty chain. {Locked="mstest:execution:dependencies:chains"} + + + The test reference '{0}' in the 'mstest:execution:dependencies' configuration section is not valid. Use 'Namespace.Class.Method', or 'Namespace.Class.*' for every test of a class. + The test reference '{0}' in the 'mstest:execution:dependencies' configuration section is not valid. Use 'Namespace.Class.Method', or 'Namespace.Class.*' for every test of a class. + {0} is the invalid reference. {Locked="mstest:execution:dependencies"} + + + The 'mstest:execution:dependencies' configuration section declares a node for '{0}' with no 'dependsOn' entries, which has no effect. + The 'mstest:execution:dependencies' configuration section declares a node for '{0}' with no 'dependsOn' entries, which has no effect. + {0} is the declared test reference. {Locked="mstest:execution:dependencies"}{Locked="dependsOn"} + + + The dependency node at index {0} of the 'mstest:execution:dependencies:nodes' configuration section has no 'test' value and is ignored. + The dependency node at index {0} of the 'mstest:execution:dependencies:nodes' configuration section has no 'test' value and is ignored. + {0} is the zero-based index of the node. {Locked="mstest:execution:dependencies:nodes"}{Locked="test"} + + + Dependencies between the following test classes cannot be scheduled with class-level parallelization because the classes depend on each other: {0}. Their tests are run sequentially, in dependency order, so the declared order is still honored. Use <Scope>MethodLevel</Scope> (or [Parallelize(Scope = ExecutionScope.MethodLevel)]) to schedule each test individually and run them in parallel again. + Dependencies between the following test classes cannot be scheduled with class-level parallelization because the classes depend on each other: {0}. Their tests are run sequentially, in dependency order, so the declared order is still honored. Use <Scope>MethodLevel</Scope> (or [Parallelize(Scope = ExecutionScope.MethodLevel)]) to schedule each test individually and run them in parallel again. + {0} is a comma-separated list of test class names. {Locked="MethodLevel"}{Locked="[Parallelize(Scope = ExecutionScope.MethodLevel)]"}{Locked="Scope"} + + + Dependency cycle detected between tests: {0}. The tests in the cycle cannot be ordered and are reported as failed. Remove one of the dependency declarations to break the cycle. + Dependency cycle detected between tests: {0}. The tests in the cycle cannot be ordered and are reported as failed. Remove one of the dependency declarations to break the cycle. + {0} is the cycle rendered as a path, for example 'A > B > A'. + + + Test skipped because it depends on '{0}', which did not pass. + Test skipped because it depends on '{0}', which did not pass. + {0} is the fully qualified name of the test that did not pass. + + + Test '{0}' declares a dependency on '{1}', which matches no test in this test source. Dependencies are resolved within a single source, so a target in another test assembly is not supported. The dependency is ignored. + Test '{0}' declares a dependency on '{1}', which matches no test in this test source. Dependencies are resolved within a single source, so a target in another test assembly is not supported. The dependency is ignored. + {0} is the fully qualified name of the test declaring the dependency. {1} is the declared target. + Test Run deployment issue: Failed to load the assembly or module '{0}'. Reason: {1} Проблема развертывания тестового запуска: не удалось загрузить сборку или модуль "{0}". Причина: {1} diff --git a/src/Adapter/MSTestAdapter.PlatformServices/Resources/xlf/Resource.tr.xlf b/src/Adapter/MSTestAdapter.PlatformServices/Resources/xlf/Resource.tr.xlf index 6706b06109..e9cf0aa911 100644 --- a/src/Adapter/MSTestAdapter.PlatformServices/Resources/xlf/Resource.tr.xlf +++ b/src/Adapter/MSTestAdapter.PlatformServices/Resources/xlf/Resource.tr.xlf @@ -101,6 +101,51 @@ ancak, '{5}' türünde {4} bağımsız değişken aldı. Hata Ayıklama İzleyici: + + The 'mstest:execution:dependencies' configuration section declares dependencies for '{0}', which matches no test in this run. They are ignored. + The 'mstest:execution:dependencies' configuration section declares dependencies for '{0}', which matches no test in this run. They are ignored. + {0} is the declared test reference. {Locked="mstest:execution:dependencies"} + + + The chain at index {0} of the 'mstest:execution:dependencies:chains' configuration section is empty and declares no dependency. Remove it: a chain needs at least two tests to express an order. + The chain at index {0} of the 'mstest:execution:dependencies:chains' configuration section is empty and declares no dependency. Remove it: a chain needs at least two tests to express an order. + {0} is the zero-based index of the empty chain. {Locked="mstest:execution:dependencies:chains"} + + + The test reference '{0}' in the 'mstest:execution:dependencies' configuration section is not valid. Use 'Namespace.Class.Method', or 'Namespace.Class.*' for every test of a class. + The test reference '{0}' in the 'mstest:execution:dependencies' configuration section is not valid. Use 'Namespace.Class.Method', or 'Namespace.Class.*' for every test of a class. + {0} is the invalid reference. {Locked="mstest:execution:dependencies"} + + + The 'mstest:execution:dependencies' configuration section declares a node for '{0}' with no 'dependsOn' entries, which has no effect. + The 'mstest:execution:dependencies' configuration section declares a node for '{0}' with no 'dependsOn' entries, which has no effect. + {0} is the declared test reference. {Locked="mstest:execution:dependencies"}{Locked="dependsOn"} + + + The dependency node at index {0} of the 'mstest:execution:dependencies:nodes' configuration section has no 'test' value and is ignored. + The dependency node at index {0} of the 'mstest:execution:dependencies:nodes' configuration section has no 'test' value and is ignored. + {0} is the zero-based index of the node. {Locked="mstest:execution:dependencies:nodes"}{Locked="test"} + + + Dependencies between the following test classes cannot be scheduled with class-level parallelization because the classes depend on each other: {0}. Their tests are run sequentially, in dependency order, so the declared order is still honored. Use <Scope>MethodLevel</Scope> (or [Parallelize(Scope = ExecutionScope.MethodLevel)]) to schedule each test individually and run them in parallel again. + Dependencies between the following test classes cannot be scheduled with class-level parallelization because the classes depend on each other: {0}. Their tests are run sequentially, in dependency order, so the declared order is still honored. Use <Scope>MethodLevel</Scope> (or [Parallelize(Scope = ExecutionScope.MethodLevel)]) to schedule each test individually and run them in parallel again. + {0} is a comma-separated list of test class names. {Locked="MethodLevel"}{Locked="[Parallelize(Scope = ExecutionScope.MethodLevel)]"}{Locked="Scope"} + + + Dependency cycle detected between tests: {0}. The tests in the cycle cannot be ordered and are reported as failed. Remove one of the dependency declarations to break the cycle. + Dependency cycle detected between tests: {0}. The tests in the cycle cannot be ordered and are reported as failed. Remove one of the dependency declarations to break the cycle. + {0} is the cycle rendered as a path, for example 'A > B > A'. + + + Test skipped because it depends on '{0}', which did not pass. + Test skipped because it depends on '{0}', which did not pass. + {0} is the fully qualified name of the test that did not pass. + + + Test '{0}' declares a dependency on '{1}', which matches no test in this test source. Dependencies are resolved within a single source, so a target in another test assembly is not supported. The dependency is ignored. + Test '{0}' declares a dependency on '{1}', which matches no test in this test source. Dependencies are resolved within a single source, so a target in another test assembly is not supported. The dependency is ignored. + {0} is the fully qualified name of the test declaring the dependency. {1} is the declared target. + Test Run deployment issue: Failed to load the assembly or module '{0}'. Reason: {1} Test Çalıştırması dağıtım sorunu: Derleme veya modül '{0}' yüklenemedi. Neden: {1} diff --git a/src/Adapter/MSTestAdapter.PlatformServices/Resources/xlf/Resource.zh-Hans.xlf b/src/Adapter/MSTestAdapter.PlatformServices/Resources/xlf/Resource.zh-Hans.xlf index 42c79f108d..87e59b20c9 100644 --- a/src/Adapter/MSTestAdapter.PlatformServices/Resources/xlf/Resource.zh-Hans.xlf +++ b/src/Adapter/MSTestAdapter.PlatformServices/Resources/xlf/Resource.zh-Hans.xlf @@ -101,6 +101,51 @@ but received {4} argument(s), with types '{5}'. 调试跟踪: + + The 'mstest:execution:dependencies' configuration section declares dependencies for '{0}', which matches no test in this run. They are ignored. + The 'mstest:execution:dependencies' configuration section declares dependencies for '{0}', which matches no test in this run. They are ignored. + {0} is the declared test reference. {Locked="mstest:execution:dependencies"} + + + The chain at index {0} of the 'mstest:execution:dependencies:chains' configuration section is empty and declares no dependency. Remove it: a chain needs at least two tests to express an order. + The chain at index {0} of the 'mstest:execution:dependencies:chains' configuration section is empty and declares no dependency. Remove it: a chain needs at least two tests to express an order. + {0} is the zero-based index of the empty chain. {Locked="mstest:execution:dependencies:chains"} + + + The test reference '{0}' in the 'mstest:execution:dependencies' configuration section is not valid. Use 'Namespace.Class.Method', or 'Namespace.Class.*' for every test of a class. + The test reference '{0}' in the 'mstest:execution:dependencies' configuration section is not valid. Use 'Namespace.Class.Method', or 'Namespace.Class.*' for every test of a class. + {0} is the invalid reference. {Locked="mstest:execution:dependencies"} + + + The 'mstest:execution:dependencies' configuration section declares a node for '{0}' with no 'dependsOn' entries, which has no effect. + The 'mstest:execution:dependencies' configuration section declares a node for '{0}' with no 'dependsOn' entries, which has no effect. + {0} is the declared test reference. {Locked="mstest:execution:dependencies"}{Locked="dependsOn"} + + + The dependency node at index {0} of the 'mstest:execution:dependencies:nodes' configuration section has no 'test' value and is ignored. + The dependency node at index {0} of the 'mstest:execution:dependencies:nodes' configuration section has no 'test' value and is ignored. + {0} is the zero-based index of the node. {Locked="mstest:execution:dependencies:nodes"}{Locked="test"} + + + Dependencies between the following test classes cannot be scheduled with class-level parallelization because the classes depend on each other: {0}. Their tests are run sequentially, in dependency order, so the declared order is still honored. Use <Scope>MethodLevel</Scope> (or [Parallelize(Scope = ExecutionScope.MethodLevel)]) to schedule each test individually and run them in parallel again. + Dependencies between the following test classes cannot be scheduled with class-level parallelization because the classes depend on each other: {0}. Their tests are run sequentially, in dependency order, so the declared order is still honored. Use <Scope>MethodLevel</Scope> (or [Parallelize(Scope = ExecutionScope.MethodLevel)]) to schedule each test individually and run them in parallel again. + {0} is a comma-separated list of test class names. {Locked="MethodLevel"}{Locked="[Parallelize(Scope = ExecutionScope.MethodLevel)]"}{Locked="Scope"} + + + Dependency cycle detected between tests: {0}. The tests in the cycle cannot be ordered and are reported as failed. Remove one of the dependency declarations to break the cycle. + Dependency cycle detected between tests: {0}. The tests in the cycle cannot be ordered and are reported as failed. Remove one of the dependency declarations to break the cycle. + {0} is the cycle rendered as a path, for example 'A > B > A'. + + + Test skipped because it depends on '{0}', which did not pass. + Test skipped because it depends on '{0}', which did not pass. + {0} is the fully qualified name of the test that did not pass. + + + Test '{0}' declares a dependency on '{1}', which matches no test in this test source. Dependencies are resolved within a single source, so a target in another test assembly is not supported. The dependency is ignored. + Test '{0}' declares a dependency on '{1}', which matches no test in this test source. Dependencies are resolved within a single source, so a target in another test assembly is not supported. The dependency is ignored. + {0} is the fully qualified name of the test declaring the dependency. {1} is the declared target. + Test Run deployment issue: Failed to load the assembly or module '{0}'. Reason: {1} 测试运行部署问题：未能加载程序集或模块 "{0}"。原因: {1} diff --git a/src/Adapter/MSTestAdapter.PlatformServices/Resources/xlf/Resource.zh-Hant.xlf b/src/Adapter/MSTestAdapter.PlatformServices/Resources/xlf/Resource.zh-Hant.xlf index 0eb64b852e..0e81c7a4ae 100644 --- a/src/Adapter/MSTestAdapter.PlatformServices/Resources/xlf/Resource.zh-Hant.xlf +++ b/src/Adapter/MSTestAdapter.PlatformServices/Resources/xlf/Resource.zh-Hant.xlf @@ -101,6 +101,51 @@ but received {4} argument(s), with types '{5}'. 偵錯追蹤: + + The 'mstest:execution:dependencies' configuration section declares dependencies for '{0}', which matches no test in this run. They are ignored. + The 'mstest:execution:dependencies' configuration section declares dependencies for '{0}', which matches no test in this run. They are ignored. + {0} is the declared test reference. {Locked="mstest:execution:dependencies"} + + + The chain at index {0} of the 'mstest:execution:dependencies:chains' configuration section is empty and declares no dependency. Remove it: a chain needs at least two tests to express an order. + The chain at index {0} of the 'mstest:execution:dependencies:chains' configuration section is empty and declares no dependency. Remove it: a chain needs at least two tests to express an order. + {0} is the zero-based index of the empty chain. {Locked="mstest:execution:dependencies:chains"} + + + The test reference '{0}' in the 'mstest:execution:dependencies' configuration section is not valid. Use 'Namespace.Class.Method', or 'Namespace.Class.*' for every test of a class. + The test reference '{0}' in the 'mstest:execution:dependencies' configuration section is not valid. Use 'Namespace.Class.Method', or 'Namespace.Class.*' for every test of a class. + {0} is the invalid reference. {Locked="mstest:execution:dependencies"} + + + The 'mstest:execution:dependencies' configuration section declares a node for '{0}' with no 'dependsOn' entries, which has no effect. + The 'mstest:execution:dependencies' configuration section declares a node for '{0}' with no 'dependsOn' entries, which has no effect. + {0} is the declared test reference. {Locked="mstest:execution:dependencies"}{Locked="dependsOn"} + + + The dependency node at index {0} of the 'mstest:execution:dependencies:nodes' configuration section has no 'test' value and is ignored. + The dependency node at index {0} of the 'mstest:execution:dependencies:nodes' configuration section has no 'test' value and is ignored. + {0} is the zero-based index of the node. {Locked="mstest:execution:dependencies:nodes"}{Locked="test"} + + + Dependencies between the following test classes cannot be scheduled with class-level parallelization because the classes depend on each other: {0}. Their tests are run sequentially, in dependency order, so the declared order is still honored. Use <Scope>MethodLevel</Scope> (or [Parallelize(Scope = ExecutionScope.MethodLevel)]) to schedule each test individually and run them in parallel again. + Dependencies between the following test classes cannot be scheduled with class-level parallelization because the classes depend on each other: {0}. Their tests are run sequentially, in dependency order, so the declared order is still honored. Use <Scope>MethodLevel</Scope> (or [Parallelize(Scope = ExecutionScope.MethodLevel)]) to schedule each test individually and run them in parallel again. + {0} is a comma-separated list of test class names. {Locked="MethodLevel"}{Locked="[Parallelize(Scope = ExecutionScope.MethodLevel)]"}{Locked="Scope"} + + + Dependency cycle detected between tests: {0}. The tests in the cycle cannot be ordered and are reported as failed. Remove one of the dependency declarations to break the cycle. + Dependency cycle detected between tests: {0}. The tests in the cycle cannot be ordered and are reported as failed. Remove one of the dependency declarations to break the cycle. + {0} is the cycle rendered as a path, for example 'A > B > A'. + + + Test skipped because it depends on '{0}', which did not pass. + Test skipped because it depends on '{0}', which did not pass. + {0} is the fully qualified name of the test that did not pass. + + + Test '{0}' declares a dependency on '{1}', which matches no test in this test source. Dependencies are resolved within a single source, so a target in another test assembly is not supported. The dependency is ignored. + Test '{0}' declares a dependency on '{1}', which matches no test in this test source. Dependencies are resolved within a single source, so a target in another test assembly is not supported. The dependency is ignored. + {0} is the fully qualified name of the test declaring the dependency. {1} is the declared target. + Test Run deployment issue: Failed to load the assembly or module '{0}'. Reason: {1} 測試執行部署問題: 無法載入組件或模組 '{0}'。原因: {1} diff --git a/src/TestFramework/TestFramework/Attributes/Lifecycle/DependsOnAttribute.cs b/src/TestFramework/TestFramework/Attributes/Lifecycle/DependsOnAttribute.cs new file mode 100644 index 0000000000..5571450fe5 --- /dev/null +++ b/src/TestFramework/TestFramework/Attributes/Lifecycle/DependsOnAttribute.cs @@ -0,0 +1,203 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT license. See LICENSE file in the project root for full license information. + +namespace Microsoft.VisualStudio.TestTools.UnitTesting; + +/// +/// Declares that a test (or every test of a test class) must not start until one or more other tests +/// have finished, forming a dependency graph that the in-assembly scheduler executes in topological +/// order. Unlike a flat ordering attribute, independent branches of the graph still run in parallel. +/// +/// +/// +/// The attribute declares an edge "this test depends on that test". Applying it several times is how +/// a test declares more than one prerequisite (fan-in); several tests naming the same prerequisite is +/// how a graph fans out. Fan-out is the point of the feature: once the shared prerequisite has passed, +/// every dependent becomes runnable at the same time and the scheduler is free to run them +/// concurrently, subject to and worker availability. +/// +/// +/// When applied to a class, the dependency applies to every test in that class. When the target is a +/// type (rather than a specific method), the edge points at every test of that type, +/// so the dependent starts only once all of them have finished. +/// +/// +/// Scope. Dependencies are resolved within a single test source. A target in another +/// assembly cannot be waited on, even though and +/// will happily accept a type from one: such an edge +/// matches nothing and is reported as an ignored dependency rather than silently ordering anything. +/// +/// +/// Failure semantics. If a prerequisite does not pass, the dependent is +/// skipped, not failed, and the skip propagates transitively down the graph. Skipping is the +/// established convention (TestNG's dependsOnMethods, TUnit's [DependsOn], +/// pytest-dependency) and it keeps the signal readable: one root cause is reported as one failure plus +/// a set of clearly-labelled skips, instead of a wall of failures. Set +/// to for a dependent that must run anyway -- +/// typically an audit or cleanup test. +/// +/// +/// Cycles. A dependency cycle is a configuration error and is reported before any +/// test in the assembly runs. Every test in the cycle is failed with a message naming the cycle path. +/// +/// +/// Interaction with parallelization. Ordering is enforced regardless of +/// . Under (the +/// default) the scheduling unit is the whole class, so dependencies between two classes are honored at +/// class granularity; a dependency between two methods of the same class orders them within +/// that class's run. Under every test is scheduled +/// individually, which gives the most parallelism and the finest ordering. If class-level scheduling +/// would require running two classes before each other (class A's test depends on class B's and vice +/// versa), the declared order is still honored - those classes' tests are run sequentially, in +/// dependency order, and a warning names them - but they lose their parallelism; switching to +/// gets it back. +/// +/// +/// A test marked runs in the sequential phase, which happens +/// after the parallel phase. A parallelizable test that depends on such a test is therefore moved into +/// the sequential phase as well (transitively), so that its prerequisite really has run first. +/// +/// +/// Data-driven tests. Naming a test that expands into several test cases (for example +/// via [DataRow]) creates an edge to all of its cases: the dependent waits for every +/// row and is skipped if any row does not pass. Per-row matching is not supported. +/// +/// +/// Inheritance. A test method declared on a base class runs as a test of every derived +/// test class, and the dependency it declares travels with it: the edge is resolved against the +/// derived class, so each derived class gets its own edge between its own copies of the two +/// tests. Dropping the edge there would silently discard the declared ordering in every concrete test +/// class. What Inherited = false opts out of is override chains: a method that overrides a +/// dependent test without re-declaring the attribute has no dependency, because re-pointing a prerequisite +/// onto a method the author rewrote tends to create edges nobody asked for. +/// +/// +/// Test dependencies couple tests together and make it impossible to run a dependent in isolation, so +/// they are a poor fit for unit tests. They exist for multi-step integration and end-to-end suites, +/// where re-establishing expensive state in every test is impractical. +/// +/// +/// +/// +/// [TestClass] +/// public class CheckoutTests +/// { +/// [TestMethod] +/// public void CreateCart() { } +/// +/// // Fan-out: both of these wait for CreateCart, then may run in parallel with each other. +/// [TestMethod] +/// [DependsOn(nameof(CreateCart))] +/// public void AddItem() { } +/// +/// [TestMethod] +/// [DependsOn(nameof(CreateCart))] +/// public void ApplyCoupon() { } +/// +/// // Fan-in: waits for both. +/// [TestMethod] +/// [DependsOn(nameof(AddItem))] +/// [DependsOn(nameof(ApplyCoupon))] +/// public void Checkout() { } +/// +/// // Runs even when its prerequisite failed. +/// [TestMethod] +/// [DependsOn(nameof(Checkout), ProceedOnFailure = true)] +/// public void WriteAuditRecord() { } +/// } +/// +/// +[AttributeUsage(AttributeTargets.Class | AttributeTargets.Method, AllowMultiple = true, Inherited = false)] +public sealed class DependsOnAttribute : Attribute +{ + /// + /// Initializes a new instance of the class declaring a dependency + /// on another test method of the same test class. + /// + /// + /// The name of the prerequisite test method. Use nameof so that renaming the method is + /// caught by the compiler. + /// + /// is . + /// is empty or whitespace. + public DependsOnAttribute(string testMethodName) + { + if (testMethodName is null) + { + throw new ArgumentNullException(nameof(testMethodName)); + } + + if (testMethodName.Trim().Length == 0) + { + throw new ArgumentException(FrameworkMessages.InvalidDependsOnTestMethodName, nameof(testMethodName)); + } + + TestMethodName = testMethodName; + } + + /// + /// Initializes a new instance of the class declaring a dependency + /// on every test of another test class. + /// + /// The prerequisite test class. + /// is . + public DependsOnAttribute(Type testClass) + => TestClass = testClass ?? throw new ArgumentNullException(nameof(testClass)); + + /// + /// Initializes a new instance of the class declaring a dependency + /// on a specific test method of another test class. + /// + /// The test class declaring the prerequisite. + /// + /// The name of the prerequisite test method. Use nameof so that renaming the method is + /// caught by the compiler. + /// + /// + /// or is . + /// + /// is empty or whitespace. + public DependsOnAttribute(Type testClass, string testMethodName) + { + if (testMethodName is null) + { + throw new ArgumentNullException(nameof(testMethodName)); + } + + if (testMethodName.Trim().Length == 0) + { + throw new ArgumentException(FrameworkMessages.InvalidDependsOnTestMethodName, nameof(testMethodName)); + } + + TestClass = testClass ?? throw new ArgumentNullException(nameof(testClass)); + TestMethodName = testMethodName; + } + + /// + /// Gets the test class declaring the prerequisite, or when the prerequisite + /// is a method of the same class as the dependent. + /// + public Type? TestClass { get; } + + /// + /// Gets the name of the prerequisite test method, or when the dependency + /// targets every test of . + /// + public string? TestMethodName { get; } + + /// + /// Gets or sets a value indicating whether the dependent still runs when a prerequisite does not + /// pass. Defaults to , meaning the dependent is skipped. Set it to + /// for a test that must run regardless of its prerequisites' outcome, such + /// as an audit or cleanup test; ordering is still enforced. + /// + /// + /// The flag is evaluated per dependent test rather than per edge: a test that declares several + /// prerequisites runs past a failure only when every one of its declarations sets this to + /// . One declaration left at the default is therefore enough to skip the test + /// when any of its prerequisites does not pass. That conservative direction is deliberate - opting + /// out of waiting on one prerequisite says nothing about the others, and running a test whose + /// remaining preconditions were never established just produces a second, misleading failure. + /// + public bool ProceedOnFailure { get; set; } +} diff --git a/src/TestFramework/TestFramework/PublicAPI/PublicAPI.Unshipped.txt b/src/TestFramework/TestFramework/PublicAPI/PublicAPI.Unshipped.txt index 6f5cbdc45d..50d21a6faf 100644 --- a/src/TestFramework/TestFramework/PublicAPI/PublicAPI.Unshipped.txt +++ b/src/TestFramework/TestFramework/PublicAPI/PublicAPI.Unshipped.txt @@ -1,4 +1,12 @@ #nullable enable +Microsoft.VisualStudio.TestTools.UnitTesting.DependsOnAttribute +Microsoft.VisualStudio.TestTools.UnitTesting.DependsOnAttribute.DependsOnAttribute(System.Type! testClass) -> void +Microsoft.VisualStudio.TestTools.UnitTesting.DependsOnAttribute.DependsOnAttribute(System.Type! testClass, string! testMethodName) -> void +Microsoft.VisualStudio.TestTools.UnitTesting.DependsOnAttribute.DependsOnAttribute(string! testMethodName) -> void +Microsoft.VisualStudio.TestTools.UnitTesting.DependsOnAttribute.ProceedOnFailure.get -> bool +Microsoft.VisualStudio.TestTools.UnitTesting.DependsOnAttribute.ProceedOnFailure.set -> void +Microsoft.VisualStudio.TestTools.UnitTesting.DependsOnAttribute.TestClass.get -> System.Type? +Microsoft.VisualStudio.TestTools.UnitTesting.DependsOnAttribute.TestMethodName.get -> string? Microsoft.VisualStudio.TestTools.UnitTesting.ResourceAccessMode Microsoft.VisualStudio.TestTools.UnitTesting.ResourceAccessMode.Read = 1 -> Microsoft.VisualStudio.TestTools.UnitTesting.ResourceAccessMode Microsoft.VisualStudio.TestTools.UnitTesting.ResourceAccessMode.ReadWrite = 0 -> Microsoft.VisualStudio.TestTools.UnitTesting.ResourceAccessMode diff --git a/src/TestFramework/TestFramework/Resources/FrameworkMessages.resx b/src/TestFramework/TestFramework/Resources/FrameworkMessages.resx index 47b283edea..a7055de58c 100644 --- a/src/TestFramework/TestFramework/Resources/FrameworkMessages.resx +++ b/src/TestFramework/TestFramework/Resources/FrameworkMessages.resx @@ -493,6 +493,9 @@ Actual: {2} Invalid GitHub ticket URL + + The test method name of a [DependsOn] must be a non-empty, non-whitespace string. + The resource key of a [ResourceLock] must be a non-empty, non-whitespace string. An empty key would silently serialize unrelated tests that also use an empty key. diff --git a/src/TestFramework/TestFramework/Resources/xlf/FrameworkMessages.cs.xlf b/src/TestFramework/TestFramework/Resources/xlf/FrameworkMessages.cs.xlf index e878f990b8..d1f9e51601 100644 --- a/src/TestFramework/TestFramework/Resources/xlf/FrameworkMessages.cs.xlf +++ b/src/TestFramework/TestFramework/Resources/xlf/FrameworkMessages.cs.xlf @@ -537,6 +537,11 @@ Skutečnost: {2} Vlastnost TestContext.{0} souvisí s aktuálním testem a není k dispozici během sestavení nebo používání testovacích přípravků tříd. {0} is the {Locked="TestContext"} property name. Do not localize the API prefix {Locked="TestContext."}. + + The test method name of a [DependsOn] must be a non-empty, non-whitespace string. + The test method name of a [DependsOn] must be a non-empty, non-whitespace string. + + Invalid GitHub ticket URL Neplatná adresa URL lístku GitHubu diff --git a/src/TestFramework/TestFramework/Resources/xlf/FrameworkMessages.de.xlf b/src/TestFramework/TestFramework/Resources/xlf/FrameworkMessages.de.xlf index bd8f892cd7..f9d19b283a 100644 --- a/src/TestFramework/TestFramework/Resources/xlf/FrameworkMessages.de.xlf +++ b/src/TestFramework/TestFramework/Resources/xlf/FrameworkMessages.de.xlf @@ -537,6 +537,11 @@ Tatsächlich: {2} Die Eigenschaft „TestContext.{0}“ im Zusammenhang mit dem aktuellen Test steht während Assembly- oder Klassenfixierungen nicht zur Verfügung. {0} is the {Locked="TestContext"} property name. Do not localize the API prefix {Locked="TestContext."}. + + The test method name of a [DependsOn] must be a non-empty, non-whitespace string. + The test method name of a [DependsOn] must be a non-empty, non-whitespace string. + + Invalid GitHub ticket URL Ungültige GitHub-Ticket-URL. diff --git a/src/TestFramework/TestFramework/Resources/xlf/FrameworkMessages.es.xlf b/src/TestFramework/TestFramework/Resources/xlf/FrameworkMessages.es.xlf index ce864b3210..36a08f1982 100644 --- a/src/TestFramework/TestFramework/Resources/xlf/FrameworkMessages.es.xlf +++ b/src/TestFramework/TestFramework/Resources/xlf/FrameworkMessages.es.xlf @@ -537,6 +537,11 @@ Real: {2} La propiedad "TestContext.{0}" está relacionado con la prueba actual y no está disponible durante los accesorios de ensamblado o clase. {0} is the {Locked="TestContext"} property name. Do not localize the API prefix {Locked="TestContext."}. + + The test method name of a [DependsOn] must be a non-empty, non-whitespace string. + The test method name of a [DependsOn] must be a non-empty, non-whitespace string. + + Invalid GitHub ticket URL Dirección URL de vale de GitHub no válida diff --git a/src/TestFramework/TestFramework/Resources/xlf/FrameworkMessages.fr.xlf b/src/TestFramework/TestFramework/Resources/xlf/FrameworkMessages.fr.xlf index 83b4ed0dc6..bde93256c4 100644 --- a/src/TestFramework/TestFramework/Resources/xlf/FrameworkMessages.fr.xlf +++ b/src/TestFramework/TestFramework/Resources/xlf/FrameworkMessages.fr.xlf @@ -537,6 +537,11 @@ Réel : {2} La propriété « TestContext.{0} », liée au test en cours, n’est pas disponible pendant les fixtures d’assembly ou de classe. {0} is the {Locked="TestContext"} property name. Do not localize the API prefix {Locked="TestContext."}. + + The test method name of a [DependsOn] must be a non-empty, non-whitespace string. + The test method name of a [DependsOn] must be a non-empty, non-whitespace string. + + Invalid GitHub ticket URL URL de ticket GitHub non valide diff --git a/src/TestFramework/TestFramework/Resources/xlf/FrameworkMessages.it.xlf b/src/TestFramework/TestFramework/Resources/xlf/FrameworkMessages.it.xlf index f9b30daa1a..120fb70716 100644 --- a/src/TestFramework/TestFramework/Resources/xlf/FrameworkMessages.it.xlf +++ b/src/TestFramework/TestFramework/Resources/xlf/FrameworkMessages.it.xlf @@ -537,6 +537,11 @@ Effettivo: {2} La proprietà 'TestContext.{0}' relativa al test corrente non è disponibile durante le fixture di assembly o classe. {0} is the {Locked="TestContext"} property name. Do not localize the API prefix {Locked="TestContext."}. + + The test method name of a [DependsOn] must be a non-empty, non-whitespace string. + The test method name of a [DependsOn] must be a non-empty, non-whitespace string. + + Invalid GitHub ticket URL L'URL del ticket GitHub non è valido diff --git a/src/TestFramework/TestFramework/Resources/xlf/FrameworkMessages.ja.xlf b/src/TestFramework/TestFramework/Resources/xlf/FrameworkMessages.ja.xlf index a6b6377aa9..71499af994 100644 --- a/src/TestFramework/TestFramework/Resources/xlf/FrameworkMessages.ja.xlf +++ b/src/TestFramework/TestFramework/Resources/xlf/FrameworkMessages.ja.xlf @@ -537,6 +537,11 @@ Actual: {2} プロパティ 'TestContext.{0}' は現在のテストに関連しており、アセンブリまたはクラス フィクスチャの間は使用できません。 {0} is the {Locked="TestContext"} property name. Do not localize the API prefix {Locked="TestContext."}. + + The test method name of a [DependsOn] must be a non-empty, non-whitespace string. + The test method name of a [DependsOn] must be a non-empty, non-whitespace string. + + Invalid GitHub ticket URL GitHub チケット URL が無効です diff --git a/src/TestFramework/TestFramework/Resources/xlf/FrameworkMessages.ko.xlf b/src/TestFramework/TestFramework/Resources/xlf/FrameworkMessages.ko.xlf index d239c8ff12..399a489e11 100644 --- a/src/TestFramework/TestFramework/Resources/xlf/FrameworkMessages.ko.xlf +++ b/src/TestFramework/TestFramework/Resources/xlf/FrameworkMessages.ko.xlf @@ -537,6 +537,11 @@ Actual: {2} 현재 테스트와 관련된 'TestContext.{0}' 속성은 어셈블리나 클래스 픽스처 실행 중에는 사용할 수 없습니다. {0} is the {Locked="TestContext"} property name. Do not localize the API prefix {Locked="TestContext."}. + + The test method name of a [DependsOn] must be a non-empty, non-whitespace string. + The test method name of a [DependsOn] must be a non-empty, non-whitespace string. + + Invalid GitHub ticket URL 잘못된 GitHub 티켓 URL diff --git a/src/TestFramework/TestFramework/Resources/xlf/FrameworkMessages.pl.xlf b/src/TestFramework/TestFramework/Resources/xlf/FrameworkMessages.pl.xlf index b555531574..7df2d75bc6 100644 --- a/src/TestFramework/TestFramework/Resources/xlf/FrameworkMessages.pl.xlf +++ b/src/TestFramework/TestFramework/Resources/xlf/FrameworkMessages.pl.xlf @@ -537,6 +537,11 @@ Rzeczywiste: {2} Właściwość „TestContext.{0}” jest powiązana z bieżącym testem i nie jest dostępna podczas montażu lub konfiguracji klasy. {0} is the {Locked="TestContext"} property name. Do not localize the API prefix {Locked="TestContext."}. + + The test method name of a [DependsOn] must be a non-empty, non-whitespace string. + The test method name of a [DependsOn] must be a non-empty, non-whitespace string. + + Invalid GitHub ticket URL Nieprawidłowy adres URL biletu usługi GitHub diff --git a/src/TestFramework/TestFramework/Resources/xlf/FrameworkMessages.pt-BR.xlf b/src/TestFramework/TestFramework/Resources/xlf/FrameworkMessages.pt-BR.xlf index e9dab6e4b9..b1ee44bf14 100644 --- a/src/TestFramework/TestFramework/Resources/xlf/FrameworkMessages.pt-BR.xlf +++ b/src/TestFramework/TestFramework/Resources/xlf/FrameworkMessages.pt-BR.xlf @@ -537,6 +537,11 @@ Real: {2} A propriedade "TestContext.{0}" está relacionada ao teste atual não está disponível durante os acessórios de assembly ou classe. {0} is the {Locked="TestContext"} property name. Do not localize the API prefix {Locked="TestContext."}. + + The test method name of a [DependsOn] must be a non-empty, non-whitespace string. + The test method name of a [DependsOn] must be a non-empty, non-whitespace string. + + Invalid GitHub ticket URL URL de tíquete do GitHub inválida diff --git a/src/TestFramework/TestFramework/Resources/xlf/FrameworkMessages.ru.xlf b/src/TestFramework/TestFramework/Resources/xlf/FrameworkMessages.ru.xlf index 4aa6ebc61d..4b9a8dda94 100644 --- a/src/TestFramework/TestFramework/Resources/xlf/FrameworkMessages.ru.xlf +++ b/src/TestFramework/TestFramework/Resources/xlf/FrameworkMessages.ru.xlf @@ -537,6 +537,11 @@ Actual: {2} Свойство "TestContext.{0}" связано с текущим тестом и недоступно во время выполнения средств тестирования сборок или классов. {0} is the {Locked="TestContext"} property name. Do not localize the API prefix {Locked="TestContext."}. + + The test method name of a [DependsOn] must be a non-empty, non-whitespace string. + The test method name of a [DependsOn] must be a non-empty, non-whitespace string. + + Invalid GitHub ticket URL Недопустимый URL-адрес билета GitHub diff --git a/src/TestFramework/TestFramework/Resources/xlf/FrameworkMessages.tr.xlf b/src/TestFramework/TestFramework/Resources/xlf/FrameworkMessages.tr.xlf index 731990e868..4cdd76f0b2 100644 --- a/src/TestFramework/TestFramework/Resources/xlf/FrameworkMessages.tr.xlf +++ b/src/TestFramework/TestFramework/Resources/xlf/FrameworkMessages.tr.xlf @@ -537,6 +537,11 @@ Gerçekte olan: {2} 'TestContext.{0}' özelliği, derleme veya sınıf sabitlemeleri sırasında mevcut testle ilişkili değildir. {0} is the {Locked="TestContext"} property name. Do not localize the API prefix {Locked="TestContext."}. + + The test method name of a [DependsOn] must be a non-empty, non-whitespace string. + The test method name of a [DependsOn] must be a non-empty, non-whitespace string. + + Invalid GitHub ticket URL Geçersiz GitHub anahtar URL'si diff --git a/src/TestFramework/TestFramework/Resources/xlf/FrameworkMessages.zh-Hans.xlf b/src/TestFramework/TestFramework/Resources/xlf/FrameworkMessages.zh-Hans.xlf index 35026dba9a..b1acae7099 100644 --- a/src/TestFramework/TestFramework/Resources/xlf/FrameworkMessages.zh-Hans.xlf +++ b/src/TestFramework/TestFramework/Resources/xlf/FrameworkMessages.zh-Hans.xlf @@ -537,6 +537,11 @@ Actual: {2} 属性‘TestContext.{0}’与当前测试相关，在程序集或类固定例程期间不可用。 {0} is the {Locked="TestContext"} property name. Do not localize the API prefix {Locked="TestContext."}. + + The test method name of a [DependsOn] must be a non-empty, non-whitespace string. + The test method name of a [DependsOn] must be a non-empty, non-whitespace string. + + Invalid GitHub ticket URL GitHub 票证 URL 无效 diff --git a/src/TestFramework/TestFramework/Resources/xlf/FrameworkMessages.zh-Hant.xlf b/src/TestFramework/TestFramework/Resources/xlf/FrameworkMessages.zh-Hant.xlf index 30c01831da..249c875e38 100644 --- a/src/TestFramework/TestFramework/Resources/xlf/FrameworkMessages.zh-Hant.xlf +++ b/src/TestFramework/TestFramework/Resources/xlf/FrameworkMessages.zh-Hant.xlf @@ -537,6 +537,11 @@ Actual: {2} 屬性 'TestContext.{0}' 與目前測試相關，無法在組件或類別測試夾具期間使用。 {0} is the {Locked="TestContext"} property name. Do not localize the API prefix {Locked="TestContext."}. + + The test method name of a [DependsOn] must be a non-empty, non-whitespace string. + The test method name of a [DependsOn] must be a non-empty, non-whitespace string. + + Invalid GitHub ticket URL 無效的 GitHub 票證 URL diff --git a/test/IntegrationTests/MSTest.Acceptance.IntegrationTests/TestDependencyExecutionTests.cs b/test/IntegrationTests/MSTest.Acceptance.IntegrationTests/TestDependencyExecutionTests.cs new file mode 100644 index 0000000000..f8d6d5fc74 --- /dev/null +++ b/test/IntegrationTests/MSTest.Acceptance.IntegrationTests/TestDependencyExecutionTests.cs @@ -0,0 +1,512 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT license. See LICENSE file in the project root for full license information. + +using System.Globalization; + +using Microsoft.Testing.Platform.Acceptance.IntegrationTests; +using Microsoft.Testing.TestInfrastructure; + +namespace MSTest.Acceptance.IntegrationTests; + +/// +/// Acceptance tests for [DependsOn] and its testconfig.json equivalent. The generated assets +/// record, for every test body, the millisecond at which it entered and left, which makes both claims +/// checkable end to end: that a dependent never starts before its prerequisite finishes, and - the point of +/// a graph rather than a flat order - that two tests sharing a prerequisite really do overlap. +/// +/// +/// The overlap claim is asserted by a bounded rendezvous inside the asset rather than by comparing elapsed +/// times, so it does not depend on how busy the machine is and this class needs no isolation from the rest +/// of the suite. Each test method also generates and runs its own project, so nothing is shared between them. +/// +[TestClass] +public sealed class TestDependencyExecutionTests : AcceptanceTestBase +{ + private const string GraphProjectName = "TestDependencyGraphTestProject"; + private const string FailureProjectName = "TestDependencyFailureTestProject"; + private const string CycleProjectName = "TestDependencyCycleTestProject"; + private const string TestConfigProjectName = "TestDependencyConfigTestProject"; + + public TestContext TestContext { get; set; } = default!; + + [TestMethod] + [DynamicData(nameof(TargetFrameworks.AllForDynamicData), typeof(TargetFrameworks))] + public async Task DependsOn_RunsPrerequisitesFirst_AndLetsIndependentBranchesOverlap(string tfm) + { + TestHost testHost = AssetFixture.GetTestHost(GraphProjectName, tfm); + + TestHostResult result = await testHost.ExecuteAsync("--output detailed", cancellationToken: TestContext.CancellationToken); + + Assert.AreEqual(0, result.ExitCode, result.StandardOutput); + Assert.Contains("succeeded: 4", result.StandardOutput); + + IReadOnlyDictionary spans = AssetFixture.ReadProbe(GraphProjectName, tfm); + + // Ordering: a dependent may not start before its prerequisite has finished. + Assert.IsGreaterThanOrEqualTo(spans["Root"].Exit, spans["BranchA"].Enter, "BranchA started before Root finished"); + Assert.IsGreaterThanOrEqualTo(spans["Root"].Exit, spans["BranchB"].Enter, "BranchB started before Root finished"); + Assert.IsGreaterThanOrEqualTo(spans["BranchA"].Exit, spans["Join"].Enter, "Join started before BranchA finished"); + Assert.IsGreaterThanOrEqualTo(spans["BranchB"].Exit, spans["Join"].Enter, "Join started before BranchB finished"); + + // Fan-out: the two branches share a prerequisite but not each other, so they must be allowed to run at + // the same time. This is what a dependency graph buys over a flat ordering attribute, which would have + // serialized them. + // + // Asserted by rendezvous rather than by comparing timestamps: each branch signals its own arrival and + // then waits for the other, so the run only completes if both were genuinely dispatched concurrently. + // Inferring overlap from elapsed times would be a race - under CI load the second branch might not + // start before the first one's sleep elapsed, failing a correct scheduler. If the branches are + // serialized the barrier is never satisfied, the waits hit their timeout, and the asset fails the + // tests rather than this assertion having to notice a suspicious ordering. + Assert.Contains("BranchA:rendezvous-ok", result.StandardOutput, "BranchA did not overlap BranchB"); + Assert.Contains("BranchB:rendezvous-ok", result.StandardOutput, "BranchB did not overlap BranchA"); + } + + [TestMethod] + [DynamicData(nameof(TargetFrameworks.AllForDynamicData), typeof(TargetFrameworks))] + public async Task DependsOn_WhenAPrerequisiteFails_SkipsDependentsTransitively_ButHonorsProceedOnFailure(string tfm) + { + TestHost testHost = AssetFixture.GetTestHost(FailureProjectName, tfm); + + TestHostResult result = await testHost.ExecuteAsync("--output detailed", cancellationToken: TestContext.CancellationToken); + + Assert.AreNotEqual(0, result.ExitCode, result.StandardOutput); + + // One real failure, its two (direct and transitive) dependents skipped rather than failed, and the two + // tests that must still run - the ProceedOnFailure audit step and the unrelated test - passing. + Assert.Contains("failed: 1", result.StandardOutput); + Assert.Contains("skipped: 2", result.StandardOutput); + Assert.Contains("succeeded: 2", result.StandardOutput); + + // The skip must say why, otherwise a skipped test is indistinguishable from one that was filtered out. + Assert.Contains("Failing", result.StandardOutput); + } + + [TestMethod] + [DynamicData(nameof(TargetFrameworks.AllForDynamicData), typeof(TargetFrameworks))] + public async Task DependsOn_WhenDependenciesFormACycle_FailsTheTestsInTheCycle_AndStillRunsTheRest(string tfm) + { + TestHost testHost = AssetFixture.GetTestHost(CycleProjectName, tfm); + + TestHostResult result = await testHost.ExecuteAsync("--output detailed", cancellationToken: TestContext.CancellationToken); + + Assert.AreNotEqual(0, result.ExitCode, result.StandardOutput); + + // The cycle is a configuration error, so it is loud: the two tests in it fail with the cycle path, and + // the unrelated test still runs, so one bad declaration does not cost the whole run. + Assert.Contains("failed: 2", result.StandardOutput); + Assert.Contains("succeeded: 1", result.StandardOutput); + Assert.Contains("Dependency cycle detected", result.StandardOutput); + } + + [TestMethod] + [DynamicData(nameof(TargetFrameworks.AllForDynamicData), typeof(TargetFrameworks))] + public async Task DependencyConfiguration_OrdersTestsThatDeclareNoAttribute(string tfm) + { + TestHost testHost = AssetFixture.GetTestHost(TestConfigProjectName, tfm); + + string probePath = Path.Combine(testHost.DirectoryName, "OrderProbe.txt"); + File.Delete(probePath); + + // The asset ships a testconfig.json declaring the order; no test carries an attribute, and the + // classes are declared in source in the reverse of the required order. + string testConfigPath = Path.Combine(testHost.DirectoryName, "pipeline.testconfig.json"); + + TestHostResult result = await testHost.ExecuteAsync($"--config-file \"{testConfigPath}\"", cancellationToken: TestContext.CancellationToken); + + Assert.AreNotEqual(0, result.ExitCode, result.StandardOutput); + + // 'chains' ordered First -> Second -> Third, and 'nodes' hung Verify and Audit off Second. + // Second fails, so the skip propagates to everything downstream that did not opt out: Third (next + // in the chain) and Verify (a node). Audit runs anyway because its node set proceedOnFailure, and + // First already ran. Hence 1 failed, 2 skipped, 2 succeeded. + Assert.Contains("failed: 1", result.StandardOutput); + Assert.Contains("skipped: 2", result.StandardOutput); + Assert.Contains("succeeded: 2", result.StandardOutput); + + // Both skips name the prerequisite that did not pass. + Assert.Contains("skipped Verify", result.StandardOutput); + Assert.Contains("skipped Third", result.StandardOutput); + Assert.Contains("Test skipped because it depends on 'Contoso.Tests.StepTwo.Second', which did not pass.", result.StandardOutput); + + string[] order = [.. File.ReadAllLines(probePath).Where(l => l.Length > 0)]; + + // The classes are declared in source in the reverse of this order and carry no attribute, so only + // the configuration can be responsible for First running before Second. + CollectionAssert.AreEqual(new[] { "First", "Second" }, order.Take(2).ToArray(), string.Join(",", order)); + + // Neither skipped test executed its body. + CollectionAssert.DoesNotContain(order, "Verify"); + CollectionAssert.DoesNotContain(order, "Third"); + + // Audit ran anyway, because its node set proceedOnFailure. + CollectionAssert.Contains(order, "Audit"); + } + + public sealed class TestAssetFixture : ITestAssetFixture + { + private readonly TempDirectory _tempDirectory = new(); + private readonly Dictionary _assets = []; + + public TestHost GetTestHost(string projectName, string tfm) + => TestHost.LocateFrom(_assets[projectName].TargetAssetPath, projectName, tfm); + + /// + /// Reads the enter/exit millisecond of each recorded test body from the probe file written by the + /// generated asset. + /// + public IReadOnlyDictionary ReadProbe(string projectName, string tfm) + { + string probePath = Path.Combine(GetTestHost(projectName, tfm).DirectoryName, "SpanProbe.txt"); + var spans = new Dictionary(StringComparer.Ordinal); + foreach (string line in File.ReadAllLines(probePath)) + { + string[] parts = line.Split(','); + if (parts.Length == 3) + { + spans[parts[0]] = ( + int.Parse(parts[1], CultureInfo.InvariantCulture), + int.Parse(parts[2], CultureInfo.InvariantCulture)); + } + } + + return spans; + } + + public async Task InitializeAsync(CancellationToken cancellationToken) + { + await GenerateAsync(GraphProjectName, GraphSourceCode, cancellationToken).ConfigureAwait(false); + await GenerateAsync(FailureProjectName, FailureSourceCode, cancellationToken).ConfigureAwait(false); + await GenerateAsync(CycleProjectName, CycleSourceCode, cancellationToken).ConfigureAwait(false); + await GenerateAsync(TestConfigProjectName, TestConfigSourceCode, cancellationToken).ConfigureAwait(false); + } + + private async Task GenerateAsync(string projectName, string sourceCode, CancellationToken cancellationToken) + { + string patched = sourceCode + .PatchTargetFrameworks(TargetFrameworks.All) + .PatchCodeWithReplace("$MSTestVersion$", MSTestVersion) + .PatchCodeWithReplace("$ProjectName$", projectName); + TestAsset asset = await TestAsset.GenerateAssetAsync(projectName, patched, _tempDirectory); + await DotnetCli.RunAsync($"build \"{asset.TargetAssetPath}\" -c Release", callerMemberName: projectName, cancellationToken: cancellationToken); + _assets.Add(projectName, asset); + } + + public void Dispose() + { + foreach (TestAsset asset in _assets.Values) + { + asset.Dispose(); + } + + _tempDirectory.Dispose(); + } + + private const string ProjectFile = """ +#file $ProjectName$.csproj + + + + Exe + true + $TargetFrameworks$ + latest + + + + + + + + + +"""; + + private const string GraphSourceCode = ProjectFile + """ +#file SpanProbe.cs +using System; +using System.Collections.Generic; +using System.Diagnostics; +using System.Globalization; +using System.IO; +using System.Text; +using System.Threading; + +using Microsoft.VisualStudio.TestTools.UnitTesting; + +// Records when each test body ran, so the acceptance test can check both the ordering guarantee and the +// overlap that proves independent branches were not serialized. +internal static class SpanProbe +{ + private static readonly Stopwatch s_clock = Stopwatch.StartNew(); + private static readonly object s_gate = new object(); + private static readonly List s_spans = new List(); + + public static void Run(string name, int milliseconds) + { + int enter = (int)s_clock.ElapsedMilliseconds; + Thread.Sleep(milliseconds); + int exit = (int)s_clock.ElapsedMilliseconds; + lock (s_gate) + { + s_spans.Add(string.Format(CultureInfo.InvariantCulture, "{0},{1},{2}", name, enter, exit)); + } + } + + private static readonly Dictionary s_entered = new Dictionary(StringComparer.Ordinal); + + public static void Enter(string name) + { + lock (s_gate) + { + s_entered[name] = (int)s_clock.ElapsedMilliseconds; + } + } + + public static void Exit(string name) + { + lock (s_gate) + { + s_spans.Add(string.Format(CultureInfo.InvariantCulture, "{0},{1},{2}", name, s_entered[name], (int)s_clock.ElapsedMilliseconds)); + } + } + + public static void Flush() + { + lock (s_gate) + { + File.WriteAllText("SpanProbe.txt", string.Join(Environment.NewLine, s_spans)); + } + } +} + +[TestClass] +public sealed class ProbeLifecycle +{ + [AssemblyCleanup] + public static void Flush() => SpanProbe.Flush(); +} + +#file Parallelization.cs +using Microsoft.VisualStudio.TestTools.UnitTesting; + +// MethodLevel makes every test its own scheduling unit, which is what allows the two branches of the fan-out +// to be handed to different workers. Four workers guarantee there is capacity for them to overlap. +[assembly: Parallelize(Workers = 4, Scope = ExecutionScope.MethodLevel)] + +#file GraphTests.cs +using System; +using System.Threading; + +using Microsoft.VisualStudio.TestTools.UnitTesting; + +[TestClass] +public sealed class GraphTests +{ + // The two branches rendezvous instead of sleeping a fixed span: each signals its arrival and waits for + // the other, so concurrent dispatch is proven rather than inferred from timestamps. A generous timeout + // keeps a serialized scheduler from hanging the run - it fails the test instead. + private static readonly CountdownEvent Rendezvous = new(2); + + [TestMethod] + public void Root() => SpanProbe.Run("Root", 300); + + [TestMethod] + [DependsOn(nameof(Root))] + public void BranchA() => RunBranch("BranchA"); + + [TestMethod] + [DependsOn(nameof(Root))] + public void BranchB() => RunBranch("BranchB"); + + [TestMethod] + [DependsOn(nameof(BranchA))] + [DependsOn(nameof(BranchB))] + public void Join() => SpanProbe.Run("Join", 50); + + private static void RunBranch(string name) + { + SpanProbe.Enter(name); + Rendezvous.Signal(); + bool met = Rendezvous.Wait(TimeSpan.FromSeconds(60)); + SpanProbe.Exit(name); + + Assert.IsTrue(met, name + " timed out waiting for the other branch, so the two never ran concurrently."); + Console.WriteLine(name + ":rendezvous-ok"); + } +} +"""; + + private const string FailureSourceCode = ProjectFile + """ +#file Parallelization.cs +using Microsoft.VisualStudio.TestTools.UnitTesting; + +[assembly: Parallelize(Workers = 4, Scope = ExecutionScope.MethodLevel)] + +#file FailureTests.cs +using Microsoft.VisualStudio.TestTools.UnitTesting; + +[TestClass] +public sealed class FailureTests +{ + [TestMethod] + public void Failing() => Assert.Fail("deliberate failure"); + + [TestMethod] + [DependsOn(nameof(Failing))] + public void DirectDependent() { } + + [TestMethod] + [DependsOn(nameof(DirectDependent))] + public void TransitiveDependent() { } + + // Must run even though its prerequisite failed: this is the audit/cleanup escape hatch. + [TestMethod] + [DependsOn(nameof(Failing), ProceedOnFailure = true)] + public void Audit() { } + + [TestMethod] + public void Unrelated() { } +} +"""; + + private const string CycleSourceCode = ProjectFile + """ +#file CycleTests.cs +using Microsoft.VisualStudio.TestTools.UnitTesting; + +[TestClass] +public sealed class CycleTests +{ + [TestMethod] + [DependsOn(nameof(Two))] + public void One() { } + + [TestMethod] + [DependsOn(nameof(One))] + public void Two() { } + + [TestMethod] + public void Unrelated() { } +} +"""; + + private const string TestConfigSourceCode = """ +#file $ProjectName$.csproj + + + + Exe + true + $TargetFrameworks$ + latest + + + + + + + + + + Always + + + + + +#file pipeline.testconfig.json +{ + "mstest": { + "execution": { + "dependencies": { + "chains": [ + [ + "Contoso.Tests.StepOne.First", + "Contoso.Tests.StepTwo.Second", + "Contoso.Tests.StepThree.Third" + ] + ], + "nodes": [ + { + "test": "Contoso.Tests.VerifyStep.Verify", + "dependsOn": [ "Contoso.Tests.StepTwo.Second" ] + }, + { + "test": "Contoso.Tests.AuditStep.Audit", + "dependsOn": [ "Contoso.Tests.StepTwo.Second" ], + "proceedOnFailure": true + } + ] + } + } + } +} + +#file OrderProbe.cs +using System.IO; + +using Microsoft.VisualStudio.TestTools.UnitTesting; + +internal static class OrderProbe +{ + private static readonly object s_gate = new object(); + + public static void Record(string name) + { + lock (s_gate) + { + File.AppendAllText("OrderProbe.txt", name + "\n"); + } + } +} + +#file Steps.cs +using Microsoft.VisualStudio.TestTools.UnitTesting; + +namespace Contoso.Tests; + +// Declared in the reverse of the order the configuration asks for, and carrying no dependency attribute, +// so a correct run can only be explained by testconfig.json. +[TestClass] +public sealed class AuditStep +{ + // Runs even though its prerequisite fails, because its node sets proceedOnFailure. + [TestMethod] + public void Audit() => OrderProbe.Record("Audit"); +} + +[TestClass] +public sealed class VerifyStep +{ + // Must be skipped: same prerequisite, but no proceedOnFailure. + [TestMethod] + public void Verify() => OrderProbe.Record("Verify"); +} + +[TestClass] +public sealed class StepThree +{ + [TestMethod] + public void Third() => OrderProbe.Record("Third"); +} + +[TestClass] +public sealed class StepTwo +{ + [TestMethod] + public void Second() + { + OrderProbe.Record("Second"); + Assert.Fail("deliberate failure to exercise skip propagation through configuration"); + } +} + +[TestClass] +public sealed class StepOne +{ + [TestMethod] + public void First() => OrderProbe.Record("First"); +} +"""; + } +} diff --git a/test/UnitTests/MSTestAdapter.PlatformServices.UnitTests/Discovery/TypeEnumeratorTests.cs b/test/UnitTests/MSTestAdapter.PlatformServices.UnitTests/Discovery/TypeEnumeratorTests.cs index a45bce3a57..9730abf1ce 100644 --- a/test/UnitTests/MSTestAdapter.PlatformServices.UnitTests/Discovery/TypeEnumeratorTests.cs +++ b/test/UnitTests/MSTestAdapter.PlatformServices.UnitTests/Discovery/TypeEnumeratorTests.cs @@ -229,6 +229,101 @@ public void GetTestFromMethodShouldInitiateTestMethodWithCorrectParameters() testElement.TestMethod.AssemblyName.Should().Be("DummyAssemblyName"); } + /// + /// A class-level [DependsOn(nameof(Setup))] is expanded onto every method of the class, including + /// Setup itself. That generated self-edge is dropped, because the declaration plainly means "every + /// *other* test waits for Setup"; keeping it would make the graph report a cycle the user never wrote, + /// fail Setup, and cascade a skip over the whole class. + /// + public void GetTestFromMethodShouldNotMakeAClassLevelDependencyTargetTargetItself() + { + SetupTestClassAndTestMethods(isValidTestClass: true, isValidTestMethod: true); + TypeEnumerator typeEnumerator = GetTypeEnumeratorInstance(typeof(DummyTestClassWithClassLevelDependsOn), "DummyAssemblyName"); + + MSTest.TestAdapter.ObjectModel.UnitTestElement target = typeEnumerator.GetTestFromMethod( + typeof(DummyTestClassWithClassLevelDependsOn).GetMethod(nameof(DummyTestClassWithClassLevelDependsOn.Setup))!, + classDisablesParallelization: false, + _warnings); + + target.Dependencies.Should().BeNull(); + + // Every other method of the class still gets the dependency. + MSTest.TestAdapter.ObjectModel.UnitTestElement dependent = typeEnumerator.GetTestFromMethod( + typeof(DummyTestClassWithClassLevelDependsOn).GetMethod(nameof(DummyTestClassWithClassLevelDependsOn.PlaceOrder))!, + classDisablesParallelization: false, + _warnings); + + dependent.Dependencies.Should().ContainSingle(); + dependent.Dependencies![0].TargetMethodName.Should().Be(nameof(DummyTestClassWithClassLevelDependsOn.Setup)); + } + + /// + /// When the same prerequisite is declared at both class and method scope with different + /// ProceedOnFailure values, the conservative value must win - matching the rule applied across + /// distinct prerequisites. Merging permissively would let a class-level opt-out silently override a + /// method-level default and run a test whose precondition did not hold. + /// + public void GetTestFromMethodShouldMergeDuplicateDependenciesConservatively() + { + SetupTestClassAndTestMethods(isValidTestClass: true, isValidTestMethod: true); + TypeEnumerator typeEnumerator = GetTypeEnumeratorInstance(typeof(DummyTestClassWithConflictingDependsOn), "DummyAssemblyName"); + + MSTest.TestAdapter.ObjectModel.UnitTestElement element = typeEnumerator.GetTestFromMethod( + typeof(DummyTestClassWithConflictingDependsOn).GetMethod(nameof(DummyTestClassWithConflictingDependsOn.PlaceOrder))!, + classDisablesParallelization: false, + _warnings); + + // One edge, because both declarations name the same prerequisite - and it must not proceed on + // failure, because the method-level declaration did not ask to. + element.Dependencies.Should().ContainSingle(); + element.Dependencies![0].TargetMethodName.Should().Be(nameof(DummyTestClassWithConflictingDependsOn.Setup)); + element.Dependencies[0].ProceedOnFailure.Should().BeFalse(); + } + + /// + /// A test method declared on a base class runs as a test of every derived test class, so the dependency + /// it declares has to travel with it: the edge resolves against the derived class, where both + /// the dependent and its prerequisite exist. Dropping it there would silently discard the declared + /// ordering in every concrete test class - the same silent-loss failure mode as an unmatched edge. + /// This is unrelated to Inherited = false, which governs override chains (see the test below). + /// + public void GetTestFromMethodShouldResolveAnInheritedDependencyAgainstTheDerivedClass() + { + SetupTestClassAndTestMethods(isValidTestClass: true, isValidTestMethod: true); + TypeEnumerator typeEnumerator = GetTypeEnumeratorInstance(typeof(DummyTestClassInheritingADependency), "DummyAssemblyName"); + + MSTest.TestAdapter.ObjectModel.UnitTestElement element = typeEnumerator.GetTestFromMethod( + typeof(DummyTestClassInheritingADependency).GetMethod(nameof(DummyTestClassInheritingADependency.PlaceOrder))!, + classDisablesParallelization: false, + _warnings); + + element.Dependencies.Should().ContainSingle(); + + // A null target class is what keeps the edge inside the derived class: it is resolved against the + // dependent's own class, so DerivedA.PlaceOrder waits for DerivedA.Setup rather than the base's. + element.Dependencies![0].TargetClassFullName.Should().BeNull(); + element.Dependencies[0].TargetMethodName.Should().Be(nameof(DummyTestClassBaseDeclaringADependency.Setup)); + element.TestMethod.FullClassName.Should().Be(typeof(DummyTestClassInheritingADependency).FullName); + } + + /// + /// This is what Inherited = false buys: an override that does not re-declare the attribute does + /// not pick up the base method's dependency. Re-pointing a prerequisite onto a method the author + /// rewrote is exactly the unintended edge the attribute opts out of. + /// + public void GetTestFromMethodShouldNotCarryADependencyOntoAnOverrideThatDoesNotRedeclareIt() + { + SetupTestClassAndTestMethods(isValidTestClass: true, isValidTestMethod: true); + TypeEnumerator typeEnumerator = GetTypeEnumeratorInstance(typeof(DummyTestClassOverridingADependentTest), "DummyAssemblyName"); + + MSTest.TestAdapter.ObjectModel.UnitTestElement element = typeEnumerator.GetTestFromMethod( + typeof(DummyTestClassOverridingADependentTest).GetMethod(nameof(DummyTestClassOverridingADependentTest.PlaceOrder))!, + classDisablesParallelization: false, + _warnings); + + element.Dependencies.Should().BeNull(); + } + public void GetTestFromMethodShouldUseClosedFullClassNameAndOpenManagedTypeNameForGenericTypes() { Type closedType = typeof(DummyGenericTestClass); @@ -554,4 +649,63 @@ public void GenericTestMethod() } } +[TestClass] +[DependsOn(nameof(Setup))] +internal class DummyTestClassWithClassLevelDependsOn +{ + [TestMethod] + public void Setup() + { + } + + [TestMethod] + public void PlaceOrder() + { + } +} + +[TestClass] +[DependsOn(nameof(Setup), ProceedOnFailure = true)] +internal class DummyTestClassWithConflictingDependsOn +{ + [TestMethod] + public void Setup() + { + } + + [TestMethod] + [DependsOn(nameof(Setup))] + public void PlaceOrder() + { + } +} + +internal class DummyTestClassBaseDeclaringADependency +{ + [TestMethod] + public void Setup() + { + } + + [TestMethod] + [DependsOn(nameof(Setup))] + public virtual void PlaceOrder() + { + } +} + +[TestClass] +internal class DummyTestClassInheritingADependency : DummyTestClassBaseDeclaringADependency +{ +} + +[TestClass] +internal class DummyTestClassOverridingADependentTest : DummyTestClassBaseDeclaringADependency +{ + [TestMethod] + public override void PlaceOrder() + { + } +} + #endregion diff --git a/test/UnitTests/MSTestAdapter.PlatformServices.UnitTests/Execution/TestDependencyDeclarationTests.cs b/test/UnitTests/MSTestAdapter.PlatformServices.UnitTests/Execution/TestDependencyDeclarationTests.cs new file mode 100644 index 0000000000..229eb52f46 --- /dev/null +++ b/test/UnitTests/MSTestAdapter.PlatformServices.UnitTests/Execution/TestDependencyDeclarationTests.cs @@ -0,0 +1,179 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT license. See LICENSE file in the project root for full license information. + +using AwesomeAssertions; + +using Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution; +using Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.ObjectModel; +using Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.Interface; + +using TestFramework.ForTestingMSTest; + +using MessageLevel = Microsoft.VisualStudio.TestTools.UnitTesting.MessageLevel; + +namespace MSTestAdapter.PlatformServices.UnitTests.Execution; + +/// +/// Tests for the testconfig.json side of test dependencies - the edges declared outside the test +/// source, which have to end up indistinguishable from attribute-declared ones. +/// +public sealed class TestDependencyDeclarationTests : TestContainer +{ + private const string ClassA = "Ns.ClassA"; + + private static UnitTestElement CreateElement(string className, string methodName) + => new(new TestMethod(methodName, className, "DummyAssembly", displayName: null)); + + public void ApplyAll_AttachesTheDeclaredEdgeToTheNamedTest() + { + UnitTestElement[] tests = [CreateElement(ClassA, "Setup"), CreateElement(ClassA, "PlaceOrder")]; + + bool applied = TestDependencyDeclaration.ApplyAll( + [new TestDependencyDeclaration($"{ClassA}.PlaceOrder", $"{ClassA}.Setup", proceedOnFailure: false)], + tests, + adapterMessageLogger: null); + + applied.Should().BeTrue(); + tests[0].Dependencies.Should().BeNull(); + tests[1].Dependencies.Should().ContainSingle(); + tests[1].Dependencies![0].TargetMethodName.Should().Be("Setup"); + } + + public void ApplyAll_WhenAWildcardDependentCoversThePrerequisite_DoesNotMakeItDependOnItself() + { + // "Ns.ClassA.*" expands onto every test of the class, including Setup - the prerequisite itself. The + // declaration means "every test of this class waits for Setup", so that generated self-edge has to be + // dropped, exactly as discovery drops it for a class-level [DependsOn]. Keeping it would report Setup + // as a cycle and skip the whole class. + UnitTestElement[] tests = [CreateElement(ClassA, "Setup"), CreateElement(ClassA, "PlaceOrder")]; + + TestDependencyDeclaration.ApplyAll( + [new TestDependencyDeclaration($"{ClassA}.*", $"{ClassA}.Setup", proceedOnFailure: false)], + tests, + adapterMessageLogger: null); + + tests[0].Dependencies.Should().BeNull(); + tests[1].Dependencies.Should().ContainSingle(); + tests[1].Dependencies![0].TargetMethodName.Should().Be("Setup"); + } + + public void ApplyAll_WhenAWildcardDependentTargetsAnotherClass_KeepsEveryEdge() + { + // The self-edge suppression must not fire across classes: a same-named method elsewhere is a genuine + // prerequisite. + UnitTestElement[] tests = [CreateElement(ClassA, "Setup"), CreateElement(ClassA, "PlaceOrder")]; + + TestDependencyDeclaration.ApplyAll( + [new TestDependencyDeclaration($"{ClassA}.*", "Ns.Other.Setup", proceedOnFailure: false)], + tests, + adapterMessageLogger: null); + + tests[0].Dependencies.Should().ContainSingle(); + tests[0].Dependencies![0].TargetClassFullName.Should().Be("Ns.Other"); + tests[0].Dependencies![0].TargetMethodName.Should().Be("Setup"); + tests[1].Dependencies.Should().ContainSingle(); + tests[1].Dependencies![0].TargetClassFullName.Should().Be("Ns.Other"); + tests[1].Dependencies![0].TargetMethodName.Should().Be("Setup"); + } + + public void ApplyAll_WhenTheDependentMatchesNothing_ReportsNothingApplied() + { + UnitTestElement[] tests = [CreateElement(ClassA, "Setup")]; + + bool applied = TestDependencyDeclaration.ApplyAll( + [new TestDependencyDeclaration("Ns.Missing.Test", $"{ClassA}.Setup", proceedOnFailure: false)], + tests, + adapterMessageLogger: null); + + applied.Should().BeFalse(); + tests[0].Dependencies.Should().BeNull(); + } + + public void ReportUnmatchedDeclarations_JudgesTheDependentAgainstTheWholeRun() + { + // ApplyAll runs once per source, so it cannot tell "names a test in another assembly" from "names + // nothing at all". Diagnosing per source would report every valid declaration as unmatched once for + // each *other* assembly in the run; this pass sees them all at once. + var logger = new RecordingLogger(); + UnitTestElement[] assemblyA = [CreateElement(ClassA, "Setup")]; + UnitTestElement[] assemblyB = [CreateElement("Ns.ClassB", "Other")]; + + TestDependencyDeclaration[] declarations = + [ + new TestDependencyDeclaration($"{ClassA}.Setup", "Ns.ClassB.Other", proceedOnFailure: false), + ]; + + TestDependencyDeclaration.ReportUnmatchedDeclarations(declarations, [.. assemblyA, .. assemblyB], logger); + + logger.Warnings.Should().BeEmpty("the dependent exists in the run, just not in every source"); + } + + public void ReportUnmatchedDeclarations_StillWarnsWhenNothingInTheRunMatches() + { + var logger = new RecordingLogger(); + UnitTestElement[] tests = [CreateElement(ClassA, "Setup")]; + + TestDependencyDeclaration.ReportUnmatchedDeclarations( + [new TestDependencyDeclaration("Ns.Nowhere.Test", $"{ClassA}.Setup", proceedOnFailure: false)], + tests, + logger); + + logger.Warnings.Should().ContainSingle(); + logger.Warnings[0].Should().Contain("Ns.Nowhere.Test"); + } + + public void ReportUnmatchedDeclarations_WarnsOnAMalformedReference() + { + var logger = new RecordingLogger(); + + TestDependencyDeclaration.ReportUnmatchedDeclarations( + [new TestDependencyDeclaration("PlaceOrder", "Setup", proceedOnFailure: false)], + [CreateElement(ClassA, "PlaceOrder")], + logger); + + // Naming the offending reference is the whole value of the diagnostic - a bare count would still + // pass if the message pointed at the wrong one. Both parts of this declaration are malformed; the + // dependent is checked first, so it is the one that must be reported. + logger.Warnings.Should().ContainSingle(); + logger.Warnings[0].Should().Contain("PlaceOrder"); + } + + private sealed class RecordingLogger : IAdapterMessageLogger + { + public List Warnings { get; } = []; + + public void SendMessage(MessageLevel level, string message) + { + if (level == MessageLevel.Warning) + { + Warnings.Add(message); + } + } + } + + public void ApplyAll_WhenAReferenceIsMalformed_SkipsThatDeclaration() + { + // A bare identifier could be a class or a method; guessing would point the edge at the wrong thing. + UnitTestElement[] tests = [CreateElement(ClassA, "Setup"), CreateElement(ClassA, "PlaceOrder")]; + + bool applied = TestDependencyDeclaration.ApplyAll( + [new TestDependencyDeclaration("PlaceOrder", "Setup", proceedOnFailure: false)], + tests, + adapterMessageLogger: null); + + applied.Should().BeFalse(); + tests[1].Dependencies.Should().BeNull(); + } + + public void ApplyAll_PreservesProceedOnFailure() + { + UnitTestElement[] tests = [CreateElement(ClassA, "Setup"), CreateElement(ClassA, "Audit")]; + + TestDependencyDeclaration.ApplyAll( + [new TestDependencyDeclaration($"{ClassA}.Audit", $"{ClassA}.Setup", proceedOnFailure: true)], + tests, + adapterMessageLogger: null); + + tests[1].Dependencies![0].ProceedOnFailure.Should().BeTrue(); + } +} diff --git a/test/UnitTests/MSTestAdapter.PlatformServices.UnitTests/Execution/TestDependencyGraphTests.cs b/test/UnitTests/MSTestAdapter.PlatformServices.UnitTests/Execution/TestDependencyGraphTests.cs new file mode 100644 index 0000000000..5564b7ffb7 --- /dev/null +++ b/test/UnitTests/MSTestAdapter.PlatformServices.UnitTests/Execution/TestDependencyGraphTests.cs @@ -0,0 +1,523 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT license. See LICENSE file in the project root for full license information. + +using AwesomeAssertions; + +using Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution; +using Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.ObjectModel; +using Microsoft.VisualStudio.TestTools.UnitTesting; + +using TestFramework.ForTestingMSTest; + +using ExecutionScope = Microsoft.VisualStudio.TestTools.UnitTesting.ExecutionScope; + +namespace MSTestAdapter.PlatformServices.UnitTests.Execution; + +public sealed class TestDependencyGraphTests : TestContainer +{ + private const string ClassA = "Ns.ClassA"; + private const string ClassB = "Ns.ClassB"; + + private static UnitTestElement CreateElement(string className, string methodName, params TestDependencyInfo[] dependencies) + => new(new TestMethod(methodName, className, "DummyAssembly", displayName: null)) + { + Dependencies = dependencies.Length == 0 ? null : dependencies, + }; + + private static TestDependencyInfo DependsOnMethod(string methodName, bool proceedOnFailure = false) + => new(null, methodName, proceedOnFailure); + + private static TestDependencyInfo DependsOnClass(string className) + => new(className, null, proceedOnFailure: false); + + private static string[] NamesOf(IEnumerable elements) + => [.. elements.Select(e => e.TestMethod.Name)]; + + private static string[] NamesOfBroken(IEnumerable broken) + => [.. broken.Select(b => b.Element.TestMethod.Name)]; + + public void Build_WhenNoTestDeclaresDependencies_ReturnsNull() + { + // The null fast path is what keeps every run that does not use the feature on the code it uses today. + var graph = TestDependencyGraph.Build( + [CreateElement(ClassA, "A"), CreateElement(ClassA, "B")], + ExecutionScope.MethodLevel, + parallelizationEnabled: true); + + graph.Should().BeNull(); + } + + public void Build_OrdersDependentAfterItsPrerequisiteWithinAChunk() + { + // Declared in reverse order on purpose: only the dependency, not the declaration order, may decide. + UnitTestElement[] tests = + [ + CreateElement(ClassA, "Second", DependsOnMethod("First")), + CreateElement(ClassA, "First"), + ]; + + TestDependencyGraph graph = TestDependencyGraph.Build(tests, ExecutionScope.ClassLevel, parallelizationEnabled: true)!; + + graph.ParallelChunks.Should().ContainSingle(); + NamesOf(graph.ParallelChunks[0]).Should().Equal("First", "Second"); + graph.Errors.Should().BeEmpty(); + } + + public void Build_UnderClassLevelScope_MakesTheDependentClassWaitForThePrerequisiteClass() + { + UnitTestElement[] tests = + [ + CreateElement(ClassB, "NeedsA", new TestDependencyInfo(ClassA, "Setup", false)), + CreateElement(ClassA, "Setup"), + ]; + + TestDependencyGraph graph = TestDependencyGraph.Build(tests, ExecutionScope.ClassLevel, parallelizationEnabled: true)!; + + graph.ParallelChunks.Length.Should().Be(2); + int classBChunk = Array.FindIndex(graph.ParallelChunks, c => c[0].TestMethod.FullClassName == ClassB); + int classAChunk = Array.FindIndex(graph.ParallelChunks, c => c[0].TestMethod.FullClassName == ClassA); + + graph.ParallelChunkPrerequisites[classBChunk].Should().Equal(classAChunk); + graph.ParallelChunkPrerequisites[classAChunk].Should().BeEmpty(); + } + + public void Build_WhenTwoTestsShareAPrerequisite_LeavesThemIndependentSoTheyCanRunInParallel() + { + // Fan-out is the whole point of a graph over a flat order: neither branch may wait for the other. + UnitTestElement[] tests = + [ + CreateElement(ClassA, "Root"), + CreateElement(ClassA, "BranchOne", DependsOnMethod("Root")), + CreateElement(ClassA, "BranchTwo", DependsOnMethod("Root")), + ]; + + TestDependencyGraph graph = TestDependencyGraph.Build(tests, ExecutionScope.MethodLevel, parallelizationEnabled: true)!; + + int root = Array.FindIndex(graph.ParallelChunks, c => c[0].TestMethod.Name == "Root"); + int one = Array.FindIndex(graph.ParallelChunks, c => c[0].TestMethod.Name == "BranchOne"); + int two = Array.FindIndex(graph.ParallelChunks, c => c[0].TestMethod.Name == "BranchTwo"); + + graph.ParallelChunkPrerequisites[one].Should().Equal(root); + graph.ParallelChunkPrerequisites[two].Should().Equal(root); + graph.ParallelChunkPrerequisites[one].Should().NotContain(two); + graph.ParallelChunkPrerequisites[two].Should().NotContain(one); + } + + public void Build_WhenDependencyTargetsAWholeClass_WaitsForEveryTestOfThatClass() + { + UnitTestElement[] tests = + [ + CreateElement(ClassB, "Dependent", DependsOnClass(ClassA)), + CreateElement(ClassA, "One"), + CreateElement(ClassA, "Two"), + ]; + + TestDependencyGraph graph = TestDependencyGraph.Build(tests, ExecutionScope.MethodLevel, parallelizationEnabled: true)!; + + graph.TestPrerequisites[0].Should().BeEquivalentTo([1, 2]); + } + + public void Build_WhenAClassDependsOnItself_DoesNotCreateASelfEdge() + { + // A whole-class dependency naturally covers the declaring test too; treating that as a cycle would + // make the common "this class runs after that class" declaration unusable from a base class. + UnitTestElement[] tests = + [ + CreateElement(ClassA, "One", DependsOnClass(ClassA)), + CreateElement(ClassA, "Two", DependsOnClass(ClassA)), + ]; + + TestDependencyGraph graph = TestDependencyGraph.Build(tests, ExecutionScope.MethodLevel, parallelizationEnabled: true)!; + + // Neither test waits for itself... + graph.TestPrerequisites[0].Should().NotContain(0); + graph.TestPrerequisites[1].Should().NotContain(1); + + // ...but the edges to the *other* tests of the class survive. Asserting only the absence above would + // still hold if the suppression had deleted every edge, which is the more likely way to get this + // wrong: over-deleting silently discards the declaration. + graph.TestPrerequisites[0].Should().Equal(1); + graph.TestPrerequisites[1].Should().Equal(0); + + // And because each now waits for the other, this particular declaration is a genuine mutual cycle - + // which is the correct outcome, not something the self-edge suppression should have hidden. + NamesOfBroken(graph.BrokenTests).Should().BeEquivalentTo(["One", "Two"]); + } + + public void Build_WhenDependenciesFormACycle_ReportsItAndMarksTheTestsBroken() + { + UnitTestElement[] tests = + [ + CreateElement(ClassA, "One", DependsOnMethod("Two")), + CreateElement(ClassA, "Two", DependsOnMethod("One")), + ]; + + TestDependencyGraph graph = TestDependencyGraph.Build(tests, ExecutionScope.MethodLevel, parallelizationEnabled: true)!; + + graph.Errors.Should().ContainSingle(); + graph.Errors[0].Should().Contain("One").And.Contain("Two"); + NamesOfBroken(graph.BrokenTests).Should().BeEquivalentTo(["One", "Two"]); + + // Broken tests are reported as failed instead of being scheduled, so they must not appear anywhere. + graph.ParallelChunks.Should().BeEmpty(); + graph.SequentialTests.Should().BeEmpty(); + } + + public void Build_WhenATestDependsOnItselfByName_IsReportedAsACycle() + { + UnitTestElement[] tests = [CreateElement(ClassA, "Loop", DependsOnMethod("Loop"))]; + + TestDependencyGraph graph = TestDependencyGraph.Build(tests, ExecutionScope.MethodLevel, parallelizationEnabled: true)!; + + graph.Errors.Should().ContainSingle(); + NamesOfBroken(graph.BrokenTests).Should().Equal("Loop"); + } + + public void Build_WhenACycleExistsOnlyInTheClassLevelProjection_RunsTheAffectedTestsSequentiallyInOrder() + { + // No test depends on itself, but class A must precede B *and* B must precede A, which cannot hold + // when the class is the scheduling unit. Dropping the ordering would be worse than losing the + // parallelism: the run-time gate cannot tell "has not run yet" from "did not pass", so unordered + // dependents would be skipped nondeterministically while the run still reported success. + UnitTestElement[] tests = + [ + CreateElement(ClassA, "A1"), + CreateElement(ClassA, "A2", new TestDependencyInfo(ClassB, "B1", false)), + CreateElement(ClassB, "B1", new TestDependencyInfo(ClassA, "A1", false)), + ]; + + TestDependencyGraph graph = TestDependencyGraph.Build(tests, ExecutionScope.ClassLevel, parallelizationEnabled: true)!; + + // A recovered downgrade, not a fatal error: the declared order is still honoured, so this must not + // land on Errors, which is what stamps a failure message onto tests caught in a real cycle. + graph.Errors.Should().BeEmpty(); + graph.Warnings.Should().ContainSingle(); + graph.Warnings[0].Should().Contain("MethodLevel"); + + // The test-level graph is sound, so nothing is broken - the tests still run. + graph.BrokenTests.Should().BeEmpty(); + + // They are moved to the sequential phase, where the topological order is honoured exactly. + graph.ParallelChunks.Should().BeEmpty(); + NamesOf(graph.SequentialTests).Should().Equal("A1", "B1", "A2"); + } + + public void Build_WhenCyclesOverlap_MarksEveryMemberOfBothCycles() + { + // A depends on B and C; B on A; C on B. There are two overlapping cycles - A>B>A and C>B>A>C - and + // every one of the three tests is in one. A plain depth-first search reports whichever back edge it + // closes first (A>B>A), finishes B, and then never notices C is cyclic too, leaving C scheduled. Only + // a component-based pass finds all three. + UnitTestElement[] tests = + [ + CreateElement(ClassA, "A", DependsOnMethod("B"), DependsOnMethod("C")), + CreateElement(ClassA, "B", DependsOnMethod("A")), + CreateElement(ClassA, "C", DependsOnMethod("B")), + ]; + + TestDependencyGraph graph = TestDependencyGraph.Build(tests, ExecutionScope.MethodLevel, parallelizationEnabled: true)!; + + NamesOfBroken(graph.BrokenTests).Should().BeEquivalentTo(["A", "B", "C"]); + + // All three are in one strongly connected component, so they share a single description. The message + // renders one real cycle path (A > B > A here), which need not name every member - membership is + // asserted above. Comparing against the single error is what catches a member being handed a + // different, or an unrelated cycle's, description. + graph.Errors.Should().ContainSingle(); + foreach (TestDependencyGraph.BrokenTest broken in graph.BrokenTests) + { + broken.CycleMessage.Should().Be(graph.Errors[0]); + } + + // Nothing cyclic may be scheduled. + graph.ParallelChunks.Should().BeEmpty(); + graph.SequentialTests.Should().BeEmpty(); + } + + public void Build_WhenCyclesOverlap_ReportsOnlyDeclaredEdgesInThePath() + { + // The component is {A, B, C}, but B has no edge to C. Rendering the component as a path could claim + // "A > B > C > A" and tell the user to remove a B -> C edge that was never declared. Every arrow in + // the message has to correspond to a real prerequisite. + UnitTestElement[] tests = + [ + CreateElement(ClassA, "A", DependsOnMethod("B"), DependsOnMethod("C")), + CreateElement(ClassA, "B", DependsOnMethod("A")), + CreateElement(ClassA, "C", DependsOnMethod("B")), + ]; + + TestDependencyGraph graph = TestDependencyGraph.Build(tests, ExecutionScope.MethodLevel, parallelizationEnabled: true)!; + + graph.Errors.Should().ContainSingle(); + + // Declared edges, as "dependent -> prerequisite" pairs. + var declared = new HashSet { "A>B", "A>C", "B>A", "C>B" }; + + string path = graph.Errors[0]; + int start = path.IndexOf(ClassA + ".", StringComparison.Ordinal); + string[] hops = [.. path.Substring(start).Split([" > "], StringSplitOptions.None) + .Select(h => h.Trim().Replace(ClassA + ".", string.Empty)) + .Select(h => new string([.. h.TakeWhile(char.IsLetterOrDigit)]))]; + + hops.Length.Should().BeGreaterThan(2, "a cycle path needs at least two hops plus the closing repeat"); + hops[0].Should().Be(hops[^1], "the path must close on the test it started from"); + + for (int i = 0; i < hops.Length - 1; i++) + { + declared.Should().Contain($"{hops[i]}>{hops[i + 1]}", $"the path claims a {hops[i]} -> {hops[i + 1]} edge"); + } + } + + public void Build_WhenTwoDisjointCyclesExist_ReportsEachFailureAgainstItsOwnCycle() + { + // Joining every cycle description onto every failure would tell whoever is reading One's failure + // about a cycle it has nothing to do with - the same conflation the class-level notice was moved out + // of Errors to avoid. + UnitTestElement[] tests = + [ + CreateElement(ClassA, "One", DependsOnMethod("Two")), + CreateElement(ClassA, "Two", DependsOnMethod("One")), + CreateElement(ClassB, "Three", new TestDependencyInfo(ClassB, "Four", false)), + CreateElement(ClassB, "Four", new TestDependencyInfo(ClassB, "Three", false)), + ]; + + TestDependencyGraph graph = TestDependencyGraph.Build(tests, ExecutionScope.MethodLevel, parallelizationEnabled: true)!; + + graph.Errors.Count.Should().Be(2); + NamesOfBroken(graph.BrokenTests).Should().BeEquivalentTo(["One", "Two", "Three", "Four"]); + + foreach (TestDependencyGraph.BrokenTest broken in graph.BrokenTests) + { + string name = broken.Element.TestMethod.Name; + if (name is "One" or "Two") + { + broken.CycleMessage.Should().Contain("One").And.Contain("Two"); + broken.CycleMessage.Should().NotContain("Three").And.NotContain("Four"); + } + else + { + broken.CycleMessage.Should().Contain("Three").And.Contain("Four"); + broken.CycleMessage.Should().NotContain("One").And.NotContain("Two"); + } + } + } + + public void Build_WhenAClassIsMerelyDownstreamOfAProjectionCycle_IsDemotedButNotNamedAsCyclic() + { + // Kahn cannot settle a chunk that is downstream of a cycle either, but saying it "depends on each + // other" with the cycle members would be false. ClassC only depends on the cycle; it must still be + // demoted, and must not appear in the warning. + UnitTestElement[] tests = + [ + CreateElement(ClassA, "A1"), + CreateElement(ClassA, "A2", new TestDependencyInfo(ClassB, "B1", false)), + CreateElement(ClassB, "B1", new TestDependencyInfo(ClassA, "A1", false)), + CreateElement("Ns.ClassC", "C1", new TestDependencyInfo(ClassB, "B1", false)), + ]; + + TestDependencyGraph graph = TestDependencyGraph.Build(tests, ExecutionScope.ClassLevel, parallelizationEnabled: true)!; + + graph.Warnings.Should().ContainSingle(); + graph.Warnings[0].Should().Contain(ClassA).And.Contain(ClassB); + graph.Warnings[0].Should().NotContain("ClassC"); + + // Demoted all the same - it cannot run in the parallel phase once its prerequisite left it. + NamesOf(graph.SequentialTests).Should().Contain("C1"); + } + + public void Build_WhenAClassLinksTwoProjectionCycles_IsDemotedButNotNamedAsCyclic() + { + // ClassX sits on a one-way path from one mutual pair to another: ClassA depends on it, and it + // depends on ClassC. Nothing returns to ClassX, so it is in neither cycle - but it does have a + // blocked dependent (ClassA), which is why narrowing the blocked set by peeling chunks that nothing + // blocked depends on cannot isolate cycle membership. Both cycles keep every one of their chunks + // from ever being peeled, so ClassX would survive and be named as mutually dependent. + UnitTestElement[] tests = + [ + CreateElement(ClassA, "A1"), + CreateElement(ClassA, "A2", new TestDependencyInfo(ClassB, "B1", false)), + CreateElement(ClassA, "A3", new TestDependencyInfo("Ns.ClassX", "X1", false)), + CreateElement(ClassB, "B1", new TestDependencyInfo(ClassA, "A1", false)), + CreateElement("Ns.ClassX", "X1", new TestDependencyInfo("Ns.ClassC", "C1", false)), + CreateElement("Ns.ClassC", "C1"), + CreateElement("Ns.ClassC", "C2", new TestDependencyInfo("Ns.ClassD", "D1", false)), + CreateElement("Ns.ClassD", "D1", new TestDependencyInfo("Ns.ClassC", "C1", false)), + ]; + + TestDependencyGraph graph = TestDependencyGraph.Build(tests, ExecutionScope.ClassLevel, parallelizationEnabled: true)!; + + // No test-level cycle exists here - every cycle is created by the class-level projection. + graph.Errors.Should().BeEmpty(); + + graph.Warnings.Should().ContainSingle(); + graph.Warnings[0].Should().Contain(ClassA).And.Contain(ClassB).And.Contain("ClassC").And.Contain("ClassD"); + graph.Warnings[0].Should().NotContain("ClassX"); + + // Demoted all the same: it cannot run in the parallel phase while its prerequisite chunk is + // unschedulable there. + NamesOf(graph.SequentialTests).Should().Contain("X1"); + } + + public void Build_WhenAProjectionCycleIsDemoted_LeavesUnrelatedTestsInTheParallelPhase() + { + // Only the classes caught in the projection cycle lose their parallelism; everything else keeps it. + UnitTestElement[] tests = + [ + CreateElement(ClassA, "A1"), + CreateElement(ClassA, "A2", new TestDependencyInfo(ClassB, "B1", false)), + CreateElement(ClassB, "B1", new TestDependencyInfo(ClassA, "A1", false)), + CreateElement("Ns.ClassC", "C1"), + ]; + + TestDependencyGraph graph = TestDependencyGraph.Build(tests, ExecutionScope.ClassLevel, parallelizationEnabled: true)!; + + graph.ParallelChunks.Should().ContainSingle(); + NamesOf(graph.ParallelChunks[0]).Should().Equal("C1"); + NamesOf(graph.SequentialTests).Should().Equal("A1", "B1", "A2"); + } + + public void Build_WhenTheSameGraphUsesMethodLevelScope_HasNoProjectionCycle() + { + UnitTestElement[] tests = + [ + CreateElement(ClassA, "A1"), + CreateElement(ClassA, "A2", new TestDependencyInfo(ClassB, "B1", false)), + CreateElement(ClassB, "B1", new TestDependencyInfo(ClassA, "A1", false)), + ]; + + TestDependencyGraph graph = TestDependencyGraph.Build(tests, ExecutionScope.MethodLevel, parallelizationEnabled: true)!; + + graph.Errors.Should().BeEmpty(); + graph.Warnings.Should().BeEmpty(); + + // Asserting the absence of a diagnostic is not enough: it would still hold if the graph came back + // with the wrong chunks. Under MethodLevel each test is its own chunk, so the same declarations that + // deadlocked the class-level projection schedule cleanly, with every test still in the parallel phase + // and the real edges preserved. + graph.ParallelChunks.Length.Should().Be(3); + graph.SequentialTests.Should().BeEmpty(); + graph.BrokenTests.Should().BeEmpty(); + + int a1 = Array.FindIndex(graph.ParallelChunks, c => c[0].TestMethod.Name == "A1"); + int a2 = Array.FindIndex(graph.ParallelChunks, c => c[0].TestMethod.Name == "A2"); + int b1 = Array.FindIndex(graph.ParallelChunks, c => c[0].TestMethod.Name == "B1"); + + graph.ParallelChunkPrerequisites[a1].Should().BeEmpty(); + graph.ParallelChunkPrerequisites[b1].Should().Equal(a1); + graph.ParallelChunkPrerequisites[a2].Should().Equal(b1); + } + + public void Build_WhenAPrerequisiteIsNotParallelizable_DemotesItsDependentsToTheSequentialPhase() + { + // The sequential phase runs after the parallel one, so a parallel test waiting on a sequential + // prerequisite could never observe it complete. The dependent moves instead. + UnitTestElement prerequisite = CreateElement(ClassA, "Serial"); + prerequisite.DoNotParallelize = true; + + UnitTestElement[] tests = + [ + prerequisite, + CreateElement(ClassA, "Direct", DependsOnMethod("Serial")), + CreateElement(ClassB, "Transitive", new TestDependencyInfo(ClassA, "Direct", false)), + CreateElement(ClassB, "Unrelated"), + ]; + + TestDependencyGraph graph = TestDependencyGraph.Build(tests, ExecutionScope.MethodLevel, parallelizationEnabled: true)!; + + NamesOf(graph.SequentialTests).Should().Equal("Serial", "Direct", "Transitive"); + graph.ParallelChunks.Should().ContainSingle(); + graph.ParallelChunks[0][0].TestMethod.Name.Should().Be("Unrelated"); + } + + public void Build_WhenParallelizationIsDisabled_PutsEveryTestInTheSequentialPhaseInDependencyOrder() + { + UnitTestElement[] tests = + [ + CreateElement(ClassA, "Last", DependsOnMethod("Middle")), + CreateElement(ClassA, "Middle", DependsOnMethod("First")), + CreateElement(ClassA, "First"), + ]; + + TestDependencyGraph graph = TestDependencyGraph.Build(tests, ExecutionScope.ClassLevel, parallelizationEnabled: false)!; + + graph.ParallelChunks.Should().BeEmpty(); + NamesOf(graph.SequentialTests).Should().Equal("First", "Middle", "Last"); + } + + public void Build_WhenADependencyMatchesNoTestInTheRun_WarnsAndIgnoresTheEdge() + { + // Running a subset (a filter, or a single test from an IDE) must stay possible, so an unmatched + // reference cannot block the dependent. + UnitTestElement[] tests = [CreateElement(ClassA, "Only", DependsOnMethod("FilteredOut"))]; + + TestDependencyGraph graph = TestDependencyGraph.Build(tests, ExecutionScope.MethodLevel, parallelizationEnabled: true)!; + + graph.Warnings.Should().ContainSingle(); + graph.Warnings[0].Should().Contain("FilteredOut"); + graph.TestPrerequisites[0].Should().BeEmpty(); + graph.Errors.Should().BeEmpty(); + } + + public void Build_MergesProceedOnFailureAcrossEdges_HoldingBackWhenAnyEdgeAsksToSkip() + { + UnitTestElement[] tests = + [ + CreateElement(ClassA, "First"), + CreateElement(ClassA, "Second"), + CreateElement(ClassA, "Mixed", DependsOnMethod("First", proceedOnFailure: true), DependsOnMethod("Second")), + CreateElement(ClassA, "AllProceed", DependsOnMethod("First", proceedOnFailure: true), DependsOnMethod("Second", proceedOnFailure: true)), + ]; + + TestDependencyGraph graph = TestDependencyGraph.Build(tests, ExecutionScope.MethodLevel, parallelizationEnabled: true)!; + + graph.ProceedOnFailure[2].Should().BeFalse(); + graph.ProceedOnFailure[3].Should().BeTrue(); + } + + public void Build_TieBreaksReadyTestsByDeclarationOrder() + { + // Determinism matters: two runs of the same suite must schedule the same way. + UnitTestElement[] tests = + [ + CreateElement(ClassA, "Alpha"), + CreateElement(ClassA, "Beta"), + CreateElement(ClassA, "Gamma", DependsOnMethod("Beta")), + ]; + + TestDependencyGraph graph = TestDependencyGraph.Build(tests, ExecutionScope.ClassLevel, parallelizationEnabled: true)!; + + NamesOf(graph.ParallelChunks[0]).Should().Equal("Alpha", "Beta", "Gamma"); + } + + public void EncodeDecode_RoundTripsEveryTargetShape() + { + AssertRoundTrip(new TestDependencyInfo(ClassA, "Method", proceedOnFailure: false)); + AssertRoundTrip(new TestDependencyInfo(ClassA, null, proceedOnFailure: true)); + AssertRoundTrip(new TestDependencyInfo(null, "Method", proceedOnFailure: true)); + + static void AssertRoundTrip(TestDependencyInfo original) + { + var decoded = TestDependencyInfo.Decode(TestDependencyInfo.Encode(original)); + decoded.TargetClassFullName.Should().Be(original.TargetClassFullName); + decoded.TargetMethodName.Should().Be(original.TargetMethodName); + decoded.ProceedOnFailure.Should().Be(original.ProceedOnFailure); + } + } + + public void Decode_WhenFlagIsUnrecognized_FailsClosedToSkipAndKeepsTheWholePayload() + { + // Proceeding past a failed prerequisite must never be the result of a decoding glitch, and the + // payload must still name the test it was meant to name. + var decoded = TestDependencyInfo.Decode("Ns.ClassA\nMethod"); + decoded.ProceedOnFailure.Should().BeFalse(); + decoded.TargetClassFullName.Should().Be("Ns.ClassA"); + decoded.TargetMethodName.Should().Be("Method"); + } + + public void DescribeTarget_RendersEachShapeReadably() + { + new TestDependencyInfo(ClassA, "Method", false).DescribeTarget().Should().Be("Ns.ClassA.Method"); + new TestDependencyInfo(ClassA, null, false).DescribeTarget().Should().Be("Ns.ClassA.*"); + new TestDependencyInfo(null, "Method", false).DescribeTarget().Should().Be("Method"); + } +} diff --git a/test/UnitTests/MSTestAdapter.PlatformServices.UnitTests/Execution/TestExecutionManagerTests.cs b/test/UnitTests/MSTestAdapter.PlatformServices.UnitTests/Execution/TestExecutionManagerTests.cs index 9eebc64f81..7424c60848 100644 --- a/test/UnitTests/MSTestAdapter.PlatformServices.UnitTests/Execution/TestExecutionManagerTests.cs +++ b/test/UnitTests/MSTestAdapter.PlatformServices.UnitTests/Execution/TestExecutionManagerTests.cs @@ -757,6 +757,197 @@ public async Task RunTestsForTestShouldRunNonParallelizableTestsSeparately() } } + /// + /// A chunk that throws outside the test body (here: a host-side recording failure) must not strand the + /// workers waiting on it. Before the scheduler released its dependents in a finally, an exception + /// skipped both the dependent release and the completion count, so every other worker blocked forever on + /// a permit that would never be issued and the whole run hung instead of failing. + /// + public async Task RunTestsWhenADependencyChunkThrowsShouldNotHang() + { + TestCase root = GetTestCase(typeof(DummyTestClassForParallelize), "TestMethod1"); + TestCase branchA = GetTestCase(typeof(DummyTestClassForParallelize2), "TestMethod1"); + TestCase branchB = GetTestCase(typeof(DummyTestClassForParallelize2), "TestMethod2"); + + _frameworkHandle.ThrowOnRecordStartForTest = root.DisplayName; + + TestCase[] tests = [root, branchA, branchB]; + _runContext.MockRunSettings.Setup(rs => rs.SettingsXml).Returns( + """ + + + True + + + + 2 + MethodLevel + + + + """); + + UnitTestElement[] elements = ToUnitTestElements(tests); + + // BranchA and BranchB both wait for the chunk that is about to throw, so with the bug present there + // is nothing left to release them and the second worker waits forever. The dependency is set on the + // element directly rather than through the transport property because these elements are already + // materialized (the property round-trip has its own tests); what is under test here is the scheduler. + TestDependencyInfo[] dependsOnRoot = [new TestDependencyInfo(typeof(DummyTestClassForParallelize).FullName, "TestMethod1", proceedOnFailure: false)]; + elements[1].Dependencies = dependsOnRoot; + elements[2].Dependencies = dependsOnRoot; + + try + { + MSTestSettings.PopulateSettings(_runContext.RunSettings?.SettingsXml, _mockMessageLogger.Object.ToAdapterMessageLogger(), null); + + Task run = _testExecutionManager.RunTestsAsync(elements, CurrentDeploymentContext, _frameworkHandle.ToAdapterMessageLogger(), TestResultRecorder, new TestElementFilterProvider(_runContext), new TestRunCancellationToken()); + + // The run must finish - failing is fine, hanging is not. The timeout is what actually asserts the + // fix: without it the test would deadlock rather than fail. + Task completed = await Task.WhenAny(run, Task.Delay(TimeSpan.FromSeconds(60))); + completed.Should().BeSameAs(run, "the run must not hang when a chunk throws"); + + // Reading Exception is what actually observes the fault - awaiting a continuation that ignores + // its antecedent does not, leaving the run to resurface as an UnobservedTaskException later and + // destabilize unrelated tests. It doubles as the assertion that the injected failure propagates + // out of the run rather than being swallowed by the scheduler. + run.IsFaulted.Should().BeTrue("the injected RecordStart failure must surface, not be swallowed"); + run.Exception.Should().NotBeNull(); + } + finally + { + _frameworkHandle.ThrowOnRecordStartForTest = null; + DummyTestClassForParallelize.Cleanup(); + DummyTestClassForParallelize2.Cleanup(); + } + } + + /// + /// A dependency graph must not switch OrderTestsByNameInClass off for the whole source. Tests that + /// are simultaneously ready still run in name order; the graph only constrains the pairs that actually + /// declare an edge. + /// + public async Task RunTestsWithDependenciesShouldStillOrderUnconstrainedTestsByName() + { + // Passed in the order 2, 1, 4 so that declaration order and name order disagree. Only TestMethod4 is + // constrained (it waits for TestMethod2), leaving 1 and 2 simultaneously ready - and they must run in + // name order, not the order they were handed over in. + TestCase testCase2 = GetTestCase(typeof(DummyTestClassWithDoNotParallelizeMethods), "TestMethod2"); + TestCase testCase1 = GetTestCase(typeof(DummyTestClassWithDoNotParallelizeMethods), "TestMethod1"); + TestCase testCase4 = GetTestCase(typeof(DummyTestClassWithDoNotParallelizeMethods), "TestMethod4"); + + _runContext.MockRunSettings.Setup(rs => rs.SettingsXml).Returns( + """ + + + True + + + true + + + """); + + UnitTestElement[] elements = ToUnitTestElements(testCase2, testCase1, testCase4); + elements[2].Dependencies = [new TestDependencyInfo(typeof(DummyTestClassWithDoNotParallelizeMethods).FullName, "TestMethod2", proceedOnFailure: false)]; + + try + { + MSTestSettings.PopulateSettings(_runContext.RunSettings?.SettingsXml, _mockMessageLogger.Object.ToAdapterMessageLogger(), null); + await _testExecutionManager.RunTestsAsync(elements, CurrentDeploymentContext, _frameworkHandle.ToAdapterMessageLogger(), TestResultRecorder, new TestElementFilterProvider(_runContext), new TestRunCancellationToken()); + + _frameworkHandle.TestCaseStartList.Should().Equal("TestMethod1", "TestMethod2", "TestMethod4"); + } + finally + { + DummyTestClassWithDoNotParallelizeMethods.Cleanup(); + } + } + + /// + /// A dependency-skipped test is still one of the tests the class-cleanup countdown was built from, so it + /// has to be accounted for even though it never runs. Otherwise the class never completes: its + /// [ClassCleanup] is silently lost, and because end-of-assembly cleanup waits on every class + /// completing, the assembly's [AssemblyCleanup] is lost with it. + /// + public async Task RunTestsWhenADependentIsSkippedShouldStillRunClassCleanup() + { + TestCase prereq = GetTestCase(typeof(DummyTestClassWithCleanupAndDependency), "Prereq"); + TestCase dependent = GetTestCase(typeof(DummyTestClassWithCleanupAndDependency), "Dependent"); + + UnitTestElement[] elements = ToUnitTestElements(prereq, dependent); + elements[1].Dependencies = [new TestDependencyInfo(typeof(DummyTestClassWithCleanupAndDependency).FullName, "Prereq", proceedOnFailure: false)]; + + // The assertion reads a static of the test class, which lives in whichever app domain ran it, so the + // run has to stay in this one - the same reason every other static-observing test here disables them. + _runContext.MockRunSettings.Setup(rs => rs.SettingsXml).Returns( + """ + + + True + + + """); + + try + { + MSTestSettings.PopulateSettings(_runContext.RunSettings?.SettingsXml, _mockMessageLogger.Object.ToAdapterMessageLogger(), null); + await _testExecutionManager.RunTestsAsync(elements, CurrentDeploymentContext, _frameworkHandle.ToAdapterMessageLogger(), TestResultRecorder, new TestElementFilterProvider(_runContext), new TestRunCancellationToken()); + + // Prereq ran (and failed), so [ClassInitialize] executed and [ClassCleanup] is genuinely owed. + DummyTestClassWithCleanupAndDependency.ClassCleanupCount.Should().Be(1); + + // The skip itself must still be reported - the cleanup accounting must not swallow it. + _frameworkHandle.TestCaseEndList.Should().Contain("Dependent:Skipped"); + } + finally + { + DummyTestClassWithCleanupAndDependency.Cleanup(); + } + } + + /// + /// Same accounting requirement for tests failed by a dependency cycle: they are reported without ever + /// reaching the runner, so the countdown still owes their decrement. + /// + public async Task RunTestsWhenTestsAreBrokenByACycleShouldStillRunClassCleanup() + { + TestCase ok = GetTestCase(typeof(DummyTestClassWithCleanupAndCycle), "Ok"); + TestCase inCycleA = GetTestCase(typeof(DummyTestClassWithCleanupAndCycle), "InCycleA"); + TestCase inCycleB = GetTestCase(typeof(DummyTestClassWithCleanupAndCycle), "InCycleB"); + + UnitTestElement[] elements = ToUnitTestElements(ok, inCycleA, inCycleB); + string className = typeof(DummyTestClassWithCleanupAndCycle).FullName!; + elements[1].Dependencies = [new TestDependencyInfo(className, "InCycleB", proceedOnFailure: false)]; + elements[2].Dependencies = [new TestDependencyInfo(className, "InCycleA", proceedOnFailure: false)]; + + // See the sibling test: the static counter is only observable when the run stays in this app domain. + _runContext.MockRunSettings.Setup(rs => rs.SettingsXml).Returns( + """ + + + True + + + """); + + try + { + MSTestSettings.PopulateSettings(_runContext.RunSettings?.SettingsXml, _mockMessageLogger.Object.ToAdapterMessageLogger(), null); + await _testExecutionManager.RunTestsAsync(elements, CurrentDeploymentContext, _frameworkHandle.ToAdapterMessageLogger(), TestResultRecorder, new TestElementFilterProvider(_runContext), new TestRunCancellationToken()); + + // Ok is not in the cycle, so it runs and initializes the class; the two cycle members never reach + // the runner, yet the class must still complete. + DummyTestClassWithCleanupAndCycle.ClassCleanupCount.Should().Be(1); + _frameworkHandle.TestCaseEndList.Should().Contain("InCycleA:Failed").And.Contain("InCycleB:Failed"); + } + finally + { + DummyTestClassWithCleanupAndCycle.Cleanup(); + } + } + public async Task RunTestsForTestShouldPreferParallelSettingsFromRunSettingsOverAssemblyLevelAttributes() { TestCase testCase1 = GetTestCase(typeof(DummyTestClassForParallelize), "TestMethod1"); @@ -1116,6 +1307,51 @@ public void TestMethod() } } + [DummyTestClass] + private class DummyTestClassWithCleanupAndDependency + { + public static int ClassCleanupCount { get; private set; } + + public static void Cleanup() => ClassCleanupCount = 0; + + [ClassCleanup] + public static void ClassCleanup() => ClassCleanupCount++; + + [TestMethod] + public void Prereq() => throw new Exception("Prereq failed on purpose"); + + [TestMethod] + public void Dependent() + { + } + } + + [DummyTestClass] + private class DummyTestClassWithCleanupAndCycle + { + public static int ClassCleanupCount { get; private set; } + + public static void Cleanup() => ClassCleanupCount = 0; + + [ClassCleanup] + public static void ClassCleanup() => ClassCleanupCount++; + + [TestMethod] + public void Ok() + { + } + + [TestMethod] + public void InCycleA() + { + } + + [TestMethod] + public void InCycleB() + { + } + } + [DummyTestClass] private class DummyTestClassWithCleanupMethods { @@ -1303,6 +1539,13 @@ public TestableFrameworkHandle() public List TestDisplayNameList { get; } + /// + /// When set, throws for the test whose display name matches. Used to simulate a + /// chunk that fails outside the test body (a host-side recording failure), which the dependency scheduler + /// must survive without stranding the workers waiting on that chunk. + /// + public string? ThrowOnRecordStartForTest { get; set; } + public void RecordResult(TestResult testResult) { ResultsList.Add(testResult.ToString()); @@ -1311,7 +1554,15 @@ public void RecordResult(TestResult testResult) public void SendMessage(TestMessageLevel testMessageLevel, string message) => MessageList.Add($"{testMessageLevel}:{message}"); - public void RecordStart(TestCase testCase) => TestCaseStartList.Add(testCase.DisplayName); + public void RecordStart(TestCase testCase) + { + if (ThrowOnRecordStartForTest is { } failing && string.Equals(testCase.DisplayName, failing, StringComparison.Ordinal)) + { + throw new InvalidOperationException($"Simulated host failure while recording the start of '{failing}'."); + } + + TestCaseStartList.Add(testCase.DisplayName); + } public void RecordEnd(TestCase testCase, TestOutcome outcome) => TestCaseEndList.Add($"{testCase.DisplayName}:{outcome}"); diff --git a/test/UnitTests/MSTestAdapter.PlatformServices.UnitTests/MSTestSettingsTests.cs b/test/UnitTests/MSTestAdapter.PlatformServices.UnitTests/MSTestSettingsTests.cs index 44bc3b785d..7a19af5919 100644 --- a/test/UnitTests/MSTestAdapter.PlatformServices.UnitTests/MSTestSettingsTests.cs +++ b/test/UnitTests/MSTestAdapter.PlatformServices.UnitTests/MSTestSettingsTests.cs @@ -1619,4 +1619,129 @@ public void ConfigJson_WithValidValues_MethodScope() } #endregion + + #region Dependency declaration configuration tests + + /// + /// Parses a flattened key/value configuration - the shape the JSON provider produces - through the real + /// settings parser, so these tests exercise the same indexer-only surface it sees at run time. + /// + private MSTestSettings ParseDependencies(Dictionary values) + { + var mockConfig = new Mock(); + mockConfig.Setup(config => config[It.IsAny()]) + .Returns((string key) => values.TryGetValue(key, out string? value) ? value : null); + + var settings = new MSTestSettings(); + MSTestSettings.SetSettingsFromConfig(mockConfig.Object, _mockMessageLogger.Object.ToAdapterMessageLogger(), settings); + return settings; + } + + public void DependencyConfig_ChainsProduceAnEdgePerAdjacentPair() + { + MSTestSettings settings = ParseDependencies(new() + { + { "mstest:execution:dependencies:chains:0:0", "Ns.C.First" }, + { "mstest:execution:dependencies:chains:0:1", "Ns.C.Second" }, + { "mstest:execution:dependencies:chains:0:2", "Ns.C.Third" }, + }); + + settings.DeclaredDependencies.Should().NotBeNull(); + settings.DeclaredDependencies!.Length.Should().Be(2); + settings.DeclaredDependencies[0].Dependent.Should().Be("Ns.C.Second"); + settings.DeclaredDependencies[0].Prerequisite.Should().Be("Ns.C.First"); + settings.DeclaredDependencies[1].Dependent.Should().Be("Ns.C.Third"); + settings.DeclaredDependencies[1].Prerequisite.Should().Be("Ns.C.Second"); + } + + public void DependencyConfig_AChainOfOneDeclaresNoEdge() + { + MSTestSettings settings = ParseDependencies(new() + { + { "mstest:execution:dependencies:chains:0:0", "Ns.C.Only" }, + }); + + settings.DeclaredDependencies.Should().BeNull(); + } + + public void DependencyConfig_AnEmptyChainIsWarnedAboutAndDoesNotTruncateTheRest() + { + // An empty element stores nothing under its index, which through the indexer is indistinguishable + // from the end of the array - so without the lookahead every later chain would silently vanish. + MSTestSettings settings = ParseDependencies(new() + { + { "mstest:execution:dependencies:chains:1:0", "Ns.C.First" }, + { "mstest:execution:dependencies:chains:1:1", "Ns.C.Second" }, + }); + + settings.DeclaredDependencies.Should().ContainSingle(); + settings.DeclaredDependencies![0].Dependent.Should().Be("Ns.C.Second"); + _mockMessageLogger.Verify( + lm => lm.SendMessage(TestMessageLevel.Warning, It.Is(m => m.Contains("chain at index 0"))), + Times.Once); + } + + public void DependencyConfig_NodesCarryEveryPrerequisiteAndProceedOnFailure() + { + MSTestSettings settings = ParseDependencies(new() + { + { "mstest:execution:dependencies:nodes:0:test", "Ns.C.Audit" }, + { "mstest:execution:dependencies:nodes:0:dependsOn:0", "Ns.C.One" }, + { "mstest:execution:dependencies:nodes:0:dependsOn:1", "Ns.C.Two" }, + { "mstest:execution:dependencies:nodes:0:proceedOnFailure", "true" }, + }); + + settings.DeclaredDependencies!.Length.Should().Be(2); + settings.DeclaredDependencies.Should().OnlyContain(d => d.Dependent == "Ns.C.Audit" && d.ProceedOnFailure); + } + + public void DependencyConfig_ProceedOnFailureDefaultsToFalseAndWarnsWhenNotABoolean() + { + MSTestSettings settings = ParseDependencies(new() + { + { "mstest:execution:dependencies:nodes:0:test", "Ns.C.Dependent" }, + { "mstest:execution:dependencies:nodes:0:dependsOn:0", "Ns.C.Prereq" }, + { "mstest:execution:dependencies:nodes:0:proceedOnFailure", "maybe" }, + }); + + // Fails closed: an unparseable value must not become "run anyway". + settings.DeclaredDependencies!.Single().ProceedOnFailure.Should().BeFalse(); + _mockMessageLogger.Verify( + lm => lm.SendMessage(TestMessageLevel.Warning, It.Is(m => m.Contains("maybe"))), + Times.Once); + } + + public void DependencyConfig_ANodeWithoutPrerequisitesIsWarnedAbout() + { + MSTestSettings settings = ParseDependencies(new() + { + { "mstest:execution:dependencies:nodes:0:test", "Ns.C.Lonely" }, + }); + + settings.DeclaredDependencies.Should().BeNull(); + _mockMessageLogger.Verify( + lm => lm.SendMessage(TestMessageLevel.Warning, It.Is(m => m.Contains("Ns.C.Lonely"))), + Times.Once); + } + + public void DependencyConfig_ANodeMissingItsTestIsWarnedAboutAndDoesNotTruncateTheRest() + { + MSTestSettings settings = ParseDependencies(new() + { + { "mstest:execution:dependencies:nodes:1:test", "Ns.C.Dependent" }, + { "mstest:execution:dependencies:nodes:1:dependsOn:0", "Ns.C.Prereq" }, + }); + + settings.DeclaredDependencies.Should().ContainSingle(); + settings.DeclaredDependencies![0].Dependent.Should().Be("Ns.C.Dependent"); + _mockMessageLogger.Verify( + lm => lm.SendMessage(TestMessageLevel.Warning, It.Is(m => m.Contains("node at index 0"))), + Times.Once); + } + + public void DependencyConfig_WhenNoDependenciesAreDeclaredNothingIsSet() + => ParseDependencies(new() { { "mstest:execution:randomizeTestOrder", "true" } }) + .DeclaredDependencies.Should().BeNull(); + + #endregion } diff --git a/test/UnitTests/MSTestAdapter.PlatformServices.UnitTests/ObjectModel/UnitTestElementTests.cs b/test/UnitTests/MSTestAdapter.PlatformServices.UnitTests/ObjectModel/UnitTestElementTests.cs index 29e47f1e26..ee15df9d91 100644 --- a/test/UnitTests/MSTestAdapter.PlatformServices.UnitTests/ObjectModel/UnitTestElementTests.cs +++ b/test/UnitTests/MSTestAdapter.PlatformServices.UnitTests/ObjectModel/UnitTestElementTests.cs @@ -80,6 +80,57 @@ public void ResourceLocksShouldRemainNullWhenNoneAreDeclared() roundTripped.ResourceLocks.Should().BeNull(); } + public void DependenciesShouldRoundTripThroughTestCaseProperty() + { + // The acceptance assets run on the native MSTest runner, so nothing else exercises this hidden VSTest + // TestProperty transport; a registration or conversion regression would silently make [DependsOn] + // ineffective under VSTest while every other test still passed. + // + // LocalExtensionData is cleared deliberately. ToTestCase caches the element on the test case, and + // ToUnitTestElementWithUpdatedSource returns that cached instance when it is present - which is what + // happens in-process, and which would make this test pass without the encoded property being read at + // all. Clearing it reproduces the case the encoding exists for: a test case that crossed a process or + // AppDomain boundary, where only the serialized properties survive. + var element = new UnitTestElement(new TestMethod("M", "C", "A", displayName: null)) + { + Dependencies = + [ + new TestDependencyInfo("Ns.Other", "Prereq", proceedOnFailure: false), + new TestDependencyInfo("Ns.Whole", null, proceedOnFailure: true), + new TestDependencyInfo(null, "SameClassPrereq", proceedOnFailure: false), + ], + }; + + var testCase = element.ToTestCase(); + testCase.LocalExtensionData = null; + UnitTestElement roundTripped = testCase.ToUnitTestElementWithUpdatedSource("A"); + + roundTripped.Dependencies.Should().NotBeNull(); + roundTripped.Dependencies!.Length.Should().Be(3); + + roundTripped.Dependencies[0].TargetClassFullName.Should().Be("Ns.Other"); + roundTripped.Dependencies[0].TargetMethodName.Should().Be("Prereq"); + roundTripped.Dependencies[0].ProceedOnFailure.Should().BeFalse(); + + // A whole-class target keeps its null method name, which is what distinguishes it from a named one. + roundTripped.Dependencies[1].TargetClassFullName.Should().Be("Ns.Whole"); + roundTripped.Dependencies[1].TargetMethodName.Should().BeNull(); + roundTripped.Dependencies[1].ProceedOnFailure.Should().BeTrue(); + + // A same-class target keeps its null class name, so it still resolves against the dependent's class. + roundTripped.Dependencies[2].TargetClassFullName.Should().BeNull(); + roundTripped.Dependencies[2].TargetMethodName.Should().Be("SameClassPrereq"); + roundTripped.Dependencies[2].ProceedOnFailure.Should().BeFalse(); + } + + public void DependenciesShouldRemainNullWhenNoneAreDeclared() + { + var testCase = _unitTestElement.ToTestCase(); + testCase.LocalExtensionData = null; + + testCase.ToUnitTestElementWithUpdatedSource("A").Dependencies.Should().BeNull(); + } + #endregion #region Source resolution / host test case tests diff --git a/test/UnitTests/TestFramework.UnitTests/Attributes/DependsOnAttributeTests.cs b/test/UnitTests/TestFramework.UnitTests/Attributes/DependsOnAttributeTests.cs new file mode 100644 index 0000000000..a55e0d69b6 --- /dev/null +++ b/test/UnitTests/TestFramework.UnitTests/Attributes/DependsOnAttributeTests.cs @@ -0,0 +1,97 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT license. See LICENSE file in the project root for full license information. + +using AwesomeAssertions; + +using TestFramework.ForTestingMSTest; + +namespace Microsoft.VisualStudio.TestPlatform.TestFramework.UnitTests.Attributes; + +/// +/// Tests for . +/// +public class DependsOnAttributeTests : TestContainer +{ + public void Constructor_WithMethodName_TargetsAMethodOfTheSameClass() + { + var attribute = new DependsOnAttribute("Setup"); + + attribute.TestMethodName.Should().Be("Setup"); + attribute.TestClass.Should().BeNull(); + attribute.ProceedOnFailure.Should().BeFalse(); + } + + public void Constructor_WithType_TargetsEveryTestOfThatClass() + { + var attribute = new DependsOnAttribute(typeof(DependsOnAttributeTests)); + + attribute.TestClass.Should().Be(); + + // A null method name is what distinguishes "every test of the class" from "this one test". + attribute.TestMethodName.Should().BeNull(); + } + + public void Constructor_WithTypeAndMethodName_TargetsThatMethodOfThatClass() + { + var attribute = new DependsOnAttribute(typeof(DependsOnAttributeTests), "Setup"); + + attribute.TestClass.Should().Be(); + attribute.TestMethodName.Should().Be("Setup"); + } + + public void ProceedOnFailure_CanBeSet() + { + // The default is asserted here too, not only in the constructor tests: without it, a property that + // ignored its setter and always returned true would still satisfy this test. + new DependsOnAttribute("Setup").ProceedOnFailure.Should().BeFalse(); + + var attribute = new DependsOnAttribute("Setup") { ProceedOnFailure = true }; + + attribute.ProceedOnFailure.Should().BeTrue(); + } + + public void Constructor_WhenMethodNameIsNull_Throws() + { + Action nameOnly = () => _ = new DependsOnAttribute((string)null!); + nameOnly.Should().Throw(); + + Action withType = () => _ = new DependsOnAttribute(typeof(DependsOnAttributeTests), null!); + withType.Should().Throw(); + } + + public void Constructor_WhenMethodNameIsEmptyOrWhitespace_Throws() + { + // An empty target is never intentional and would otherwise become an edge that silently matches + // nothing, so it is rejected at the source rather than warned about at run time. + Action empty = () => _ = new DependsOnAttribute(string.Empty); + empty.Should().Throw(); + + Action whitespace = () => _ = new DependsOnAttribute(" "); + whitespace.Should().Throw(); + } + + public void Constructor_WhenTypeIsNull_Throws() + { + Action typeOnly = () => _ = new DependsOnAttribute((Type)null!); + typeOnly.Should().Throw(); + + Action withMethod = () => _ = new DependsOnAttribute(null!, "Setup"); + withMethod.Should().Throw(); + } + + public void Attribute_AllowsMultiple_SoATestCanDeclareSeveralPrerequisites() + { + AttributeUsageAttribute usage = typeof(DependsOnAttribute) + .GetCustomAttributes(typeof(AttributeUsageAttribute), inherit: false) + .Cast() + .Single(); + + // Fan-in is expressed by repeating the attribute, so AllowMultiple is part of the contract. + usage.AllowMultiple.Should().BeTrue(); + + // Deliberately not inherited: a dependency states one concrete test's prerequisites, and + // re-pointing it at every derived class would invent edges nobody declared. + usage.Inherited.Should().BeFalse(); + usage.ValidOn.Should().Be(AttributeTargets.Class | AttributeTargets.Method); + } +}