Add dotnet test --list-devices support for MAUI

[jonathanpeppers]

Expands on #54295 (which added dotnet test --device) to implement the --list-devices switch on dotnet test, as described in dotnet-run-for-maui.md. This lets users discover the device identifiers they can pass to --device for MAUI/Android/iOS test projects without first having to run a build.

Behavior

dotnet test --list-devices mirrors dotnet run --list-devices:

• Resolves the project (rejects solutions, since each project may have its own device list).
• Prompts for $(TargetFramework) for multi-targeted projects in interactive mode; errors with a --framework suggestion in non-interactive mode.
• Calls the ComputeAvailableDevices MSBuild target via RunCommandSelector and prints the available devices with a friendly dotnet test --device <id> example.
• Exits early without building, deploying, or running any tests.
• If the project has no ComputeAvailableDevices target, exits silently with success (matches dotnet run --list-devices).

Implementation notes

To keep the listing flow structurally identical to dotnet run --list-devices, HandleListDevices reuses a single RunCommandSelector instance for TrySelectTargetFramework -> InvalidateGlobalProperties -> TrySelectDevice(listDevices: true), the same sequence as RunCommand.TrySelectTargetFrameworkAndDeviceIfNeeded.

RunCommandSelector.TrySelectDevice gained a required commandName parameter so the listed/error example renders dotnet test --device <id> instead of the previously hard-coded dotnet run --device <id>. The two existing call sites (dotnet run, per-TFM device selection in tests) pass "dotnet run" and "dotnet test" respectively.

Code review surfaced a separate latent bug from #54295: when --device was used against a solution, the error told users to specify --framework even though that wouldn't fix the failure. Both --device and --list-devices now throw TestCommandUseProject (Specifying a project for 'dotnet test' should be via '--project'.). The --device guard applies unconditionally, including when -f/--framework is also passed, matching --list-devices behavior.

Tests

Six new integration tests in GivenDotnetTestSelectsDevice:

• lists devices for a multi-TFM project with -f
• lists a single device when only one is available
• fails non-interactively when multi-targeted without -f
• exits silently for projects without ComputeAvailableDevices
• errors with the --project hint when --list-devices is run against a solution
• errors with the --project hint when --device is used against a solution (parameterized with and without -f)

The MTP help snapshot is also updated for the new --list-devices line.

Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com

[Copilot]

Pull request overview

This PR extends the dotnet test Microsoft Testing Platform (MTP) command to support --list-devices, mirroring the existing dotnet run --list-devices behavior so MAUI/Android/iOS test projects can enumerate device IDs without building or running tests.

Changes:

• Added dotnet test --list-devices option and an early-exit listing flow that calls ComputeAvailableDevices via RunCommandSelector.
• Updated RunCommandSelector.TrySelectDevice to accept a commandName so the printed example uses the invoking command (dotnet run vs dotnet test).
• Added integration tests for listing behavior and updated the MTP help snapshot.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
test/dotnet.Tests/CommandTests/Test/snapshots/MTPHelpSnapshotTests.VerifyMTPHelpOutput.verified.txt	Updates dotnet test MTP help snapshot to include --list-devices.
test/dotnet.Tests/CommandTests/Test/GivenDotnetTestSelectsDevice.cs	Adds integration tests covering --list-devices scenarios (multi-TFM, single device, missing target, solution error).
src/Cli/Microsoft.DotNet.Cli.Definitions/Commands/Test/TestCommandDefinition.MicrosoftTestingPlatform.cs	Adds the --list-devices option to the MTP dotnet test command definition.
src/Cli/dotnet/Commands/Test/MTP/SolutionAndProjectUtility.cs	Passes commandName: "dotnet test" into device selection to render correct examples.
src/Cli/dotnet/Commands/Test/MTP/Options.cs	Extends BuildOptions to carry the ListDevices flag.
src/Cli/dotnet/Commands/Test/MTP/MSBuildUtility.cs	Wires --list-devices parse result into BuildOptions.
src/Cli/dotnet/Commands/Test/MTP/MicrosoftTestingPlatformTestCommand.cs	Implements the --list-devices early-exit flow and improves solution handling for --device.
src/Cli/dotnet/Commands/Run/RunCommandSelector.cs	Adds commandName parameter to TrySelectDevice and uses it in example output.
src/Cli/dotnet/Commands/Run/RunCommand.cs	Updates dotnet run to pass commandName: "dotnet run" into TrySelectDevice.

[Evangelink]

⚠️ Disclaimer: This review was produced by a code-review subagent of GitHub Copilot CLI acting as a testfx / Microsoft.Testing.Platform expert reviewer, posted under my identity. Treat findings as drafted suggestions to verify rather than authoritative statements from me personally. I will follow up on anything that lands.

Reviewed the diff against main. Focused on parity with dotnet run --list-devices, option wiring, exit codes, and the new HandleListDevices flow.

🐛 Bugs / Correctness

1. ListDevicesOption is missing Arity = ArgumentArity.Zero

File: src/Cli/Microsoft.DotNet.Cli.Definitions/Commands/Test/TestCommandDefinition.MicrosoftTestingPlatform.cs

public readonly Option<bool> ListDevicesOption = new("--list-devices")
{
    Description = CommandDefinitionStrings.CommandOptionListDevicesDescriptionForTest,
};

The dotnet run counterpart deliberately sets Arity = ArgumentArity.Zero (RunCommandDefinition.cs:57‑61), and so does every other boolean flag added to this very class — NoAnsiOption, ListTestsOption, NoLaunchProfileOption. Without Arity.Zero, an Option<bool> defaults to ZeroOrOne, meaning System.CommandLine will try to bind the next positional token (e.g. dotnet test --list-devices SomeProject.csproj) as the option value and fail bool parsing rather than treating it as a flag. The mirrored dotnet run option exists precisely to prevent that ambiguity — the test command should match.

⚠️ Behavioral / Parity Concerns

2. TrySelectTargetFramework still suggests "dotnet run --framework …" when invoked from dotnet test

File: src/Cli/dotnet/Commands/Run/RunCommandSelector.cs:280, reached from the new HandleListDevices.

The PR went out of its way to thread a commandName parameter through TrySelectDevice so the "Example: dotnet test --device …" hint is correct. The exact same hard‑coded "dotnet run --framework {frameworks[0]}" string survives in TrySelectTargetFramework and now fires from two dotnet test code paths:

• HandleListDevices → selector.TrySelectTargetFramework(...) (new in this PR)
• HandleDeviceWithTargetFrameworkSelection → RunCommandSelector.TrySelectTargetFramework(...) (pre‑existing, untouched here)

So a user running dotnet test --list-devices (or dotnet test --device) against a multi‑TFM project in a non‑interactive shell still sees:

Example: dotnet run --framework net10.0-android

This is exactly the latent bug the PR claims to be cleaning up for --device on a solution. The new test ItFailsToListDevicesWhenMultipleTargetFrameworks_InNonInteractiveMode only asserts the first error string and therefore doesn't catch it. Consider plumbing commandName through TrySelectTargetFramework (and/or its static overload) the same way the PR did for TrySelectDevice, and adding a stderr.Should().NotContain("dotnet run") assertion.

3. --no-build silently implies --no-restore for device discovery in dotnet test, but not in dotnet run

File: MicrosoftTestingPlatformTestCommand.cs

noRestore: buildOptions.HasNoRestore || buildOptions.HasNoBuild,

RunCommand.TrySelectTargetFrameworkAndDeviceIfNeeded passes only NoRestore (RunCommand.cs:303). The MTP version conflates --no-build with --no-restore for the ComputeAvailableDevices evaluation. This matches the pre‑existing SelectDeviceForTfm in SolutionAndProjectUtility.cs:429, so it's at least internally consistent, but it diverges from the stated "mirror dotnet run --list-devices" goal and means that on a clean project dotnet test --list-devices --no-build will skip the restore that dotnet run --list-devices --no-build would perform — and ComputeAvailableDevices may then fail on a missing obj/project.assets.json. Worth a deliberate decision rather than an accidental one.

4. --list-devices silently swallows --device and --test-modules

File: MicrosoftTestingPlatformTestCommand.cs

The PR (correctly) added a hard error for --list-devices + --list-tests. But:

• --list-devices --device X → ListDevices short‑circuits at the top of Run, Device is silently ignored. dotnet run has the same behavior, so this is parity, but it's worth being explicit (either a third mutually‑exclusive check or a documented "device is ignored").
• --list-devices --test-modules foo → ValidationUtility.ValidateOptionsIrrelevantToModulesFilter only screens Configuration/Framework/Architecture/OS/Runtime, not Device/ListDevices. HandleListDevices then calls ValidateBuildPathOptions, which ignores TestModules entirely and falls back to scanning the current directory for a project — surprising and almost certainly not what the user asked for. Pre‑existing for --device too, but the PR is a natural place to tighten it.

🧪 Test Gaps

The five new tests cover the happy path and the "must‑error" cases reasonably well, but a few high‑value cases are missing:

• No regression test for the "dotnet test --device" hint added in the new diff. The positive ItListsDevicesAndExits asserts StdOut.Should().Contain("dotnet test --device") (good), but there's no equivalent for the non‑interactive multi‑device fallback path. Worth a test that exercises the else branch of TrySelectDevice so the rendered example doesn't silently regress.
• --list-devices on .slnf: only .slnx is tested (ItErrorsWhenListingDevicesForSolution). .slnf is in CliConstants.SolutionExtensions so it should take the same branch, but the --solution parsing for .slnf has had its own bugs historically.
• --list-devices with explicit -f on a multi‑TFM device project: confirms the "non‑interactive TFM prompt failure" doesn't fire when the user is doing the right thing. The current ItListsDevicesAndExits uses ToolsetInfo.CurrentTargetFramework which the asset declares as the only TFM, so we never actually validate the "two TFMs + explicit --framework" path that MAUI users will hit in practice.
• --list-devices + --test-modules (see concern 4).
• --list-devices + --device (see concern 4) — assert that --device is ignored / no error, so any future stricter behavior is a deliberate decision.

💡 Optional Observations

• HandleListDevices reproduces the isInteractive heuristic !Console.IsOutputRedirected && !new CIEnvironmentDetectorForTelemetry().IsCIEnvironment(). dotnet run instead honors the explicit --interactive option (RunCommand.cs:948). The MTP command does expose interactivity differently, so this matches the surrounding code, but it does mean dotnet test --interactive --list-devices on a TTY behaves identically to dotnet test --list-devices. Possibly fine, possibly a future paper cut.
• commandName is a required positional string placed at the end of TrySelectDevice after three out parameters. All call sites use named‑arg syntax so it works, but string commandName = "dotnet" with explicit overrides at the two callers would be a smaller, less surprising signature change — and would defuse any future caller (e.g., a unit test using positional args) accidentally compiling against a stale shape. Not worth blocking on.
• The --list-devices/--list-tests mutual‑exclusion message lives in CliCommandStrings.resx (runtime layer). Correct choice — the check executes in MicrosoftTestingPlatformTestCommand.Run, not at definition time. ✅
• Placeholders in the two new CommandOptionListDevicesDescriptionForTest / CommandOptionDeviceDescriptionForTest strings: both are plain sentences with no {0}/{1}, so no format‑string mismatch risk. ✅

[jonathanpeppers]

Thanks for the thorough review! Pushed a follow-up commit (squashed into the single commit on the branch). Summary of what I addressed:

Adopted:

• 1) Arity = ArgumentArity.Zero on ListDevicesOption - added. Matches every other bool option in TestCommandDefinition.MicrosoftTestingPlatform.cs and prevents the option from greedily consuming the next positional token.
• 2) Hard-coded "dotnet run --framework" in error output - fixed. RunCommandSelector now takes commandName via its constructor (stored as _commandName), and the static TrySelectTargetFramework overload takes it as a parameter. Callers from MicrosoftTestingPlatformTestCommand / SolutionAndProjectUtility pass "dotnet test"; the RunCommand callers pass "dotnet run". Strengthened the existing ItFailsToListDevicesWhenMultipleTargetFrameworks_InNonInteractiveMode test to assert stderr contains "dotnet test --framework" and does not contain "dotnet run --framework".
• 4) --list-devices / --device swallowing --test-modules - added explicit validation in MicrosoftTestingPlatformTestCommand.Run that fails fast with a clear error (new resx string CmdDeviceOptionsRequireProject) when these are combined. New test ItErrorsWhenListDevicesIsCombinedWithTestModules covers it.

Skipped (with rationale):

• 3) noRestore conflated with HasNoBuild - this matches the existing pattern in SelectDeviceForTfm and dotnet run's own behavior. Decoupling them would be a behavior change worth its own PR rather than rolling into this one.
• Test gap on .slnf solution filters - .slnx coverage exercises the same SolutionAndProjectUtility path; .slnf adds marginal value here.
• Test gap on --list-devices --device combination - parity with dotnet run, which has the same shape.
• Optional Observations - all noted by the reviewer as non-blocking; left as-is to keep the diff focused.

[Evangelink]

Note

This review was generated with AI assistance and reviewed before posting.

Overall this is a clean change that mirrors dotnet run --list-devices closely — signatures and call sites are consistent, the new resource strings exist, .xlf files are regenerated correctly, and the help snapshot matches. A few items below, mostly around test coverage and a behavioral inconsistency between the --device and --list-devices solution guards.

[Evangelink]

Low confidence / pre-existing: TryComputeAvailableDevices (in RunCommandSelector) returns false identically for three distinct cases — target absent, restore failed, and ComputeAvailableDevices build failed — and TrySelectDevice treats a false target result as "no device support" and returns true. So dotnet test --list-devices on a project that has the target but fails to restore/build exits 0 with no output rather than surfacing the failure. This is inherited from the dotnet run selector, but the behavior is now also on the test path — flagging as an opportunity rather than a blocker.

[jonathanpeppers]

Agree this is worth documenting, but keeping the current behavior intentionally: it matches dotnet run --list-devices exactly (the whole point of using RunCommandSelector here is parity with the run path). Improving the diagnostics on restore/build failure would be a good separate change in RunCommandSelector itself, benefiting both dotnet run and dotnet test users.

[Evangelink]

Note

This review was generated with AI assistance and reviewed before posting.

Re-reviewed after the latest push. The two earlier findings look correctly addressed: the --device + --solution guard was moved out of the TargetFramework check so it now rejects unconditionally (matching --list-devices), and ItErrorsWhenUsingDeviceForSolution covers both the -f and no--f cases. One new blocking issue from the merge with main below.

[Evangelink]

Blocking (compile break): the merge from main migrated this file to MSTest.Sdk — the pre-existing tests above (lines ~18-196) are now [TestClass]/[TestMethod]/[DataRow], but every PR-added test from here down (lines 198-379) still uses xUnit [Fact]/[Theory]/[InlineData]. dotnet.Tests.csproj references Microsoft.NET.TestFramework.MSTest.csproj and has no using Xunit (the Xunit global using only lives in the non-MSTest Microsoft.NET.TestFramework.csproj), so these attributes won't resolve — this is now the only file in dotnet.Tests containing [Fact]. Note the Finding-1 fix ItErrorsWhenUsingDeviceForSolution also uses [Theory]/[InlineData], so it won't compile either. Convert all PR-added tests: [Fact] -> [TestMethod], [Theory] -> [TestMethod], [InlineData(x)] -> [DataRow(x)]. The tests use FluentAssertions and no Assert.Throws, so the attributes are the only change needed.

[jonathanpeppers]

Done in 1763335 — converted every PR-added test in this file to MSTest: [Fact] -> [TestMethod], [Theory] + [InlineData(x)] -> [TestMethod] + [DataRow(x)], added [TestClass] on the class, dropped the ITestOutputHelper ctor injection in favor of the parameterless ctor + inherited Log property (matches the pattern the rest of dotnet.Tests uses post-migration). Assertions and test logic are unchanged. Verified dotnet build test/dotnet.Tests/dotnet.Tests.csproj compiles clean locally.

[Evangelink]

Note

This review was generated with AI assistance and reviewed before posting.

Re-reviewed the latest push. The blocking compile break is fully resolved — all PR-added tests now use MSTest attributes ([TestMethod]/[DataRow]), and there are no remaining [Fact]/[Theory]/[InlineData] or other xUnit references anywhere in dotnet.Tests. Combined with the earlier rounds, everything I raised is addressed:

• --device + --solution is now rejected unconditionally (parity with --list-devices + --solution), covered by the parameterized ItErrorsWhenUsingDeviceForSolution.
• The --device-on-solution error-message fix has regression coverage.
• The restore/build-failure silent-success behavior is intentional parity with dotnet run --list-devices, reasonably deferred to a separate change in RunCommandSelector.

The --list-devices flow mirrors dotnet run --list-devices cleanly, signatures/call sites are consistent, resource strings and .xlf regeneration are correct, and the help snapshot matches. LGTM.