Dotnetup: Add support for update, uninstall, and garbage collection - #53290
Conversation
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Migrate the dotnetup shared manifest from a flat List<DotnetInstall> to a structured format with schema versioning, dotnet roots, install specs, and installations with subcomponents. New types: - DotnetupManifestData: top-level with schemaVersion and dotnetRoots - DotnetRootEntry: groups install specs and installations per root - InstallSpec: tracks what user requested (component, channel, source) - Installation: tracks what's installed (component, version, subcomponents) - InstallSource enum: Explicit, GlobalJson, Previous Key changes: - DotnetupSharedManifest reads/writes new format with legacy migration - IDotnetupManifest extended with install spec and installation APIs - InstallerOrchestratorSingleton records both specs and installations - 8 new unit tests covering serialization, migration, and CRUD ops Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Implement the remaining pieces of the installation tracking design: Subcomponent tracking: - SubcomponentResolver maps archive entry paths to subcomponent IDs based on known folder depths (sdk=2, shared=3, host=3, packs=3, templates=2, sdk-manifests=4) - DotnetArchiveExtractor now collects subcomponent IDs during extraction - InstallerOrchestratorSingleton populates Installation.Subcomponents Garbage collection: - GarbageCollector class refreshes global.json specs, resolves latest matching installation per spec, removes unmarked installations, and deletes orphaned subcomponent folders from disk Update command: - SdkUpdateCommand resolves latest version per install spec, installs if needed, then runs GC to clean old versions Uninstall command: - SdkUninstallCommand removes matching install spec, runs GC, warns if installation still referenced by other specs Pre-existing root detection: - PreexistingRootDetector scans dotnet root for existing SDK/runtime folders and creates Previous-sourced install specs on first use Tests: 30 new tests covering SubcomponentResolver, GarbageCollector, and PreexistingRootDetector. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
By default, the uninstall command only removes install specs with InstallSource.Explicit. A new --source option accepts explicit (default), previous, globaljson, or all to target other install sources. When no explicit spec matches but specs with other sources exist, a helpful message directs the user to use the --source option. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…l.json resolver, list improvements - Create shared UpdateWorkflow and UninstallWorkflow (#4A) - Refactor SdkUpdateCommand and SdkUninstallCommand to thin wrappers (#4B) - Fix uninstall error message to use actual source filter (dotnet#2) - Fix GC deletion assumption with targeted installation snapshot (dotnet#3) - Include channel/spec in update status messages (dotnet#8) - Create RuntimeUpdateCommand and RuntimeUninstallCommand (#4C) - Wire runtime update/uninstall in parsers (#4D) - Implement GlobalJsonChannelResolver and use in GC RefreshGlobalJsonSpecs (dotnet#16) - List command now shows both install specs and installed versions (dotnet#12) - Fix SourceOption naming/syntax for System.CommandLine compatibility Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Changes include: - Rename 'managed installations' to 'tracked installations' (dotnet#10) - Fix '.metadata' -> 'metadata' folder name (dotnet#18) - Include component type in error/warning messages (dotnet#11) - Remove component spec from runtime update (dotnet#12) - Remove GetInstallations wrapper (dotnet#6) - ScopedMutex throws TimeoutException on failure (dotnet#22) - Make InstallResult.Install non-nullable (dotnet#17) - Remove ComponentFilesExist shortcut (dotnet#19) - Wire up InstallSource enum (dotnet#21) - ListCommand: enum types, narrowed mutex, read-only verify, spacing (dotnet#3,dotnet#4,dotnet#5,#1) - Show global.json path in list output (dotnet#15) - Add doc comments to ListData types (dotnet#7) - Move SourceOption to CommonOptions, remove alias fields (dotnet#9) - Remove ResolveChannelFromGlobalJson test hook wrapper (dotnet#14) - Root 'dotnetup update' updates all components (dotnet#16) - GlobalJsonChannelResolver: support rollForward policy (dotnet#13) - Error when installing to unmanaged dotnet root (dotnet#20) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Changes include: - UpdateWorkflow.Execute returns non-zero exit code on failure (#1) - InfoCommand JSON output includes InstallSpecs (dotnet#2) - Wire --update-global-json through SdkUpdateCommand to UpdateWorkflow (dotnet#3) - Implement DotnetInstallManager.UpdateGlobalJson using Utf8JsonReader (dotnet#3) - Fix ResolveEntryDestPath to use normalized entry name (dotnet#4) - Fix test: MajorMinor test uses correct channel '9.0' (dotnet#5) - Fix test: LTS assertion checks Major instead of Minor (dotnet#6) - Fix ScopedMutex.Dispose to not leak mutex on ReleaseMutex throw (dotnet#7) - Make CommonOptions fields static readonly (dotnet#9) - Remove unused --no-progress from uninstall parsers (dotnet#10) - GarbageCollector catches Exception instead of just IOException (dotnet#11) - Capitalize 'SDK' in SdkCommandParser description (dotnet#12) - Eliminate redundant manifest reads in InstallerOrchestratorSingleton (dotnet#13) - Fix STS test error message to say 'major' not 'minor' (dotnet#14) - Make DotnetupUtilities.ExeSuffix readonly (dotnet#16) - Fix FormatBytes to use floating-point with one decimal (dotnet#17) - Remove dead ReleaseManifest variable in DotnetInstaller (dotnet#18) - Make UpdateChannel.Name immutable (dotnet#19) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Changes: - Make GlobalJsonContents.Sdk and SdkSection public (fixes source-gen deserialization) - Add path traversal (zip slip) check in archive extraction - Add symlink and hard link support in tar extraction - Fix IsAdminInstallPath to require directory boundary match - Use MarkupLineInterpolated to prevent Spectre.Console markup injection - Fix DownloadProgress divide-by-zero when TotalBytes is 0 - Use case-sensitive path comparison on Linux in PathsEqual - Normalize paths in IsRootInManifest with Path.GetFullPath - Add clarifying comments for UpdateChannel.Matches broad matching - Make SpectreProgressTarget.Dispose idempotent with TrySetResult - Report actual file size for cached downloads instead of 100/100 - Scope anyUpdated/anyFailed per-root in UpdateWorkflow - Fix RegistryKey leak from OpenBaseKey in WindowsPathHelper - Cache ChannelVersionResolver in DotnetReleaseInfoProvider - Filter null versions before sorting in GarbageCollector - Normalize tar directory entries with ResolveEntryDestPath - Filter WindowsDesktop component on non-Windows in RuntimeUpdateCommand - Handle @ at position 0 in RuntimeInstallCommand.ParseComponentSpec - Remove unused allowPrerelease/rollForward params from UpdateGlobalJson - Add null guard in DotnetArchiveExtractor.Commit for _archivePath - Write manifest before deleting directories in GarbageCollector Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Add install spec even when version is already installed via another channel - Show message when uninstall removes no files due to shared subcomponents - Fix runtime install incorrectly tagged as GlobalJson source - Fix broken Spectre.Console markup in uninstall source mismatch message - Explicit version/channel now takes priority over global.json in install workflow - Only set GlobalJsonPath on install request when source is actually GlobalJson Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
This PR extends dotnetup to support update/uninstall flows and introduces manifest-driven garbage collection (GC) to remove unreferenced installed subcomponent folders. It refactors installation tracking from a legacy flat list into a structured manifest containing install specs (what the user requested) and installations (what’s actually installed + subcomponents).
Changes:
- Introduces a new dotnetup shared manifest schema (
DotnetupManifestData) withInstallSpec+Installationrecords and JSON source-gen updates. - Adds
update/uninstallcommand paths and shared workflows (UpdateWorkflow,UninstallWorkflow) that operate over install specs and run GC. - Adds GC + archive extraction subcomponent tracking (
GarbageCollector,SubcomponentResolver, extractor changes) and updates tests/docs accordingly.
Reviewed changes
Copilot reviewed 57 out of 57 changed files in this pull request and generated 8 comments.
Show a summary per file
| File | Description |
|---|---|
| test/dotnetup.Tests/SubcomponentResolverTests.cs | Unit tests for subcomponent path resolution. |
| test/dotnetup.Tests/SdkUninstallCommandTests.cs | Tests uninstall source filtering behavior. |
| test/dotnetup.Tests/RuntimeInstallTests.cs | Updates manifest tracking assertions for new schema. |
| test/dotnetup.Tests/ManifestTests.cs | Tests manifest schema, roundtrip, and operations. |
| test/dotnetup.Tests/ListCommandTests.cs | Updates list tests for new ListData output. |
| test/dotnetup.Tests/LibraryTests.cs | Normalizes file header/BOM change. |
| test/dotnetup.Tests/GarbageCollectorTests.cs | Tests GC behavior against manifest + filesystem. |
| test/dotnetup.Tests/DnupE2Etest.cs | Updates E2E manifest assertions to new format. |
| test/dotnetup.Tests/ChannelVersionResolverTests.cs | Fixes channel parsing expectations and assertions. |
| src/Installer/dotnetup/WindowsPathHelper.cs | Minor registry key handling refactor. |
| src/Installer/dotnetup/SpectreProgressTarget.cs | Makes progress completion idempotent. |
| src/Installer/dotnetup/Parser.cs | Registers new SDK uninstall command parser. |
| src/Installer/dotnetup/InstallerOrchestratorSingleton.cs | Records install specs/installations; adds root safety guard. |
| src/Installer/dotnetup/IDotnetupManifest.cs | Replaces legacy installed-version APIs with spec/installation APIs. |
| src/Installer/dotnetup/IDotnetInstallManager.cs | Simplifies UpdateGlobalJson signature. |
| src/Installer/dotnetup/GlobalJsonContents.cs | Makes SDK section public for JSON deserialization. |
| src/Installer/dotnetup/GlobalJsonChannelResolver.cs | Resolves channel from global.json + rollForward policy. |
| src/Installer/dotnetup/GarbageCollector.cs | GC implementation: prune manifest + delete orphan dirs. |
| src/Installer/dotnetup/DotnetupSharedManifest.cs | New manifest read/write + CRUD for specs/installations. |
| src/Installer/dotnetup/DotnetupManifestJsonContext.cs | Updates JSON source-gen types/options for new schema. |
| src/Installer/dotnetup/DotnetupManifestData.cs | Defines schema types: roots/specs/installations/sources. |
| src/Installer/dotnetup/DotnetInstallManager.cs | Implements global.json SDK version replacement logic. |
| src/Installer/dotnetup/Constants.cs | Renames global mutex identifier. |
| src/Installer/dotnetup/CommonOptions.cs | Makes options readonly; adds --source option. |
| src/Installer/dotnetup/Commands/Shared/UpdateWorkflow.cs | Shared update flow across specs + optional global.json update. |
| src/Installer/dotnetup/Commands/Shared/UninstallWorkflow.cs | Shared uninstall flow + GC + reporting. |
| src/Installer/dotnetup/Commands/Shared/InstallWorkflow.cs | Plumbs install source info into install requests. |
| src/Installer/dotnetup/Commands/Shared/InstallWalkthrough.cs | Ensures explicit channel overrides global.json channel. |
| src/Installer/dotnetup/Commands/Shared/InstallExecutor.cs | Switches to exception-based failure handling; improves path checks. |
| src/Installer/dotnetup/Commands/Sdk/Update/SdkUpdateCommandParser.cs | Adds root update variant + options wiring. |
| src/Installer/dotnetup/Commands/Sdk/Update/SdkUpdateCommand.cs | Implements sdk update using shared workflow. |
| src/Installer/dotnetup/Commands/Sdk/Uninstall/SdkUninstallCommandParser.cs | Adds sdk uninstall command parser. |
| src/Installer/dotnetup/Commands/Sdk/Uninstall/SdkUninstallCommand.cs | Implements sdk uninstall using shared workflow. |
| src/Installer/dotnetup/Commands/Sdk/SdkCommandParser.cs | Wires sdk uninstall into sdk command group. |
| src/Installer/dotnetup/Commands/Sdk/Install/SdkInstallCommandParser.cs | Reuses common options instead of local aliases. |
| src/Installer/dotnetup/Commands/Sdk/Install/SdkInstallCommand.cs | Uses GlobalJsonChannelResolver for channel derivation. |
| src/Installer/dotnetup/Commands/Runtime/Update/RuntimeUpdateCommandParser.cs | Adds runtime update command parser. |
| src/Installer/dotnetup/Commands/Runtime/Update/RuntimeUpdateCommand.cs | Updates runtime components via shared workflow. |
| src/Installer/dotnetup/Commands/Runtime/Uninstall/RuntimeUninstallCommandParser.cs | Adds runtime uninstall command parser. |
| src/Installer/dotnetup/Commands/Runtime/Uninstall/RuntimeUninstallCommand.cs | Implements runtime uninstall using shared workflow. |
| src/Installer/dotnetup/Commands/Runtime/RuntimeCommandParser.cs | Wires runtime update/uninstall into runtime group. |
| src/Installer/dotnetup/Commands/Runtime/Install/RuntimeInstallCommand.cs | Validates @ component spec edge case. |
| src/Installer/dotnetup/Commands/List/ListCommand.cs | Changes list output to include specs + validity metadata. |
| src/Installer/dotnetup/Commands/Info/InfoCommand.cs | Extends info JSON to include install specs too. |
| src/Installer/Microsoft.Dotnet.Installation/Internal/UpdateChannel.cs | Adds Matches() for channel-to-version matching. |
| src/Installer/Microsoft.Dotnet.Installation/Internal/SubcomponentResolver.cs | Adds subcomponent classification and depth mapping. |
| src/Installer/Microsoft.Dotnet.Installation/Internal/ScopedMutex.cs | Makes mutex re-entrant and throws on timeout. |
| src/Installer/Microsoft.Dotnet.Installation/Internal/DownloadProgressReporter.cs | Improves human-readable byte formatting precision. |
| src/Installer/Microsoft.Dotnet.Installation/Internal/DotnetupUtilities.cs | Makes exe suffix readonly; fixes Unix path equality semantics. |
| src/Installer/Microsoft.Dotnet.Installation/Internal/DotnetReleaseInfoProvider.cs | Reuses resolver instance instead of recreating it. |
| src/Installer/Microsoft.Dotnet.Installation/Internal/DotnetInstaller.cs | Removes unused manifest allocation. |
| src/Installer/Microsoft.Dotnet.Installation/Internal/DotnetInstall.cs | Adds resolved version + install source/global.json path options. |
| src/Installer/Microsoft.Dotnet.Installation/Internal/DotnetArchiveExtractor.cs | Tracks extracted subcomponents; strengthens extraction safety. |
| src/Installer/Microsoft.Dotnet.Installation/Internal/DotnetArchiveDownloader.cs | Reports cached download progress using bytes. |
| src/Installer/Microsoft.Dotnet.Installation/DownloadProgress.cs | Fixes percent calc when total bytes is 0. |
| documentation/general/dotnetup/installation-tracking.md | Documents intended tracking/GC approach and schema. |
| .vscode/mcp.json | Renames MCP server key. |
You can also share your feedback on Copilot code review. Take the survey.
Add a new --untracked flag to sdk install and runtime install commands that installs .NET without recording in the dotnetup manifest. When set: - Bypasses the untracked-installation guard (no error for pre-existing .NET artifacts not tracked by the manifest) - Skips AddInstallSpec() and AddInstallation() manifest writes - Still performs the actual download, extraction, and validation This enables scenarios where users want dotnetup to install .NET to a directory without taking ownership of it in the manifest. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The shared frameworks installed by restore-toolset are for testing only and should not be tracked in the dotnetup manifest. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
nagilson
left a comment
There was a problem hiding this comment.
Very excited to see this working! Please see my feedback which for now covers up until src/Installer/dotnetup/InstallerOrchestratorSingleton.cs
|
Is there a place to look where I can see why garbage collection is needed? Dnvm uses channels and installs. You have to manage those manually. Is this because of workloads? And how does it relate to the workloads fix that the ask already has? |
#52834 This spec should be helpful. The philosophy of how installs are managed here is somewhat different. The 'default' Update experience here (when not providing a specific install/channel) refers to updating every install based on their specified channels. We expect workload commands themselves to trigger the workload garbage collection after this. |
nagilson
left a comment
There was a problem hiding this comment.
I think there are some issues with the unix path handling:
Warning: Unrecognized subcomponent path './LICENSE.txt' in archive. This file will not be tracked by dotnetup.
Warning: Unrecognized subcomponent path './ThirdPartyNotices.txt' in archive. This file will not be tracked by dotnetup.
Warning: Unrecognized subcomponent path './dotnet' in archive. This file will not be tracked by dotnetup.
Warning: Unrecognized subcomponent path './host/' in archive. This file will not be tracked by dotnetup.
Warning: Unrecognized subcomponent path './host/fxr/' in archive. This file will not be tracked by dotnetup.
Warning: Unrecognized subcomponent path './host/fxr/9.0.13/' in archive. This file will not be tracked by dotnetup.
- Register InstallPathOption and ManifestPathOption in ListCommandParser so the list command accepts --install-path and --manifest-path flags - Plumb manifestPath through ListCommand to InstallationLister.GetListData so the list command reads from the correct manifest file - Remove --interactive false from BuildSdkUninstallArguments and BuildRuntimeUninstallArguments since uninstall commands don't register the --interactive option Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The GarbageCollector removes specific SDK version subdirectories (e.g., sdk/9.0.103/) but does not delete the parent sdk/ directory. The two failing tests incorrectly asserted Directory.Exists(sdkDir).BeFalse() on the parent directory. Fix both InstallThenUninstall_FolderIsCleanedUp and SdkUninstall_DoesNotRemoveExplicitlyInstalledRuntime to check that no version subdirectories remain inside sdk/ instead, matching the pattern used in GarbageCollectorTests. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ements - Move ArchiveInstallationValidator outside loop in ListCommand - Split ListCommand display into DisplayInstallRoots/DisplayInstallations - ParseComponentSpec throws DotnetInstallException instead of returning error tuples - Assert mutex held in GarbageCollectionRunner; conditional deletion output - Use DotnetupUtilities.PathsEqual and GetDotnetExeName in InstallerOrchestrator - Use GlobalJsonFileHelper for UTF-16 support in GlobalJsonChannelResolver/UpdateWorkflow - Extract GlobalJsonModifier class from DotnetInstallManager - Migrate ScopedMutex from ThreadLocal to AsyncLocal; rename s_heldMutexCounts - Add UseStringEnumConverter for manifest JSON serialization - Ensure ManifestPath is always a full path - Use FormattableString.Invariant in FormatBytes for culture safety - Add telemetry for GC delete failures with relative path tags - Add ListCommand support for --install-path/--manifest-path options - Skip extraction of existing subcomponents during archive install - Add InstallerOrchestratorTests (15 tests) - Add LifecycleEndToEndTests (4 E2E scenarios) - Add SubcomponentResolver backslash normalization tests - Add DotnetArchiveExtractor skip-existing tests - Rename test VersionMatchingWorksForExplicitChannelPatterns Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The list command reads install roots from the manifest, not from a CLI argument. Only ManifestPathOption is needed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Leverage existing ErrorCodeMapper infrastructure to classify IO exceptions (permission denied, sharing violation, disk full, etc.) instead of manually building activity event tags. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This is informational output, not a warning. Stderr should be reserved for actual warnings/errors. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
so the current preview test failure is because right now release-index is wrong and has so the sources of truth are misaligned. only some edges in the cdn return the wrong result, so the results may vary. |
|
/azp run |
|
cdn edge with invalid release data got purged |
|
Azure Pipelines successfully started running 2 pipeline(s). |
fixes the linux code I messed up by trying to call get full path on a fake path
nagilson
left a comment
There was a problem hiding this comment.
🎉 Let's get this merged! Thanks for working diligently on this, excited as this is a core feature set of dotnetup!
When configureEnvironment is true, TestEnvironment now also sets DOTNET_TESTHOOK_MANIFEST_PATH to the per-test manifest path. This prevents parallel tests from sharing the same default manifest file, which could cause flaky CI failures. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…paths On macOS, Path.GetTempPath() returns paths through /var which is a symlink to /private/var. When a child dotnetup process resolves its working directory via getcwd(), it gets the real /private/var path, causing GlobalJsonPath mismatches when the test tries to find install specs by path comparison. Fix by normalizing the temp root path at creation time. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Replace hardcoded macOS /var -> /private/var prefix with ResolveRealPath helper that walks path components and calls Directory.ResolveLinkTarget on each. This handles symlinks on any platform generically. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Spectre.Console emits ANSI color codes that break string assertions when error messages wrap across lines, causing color reset/re-apply sequences to be inserted mid-phrase. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The error message can wrap across console lines, inserting newlines mid-phrase. Collapse newlines to spaces before asserting substring containment. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
When stdout is piped/redirected, Spectre.Console defaults to 80-column wrapping which is not useful for non-interactive consumers. Set Profile.Width to int.MaxValue when Console.IsOutputRedirected is true. Reverts the ReplaceLineEndings workaround in the test since the product code fix handles it properly. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
/ba-g SDK tests are flaky in this branch |
No description provided.