Fail fast when the MSBuildCache graph build hits real build errors

azure-pipelines.yml

@@ -291,7 +291,27 @@ stages:
 
         # MSBuildCache requires a direct static graph build; Arcade's outer Build.proj discovers its projects too late
         # for the project-cache plugin. Keep the regular Arcade build below as the correctness fallback during rollout.
+        #
+        # This step reports one of three outcomes, and the fallback below keys off it:
+        #   succeeded              - the cache supplied the outputs, the Arcade 'Build' step is skipped.
+        #   succeeded with warning - the cache build broke for a reason the fallback can plausibly fix, so the
+        #                            Arcade 'Build' step runs. The step deliberately stays green: a step marked
+        #                            'SucceededWithIssues' makes the job PartiallySucceeded, which Azure Repos
+        #                            build-validation policies treat as a failure and would block the merge on a
+        #                            run that ultimately builds fine (same reason as _MacOSNonBlockingTrailer).
+        #   failed                 - the sources genuinely do not build, see the classification below.
         - pwsh: |
+            # This step no longer sets continueOnError, so its result is meaningful and an *unexpected*
+            # terminating error (Tee-Object, log reading, the classification below) must not fail the job:
+            # that is an infrastructure failure, which belongs on the fallback path exactly like a crash of
+            # the build itself. `exit` is flow control rather than an error, so the deliberate `exit 1` for a
+            # classified source break further down still fails the step as intended.
+            trap {
+              Write-Host "##vso[task.setvariable variable=MSBuildCacheBuildSucceeded]false"
+              Write-Host "##vso[task.logissue type=warning]The MSBuildCache step itself failed unexpectedly ($($_.Exception.Message)); continuing with the regular Arcade build."
+              exit 0
+            }
+
             $pwsh = Join-Path $PSHOME "pwsh.exe"
             # Azure Pipeline cache entries are immutable. The batched main-merge stage is the only remote publisher;
             # keeping this canary read-only prevents manual or scheduled runs from racing with it.
@@ -321,29 +341,90 @@ stages:
               "/p:MSBuildCacheLogDirectory=$(Agent.TempDirectory)\MSBuildCache"
             )
 
+            # Tee MSBuild's console output (errors included) so the failure can be classified below without
+            # re-running anything. stderr is deliberately left unredirected, exactly as before, so PowerShell's
+            # native-command error handling in this step is unchanged.
+            $logPath = Join-Path "$(Agent.TempDirectory)" "MSBuildCacheGraphBuild.log"
+
             $previousPSNativeCommandUseErrorActionPreference = $PSNativeCommandUseErrorActionPreference
             try {
               # Unlike Start-Process -Wait, direct native invocation does not wait for detached telemetry descendants.
               $PSNativeCommandUseErrorActionPreference = $false
-              & $pwsh @arguments
+              & $pwsh @arguments | Tee-Object -FilePath $logPath
               $exitCode = $LASTEXITCODE
             }
             finally {
               $PSNativeCommandUseErrorActionPreference = $previousPSNativeCommandUseErrorActionPreference
             }
 
-            if ($exitCode -ne 0) {
-              Write-Host "##vso[task.logissue type=warning]MSBuildCache canary failed with exit code $exitCode; continuing with the regular Arcade build."
-              Write-Host "##vso[task.setvariable variable=MSBuildCacheBuildSucceeded]false"
-            }
-            else {
+            if ($exitCode -eq 0) {
               Write-Host "##vso[task.setvariable variable=MSBuildCacheBuildSucceeded]true"
+              Write-Host "MSBuildCache produced the build outputs. The Arcade 'Build' step will be skipped."
+              exit 0
             }
 
+            Write-Host "##vso[task.setvariable variable=MSBuildCacheBuildSucceeded]false"
+
+            # Decide whether the Arcade fallback can still reach a different result than this build did.
+            #
+            # A cache hit only materializes outputs that were stored earlier; it never runs the compiler. Every
+            # "error XXnnnn:" below was therefore emitted by a project that actually executed - exactly as the
+            # fallback would execute it - so running the whole solution through Arcade again only reproduces the
+            # same errors several minutes later. Fail here instead, with the errors surfaced on this step.
+            #
+            # Three classes of failure stay on the fallback path because the non-cached build can legitimately
+            # behave differently: anything MSBuildCache itself reports (plugin, cache or file-access problems),
+            # restore failures (NU*, which are often transient), and the engine/graph failures classified below -
+            # whether or not they reached MSBuild's error summary.
+            $log = if (Test-Path $logPath) { @(Get-Content -LiteralPath $logPath) } else { @() }
+
+            $reportedErrorCount = -1
+            # Matches MSBuild's English summary. A localized agent leaves the count at -1, which falls through
+            # to the fallback below - the safe direction, and the same outcome as a build that never reached a
+            # summary at all.
+            $errorSummary = @($log | Select-String -Pattern '(\d+)\s+Error\(s\)') | Select-Object -Last 1
+            if ($errorSummary) {
+              $reportedErrorCount = [int]$errorSummary.Matches[0].Groups[1].Value
+            }
+
+            $errorLines = @($log | Where-Object { $_ -match '(?i):\s*error(\s+[a-z]+\d+)?\s*:\s' } | Select-Object -Unique)
+            $fallbackMayHelp = '(?i)MSBuildCache|ProjectCache|project cache|cache plugin|CacheClient|file access|\bNU\d{4}\b'
+            $mayHelpLines = @($errorLines | Where-Object { $_ -match $fallbackMayHelp })
+
+            # Some failures do reach the error summary but still are not source breaks the fallback would
+            # reproduce, so they must not fail fast. These are matched against the whole log rather than against
+            # $errorLines, because their diagnostic detail usually continues on following lines that are not
+            # themselves formatted as errors:
+            #   - engine crashes and OOM (MSB4166 "exited prematurely", MSB0001/MSB1025 internal errors,
+            #     MSB4017 logger failure). The node died before the projects it owned could report their real
+            #     result, so the Arcade invocation can legitimately succeed where this one did not.
+            #   - MSB4260, a project reference that cannot be resolved with a static graph. That is a
+            #     limitation of the /graph switch this step passes; the Arcade fallback does not build the
+            #     graph statically, so it can resolve the same reference and succeed.
+            #   - MSBuildCache's own duplicate-output diagnostic, worded "Node ... produced output ... which was
+            #     already produced by another node ...". It carries no plugin or cache token of its own, so
+            #     without this marker it reads as an ordinary build error and would defeat the intent that
+            #     everything the plugin reports goes to the fallback.
+            #   - transient file locks (MSB3021/MSB3027, "used by another process"). Copy contention is timing
+            #     dependent, and copying is heavier here because the cache uses copy rather than hardlink
+            #     semantics, so a differently scheduled build can legitimately succeed.
+            $fallbackCanDiffer = '(?i)\bMSB4166\b|\bMSB0001\b|\bMSB1025\b|\bMSB4017\b|\bMSB4260\b|\bMSB3021\b|\bMSB3027\b|exited prematurely|OutOfMemoryException|already produced by another node|being used by another process'
+            $fallbackCanDifferLines = @($log | Where-Object { $_ -match $fallbackCanDiffer })
+
+            if ($reportedErrorCount -gt 0 -and $errorLines.Count -gt 0 -and $mayHelpLines.Count -eq 0 -and $fallbackCanDifferLines.Count -eq 0) {
+              foreach ($line in @($errorLines | Select-Object -First 20)) {
+                Write-Host "##vso[task.logissue type=error]$line"
+              }
+
+              Write-Host "The solution does not build. The Arcade 'Build' step would report the same $reportedErrorCount error(s), so the build is failed here instead of repeating it."
+              Write-Host "If you believe an error above is an artifact of a cached dependency rather than a source break, the cache diagnostics for this run are published under artifacts/log/$(_BuildConfig)/MSBuildCache."
+              exit 1
+            }
+
+            Write-Host "##vso[task.logissue type=warning]MSBuildCache build failed with exit code $exitCode without a build error the fallback would reproduce; continuing with the regular Arcade build."
             exit 0
           displayName: Build solution graph with MSBuildCache
           condition: and(succeeded(), or(and(eq(variables['Build.Reason'], 'PullRequest'), ne(variables['System.PullRequest.IsFork'], 'True')), and(ne(variables['Build.Reason'], 'PullRequest'), eq(variables['Build.SourceBranch'], 'refs/heads/main'))))
-          continueOnError: true
           env:
             SYSTEM_ACCESSTOKEN: $(System.AccessToken)
             # Must match the seeding stage exactly. MSBuildCache normalizes observed input paths against
@@ -362,6 +443,15 @@ stages:
         # consumed by the acceptance tests. Run those phases without the Build phase when the cache supplied the
         # compiled outputs. Keep this non-blocking so any failure switches the regular Arcade build back on.
         - pwsh: |
+            # As above: with continueOnError gone, an unexpected terminating error here would fail the job
+            # instead of handing over to the Arcade build. This step has no deliberate failure path - every
+            # outcome either uses the cached outputs or falls back - so any error routes to the fallback.
+            trap {
+              Write-Host "##vso[task.setvariable variable=MSBuildCacheBuildSucceeded]false"
+              Write-Host "##vso[task.logissue type=warning]Preparing cached outputs failed unexpectedly ($($_.Exception.Message)); continuing with the regular Arcade build."
+              exit 0
+            }
+
             Write-Host "##vso[task.setvariable variable=MSBuildCacheBuildSucceeded]false"
 
             $pwsh = Join-Path $PSHOME "pwsh.exe"
@@ -393,16 +483,17 @@ stages:
             }
 
             if ($exitCode -ne 0) {
+              # Same reasoning as the graph build above: staying green keeps a run that the fallback repairs from
+              # being reported as PartiallySucceeded and blocking the merge.
               Write-Host "##vso[task.logissue type=warning]Preparing cached outputs failed with exit code $exitCode; continuing with the regular Arcade build."
-            }
-            else {
-              Write-Host "##vso[task.setvariable variable=MSBuildCacheBuildSucceeded]true"
+              exit 0
             }
 
+            Write-Host "##vso[task.setvariable variable=MSBuildCacheBuildSucceeded]true"
+            Write-Host "Cached build outputs are restored, signed and packed. The Arcade 'Build' step will be skipped."
             exit 0
           displayName: Prepare and pack cached build outputs
           condition: and(succeeded(), eq(variables['MSBuildCacheBuildSucceeded'], 'true'))
-          continueOnError: true
 
         # Publish only the project-level diagnostics; CacheClient.log and the OAuth-bearing cache environment stay
         # outside published artifacts. A successful cache build supplies the outputs consumed by the test steps.
@@ -475,6 +566,8 @@ stages:
               New-ItemProperty -Path "HKLM:\SOFTWARE\Microsoft\Windows\Windows Error Reporting\LocalDumps" -Name "DumpFolder" -Value "$(Build.SourcesDirectory)\artifacts\CrashDumps" -PropertyType ExpandString -Force
               New-ItemProperty -Path "HKLM:\SOFTWARE\Microsoft\Windows\Windows Error Reporting\LocalDumps" -Name "DumpCount" -Value 10 -PropertyType DWord -Force
 
+        # Fallback build. Skipped when the MSBuildCache steps above already produced, signed and packed the
+        # outputs; also skipped when they failed with build errors, because this step would only reproduce them.
         - script: eng\common\CIBuild.cmd
             -configuration $(_BuildConfig)
             -prepareMachine

docs/dev-guide.md

@@ -86,7 +86,13 @@ For more information about all the different options available, supply the argum
 
 ### MSBuildCache
 
-The Windows PR pipeline experimentally runs [MSBuildCache](https://github.com/microsoft/MSBuildCache). The cache-aware build uses Arcade's `eng/common/msbuild.ps1` launcher to invoke the solution directly because Arcade's outer `Build.proj` discovers projects dynamically and cannot expose the repository's static project graph to the cache plugin. PR builds consume the immutable Azure Pipeline cache read-only, and fork PRs skip this step because they do not receive the required token scope. When the cache build succeeds, the pipeline runs the remaining Arcade restore, sign, and pack phases without rebuilding, then uses the cached outputs for the test steps. When the cache build or preparation phases fail, the pipeline preserves the cache diagnostics, removes partial outputs, and runs the regular Arcade build as a fallback.
+The Windows PR pipeline experimentally runs [MSBuildCache](https://github.com/microsoft/MSBuildCache). The cache-aware build uses Arcade's `eng/common/msbuild.ps1` launcher to invoke the solution directly because Arcade's outer `Build.proj` discovers projects dynamically and cannot expose the repository's static project graph to the cache plugin. PR builds consume the immutable Azure Pipeline cache read-only, and fork PRs skip this step because they do not receive the required token scope.
+
+The cache steps have three outcomes:
+
+- **The cache build succeeds.** The pipeline runs the remaining Arcade restore, sign, and pack phases without rebuilding, then uses the cached outputs for the test steps. The regular Arcade `Build` step is skipped.
+- **The cache build fails with errors the regular build would only reproduce.** A cache hit materializes previously stored outputs and never runs the compiler, so an ordinary build error came from a project that genuinely executed, exactly as the fallback would execute it. Re-running the whole solution through Arcade would report the same errors several minutes later, so the pipeline surfaces them on the cache step and fails the job there instead. Should such a verdict ever be wrong, the cache diagnostics published under `artifacts\log\<configuration>\MSBuildCache` show what the plugin did for that run.
+- **The cache build or the preparation phase fails for any other reason.** The pipeline preserves the cache diagnostics, removes partial outputs, and runs the regular Arcade build as a fallback. This covers anything MSBuildCache itself reports (plugin, cache, or file-access problems, including its duplicate-output diagnostic), `NU*` restore failures, engine crashes and OOM (`MSB4166`, `MSB0001`, `MSB1025`, `MSB4017`), `MSB4260` project references that cannot be resolved with a static graph (a `/graph` limitation the fallback does not have), transient file locks (`MSB3021`, `MSB3027`), an unexpected failure of the wrapper script itself, and any failure that never reached MSBuild's error summary. The cache step stays green in this case: marking it `SucceededWithIssues` would make the job `PartiallySucceeded`, which Azure Repos build-validation policies treat as a failure even when the fallback build then succeeds.
 
 Every merge to `main` that touches product build inputs runs a dedicated, batched seed stage for both Debug and Release. This is required: the cache fingerprints project inputs, so entries become stale whenever shared build inputs such as `global.json`, `eng/Versions.props`, or Arcade change. This stage is the only remote cache publisher; PR, manual, and nightly canary builds consume the cache read-only so they cannot race to publish immutable entries. Debug and Release use separate cache universes because configuration-independent projects can otherwise race while the two configurations publish in parallel.