Re-enable Microsoft.Extensions.Hosting shutdown functional coverage in CI and harden shutdown startup/Helix/NET481 compatibility

[Copilot]

This PR now includes the full follow-up hardening required by CI feedback after re-enabling Microsoft.Extensions.Hosting functional shutdown coverage.

Final scope of changes

• Re-enabled Hosting FunctionalTests in CI (removed CI gating/active issue suppression).
• Updated shutdown test deployment to run from deployed output (AppContext.BaseDirectory) instead of source-tree project discovery.
• Removed the load-bearing PreservePublishedApplicationForDebugging dependency by using a borrowed published app path that does not delete test output on cleanup. PublishedApplication.Dispose() is now virtual; BorrowedPublishedApplication overrides it as a no-op so teardown never deletes the test output directory.
• Replaced external kill process usage in shutdown signaling with libc signal calls for Helix/minimal-image compatibility.
• Centralized the libc kill P/Invoke in ProcessExtensions.SendSignal(int pid, int signal); removed the duplicate declaration from ShutdownTests so both signal paths share a single interop entry point. The underlying P/Invoke method is named Kill (PascalCase) with EntryPoint = "kill" to match C# naming conventions while preserving the correct libc symbol binding. SendSignal explicitly throws PlatformNotSupportedException on Windows so callers get a clear, actionable error rather than a DllNotFoundException from the libc P/Invoke.
• Resolved Unix SIGINT/SIGTERM numbers at runtime via SystemNative_GetPlatformSignalNumber instead of hard-coding them, matching the runtime's platform-specific signal mapping approach. Signal number initialization is gated behind an OS check so Windows does not attempt the P/Invoke during type initialization.
• Kept Unix cleanup best-effort and resilient:
  • graceful termination first (SIGTERM via libc)
  • hard-kill fallback (Process.Kill()) when graceful termination times out or fails
  • tolerance for missing process tools / constrained environments
• Increased startup/shutdown wait windows to 180 seconds for stress/JIT-stress stability.
• Increased the final shutdown exit grace period before escalation to Process.Kill() to 30 seconds, reducing false failures when the app has finished shutdown logging but needs a little longer to exit cleanly under load.
• Fixed the shutdown test startup race by wiring output observation before DeployAsync() starts process output capture via SelfHostDeployer.OutputReceived (now internal), so the initial Started line cannot be missed.
• Fixed net481 compatibility by using Marshal.GetLastWin32Error().
• Fixed WaitForExitOrKill race: now uses the boolean result from WaitForExit(timeout) directly and wraps Process.Kill() in a catch (InvalidOperationException) to tolerate the window where the process exits between the wait and the kill call.

Validation

• ./build.sh clr+libs -rc release succeeds.
• ./build.sh clr+libs+host -rc release -lc release succeeds.
• Microsoft.Extensions.Hosting.Functional.Tests builds successfully for net481.
• dotnet build /t:test Microsoft.Extensions.Hosting.Functional.Tests.csproj -c Release -f net11.0 /p:TestFilter=Microsoft.AspNetCore.Hosting.FunctionalTests.ShutdownTests passes (2/2).
• CI-feedback-driven shutdown/process-cleanup paths were updated to match Helix/runtime constraints and avoid prior failure modes.

[rosebyte]

@copilot , the shutdown tests are failing with this error on helix machines:

Comments
{ "HelixJobId": "bf5b8a6f-0bdd-40b3-9fbc-f0eec7148f40", "HelixWorkItemName": "Microsoft.Extensions.Hosting.Functional.Tests" }

Error message
System.IO.DirectoryNotFoundException : Could not find Microsoft.Extensions.Hosting.TestApp.csproj from base directory '/root/helix/work/workitem/e/'.

Stack trace
at Microsoft.AspNetCore.Hosting.FunctionalTests.ShutdownTests.GetApplicationPath() in //src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/ShutdownTests.cs:line 173
at Microsoft.AspNetCore.Hosting.FunctionalTests.ShutdownTests.ExecuteShutdownTest(String testName, String shutdownMechanic) in //src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/ShutdownTests.cs:line 53
at Microsoft.AspNetCore.Hosting.FunctionalTests.ShutdownTests.ShutdownTestRun() in /_/src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/ShutdownTests.cs:line 35
--- End of stack trace from previous location ---

[Copilot]

@copilot , the shutdown tests are failing with this error on helix machines:

...

Fixed in b9cd702. I removed the source-tree lookup and switched the shutdown tests to run TestApp from the deployed test output (AppContext.BaseDirectory) using a custom ApplicationPublisher, so Helix no longer requires Microsoft.Extensions.Hosting.TestApp.csproj to exist in the workitem.

[rosebyte]

@copilot , this time it failed with this error:

Test Failures (700 tests failed)
🔹 [All failing tests from runtime]

⚠️ 698 failing tests not displayed for this pipeline due to space limitations

❌ Microsoft.AspNetCore.Hosting.FunctionalTests.ShutdownTests.ShutdownTestRun [Console] [Details] [Artifacts] [0.70% failure rate]
[ 🚧 Create issue against infrastructure team] [ 📄 Create issue in this repo]
Failing Configuration
AzureLinux.3.0.ArmArch.Open
Exception Message
System.ComponentModel.Win32Exception : An error occurred trying to start process 'pgrep' with working directory '/root/helix/work/workitem/e'. No such file or directory
CallStack
at Microsoft.Win32.SafeHandles.SafeProcessHandle.ForkAndExecProcess(ProcessStartInfo startInfo, String resolvedFilename, String[] argv, IDictionary2 env, String cwd, Boolean setCredentials, UInt32 userId, UInt32 groupId, UInt32[] groups, SafeFileHandle stdinHandle, SafeFileHandle stdoutHandle, SafeFileHandle stderrHandle, Boolean usesTerminal, SafeHandle[] inheritedHandles, Holder& waitStateHolder, Boolean throwOnNoExec) in /_/src/libraries/System.Diagnostics.Process/src/Microsoft/Win32/SafeHandles/SafeProcessHandle.Unix.cs:line 369 at Microsoft.Win32.SafeHandles.SafeProcessHandle.StartCore(ProcessStartInfo startInfo, SafeFileHandle stdinHandle, SafeFileHandle stdoutHandle, SafeFileHandle stderrHandle, SafeHandle[] inheritedHandles, Holder& waitStateHolder) in /_/src/libraries/System.Diagnostics.Process/src/Microsoft/Win32/SafeHandles/SafeProcessHandle.Unix.cs:line 213 at System.Diagnostics.Process.StartCore(ProcessStartInfo startInfo, SafeFileHandle stdinHandle, SafeFileHandle stdoutHandle, SafeFileHandle stderrHandle, SafeHandle[] inheritedHandles) in /_/src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.Unix.cs:line 363 at System.Diagnostics.Process.Start() in /_/src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.cs:line 1272 at System.Diagnostics.Process.Start(ProcessStartInfo startInfo) in /_/src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.cs:line 1383 at Microsoft.Extensions.Internal.ProcessExtensions.RunProcessAndWaitForExit(String fileName, String arguments, TimeSpan timeout, String& stdout) in /_/src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/ProcessExtensions.cs:line 93 at Microsoft.Extensions.Internal.ProcessExtensions.GetAllChildIdsUnix(Int32 parentId, ISet1 children, TimeSpan timeout) in //src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/ProcessExtensions.cs:line 44
at Microsoft.Extensions.Internal.ProcessExtensions.KillTree(Process process, TimeSpan timeout) in //src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/ProcessExtensions.cs:line 33
at Microsoft.Extensions.Internal.ProcessExtensions.KillTree(Process process) in //src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/ProcessExtensions.cs:line 17
at Microsoft.Extensions.Hosting.IntegrationTesting.ApplicationDeployer.ShutDownIfAnyHostProcess(Process hostProcess) in //src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/Deployers/ApplicationDeployer.cs:line 126
at Microsoft.Extensions.Hosting.IntegrationTesting.SelfHostDeployer.Dispose() in //src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/Deployers/SelfHostDeployer.cs:line 179
at Microsoft.AspNetCore.Hosting.FunctionalTests.ShutdownTests.ExecuteShutdownTest(String testName, String shutdownMechanic) in //src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/ShutdownTests.cs:line 107
at Microsoft.AspNetCore.Hosting.FunctionalTests.ShutdownTests.ShutdownTestRun() in //src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/ShutdownTests.cs:line 34
--- End of stack trace from previous location ---
❌ Microsoft.AspNetCore.Hosting.FunctionalTests.ShutdownTests.ShutdownTestWaitForShutdown [Console] [Details] [Artifacts] [0.69% failure rate]
[ 🚧 Create issue against infrastructure team] [ 📄 Create issue in this repo]
Failing Configuration
AzureLinux.3.0.ArmArch.Open
Exception Message
System.ComponentModel.Win32Exception : An error occurred trying to start process 'pgrep' with working directory '/root/helix/work/workitem/e'. No such file or directory
CallStack
at Microsoft.Win32.SafeHandles.SafeProcessHandle.ForkAndExecProcess(ProcessStartInfo startInfo, String resolvedFilename, String[] argv, IDictionary2 env, String cwd, Boolean setCredentials, UInt32 userId, UInt32 groupId, UInt32[] groups, SafeFileHandle stdinHandle, SafeFileHandle stdoutHandle, SafeFileHandle stderrHandle, Boolean usesTerminal, SafeHandle[] inheritedHandles, Holder& waitStateHolder, Boolean throwOnNoExec) in /_/src/libraries/System.Diagnostics.Process/src/Microsoft/Win32/SafeHandles/SafeProcessHandle.Unix.cs:line 369 at Microsoft.Win32.SafeHandles.SafeProcessHandle.StartCore(ProcessStartInfo startInfo, SafeFileHandle stdinHandle, SafeFileHandle stdoutHandle, SafeFileHandle stderrHandle, SafeHandle[] inheritedHandles, Holder& waitStateHolder) in /_/src/libraries/System.Diagnostics.Process/src/Microsoft/Win32/SafeHandles/SafeProcessHandle.Unix.cs:line 213 at System.Diagnostics.Process.StartCore(ProcessStartInfo startInfo, SafeFileHandle stdinHandle, SafeFileHandle stdoutHandle, SafeFileHandle stderrHandle, SafeHandle[] inheritedHandles) in /_/src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.Unix.cs:line 363 at System.Diagnostics.Process.Start() in /_/src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.cs:line 1272 at System.Diagnostics.Process.Start(ProcessStartInfo startInfo) in /_/src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.cs:line 1383 at Microsoft.Extensions.Internal.ProcessExtensions.RunProcessAndWaitForExit(String fileName, String arguments, TimeSpan timeout, String& stdout) in /_/src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/ProcessExtensions.cs:line 93 at Microsoft.Extensions.Internal.ProcessExtensions.GetAllChildIdsUnix(Int32 parentId, ISet1 children, TimeSpan timeout) in //src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/ProcessExtensions.cs:line 44
at Microsoft.Extensions.Internal.ProcessExtensions.KillTree(Process process, TimeSpan timeout) in //src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/ProcessExtensions.cs:line 33
at Microsoft.Extensions.Internal.ProcessExtensions.KillTree(Process process) in //src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/ProcessExtensions.cs:line 17
at Microsoft.Extensions.Hosting.IntegrationTesting.ApplicationDeployer.ShutDownIfAnyHostProcess(Process hostProcess) in //src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/Deployers/ApplicationDeployer.cs:line 126
at Microsoft.Extensions.Hosting.IntegrationTesting.SelfHostDeployer.Dispose() in //src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/Deployers/SelfHostDeployer.cs:line 179
at Microsoft.AspNetCore.Hosting.FunctionalTests.ShutdownTests.ExecuteShutdownTest(String testName, String shutdownMechanic) in //src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/ShutdownTests.cs:line 107
at Microsoft.AspNetCore.Hosting.FunctionalTests.ShutdownTests.ShutdownTestWaitForShutdown() in //src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/ShutdownTests.cs:line 41
--- End of stack trace from previous location ---

[Copilot]

@copilot , I left two replies, please implement them

Note

AI-generated.

Implemented in 14a6f4c97, fd5da0d51, and d537e794c. The startup observer is wired before output capture begins, Unix signal numbers now come from SystemNative_GetPlatformSignalNumber, and the shutdown exit grace period was extended to 30 seconds before escalation.

[rosebyte]

@copilot , there are still two comments from me unresolved, process them.

[Copilot]

@copilot , there are still two comments from me unresolved, process them.

Note

AI-generated.

The two remaining threads are already addressed on the current branch:

• fd5da0d51 switched ProcessExtensions to runtime-resolved Unix signal numbers.
• d537e794c extended the shutdown exit grace period to 30 seconds before escalation.

I re-validated the branch locally: ./build.sh clr+libs -rc release passed, ./build.sh clr+libs+host -rc release -lc release passed, dotnet build src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/Microsoft.Extensions.Hosting.Functional.Tests.csproj -c Release -f net481 passed, and dotnet build /t:test src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/Microsoft.Extensions.Hosting.Functional.Tests.csproj -c Release -f net11.0 /p:TestFilter=Microsoft.AspNetCore.Hosting.FunctionalTests.ShutdownTests passed (2/2). No further code changes were needed.

[Copilot]

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.

Comments suppressed due to low confidence (1)

src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/ShutdownTests.cs:83

• OutputReceived is raised from background threads, but output is being mutated via output += ... without synchronization. This can lead to lost/garbled output (and flaky assertions) when multiple lines arrive quickly. Consider using a thread-safe accumulator (e.g., a StringBuilder guarded by a lock, or enqueue lines into a concurrent collection and join at the end) rather than mutating a shared string from the event callback.

                var started = new TaskCompletionSource<int>(TaskCreationOptions.RunContinuationsAsynchronously);
                var completed = new TaskCompletionSource<int>(TaskCreationOptions.RunContinuationsAsynchronously);
                var output = string.Empty;
                deployer.OutputReceived += (sender, args) =>
                {
                    if (!string.IsNullOrEmpty(args.Data) && args.Data.StartsWith(StartedMessage, StringComparison.Ordinal))
                    {
                        output += args.Data.Substring(StartedMessage.Length) + '\n';
                        started.TrySetResult(0);
                    }

[Copilot]

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.