Add wasm/CoreCLR libraries tests to outerloop CI#129436
Draft
radekdoulik wants to merge 5 commits into
Draft
Conversation
Add a scheduled outerloop leg that runs the slow [OuterLoop] library suites on Browser+CoreCLR (Chrome). There was previously no wasm/CoreCLR coverage in the outerloop pipeline, only CoreCLR desktop and Mono browser_wasm. To make the slow suites complete, enable the previously-excluded long-running suites and extend the Helix work-item timeouts - both conditionally for the outerloop (and 'all') TestScope, preserving the existing innerloop behavior: - eng/pipelines/libraries/outerloop.yml: new browser_wasm + coreclr job mirroring the Mono browser_wasm outerloop job. - src/libraries/tests.proj: gate the long-running suite exclusions behind a new RunLongRunningWasmTests property (true for outerloop/all). Nrbf stays unconditionally excluded. - src/tasks/HelixTestTasks/ComputeBatchTimeout.cs: parameterize the per-batch timeout (MinutesPerSuite/MinimumMinutes/MaximumMinutes); the defaults reproduce the previous behavior exactly. - src/libraries/sendtohelix-browser.targets: raise the batch timeout budget for outerloop, keeping the maximum below the AzDO job timeout. - src/libraries/sendtohelixhelp.proj: extend the non-batched browser+ CoreCLR work-item timeout for outerloop. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
|
Tagging subscribers to this area: @dotnet/area-infrastructure-libraries |
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds missing CI coverage for browser-wasm + CoreCLR by introducing an outerloop leg that runs slower [OuterLoop] library test suites on Chrome, and adjusts Helix batching/work-item timeout behavior so those suites can complete without impacting innerloop defaults.
Changes:
- Add a new outerloop pipeline job for browser_wasm + CoreCLR (Chrome) in libraries outerloop CI.
- Gate previously-excluded long-running browser/CoreCLR suites behind a new
RunLongRunningWasmTestsproperty enabled forTestScope=outerloop|all. - Parameterize and extend Helix timeout budgeting for batched browser test work-items (outerloop only), plus increase the non-batched browser/CoreCLR work-item timeout for outerloop.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| eng/pipelines/libraries/outerloop.yml | Adds a scheduled outerloop job to run browser-wasm CoreCLR library tests on Chrome with an increased job timeout. |
| src/libraries/tests.proj | Enables long-running browser/CoreCLR suites for outerloop/all via RunLongRunningWasmTests, while keeping the current default behavior for innerloop/local runs. |
| src/tasks/HelixTestTasks/ComputeBatchTimeout.cs | Makes per-batch Helix timeout calculation configurable via new MSBuild-settable parameters. |
| src/libraries/sendtohelix-browser.targets | Plumbs new per-batch timeout knobs into ComputeBatchTimeout, with larger budgets for outerloop/all. |
| src/libraries/sendtohelixhelp.proj | Increases browser/CoreCLR work-item timeout specifically for outerloop/all as a safety net for non-batched paths. |
Comment on lines
+64
to
+68
| // MinutesPerSuite minutes per suite to account for WASM startup overhead + test | ||
| // execution; MinimumMinutes floor handles the heaviest individual suites | ||
| // (e.g. Cryptography ~17m); capped at MaximumMinutes (kept below 24h to prevent | ||
| // hh format wrapping, and intended to stay under the AzDO job timeout). | ||
| int totalMinutes = Math.Min(MaximumMinutes, Math.Max(MinimumMinutes, count * MinutesPerSuite)); |
Member
Author
|
/azp run runtime-coreclr outerloop |
|
Azure Pipelines successfully started running 1 pipeline(s). |
Member
Author
|
/azp run runtime-libraries outerloop |
|
No pipelines are associated with this pull request. |
Member
Author
|
/azp run runtime-libraries outerloop |
|
No pipelines are associated with this pull request. |
The new browser_wasm + CoreCLR outerloop job was emitted unconditionally, so it ran in all four scheduled libraries-outerloop definitions (the umbrella plus the -windows/-linux/-osx platform splits), executing the slow wasm suites 4x per day. Gate it on includeLinuxOuterloop (browser_wasm builds on a Linux host) so it runs only in the umbrella and -linux definitions, matching how the existing Linux desktop CoreCLR suites are scoped. The job stays PR-runnable via /azp run since it is not gated on isRollingBuild. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Member
Author
|
/azp run runtime-libraries-coreclr outerloop-linux |
|
Azure Pipelines successfully started running 1 pipeline(s). |
The 6 long-running library suites enabled for the wasm/CoreCLR outerloop job were bin-packed alongside ~9 other suites per Helix batch, so a single slow suite could inflate a shared batch's wall-clock and risk a timeout. Add a SoloItems parameter to the GroupWorkItems task that forces matching work items into their own solo batch (the same path used for oversized items), and wire it up in sendtohelix-browser.targets with the _WasmSoloBatchSuite list mirroring the long-running exclusions in tests.proj. The list is a no-op in innerloop, where those suites are excluded from the test archive. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The slow [OuterLoop] library suites were killed mid-run: the per-suite xharness timeout (WasmXHarnessTestsTimeout) defaulted to 90 min for wasm/CoreCLR, so suites such as System.Memory.Tests and System.Xml.Linq.xNodeBuilder.Tests hit xharness exit code 71 at exactly 5401s, and three batches were still running when the AzDO job timed out. Coordinate the nested timeout layers (suite <= batch <= job), conditional on outerloop only so innerloop budgets are preserved: - tests.wasm.targets: per-suite xharness timeout 04:00:00 for outerloop wasm/CoreCLR (innerloop keeps 01:30:00 / 00:30:00). - sendtohelix-browser.targets: raise the per-batch Helix floor to 270 min (above the 240-min suite timeout, so a solo slow-suite batch is bounded by the suite's own timeout) and the cap to 300 min. - sendtohelix-browser.targets: gate the solo-batch list on outerloop and add System.Xml.Linq.xNodeBuilder.Tests (an outerloop-slow suite that still runs in innerloop). - outerloop.yml: raise the AzDO job timeout to 600 min so it outlasts the slowest batch plus build/queue overhead. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
3 tasks
Member
Author
|
/azp run runtime-libraries-coreclr outerloop-linux |
|
Azure Pipelines successfully started running 1 pipeline(s). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a scheduled outerloop leg that runs the slow
[OuterLoop]library suites on Browser + CoreCLR (Chrome). There was previously no wasm/CoreCLR coverage in the outerloop pipeline — only CoreCLR desktop and Monobrowser_wasm.To make the slow suites actually complete, this also enables the previously-excluded long-running suites and extends the Helix work-item timeouts — conditionally for the
outerloop(andall)TestScopeonly, preserving existing innerloop behavior.Changes
eng/pipelines/libraries/outerloop.yml— newbrowser_wasm+coreclrjob mirroring the Monobrowser_wasmouterloop job (WasmTestOnChrome, full non-smoke run,timeoutInMinutes: 420).src/libraries/tests.proj— gate the long-running suite exclusions (System.Linq,System.Memory,System.Private.Uri,System.Private.Xml,System.Runtime.Numerics,System.Text.Json) behind a newRunLongRunningWasmTestsproperty (truefor outerloop/all).System.Formats.Nrbfstays unconditionally excluded.src/tasks/HelixTestTasks/ComputeBatchTimeout.cs— parameterize the per-batch timeout (MinutesPerSuite/MinimumMinutes/MaximumMinutes); the defaults reproduce the previous behavior exactly.src/libraries/sendtohelix-browser.targets— raise the batch timeout budget for outerloop, keeping the maximum below the AzDO job timeout so the synchronous send-to-Helix wait fits within the job.src/libraries/sendtohelixhelp.proj— extend the non-batched browser + CoreCLR work-item timeout for outerloop (safety net for non-batched paths such as sample apps; the batched suites get their timeout fromComputeBatchTimeout).Notes / things to watch on the first runs
timeout (>30 min)), not crashes (tracked under [wasm][coreclr] libraries tests tracking issue #125495). Extending timeouts is the intended remedy, but a slow suite could still OOM/fail under heavier outerloop load — watch the first scheduled runs and, if any persistently crash, move them to the always-excluded "test failures" group.Draft for CI validation.
Note
This pull request was created with assistance from GitHub Copilot.