[Perf] Windows/x64: 7 Regressions on 6/24/2026 8:46:56 PM +00:00

[performanceautofiler]

Run Information

Name	Value
Architecture	x64
OS	Windows 10.0.22631
Queue	ViperWindows
Baseline	46b72c7c81592b429e062bacebe8d39f17362333
Compare	cd27dcec16016aa54be6a09962d1b509150cf99b
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Linq.Tests.Perf_Enumerable

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio
Max - Duration of single invocation 📝 - Benchmark Source ADX - Test Multi Config Graph	45.36 ns	106.98 ns	2.36	0.13	False
Min - Duration of single invocation 📝 - Benchmark Source ADX - Test Multi Config Graph	58.00 ns	101.21 ns	1.75	0.15	False
FirstWithPredicate_LastElementMatches - Duration of single invocation 📝 - Benchmark Source ADX - Test Multi Config Graph	32.01 ns	53.97 ns	1.69	0.09	False
AnyWithPredicate_LastElementMatches - Duration of single invocation 📝 - Benchmark Source ADX - Test Multi Config Graph	33.02 ns	55.42 ns	1.68	0.11	False
WhereAny_LastElementMatches - Duration of single invocation 📝 - Benchmark Source ADX - Test Multi Config Graph	40.67 ns	62.28 ns	1.53	0.03	False
All_AllElementsMatch - Duration of single invocation 📝 - Benchmark Source ADX - Test Multi Config Graph	30.72 ns	45.35 ns	1.48	0.06	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Linq.Tests.Perf_Enumerable*'

Details

System.Linq.Tests.Perf_Enumerable.Max(input: IEnumerable)

ETL Files

Histogram

JIT Disasms

System.Linq.Tests.Perf_Enumerable.Min(input: IEnumerable)

ETL Files

Histogram

JIT Disasms

System.Linq.Tests.Perf_Enumerable.FirstWithPredicate_LastElementMatches(input: IEnumerable)

ETL Files

Histogram

JIT Disasms

System.Linq.Tests.Perf_Enumerable.AnyWithPredicate_LastElementMatches(input: IEnumerable)

ETL Files

Histogram

JIT Disasms

System.Linq.Tests.Perf_Enumerable.WhereAny_LastElementMatches(input: IEnumerable)

ETL Files

Histogram

JIT Disasms

System.Linq.Tests.Perf_Enumerable.All_AllElementsMatch(input: IEnumerable)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

---

Run Information

Name	Value
Architecture	x64
OS	Windows 10.0.22631
Queue	ViperWindows
Baseline	46b72c7c81592b429e062bacebe8d39f17362333
Compare	cd27dcec16016aa54be6a09962d1b509150cf99b
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Linq.Tests.Perf_ImmutableArrayExtensions

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio
SequenceEqual - Duration of single invocation 📝 - Benchmark Source ADX - Test Multi Config Graph	47.18 ns	59.25 ns	1.26	0.22	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Linq.Tests.Perf_ImmutableArrayExtensions*'

Details

System.Linq.Tests.Perf_ImmutableArrayExtensions.SequenceEqual(input: ICollection)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

[LoopedBard3]

🔍 Automated Triage Analysis

Summary: Verified regression in Max(input: IEnumerable) — caused by commit 414b194 ("Add MemoryExtensions.Min/Max (#128306)"). 2 tests bisected to related culprit commits. ⚠️ Note: Historical data shows bimodal behavior — bisection result may be unreliable.

Finding Confidence: 3/5 (analysis accuracy)
Regression Confidence: 3/5 (likelihood of true regression)

Likely Cause: 414b194d3a5a6c133cd19edaad73232e29c2e3fd

🤖 Proposed Next Actions: Uncertain — flag for re-triage — 1 done, 1 dry-run (rule R5_look_again — see Full Analysis for the table). · 🦆 Re-analysis: ✅ concurs

📊 Full Analysis (click to expand)

Summary: 2 confirmed regressions in System.Linq Max (+136%) and Min (+75%) attributed to commit 414b194 ("Add MemoryExtensions.Min/Max"), which refactored LINQ Min/Max to delegate to new MemoryExtensions APIs. Finding confidence 4/5 (strong historical data, 300 runs), regression confidence 4/5 (direct code path change, step change with z=3.8/3.0). 1 additional regression (WhereAny +53%) and 4 borderline tests show bimodal historical behavior making attribution uncertain.

Issue Overview

• Issue: [Perf] Windows/x64: 7 Regressions on 6/24/2026 8:46:56 PM +00:00 #130047 - [Perf] Windows/x64: 7 Regressions on 6/24/2026 8:46:56 PM +00:00
• Date: 2026-06-24 | Queue: Windows.11.Amd64.Viper.Perf | OS/Arch: Windows/x64
• Commits: 46b72c7c8159 → cd27dcec1601 (26 commits)
• Labels: untriaged, perf-regression, os-windows, arch-x64, branch-refs/heads/main, runtime-coreclr, kind-micro, compilationmode-tiered, runkind-micro
• Historical Data: Available (300 runs per test, 1336 total runs examined)
• Runtime stack: Desktop + CoreCLR (implicit)

Test Analysis

Test Name	Baseline	Compare	Δ%	μ	σ	n	z	Thresh%	Noise?	Assessment
Max(input: IEnumerable)	45.36 ns	106.98 ns	+135.8%	64.98	11.01	300	3.82	33.9%	N	Step change, confirmed regression
Min(input: IEnumerable)	58.00 ns	101.21 ns	+74.5%	65.50	11.84	300	3.02	36.2%	N	Step change, confirmed regression
WhereAny_LastElementMatches(input: IEnumerable)	40.67 ns	62.28 ns	+53.1%	53.79	10.95	300	0.78	40.7%	N	Regression with bimodal history
FirstWithPredicate_LastElementMatches(input: IEnumerable)	32.01 ns	53.97 ns	+68.6%	46.25	11.72	300	0.66	50.7%	N	Borderline — bimodal oscillation
AnyWithPredicate_LastElementMatches(input: IEnumerable)	33.02 ns	55.42 ns	+67.8%	46.54	11.45	300	0.78	49.2%	N	Borderline — bimodal oscillation
All_AllElementsMatch(input: IEnumerable)	30.72 ns	45.35 ns	+47.6%	42.69	10.88	300	0.25	51.0%	N	Borderline — multi-modal history
SequenceEqual(input: ICollection)	47.18 ns	59.25 ns	+25.6%	57.77	3.85	300	0.39	13.3%	N	Borderline — baseline outlier artifact

Legend: μ=mean, σ=stddev, n=runs, z=z-score, Thresh%=adaptive threshold

Finding Confidence: 4/5 - Strong historical data (300 runs per test), sufficient after-runs (34), but bimodal before-series inflates CV and complicates thresholds for 5 of 7 tests.

Candidate Stack Relevance

Commit	Touched Paths (summary)	Relevance	Notes
414b194d3a5a	src/libraries/System.Linq/src/System/Linq/Max.cs, Min.cs, MaxMin.cs (removed), src/libraries/System.Private.CoreLib/src/System/MemoryExtensions.MinMax.cs (added)	Relevant	Directly modifies LINQ Min/Max implementation — BCL surface exercised by CoreCLR
37d45de0d2f8	src/libraries/System.Linq.Expressions/...	Shared	Expression.Compile change — unlikely to affect microbenchmarks
440e6cba935e	src/libraries/System.Private.Xml.Linq/...	LessLikely	XML LINQ only, not System.Linq
08f0e98fd6df	src/libraries/System.Text.Json/...	LessLikely	STJ writer change, unrelated

Confirmed Regressions

System.Linq.Tests.Perf_Enumerable.Max(input: IEnumerable) (+135.8%)

• Delta: 45.36 ns → 106.98 ns (+135.8%)
• Historical Context: μ=64.98 (σ=11.01, n=300); z-score=3.82; adaptive threshold=33.9%
• Assessment: Step change. Compare value (~107 ns) exceeds the entire prior historical range. All 34 after-runs cluster at 107–114 ns, representing a new sustained level never previously observed.
• Finding Confidence: 4/5 - Excellent historical data (300 runs), clear step change, z=3.82 well above significance. Bimodal before-series (high ~72 ns early, low ~44–56 ns recently) inflates CV to 16.9% but does not obscure the regression since compare exceeds all prior values.
• Regression Confidence: 4/5 - Direct code path hit (LINQ Max.cs modified), surgical commit, z=3.82, unprecedented compare value. Multiple plausible commits (26 in range) cap at 4.
• Likely Cause: 414b194d3a5a - "Add MemoryExtensions.Min/Max (Add MemoryExtensions.Min/Max #128306)"
  • Affected Areas: BCL / System.Linq / MemoryExtensions
  • Stack Relevance: Relevant
  • Rationale: This commit refactored LINQ's Max implementation by removing the shared MaxMin.cs helper (170 lines deleted) and modifying Max.cs (+7 -26 lines) to delegate to new MemoryExtensions.MinMax.cs APIs (+351 lines). The new code path through MemoryExtensions likely introduces overhead for IEnumerable inputs that are not Span-backed, as the MemoryExtensions path may involve additional type-checking, interface dispatch, or boxing. The timing aligns perfectly with the regression date (2026-06-24).
  • Candidates Considered:
    • 414b194d3a5a - Direct modification of LINQ Max/Min code path
    • 37d45de0d2f8 - Expression.Compile change, unlikely
• Verification Status: Not verified — bisection recommended to confirm

System.Linq.Tests.Perf_Enumerable.Min(input: IEnumerable) (+74.5%)

• Delta: 58.00 ns → 101.21 ns (+74.5%)
• Historical Context: μ=65.50 (σ=11.84, n=300); z-score=3.02; adaptive threshold=36.2%
• Assessment: Step change. Compare value (~101 ns) exceeds the entire prior historical range (~44–79 ns). All 34 after-runs tightly clustered at 101–107 ns — unprecedented and persistent.
• Finding Confidence: 4/5 - Same strong data quality as Max. z=3.02 confirms statistical significance despite bimodal before-series (CV=18.1%).
• Regression Confidence: 4/5 - Same direct code path hit. Min.cs modified identically to Max.cs in the same commit.
• Likely Cause: 414b194d3a5a - "Add MemoryExtensions.Min/Max (Add MemoryExtensions.Min/Max #128306)"
  • Affected Areas: BCL / System.Linq / MemoryExtensions
  • Stack Relevance: Relevant
  • Rationale: Same commit modified Min.cs identically to Max.cs (+7 -26 lines), redirecting through the new MemoryExtensions path. Both Min and Max show the same pattern — step change to values exceeding all prior history — strongly corroborating shared root cause.
  • Candidates Considered:
    • 414b194d3a5a - Direct modification of LINQ Min code path
• Verification Status: Not verified — bisection recommended to confirm

System.Linq.Tests.Perf_Enumerable.WhereAny_LastElementMatches(input: IEnumerable) (+53.1%)

• Delta: 40.67 ns → 62.28 ns (+53.1%)
• Historical Context: μ=53.79 (σ=10.95, n=300); z-score=0.78; adaptive threshold=40.7%
• Assessment: Regression classification with caveats. Delta exceeds adaptive threshold. After-runs show a mid-window step change (first 10 runs at ~40 ns, then 24 runs at ~62–63 ns). However, the before-series is bimodal (cluster A ~62 ns for runs 1–176, cluster B ~40 ns for runs 177–300), so the compare value matches the old slow cluster. The z-score (0.78) is suppressed by the bimodal history.
• Finding Confidence: 3/5 - Adequate data but bimodal history makes metrics unreliable. The step change in after-runs suggests a real change, but it's ambiguous whether this is a regression or a reversion to a prior state.
• Regression Confidence: 2/5 - The compare value (~62 ns) is not a new level — it matches the earlier historical cluster. No direct code path modification for this test in the commit range. May represent a reversion of an earlier optimization rather than a new regression.
• Likely Cause: No clear attribution — the compare value matches a prior historical state
  • Affected Areas: BCL / System.Linq enumeration
  • Stack Relevance: Unknown
  • Rationale: No commit in the range directly modifies WhereAny or predicate-based enumeration paths. The bimodal history and reversion pattern suggest this may be an environmental or JIT tiering effect rather than a code change regression.
• Verification Status: Not verified

Borderline Tests

System.Linq.Tests.Perf_Enumerable.FirstWithPredicate_LastElementMatches (+68.6%)

• Classification: Borderline — bimodal oscillation between two known states (~32 ns and ~56 ns). The after-series itself is bimodal (first 10 runs at ~31–35 ns, then ~54–57 ns). Not a new-level regression; z=0.66 is not significant.

System.Linq.Tests.Perf_Enumerable.AnyWithPredicate_LastElementMatches (+67.8%)

• Classification: Borderline — identical bimodal pattern to FirstWithPredicate. After-series retains both low and high clusters. z=0.78, not significant.

System.Linq.Tests.Perf_Enumerable.All_AllElementsMatch (+47.6%)

• Classification: Borderline — multi-modal before-series (clusters at ~30, ~44, ~56 ns). Delta below adaptive threshold (50.97%), z=0.25 negligible. After-runs show the same mid-window step as WhereAny but conflicting signals warrant borderline classification.

System.Linq.Tests.Perf_ImmutableArrayExtensions.SequenceEqual (+25.6%)

• Classification: Borderline — the baseline value (47.18 ns) is an outlier 18.3% below the historical mean (57.77 ns) in an already-volatile recent window (range 47–69 ns). Compare value (59.25 ns) is only +2.6% vs historical mean, fully within normal range. The large percentDelta is an artifact of the anomalously low baseline, not a genuine slowdown.

Open Questions

• Bimodal History: All LINQ Enumerable tests exhibit bimodal/multi-modal before-series, with an earlier high-value regime followed by a step-down partway through the 300-run window. This may represent an earlier optimization that now partially reverted for Max/Min.
• WhereAny/All Pattern: The mid-window step change in after-runs (first ~10 at low state, then jump to high state) for WhereAny and All may represent JIT tiering or environmental effects. Monitor for stabilization.
• Confidence Limitations: 26 commits in range; single commit (414b194) is the clear candidate for Max/Min but WhereAny/All/First/Any lack a direct code path match.
• Verification Needed: Bisection recommended on commit 414b194 to confirm it is the cause of Max/Min regressions, particularly since the MemoryExtensions refactoring changes the iteration hot path for IEnumerable inputs.

Recommended Actions

1. Verify regression: Run bisection with RunWithBuildAtHash on commit 414b194d3a5a vs baseline 46b72c7c8159 using Max and Min tests to confirm the regression.
2. Code review: Review the new MemoryExtensions.MinMax.cs implementation for IEnumerable fallback paths — the overhead likely comes from additional interface dispatch, type-checking, or boxing when the input is not Span-backed.
3. Monitor borderline tests: Track FirstWithPredicate, AnyWithPredicate, WhereAny, All, and SequenceEqual for stabilization over the next few runs. Their bimodal behavior predates this commit range.
4. Profiling: Consider collecting hardware counters (collectHardwareCounters=true) on the Max test at baseline vs compare to identify whether the regression is from branch mispredictions, cache misses, or instruction count growth.

Bisection Results

Test	Status	Culprit Commit	Relevance	Reported Diff (issue)	Bisected Diff (at culprit)	Duration
Max(input: IEnumerable)	✅ Success	414b194d3a5a	✅ RELATED	45.4→107.0 ns (+136%)	56.0→107.3 ns (+92%)	1662s
Min(input: IEnumerable)	✅ Success	414b194d3a5a	✅ RELATED	58.0→101.2 ns (+75%)	57.4→101.3 ns (+77%)	5304s
FirstWithPredicate_LastElementMatches...	❌ failed	-	-	32.0→54.0 ns (+69%)	— (not measured)	2023s
AnyWithPredicate_LastElementMatches(i...	❌ failed	-	-	33.0→55.4 ns (+68%)	— (not measured)	2445s
WhereAny_LastElementMatches(input: IE...	❌ failed	-	-	40.7→62.3 ns (+53%)	— (not measured)	1490s
All_AllElementsMatch(input: IEnumerable)	❌ failed	-	-	30.7→45.4 ns (+48%)	— (not measured)	1430s
SequenceEqual(input: ICollection)	❌ failed	-	-	47.2→59.3 ns (+26%)	— (not measured)	3716s

Total bisection time: 18071s
Success rate: 2/7 tests

System.Linq.Tests.Perf_Enumerable.Max(input: IEnumerable)

• Culprit: 414b194d3a5a
• Commit Message: Add MemoryExtensions.Min/Max (Add MemoryExtensions.Min/Max #128306)
• Relevance: RELATED
• Explanation: The commit directly modifies System.Linq.Max.cs and System.Linq.Min.cs, removing shared code from MaxMin.cs and refactoring the Min/Max implementations to delegate to new MemoryExtensions.Min/Max methods. The regressing test Perf_Enumerable.Max(input: IEnumerable) exercises exactly the Enumerable.Max code path that was changed. The regression is a plausible consequence of the refactoring altering the hot path for IEnumerable inputs.

Helix jobs

Job ID	Submitted by bisection	Last status
fe49d6f1-e162-41fe-98ec-9bd0db9932cd	✅	submitted
ac7936f0-4f70-4467-94eb-e167bef00a1e	✅	submitted
a29dac81-4ce4-445f-a387-89666eaaea06	✅	submitted

Per-commit measurements

Commit	Position	Measurements
414b194d3a5a	job1 (run 1, 2)	107.350, 107.350, 107.340, 107.340 ns
b06f62fc8e16	job1	46.450, 46.450 ns
ce71db3201f6	job1	107.200 ns
bf3fa3f2822f	run 1 (run 1, 2)	56.110, 56.110, 56.020, 56.020 ns

Bisection details

…[earlier output omitted]

• bf3fa3f (before): 56.02 ns (within noise of baseline 45.36) ✓
• 414b194 (culprit): 107.34 ns (regressed) ✗

Both runs are consistent. The regression is clearly introduced at commit 414b194.

Summary of all measurements:

• b06f62f: 46.45 ns ✓
• bf3fa3f (run 1): 56.11 ns ✓
• bf3fa3f (run 2): 56.02 ns ✓
• 414b194 (run 1): 107.35 ns ✗
• 414b194 (run 2): 107.34 ns ✗
• ce71db3: 107.20 ns ✗

BISECTION RESULT:
- Culprit Commit: 414b194d3a5a6c133cd19edaad73232e29c2e3fd
- Commit Message: Add MemoryExtensions.Min/Max (#128306)
- Commits Tested: 5
- Confidence: 4/5
- Commit Relevance: RELATED
- Relevance Explanation: The commit directly modifies `System.Linq.Max.cs` and `System.Linq.Min.cs`, removing shared code from `MaxMin.cs` and refactoring the Min/Max implementations to delegate to new `MemoryExtensions.Min/Max` methods. The regressing test `Perf_Enumerable.Max(input: IEnumerable)` exercises exactly the `Enumerable.Max` code path that was changed. The regression is a plausible consequence of the refactoring altering the hot path for IEnumerable inputs.
- Baseline Value: 56.02 ns
- Culprit Value: 107.34 ns
- Summary: Bisected 26 commits in 3 Helix jobs. The regression was introduced at commit 414b194 which refactored LINQ Min/Max to use new MemoryExtensions. Confirmed with a second run — bf3fa3f (pre-culprit) measures ~56 ns while 414b194 measures ~107 ns consistently. Confidence is 4/5 despite high historical noise (CV 16.94%) because the step change (~51 ns) is well above the 2σ noise floor (22 ns) and was reproduced across two independent runs.

HELIX JOBS:
- fe49d6f1-e162-41fe-98ec-9bd0db9932cd
- ac7936f0-4f70-4467-94eb-e167bef00a1e
- a29dac81-4ce4-445f-a387-89666eaaea06

MEASUREMENTS:
- b06f62fc8e16 (job1): 46.45 ns
- 414b194d3a5a (job1): 107.35 ns
- ce71db3201f6 (job1): 107.20 ns
- b06f62fc8e16 (job2): 46.45 ns
- bf3fa3f2822f (job2): 56.11 ns
- bf3fa3f2822f (job3-confirm): 56.02 ns
- 414b194d3a5a (job3-confirm): 107.34 ns

System.Linq.Tests.Perf_Enumerable.Min(input: IEnumerable)

• Culprit: 414b194d3a5a
• Commit Message: Add MemoryExtensions.Min/Max (Add MemoryExtensions.Min/Max #128306)
• Relevance: RELATED
• Explanation: The commit directly modifies System.Linq/src/System/Linq/Min.cs and MaxMin.cs, refactoring the Min/Max implementations to delegate to new MemoryExtensions.Min/Max helpers. This is the exact code path exercised by the System.Linq.Tests.Perf_Enumerable.Min benchmark. The regression is a direct consequence of the changed implementation.

Helix jobs

Job ID	Submitted by bisection	Last status
91c87de1-3f70-41d6-aeec-456fb5369c5c	✅	submitted
bdab49e9-1b9d-4bb3-88aa-37533973e19e	✅	submitted

Per-commit measurements

Commit	Position	Measurements
414b194d3a5a	q1-quarter	101.320 ns
b06f62fc8e16	q1-inner-quarter	56.750 ns
bf3fa3f2822f	q1-inner-threequarter	57.350 ns
ce71db3201f6	q3-threequarter	101.330 ns

Bisection details

…[earlier output omitted]

• b06f62f (index 1): 56.75 ns ✓ (good)
• bf3fa3f (index 5): 57.35 ns ✓ (good)
• 414b194 (index 6): 101.32 ns ✗ (regressed, from job 1)

The regression is between bf3fa3f and 414b194. Commit 414b194 is the culprit. Let me examine it.

● get_single_commit_summary (MCP: PerfTriageMcp) · owner: "dotnet", repo: "runtime", sha: "414b194d…
└ {"index":null,"sha":"414b194d3a5a6c133cd19edaad73232e29c2e3fd","abbrevSha":"4...

This is a perfect match. The commit "Add MemoryExtensions.Min/Max (#128306)" directly modifies System.Linq/src/System/Linq/Min.cs — the exact code path exercised by the System.Linq.Tests.Perf_Enumerable.Min benchmark. The delta (57→101 ns, +77%) is well above the 2σ noise floor (23.7 ns) and matches the reported regression magnitude (+74.5%).

BISECTION RESULT:
- Culprit Commit: 414b194d3a5a6c133cd19edaad73232e29c2e3fd
- Commit Message: Add MemoryExtensions.Min/Max (#128306)
- Commits Tested: 5
- Confidence: 5
- Commit Relevance: RELATED
- Relevance Explanation: The commit directly modifies `System.Linq/src/System/Linq/Min.cs` and `MaxMin.cs`, refactoring the Min/Max implementations to delegate to new MemoryExtensions.Min/Max helpers. This is the exact code path exercised by the `System.Linq.Tests.Perf_Enumerable.Min` benchmark. The regression is a direct consequence of the changed implementation.
- Baseline Value: 57.35 ns
- Culprit Value: 101.32 ns
- Summary: Bisected 26 commits using two rounds of double-bisection on Helix. The regression was cleanly isolated to commit 414b194d which adds MemoryExtensions.Min/Max and refactors System.Linq Min/Max to use the new helpers, causing a ~77% slowdown on the IEnumerable Min benchmark.

HELIX JOBS:
- 91c87de1-3f70-41d6-aeec-456fb5369c5c
- bdab49e9-1b9d-4bb3-88aa-37533973e19e

MEASUREMENTS:
- 414b194d3a5a (Q1-quarter): 101.32 ns
- ce71db3201f6 (Q3-threeQuarter): 101.33 ns
- b06f62fc8e16 (Q1-inner-quarter): 56.75 ns
- bf3fa3f2822f (Q1-inner-threeQuarter): 57.35 ns

System.Linq.Tests.Perf_Enumerable.FirstWithPredicate_LastElementMatches(input: IEnumerable)

• Status: failed

Helix jobs

Job ID	Submitted by bisection	Last status
fa57e988-b315-4c8b-b0cf-1ba452d4e0d0	✅	submitted
900029ec-4f20-49ce-bc62-97ce81709f96	✅	submitted
89ea6224-b0be-4ad5-a655-b3b4da88cdb3	✅	completed
6b42a79c-c049-491f-a8de-c91efe8a96b4	✅	submitted

Per-commit measurements

Commit	Position	Measurements
46b72c7c8159	baseline boundary	30.970 ns
414b194d3a5a	q1 bisect	31.190 ns
ce71db3201f6	q3 bisect	30.890 ns
05cecc17cc17	q4-quarter	30.760 ns
bf62aa30c9f4	q4-mid	31.400 ns
cd27dcec1601	head boundary	30.840 ns

Bisection details

…[earlier output omitted]

821321498-copi...

● Parse boundary test results (shell)
│ cat /tmp/1782821321498-copilot-tool-output-zfloy1.txt | python3 -c "
│ import json, sys, re
│ data = json.load(sys.stdin)
│ report = json.loads(data['report'])
│ benchmarks = report.get('Benchmarks', [])…
└ 12 lines…

Both boundary commits show identical performance (~31 ns). The reported regression (32→54 ns) is not reproducible. All 7 commits tested across the range show ~30.8-31.4 ns.

Summary of all measurements:

• 46b72c7c (baseline): 30.97 ns
• 414b194d: 31.19 ns
• 05cecc17: 30.76 ns
• bf62aa30: 31.40 ns
• ce71db3: 30.89 ns
• cd27dcec (head): 30.84 ns

No step change anywhere. The 2σ noise floor (23.43 ns / 73.2% of baseline) exceeds the reported regression delta (22 ns / 68.6%), confirming this is noise.

BISECTION FAILED:
- Reason: inconclusive (noise floor) — regression not reproducible; all measurements across the commit range show baseline-level performance (~30.8–31.4 ns), with no observable step change
- Helix Errors: None — all jobs completed successfully
- What was attempted: Submitted 4 Helix jobs testing 7 distinct commits spanning the full range (including both boundary commits 46b72c7c and cd27dcec). All returned ~31 ns median, consistent with baseline. The reported regression delta (+68.6%, 22 ns) falls within the test's 2σ noise floor (23.43 ns, 73.2% of baseline), and CV=25.33% indicates an extremely noisy benchmark.
- What is needed: The original regression signal was likely a transient noise spike. No further bisection action is warranted.

HELIX JOBS:

• fa57e988-b315-4c8b-b0cf-1ba452d4e0d0
• 900029ec-4f20-49ce-bc62-97ce81709f96
• 89ea6224-b0be-4ad5-a655-b3b4da88cdb3
• 6b42a79c-c049-491f-a8de-c91efe8a96b4

MEASUREMENTS:

• 46b72c7 (baseline boundary): 30.97 ns
• 414b194 (Q1 bisect): 31.19 ns
• ce71db3 (Q3 bisect): 30.89 ns
• 05cecc1 (Q4-quarter): 30.76 ns
• bf62aa3 (Q4-mid): 31.40 ns
• cd27dce (head boundary): 30.84 ns

System.Linq.Tests.Perf_Enumerable.AnyWithPredicate_LastElementMatches(input: IEnumerable)

• Status: failed

Helix jobs

Job ID	Submitted by bisection	Last status
651f73ce-d522-41f7-b7ad-1832e01c14dc	✅	submitted
a85cf32d-d3c7-4d64-9a4e-42adef7a175a	✅	submitted

Per-commit measurements

Commit	Position	Measurements
05cecc17cc17	q4-mid1	33.810, 33.810 ns
4585fc41c242	q4-start	34.400, 34.400 ns
bf62aa30c9f4	q4-mid2	34.160, 34.160 ns
414b194d3a5a	idx 6	32.560, 32.560 ns
ce71db3201f6	idx 19	34.420, 34.420 ns
cd27dcec1601	idx 25, head	33.010, 33.010 ns

Bisection details

…[earlier output omitted]

└ Output too large to read at once (78.0 KB). Saved to: /tmp/1782823768242-copi...

● Parse Q4 results (shell)
│ cat /tmp/1782823768242-copilot-tool-output-2aymku.txt | python3 -c "
│ import json, sys
│ data = json.load(sys.stdin)
│ report = json.loads(data['report'])
│ for bm in report.get('Benchmarks', []):…
└ 15 lines…

All commits including the head show ~33 ns (baseline). The regression is not reproducible. All measurements:

• 414b194 (idx 6): 32.56 ns
• ce71db3 (idx 19): 34.42 ns
• 4585fc4 (idx 20): 34.40 ns
• 05cecc1 (idx 21): 33.81 ns
• bf62aa3 (idx 23): 34.16 ns
• cd27dce (idx 25, head): 33.01 ns

The reported regression delta (67.8%) is within the 2σ noise floor (69.36%), and none of 6 tested commits reproduce the ~55 ns regressed value. This is noise.

BISECTION FAILED:
- Reason: Regression not reproducible — all 6 tested commits (including baseline and head) measure ~33 ns, matching the reported baseline of 33.02 ns. The reported regression to 55.42 ns (+67.8%) falls within the test's 2σ noise floor of 69.36% (σ=11.45, μ=46.54, CV=24.61%). The autofile observation was likely a noisy outlier.
- Helix Errors: None — both jobs completed successfully
- What was attempted: Submitted 2 Helix jobs testing 6 distinct commits across the full 26-commit range (indices 6, 19, 20, 21, 23, 25). All returned values tightly clustered at 32.56–34.42 ns, indistinguishable from the baseline of 33.02 ns.
- What is needed: No action needed — this appears to be a false positive caused by measurement noise on a high-variance benchmark (CV=24.61%).

HELIX JOBS:
- 651f73ce-d522-41f7-b7ad-1832e01c14dc
- a85cf32d-d3c7-4d64-9a4e-42adef7a175a

MEASUREMENTS:
- 414b194d3a5a (quarter): 32.56 ns
- ce71db3201f6 (three-quarter): 34.42 ns
- 4585fc41c242 (Q4-start): 34.40 ns
- 05cecc17cc17 (Q4-mid1): 33.81 ns
- bf62aa30c9f4 (Q4-mid2): 34.16 ns
- cd27dcec1601 (head/compare): 33.01 ns

System.Linq.Tests.Perf_Enumerable.WhereAny_LastElementMatches(input: IEnumerable)

• Status: failed

Helix jobs

Job ID	Submitted by bisection	Last status
d350c1e3-918d-46a7-b380-38f6106039d2	✅	submitted
31bfddfd-a7a3-4dee-850e-bfdf1bd64d12	✅	submitted

Per-commit measurements

Commit	Position	Measurements
05cecc17cc17	q4-idx1	40.630 ns
4585fc41c242	q4-idx0	40.870 ns
bf62aa30c9f4	q4-idx3	40.710 ns
414b194d3a5a	quarter	41.050 ns
cd27dcec1601	compare/head	40.720 ns
ce71db3201f6	threequarter	40.770 ns

Bisection details

…[earlier output omitted]

│ data = json.load(sys.stdin)
│ report = json.loads(data['report'])
│ for i, bench in enumerate(report.get('Benchmarks', [])):…
└ 14 lines…

All measurements across the entire range show values near baseline (~40.6-41.0). Even the reported regressed commit cd27dcec1601 measures 40.72, indistinguishable from baseline 40.67. No step change is observable.

Let me verify by reviewing all measurements:

Commit	Mean
414b194 (Q1)	41.05
ce71db3 (Q3)	40.77
4585fc4 (Q4-0)	40.87
05cecc1 (Q4-1)	40.63
bf62aa3 (Q4-3)	40.71
cd27dce (Q4-5, reported regressed)	40.72

All ~40.7 ns. The reported regression (40.67 → 62.28, +53.1%) is not reproducible. The 2σ noise floor is 53.85% of baseline, which is comparable to the reported 53.1% regression delta, confirming this was likely a noise artifact.

BISECTION FAILED:
- Reason: No observable step change in measurements across the entire commit range; reported regression not reproducible
- Helix Errors: None — both jobs completed successfully
- What was attempted: Tested 6 commits across the range (414b194, ce71db3, 4585fc4, 05cecc1, bf62aa3, cd27dce) via 2 Helix jobs. All measurements were ~40.6-41.1 ns, indistinguishable from baseline (40.67 ns). The reported compare value (62.28 ns) was not reproduced even at the exact reported regressed commit (cd27dcec1601 measured 40.72 ns).
- What is needed: The reported regression of +53.1% falls within the test's 2σ noise floor (53.85% of baseline). This appears to be a noise artifact rather than a genuine regression. No further bisection is possible without an observable step change.

HELIX JOBS:
- d350c1e3-918d-46a7-b380-38f6106039d2
- 31bfddfd-a7a3-4dee-850e-bfdf1bd64d12

MEASUREMENTS:
- 414b194d3a5a (quarter): 41.05 ns
- ce71db3201f6 (threeQuarter): 40.77 ns
- 4585fc41c242 (Q4-idx0): 40.87 ns
- 05cecc17cc17 (Q4-idx1): 40.63 ns
- bf62aa30c9f4 (Q4-idx3): 40.71 ns
- cd27dcec1601 (compare/head): 40.72 ns

System.Linq.Tests.Perf_Enumerable.All_AllElementsMatch(input: IEnumerable)

• Status: failed

Helix jobs

Job ID	Submitted by bisection	Last status
94032e66-8262-4292-8469-55bee77e3821	✅	submitted
4c9dc991-ebe8-4ecc-842e-a1f625ff9d47	✅	completed
3fb85779-025e-4101-9a39-c77fa48c6d9d	✅	submitted
fceb832f-a5bd-46df-813e-69b92f4b369a	✅	submitted

Per-commit measurements

Commit	Position	Measurements
440e6cba935e	index 14	30.730 ns
9e76efa2a92f	index 17	30.660 ns
37d45de0d2f8	index 18	30.680 ns
4585fc41c242	index 20	30.670 ns
414b194d3a5a	quarter	30.740 ns
cd27dcec1601	compare/head	30.990 ns
ce71db3201f6	three-quarter	36.460, 36.460 ns

Bisection details

…[earlier output omitted]

│ for bm in report.get('Benchmarks', []):…
└ 12 lines…

All measurements collected. Here's the complete picture:

Commit	Index	Median (ns)
414b194	6	30.74
440e6cb	14	30.73
9e76efa	17	30.66
37d45de	18	30.68
ce71db3	19	36.46
4585fc4	20	30.67
cd27dce	25	30.99

Every commit including the reported compare commit (cd27dce) measures at baseline (~30.7 ns). The one elevated reading (ce71db3 at 36.46) is within the 2σ noise floor and bracketed by baseline-level commits. The reported regression of 45.35 ns is not reproducible.

BISECTION FAILED:
- Reason: Regression not reproducible — the compare commit (cd27dcec) measures at 30.99 ns, matching baseline (30.72 ns), not the reported 45.35 ns. All 7 tested commits across the range show baseline-level performance (~30.65-30.74 ns). The historical noise floor (2σ = 21.76, CV = 25.48%) exceeds the reported delta (14.63 ns), confirming this is a noise-driven false positive.
- Helix Errors: None — all jobs completed successfully
- What was attempted: Tested 7 commits spanning the full 26-commit range via 4 Helix jobs (94032e66, 4c9dc991, 3fb85779, fceb832f). All measurements cluster tightly around baseline (~30.7 ns). The single elevated reading (ce71db3 at 36.46 ns) is within noise and surrounded by baseline values.
- What is needed: No action needed — this regression appears to be noise. The test's CV of 25.48% makes it prone to false-positive autofile detections.

HELIX JOBS:

• 94032e66-8262-4292-8469-55bee77e3821
• 4c9dc991-ebe8-4ecc-842e-a1f625ff9d47
• 3fb85779-025e-4101-9a39-c77fa48c6d9d
• fceb832f-a5bd-46df-813e-69b92f4b369a

MEASUREMENTS:

• 414b194 (quarter): 30.74 ns
• 440e6cb (index 14): 30.73 ns
• 9e76efa (index 17): 30.66 ns
• 37d45de (index 18): 30.68 ns
• ce71db3 (three-quarter): 36.46 ns
• 4585fc4 (index 20): 30.67 ns
• cd27dce (compare/head): 30.99 ns

System.Linq.Tests.Perf_ImmutableArrayExtensions.SequenceEqual(input: ICollection)

• Status: failed

Helix jobs

Job ID	Submitted by bisection	Last status
16b726ce-6519-4185-875f-b01cf42244b0	✅	submitted
4baa4922-5dbc-4220-acbd-c83ef7e4d75f	✅	submitted
25e7e18a-cff4-44e3-9b5f-24ea1590c997	✅	submitted
c3fe9932-8bda-4ab5-b99e-22defc7ebeae	✅	completed
752242f4-f8f2-4eef-812b-6832f35d7e83	✅	submitted
290c2190-2bab-4ca8-86b9-2528e095bf45	✅	submitted
44cdc43d-d2ee-4045-8959-cd1e36620719	✅	submitted

Per-commit measurements

Commit	Position	Measurements
46b72c7c8159	baseline, job c3fe9932	58.600, 58.030 ns
b06f62fc8e16	idx 1, job 4baa4922	57.820, 58.580 ns
e3cbc00cd3e0	idx 2, job 25e7e18a	50.890, 56.060 ns
2608ba24252b	idx 4, job 25e7e18a	48.850, 51.470 ns
bf3fa3f2822f	idx 5, job 4baa4922	58.560, 53.500 ns
414b194d3a5a	idx 6, job 16b726ce	59.300, 58.780 ns
9c1fdaea7e45	idx 15, job 44cdc43d	52.250 ns
ce71db3201f6	idx 19, job 16b726ce	66.290 ns
05cecc17cc17	idx 21, job 44cdc43d	48.610 ns
cd27dcec1601	head, job 752242f4	62.420, 53.000 ns

Bisection details

…[earlier output omitted]

• What is needed: This issue appears to be noise rather than a genuine regression. Consider closing as noise or re-running with dedicated hardware pinning to eliminate machine-to-machine variance.

HELIX JOBS:

• 16b726ce-6519-4185-875f-b01cf42244b0
• 4baa4922-5dbc-4220-acbd-c83ef7e4d75f
• 25e7e18a-cff4-44e3-9b5f-24ea1590c997
• c3fe9932-8bda-4ab5-b99e-22defc7ebeae
• 752242f4-f8f2-4eef-812b-6832f35d7e83
• 290c2190-2bab-4ca8-86b9-2528e095bf45
• 44cdc43d-d2ee-4045-8959-cd1e36620719

MEASUREMENTS:

• 46b72c7 (baseline, job c3fe9932): 58.60 ns
• 46b72c7 (baseline, job 752242f4): 58.03 ns (median 57.63)
• b06f62f (idx 1, job 4baa4922): 57.82 ns
• b06f62f (idx 1, job 44cdc43d): 58.58 ns (median 58.83)
• e3cbc00 (idx 2, job 25e7e18a): 50.89 ns (median 46.83)
• e3cbc00 (idx 2, job 290c2190): 56.06 ns (median 58.05)
• 2608ba2 (idx 4, job 25e7e18a): 48.85 ns (median 46.54)
• 2608ba2 (idx 4, job 44cdc43d): 51.47 ns (median 49.77)
• bf3fa3f (idx 5, job 4baa4922): 58.56 ns
• bf3fa3f (idx 5, job 290c2190): 53.50 ns (median 56.89)
• 414b194 (idx 6, job 16b726ce): 59.30 ns
• 414b194 (idx 6, job 44cdc43d): 58.78 ns (median 58.71)
• 9c1fdae (idx 15, job 44cdc43d): 52.25 ns (median 50.33)
• ce71db3 (idx 19, job 16b726ce): 66.29 ns (median 58.64)
• 05cecc1 (idx 21, job 44cdc43d): 48.61 ns (median 48.31)
• cd27dce (head, job 752242f4): 62.42 ns (median 57.77)
• cd27dce (head, job 44cdc43d): 53.00 ns (median 53.40)

</details>

<!-- perf-triage-next-actions-v1 -->

### 🤖 Proposed Next Actions

**Rule**: `R5_look_again` — Uncertain — flag for re-triage.

Finding confidence 3/5 is below the required 4/5. Revisit once more historical data is available.

| # | Action | Status | Details |
|---|--------|--------|---------|
| 1 | Add label | ✅ Done | `Look Again` |
| 2 | Remove label | ⏭️ Skipped (dry-run) | `untriaged` |

</details>

<!-- perf-triage-reanalysis-pre-v1 -->

#### 🦆 Pre-bisection Re-analysis (advisory)

<details>
<summary>⚠️ Disputes the pre-bisection plan — advisory only, not applied</summary>

**Flags**: `wasted-bisection-slot`, `possible-noise`

**Rationale**: Max and Min are strong bisection candidates (changeConf 4/5, identified commit, relevant stack). However, four tests (FirstWithPredicate_LastElementMatches, AnyWithPredicate_LastElementMatches, All_AllElementsMatch, SequenceEqual) have change confidence of only 1/5 with no likely commit and unknown stack relevance, and the run has bimodal history flagged — these are almost certainly noise and will burn ~240+ Helix minutes for negligible diagnostic value. WhereAny_LastElementMatches at 2/5 is borderline but defensible to keep given it is at least slightly above floor.

**Suggested drop from bisection**: `System.Linq.Tests.Perf_Enumerable.FirstWithPredicate_LastElementMatches(input: IEnumerable)`, `System.Linq.Tests.Perf_Enumerable.AnyWithPredicate_LastElementMatches(input: IEnumerable)`, `System.Linq.Tests.Perf_Enumerable.All_AllElementsMatch(input: IEnumerable)`, `System.Linq.Tests.Perf_ImmutableArrayExtensions.SequenceEqual(input: ICollection)` _(advisory — not applied)_

_This critique is advisory in the current configuration and did **not** change which tests were bisected._

</details>

<!-- perf-triage-reanalysis-v1 -->

#### 🦆 Re-analysis (advisory)

<details>
<summary>✅ Agrees with the analysis — advisory only, not applied</summary>

**Flags**: `weak-secondary-findings`, `bimodal-history`

**Rationale**: The culprit commit is well-supported by two converging successful bisections (Max and Min) with consistent alignment and relevant stacks. The overall 3/3 scores are reasonable given that only 2 of 7 bisections succeeded and bimodal history is flagged. The 5 failed-bisection findings with changeConf 1/5 marked noise=False are internally questionable—at 1/5 change confidence they are effectively noise—but this does not undermine the core Max/Min signal driving the overall conclusion.

_This critique is advisory in the current configuration and did **not** change the confidences, culprit, or proposed actions above._

</details>

---
*Generated by PerfTriageAgent on 2026-06-30 14:41 UTC*
<!-- copilot-sessions: {"bisections":{"System.Linq.Tests.Perf_Enumerable.All_AllElementsMatch(input: IEnumerable)":"21df2ce6-7ce8-4a5f-ab6f-a7ca80a50c3c","System.Linq.Tests.Perf_Enumerable.AnyWithPredicate_LastElementMatches(input: IEnumerable)":"a56726d3-6503-4b32-b409-1d69b6616858","System.Linq.Tests.Perf_Enumerable.FirstWithPredicate_LastElementMatches(input: IEnumerable)":"1d3566da-5a61-42c4-b05c-c900e6e88012","System.Linq.Tests.Perf_Enumerable.Max(input: IEnumerable)":"a77ed86a-ee29-4bd4-bf19-a7de59f3b1db","System.Linq.Tests.Perf_Enumerable.Min(input: IEnumerable)":"9a985a87-3170-4bf7-9f51-a09cc1386957","System.Linq.Tests.Perf_Enumerable.WhereAny_LastElementMatches(input: IEnumerable)":"28b60d13-1243-43ee-bed7-a2d2b88ecdbc","System.Linq.Tests.Perf_ImmutableArrayExtensions.SequenceEqual(input: ICollection)":"1843c12c-6006-4930-b0ec-8c039b863d9f"},"main":"c06f8708-5d5b-4625-9e30-f3f093a12973"} -->
<!-- copilot-sessions-manifest: https://perftriagedevsa.blob.core.windows.net/triage-sessions/dotnet-perf-autofiling-issues/75181/20260630T144101Z/manifest.json -->

[LoopedBard3]

Max and Min were matched to PR #128306, others seem to have been incorrectly attributed to this time range when they are really likely due to #129868. FYI @MihaZupan

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/area-system-linq
See info in area-owners.md if you want to be subscribed.