Skip to content

Fix flaky Linux Project25 test#19373

Merged
T-Gro merged 1 commit intodotnet:mainfrom
T-Gro:fix/flaky-linux
Mar 2, 2026
Merged

Fix flaky Linux Project25 test#19373
T-Gro merged 1 commit intodotnet:mainfrom
T-Gro:fix/flaky-linux

Conversation

@T-Gro
Copy link
Member

@T-Gro T-Gro commented Mar 2, 2026

Fix flaky Test Project25 symbol uses of type-provided members on Linux

Fixes #19364

Root cause

PR #19254 replaced a checked-in FSharp.Data.dll with NuGet-restored packages. The setup copied multiple DLLs (runtime + design-time + transitive deps) into a staging directory and added them all as -r: references via Directory.GetFiles(stagingDir, "*.dll").

Directory.GetFiles does not guarantee ordering across platforms. On Linux (ext4/btrfs), files are returned in inode order — effectively random between runs. On Windows (NTFS), they're typically alphabetical and stable.

The FSharp.Data namespace entity exists in two places: the runtime assembly (regular namespace, IsProvided = false) and the type provider registration (TProvidedNamespaceRepr, IsProvided = true). Which entity surfaces in the symbol API depends on reference processing order. When the order varies on Linux, the namespace is sometimes tagged ["namespace"] and sometimes ["namespace"; "provided"] — causing the flaky assertion.

This is not a TransparentCompiler issue (Linux CI does not use it), and the "provided" namespace tag is technically correct — it's a pre-existing ambiguity when the same namespace comes from both a regular assembly and a type provider.

Fix

Replace FSharp.Data (NuGet) with the built-in TestTP type provider, which is already compiled as part of the solution build. This:

  • Eliminates the 70-line NuGet restore/staging setup (net -40 lines)
  • Removes the non-deterministic Directory.GetFiles ordering issue entirely
  • Removes the FactSkipOnSignedBuild attribute (TestTP is always available)
  • Replaces brittle exact-match symbol list with targeted assertions for provided types, methods, and namespaces
  • Keeps testing the same symbol API surface: GetAllUsesOfAllSymbolsInFile, GetSymbolUseAtLocation, GetUsesOfSymbolInFile for provided types and regular F# types side by side

@T-Gro T-Gro requested a review from a team as a code owner March 2, 2026 11:39
@T-Gro T-Gro added the NO_RELEASE_NOTES Label for pull requests which signals, that user opted-out of providing release notes label Mar 2, 2026
@github-actions
Copy link
Contributor

github-actions bot commented Mar 2, 2026

⚠️ Release notes required, but author opted out

Warning

Author opted out of release notes, check is disabled for this pull request.
cc @dotnet/fsharp-team-msft

@T-Gro T-Gro force-pushed the fix/flaky-linux branch from f203cb5 to ad9ea77 Compare March 2, 2026 12:39
@T-Gro
Fix flaky Project25 TP test by replacing FSharp.Data NuGet with TestTP (
917c82a 
dotnet#19364)

Replace FSharp.Data (resolved via NuGet at runtime) with the built-in
TestTP type provider for the Project25 symbol API tests. This fixes
non-deterministic test failures on Linux CI caused by
Directory.GetFiles returning DLLs in random inode order on ext4,
which varied whether the FSharp.Data namespace was tagged as
'provided' or not.

Changes:
- Replace 70-line NuGet restore/staging setup with 2-line TestTP reference
- Update source to use ErasedWithConstructor.Provided.MyType instead
  of FSharp.Data.XmlProvider
- Replace brittle exact-match symbol list with targeted assertions
  for provided types, methods, and namespaces
- Remove FactSkipOnSignedBuild (TestTP is always available after build)
- Rename test variables to match the types being tested

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@T-Gro T-Gro force-pushed the fix/flaky-linux branch from ad9ea77 to 917c82a Compare March 2, 2026 12:44
@T-Gro T-Gro enabled auto-merge (squash) March 2, 2026 13:59
@T-Gro T-Gro requested a review from abonie March 2, 2026 14:00
@github-project-automation github-project-automation bot moved this from New to In Progress in F# Compiler and Tooling Mar 2, 2026
@T-Gro T-Gro merged commit 431a3b0 into dotnet:main Mar 2, 2026
45 checks passed
This was referenced Mar 2, 2026
Merged
Merged
Merged
T-Gro added a commit that referenced this pull request Mar 6, 2026
@github-actions @auduchinok
@dotnet-maestro @T-Gro @abonie @majocha @evgTSV @Youssef1313 @bbatsov @aw0lid @brianrourkeboll @apoorvdarshan
[automated] Merge branch 'main' => 'feature/net11-scouting' (#19376)
f52adae 
* Type checker: recover on argument/overload checking (#19314)

* [main] Update dependencies from dotnet/arcade (#19333)

* Update dependencies from https://github.com/dotnet/arcade build 20260219.2
On relative base path root
Microsoft.DotNet.Arcade.Sdk From Version 10.0.0-beta.26117.6 -> To Version 10.0.0-beta.26119.2

* Update dependencies from https://github.com/dotnet/arcade build 20260223.2
On relative base path root
Microsoft.DotNet.Arcade.Sdk From Version 10.0.0-beta.26117.6 -> To Version 10.0.0-beta.26123.2

---------

Co-authored-by: dotnet-maestro[bot] <dotnet-maestro[bot]@users.noreply.github.com>

* [main] Source code updates from dotnet/dotnet (#19343)

* Backflow from https://github.com/dotnet/dotnet / 854c152 build 302768

[[ commit created by automation ]]

* Update dependencies from build 302768
No dependency updates to commit
[[ commit created by automation ]]

* Backflow from https://github.com/dotnet/dotnet / 51587e2 build 302820

[[ commit created by automation ]]

* Update dependencies from build 302820
No dependency updates to commit
[[ commit created by automation ]]

---------

Co-authored-by: dotnet-maestro[bot] <dotnet-maestro[bot]@users.noreply.github.com>
Co-authored-by: Tomas Grosup <Tomas.Grosup@gmail.com>

* Update image used for insert into VS step (#19357)

* Fix :: FAR :: Remove corrupted .ctor symbol reference (#19358)

* DotNetBuildUseMonoRuntime stackguard (#19360)

* Fsharp.Core :: {Array;List;Set;Array.Parallel} partitionWith (taking Choice<T,U> partitioner) (#19335)

* Sort out some outstanding issues after xUnit upgrade  (#19363)

* Improve collection comparison diagnostics in tests (#19365)

Added shouldBeEqualCollections to Assert.fs for detailed collection comparison, including reporting missing/unexpected items and positional differences. Updated Project25 symbol uses test to use this helper and print actual/expected values for easier debugging.

* Enhance the compiler to produce a FS0750 error on let! or use! outside a CE (#19347)

* Update to latest .NET 10 SDK patch (#19350)

* Feature: `#elif` preprocessor directive (#19323)

* Support #exit;; as alias to #quit;; in fsi (#19329)

* Checker: prevent reporting optional parameter rewritten tree symbols (#19353)

* Improve static compilation of state machines (#19297)

* Add FSC compiler options tests (#19348)

* Update dependencies from https://dev.azure.com/dnceng/internal/_git/dotnet-optimization build 20260226.1 (#19366)

On relative base path root
optimization.linux-arm64.MIBC.Runtime , optimization.linux-x64.MIBC.Runtime , optimization.windows_nt-arm64.MIBC.Runtime , optimization.windows_nt-x64.MIBC.Runtime , optimization.windows_nt-x86.MIBC.Runtime From Version 1.0.0-prerelease.26117.2 -> To Version 1.0.0-prerelease.26126.1

Co-authored-by: dotnet-maestro[bot] <dotnet-maestro[bot]@users.noreply.github.com>

* Fix Get-PrBuildIds.ps1 reporting SUCCESS while builds in progress (#19313)

* Fix flaky Project25 TP test by replacing FSharp.Data NuGet with TestTP (#19364) (#19373)

Replace FSharp.Data (resolved via NuGet at runtime) with the built-in
TestTP type provider for the Project25 symbol API tests. This fixes
non-deterministic test failures on Linux CI caused by
Directory.GetFiles returning DLLs in random inode order on ext4,
which varied whether the FSharp.Data namespace was tagged as
'provided' or not.

Changes:
- Replace 70-line NuGet restore/staging setup with 2-line TestTP reference
- Update source to use ErasedWithConstructor.Provided.MyType instead
  of FSharp.Data.XmlProvider
- Replace brittle exact-match symbol list with targeted assertions
  for provided types, methods, and namespaces
- Remove FactSkipOnSignedBuild (TestTP is always available after build)
- Rename test variables to match the types being tested

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Add automatic merge flow from main to feature/net11-scouting (#19283)

* Optimize Set.intersect symmetry and add release notes (#19292)

Co-authored-by: Ahmed <w0lf@192.168.1.8>

* Fix strong name signature size to align with Roslyn for public signing (#11887) (#19242)

* Use culture-independent `IndexOf` in interpolated string parsing (#19370)

* Rename "inline hints" to "inlay hints" (#19318)

* Rename "inline hints" to "inlay hints" for LSP consistency

Aligns F#-owned terminology with LSP and VS Code conventions.
The Roslyn ExternalAccess types (IFSharpInlineHintsService, etc.)
are left unchanged as they are owned by Roslyn.

Fixes #16608

* Add release note for inline-to-inlay hints rename

---------

Co-authored-by: Tomas Grosup <Tomas.Grosup@gmail.com>

* FCS: capture additional types during analysis (#19305)

* remove duplicate FlatErrors file (#19383)

* Update dependencies from https://github.com/dotnet/arcade build 20260302.1 (#19379)

[main] Update dependencies from dotnet/arcade

* [main] Update dependencies from dotnet/msbuild (#19021)

* Update dependencies from https://github.com/dotnet/msbuild build 20251021.3
On relative base path root
Microsoft.Build , Microsoft.Build.Framework , Microsoft.Build.Tasks.Core , Microsoft.Build.Utilities.Core From Version 18.1.0-preview-25515-01 -> To Version 18.1.0-preview-25521-03

* Update dependencies from https://github.com/dotnet/msbuild build 20251023.2
On relative base path root
Microsoft.Build , Microsoft.Build.Framework , Microsoft.Build.Tasks.Core , Microsoft.Build.Utilities.Core From Version 18.1.0-preview-25515-01 -> To Version 18.1.0-preview-25523-02

* Update dependencies from https://github.com/dotnet/msbuild build 20251024.3
On relative base path root
Microsoft.Build , Microsoft.Build.Framework , Microsoft.Build.Tasks.Core , Microsoft.Build.Utilities.Core From Version 18.1.0-preview-25515-01 -> To Version 18.1.0-preview-25524-03

* Update dependencies from https://github.com/dotnet/msbuild build 20251027.5
On relative base path root
Microsoft.Build , Microsoft.Build.Framework , Microsoft.Build.Tasks.Core , Microsoft.Build.Utilities.Core From Version 18.1.0-preview-25515-01 -> To Version 18.1.0-preview-25527-05

* Update dependencies from https://github.com/dotnet/msbuild build 20251027.6
On relative base path root
Microsoft.Build , Microsoft.Build.Framework , Microsoft.Build.Tasks.Core , Microsoft.Build.Utilities.Core From Version 18.1.0-preview-25515-01 -> To Version 18.1.0-preview-25527-06

* Update dependencies from https://github.com/dotnet/msbuild build 20251104.4
On relative base path root
Microsoft.Build , Microsoft.Build.Framework , Microsoft.Build.Tasks.Core , Microsoft.Build.Utilities.Core From Version 18.1.0-preview-25515-01 -> To Version 18.1.0-preview-25554-04

* Bump dependency versions in Version.Details.props

Updated package versions for several dependencies.

---------

Co-authored-by: dotnet-maestro[bot] <dotnet-maestro[bot]@users.noreply.github.com>
Co-authored-by: Tomas Grosup <Tomas.Grosup@gmail.com>

* Add fsharp-release-announcement agent (#19390)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix Seq.empty rendering as "EmptyEnumerable" in serializers (#19317)

* Update FileContentMapping.fs (#19391)

* Fix flaky help_options test: restore enableConsoleColoring after mutation (#19385)

* Initial plan

* Fix flaky help_options test by saving/restoring enableConsoleColoring global

The `fsc --consolecolors switch` test mutates the global
`enableConsoleColoring` via `--consolecolors-`. When help_options tests
run after this test, the help output says "(off by default)" instead of
"(on by default)", causing baseline mismatches.

Fix: save and restore `enableConsoleColoring` around the test.

Co-authored-by: T-Gro <46543583+T-Gro@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: T-Gro <46543583+T-Gro@users.noreply.github.com>
Co-authored-by: Tomas Grosup <Tomas.Grosup@gmail.com>

* isolate checker (#19393)

Isolate type-provider tests with dedicated FSharpChecker

Introduce Project25.checker to avoid shared state races in type-provider tests. All relevant test cases now use this dedicated instance, improving test isolation, reliability, and determinism without changing test functionality.

* Introduce CallRelatedSymbolSink to avoid affect name resolution with related symbols (#19361)

* Add RelatedSymbolUseKind flags enum and separate sink for related symbols

Address auduchinok's review comments on PR #19252: related symbols (union case
testers, copy-and-update record types) are now reported via a separate
NotifyRelatedSymbolUse sink instead of abusing NotifyNameResolution.

- Add [<Flags>] RelatedSymbolUseKind enum (None/UnionCaseTester/CopyAndUpdateRecord/All)
- Add NotifyRelatedSymbolUse to ITypecheckResultsSink interface
- Refactor RegisterUnionCaseTesterForProperty to use the new sink
- Refactor copy-and-update in TcRecdExpr to use CallRelatedSymbolSink
- Add ?relatedSymbolKinds parameter to GetUsesOfSymbolInFile (default: None)
- Wire up VS FAR to pass relatedSymbolKinds=All
- Write related symbols to ItemKeyStore for background FAR
- Update semantic classification tests: tester properties now classified as
  Property (not UnionCase) since they're no longer in capturedNameResolutions

* Add release notes for FSharp.Compiler.Service 11.0.100

---------

Co-authored-by: Eugene Auduchinok <eugene.auduchinok@gmail.com>
Co-authored-by: dotnet-maestro[bot] <42748379+dotnet-maestro[bot]@users.noreply.github.com>
Co-authored-by: dotnet-maestro[bot] <dotnet-maestro[bot]@users.noreply.github.com>
Co-authored-by: Tomas Grosup <Tomas.Grosup@gmail.com>
Co-authored-by: Adam Boniecki <20281641+abonie@users.noreply.github.com>
Co-authored-by: Jakub Majocha <1760221+majocha@users.noreply.github.com>
Co-authored-by: Evgeny Tsvetkov <61620612+evgTSV@users.noreply.github.com>
Co-authored-by: Youssef Victor <youssefvictor00@gmail.com>
Co-authored-by: Bozhidar Batsov <bozhidar@batsov.dev>
Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Ahmed Waleed <ahmedwalidahmed.0@gmail.com>
Co-authored-by: Ahmed <w0lf@192.168.1.8>
Co-authored-by: Brian Rourke Boll <brianrourkeboll@users.noreply.github.com>
Co-authored-by: Apoorv Darshan <ad13dtu@gmail.com>
Co-authored-by: T-Gro <46543583+T-Gro@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@abonie abonie abonie approved these changes

Assignees

No one assigned

Labels

NO_RELEASE_NOTES Label for pull requests which signals, that user opted-out of providing release notes

Projects

F# Compiler and Tooling
Archived in project

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Some tests are flaky after xUnit upgrade

2 participants

@T-Gro @abonie