[mono] Fix re-entrant AssemblyLoadContext resolution stack overflow under full-AOT

[pavelsavara]

Summary

Fixes an unbounded recursion (stack overflow) in Mono assembly resolution that reproduces under full-AOT when an assembly reference is resolved through a managed AssemblyLoadContext resolve hook.

Root cause

invoke_resolve_method invokes a managed ALC resolve hook (e.g. AssemblyLoadContext.MonoResolveUsingLoad). That hook constructs an AssemblyName, whose parsing uses a generic method (MemoryExtensions.Split<…>). Under full-AOT, JIT-compiling that generic instance can itself trigger resolution of the same assembly, which re-invokes the same hook, and so on without bound until the stack overflows.

The crash only reproduces under full-AOT. In JIT/interpreter the generic method is already compiled and cached, so the nested resolution does not recur.

Observed native recursion cycle (one lap, repeating identically):

mono_class_get_checked
  → mono_class_from_typeref_checked
  → mono_assembly_load_reference
  → mono_assembly_request_byname
  → mono_runtime_try_invoke[_handle]
  → wrapper_runtime_invoke
  → AssemblyLoadContext.MonoResolveUsingLoad        (managed resolve hook)
  → new AssemblyName(string) → AssemblyNameParser.Parse → …
  → MemoryExtensions.Split → SplitCore<…>           (generic method)
  → generic_trampoline_jit                          (JIT the generic instance under full-AOT)
  → native class/type setup → mono_class_get_checked → ∞

Fix

Add a thread-local re-entrancy guard in invoke_resolve_method, keyed by (resolve_method, assembly_name). This mirrors the existing TLS recursion guard in mono_class_setup_fields (setup_fields_tls_id). When resolution re-enters the same hook for the same assembly name on the current thread, the nested call returns NULL to break the cycle.

Validation

Reproduced and verified against a merged JIT regression assembly that loads itself into a collectible AssemblyLoadContext and invokes a method via reflection:

• Before: deterministic SIGSEGV (stack overflow) under Mono full-AOT.
• After: the test passes (exit 100), behaving the same as JIT/interpreter.
• The full merged regression assembly (hundreds of sub-tests that load many assemblies normally) passes under full-AOT with no regressions, confirming the guard does not affect normal, non-re-entrant assembly loading.

Note

This description was generated with the assistance of GitHub Copilot.

---

Part of the MonoAOT LLVM 23 full-AOT regression set tracked by #129508. Built and tested together with the Emscripten 5.0.6 / LLVM 23 bump on #129396, where the re-enabled Runtime_105619 test passes on the runtime-llvm AllSubsets_Mono_LLVMFULLAOT_RuntimeTests leg.

Contributes to #129508.

[pavelsavara]

/azp run runtime-extra-platforms

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[Copilot]

Pull request overview

This PR adds a thread-local re-entrancy guard around managed AssemblyLoadContext resolve hook invocation in Mono, preventing unbounded recursion (and eventual stack overflow) when assembly resolution re-enters the same resolve path on the same thread.

Changes:

• Introduces a per-thread “resolve in progress” tracking list (TLS) to detect and short-circuit re-entrant resolve calls.
• Allocates the new TLS key during mono_alcs_init.
• Adds a debug trace when re-entrant resolution is skipped.

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

[BrzVlad]

Why is this a llvm bump regression ?

[pavelsavara]

Why is this a llvm bump regression ?

@BrzVlad the LLVM-23 bump's main contribution to #129693 was turning on a test leg that finally ran the offending test in the only mode that crashes — the bump branch went red, we triaged the new failures, and this latent ALC re-entrancy fell out of that.

More details here https://gist.github.com/pavelsavara/87f39ec5f1387a3770d11b2e2793b96f

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

src/tests/JIT/Regression/JitBlue/Runtime_105619/Runtime_105619.cs:125

• TestEntryPoint currently swallows all exceptions, which means the test can still pass even if the collectible ALC load or reflection lookup fails (e.g., type/method not found) as long as the process doesn’t crash. Now that the MonoFULLAOT ActiveIssue is removed, it would be better for the test to fail on unexpected load/lookup errors and only ignore exceptions thrown by the invoked payload (typically wrapped in TargetInvocationException).

        try
        {
            CollectibleALC alc = new CollectibleALC();
            System.Reflection.Assembly asm = alc.LoadFromAssemblyPath(System.Reflection.Assembly.GetExecutingAssembly().Location);
            System.Reflection.MethodInfo mi = asm.GetType(typeof(Program).FullName).GetMethod(nameof(MainInner));