Skip to content

[mono] AOT-compile marshalling wrappers for layout classes with class-typed fields#129710

Merged
pavelsavara merged 4 commits into
dotnet:mainfrom
pavelsavara:mono-aot-marshal-layout-class
Jun 24, 2026
Merged

[mono] AOT-compile marshalling wrappers for layout classes with class-typed fields#129710
pavelsavara merged 4 commits into
dotnet:mainfrom
pavelsavara:mono-aot-marshal-layout-class

Conversation

@pavelsavara

@pavelsavara pavelsavara commented Jun 22, 2026

Copy link
Copy Markdown
Member

Summary

The Mono AOT compiler emits per-type StructureToPtr/PtrToStructure marshalling wrappers for [StructLayout] value types and sequential-layout classes, but only when can_marshal_struct() considers the type marshalable. That helper inspects each instance field and rejects the type if it encounters a field type it doesn't explicitly allow.

can_marshal_struct() had no case for MONO_TYPE_CLASS, so any layout class whose fields are themselves marshalable layout classes fell through to the default: branch and was treated as non-marshalable. A representative example is:

[StructLayout(LayoutKind.Sequential)]
class POINT { public int x, y; }

[StructLayout(LayoutKind.Sequential)]
class MINMAXINFO { public POINT ptMinTrackSize, ptMaxTrackSize; }

Because MINMAXINFO was rejected, its StructureToPtr/PtrToStructure wrappers were never emitted into the AOT image. The runtime marshals such a type just fine (it marshals the nested layout class as an embedded struct), so the call works under JIT, but under full-AOT the wrapper is generated on demand:

System.ExecutionEngineException: Attempting to JIT compile method
'(wrapper other) void MINMAXINFO:StructureToPtr (object,intptr,bool)'
while running in aot-only mode.

Fix

Handle MONO_TYPE_CLASS fields the same way as MONO_TYPE_VALUETYPE ΓÇö recurse into can_marshal_struct(). The existing mono_class_is_auto_layout() check at the top of the helper keeps the analysis conservative: non-layout class fields (e.g. object) still cause the type to be considered non-marshalable, while sequential/explicit-layout class fields recurse and are accepted when they are themselves marshalable.

With the fix, the MINMAXINFO/POINT StructureToPtr/PtrToStructure wrappers are present in the AOT image and the call succeeds under full-AOT.

Testing

Re-enables the JIT/Regression/JitBlue/WPF_3226 test on Mono (the [ActiveIssue] annotation is removed so CI exercises the fix). Validated locally with an x64 Mono LLVM full-AOT build: the wrappers now appear in the AOT image and the test passes (it previously crashed with the ExecutionEngineException above).

Note

This change was authored with the assistance of GitHub Copilot.


Part of the MonoAOT LLVM 23 full-AOT regression set tracked by #129508. Built and tested together with the Emscripten 5.0.6 / LLVM 23 bump on #129396, where the re-enabled WPF_3226 test passes on the runtime-llvm AllSubsets_Mono_LLVMFULLAOT_RuntimeTests leg.

Contributes to #129508.

…-typed fields

can_marshal_struct() in the AOT compiler did not handle MONO_TYPE_CLASS fields, so a [StructLayout] class whose fields are themselves marshalable layout classes (e.g. MINMAXINFO with two POINT fields) was considered not marshalable. Its StructureToPtr/PtrToStructure wrappers were therefore not emitted into the AOT image, and calling Marshal.StructureToPtr/PtrToStructure under full-AOT failed with 'Attempting to JIT compile method ... while running in aot-only mode'.

Handle MONO_TYPE_CLASS the same way as MONO_TYPE_VALUETYPE by recursing into can_marshal_struct; the existing auto-layout check keeps non-layout class fields conservative.

Re-enables the JIT/Regression/JitBlue/WPF_3226 test on Mono.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates Mono’s AOT wrapper inclusion logic so sequential/explicit-layout classes with class-typed fields can be considered marshalable (enabling StructureToPtr/PtrToStructure wrappers to be emitted ahead-of-time), and re-enables a regression test on Mono.

Changes:

  • Extend can_marshal_struct() to handle MONO_TYPE_CLASS fields by recursing into the nested class.
  • Re-enable WPF_3226 regression coverage on Mono by removing the [ActiveIssue] skip.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File Description
src/mono/mono/mini/aot-compiler.c Adds MONO_TYPE_CLASS handling in can_marshal_struct() to allow nested layout classes to qualify for AOT marshalling wrappers.
src/tests/JIT/Regression/JitBlue/WPF_3226/CSharpRepro/WPF_3226.cs Removes Mono-specific [ActiveIssue] skip so CI exercises the scenario again.

Comment thread src/mono/mono/mini/aot-compiler.c Outdated
Comment thread src/mono/mono/mini/aot-compiler.c Outdated
pavelsavara added a commit that referenced this pull request Jun 22, 2026
Re-link the disabled Mono full-AOT tests from the #129508 tracking issue to the individual PRs that fix them (nullabletypes -> #129702, call05_large/small -> #129708, WPF_3226 -> #129710, b143840 -> #129713, UnitTest_GVM_TypeLoadException -> #129715). The tests stay disabled; Runtime_105619 keeps the #129508 link.
The new MONO_TYPE_CLASS handling recurses into class-typed fields, but unlike value types a reference-class field can point back to its declaring class (directly or indirectly). On a self-referential or cyclic [StructLayout] class this recursed without bound and overflowed the stack, crashing the AOT compiler (observed while AOT-compiling framework assemblies such as System.Net.Security and System.Private.CoreLib). Add a recursion-depth guard: such types cannot be marshalled as a flat struct anyway, so treat them as non-marshalable instead of overflowing.
Copilot AI review requested due to automatic review settings June 23, 2026 10:17

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

Comment thread src/mono/mono/mini/aot-compiler.c Outdated
Use the codebase's standard spelling 'marshalable'.
pavelsavara added a commit to pavelsavara/runtime that referenced this pull request Jun 23, 2026
…ine jmp comment, fix marshalable typo

Integrates the minor review follow-ups from dotnet#129708 and dotnet#129710.
@pavelsavara

Copy link
Copy Markdown
Member Author

/ba-g unrelated failures

@pavelsavara pavelsavara merged commit 43b739f into dotnet:main Jun 24, 2026
127 of 130 checks passed
@pavelsavara pavelsavara deleted the mono-aot-marshal-layout-class branch June 24, 2026 10:13
@dotnet-milestone-bot dotnet-milestone-bot Bot added this to the 11.0-preview7 milestone Jun 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants