Skip to content

[Repo Assist] Add more stress benchmarks and perf improvements (transTypeRef + transMethRef caching) - #457

Merged
dsyme merged 2 commits into
masterfrom
repo-assist/benchmarks-stress-and-perf-improvements-ee06ccfc1d1b4e6f
Feb 26, 2026
Merged

[Repo Assist] Add more stress benchmarks and perf improvements (transTypeRef + transMethRef caching)#457
dsyme merged 2 commits into
masterfrom
repo-assist/benchmarks-stress-and-perf-improvements-ee06ccfc1d1b4e6f

Conversation

@github-actions

Copy link
Copy Markdown
Contributor

🤖 This PR was created by Repo Assist, an automated AI assistant. Closes / contributes to #341.

What this PR does

Continues the perf-improvement work started in PR #443 by adding two new memoization caches in AssemblyCompiler.Compile and significantly expanding the benchmark suite with realistic stress scenarios.


Performance improvements in ProvidedTypes.fs

1. transTypeRefCache — cache ILTypeRef per System.Type

let transTypeRefCache = Dictionary(Type, ILTypeRef)()
...
and transTypeRef (ty: Type) = 
    let ty = if ty.IsGenericType then ty.GetGenericTypeDefinition() else ty
    match transTypeRefCache.TryGetValue(ty) with
    | true, tref -> tref
    | false, _ ->
    let tref = ILTypeRef(transTypeRefScope ty, ...)
    transTypeRefCache.[ty] <- tref
    tref

Previously transTypeRef allocated a new ILTypeRef object on every call even for the same type. transType caches the full ILType, but transTypeRef is also called independently from transFieldSpec, transMethRef, etc. — so the same ILTypeRef was being reconstructed many times per compilation. The cache eliminates these redundant allocations.

2. transMethRefCache — cache ILMethodRef per MethodInfo

let transMethRefCache = Dictionary(MethodInfo, ILMethodRef)()

let transMethRef (m:MethodInfo) = 
    ...
    let m2 = m.GetDefinition()
    match transMethRefCache.TryGetValue(m2) with
    | true, mref -> mref
    | false, _ ->
    ...
    transMethRefCache.[m2] <- mref
    mref
```

In a type provider that generates many method bodies calling the same BCL methods (e.g. `String.IsNullOrEmpty` for null-checking in data-validation helpers), `transMethRef` was called once per generated method, producing identical `ILMethodRef` objects every time. The cache reduces this to one allocation per distinct BCL method signature per compilation.

Both caches are local to each `Compile()` invocation (fresh `Dictionary` created per assembly emission), so there is no cross-compilation state leakage.

---

### New benchmark scenarios

| Scenario | What it covers |
|----------|----------------|
| `PropertyHeavyBenchmark` (50/200 types × 10 props) | `ProvidedField` backing + `FieldGet`/`FieldSet` getter/setter bodies — simulates REST API DTOs |
| `ComplexBodyBenchmark` (100/300 types × 10 methods) | Method bodies calling real BCL methods (`String.IsNullOrEmpty`, `Object.ToString`) — exercises `transMeth`/`transMethRef` |
| `NestedTypesBenchmark` (30/100 outer types × depth 4) | Recursive nested-type hierarchyexercises the `defineNestedTypes` walk |
| `XLargeStressBenchmark` (1 000 types × 10 methods) | Amplifies any O() behaviour that smaller scenarios miss |

`runQuickMeasurement` (`--quick` mode) now covers all four new scenario families, with output like:

```
Scenario                                                          Mean
---------------------------------------------------------------------------
Methods  50 types × 20 methods                            939.2 ms/iter
Methods 200 types × 20 methods                           3737.0 ms/iter
Methods 500 types × 10 methods                           4556.3 ms/iter
Methods 1000 types × 10 methods                          9323.0 ms/iter

Props    50 types × 10 props                             1056.4 ms/iter
Props   200 types × 10 props                             4074.6 ms/iter
Props   500 types × 10 props                            10084.7 ms/iter

ComplexBody 100 types × 10 methods                        924.4 ms/iter
ComplexBody 300 types × 10 methods                       2749.3 ms/iter

Nested  30 outer × depth 4 × 5 methods                    691.0 ms/iter
Nested 100 outer × depth 4 × 5 methods                   2216.0 ms/iter

(Numbers from a GitHub Actions runner — useful as relative comparison baselines.)


Test Status

✅ All 102 existing tests pass (dotnet test tests/ -c Release), 1 pre-existing skip unchanged.
✅ Benchmarks build cleanly (dotnet build benchmarks/... -c Release).
--quick mode runs to completion with correct output for all scenarios.

Generated by Repo Assist for issue #341

To install this workflow, run gh aw add githubnext/agentics/workflows/repo-assist.md@f2c5cf1e4af58e09a93ba0703c6bf084711b265f. View source at https://github.com/githubnext/agentics/tree/f2c5cf1e4af58e09a93ba0703c6bf084711b265f/workflows/repo-assist.md.

…sMethRef caching)

- Add transTypeRefCache in AssemblyCompiler.Compile: caches ILTypeRef per
  System.Type (using generic type definition as key). Avoids allocating
  duplicate ILTypeRef objects when the same BCL type appears repeatedly as
  field/parameter/declaring type across many schema members.

- Add transMethRefCache in AssemblyCompiler.Compile: caches ILMethodRef
  per MethodInfo (using GetDefinition() as key). Eliminates redundant
  ILMethodRef allocations when the same BCL method is called from many
  generated method bodies (e.g. String.IsNullOrEmpty in data validation).

- New benchmark helpers:
  * buildProviderWithProperties: types with ProvidedField backing +
    getter (FieldGet) and setter (FieldSet) bodies — simulates REST API
    data-transfer-object schemas
  * buildProviderWithComplexBodies: methods whose bodies call real BCL
    methods (String.IsNullOrEmpty / Object.ToString) — exercises the
    transMeth/transMethRef path
  * buildProviderWithNestedTypes: outer types with configurable nesting
    depth — exercises the recursive type-definition walk in Compile()

- New BenchmarkDotNet classes: PropertyHeavyBenchmark,
  ComplexBodyBenchmark, NestedTypesBenchmark, XLargeStressBenchmark
  (1000 types × 10 methods)

- Updated runQuickMeasurement to cover all new scenarios with clear
  labelled output

- Updated README with new benchmark table and optimisation notes

All 102 existing tests pass unchanged.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@dsyme
dsyme marked this pull request as ready for review February 26, 2026 11:26
@dsyme
dsyme merged commit 6b3cafc into master Feb 26, 2026
2 checks passed
dsyme pushed a commit that referenced this pull request Mar 7, 2026
🤖 *This PR was created by Repo Assist, an automated AI assistant.*

Prepares the RELEASE_NOTES.md for version **8.3.0** — a minor release
capturing significant improvements merged since 8.2.0 (February 24).

## Changes since 8.2.0

| Type | PR | Description |
|------|----|-------------|
| 🐛 Bug fix | #432 | Fix custom attributes on nested erased types |
| 🐛 Bug fix | #458 | Fix `GetNestedType` on
`TypeSymbol`/`ProvidedTypeSymbol` for generic provided types |
| 🐛 Bug fix | #459 | Fix mutable variable captures in
`QuotationSimplifier` — promote to ref cells |
| ⚡ Performance | #443 | Memoize `transType` in `AssemblyCompiler` to
reduce redundant type translation |
| ⚡ Performance | #457 | Cache `transTypeRef` and `transMethRef` in
assembly compiler |
| ✨ Feature | #428 | New warning when all static parameters in a type
provider are optional |
| 📚 Docs | #455 | Documentation guide overhaul |
| 🧪 Tests | #442 | Add coverage tests and Coverage build target |
| 🔧 Toolchain | #431 | Update to .NET 8 SDK and toolchain |

This is a minor version bump (8.2.0 → 8.3.0) due to the new feature
(#428) and significant improvements. If preferred, a patch release
(8.2.1) is also reasonable given the emphasis on bug fixes.

## Test Status

This PR only modifies RELEASE_NOTES.md. The build/test status for the
underlying changes is tracked in the individual PRs listed above (all
passed CI before merging).

---

*To release: merge this PR, then tag `v8.3.0` and publish the NuGet
package via the existing build pipeline.*




> Generated by [Repo
Assist](https://github.com/fsprojects/FSharp.TypeProviders.SDK/actions/runs/22448247879)
>
> To install this [agentic
workflow](https://github.com/githubnext/agentics/tree/afb00b92a9514fee9a14c583f059a03d05738f70/workflows/repo-assist.md),
run
> ```
> gh aw add
githubnext/agentics@afb00b9
> ```

<!-- gh-aw-agentic-workflow: Repo Assist, engine: copilot, id:
22448247879, workflow_id: repo-assist, run:
https://github.com/fsprojects/FSharp.TypeProviders.SDK/actions/runs/22448247879
-->

<!-- gh-aw-workflow-id: repo-assist -->

---------

Co-authored-by: Repo Assist <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
dsyme added a commit that referenced this pull request Jul 29, 2026
…er body (issue #341) (#520)

## Problem

Issue #341 reported that generative compilation
(`AssemblyCompiler.Compile`) is still very slow for large providers
(SwaggerProvider + Stripe schema), even after the memoization work in
#443/#457/#471/#493/#497/#502/#509/#513.

Profiling the benchmark suite with dotnet-trace showed that with those
caches in place, **~80% of `Compile` time was FSharp.Core quotation
deserialization** (`PatternsModule.deserialize` → `instMeth` →
`GetGenericArguments`/`MakeGenericType`), attributed to the
`CodeGenerator` constructor.

Root cause: `CodeGenerator` defines ~63 active patterns from quotation
literals — `let (|Addition|_|) = (|SpecificCall|_|) <@ (+) @>`, `<@ max
@>`, `<@ sqrt @>`, etc. Evaluating a quotation literal deserializes it
(reflection-binding every method it mentions), and a `CodeGenerator` is
constructed **once per emitted member body**. A provider with 10,000
members therefore deserialized those quotations ~630,000 times.
`lessThan` was worse still: it deserialized `<@@ (<) @@>` on every call.

## Fix

* New `CodeGenPatterns(convTypeToTgt)` type holding all these patterns
as `member val` properties, so they are computed once at construction (a
plain property getter would re-evaluate per access).
* `AssemblyCompiler.Compile` constructs one instance and passes it to
every `CodeGenerator`; the constructor now just re-binds the local
active patterns from it, so the large `emitExpr` match code is
untouched.
* `lessThan` uses a precomputed generic method definition.

## Results

Quick benchmark (`benchmarks -- --quick`, ms/iter, before → after):

| Scenario | Before | After | Speedup |
|---|---|---|---|
| Methods 50×20 | 880 | 86 | 10× |
| Methods 200×20 | 2893 | 180 | 16× |
| Methods 1000×10 | 6892 | 535 | 13× |
| Props 500×10 | 8074 | 1259 | 6.4× |
| ComplexBody 300×10 | 2528 | 158 | 16× |
| Nested 100×4×5 | 1668 | 127 | 13× |

End-to-end with the exact repro from #341 (SwaggerProvider built from
source against this branch, Stripe `spec3.sdk.json`, `dotnet fsi`):
**62–72 s → 19–20 s** wall clock; the remainder is dominated by schema
download, OpenApi parsing and fsi startup rather than the TPSDK.

Re-profiling after the change confirms the SDK-side deserialization is
gone; the remaining `deserialize` samples come from the provider
author's own `<@@ ... @@>` literals in `invokeCode` (once per member,
inherent to user code).

## Testing

* Full test suite passes: 160/160.
* Benchmark suite builds and runs (numbers above).
* SwaggerProvider builds unmodified against the patched
`ProvidedTypes.fs` and the generated Stripe client works
(`Stripe.Client()` constructed successfully).

Fixes #341 (the remaining slowness reported there after the earlier
rounds of caching).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Don Syme <dsyme@github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Provided Types performance, parsing is still slow

1 participant