Add APIs to BlobBuilder for customizing the underlying byte array et al.

[teo-tsirpanis]

Fixes #99244
Fixes #100418

This PR builds on top of @jaredpar's branch to add APIs for customizing the underlying buffer of a BlobBuilder. The chunking logic of BlobBuilder was updated to allocate multiple additional chunks with a user-customizable maximum size each. As part of this, we use APIs from System.Text.Unicode.Utf8 to encode UTF-8 strings, which increases performance and safety, and reduces duplicate code.

[ghost]

Note regarding the new-api-needs-documentation label:

This serves as a reminder for when your PR is modifying a ref *.cs file and adding/modifying public APIs, please make sure the API implementation in the src *.cs file is documented with triple slash comments, so the PR reviewers can sign off that change.

[ghost]

Note regarding the new-api-needs-documentation label:

This serves as a reminder for when your PR is modifying a ref *.cs file and adding/modifying public APIs, please make sure the API implementation in the src *.cs file is documented with triple slash comments, so the PR reviewers can sign off that change.

[Copilot]

Pull Request Overview

This PR adds new APIs to BlobBuilder for customizing the underlying byte array and updates related encoding and buffer-handling logic across metadata and core libraries. Key changes include replacing legacy UTF-8 encoding code with calls to the new System.Text.Unicode.Utf8 APIs, updating BlobBuilder’s API surface (including new constructors and properties), and adding NET-specific intrinsics support across several core modules.

Reviewed Changes

Copilot reviewed 31 out of 36 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
System/Reflection/Internal/Utilities/StreamExtensions.cs	Removed obsolete TryReadAll overload for Span to rely on newer API paths.
System/Reflection/Internal/Utilities/BlobUtilities.cs	Rewrote WriteUtf8 to use Utf8.FromUtf16 for encoding, replacing manual UTF-8 encoding logic.
System/Reflection/Metadata.cs	Added new BlobBuilder constructors, properties, and APIs including ReadOnlySpan/WriteBytes overloads.
System.Private.CoreLib (various files)	Updated intrinsics and preprocessor conditions (#if NET, #if SYSTEM_PRIVATE_CORELIB) for newer vectorized and ASCII helper routines.
Microsoft.Bcl.Memory (PACKAGE.md and others)	Updated documentation and type forwarding to include UTF-8 APIs for NET platforms.

Files not reviewed (5)

• src/libraries/Microsoft.Bcl.Memory/src/Microsoft.Bcl.Memory.csproj: Language not supported
• src/libraries/Microsoft.Bcl.Memory/tests/Microsoft.Bcl.Memory.Tests.csproj: Language not supported
• src/libraries/System.Reflection.Metadata/System.Reflection.Metadata.sln: Language not supported
• src/libraries/System.Reflection.Metadata/src/Resources/Strings.resx: Language not supported
• src/libraries/System.Reflection.Metadata/src/System.Reflection.Metadata.csproj: Language not supported

[jkotas]

As part of this, we use APIs from System.Text.Unicode.Utf8 to encode UTF-8 strings, which increases performance and safety, and reduces duplicate code.

Is this needed to introduce the new APIs?

System.Text.Unicode.Utf8 change introduces a new dependency for System.Reflection.Metadata on .NET Framework that will be an extra work to push through the system. It would be better to avoid bundling the two changes together in a single PR.

[am11]

As part of this, we use APIs from System.Text.Unicode.Utf8 to encode UTF-8 strings, which increases performance and safety, and reduces duplicate code.

Is this needed to introduce the new APIs?

The other PR is approved and ready to merge #111292. After the merge and rebase this branch against main, those commits will disappear.

[jkotas]

The other PR is approved and ready to merge #111292. After the merge and rebase this branch against main, those commits will disappear.

The change that introduces System.Reflection.Metadata dependency on Microsoft.Bcl.Memory won't disappear.

[teo-tsirpanis]

Switched back to the old WriteUTF8 implementation for downlevel frameworks, which was rewritten to use spans and remove unsafe code. On modern .NET we still use System.Text.Unicode.Utf8.

Tests pass locally. This is ready for review.

[jkotas]

WriteUTF8 implementation for downlevel frameworks, which was rewritten to use spans and remove unsafe code

What''s the performance regression introduced by this rewrite on .NET Framework?

Our primary interest in removing unsafe code is on latest .NET. It is fine to keep unsafe code for .NET Framework if it is required for decent performance.

[teo-tsirpanis]

I updated the function to use unsafe code and wrote a benchmark to compare it with my initial safe edition. We cannot compare it with the existing unsafe implementation since the functions don't have the same signature and semantics. The numbers look promising so I switched to the unsafe edition:

BenchmarkDotNet v0.14.0, Windows 10 (10.0.19045.5796/22H2/2022Update)
Intel Core i7-7700HQ CPU 2.80GHz (Kaby Lake), 1 CPU, 8 logical and 4 physical cores
  [Host]     : .NET Framework 4.8.1 (4.8.9310.0), X64 RyuJIT VectorSize=256
  DefaultJob : .NET Framework 4.8.1 (4.8.9310.0), X64 RyuJIT VectorSize=256

Method	N	Mean	Error	StdDev	Ratio	RatioSD
Safe	16	276.9 ns	5.38 ns	4.77 ns	1.00	0.02
Unsafe	16	173.1 ns	3.05 ns	2.85 ns	0.63	0.01
Safe	128	1,946.7 ns	35.89 ns	31.81 ns	1.00	0.02
Unsafe	128	1,210.2 ns	17.84 ns	15.81 ns	0.62	0.01

Benchmark code

// See https://aka.ms/new-console-template for more information
using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Running;
using System.Runtime.InteropServices;
using System.Text;

BenchmarkRunner.Run<Utf8Bench>();

public class Utf8Bench
{
    public string TestString = null!;

    public byte[] TestBytes = new byte[2048];

    [Params(16, 128)]
    public int N { get; set; }

    [GlobalSetup]
    public void Setup()
    {
        var sb = new StringBuilder();
        for (int i = 0; i < N; i++)
        {
            sb.Append('a');
        }
        for (int i = 0; i < N; i++)
        {
            sb.Append('Θ');
        }
        for (int i = 0; i < N; i++)
        {
            sb.Append("😂");
        }
        TestString = sb.ToString();
        TestBytes = new byte[2048];
    }

    [Benchmark(Baseline = true)]
    public int Safe()
    {
        WriteUtf8Safe(TestString.AsSpan(), TestBytes, out int charsRead, out int bytesWritten, true);
        return charsRead + bytesWritten;
    }

    [Benchmark]
    public int Unsafe()
    {
        WriteUtf8Unsafe(TestString.AsSpan(), TestBytes, out int charsRead, out int bytesWritten, true);
        return charsRead + bytesWritten;
    }

    public static void WriteUtf8Safe(ReadOnlySpan<char> source, Span<byte> destination, out int charsRead, out int bytesWritten, bool allowUnpairedSurrogates)
    {
        // Copy from PR
    }

    public static unsafe void WriteUtf8Unsafe(ReadOnlySpan<char> source, Span<byte> destination, out int charsRead, out int bytesWritten, bool allowUnpairedSurrogates)
    {
        // Copy from PR
    }
}

[teo-tsirpanis]

I don't understand how subclasses are supposed to override this. They will reassign the Buffer property, but according to a comment in #100418, reassigning Buffer clears the head chunk.

What are the semantics? Should Capacity's setter have additional logic?

[stephentoub]

cc: @jaredpar

[teo-tsirpanis]

I came up with an alternative design that removes the need for the SetCapacity method. A buffer is resized by allocating a new BlobBuilder, swapping the buffers of this and the new BlobBuilder (while copying any data), and freeing the new BlobBuilder that now contains the old buffer.

This results in a simplified set of methods to extend, with the caveat that linking pooled and non-pooled BlobBuilders will not be supported. Applications will have to ensure that they always use pooled builders, and override OnLinking to throw if that's not the case. I can imagine this is possible to guarantee in Roslyn.

Let me know what you think.

[jkotas]

I would like to see the numbers for the implementation in this PR and including walk clock time.

[agocke]

It sounds like there's an assumption that we need to bring this new API surface to the .NET Framework polyfill. Why is that the case?

[jkotas]

It sounds like there's an assumption that we need to bring this new API surface to the .NET Framework polyfill. Why is that the case?

This is a question for Roslyn (@jaredpar). These APIs are designed to optimize Roslyn. If Roslyn is fine with missing these APIs on netfx, we can certainly drop them there.

[teo-tsirpanis]

I would like to see the numbers for the implementation in this PR and including walk clock time.

I wrote a benchmark of BlobBuilder.WriteUTF8 similar to my previous one, comparing between SRM from NuGet and a local build of this branch, running on .NET Framework and .NET 10. I also included writing ASCII-only strings, which is what the overwhelming majority of metadata identifiers are going to be in a compilation.

.NET Framework

BenchmarkDotNet v0.15.8, Windows 10 (10.0.19045.6456/22H2/2022Update)
Intel Core i7-7700HQ CPU 2.80GHz (Max: 2.81GHz) (Kaby Lake), 1 CPU, 8 logical and 4 physical cores
  [Host]     : .NET Framework 4.8.1 (4.8.9310.0), X64 RyuJIT VectorSize=256
  DefaultJob : .NET Framework 4.8.1 (4.8.9310.0), X64 RyuJIT VectorSize=256

Method	AsciiOnly	N	Mean	Error	StdDev
Write_Before	False	16	222.5 ns	3.81 ns	3.38 ns
Write_Before	True	16	164.9 ns	0.71 ns	0.67 ns
Write_Before	False	128	1,474.0 ns	7.12 ns	6.31 ns
Write_Before	True	128	993.3 ns	5.79 ns	5.13 ns
Write_After	False	16	219.7 ns	1.62 ns	1.26 ns
Write_After	True	16	148.4 ns	1.09 ns	0.91 ns
Write_After	False	128	1,291.5 ns	6.24 ns	5.53 ns
Write_After	True	128	698.3 ns	4.26 ns	3.99 ns

.NET 10

BenchmarkDotNet v0.15.8, Windows 10 (10.0.19045.6456/22H2/2022Update)
Intel Core i7-7700HQ CPU 2.80GHz (Max: 2.81GHz) (Kaby Lake), 1 CPU, 8 logical and 4 physical cores
.NET SDK 10.0.102
  [Host] : .NET 10.0.2 (10.0.2, 10.0.225.61305), X64 RyuJIT x86-64-v3

Job=InProcess  Toolchain=InProcessEmitToolchain  

Method	AsciiOnly	N	Mean	Error	StdDev
Write_Before	False	16	154.05 ns	1.515 ns	1.265 ns
Write_Before	True	16	65.35 ns	0.233 ns	0.206 ns
Write_Before	False	128	1,159.08 ns	7.941 ns	7.428 ns
Write_Before	True	128	467.17 ns	2.684 ns	2.241 ns
Write_After	False	16	85.51 ns	1.687 ns	2.675 ns
Write_After	True	16	19.53 ns	0.146 ns	0.114 ns
Write_After	False	128	532.97 ns	7.474 ns	6.991 ns
Write_After	True	128	28.57 ns	0.308 ns	0.257 ns

Sources are attached. I manually changed the TFM, the UseStockSrm property, and the path to the local SRM assembly, and concatenated the results into the tables above.

[Copilot]

Pull request overview

Copilot reviewed 20 out of 20 changed files in this pull request and generated 6 comments.

[Copilot]

The Buffer property setter doesn't validate that the new buffer is at least MinChunkSize (16 bytes) long. This validation is performed in the constructor that accepts a buffer parameter, but not in the property setter. If a derived type sets Buffer to an array smaller than 16 bytes, subsequent operations may fail or cause undefined behavior.

[teo-tsirpanis]

Added comment on why this is by design.

[Copilot]

Pull request overview

Copilot reviewed 20 out of 20 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 20 out of 20 changed files in this pull request and generated 4 comments.

[Copilot]

ResizeChunk calls OnLinking(this, newChunk) even though the operation is just a buffer resize/reallocation (not truly linking two builders). The newChunk returned from AllocateChunk is never added to the chain—its buffer is stolen and it's immediately freed. This will trigger the OnLinking override (intended for detecting linking between different BlobBuilder types) on every capacity resize or Capacity set, which is semantically incorrect. Derived types that override OnLinking to enforce pool consistency will get spurious notifications. Consider separating the "allocate a new buffer" concern from the "link two builders" notification.

Suggested change

OnLinking(this, newChunk);

[teo-tsirpanis]

We are calling OnLinking out of abundance of caution, to enforce pool consistency between the newly allocated chunk, because we are going to swap their buffers. Added comment explaining it.