Performance: Limit maximum ASCII byte scan range to codepoint limit it _wp_scan_utf8

[sirreal]

When _wp_scan_utf8() is invoked with a maximum number of codepoints, the ASCII fast-path will still scan the entire input string. In the case of long ASCII-only strings searching for a limited number of codepoints, this is wasteful and can be optimized by only searching up to remaining number of desired codepoints.

ASCII are all 1-byte UTF-8 codepoints, so the maximum number of ASCII bytes can be limited by the maximum number of remaining codepoints.

See the demonstration below.

wordpress-develop/src/wp-includes/compat-utf8.php

Lines 56 to 69 in c710ca6

	/*
	* Quickly skip past US-ASCII bytes, all of which are valid UTF-8.
	*
	* This optimization step improves the speed from 10x to 100x
	* depending on whether the JIT has optimized the function.
	*/
	$ascii_byte_count = strspn(
	$bytes,
	"\x00\x01\x02\x03\x04\x05\x06\x07\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" .
	"\x10\x11\x12\x13\x14\x15\x16\x17\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f" .
	" !\"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\\]^_`abcdefghijklmnopqrstuvwxyz{|}~\x7f",
	$i,
	$end - $i
	);

Demonstration

php \
    -d memory_limit=8G \
    -d opcache.enable_cli=1 \
    -d opcache.enable=1 \
    -d opcache.jit_buffer_size=128M \
    -d opcache.jit=tracing \
test.php

// test.php
<?php
require_once 'src/wp-includes/compat-utf8.php';

                   //"Hello, 🌎!"
$s                 = "Hello, \xF0\x9F\x8C\x8E!" .
                     str_repeat( 'abc123', 512 * 1024 * 1024 ) .
                   //Invalid UTF-8 byte
                     "\xF1";
$at                = 0;
$invalid_length    = 0;
$max_bytes         = null;
$max_codepoints    = null;
$has_noncharacters = false;
$dur               = -hrtime( true );
$scanned           = _wp_scan_utf8( $s, $at, $invalid_length, $max_bytes, $max_codepoints, $has_noncharacters );
$dur              += hrtime( true );

echo <<<"TEXT"
Found {$invalid_length} invalid UTF-8 bytes up to position {$scanned} in {$dur}ms.
Max bytes: {$max_bytes}.
Max codepoints: {$max_codepoints}.
===

TEXT;

$at                = 0;
$invalid_length    = 0;
$max_bytes         = 12;
$max_codepoints    = null;
$has_noncharacters = false;
$dur               = -hrtime( true );
$scanned           = _wp_scan_utf8( $s, $at, $invalid_length, $max_bytes, $max_codepoints, $has_noncharacters );
$dur              += hrtime( true );

echo <<<"TEXT"
Found {$invalid_length} invalid UTF-8 bytes up to position {$scanned} in {$dur}ms.
Max bytes: {$max_bytes}.
Max codepoints: {$max_codepoints}.
===

TEXT;

$at                = 0;
$invalid_length    = 0;
$max_bytes         = null;
$max_codepoints    = 9;
$has_noncharacters = false;
$dur               = -hrtime( true );
$scanned           = _wp_scan_utf8( $s, $at, $invalid_length, $max_bytes, $max_codepoints, $has_noncharacters );
$dur              += hrtime( true );

echo <<<"TEXT"
Found {$invalid_length} invalid UTF-8 bytes up to position {$scanned} in {$dur}ms.
Max bytes: {$max_bytes}.
Max codepoints: {$max_codepoints}.
===

TEXT;

$dur   = -hrtime( true );
$count = _wp_utf8_codepoint_count( $s, 0, 9 );
$span  = _wp_utf8_codepoint_span( $s, 0, 9 );
$dur  += hrtime( true );
$text  = substr( $s, 0, $span );

echo <<<"TEXT"
Counted {$count} codepoints in {$span} bytes in {$dur}ms:
{$text}
TEXT;

Output diff:

• trunk@ c710ca6
• this branch@ 5829e04

diff --git 1/trunk.txt 2/branch.txt
index 70190c68b8..535b49e7df 100644
--- 1/trunk.txt
+++ 2/branch.txt
@@ -1,14 +1,14 @@
-Found 1 invalid UTF-8 bytes up to position 3221225481 in 1453752292ms.
+Found 1 invalid UTF-8 bytes up to position 3221225481 in 1436277917ms.
 Max bytes: .
 Max codepoints: .
 ===
-Found 0 invalid UTF-8 bytes up to position 9 in 4417ms.
+Found 0 invalid UTF-8 bytes up to position 9 in 8208ms.
 Max bytes: 12.
 Max codepoints: .
 ===
-Found 0 invalid UTF-8 bytes up to position 9 in 1005872250ms.
+Found 0 invalid UTF-8 bytes up to position 9 in 833ms.
 Max bytes: .
 Max codepoints: 9.
 ===
-Counted 8 codepoints in 12 bytes in 978311000ms:
+Counted 8 codepoints in 12 bytes in 8084ms:
 Hello, 🌎!

This seems to address a performance issue noted by @adamziel when reading codepoints in a 10MB XML file. The file is mostly ASCII (>98%), and at each codepoint boundary the rest of the document was scanned to find the ASCII code point count, often reading very large chunks of the document repeatedly. This is fixed by recognizing the max codepoint limit in the ASCII fast path.

performance benchmark results

Before (trunk@ c710ca6 )

Timed: 8.93s user 0.02s system 99% cpu 8.982 total

Parsing a 10MB XML file with next_codepoint_wp_scan_utf8...
Starting at: 1781785757.8389
Parsed 100000 codepoints in 0.56971597671509 seconds
Parsed 200000 codepoints in 2.5511300563812 seconds
Parsed 300000 codepoints in 4.1061329841614 seconds
Parsed 400000 codepoints in 4.4027769565582 seconds
Parsed 500000 codepoints in 4.5246770381927 seconds
Parsed 600000 codepoints in 4.7350499629974 seconds
Parsed 700000 codepoints in 5.7863998413086 seconds
Parsed 800000 codepoints in 7.3096778392792 seconds
Parsed 900000 codepoints in 8.7035548686981 seconds
Parsed 1000000 codepoints in 8.9111640453339 seconds

After (branch@ 5829e04 )

Timed: 0.19s user 0.01s system 96% cpu 0.214 total

Parsing a 10MB XML file with next_codepoint_wp_scan_utf8...
Starting at: 1781785739.3894
Parsed 100000 codepoints in 0.016802072525024 seconds
Parsed 200000 codepoints in 0.032394170761108 seconds
Parsed 300000 codepoints in 0.047827005386353 seconds
Parsed 400000 codepoints in 0.063305139541626 seconds
Parsed 500000 codepoints in 0.078748941421509 seconds
Parsed 600000 codepoints in 0.094329118728638 seconds
Parsed 700000 codepoints in 0.10992097854614 seconds
Parsed 800000 codepoints in 0.12537217140198 seconds
Parsed 900000 codepoints in 0.14099717140198 seconds
Parsed 1000000 codepoints in 0.15646696090698 seconds

Follow-up to: r60768
Trac ticket: https://core.trac.wordpress.org/ticket/65483

Use of AI Tools

AI assistance: Yes
Tool(s): Claude (Fable 5, Opus 4.8, etc.), Codex (GPT 5.5)
Used for: Fuzz testing and discovery, implementation, testing..

---

This Pull Request is for code review only. Please keep all other discussion in the Trac ticket. Do not merge this Pull Request. See GitHub Pull Requests for Code Review in the Core Handbook for more details.

[github-actions]

Hi there! 👋

Thank you for your contribution to WordPress! 💖

It looks like this is your first pull request to wordpress-develop. Here are a few things to be aware of that may help you out!

No one monitors this repository for new pull requests. Pull requests must be attached to a Trac ticket to be considered for inclusion in WordPress Core. To attach a pull request to a Trac ticket, please include the ticket's full URL in your pull request description.

Pull requests are never merged on GitHub. The WordPress codebase continues to be managed through the SVN repository that this GitHub repository mirrors. Please feel free to open pull requests to work on any contribution you are making.

More information about how GitHub pull requests can be used to contribute to WordPress can be found in the Core Handbook.

Please include automated tests. Including tests in your pull request is one way to help your patch be considered faster. To learn about WordPress' test suites, visit the Automated Testing page in the handbook.

If you have not had a chance, please review the Contribute with Code page in the WordPress Core Handbook.

The Developer Hub also documents the various coding standards that are followed:

• PHP Coding Standards
• CSS Coding Standards
• HTML Coding Standards
• JavaScript Coding Standards
• Accessibility Coding Standards
• Inline Documentation Standards

Thank you,
The WordPress Project

[github-actions]

Test using WordPress Playground

The changes in this pull request can previewed and tested using a WordPress Playground instance.

WordPress Playground is an experimental project that creates a full WordPress instance entirely within the browser.

Some things to be aware of

• All changes will be lost when closing a tab with a Playground instance.
• All changes will be lost when refreshing the page.
• A fresh instance is created each time the link below is clicked.
• Every time this pull request is updated, a new ZIP file containing all changes is created. If changes are not reflected in the Playground instance,
  it's possible that the most recent build failed, or has not completed. Check the list of workflow runs to be sure.

For more details about these limitations and more, check out the Limitations page in the WordPress Playground documentation.

Test this pull request with WordPress Playground.

[github-actions]

The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs. You can also manually ask me to refresh this list by adding the props-bot label.

Core Committers: Use this line as a base for the props when committing in SVN:

Props jonsurrell, dmsnell.

To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook.

[dmsnell]

Noting the missing @ticket reference in the tests, but this is great work. Thanks for finding this and adding the limit.

[github-actions]

A commit was made that fixes the Trac ticket referenced in the description of this pull request.

SVN changeset: 62523
GitHub commit: 54eca7b

This PR will be closed, but please confirm the accuracy of this and reopen if there is more work to be done.