Harden C extension for memory/pointer safety, GIL=0, and speed

[devin-ai-integration]

Summary

Harden the C extension and Python wrapper for pypcre, make hot paths faster, and release v0.4.0 with expanded test coverage.

• Memory safety / bounds / refcount

  • create_match_object validates ovector allocation size against SIZE_MAX, rejects ovec_count == 0, and checks ovector != NULL.
  • match_resolve_span and match_get_group_value use size_t indexing to avoid overflow on large group indices.
  • create_groupindex_dict bounds name length with strnlen instead of relying on PCRE2 null termination.
  • pcre_memory_initialize is now atomic/lock-safe using atomic_flag and ATOMIC_VAR; pcre_malloc/pcre_free/pcre_memory_allocator_name use acquire loads.

• Free-threaded / GIL=0

  • Existing Py_MOD_GIL_NOT_USED / PyUnstable_Module_SetGIL stays in place.
  • New atomic/lock-safe allocator init and PCRE2_CALL_MAYBE_RELEASE_GIL are no-ops under Py_GIL_DISABLED, so the module remains safe for PYTHON_GIL=0.

• Performance

  • PCRE2_CALL_MAYBE_RELEASE_GIL only drops the GIL when the match subject is larger than 256 KiB, avoiding the PyEval_{Save,Restore}Thread overhead on small calls.
  • Vectorized utf8_offset_to_index by counting UTF-8 starter bytes in 8-byte chunks with popcountll.
  • Vectorized utf8_index_to_offset by skipping 8-byte chunks whose starter count is smaller than the remaining index.
  • match_get_group_value keeps the fast PyUnicode_DecodeUTF8 path for group extraction; the vectorized utf8_offset_to_index is used only for span()/start()/end().
  • Skip PCRE2's redundant UTF-8 validation (PCRE2_NO_UTF_CHECK) for Python strings and fully validated bytes subjects.
  • Fix FindIter so a JIT NOMATCH no longer falls through to a redundant pcre2_match call.
  • Add a C findall implementation (Pattern_findall, module_findall) that builds the result list directly, bypassing the Python per-match iterator overhead.

• Bug/security lapses fixed

  • pcre_memory_initialize was not thread-safe; two threads could race through dlopen/pointer assignment.
  • FindIter_iternext ran pcre2_match after every JIT NOMATCH, doubling work.
  • utf8_offset_to_index allocated a temporary PyUnicode object for every span()/start()/end() call.
  • create_groupindex_dict read names with unbounded PyUnicode_FromString.

• Build/CI

  • Pin the release workflow's setuptools install to the pyproject.toml build-system range and fix the uv pip isntall typo.

• README

  • Replaced the benchmark table with head-to-head re/regex results and added benchmarks/competitor_bench.py so the numbers are reproducible.

• Version / test coverage

  • Bumped version to 0.4.0 in pyproject.toml and pcre_ext/pcre2.c __version__, with a README release note.
  • Added tests/test_re_compat.py, tests/test_threads.py, tests/test_cache_global.py, tests/test_cache_global_inproc.py, and tests/test_coverage_gaps.py to cover the remaining Python-layer corners (compile flags, finditer/findall/sub fallback loops, re_compat helpers, thread-pool lifecycle, and the global cache strategy).
  • Total Python-layer coverage is 99% (1316 statements, 14 missed lines that are unreachable dead code or the sre_parse fallback unavailable on CPython 3.14).

Benchmarks

All numbers are from a free-threaded Python 3.14.5 build with sys._is_gil_enabled() == False, branch vs. main.

Pattern.search(..., pos=i) over every position (tests/test_bench_string.py, 50 iterations)

subject	length	main (ms)	branch (ms)	speedup
2byte	64000	2049.4	593.4	71.0%
4byte	64000	1389.7	1223.8	11.9%
ascii	64000	29.6	30.2	~0%
latin1	64000	112.0	120.9	~noise

The large 2-byte/4-byte wins come from vectorizing utf8_index_to_offset; ASCII and latin1 are unchanged within run-to-run noise.

Match.span(0) + Match.group(0) micro-benchmark (1000 calls, median of 10)

text	main (ms)	branch (ms)	speedup
ascii	1.32	1.28	3%
2byte	16.28	9.28	43%
4byte	16.35	9.26	43%
mixed	14.20	8.16	43%

Pattern.findall(...) (\w+ over 20k/10k words, median of 30)

text	main (ms)	branch (ms)	speedup
ascii	5.74	2.61	54%
2byte	0.34	0.34	~0%
4byte	2.97	1.35	54%

Head-to-head vs re and regex (README highlight workloads)

Measured on Python 3.14.5 free-threaded x86_64 Linux, compiled-pattern reuse, JIT enabled, best-of-7. Only workloads where PyPcre is decisively faster are shown.

findall

Workload	PyPcre (ms)	re (ms)	regex (ms)	PyPcre edge
Extract WARN / ERROR lines (^(?:WARN|ERROR).*?$, MULTILINE)	0.60	29.35	30.85	49x vs re, 52x vs regex
Per-line full-name extraction (^[A-Z][a-z]+ [A-Z][a-z]+, MULTILINE)	1.02	29.50	15.95	29x vs re, 16x vs regex
Lookbehind + negative-lookahead tokens ((?:(?<=foo)bar|baz)(?!qux))	3.78	11.81	10.42	3.1x vs re, 2.8x vs regex

finditer

Workload	PyPcre (ms)	re (ms)	regex (ms)	PyPcre edge
Extract WARN / ERROR lines	0.60	29.33	31.23	49x vs re, 52x vs regex
Per-line full-name extraction	1.02	30.09	16.25	29x vs re, 16x vs regex

Multi-threaded findall (8 threads)

Workload	PyPcre (ms)	re (ms)	regex (ms)	PyPcre edge
Extract WARN / ERROR lines	3.26	32.57	34.41	10.0x vs re, 10.6x vs regex
Per-line full-name extraction	3.18	31.84	17.31	10.0x vs re, 5.4x vs regex

Validation

• pytest passes on free-threaded Python 3.14.5: 275 passed, 14 skipped, 545 subtests passed.
• bash format/format.sh passes.
• Python-layer coverage: 99% (1316 statements, 14 missed: unreachable/dead code or sre_parse fallback unavailable on CPython 3.14).
• Release CI green across Python 3.10–3.14 (including 3.14t) on Ubuntu/macOS/Windows.

Link to Devin session: https://app.devin.ai/sessions/bd880d99fc244c188c268e96d90ac233
Requested by: @Qubitium

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment, CI, and merge conflict monitoring