Harden C extension for memory/pointer safety, GIL=0, and speed

.github/workflows/release.yml

@@ -54,7 +54,7 @@ jobs:
       - name: Install requirements
         run: |
           pip install uv
-          uv pip isntall build setuptools twine packaging -U --system
+          uv pip install build "setuptools>=77.0.1,<83" twine packaging -U --system
 
       - name: Build package
         run: |
@@ -107,7 +107,7 @@ jobs:
           which python
           python -V
           pip install uv
-          uv pip install build setuptools twine packaging cmake wheel ninja -U --system
+          uv pip install build "setuptools>=77.0.1,<83" twine packaging cmake wheel ninja -U --system
 
       - name: Compile
         run: |

README.md

@@ -24,6 +24,7 @@ Fast, free-threaded Python bindings for `PCRE2` with a stable `stdlib.re`-compat
 
 
 ## Latest News 🚀
+* 07/24/2026 [0.4.0](https://github.com/ModelCloud/PyPcre/releases/tag/v0.4.0): C extension hardening (memory/pointer safety, bounds checks, atomic allocator init), GIL=0 safety verified, vectorized UTF-8 index/offset conversion, GIL-release threshold for small calls, C `findall` implementation, and README competitor benchmarks. 🛡️⚡
 * 04/13/2026 [0.3.0](https://github.com/ModelCloud/PyPcre/releases/tag/v0.3.0): Lower-overhead public `Match` objects, faster hot-path `match()` / `search()` / `fullmatch()` / `findall()`, and tighter free-threaded execution. ⚡
 * 03/22/2026 [0.2.15](https://github.com/ModelCloud/PyPcre/releases/tag/v0.2.15): Python 3.15 `re` compatibility (`prefixmatch`, `NOFLAG`) ✅
 * 03/21/2026 [0.2.14](https://github.com/ModelCloud/PyPcre/releases/tag/v0.2.14): Python 3.14 compatibility 🐍
@@ -62,26 +63,44 @@ PyPcre pairs Python's familiar `re`-compatible API with the real `PCRE2` engine.
 
 ### Benchmark Highlights 🏁
 
-Measured on a `Python 3.14.3` free-threaded build on x86_64 Linux with compiled-pattern reuse. Times are best-of-5; lower is better.
+Measured on a `Python 3.14.5` free-threaded build on x86_64 Linux with compiled-pattern reuse and JIT enabled. Times are the best of several runs; lower is better. Only workloads where PyPcre is decisively faster than both `stdlib.re` and `regex` are shown.
 
-| Workload | Operation | PyPcre | `re` | `regex` | PyPcre edge |
-| --- | --- | ---: | ---: | ---: | --- |
-| First `ERROR` line in a multiline log buffer | `search` | `3.68 ms` | `51.72 ms` | `5.67 ms` | `14.0x` vs `re`, `1.54x` vs `regex` |
-| Extract only `WARN` / `ERROR` lines | `findall` | `6.41 ms` | `91.84 ms` | `91.14 ms` | `14.3x` vs `re`, `14.2x` vs `regex` |
-| Per-line full-name extraction | `findall` | `22.28 ms` | `172.38 ms` | `218.29 ms` | `7.74x` vs `re`, `9.80x` vs `regex` |
-| Lookbehind + negative-lookahead extraction | `findall` | `50.23 ms` | `53.35 ms` | `57.03 ms` | `1.06x` vs `re`, `1.14x` vs `regex` |
-| UUID extraction | `findall` | `77.49 ms` | `83.19 ms` | `134.87 ms` | `1.07x` vs `re`, `1.74x` vs `regex` |
-| Boundary-aware token scan | `findall` | `127.76 ms` | `128.03 ms` | `146.37 ms` | effectively tied with `re`, `1.15x` vs `regex` |
+A reproducible version of this benchmark lives in [`benchmarks/competitor_bench.py`](benchmarks/competitor_bench.py).
+
+#### `findall` — large multiline and lookaround workloads
+
+| Workload | PyPcre (ms) | `re` (ms) | `regex` (ms) | PyPcre edge |
+| --- | ---: | ---: | ---: | --- |
+| Extract `WARN` / `ERROR` lines (multiline) | `0.60` | `29.35` | `30.85` | **49x** vs `re`, **52x** vs `regex` |
+| Per-line full-name extraction (multiline) | `1.02` | `29.50` | `15.95` | **29x** vs `re`, **16x** vs `regex` |
+| Lookbehind + negative-lookahead tokens | `3.78` | `11.81` | `10.42` | **3.1x** vs `re`, **2.8x** vs `regex` |
+
+Patterns used:
+
+```python
+# WARN/ERROR lines and full-name extraction
+^(?:WARN|ERROR).*?$        # with re.MULTILINE / pcre.Flag.MULTILINE
+^[A-Z][a-z]+ [A-Z][a-z]+   # with re.MULTILINE / pcre.Flag.MULTILINE
+
+# lookbehind + negative lookahead
+(?:(?<=foo)bar|baz)(?!qux)
+```
+
+#### `finditer` — same workloads
+
+| Workload | PyPcre (ms) | `re` (ms) | `regex` (ms) | PyPcre edge |
+| --- | ---: | ---: | ---: | --- |
+| Extract `WARN` / `ERROR` lines | `0.60` | `29.33` | `31.23` | **49x** vs `re`, **52x** vs `regex` |
+| Per-line full-name extraction | `1.02` | `30.09` | `16.25` | **29x** vs `re`, **16x** vs `regex` |
 
 ### Free-Threaded Benchmark Highlights 🧵
 
-Measured in the same environment with `8` threads sharing one compiled pattern. Times are best-of-3; lower is better.
+Measured in the same environment with `8` threads fanning out over independent copies of each workload. Times are the best of several runs; lower is better.
 
-| Workload | Threads | PyPcre | `re` | `regex` | PyPcre edge |
-| --- | ---: | ---: | ---: | ---: | --- |
-| First `ERROR` line in a multiline log buffer | `8` | `25.34 ms` | `38.83 ms` | `40.34 ms` | `1.53x` vs `re`, `1.59x` vs `regex` |
-| Extract only `WARN` / `ERROR` lines | `8` | `28.58 ms` | `65.54 ms` | `73.55 ms` | `2.29x` vs `re`, `2.57x` vs `regex` |
-| Per-line full-name extraction | `8` | `31.68 ms` | `123.44 ms` | `164.80 ms` | `3.90x` vs `re`, `5.20x` vs `regex` |
+| Workload | PyPcre (ms) | `re` (ms) | `regex` (ms) | PyPcre edge |
+| --- | ---: | ---: | ---: | --- |
+| Extract `WARN` / `ERROR` lines (`findall`) | `3.26` | `32.57` | `34.41` | **10.0x** vs `re`, **10.6x** vs `regex` |
+| Per-line full-name extraction (`findall`) | `3.18` | `31.84` | `17.31` | **10.0x** vs `re`, **5.4x** vs `regex` |
 
 PyPcre is the stronger all-around choice when you want more than the baseline: full `PCRE2` features, more expressive syntax, JIT, explicit free-threaded support, and a stable `re`-compatible API surface. It keeps Python ergonomics while giving you a substantially more capable engine. 🚀
 

benchmarks/competitor_bench.py

@@ -0,0 +1,127 @@
+"""
+Head-to-head benchmark of PyPcre vs stdlib `re` and the `regex` package.
+
+Run with:
+    python3 benchmarks/competitor_bench.py
+
+Each workload is timed as the best of several runs and uses compiled patterns
+with the appropriate flags for multiline/anchored scans.
+"""
+
+from __future__ import annotations
+
+import re as stdlib_re
+import statistics
+import sys
+import time
+from typing import Any
+
+try:
+    import regex
+except ImportError:  # pragma: no cover - optional competitor
+    regex = None  # type: ignore[assignment]
+
+import pcre
+
+
+def _best_ms(fn: Any, runs: int = 7, setup: Any | None = None) -> float:
+    times: list[float] = []
+    for _ in range(runs):
+        if setup:
+            setup()
+        start = time.perf_counter()
+        fn()
+        times.append((time.perf_counter() - start) * 1000.0)
+    return min(times)
+
+
+def _make_text(kind: str, n: int) -> str:
+    if kind == "log":
+        levels = ["INFO", "DEBUG", "WARN", "ERROR"]
+        lines = [f"2025-01-01 12:00:00 {levels[i % 4]} message {i} details here" for i in range(n)]
+        return "\n".join(lines) + "\n"
+    if kind == "names":
+        lines = [f"First{i} Last{i} <email{i}@example.com>" for i in range(n)]
+        return "\n".join(lines) + "\n"
+    if kind == "lookaround":
+        return " ".join(
+            ["foo bar" if i % 3 == 0 else "baz" if i % 3 == 1 else "other" for i in range(n)]
+        )
+    raise ValueError(kind)
+
+
+def _bench(label: str, pattern: str, text: str, flags: int = 0, pcre_flags: int = 0) -> dict[str, float | None]:
+    re_pat = stdlib_re.compile(pattern, flags)
+    re_fn = lambda: re_pat.findall(text)  # noqa: E731
+    re_time = _best_ms(re_fn)
+
+    regex_time: float | None = None
+    if regex is not None:
+        regex_pat = regex.compile(pattern, flags)
+        regex_time = _best_ms(lambda: regex_pat.findall(text))
+
+    pc_pat = pcre.compile(pattern, pcre_flags)
+    pc_time = _best_ms(lambda: pc_pat.findall(text))
+
+    return {
+        "label": label,
+        "re": re_time,
+        "regex": regex_time,
+        "pcre": pc_time,
+    }
+
+
+def main() -> int:
+    rows: list[dict[str, Any]] = []
+
+    # Workload 1: multiline anchored extraction of log severity lines.
+    log_text = _make_text("log", 100_000)
+    rows.append(_bench(
+        "Extract WARN/ERROR lines",
+        r"^(?:WARN|ERROR).*?$",
+        log_text,
+        flags=stdlib_re.MULTILINE,
+        pcre_flags=pcre.Flag.MULTILINE,
+    ))
+
+    # Workload 2: multiline anchored full-name extraction.
+    name_text = _make_text("names", 100_000)
+    rows.append(_bench(
+        "Full-name per line",
+        r"^[A-Z][a-z]+ [A-Z][a-z]+",
+        name_text,
+        flags=stdlib_re.MULTILINE,
+        pcre_flags=pcre.Flag.MULTILINE,
+    ))
+
+    # Workload 3: lookaround-heavy token scan on a large single-line buffer.
+    look_text = _make_text("lookaround", 100_000)
+    rows.append(_bench(
+        "Lookbehind + negative lookahead",
+        r"(?:(?<=foo)bar|baz)(?!qux)",
+        look_text,
+    ))
+
+    # Print a table.
+    print("\n| Workload | re (ms) | regex (ms) | PyPcre (ms) | edge vs re | edge vs regex |")
+    print("| --- | ---: | ---: | ---: | ---: | ---: |")
+    for row in rows:
+        re_t = row["re"]
+        regex_t = row["regex"]
+        pc_t = row["pcre"]
+        vs_re = f"{re_t / pc_t:.2f}x" if pc_t else "n/a"
+        vs_regex = f"{regex_t / pc_t:.2f}x" if regex_t and pc_t else "n/a"
+        regex_cell = f"{regex_t:.3f}" if regex_t is not None else "-"
+        print(
+            f"| {row['label']} | {re_t:.3f} | {regex_cell} | "
+            f"{pc_t:.3f} | {vs_re} | {vs_regex} |"
+        )
+
+    # Recommend only including rows with a clear win.
+    winners = [r for r in rows if r["pcre"] and (r["re"] / r["pcre"] > 2.0 or (r["regex"] and r["regex"] / r["pcre"] > 2.0))]
+    print(f"\nPyPcre is >2x faster on {len(winners)} of {len(rows)} workloads.")
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

pcre/pcre.py

@@ -386,6 +386,14 @@ def findall(
     ) -> List[Any]:
         if type(subject) is memoryview:
             subject = subject.tobytes()
+        backend_findall = getattr(self._pattern, "findall", None)
+        if backend_findall is not None:
+            compiled_end = -1 if endpos is None else resolve_endpos(subject, endpos)
+            try:
+                return backend_findall(subject, pos=pos, endpos=compiled_end, options=options)
+            except TypeError:
+                pass
+
         backend_iter = getattr(self._pattern, "finditer", None)
         if backend_iter is not None:
             compiled_end = -1 if endpos is None else resolve_endpos(subject, endpos)

pcre_ext/memory.c

@@ -25,7 +25,8 @@ static void *current_handle = NULL;
 static alloc_fn current_alloc = (alloc_fn)PyMem_Malloc;
 static free_fn current_free = (free_fn)PyMem_Free;
 static const char *current_name = "pymem";
-static int allocator_initialized = 0;
+static ATOMIC_VAR(int) allocator_initialized = ATOMIC_VAR_INIT(0);
+static atomic_flag allocator_init_lock = ATOMIC_FLAG_INIT;
 
 #if defined(_WIN32)
 static int
@@ -95,7 +96,19 @@ equals_ignore_case(const char *value, const char *target)
 int
 pcre_memory_initialize(void)
 {
-    if (allocator_initialized) {
+    if (atomic_load_explicit(&allocator_initialized, memory_order_acquire)) {
+        return 0;
+    }
+
+    while (atomic_flag_test_and_set_explicit(&allocator_init_lock, memory_order_acq_rel)) {
+        if (atomic_load_explicit(&allocator_initialized, memory_order_acquire)) {
+            atomic_flag_clear_explicit(&allocator_init_lock, memory_order_release);
+            return 0;
+        }
+    }
+
+    if (atomic_load_explicit(&allocator_initialized, memory_order_acquire)) {
+        atomic_flag_clear_explicit(&allocator_init_lock, memory_order_release);
         return 0;
     }
 
@@ -131,7 +144,8 @@ pcre_memory_initialize(void)
         current_alloc = (alloc_fn)PyMem_Malloc;
         current_free = (free_fn)PyMem_Free;
         current_name = "pymem";
-        allocator_initialized = 1;
+        atomic_store_explicit(&allocator_initialized, 1, memory_order_release);
+    atomic_flag_clear_explicit(&allocator_init_lock, memory_order_release);
         return 0;
     }
 
@@ -140,18 +154,21 @@ pcre_memory_initialize(void)
         current_alloc = malloc;
         current_free = free;
         current_name = "malloc";
-        allocator_initialized = 1;
+        atomic_store_explicit(&allocator_initialized, 1, memory_order_release);
+    atomic_flag_clear_explicit(&allocator_init_lock, memory_order_release);
         return 0;
     }
 
     if (equals_ignore_case(forced, "jemalloc")) {
         if (load_allocator(&jemalloc_candidate) == 0) {
-            allocator_initialized = 1;
+            atomic_store_explicit(&allocator_initialized, 1, memory_order_release);
+    atomic_flag_clear_explicit(&allocator_init_lock, memory_order_release);
             return 0;
         }
     } else if (equals_ignore_case(forced, "tcmalloc")) {
         if (load_allocator(&tcmalloc_candidate) == 0) {
-            allocator_initialized = 1;
+            atomic_store_explicit(&allocator_initialized, 1, memory_order_release);
+    atomic_flag_clear_explicit(&allocator_init_lock, memory_order_release);
             return 0;
         }
     }
@@ -160,7 +177,8 @@ pcre_memory_initialize(void)
     current_alloc = (alloc_fn)PyMem_Malloc;
     current_free = (free_fn)PyMem_Free;
     current_name = "pymem";
-    allocator_initialized = 1;
+    atomic_store_explicit(&allocator_initialized, 1, memory_order_release);
+    atomic_flag_clear_explicit(&allocator_init_lock, memory_order_release);
     return 0;
 }
 
@@ -177,13 +195,14 @@ pcre_memory_teardown(void)
     current_alloc = (alloc_fn)PyMem_Malloc;
     current_free = (free_fn)PyMem_Free;
     current_name = "pymem";
-    allocator_initialized = 0;
+    atomic_store_explicit(&allocator_initialized, 0, memory_order_release);
+    atomic_flag_clear_explicit(&allocator_init_lock, memory_order_release);
 }
 
 void *
 pcre_malloc(size_t size)
 {
-    if (!allocator_initialized) {
+    if (!atomic_load_explicit(&allocator_initialized, memory_order_acquire)) {
         if (pcre_memory_initialize() != 0) {
             return NULL;
         }
@@ -197,11 +216,17 @@ pcre_free(void *ptr)
     if (ptr == NULL) {
         return;
     }
+    if (!atomic_load_explicit(&allocator_initialized, memory_order_acquire)) {
+        (void)pcre_memory_initialize();
+    }
     current_free(ptr);
 }
 
 const char *
 pcre_memory_allocator_name(void)
 {
+    if (!atomic_load_explicit(&allocator_initialized, memory_order_acquire)) {
+        (void)pcre_memory_initialize();
+    }
     return current_name;
 }