diff --git a/README.md b/README.md index 44e1463..c3cec99 100644 --- a/README.md +++ b/README.md @@ -24,6 +24,8 @@ Fast, free-threaded Python bindings for `PCRE2` with a stable `stdlib.re`-compat ## Latest News 🚀 +* 08/10/2026 **Literal split/substitution/findall fast paths**: exact plain-literal `Pattern.split` calls now use the immutable built-in splitter after construction-time validation, measuring **2.1x** faster than the prior C dispatch on Python 3.10 and **1.7x** faster on free-threaded Python 3.14t/GIL=0; delimiter-heavy multi-character literals reach roughly **4.8x**. Literal `Pattern.subn` and module-level `sub`/`subn` now use native replace/count primitives, reaching about **15x** on short repeated tokens and **3x** on delimiter-heavy text. Literal `findall` uses non-overlapping native count/list construction, reaching about **9x** on short repeated tokens and **8x** on delimiter-heavy text. Regex metacharacters, explicit flags, subclasses, and buffer subjects remain on the compatibility-safe PCRE2 path. ⚡ +* 08/09/2026 **API hot-path update**: large `parallel_map(findall)` workloads now reach **11.5x** speedup on Python 3.10 and **11.25x** on free-threaded Python 3.14t/GIL=0 with 12 performance-tier workers. Ordered `parallel_map(search)` reaches **8.57x** and **7.85x**, respectively; one-item and up to eight tiny explicit `parallel_map` subjects now avoid executor setup (the one-item case measures **13.3x** faster on Python 3.10 and **27.7x** on 3.14t), default bound `Pattern.split` is another **1.6x/1.5x** faster on 3.10/3.14t, and default bound literal `Pattern.subn` is about **1.5x** faster on Python 3.10. Canonical module helpers retain their optimized dispatch while their wrapper/template caches are thread-scoped, size-bounded, and invalidated across live workers. Repeated backreference `Match.expand()` avoids reparsing within the active cache context, while captured values returned by `Match.groups()` remain call-local so a long-lived Match does not retain an additional copy of large captures. 🧵⚡ * 08/08/2026 **0.6.0**: `findall`, `finditer`, `sub`/`subn`, `split`, and `match`/`search`/`fullmatch` are now up to **46x faster** than `stdlib.re` and **48x faster** than `regex` on `finditer`/`findall` workloads, **13x** on `split`, and **2–9x** on `sub`/`subn` backref workloads, with full `re` semantics. Free-threaded `findall` reaches **13.8x** vs `re` on 8 threads. 🚀⚡ * 07/27/2026 [0.5.0](https://github.com/ModelCloud/PyPcre/releases/tag/v0.5.0): Zero-copy buffer-protocol subject support (`mmap.mmap`, `bytearray`, `array.array`) with UTF-8 validation and GIL=0-safe memory pinning. 🗂️⚡ * 07/24/2026 [0.4.0](https://github.com/ModelCloud/PyPcre/releases/tag/v0.4.0): C extension hardening (memory/pointer safety, bounds checks, atomic allocator init), GIL=0 safety verified, vectorized UTF-8 index/offset conversion, GIL-release threshold for small calls, C `findall` implementation, and README competitor benchmarks. 🛡️⚡ @@ -65,6 +67,39 @@ PyPcre pairs Python's familiar `re`-compatible API with the real `PCRE2` engine. ### Benchmark Highlights 🏁 +#### API hot paths and 12-core fan-out + +Pinned A/B measurements on an Apple M4 Max use the same `taskpolicy -t 1 -l 1` +scheduler policy for both interpreters. The host reports 12 performance logical +CPUs and 4 efficiency logical CPUs; macOS does not provide an unprivileged hard +per-process CPU mask, so the benchmark records the topology rather than claiming +hard CPU affinity. + +| Workload | Python 3.10 | Python 3.14t/GIL=0 | +| --- | ---: | ---: | +| `parallel_map(search)`, 16 × 1 MiB subjects, 12 workers | **8.57x** | **7.85x** | +| `parallel_map(findall)`, 48 × 1 MiB subjects, 12 workers | **11.51x** | **11.25x** | +| Bound one-character literal `Pattern.split` | **2.1x** | **1.7x** | +| Bound backreference `sub` hot path | **1.38 μs** | **1.14 μs** | +| Repeated call-local `Match.groups()` | **~0.05 μs** | **~0.05 μs** | +| Repeated `Match.expand(r"[\\1]")` | **5.23 μs** | **1.10 μs** | +| Repeated default `compile("(x)")` | **0.49 μs** | **0.38 μs** | +| Repeated integer-flagged `compile("x", CASELESS)` | **1.16 μs** | **0.81 μs** | + +The parallel figures are serial-to-parallel speedups and preserve input order and +exception behavior. Large `findall` scans release the GIL only around the PCRE2 +call; match data, context, and subject ownership remain worker-local. Reproduce +the fan-out benchmark with: + +```bash +taskpolicy -t 1 -l 1 env PYTHONPATH=. \ + PYPCRE_PARALLEL_WORKERS=12 PYPCRE_PARALLEL_RUNS=3 \ + python3 benchmarks/parallel_map_hotpath.py +``` + +The same script runs under Python 3.14t. The API microbenchmarks are available in +[`benchmarks/api_hotpaths.py`](benchmarks/api_hotpaths.py). + Measured on a `Python 3.14.6` free-threaded build on x86_64 Linux with compiled-pattern reuse and JIT enabled. Times are the best of several runs; lower is better. Only workloads where PyPcre is decisively faster than both `stdlib.re` and `regex` are shown. A reproducible version of this benchmark lives in [`benchmarks/competitor_bench.py`](benchmarks/competitor_bench.py). @@ -280,7 +315,13 @@ the conversion without repeating the flag. - `pcre.configure(jit=False)` disables JIT globally. `Flag.JIT` and `Flag.NO_JIT` let you override that per pattern. - `pcre.set_cache_limit()`, `pcre.get_cache_limit()`, and `pcre.clear_cache()` - control the high-level compile cache. + control every high-level compile/template helper cache in the active context. + A zero limit disables them, and `None` uses a 256-entry hard safety ceiling + rather than permitting unbounded growth. High-level cache entries never cross + thread scope in the default thread-local strategy; a clear invalidates live + workers' high-level helper caches on their next cache-backed call. Backend + scratch buffers remain thread-scoped and are released when that thread exits. + Oversized patterns and templates are not retained. - `pcre.configure_threads()`, `pcre.configure_thread_pool()`, `shutdown_thread_pool()`, `Flag.THREADS`, and `Flag.NO_THREADS` are available if you want to opt into or restrict threaded execution. diff --git a/benchmarks/api_hotpaths.py b/benchmarks/api_hotpaths.py new file mode 100644 index 0000000..9e87c12 --- /dev/null +++ b/benchmarks/api_hotpaths.py @@ -0,0 +1,86 @@ +"""Reproducible API hot-path benchmark for CPython 3.10 and 3.14t. + +Run from the repository root with either interpreter, for example:: + + PYTHONPATH=. python3 benchmarks/api_hotpaths.py + PYTHONPATH=. python3.14t benchmarks/api_hotpaths.py + +The benchmark intentionally uses short subjects so Python dispatch, template +parsing, and object-wrapper costs are visible instead of being hidden by a +large PCRE2 scan. Set ``PYPCRE_BENCH_RUNS`` to change the iteration count. +When running on a free-threaded build, an additional shared-pattern workload +checks the concurrent execution path. +""" + +from __future__ import annotations + +import concurrent.futures +import os +import sys +import time +from collections.abc import Callable + +import pcre + + +RUNS = int(os.getenv("PYPCRE_BENCH_RUNS", "50000")) + + +def _time(fn: Callable[[], object]) -> float: + started = time.perf_counter() + for _ in range(RUNS): + fn() + return (time.perf_counter() - started) * 1_000_000.0 / RUNS + + +def main() -> int: + subject = "x" * 1000 + short_subject = "x" * 10 + pattern = pcre.compile("(x)") + captured = pattern.match(short_subject) + if captured is None: + raise AssertionError("benchmark pattern failed to produce a match") + operations: list[tuple[str, Callable[[], object]]] = [ + ("bound.match", lambda: pattern.match(subject)), + ("bound.search", lambda: pattern.search(subject)), + ("bound.fullmatch", lambda: pattern.fullmatch(subject)), + ("bound.findall", lambda: pattern.findall(short_subject)), + ("bound.finditer", lambda: list(pattern.finditer(short_subject))), + ("bound.split", lambda: pattern.split("x " * 8)), + ("bound.sub.literal", lambda: pattern.sub("[X]", short_subject)), + ("bound.sub.backref", lambda: pattern.sub(r"[\1]", short_subject)), + ("match.groups", captured.groups), + ("module.match", lambda: pcre.match("(x)", subject)), + ("module.search", lambda: pcre.search("(x)", subject)), + ("module.fullmatch", lambda: pcre.fullmatch("(x)", subject)), + ("module.findall", lambda: pcre.findall("(x)", short_subject)), + ("module.finditer", lambda: list(pcre.finditer("(x)", short_subject))), + ("module.split", lambda: pcre.split("(x)", "x " * 8)), + ("module.sub.literal", lambda: pcre.sub("(x)", "[X]", short_subject)), + ] + + gil_enabled = getattr(sys, "_is_gil_enabled", lambda: True)() + print(f"runtime={sys.version.split()[0]} gil_enabled={gil_enabled} runs={RUNS}") + for name, operation in operations: + print(f"{name:22s} {_time(operation):8.3f} us") + + if not gil_enabled: + workers = 8 + per_worker = max(1, RUNS // 5) + + def shared_search(_: int) -> int: + for _ in range(per_worker): + pattern.search(subject) + return per_worker + + started = time.perf_counter() + with concurrent.futures.ThreadPoolExecutor(max_workers=workers) as pool: + completed = sum(pool.map(shared_search, range(workers))) + elapsed_ms = (time.perf_counter() - started) * 1000.0 + print(f"shared.search.{workers}T {elapsed_ms:8.3f} ms ({completed} calls)") + + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/benchmarks/parallel_map_hotpath.py b/benchmarks/parallel_map_hotpath.py new file mode 100644 index 0000000..f0b5197 --- /dev/null +++ b/benchmarks/parallel_map_hotpath.py @@ -0,0 +1,91 @@ +"""Measure the batched :func:`pcre.parallel_map` workload. + +Run from the repository root with either interpreter:: + + PYTHONPATH=. python3 benchmarks/parallel_map_hotpath.py + PYTHONPATH=. python3.14t benchmarks/parallel_map_hotpath.py + +On macOS, use the same scheduler policy for both A/B runs and use the twelve +performance-tier logical workers exposed by this host, for example:: + + taskpolicy -t 1 -l 1 env PYTHONPATH=. PYPCRE_PARALLEL_WORKERS=12 python3 benchmarks/parallel_map_hotpath.py + +macOS exposes the performance/efficiency cluster counts but not a portable +per-process CPU mask; the benchmark prints both counts so a run cannot be +mistaken for a hard CPU pin. + +The subjects are intentionally large enough for PCRE2 to amortize worker +startup and queueing. This reports the serial baseline, threaded execution, +and the speedup while preserving result order. +""" + +from __future__ import annotations + +import os +import subprocess +import sys +import time +from collections.abc import Callable + +import pcre + + +SUBJECT_COUNT = int(os.getenv("PYPCRE_PARALLEL_SUBJECTS", "16")) +SUBJECT_SIZE = int(os.getenv("PYPCRE_PARALLEL_SIZE", "1000000")) +RUNS = int(os.getenv("PYPCRE_PARALLEL_RUNS", "5")) +WORKERS = int(os.getenv("PYPCRE_PARALLEL_WORKERS", "12")) + + +def _topology() -> str: + if sys.platform != "darwin": + return "topology=non-darwin" + values: list[str] = [] + for name in ("hw.perflevel0.logicalcpu", "hw.perflevel1.logicalcpu"): + try: + value = subprocess.check_output( + ["sysctl", "-n", name], text=True, stderr=subprocess.DEVNULL + ).strip() + except (OSError, subprocess.CalledProcessError): + value = "unknown" + values.append(value) + return f"performance_logical={values[0]} efficiency_logical={values[1]}" + + +def _best_ms(fn: Callable[[], object]) -> float: + callable_fn = fn # Keep the timed call outside the loop's attribute lookup. + best = float("inf") + for _ in range(RUNS): + started = time.perf_counter() + callable_fn() + best = min(best, (time.perf_counter() - started) * 1000.0) + return best + + +def main() -> int: + pattern = pcre.compile(r"\d+", pcre.Flag.THREADS) + subjects = ["x" * SUBJECT_SIZE + "123"] * SUBJECT_COUNT + serial = lambda: [pattern.search(subject) for subject in subjects] + parallel = lambda: pattern.parallel_map( + subjects, method="search", max_workers=WORKERS + ) + + serial_result = serial() + parallel_result = parallel() + if [bool(item) for item in serial_result] != [ + bool(item) for item in parallel_result + ]: + raise AssertionError("parallel_map changed result order or match presence") + + serial_ms = _best_ms(serial) + parallel_ms = _best_ms(parallel) + print( + f"{_topology()} subjects={SUBJECT_COUNT} size={SUBJECT_SIZE} " + f"workers={WORKERS} runs={RUNS} " + f"serial={serial_ms:.3f}ms parallel={parallel_ms:.3f}ms " + f"speedup={serial_ms / parallel_ms:.2f}x" + ) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/pcre/cache.py b/pcre/cache.py index 98ae04d..0139e02 100644 --- a/pcre/cache.py +++ b/pcre/cache.py @@ -10,15 +10,64 @@ import os from enum import Enum from threading import RLock, local -from typing import Any, Callable, Dict, Tuple, TypeVar, cast +from typing import Any, Callable, TypeVar, cast +from weakref import WeakSet import pcre_ext_c as _pcre2 - T = TypeVar("T") _DEFAULT_THREAD_CACHE_LIMIT = 32 _DEFAULT_GLOBAL_CACHE_LIMIT = 128 +_HARD_CACHE_ENTRY_LIMIT = 256 +_MAX_CACHE_INPUT_UNITS = 64 * 1024 + +_CACHE_EPOCH = globals().get("_CACHE_EPOCH", 0) +_CACHE_EPOCH_LOCK = globals().get("_CACHE_EPOCH_LOCK", RLock()) +_CACHE_CONTROL_LOCK = globals().get("_CACHE_CONTROL_LOCK", RLock()) +_CACHE_CONTROL_CALLBACKS: WeakSet[Callable[[], None]] = globals().get( + "_CACHE_CONTROL_CALLBACKS", WeakSet() +) + + +def get_cache_epoch() -> int: + """Return the process-wide invalidation generation for auxiliary caches.""" + + return _CACHE_EPOCH + + +def _bump_cache_epoch() -> int: + global _CACHE_EPOCH + + with _CACHE_EPOCH_LOCK: + _CACHE_EPOCH += 1 + return _CACHE_EPOCH + + +def register_cache_control(callback: Callable[[], None]) -> None: + """Register a current-context cache clearer/trim callback.""" + + with _CACHE_CONTROL_LOCK: + _CACHE_CONTROL_CALLBACKS.add(callback) + + +def _notify_cache_control() -> None: + with _CACHE_CONTROL_LOCK: + callbacks = tuple(_CACHE_CONTROL_CALLBACKS) + for callback in callbacks: + callback() + + +def cache_input_allowed(value: Any) -> bool: + """Reject oversized immutable inputs from persistent caches.""" + + return not isinstance(value, (str, bytes)) or len(value) <= _MAX_CACHE_INPUT_UNITS + + +def _bounded_cache_limit(limit: int | None) -> int: + if limit is None: + return _HARD_CACHE_ENTRY_LIMIT + return min(limit, _HARD_CACHE_ENTRY_LIMIT) class _CacheStrategy(str, Enum): @@ -31,17 +80,18 @@ class _ThreadCacheState(local): def __init__(self) -> None: self.cache_limit: int | None = _DEFAULT_THREAD_CACHE_LIMIT - self.pattern_cache: Dict[Tuple[Any, int, bool], Any] = {} + self.pattern_cache: dict[tuple[Any, int, bool], Any] = {} + self.epoch = get_cache_epoch() class _GlobalCacheState: """Process-wide cache state mirroring the historic global cache.""" - __slots__ = ("cache_limit", "pattern_cache", "lock") + __slots__ = ("cache_limit", "lock", "pattern_cache") def __init__(self) -> None: self.cache_limit: int | None = _DEFAULT_GLOBAL_CACHE_LIMIT - self.pattern_cache: Dict[Tuple[Any, int, bool], Any] = {} + self.pattern_cache: dict[tuple[Any, int, bool], Any] = {} self.lock = RLock() @@ -56,6 +106,7 @@ def _normalize_strategy(value: str) -> _CacheStrategy: except ValueError as exc: # pragma: no cover - defensive raise ValueError("cache strategy must be 'thread-local' or 'global'") from exc + def _env_flag_is_true(value: str | None) -> bool: if value is None or value == "": return False @@ -78,8 +129,6 @@ def cache_strategy(strategy: str | None = None) -> str: :class:`RuntimeError`. """ - global _CACHE_STRATEGY - if strategy is None: return _CACHE_STRATEGY.value @@ -96,11 +145,21 @@ def cache_strategy(strategy: str | None = None) -> str: def _cached_compile_thread_local( pattern: Any, flags: int, - wrapper: Callable[["_pcre2.Pattern"], T], + wrapper: Callable[[_pcre2.Pattern], T], *, jit: bool, ) -> T: - cache_limit = _THREAD_LOCAL.cache_limit + current_epoch = get_cache_epoch() + if _THREAD_LOCAL.epoch != current_epoch: + _THREAD_LOCAL.pattern_cache.clear() + _THREAD_LOCAL.epoch = current_epoch + _pcre2.clear_pattern_cache() + _pcre2.clear_match_data_cache() + _pcre2.clear_jit_stack_cache() + + cache_limit = _bounded_cache_limit(_THREAD_LOCAL.cache_limit) + if not cache_input_allowed(pattern): + return wrapper(_pcre2.compile(pattern, flags=flags, jit=jit)) if cache_limit == 0: return wrapper(_pcre2.compile(pattern, flags=flags, jit=jit)) @@ -110,10 +169,12 @@ def _cached_compile_thread_local( cached = cache[key] except KeyError: compiled = wrapper(_pcre2.compile(pattern, flags=flags, jit=jit)) - if cache_limit != 0: - if cache_limit is not None and len(cache) >= cache_limit: - cache.pop(next(iter(cache))) - cache[key] = compiled + active_limit = _bounded_cache_limit(_THREAD_LOCAL.cache_limit) + if _THREAD_LOCAL.epoch != current_epoch or active_limit == 0: + return compiled + if len(cache) >= active_limit: + cache.pop(next(iter(cache))) + cache[key] = compiled return compiled except TypeError: return wrapper(_pcre2.compile(pattern, flags=flags, jit=jit)) @@ -124,11 +185,13 @@ def _cached_compile_thread_local( def _cached_compile_global( pattern: Any, flags: int, - wrapper: Callable[["_pcre2.Pattern"], T], + wrapper: Callable[[_pcre2.Pattern], T], *, jit: bool, ) -> T: - cache_limit = _GLOBAL_STATE.cache_limit + cache_limit = _bounded_cache_limit(_GLOBAL_STATE.cache_limit) + if not cache_input_allowed(pattern): + return wrapper(_pcre2.compile(pattern, flags=flags, jit=jit)) if cache_limit == 0: return wrapper(_pcre2.compile(pattern, flags=flags, jit=jit)) @@ -144,15 +207,17 @@ def _cached_compile_global( else: return cast(T, cached) + compile_epoch = get_cache_epoch() compiled = wrapper(_pcre2.compile(pattern, flags=flags, jit=jit)) with lock: - if _GLOBAL_STATE.cache_limit == 0: + if _GLOBAL_STATE.cache_limit == 0 or compile_epoch != get_cache_epoch(): return compiled try: existing = _GLOBAL_STATE.pattern_cache[key] except KeyError: - if _GLOBAL_STATE.cache_limit is not None and len(_GLOBAL_STATE.pattern_cache) >= _GLOBAL_STATE.cache_limit: + active_limit = _bounded_cache_limit(_GLOBAL_STATE.cache_limit) + if len(_GLOBAL_STATE.pattern_cache) >= active_limit: _GLOBAL_STATE.pattern_cache.pop(next(iter(_GLOBAL_STATE.pattern_cache))) _GLOBAL_STATE.pattern_cache[key] = compiled except TypeError: @@ -165,7 +230,7 @@ def _cached_compile_global( def cached_compile( pattern: Any, flags: int, - wrapper: Callable[["_pcre2.Pattern"], T], + wrapper: Callable[[_pcre2.Pattern], T], *, jit: bool, ) -> T: @@ -179,8 +244,11 @@ def cached_compile( def clear_cache() -> None: """Clear the cached compiled patterns and backend caches for the active strategy.""" + current_epoch = _bump_cache_epoch() + if _CACHE_STRATEGY is _CacheStrategy.THREAD_LOCAL: _THREAD_LOCAL.pattern_cache.clear() + _THREAD_LOCAL.epoch = current_epoch else: with _GLOBAL_STATE.lock: _GLOBAL_STATE.pattern_cache.clear() @@ -188,6 +256,7 @@ def clear_cache() -> None: _pcre2.clear_pattern_cache() _pcre2.clear_match_data_cache() _pcre2.clear_jit_stack_cache() + _notify_cache_control() def set_cache_limit(limit: int | None) -> None: @@ -206,29 +275,41 @@ def set_cache_limit(limit: int | None) -> None: if _CACHE_STRATEGY is _CacheStrategy.THREAD_LOCAL: _THREAD_LOCAL.cache_limit = new_limit cache = _THREAD_LOCAL.pattern_cache - if new_limit == 0: + effective_limit = _bounded_cache_limit(new_limit) + if effective_limit == 0: cache.clear() - elif new_limit is not None: - while len(cache) > new_limit: + _pcre2.clear_pattern_cache() + else: + while len(cache) > effective_limit: cache.pop(next(iter(cache))) else: + _bump_cache_epoch() with _GLOBAL_STATE.lock: _GLOBAL_STATE.cache_limit = new_limit cache = _GLOBAL_STATE.pattern_cache - if new_limit == 0: + effective_limit = _bounded_cache_limit(new_limit) + if effective_limit == 0: cache.clear() - elif new_limit is not None: - while len(cache) > new_limit: + else: + while len(cache) > effective_limit: cache.pop(next(iter(cache))) + _notify_cache_control() + def get_cache_limit() -> int | None: - """Return the current cache limit (``None`` means unlimited).""" + """Return the requested limit (``None`` uses the hard safety ceiling).""" if _CACHE_STRATEGY is _CacheStrategy.THREAD_LOCAL: return _THREAD_LOCAL.cache_limit return _GLOBAL_STATE.cache_limit +def get_effective_cache_limit() -> int: + """Return the bounded entry limit applied to every high-level cache.""" + + return _bounded_cache_limit(get_cache_limit()) + + # The backend has already been configured during module import; rely on its # reported strategy to keep this helper in sync. diff --git a/pcre/pcre.py b/pcre/pcre.py index c627b2c..2520645 100644 --- a/pcre/pcre.py +++ b/pcre/pcre.py @@ -9,16 +9,23 @@ import re as _std_re from collections.abc import Iterable, Iterator, Mapping +from functools import lru_cache +from threading import local from typing import Any, List import pcre_ext_c as _pcre2 from ._stdlib_re import RE_TEMPLATE, RE_TEMPLATE_FLAG, RE_UNICODE_FLAG, _parser -from .cache import cached_compile +from .cache import ( + cache_input_allowed, + cached_compile, + get_cache_epoch, + get_effective_cache_limit, + register_cache_control, +) from .cache import clear_cache as _clear_cache from .flags import Flag, strip_py_only_flags - # Cache frequently used flag values as plain integers to avoid the overhead of # IntFlag arithmetic in hot paths such as module-level search helpers. COMPAT_UNICODE_ESCAPE: int = int(Flag.COMPAT_UNICODE_ESCAPE) @@ -33,6 +40,7 @@ ) from .re_compat import ( TemplatePatternStub, + _cached_expand_template, coerce_group_value, coerce_subject_slice, compute_next_pos, @@ -49,11 +57,12 @@ ensure_thread_pool, get_auto_threshold, get_thread_default, + get_thread_pool_size, threading_supported, ) - _CPattern = _pcre2.Pattern +_ORIGINAL_CACHED_COMPILE = cached_compile PcreError = _pcre2.PcreError Match = getattr(_pcre2, "Match", _CompatMatch) _ATTACH_MATCH = getattr(_pcre2, "_attach_match", None) @@ -63,6 +72,8 @@ _DEFAULT_JIT = True _DEFAULT_COMPAT_REGEX = False +_DEFAULT_COMPILE_LOCAL = local() +_LOCAL_CACHE_NAMES = ("cache", "flagged_cache") _THREAD_MODE_DISABLED = "disabled" @@ -70,6 +81,62 @@ _THREAD_MODE_AUTO = "auto" +def _synchronize_local_caches() -> None: + epoch = get_cache_epoch() + if getattr(_DEFAULT_COMPILE_LOCAL, "epoch", -1) == epoch: + return + for name in ("module_lru", "replacement_lru"): + cached = getattr(_DEFAULT_COMPILE_LOCAL, name, None) + if cached is not None: + cached.cache_clear() + setattr(_DEFAULT_COMPILE_LOCAL, name, None) + _DEFAULT_COMPILE_LOCAL.module_hot_key = None + _DEFAULT_COMPILE_LOCAL.module_hot_value = None + _DEFAULT_COMPILE_LOCAL.replacement_hot_key = None + _DEFAULT_COMPILE_LOCAL.replacement_hot_value = None + for name in _LOCAL_CACHE_NAMES: + setattr(_DEFAULT_COMPILE_LOCAL, name, {}) + _DEFAULT_COMPILE_LOCAL.effective_limit = get_effective_cache_limit() + _DEFAULT_COMPILE_LOCAL.epoch = epoch + + +def _local_cache(name: str) -> dict[Any, Any]: + _synchronize_local_caches() + cache = getattr(_DEFAULT_COMPILE_LOCAL, name, None) + if cache is None: + cache = {} + setattr(_DEFAULT_COMPILE_LOCAL, name, cache) + return cache + + +def _trim_local_cache(cache: dict[Any, Any]) -> None: + limit = get_effective_cache_limit() + if limit == 0: + cache.clear() + return + while len(cache) > limit: + cache.pop(next(iter(cache))) + + +def _cache_configuration_changed() -> None: + _synchronize_local_caches() + for name in _LOCAL_CACHE_NAMES: + _trim_local_cache(_local_cache(name)) + for name in ("module_lru", "replacement_lru"): + cached = getattr(_DEFAULT_COMPILE_LOCAL, name, None) + if cached is not None: + cached.cache_clear() + setattr(_DEFAULT_COMPILE_LOCAL, name, None) + _DEFAULT_COMPILE_LOCAL.module_hot_key = None + _DEFAULT_COMPILE_LOCAL.module_hot_value = None + _DEFAULT_COMPILE_LOCAL.replacement_hot_key = None + _DEFAULT_COMPILE_LOCAL.replacement_hot_value = None + _DEFAULT_COMPILE_LOCAL.effective_limit = get_effective_cache_limit() + + +register_cache_control(_cache_configuration_changed) + + def _can_attach_match(raw: Any) -> bool: return ( _ATTACH_MATCH is not None @@ -138,7 +205,9 @@ def _apply_default_unicode_flags(pattern: Any, flags: int) -> int: def _coerce_stdlib_regexflag(flag: _std_re.RegexFlag) -> int: - unsupported_bits = int(flag) & ~(_STD_RE_FLAG_MASK | RE_TEMPLATE_FLAG | RE_UNICODE_FLAG) + unsupported_bits = int(flag) & ~( + _STD_RE_FLAG_MASK | RE_TEMPLATE_FLAG | RE_UNICODE_FLAG + ) if unsupported_bits: unsupported = _std_re.RegexFlag(unsupported_bits) raise ValueError( @@ -224,7 +293,13 @@ def _pcre2_replacement_from_parsed(parsed: Any, is_bytes: bool) -> Any: class Pattern: """High-level wrapper around the C-backed :class:`pcre_ext_c.Pattern`.""" - __slots__ = ("_pattern", "_groups_hint", "_thread_mode", "_is_c_pattern") + __slots__ = ( + "_pattern", + "_groups_hint", + "_thread_mode", + "_is_c_pattern", + "_literal_split", + ) def __init__(self, pattern: _CPattern) -> None: self._pattern = pattern @@ -235,6 +310,27 @@ def __init__(self, pattern: _CPattern) -> None: except AttributeError: # pragma: no cover - older extension fallback self._groups_hint = maybe_infer_group_count(pattern.pattern) + literal_split: str | bytes | None = None + if self._is_c_pattern: + source = pattern.pattern + if type(source) in (str, bytes) and source: + metacharacters = ( + ".^$*+?{}[]\\|()" if type(source) is str else b".^$*+?{}[]\\|()" + ) + expected_flags = ( + _pcre2.PCRE2_UTF + | _pcre2.PCRE2_UCP + | getattr(_pcre2, "PCRE2_NEVER_BACKSLASH_C", 0x00100000) + if type(source) is str + else 0 + ) + if ( + not any(char in metacharacters for char in source) + and pattern.flags == expected_flags + ): + literal_split = source + self._literal_split = literal_split + def __repr__(self) -> str: # pragma: no cover - delegated to C repr return repr(self._pattern) @@ -315,7 +411,9 @@ def match( raw = self._pattern.match(subject, pos=pos, options=options) else: resolved_end = resolve_endpos(subject, endpos) - raw = self._pattern.match(subject, pos=pos, endpos=resolved_end, options=options) + raw = self._pattern.match( + subject, pos=pos, endpos=resolved_end, options=options + ) if raw is None: return None return self._wrap_match(raw, subject, pos, resolved_end) @@ -340,7 +438,9 @@ def search( raw = self._pattern.search(subject, pos=pos, options=options) else: resolved_end = resolve_endpos(subject, endpos) - raw = self._pattern.search(subject, pos=pos, endpos=resolved_end, options=options) + raw = self._pattern.search( + subject, pos=pos, endpos=resolved_end, options=options + ) if raw is None: return None return self._wrap_match(raw, subject, pos, resolved_end) @@ -363,7 +463,9 @@ def fullmatch( raw = self._pattern.fullmatch(subject, pos=pos, options=options) else: resolved_end = resolve_endpos(subject, endpos) - raw = self._pattern.fullmatch(subject, pos=pos, endpos=resolved_end, options=options) + raw = self._pattern.fullmatch( + subject, pos=pos, endpos=resolved_end, options=options + ) if raw is None: return None return self._wrap_match(raw, subject, pos, resolved_end) @@ -385,14 +487,21 @@ def finditer( if self._is_c_pattern: # The C iterator stamps each Match with this public Pattern, so # we can return it directly without per-match Python wrapping. + fast_finditer = getattr(self._pattern, "_finditer_fast", None) + if fast_finditer is not None: + return fast_finditer(subject, pos, compiled_end, options, self) return backend_iter(subject, pos, compiled_end, options, self) raw_iter = None try: - raw_iter = backend_iter(subject, pos=pos, endpos=compiled_end, options=options, owner=self) + raw_iter = backend_iter( + subject, pos=pos, endpos=compiled_end, options=options, owner=self + ) except TypeError: # Older extensions and test doubles may not accept `owner`. try: - raw_iter = backend_iter(subject, pos=pos, endpos=compiled_end, options=options) + raw_iter = backend_iter( + subject, pos=pos, endpos=compiled_end, options=options + ) except TypeError: raw_iter = None if raw_iter is not None: @@ -404,14 +513,18 @@ def finditer( except StopIteration: return iter([]) if _can_attach_match(peek) and peek.re is self: + def _owned_iter(): yield peek yield from raw_iter + return _owned_iter() + def _wrapped_iter(): yield self._wrap_match(peek, subject, pos, resolved_end) for raw in raw_iter: yield self._wrap_match(raw, subject, pos, resolved_end) + return _wrapped_iter() search_end = resolved_end if endpos is not None else -1 @@ -422,7 +535,9 @@ def _wrapped_iter(): def _generator(): nonlocal current while True: - raw = self._pattern.search(subject, pos=current, endpos=search_end, options=options) + raw = self._pattern.search( + subject, pos=current, endpos=search_end, options=options + ) if raw is None: break @@ -451,11 +566,39 @@ def findall( ) -> List[Any]: if type(subject) is memoryview: subject = subject.tobytes() + literal_source = getattr(self, "_literal_split", None) + if ( + getattr(self, "_is_c_pattern", False) + and type(self) is Pattern + and literal_source is not None + and type(subject) is type(literal_source) + and type(pos) is int + and pos == 0 + and endpos is None + and type(options) is int + and options == 0 + ): + if subject.find(literal_source[:1]) < 0: + return [] + return [literal_source] * subject.count(literal_source) backend_findall = getattr(self._pattern, "findall", None) if backend_findall is not None: compiled_end = -1 if endpos is None else resolve_endpos(subject, endpos) + if ( + self._is_c_pattern + and type(pos) is int + and pos == 0 + and endpos is None + and type(options) is int + and options == 0 + ): + fast = getattr(self._pattern, "_findall_fast", None) + if fast is not None: + return fast(subject) try: - return backend_findall(subject, pos=pos, endpos=compiled_end, options=options) + return backend_findall( + subject, pos=pos, endpos=compiled_end, options=options + ) except TypeError: pass @@ -463,7 +606,9 @@ def findall( if backend_iter is not None: compiled_end = -1 if endpos is None else resolve_endpos(subject, endpos) try: - raw_iter = backend_iter(subject, pos=pos, endpos=compiled_end, options=options) + raw_iter = backend_iter( + subject, pos=pos, endpos=compiled_end, options=options + ) except TypeError: raw_iter = None if raw_iter is not None: @@ -477,7 +622,9 @@ def findall( return results results: List[Any] = [] - for match_obj in self.finditer(subject, pos=pos, endpos=endpos, options=options): + for match_obj in self.finditer( + subject, pos=pos, endpos=endpos, options=options + ): groups = match_obj.groups() if groups: results.append(groups[0] if len(groups) == 1 else groups) @@ -486,6 +633,36 @@ def findall( return results def split(self, subject: Any, maxsplit: Any = 0) -> List[Any]: + # A plain literal has exactly the same split semantics as the built-in + # immutable string/bytes splitter. The immutable construction check + # restricts this to canonical patterns with default options. + if ( + self._is_c_pattern + and type(self) is Pattern + and type(subject) in (str, bytes) + and type(maxsplit) is int + and type(subject) is type(self._literal_split) + ): + # Python's explicit ``maxsplit=0`` means unlimited splitting for + # ``Pattern.split`` (unlike ``str.split(sep, 0)``). + split_limit = -1 if maxsplit == 0 else (0 if maxsplit < 0 else maxsplit) + return subject.split(self._literal_split, split_limit) + + # The common immutable/default shape can go straight to the C splitter. + # Keep subclasses, buffer exporters, and non-default limits on the + # compatibility path so their coercion and override semantics remain + # unchanged. + if ( + self._is_c_pattern + and type(self) is Pattern + and type(subject) in (str, bytes) + and type(maxsplit) is int + and maxsplit == 0 + ): + fast_split = getattr(self._pattern, "_split_fast", None) + if fast_split is not None: + return fast_split(subject, 0) + subject = prepare_subject(subject) limit = normalise_count(maxsplit) if limit == 0: @@ -499,12 +676,19 @@ def split(self, subject: Any, maxsplit: Any = 0) -> List[Any]: # Empty patterns split at every code point/byte; avoid per-match overhead. if limit is None and self.pattern in ("", b""): if is_bytes_like(subject): - return [b""] + [bytes(subject[i : i + 1]) for i in range(len(subject))] + [b""] + return ( + [b""] + + [bytes(subject[i : i + 1]) for i in range(len(subject))] + + [b""] + ) return [""] + list(subject) + [""] backend_split = getattr(self._pattern, "split", None) if backend_split is not None: try: + fast_split = getattr(self._pattern, "_split_fast", None) + if fast_split is not None: + return fast_split(subject, 0 if limit is None else limit) return backend_split(subject, 0 if limit is None else limit) except TypeError: pass @@ -520,17 +704,29 @@ def split(self, subject: Any, maxsplit: Any = 0) -> List[Any]: break start, end = match_obj.span() - parts.append(coerce_subject_slice(subject, last_end, start, is_bytes=subject_is_bytes)) + parts.append( + coerce_subject_slice( + subject, last_end, start, is_bytes=subject_is_bytes + ) + ) groups = match_obj.groups() if groups: for value in groups: - parts.append(coerce_group_value(value, is_bytes=subject_is_bytes, empty=empty)) + parts.append( + coerce_group_value( + value, is_bytes=subject_is_bytes, empty=empty + ) + ) last_end = end splits_done += 1 - parts.append(coerce_subject_slice(subject, last_end, len(subject), is_bytes=subject_is_bytes)) + parts.append( + coerce_subject_slice( + subject, last_end, len(subject), is_bytes=subject_is_bytes + ) + ) return parts def sub(self, repl: Any, subject: Any, count: Any = 0) -> Any: @@ -538,6 +734,51 @@ def sub(self, repl: Any, subject: Any, count: Any = 0) -> Any: return result def subn(self, repl: Any, subject: Any, count: Any = 0) -> tuple[Any, int]: + # Plain literal patterns with literal replacements can use the built-in + # immutable replace/count primitives. This is safe only for canonical + # exact text/bytes values and preserves Python's count mapping (zero + # means unlimited replacement while negative counts perform none). + if ( + self._is_c_pattern + and type(self) is Pattern + and self._literal_split is not None + and type(subject) is type(self._literal_split) + and type(repl) is type(subject) + and type(count) is int + and ("\\" not in repl if type(repl) is str else b"\\" not in repl) + and ("$" not in repl if type(repl) is str else b"$" not in repl) + ): + if subject.find(self._literal_split[:1]) < 0: + return subject, 0 + if count < 0: + return subject, 0 + matched = subject.count(self._literal_split) + if matched == 0: + return subject, 0 + if count > 0 and matched > count: + matched = count + return ( + subject.replace(self._literal_split, repl, -1 if count <= 0 else count), + matched, + ) + + # Exact immutable literal replacements can bypass normalization and + # template setup. Keep escaped templates, callables, subclasses, + # buffers, and bounded counts on the compatibility path. + if ( + self._is_c_pattern + and type(self) is Pattern + and type(subject) in (str, bytes) + and type(repl) is type(subject) + and type(count) is int + and count == 0 + and ("\\" not in repl if type(repl) is str else b"\\" not in repl) + and ("$" not in repl if type(repl) is str else b"$" not in repl) + ): + fast_substitute = getattr(self._pattern, "_substitute_fast", None) + if fast_substitute is not None: + return fast_substitute(subject, repl) + subject = prepare_subject(subject) subject_is_bytes = is_bytes_like(subject) empty = b"" if subject_is_bytes else "" @@ -546,42 +787,95 @@ def subn(self, repl: Any, subject: Any, count: Any = 0) -> tuple[Any, int]: callable_repl = callable(repl) template = None parsed_template: List[Any] | None = None + has_extended_syntax = False if not callable_repl: if subject_is_bytes: if not is_bytes_like(repl): - raise TypeError("replacement must be bytes-like when substituting on bytes") + raise TypeError( + "replacement must be bytes-like when substituting on bytes" + ) template = bytes(repl) else: if not isinstance(repl, str): raise TypeError("replacement must be str when substituting on text") template = repl + has_extended_syntax = ( + b"\\" in template or b"$" in template + if subject_is_bytes + else str.__contains__(template, "\\") or str.__contains__(template, "$") + ) + if limit is None: backend_substitute = getattr(self._pattern, "substitute", None) if backend_substitute is not None: + # PCRE2's extended replacement syntax only treats ``\\`` + # and ``$`` specially. A replacement without either is + # already a literal Python template, so skip the costly + # ``sre_parse.parse_template`` round-trip. This path is + # immutable and per-call, so it remains safe when the same + # Pattern is used concurrently by GIL-free threads. + if not has_extended_syntax: + try: + fast_substitute = getattr( + self._pattern, "_substitute_fast", None + ) + if fast_substitute is not None: + return fast_substitute(subject, template) + return backend_substitute( + subject, replacement=template, count=0 + ) + except TypeError: + pass try: - parsed_template = _parser.parse_template( - template, - TemplatePatternStub(self.groups, self.groupindex), - ) + if type(self) is Pattern and type(template) in (str, bytes): + parsed_template, pcre2_repl = _cached_replacement_parts( + self, template, subject_is_bytes + ) + else: + parsed_template = _parser.parse_template( + template, + TemplatePatternStub(self.groups, self.groupindex), + ) + pcre2_repl = _pcre2_replacement_from_parsed( + parsed_template, subject_is_bytes + ) except (ValueError, _std_re.error, IndexError) as exc: raise PcreError(str(exc)) from exc - pcre2_repl = _pcre2_replacement_from_parsed(parsed_template, subject_is_bytes) try: - backend_result = backend_substitute(subject, replacement=pcre2_repl, count=0) + fast_substitute = getattr( + self._pattern, "_substitute_fast", None + ) + if fast_substitute is not None: + return fast_substitute(subject, pcre2_repl) + backend_result = backend_substitute( + subject, replacement=pcre2_repl, count=0 + ) except Exception: backend_result = NotImplemented - if backend_result is not NotImplemented and backend_result is not None: + if ( + backend_result is not NotImplemented + and backend_result is not None + ): return backend_result parsed_template = None - if self._groups_hint is not None: + if not has_extended_syntax: + # A flat one-item parsed template is equivalent to a literal + # replacement and avoids reparsing for bounded substitutions. + parsed_template = [template] + elif self._groups_hint is not None: try: - parsed_template = _parser.parse_template( - template, - TemplatePatternStub(self._groups_hint, self.groupindex), - ) + if type(self) is Pattern and type(template) in (str, bytes): + parsed_template, _ = _cached_replacement_parts( + self, template, subject_is_bytes + ) + else: + parsed_template = _parser.parse_template( + template, + TemplatePatternStub(self._groups_hint, self.groupindex), + ) except (ValueError, _std_re.error, IndexError) as exc: raise PcreError(str(exc)) from exc @@ -594,14 +888,20 @@ def subn(self, repl: Any, subject: Any, count: Any = 0) -> tuple[Any, int]: break start, end = match_obj.span() - parts.append(coerce_subject_slice(subject, last_end, start, is_bytes=subject_is_bytes)) + parts.append( + coerce_subject_slice( + subject, last_end, start, is_bytes=subject_is_bytes + ) + ) if not callable_repl: if parsed_template is None: try: parsed_template = _parser.parse_template( template, - TemplatePatternStub(len(match_obj.groups()), self.groupindex), + TemplatePatternStub( + len(match_obj.groups()), self.groupindex + ), ) except (ValueError, _std_re.error, IndexError) as exc: raise PcreError(str(exc)) from exc @@ -614,18 +914,23 @@ def subn(self, repl: Any, subject: Any, count: Any = 0) -> tuple[Any, int]: empty=empty, ) else: - replacement = normalise_replacement(repl(match_obj), is_bytes=subject_is_bytes) + replacement = normalise_replacement( + repl(match_obj), is_bytes=subject_is_bytes + ) parts.append(replacement) substitutions += 1 last_end = end - parts.append(coerce_subject_slice(subject, last_end, len(subject), is_bytes=subject_is_bytes)) + parts.append( + coerce_subject_slice( + subject, last_end, len(subject), is_bytes=subject_is_bytes + ) + ) result = join_parts(parts, is_bytes=subject_is_bytes) return result, substitutions - def parallel_map( self, subjects: Iterable[Any], @@ -652,6 +957,158 @@ def parallel_map( ) +# Keep stable references so optional C dispatch does not bypass runtime +# instrumentation or tests that replace a public wrapper method. +_PATTERN_METHODS_FOR_FAST = { + name: getattr(Pattern, name) for name in ("match", "search", "fullmatch", "findall") +} + + +def _replacement_parts_uncached( + pattern: Pattern, template: str | bytes, is_bytes: bool +) -> tuple[Any, Any]: + parsed = _parser.parse_template( + template, + TemplatePatternStub(pattern.groups, pattern.groupindex), + ) + return parsed, _pcre2_replacement_from_parsed(parsed, is_bytes) + + +def _cached_replacement_parts( + pattern: Pattern, template: str | bytes, is_bytes: bool +) -> tuple[Any, Any]: + """Cache immutable replacement parsing/conversion in the active thread.""" + + _synchronize_local_caches() + key = (pattern, template, is_bytes) + if getattr(_DEFAULT_COMPILE_LOCAL, "replacement_hot_key", None) == key: + return _DEFAULT_COMPILE_LOCAL.replacement_hot_value + limit = _DEFAULT_COMPILE_LOCAL.effective_limit + if ( + limit == 0 + or not cache_input_allowed(template) + or not cache_input_allowed(pattern.pattern) + ): + return _replacement_parts_uncached(pattern, template, is_bytes) + cached = getattr(_DEFAULT_COMPILE_LOCAL, "replacement_lru", None) + if cached is None: + cached = lru_cache(maxsize=limit)(_replacement_parts_uncached) + _DEFAULT_COMPILE_LOCAL.replacement_lru = cached + result = cached(pattern, template, is_bytes) + _DEFAULT_COMPILE_LOCAL.replacement_hot_key = key + _DEFAULT_COMPILE_LOCAL.replacement_hot_value = result + return result + + +def _clear_replacement_cache() -> None: + cached = getattr(_DEFAULT_COMPILE_LOCAL, "replacement_lru", None) + if cached is not None: + cached.cache_clear() + _DEFAULT_COMPILE_LOCAL.replacement_lru = None + _DEFAULT_COMPILE_LOCAL.replacement_hot_key = None + _DEFAULT_COMPILE_LOCAL.replacement_hot_value = None + + +def _replacement_cache_size() -> int: + cached = getattr(_DEFAULT_COMPILE_LOCAL, "replacement_lru", None) + return 0 if cached is None else cached.cache_info().currsize + + +_cached_replacement_parts.cache_clear = _clear_replacement_cache # type: ignore[attr-defined] + + +def _policy_wrapper(compiled: Pattern, thread_mode: str) -> Pattern: + """Create a policy-local wrapper around immutable cached PCRE2 code.""" + + result = Pattern(compiled._pattern) + if thread_mode == _THREAD_MODE_ENABLED: + result.enable_threads() + elif thread_mode == _THREAD_MODE_AUTO: + result.enable_auto_threads() + else: + result.disable_threads() + return result + + +def _compile_default_builtin(pattern: str | bytes) -> Pattern: + """Compile an exact built-in pattern through a per-thread direct cache.""" + thread_mode = _THREAD_MODE_AUTO if get_thread_default() else _THREAD_MODE_DISABLED + return _compile_default_snapshot( + pattern, + bool(_DEFAULT_JIT), + bool(_DEFAULT_COMPAT_REGEX), + thread_mode, + ) + + +def _compile_default_snapshot( + pattern: str | bytes, + jit: bool, + compat: bool, + thread_mode: str, +) -> Pattern: + """Compile using one coherent configuration snapshot.""" + + if getattr(_DEFAULT_COMPILE_LOCAL, "epoch", -1) != get_cache_epoch(): + _synchronize_local_caches() + cache = getattr(_DEFAULT_COMPILE_LOCAL, "cache", None) + if cache is None: + cache = _DEFAULT_COMPILE_LOCAL.cache = {} + + key = (pattern, jit, compat, thread_mode) + limit = _DEFAULT_COMPILE_LOCAL.effective_limit + compiled = cache.get(key) if limit else None + if compiled is not None: + return compiled + + adjusted_pattern = _apply_regex_compat(pattern, compat) + native_flags = ( + _pcre2.PCRE2_UTF | _pcre2.PCRE2_UCP if isinstance(adjusted_pattern, str) else 0 + ) + cached = cached_compile(adjusted_pattern, native_flags, Pattern, jit=jit) + compiled = _policy_wrapper(cached, thread_mode) + if limit and cache_input_allowed(pattern): + cache[key] = compiled + while len(cache) > limit: + cache.pop(next(iter(cache))) + return compiled + + +def _compile_flagged_builtin( + pattern: str | bytes, + native_source_flags: int, + jit: bool, + compat: bool, + thread_mode: str, +) -> Pattern: + if getattr(_DEFAULT_COMPILE_LOCAL, "epoch", -1) != get_cache_epoch(): + _synchronize_local_caches() + cache = getattr(_DEFAULT_COMPILE_LOCAL, "flagged_cache", None) + if cache is None: + cache = _DEFAULT_COMPILE_LOCAL.flagged_cache = {} + + key = (pattern, native_source_flags, bool(jit), bool(compat), thread_mode) + limit = _DEFAULT_COMPILE_LOCAL.effective_limit + compiled = cache.get(key) if limit else None + if compiled is None: + adjusted_pattern = _apply_regex_compat(pattern, compat) + effective_flags = _apply_default_unicode_flags( + adjusted_pattern, native_source_flags + ) + cached = cached_compile( + adjusted_pattern, + strip_py_only_flags(effective_flags), + Pattern, + jit=jit, + ) + compiled = _policy_wrapper(cached, thread_mode) + if limit and cache_input_allowed(pattern): + cache[key] = compiled + while len(cache) > limit: + cache.pop(next(iter(cache))) + return compiled + + def compile(pattern: Any, flags: FlagInput = 0) -> Pattern: # Fast path for the dominant shape: compile(pattern) with default flags. if flags == 0: @@ -666,17 +1123,21 @@ def compile(pattern: Any, flags: FlagInput = 0) -> Pattern: wrapper.disable_threads() return wrapper + if type(pattern) in (str, bytes) and cached_compile is _ORIGINAL_CACHED_COMPILE: + return _compile_default_builtin(pattern) + adjusted_pattern = _apply_regex_compat(pattern, bool(_DEFAULT_COMPAT_REGEX)) if isinstance(adjusted_pattern, str): native_flags = _pcre2.PCRE2_UTF | _pcre2.PCRE2_UCP else: native_flags = 0 - compiled = cached_compile(adjusted_pattern, native_flags, Pattern, jit=_DEFAULT_JIT) - if get_thread_default(): - compiled.enable_auto_threads() - else: - compiled.disable_threads() - return compiled + compiled = cached_compile( + adjusted_pattern, native_flags, Pattern, jit=_DEFAULT_JIT + ) + thread_mode = ( + _THREAD_MODE_AUTO if get_thread_default() else _THREAD_MODE_DISABLED + ) + return _policy_wrapper(compiled, thread_mode) resolved_flags = _normalise_flags(flags) threads_requested = bool(resolved_flags & THREADS) @@ -685,7 +1146,9 @@ def compile(pattern: Any, flags: FlagInput = 0) -> Pattern: if threads_requested and no_threads_requested: raise ValueError("Flag.THREADS and Flag.NO_THREADS cannot be combined") - resolved_flags_no_thread_markers = resolved_flags & ~(THREADS | NO_THREADS | COMPAT_UNICODE_ESCAPE) + resolved_flags_no_thread_markers = resolved_flags & ~( + THREADS | NO_THREADS | COMPAT_UNICODE_ESCAPE + ) jit_override = _extract_jit_override(resolved_flags_no_thread_markers) resolved_jit = _resolve_jit_setting(jit_override) compat_enabled = bool(_DEFAULT_COMPAT_REGEX or compat_requested) @@ -695,7 +1158,9 @@ def compile(pattern: Any, flags: FlagInput = 0) -> Pattern: elif no_threads_requested: thread_mode = _THREAD_MODE_DISABLED else: - thread_mode = _THREAD_MODE_AUTO if get_thread_default() else _THREAD_MODE_DISABLED + thread_mode = ( + _THREAD_MODE_AUTO if get_thread_default() else _THREAD_MODE_DISABLED + ) if isinstance(pattern, Pattern): if resolved_flags_no_thread_markers: @@ -714,9 +1179,13 @@ def compile(pattern: Any, flags: FlagInput = 0) -> Pattern: if isinstance(pattern, _CPattern): if resolved_flags_no_thread_markers: - raise ValueError("Cannot supply flags when using a compiled pattern instance.") + raise ValueError( + "Cannot supply flags when using a compiled pattern instance." + ) if jit_override is not None: - raise ValueError("Cannot supply jit when using a compiled pattern instance.") + raise ValueError( + "Cannot supply jit when using a compiled pattern instance." + ) if compat_requested: raise ValueError( "Cannot supply Flag.COMPAT_UNICODE_ESCAPE when using a compiled pattern instance." @@ -733,6 +1202,19 @@ def compile(pattern: Any, flags: FlagInput = 0) -> Pattern: wrapper.disable_threads() return wrapper + # Keep the direct flagged cache restricted to plain integer callers. The + # project IntFlag path retains the general cache, whose free-threaded + # lifetime/eviction semantics are already hardened for randomized inputs. + if type(pattern) in (str, bytes) and type(flags) in (int, Flag): + if cached_compile is _ORIGINAL_CACHED_COMPILE: + return _compile_flagged_builtin( + pattern, + resolved_flags_no_thread_markers, + resolved_jit, + compat_enabled, + thread_mode, + ) + adjusted_pattern = _apply_regex_compat(pattern, compat_enabled) effective_flags = _apply_default_unicode_flags( adjusted_pattern, resolved_flags_no_thread_markers @@ -740,47 +1222,157 @@ def compile(pattern: Any, flags: FlagInput = 0) -> Pattern: native_flags = strip_py_only_flags(effective_flags) compiled = cached_compile(adjusted_pattern, native_flags, Pattern, jit=resolved_jit) - if threads_requested: - compiled.enable_threads() - elif no_threads_requested: - compiled.disable_threads() - else: - if thread_mode == _THREAD_MODE_AUTO: - compiled.enable_auto_threads() - else: - compiled.disable_threads() - return compiled + return _policy_wrapper(compiled, thread_mode) + + +def _module_pattern_uncached( + pattern: str | bytes, + jit: bool, + compat_regex: bool, + thread_mode: str, +) -> Pattern: + return _compile_default_snapshot(pattern, jit, compat_regex, thread_mode) + + +def _cached_module_pattern( + pattern: str | bytes, + jit: bool, + compat_regex: bool, + thread_mode: str, +) -> Pattern: + """Reuse canonical wrappers for exact default-flag module calls. + + The public ``compile`` cache remains authoritative, so match ``.re`` + identity is unchanged. This small bounded layer removes repeated Python + cache-key construction and wrapper setup from module-level helpers while + keeping configuration changes in the cache key. + """ + _synchronize_local_caches() + limit = _DEFAULT_COMPILE_LOCAL.effective_limit + if limit == 0 or not cache_input_allowed(pattern): + return _module_pattern_uncached(pattern, jit, compat_regex, thread_mode) + cached = getattr(_DEFAULT_COMPILE_LOCAL, "module_lru", None) + if cached is None: + cached = lru_cache(maxsize=limit)(_module_pattern_uncached) + _DEFAULT_COMPILE_LOCAL.module_lru = cached + return cached(pattern, jit, compat_regex, thread_mode) + + +def _clear_module_cache() -> None: + cached = getattr(_DEFAULT_COMPILE_LOCAL, "module_lru", None) + if cached is not None: + cached.cache_clear() + _DEFAULT_COMPILE_LOCAL.module_lru = None + _DEFAULT_COMPILE_LOCAL.module_hot_key = None + _DEFAULT_COMPILE_LOCAL.module_hot_value = None + + +def _module_cache_size() -> int: + cached = getattr(_DEFAULT_COMPILE_LOCAL, "module_lru", None) + return 0 if cached is None else cached.cache_info().currsize + + +_cached_module_pattern.cache_clear = _clear_module_cache # type: ignore[attr-defined] + + +def _module_compile(pattern: Any, flags: FlagInput) -> Pattern: + if type(pattern) in (str, bytes) and type(flags) in (int, Flag) and flags == 0: + thread_mode = ( + _THREAD_MODE_AUTO if get_thread_default() else _THREAD_MODE_DISABLED + ) + key = (pattern, bool(_DEFAULT_JIT), bool(_DEFAULT_COMPAT_REGEX), thread_mode) + if ( + getattr(_DEFAULT_COMPILE_LOCAL, "epoch", -1) == get_cache_epoch() + and getattr(_DEFAULT_COMPILE_LOCAL, "module_hot_key", None) == key + ): + return _DEFAULT_COMPILE_LOCAL.module_hot_value + compiled = _cached_module_pattern( + pattern, + _DEFAULT_JIT, + _DEFAULT_COMPAT_REGEX, + thread_mode, + ) + if _DEFAULT_COMPILE_LOCAL.effective_limit > 0 and cache_input_allowed(pattern): + _DEFAULT_COMPILE_LOCAL.module_hot_key = key + _DEFAULT_COMPILE_LOCAL.module_hot_value = compiled + return compiled + return compile(pattern, flags=flags) + + +def _module_lookup(pattern: Any, string: Any, flags: FlagInput, method: str) -> Any: + compiled = _module_compile(pattern, flags) + if ( + type(pattern) in (str, bytes) + and type(flags) in (int, Flag) + and flags == 0 + and type(string) in (str, bytes) + and compiled._is_c_pattern + ): + if ( + method == "findall" + and getattr(compiled, "_literal_split", None) is not None + and type(string) is type(compiled._literal_split) + ): + return compiled.findall(string) + fast = getattr(compiled._pattern, f"_{method}_fast", None) + if fast is not None: + if method == "findall": + return fast(string) + if method == "finditer": + return fast(string, 0, -1, 0, compiled) + return fast(string, compiled) + return getattr(compiled, method)(string) def prefixmatch(pattern: Any, string: Any, flags: FlagInput = 0) -> Match | None: - return compile(pattern, flags=flags).match(string) + return _module_lookup(pattern, string, flags, "match") match = prefixmatch def search(pattern: Any, string: Any, flags: FlagInput = 0) -> Match | None: - return compile(pattern, flags=flags).search(string) + return _module_lookup(pattern, string, flags, "search") def fullmatch(pattern: Any, string: Any, flags: FlagInput = 0) -> Match | None: - return compile(pattern, flags=flags).fullmatch(string) + return _module_lookup(pattern, string, flags, "fullmatch") def finditer(pattern: Any, string: Any, flags: FlagInput = 0) -> Iterable[Match]: - return compile(pattern, flags=flags).finditer(string) + return _module_lookup(pattern, string, flags, "finditer") def findall(pattern: Any, string: Any, flags: FlagInput = 0) -> List[Any]: - return compile(pattern, flags=flags).findall(string) + return _module_lookup(pattern, string, flags, "findall") -def split(pattern: Any, string: Any, maxsplit: Any = 0, flags: FlagInput = 0) -> List[Any]: - return compile(pattern, flags=flags).split(string, maxsplit=maxsplit) +def split( + pattern: Any, string: Any, maxsplit: Any = 0, flags: FlagInput = 0 +) -> List[Any]: + compiled = _module_compile(pattern, flags) + if ( + type(pattern) in (str, bytes) + and type(flags) in (int, Flag) + and flags == 0 + and type(string) in (str, bytes) + and type(maxsplit) is int + and compiled._is_c_pattern + ): + literal_split = getattr(compiled, "_literal_split", None) + if literal_split is not None and type(string) is type(literal_split): + return compiled.split(string, maxsplit) + fast = getattr(compiled._pattern, "_split_fast", None) + if fast is not None: + return fast(string, maxsplit) + return compiled.split(string, maxsplit=maxsplit) -def sub(pattern: Any, repl: Any, string: Any, count: Any = 0, flags: FlagInput = 0) -> Any: - return compile(pattern, flags=flags).sub(repl, string, count=count) +def sub( + pattern: Any, repl: Any, string: Any, count: Any = 0, flags: FlagInput = 0 +) -> Any: + result = subn(pattern, repl, string, count=count, flags=flags) + return result[0] def subn( @@ -790,19 +1382,59 @@ def subn( count: Any = 0, flags: FlagInput = 0, ) -> tuple[Any, int]: - return compile(pattern, flags=flags).subn(repl, string, count=count) + compiled = _module_compile(pattern, flags) + literal_split = getattr(compiled, "_literal_split", None) + if ( + type(pattern) in (str, bytes) + and type(flags) in (int, Flag) + and flags == 0 + and type(string) in (str, bytes) + and type(repl) is type(string) + and type(count) is int + and compiled._is_c_pattern + and literal_split is not None + and type(string) is type(literal_split) + and ( + (type(repl) is str and "\\" not in repl and "$" not in repl) + or (type(repl) is bytes and b"\\" not in repl and b"$" not in repl) + ) + ): + return compiled.subn(repl, string, count=count) + if ( + type(pattern) in (str, bytes) + and type(flags) in (int, Flag) + and flags == 0 + and type(string) in (str, bytes) + and type(repl) is type(string) + and type(count) is int + and count == 0 + and compiled._is_c_pattern + and ( + (type(repl) is str and "\\" not in repl and "$" not in repl) + or (type(repl) is bytes and b"\\" not in repl and b"$" not in repl) + ) + ): + fast = getattr(compiled._pattern, "_substitute_fast", None) + if fast is not None: + return fast(string, repl) + return compiled.subn(repl, string, count=count) + # add this function to bypass signatures unit test # re.template() is deprecated and removed since python 3.12 def template(pattern, flags=0): import warnings - warnings.warn("The re.template() function is deprecated " - "as it is an undocumented function " - "without an obvious purpose. " - "Use re.compile() instead.", - DeprecationWarning) + + warnings.warn( + "The re.template() function is deprecated " + "as it is an undocumented function " + "without an obvious purpose. " + "Use re.compile() instead.", + DeprecationWarning, + ) return compile(pattern, flags | RE_TEMPLATE) + _PARALLEL_EXEC_METHODS = frozenset({"match", "search", "fullmatch", "findall"}) @@ -852,7 +1484,9 @@ def parallel_map( method_name = str(method) if method_name not in _PARALLEL_EXEC_METHODS: allowed = ", ".join(sorted(_PARALLEL_EXEC_METHODS)) - raise ValueError(f"parallel_map only supports {allowed} methods, got {method_name!r}") + raise ValueError( + f"parallel_map only supports {allowed} methods, got {method_name!r}" + ) pattern_obj = compile(pattern, flags=flags) try: @@ -871,6 +1505,30 @@ def parallel_map( "threading defaults." ) + # A one-item map cannot benefit from worker fan-out. Keep the documented + # list-shaped result but avoid executor creation, queueing, and a Future; + # this is also the safest path for explicit ``Flag.THREADS`` on tiny jobs. + if len(materials) == 1: + if mode == _THREAD_MODE_AUTO: + _should_use_auto_threads(materials) + return [bound_method(materials[0], pos=pos, endpos=endpos, options=options)] + + # Explicit threading enables safe fan-out, but tiny canonical C jobs are + # dominated by queue/executor overhead. Keep auto-threshold semantics + # untouched and process at most eight short built-in subjects inline. + if ( + mode == _THREAD_MODE_ENABLED + and pattern_obj._is_c_pattern + and type(pattern_obj) is Pattern + and 1 < len(materials) <= 8 + and all(type(subject) in (str, bytes) for subject in materials) + and max(len(subject) for subject in materials) <= 4096 + ): + return [ + bound_method(subject, pos=pos, endpos=endpos, options=options) + for subject in materials + ] + if mode == _THREAD_MODE_AUTO and not _should_use_auto_threads(materials): return [ bound_method(subject, pos=pos, endpos=endpos, options=options) @@ -884,11 +1542,62 @@ def parallel_map( ] executor = ensure_thread_pool(max_workers) + + # For the common default lookup shape, the C backend can execute directly + # without rebuilding the Python wrapper's keyword arguments for every + # subject. Restrict this to exact text/bytes values: the wrapper's normal + # path preserves buffer and subclass coercion semantics that the private + # vectorcall entry point intentionally does not duplicate. + fast_method = None + fast_method_takes_owner = False + if ( + pattern_obj._is_c_pattern + and method_name in {"match", "search", "fullmatch", "findall"} + # Preserve instrumentation/subclass overrides of the public wrapper. + # The private C entry points are valid only when dispatch is canonical. + and getattr(type(pattern_obj), method_name, None) + is _PATTERN_METHODS_FOR_FAST.get(method_name) + and type(pos) is int + and pos == 0 + and endpos is None + and type(options) is int + and options == 0 + and all(type(subject) in (str, bytes) for subject in materials) + ): + fast_method = getattr(pattern_obj._pattern, f"_{method_name}_fast", None) + fast_method_takes_owner = method_name != "findall" + + # Submit bounded batches instead of one Future per subject. The latter + # makes small/medium maps dominated by Future allocation and queue-lock + # traffic (especially on the free-threaded build), while the actual PCRE2 + # calls are already independent and release the interpreter lock for long + # subjects. Batches preserve input order and retain the same exception + # propagation behavior as the one-Future implementation. + worker_count = max(1, get_thread_pool_size()) + task_count = min(len(materials), worker_count * 2) + chunk_size = (len(materials) + task_count - 1) // task_count + + def _run_chunk(start: int, stop: int) -> list[Any]: + if fast_method is not None: + if fast_method_takes_owner: + return [ + fast_method(materials[index], pattern_obj) + for index in range(start, stop) + ] + return [fast_method(materials[index]) for index in range(start, stop)] + return [ + bound_method(materials[index], pos=pos, endpos=endpos, options=options) + for index in range(start, stop) + ] + futures = [ - executor.submit(bound_method, subject, pos=pos, endpos=endpos, options=options) - for subject in materials + executor.submit(_run_chunk, start, min(start + chunk_size, len(materials))) + for start in range(0, len(materials), chunk_size) ] - return [future.result() for future in futures] + results: List[Any] = [] + for future in futures: + results.extend(future.result()) + return results def configure(*, jit: bool | None = None, compat_regex: bool | None = None) -> bool: @@ -922,3 +1631,7 @@ def clear_cache() -> None: """Clear the compiled pattern cache and release cached match-data/JIT buffers.""" _clear_cache() + _cache_configuration_changed() + _cached_module_pattern.cache_clear() + _cached_replacement_parts.cache_clear() + _cached_expand_template.cache_clear() diff --git a/pcre/re_compat.py b/pcre/re_compat.py index 514e979..ea8b4da 100644 --- a/pcre/re_compat.py +++ b/pcre/re_compat.py @@ -9,14 +9,23 @@ import operator import re as _std_re +from functools import lru_cache +from threading import local from typing import Any, List import pcre_ext_c as _pcre2 from ._stdlib_re import _parser - +from .cache import ( + cache_input_allowed, + get_cache_epoch, + get_effective_cache_limit, + register_cache_control, +) _CRawMatch = _pcre2.Match +_EXPAND_TEMPLATE_LOCAL = local() +_MAX_GROUPINDEX_CACHE_UNITS = 64 * 1024 def prepare_subject(subject: Any) -> Any: @@ -79,6 +88,86 @@ def __init__(self, groups: int, groupindex: dict[str, int]): self.groupindex = groupindex +def _expand_template_uncached( + template: str | bytes, + groups: int, + groupindex_items: tuple[tuple[str, int], ...], +) -> Any: + return _parser.parse_template( + template, + TemplatePatternStub(groups, dict(groupindex_items)), + ) + + +def _cached_expand_template( + template: str | bytes, + groups: int, + groupindex_items: tuple[tuple[str, int], ...], +) -> Any: + """Cache immutable ``Match.expand`` parsing in the active thread. + + The rendered result is still produced per call because it depends on the + individual match. The key snapshots the current name table, so a caller + mutating the exposed ``groupindex`` mapping cannot reuse a stale parse. + """ + epoch = get_cache_epoch() + if getattr(_EXPAND_TEMPLATE_LOCAL, "epoch", -1) != epoch: + cached = getattr(_EXPAND_TEMPLATE_LOCAL, "lru", None) + if cached is not None: + cached.cache_clear() + _EXPAND_TEMPLATE_LOCAL.lru = None + _EXPAND_TEMPLATE_LOCAL.limit = get_effective_cache_limit() + _EXPAND_TEMPLATE_LOCAL.hot_key = None + _EXPAND_TEMPLATE_LOCAL.hot_value = None + _EXPAND_TEMPLATE_LOCAL.epoch = epoch + key = (template, groups, groupindex_items) + if getattr(_EXPAND_TEMPLATE_LOCAL, "hot_key", None) == key: + return _EXPAND_TEMPLATE_LOCAL.hot_value + names_size = sum(len(name) for name, _ in groupindex_items) + limit = getattr(_EXPAND_TEMPLATE_LOCAL, "limit", None) + if limit is None: + limit = get_effective_cache_limit() + _EXPAND_TEMPLATE_LOCAL.limit = limit + if ( + limit == 0 + or not cache_input_allowed(template) + or names_size > _MAX_GROUPINDEX_CACHE_UNITS + ): + return _expand_template_uncached(template, groups, groupindex_items) + cached = getattr(_EXPAND_TEMPLATE_LOCAL, "lru", None) + if cached is None: + cached = lru_cache(maxsize=limit)(_expand_template_uncached) + _EXPAND_TEMPLATE_LOCAL.lru = cached + result = cached(template, groups, groupindex_items) + _EXPAND_TEMPLATE_LOCAL.hot_key = key + _EXPAND_TEMPLATE_LOCAL.hot_value = result + return result + + +def _clear_expand_template_cache() -> None: + cached = getattr(_EXPAND_TEMPLATE_LOCAL, "lru", None) + if cached is not None: + cached.cache_clear() + _EXPAND_TEMPLATE_LOCAL.lru = None + _EXPAND_TEMPLATE_LOCAL.limit = get_effective_cache_limit() + _EXPAND_TEMPLATE_LOCAL.hot_key = None + _EXPAND_TEMPLATE_LOCAL.hot_value = None + _EXPAND_TEMPLATE_LOCAL.epoch = get_cache_epoch() + + +def _trim_expand_template_cache() -> None: + _clear_expand_template_cache() + + +def _expand_template_cache_size() -> int: + cached = getattr(_EXPAND_TEMPLATE_LOCAL, "lru", None) + return 0 if cached is None else cached.cache_info().currsize + + +_cached_expand_template.cache_clear = _clear_expand_template_cache # type: ignore[attr-defined] +register_cache_control(_trim_expand_template_cache) + + def coerce_group_value(value: Any, *, is_bytes: bool, empty: Any) -> Any: if value is None: return empty @@ -144,6 +233,31 @@ def render_template(parsed: Any, match: "Match", *, is_bytes: bool, empty: Any) and isinstance(parsed[1], list) ): group_slots, literals = parsed + if ( + len(group_slots) == 1 + and len(literals) == 1 + and group_slots[0][0] == 0 + and group_slots[0][1] >= 0 + and literals[0] is None + ): + return coerce_group_value( + match.group(group_slots[0][1]), + is_bytes=is_bytes, + empty=empty, + ) + if ( + len(group_slots) == 1 + and len(literals) == 3 + and group_slots[0][0] == 1 + and group_slots[0][1] >= 0 + and literals[1] is None + ): + group_value = coerce_group_value( + match.group(group_slots[0][1]), + is_bytes=is_bytes, + empty=empty, + ) + return literals[0] + group_value + literals[2] # Copy literals so repeated substitutions reuse the cached template. pieces: List[Any] = [empty if part is None else part for part in literals] for slot_index, group_index in group_slots: @@ -155,11 +269,32 @@ def render_template(parsed: Any, match: "Match", *, is_bytes: bool, empty: Any) ) return join_parts(pieces, is_bytes=is_bytes) + if len(parsed) == 1 and isinstance(parsed[0], int): + return coerce_group_value( + match.group(parsed[0]), + is_bytes=is_bytes, + empty=empty, + ) + if ( + len(parsed) == 3 + and isinstance(parsed[1], int) + and not isinstance(parsed[0], int) + and not isinstance(parsed[2], int) + ): + group_value = coerce_group_value( + match.group(parsed[1]), + is_bytes=is_bytes, + empty=empty, + ) + return parsed[0] + group_value + parsed[2] + pieces: List[Any] = [] for item in parsed: if isinstance(item, int): group_value = match.group(item) - pieces.append(coerce_group_value(group_value, is_bytes=is_bytes, empty=empty)) + pieces.append( + coerce_group_value(group_value, is_bytes=is_bytes, empty=empty) + ) else: pieces.append(item) return join_parts(pieces, is_bytes=is_bytes) @@ -176,10 +311,17 @@ def expand_match_template(match: Any, template: Any) -> Any: if not isinstance(template, str): raise TypeError("template must be str for text matches") - parsed = _parser.parse_template( - template, - TemplatePatternStub(match.re.groups, match.re.groupindex), - ) + groups = int(match.re.groups) + groupindex = match.re.groupindex + try: + parsed = _cached_expand_template(template, groups, tuple(groupindex.items())) + except (AttributeError, TypeError): + # Preserve compatibility with legacy mapping-like pattern doubles and + # unhashable custom templates. + parsed = _parser.parse_template( + template, + TemplatePatternStub(groups, groupindex), + ) return render_template(parsed, match, is_bytes=is_bytes, empty=empty) @@ -243,7 +385,9 @@ def is_capturing_group_start(source: str, index: int) -> bool: if source.startswith("(?P<", index) or source.startswith("(?P'", index): return True if source.startswith("(?<", index): - return not (source.startswith("(?<=", index) or source.startswith("(? bool: class Match: __slots__ = ("_match", "_pattern", "_string", "_pos", "_endpos") - def __init__(self, pattern: Any, match: _CRawMatch, subject: Any, pos: int, endpos: int) -> None: + def __init__( + self, pattern: Any, match: _CRawMatch, subject: Any, pos: int, endpos: int + ) -> None: self._match = match self._pattern = pattern self._string = subject diff --git a/pcre/threads.py b/pcre/threads.py index dec2759..7a18977 100644 --- a/pcre/threads.py +++ b/pcre/threads.py @@ -9,6 +9,7 @@ import atexit import os +import subprocess import sys import threading from concurrent.futures import ThreadPoolExecutor @@ -21,6 +22,7 @@ _THREAD_POOL: ThreadPoolExecutor | None = None _THREAD_POOL_WORKERS: int | None = None _THREAD_AUTO_THRESHOLD: int = 60_000 +_PERFORMANCE_CPU_TOTAL: int | None = None def _cpu_total() -> int: @@ -33,6 +35,27 @@ def threading_supported() -> bool: return _cpu_total() >= _MIN_CORES_FOR_THREADS +def _performance_cpu_total() -> int: + """Return macOS performance-tier logical CPUs when the kernel exposes it.""" + + global _PERFORMANCE_CPU_TOTAL + if _PERFORMANCE_CPU_TOTAL is not None: + return _PERFORMANCE_CPU_TOTAL + if sys.platform != "darwin": + _PERFORMANCE_CPU_TOTAL = 0 + return 0 + try: + value = subprocess.check_output( + ["sysctl", "-n", "hw.perflevel0.logicalcpu"], + text=True, + stderr=subprocess.DEVNULL, + ) + _PERFORMANCE_CPU_TOTAL = max(0, int(value.strip())) + except (OSError, ValueError, subprocess.CalledProcessError): + _PERFORMANCE_CPU_TOTAL = 0 + return _PERFORMANCE_CPU_TOTAL + + _THREADS_DEFAULT: bool = threading_supported() and not ( hasattr(sys, "_is_gil_enabled") and sys._is_gil_enabled() ) @@ -42,6 +65,9 @@ def _max_threads() -> int: if not threading_supported(): return 0 cpu_total = _cpu_total() + performance_total = _performance_cpu_total() + if performance_total > 0: + return performance_total return max(1, cpu_total // 4) @@ -92,7 +118,9 @@ def ensure_thread_pool(max_workers: int | None = None) -> ThreadPoolExecutor: return _THREAD_POOL -def configure_thread_pool(*, max_workers: int | None = None, preload: bool = False) -> int: +def configure_thread_pool( + *, max_workers: int | None = None, preload: bool = False +) -> int: """Set the shared executor size used by :func:`parallel_map`. Returns the effective worker count after applying the update. @@ -137,7 +165,9 @@ def get_thread_pool_size() -> int: return _THREAD_POOL_WORKERS -def configure_threads(*, enabled: bool | None = None, threshold: int | None = None) -> bool: +def configure_threads( + *, enabled: bool | None = None, threshold: int | None = None +) -> bool: """Adjust the global threading defaults and/or auto threshold.""" global _THREADS_DEFAULT diff --git a/pcre_ext/pcre2.c b/pcre_ext/pcre2.c index c54f4c1..a0dcd90 100644 --- a/pcre_ext/pcre2.c +++ b/pcre_ext/pcre2.c @@ -72,6 +72,7 @@ coerce_uint32_argument(PyObject *value, const char *name, uint32_t *out) * short matches the extra work is measurable, so only release for large inputs. */ #define PCRE2_GIL_RELEASE_THRESHOLD 262144ULL +#define PCRE_PATTERN_CACHE_INPUT_LIMIT (64 * 1024) #if defined(Py_GIL_DISABLED) #define PCRE2_CALL_RELEASE_GIL(call) \ @@ -252,6 +253,7 @@ Match_dealloc(MatchObject *self) Py_XDECREF(self->public_pattern); Py_XDECREF(self->subject); Py_XDECREF(self->utf8_owner); + Py_XDECREF(self->regs_cache); pcre_free(self->ovector); Py_TYPE(self)->tp_free((PyObject *)self); } @@ -301,6 +303,18 @@ match_resolve_span(MatchObject *self, return 0; } + /* UTF-8 offsets are identical to Python indexes for ASCII subjects. This + is the hot path for log/token matching: avoid rescanning the prefix for + every span/start/end accessor (which otherwise makes a late match in a + large ASCII subject O(subject length) per accessor). The subject is an + owned immutable reference for the lifetime of this match, so its ASCII + kind cannot change while this snapshot is queried, including GIL=0. */ + if (PyUnicode_IS_ASCII(self->subject)) { + *start_out = start; + *end_out = end; + return 0; + } + const char *data = self->utf8_data; *start_out = utf8_offset_to_index(data, start); *end_out = utf8_offset_to_index(data, end); @@ -329,6 +343,30 @@ resolve_group_key(MatchObject *self, PyObject *key, Py_ssize_t *index) return -1; } + /* The immutable groupindex mapping handles the overwhelmingly common + unique-name case in O(1). If the selected entry did not + participate, retain the PCRE2 name-table walk below so DUPNAMES + still selects the participating capture exactly like ``re``. */ + if (PyUnicode_CheckExact(key)) { + PyObject *mapped = PyDict_GetItemWithError(self->pattern->groupindex, key); + if (mapped != NULL) { + Py_ssize_t candidate = PyLong_AsSsize_t(mapped); + if (candidate == -1 && PyErr_Occurred()) { + return -1; + } + if (candidate >= 0 && (size_t)candidate < self->ovec_count) { + Py_ssize_t start = self->ovector[(size_t)candidate * 2]; + Py_ssize_t end = self->ovector[(size_t)candidate * 2 + 1]; + if (start >= 0 && end >= 0) { + *index = candidate; + return 0; + } + } + } else if (PyErr_Occurred()) { + return -1; + } + } + uint32_t name_count = 0; uint32_t entry_size = 0; PCRE2_SPTR name_table = NULL; @@ -494,14 +532,13 @@ match_get_group_value(MatchObject *self, Py_ssize_t index) } static PyObject * -Match_group(MatchObject *self, PyObject *args) +Match_group_fast(MatchObject *self, PyObject *const *args, Py_ssize_t nargs) { - Py_ssize_t nargs = PyTuple_GET_SIZE(args); if (nargs == 0) { return match_get_group_value(self, 0); } if (nargs == 1) { - PyObject *key = PyTuple_GET_ITEM(args, 0); + PyObject *key = args[0]; Py_ssize_t index = 0; if (resolve_group_key(self, key, &index) < 0) { return NULL; @@ -513,7 +550,7 @@ Match_group(MatchObject *self, PyObject *args) return NULL; } for (Py_ssize_t i = 0; i < nargs; ++i) { - PyObject *key = PyTuple_GET_ITEM(args, i); + PyObject *key = args[i]; Py_ssize_t index = 0; if (resolve_group_key(self, key, &index) < 0) { Py_DECREF(result); @@ -530,43 +567,37 @@ Match_group(MatchObject *self, PyObject *args) } static PyObject * -Match_groups(MatchObject *self, PyObject *args, PyObject *kwargs) +Match_span_fast(MatchObject *self, PyObject *const *args, Py_ssize_t nargs) { - static char *kwlist[] = {"default", NULL}; - PyObject *default_value = Py_None; - if (!PyArg_ParseTupleAndKeywords(args, kwargs, "|O", kwlist, &default_value)) { + if (nargs > 1) { + PyErr_Format(PyExc_TypeError, "span expected at most 1 argument, got %zd", nargs); return NULL; } - - PyObject *result = PyTuple_New(self->ovec_count - 1); - if (result == NULL) { + PyObject *key = nargs == 0 ? NULL : args[0]; + Py_ssize_t index = 0; + if (resolve_group_key(self, key, &index) < 0) { return NULL; } - - for (uint32_t i = 1; i < self->ovec_count; ++i) { - PyObject *value = match_get_group_value(self, (Py_ssize_t)i); - if (value == NULL) { - Py_DECREF(result); - return NULL; - } - if (value == Py_None && default_value != Py_None) { - Py_DECREF(value); - Py_INCREF(default_value); - value = default_value; - } - PyTuple_SET_ITEM(result, i - 1, value); + Py_ssize_t start = 0; + Py_ssize_t end = 0; + int rc = match_resolve_span(self, index, &start, &end, 1); + if (rc < 0) { + return NULL; } - - return result; + if (rc > 0) { + Py_RETURN_NONE; + } + return Py_BuildValue("(nn)", start, end); } static PyObject * -Match_span(MatchObject *self, PyObject *args) +Match_start_fast(MatchObject *self, PyObject *const *args, Py_ssize_t nargs) { - PyObject *key = NULL; - if (!PyArg_ParseTuple(args, "|O", &key)) { + if (nargs > 1) { + PyErr_Format(PyExc_TypeError, "start expected at most 1 argument, got %zd", nargs); return NULL; } + PyObject *key = nargs == 0 ? NULL : args[0]; Py_ssize_t index = 0; if (resolve_group_key(self, key, &index) < 0) { return NULL; @@ -580,33 +611,62 @@ Match_span(MatchObject *self, PyObject *args) if (rc > 0) { Py_RETURN_NONE; } - return Py_BuildValue("(nn)", start, end); + return PyLong_FromSsize_t(start); } static PyObject * -Match_start(MatchObject *self, PyObject *args) +Match_end_fast(MatchObject *self, PyObject *const *args, Py_ssize_t nargs) { - PyObject *span = Match_span(self, args); - if (span == NULL || span == Py_None) { - return span; - } - PyObject *value = PyTuple_GET_ITEM(span, 0); - Py_INCREF(value); - Py_DECREF(span); - return value; + if (nargs > 1) { + PyErr_Format(PyExc_TypeError, "end expected at most 1 argument, got %zd", nargs); + return NULL; + } + PyObject *key = nargs == 0 ? NULL : args[0]; + Py_ssize_t index = 0; + if (resolve_group_key(self, key, &index) < 0) { + return NULL; + } + Py_ssize_t start = 0; + Py_ssize_t end = 0; + int rc = match_resolve_span(self, index, &start, &end, 1); + if (rc < 0) { + return NULL; + } + if (rc > 0) { + Py_RETURN_NONE; + } + return PyLong_FromSsize_t(end); } static PyObject * -Match_end(MatchObject *self, PyObject *args) +Match_groups(MatchObject *self, PyObject *args, PyObject *kwargs) { - PyObject *span = Match_span(self, args); - if (span == NULL || span == Py_None) { - return span; - } - PyObject *value = PyTuple_GET_ITEM(span, 1); - Py_INCREF(value); - Py_DECREF(span); - return value; + static char *kwlist[] = {"default", NULL}; + PyObject *default_value = Py_None; + if (!PyArg_ParseTupleAndKeywords(args, kwargs, "|O", kwlist, &default_value)) { + return NULL; + } + + PyObject *result = PyTuple_New(self->ovec_count - 1); + if (result == NULL) { + return NULL; + } + + for (uint32_t i = 1; i < self->ovec_count; ++i) { + PyObject *value = match_get_group_value(self, (Py_ssize_t)i); + if (value == NULL) { + Py_DECREF(result); + return NULL; + } + if (value == Py_None && default_value != Py_None) { + Py_DECREF(value); + Py_INCREF(default_value); + value = default_value; + } + PyTuple_SET_ITEM(result, i - 1, value); + } + + return result; } static PyObject * @@ -958,6 +1018,17 @@ Match_get_lastgroup(MatchObject *self, void *closure) static PyObject * Match_get_regs(MatchObject *self, void *closure) { + PyObject *cached = NULL; + Py_BEGIN_CRITICAL_SECTION(self); + if (self->regs_cache != NULL) { + cached = self->regs_cache; + Py_INCREF(cached); + } + Py_END_CRITICAL_SECTION(); + if (cached != NULL) { + return cached; + } + PyObject *result = PyTuple_New(self->ovec_count); if (result == NULL) { return NULL; @@ -978,19 +1049,64 @@ Match_get_regs(MatchObject *self, void *closure) PyTuple_SET_ITEM(result, index, span); } - return result; + /* Match snapshots are immutable. Keep one tuple so repeated ``regs`` + reads avoid rebuilding every span; the critical section makes the + first publication safe on free-threaded CPython. */ + Py_BEGIN_CRITICAL_SECTION(self); + if (self->regs_cache == NULL) { + self->regs_cache = result; + Py_INCREF(result); + } + cached = self->regs_cache; + Py_INCREF(cached); + Py_END_CRITICAL_SECTION(); + Py_DECREF(result); + return cached; } static PyObject * Match_expand(MatchObject *self, PyObject *template_obj) { + /* A template without a backslash has no group references or escapes. + Return its validated text directly instead of importing the Python + parser and allocating its intermediate representation. This mirrors + ``re.Match.expand`` for literal text while keeping subclass conversion + in PyUnicode_FromObject. */ + if (!self->subject_is_bytes && PyUnicode_Check(template_obj)) { + Py_ssize_t template_length = PyUnicode_GET_LENGTH(template_obj); + if (PyUnicode_FindChar(template_obj, '\\', 0, template_length, 1) < 0 && + !PyErr_Occurred()) { + return PyUnicode_FromObject(template_obj); + } + PyErr_Clear(); + } + if (self->subject_is_bytes && + (PyBytes_Check(template_obj) || PyByteArray_Check(template_obj))) { + const char *template_data = PyBytes_Check(template_obj) + ? PyBytes_AS_STRING(template_obj) + : (const char *)PyByteArray_AS_STRING(template_obj); + Py_ssize_t template_length = PyBytes_Check(template_obj) + ? PyBytes_GET_SIZE(template_obj) + : PyByteArray_GET_SIZE(template_obj); + if (memchr(template_data, '\\', (size_t)template_length) == NULL) { + return PyBytes_FromObject(template_obj); + } + } + /* Delegate template parsing to the Python compatibility helper. */ PyObject *module = PyImport_ImportModule("pcre.re_compat"); if (module == NULL) { return NULL; } - PyObject *helper = PyObject_GetAttrString(module, "expand_match_template"); + /* The helper is a module function, so a direct dictionary lookup avoids + attribute lookup machinery on every expand() call while retaining the + module's normal import/refcount lifetime. */ + PyObject *helper = PyDict_GetItemString( + PyModule_GetDict(module), + "expand_match_template" + ); + Py_XINCREF(helper); Py_DECREF(module); if (helper == NULL) { return NULL; @@ -1045,12 +1161,12 @@ match_set_public_pattern(MatchObject *self, PyObject *public_pattern) } static PyMethodDef Match_methods[] = { - {"group", (PyCFunction)Match_group, METH_VARARGS, PyDoc_STR("Return one or more capture groups.")}, + {"group", (PyCFunction)(void(*)(void))Match_group_fast, METH_FASTCALL, PyDoc_STR("Return one or more capture groups.")}, {"groups", (PyCFunction)Match_groups, METH_VARARGS | METH_KEYWORDS, PyDoc_STR("Return all capture groups as a tuple." )}, {"groupdict", (PyCFunction)Match_groupdict, METH_VARARGS | METH_KEYWORDS, PyDoc_STR("Return a dict for named capture groups." )}, - {"span", (PyCFunction)Match_span, METH_VARARGS, PyDoc_STR("Return the (start, end) span for a group." )}, - {"start", (PyCFunction)Match_start, METH_VARARGS, PyDoc_STR("Return the start index for a group." )}, - {"end", (PyCFunction)Match_end, METH_VARARGS, PyDoc_STR("Return the end index for a group." )}, + {"span", (PyCFunction)(void(*)(void))Match_span_fast, METH_FASTCALL, PyDoc_STR("Return the (start, end) span for a group." )}, + {"start", (PyCFunction)(void(*)(void))Match_start_fast, METH_FASTCALL, PyDoc_STR("Return the start index for a group." )}, + {"end", (PyCFunction)(void(*)(void))Match_end_fast, METH_FASTCALL, PyDoc_STR("Return the end index for a group." )}, {"expand", (PyCFunction)Match_expand, METH_O, PyDoc_STR("Apply a replacement template to the match." )}, {"__copy__", (PyCFunction)Match_copy, METH_NOARGS, NULL}, {"__deepcopy__", (PyCFunction)Match_deepcopy, METH_O, NULL}, @@ -1651,6 +1767,7 @@ create_match_object(PatternObject *pattern, match->public_endpos = endpos; match->replay_options = replay_options; match->lastindex_cache = -2; + match->regs_cache = NULL; /* Anything that isn't str (bytes, or a buffer-protocol object such as mmap.mmap) is treated as raw byte data: offsets are byte offsets and group values are returned as bytes. */ @@ -2800,6 +2917,16 @@ Pattern_findall(PatternObject *self, Py_ssize_t current_byte = byte_start; Py_ssize_t current_pos = pos; int retry_nonempty = 0; + /* + * A findall scan usually performs one expensive PCRE2 call followed by + * Python object construction. Release the GIL for that first large + * scan, but stop doing so after a match: patterns such as ``.`` can + * produce millions of tiny matches and repeatedly saving/restoring the + * GIL would cost more than the matcher. Each worker owns its match data + * and keeps ``utf8_owner`` alive, so the PCRE2 call remains thread-safe. + */ + int release_gil_for_match = + subject_length_bytes > (Py_ssize_t)PCRE2_GIL_RELEASE_THRESHOLD; while (1) { if (current_byte > subject_length_bytes) { @@ -2823,15 +2950,26 @@ Pattern_findall(PatternObject *self, int use_jit = attempt_jit && !retry_nonempty; if (use_jit) { - jit_guard_acquire(); - rc = pcre2_jit_match(self->code, - (PCRE2_SPTR)utf8_data, - exec_length, - (PCRE2_SIZE)current_byte, - current_options, - match_data, - match_context); - jit_guard_release(); + if (release_gil_for_match) { + PCRE2_JIT_CALL_MAYBE_RELEASE_GIL(pcre2_jit_match(self->code, + (PCRE2_SPTR)utf8_data, + exec_length, + (PCRE2_SIZE)current_byte, + current_options, + match_data, + match_context), + exec_length); + } else { + jit_guard_acquire(); + rc = pcre2_jit_match(self->code, + (PCRE2_SPTR)utf8_data, + exec_length, + (PCRE2_SIZE)current_byte, + current_options, + match_data, + match_context); + jit_guard_release(); + } if (rc == PCRE2_ERROR_JIT_BADOPTION || rc == PCRE2_ERROR_BADOPTION) { pattern_jit_set(self, 0); @@ -2854,13 +2992,24 @@ Pattern_findall(PatternObject *self, } if (!use_jit) { - rc = pcre2_match(self->code, - (PCRE2_SPTR)utf8_data, - exec_length, - (PCRE2_SIZE)current_byte, - current_options, - match_data, - match_context); + if (release_gil_for_match) { + PCRE2_CALL_MAYBE_RELEASE_GIL(pcre2_match(self->code, + (PCRE2_SPTR)utf8_data, + exec_length, + (PCRE2_SIZE)current_byte, + current_options, + match_data, + match_context), + exec_length); + } else { + rc = pcre2_match(self->code, + (PCRE2_SPTR)utf8_data, + exec_length, + (PCRE2_SIZE)current_byte, + current_options, + match_data, + match_context); + } if (rc == PCRE2_ERROR_NOMATCH) { goto findall_no_match; @@ -2885,6 +3034,7 @@ Pattern_findall(PatternObject *self, if (value == NULL) { goto error; } + release_gil_for_match = 0; if (PyList_Append(result, value) < 0) { Py_DECREF(value); goto error; @@ -3547,6 +3697,117 @@ Pattern_split_method(PatternObject *self, PyObject *args, PyObject *kwargs) return Pattern_split(self, subject, maxsplit); } +/* + * Private vectorcall entry points used by the Python wrapper's default-shape + * dispatch. The public methods retain their keyword-compatible ABI; these + * helpers only remove temporary tuple/keyword objects from the allocation-heavy + * findall/substitute paths. Replacement and match ownership remain unchanged. + */ +static PyObject * +Pattern_findall_fast(PatternObject *self, PyObject *const *args, Py_ssize_t nargs) +{ + if (nargs != 1) { + PyErr_Format(PyExc_TypeError, + "_findall_fast() takes exactly 1 positional argument (%zd given)", + nargs); + return NULL; + } + return Pattern_findall(self, args[0], 0, -1, 0); +} + +static PyObject * +Pattern_substitute_fast(PatternObject *self, PyObject *const *args, Py_ssize_t nargs) +{ + if (nargs != 2) { + PyErr_Format(PyExc_TypeError, + "_substitute_fast() takes exactly 2 positional arguments (%zd given)", + nargs); + return NULL; + } + return Pattern_substitute(self, args[0], args[1], 0); +} + +/* + * These lookup helpers are intentionally private. They are used only by the + * high-level parallel fan-out when every optional argument has its default + * value, avoiding a Python wrapper call and its keyword dictionary per item. + * The owner is still supplied so Match.re retains the public Pattern wrapper. + */ +static PyObject * +Pattern_lookup_fast(PatternObject *self, + PyObject *const *args, + Py_ssize_t nargs, + int mode, + const char *name) +{ + if (nargs != 2) { + PyErr_Format(PyExc_TypeError, + "%s() takes exactly 2 positional arguments (%zd given)", + name, nargs); + return NULL; + } + return Pattern_execute(self, args[0], 0, -1, 0, mode, args[1]); +} + +static PyObject * +Pattern_match_fast(PatternObject *self, PyObject *const *args, Py_ssize_t nargs) +{ + return Pattern_lookup_fast(self, args, nargs, EXEC_MODE_MATCH, "_match_fast"); +} + +static PyObject * +Pattern_search_fast(PatternObject *self, PyObject *const *args, Py_ssize_t nargs) +{ + return Pattern_lookup_fast(self, args, nargs, EXEC_MODE_SEARCH, "_search_fast"); +} + +static PyObject * +Pattern_fullmatch_fast(PatternObject *self, PyObject *const *args, Py_ssize_t nargs) +{ + return Pattern_lookup_fast(self, args, nargs, EXEC_MODE_FULLMATCH, "_fullmatch_fast"); +} + +static PyObject * +Pattern_finditer_fast(PatternObject *self, PyObject *const *args, Py_ssize_t nargs) +{ + if (nargs != 5) { + PyErr_Format(PyExc_TypeError, + "_finditer_fast() takes exactly 5 positional arguments (%zd given)", + nargs); + return NULL; + } + + Py_ssize_t pos = PyLong_AsSsize_t(args[1]); + if (pos == (Py_ssize_t)-1 && PyErr_Occurred()) { + return NULL; + } + Py_ssize_t endpos = PyLong_AsSsize_t(args[2]); + if (endpos == (Py_ssize_t)-1 && PyErr_Occurred()) { + return NULL; + } + uint32_t options = 0; + if (coerce_uint32_argument(args[3], "options", &options) < 0) { + return NULL; + } + return Pattern_create_finditer(self, args[0], pos, endpos, options, args[4]); +} + +static PyObject * +Pattern_split_fast(PatternObject *self, PyObject *const *args, Py_ssize_t nargs) +{ + if (nargs != 2) { + PyErr_Format(PyExc_TypeError, + "_split_fast() takes exactly 2 positional arguments (%zd given)", + nargs); + return NULL; + } + Py_ssize_t maxsplit = PyLong_AsSsize_t(args[1]); + if (maxsplit == (Py_ssize_t)-1 && PyErr_Occurred()) { + return NULL; + } + return Pattern_split(self, args[0], maxsplit); +} + static PyMethodDef Pattern_methods[] = { {"findall", (PyCFunction)Pattern_findall_method, METH_VARARGS | METH_KEYWORDS, PyDoc_STR("Return a list of all non-overlapping matches.")}, {"substitute", (PyCFunction)Pattern_substitute_method, METH_VARARGS | METH_KEYWORDS, PyDoc_STR("Fast substitution using pcre2_substitute.")}, @@ -3555,6 +3816,13 @@ static PyMethodDef Pattern_methods[] = { {"match", (PyCFunction)Pattern_match_method, METH_VARARGS | METH_KEYWORDS, PyDoc_STR("Match the pattern at the start of the subject.")}, {"search", (PyCFunction)Pattern_search_method, METH_VARARGS | METH_KEYWORDS, PyDoc_STR("Search the subject for the pattern." )}, {"fullmatch", (PyCFunction)Pattern_fullmatch_method, METH_VARARGS | METH_KEYWORDS, PyDoc_STR("Require the pattern to match the entire subject." )}, + {"_findall_fast", (PyCFunction)(void(*)(void))Pattern_findall_fast, METH_FASTCALL, NULL}, + {"_substitute_fast", (PyCFunction)(void(*)(void))Pattern_substitute_fast, METH_FASTCALL, NULL}, + {"_match_fast", (PyCFunction)(void(*)(void))Pattern_match_fast, METH_FASTCALL, NULL}, + {"_search_fast", (PyCFunction)(void(*)(void))Pattern_search_fast, METH_FASTCALL, NULL}, + {"_fullmatch_fast", (PyCFunction)(void(*)(void))Pattern_fullmatch_fast, METH_FASTCALL, NULL}, + {"_finditer_fast", (PyCFunction)(void(*)(void))Pattern_finditer_fast, METH_FASTCALL, NULL}, + {"_split_fast", (PyCFunction)(void(*)(void))Pattern_split_fast, METH_FASTCALL, NULL}, {NULL, NULL, 0, NULL}, }; @@ -3767,6 +4035,14 @@ Pattern_compile_cached(PyObject *pattern_obj, uint32_t flags, int jit, int jit_e PyObject *jit_explicit_bool = NULL; PyObject *cache_key = NULL; int use_cache = PyUnicode_CheckExact(pattern_obj) || PyBytes_CheckExact(pattern_obj); + if (use_cache) { + Py_ssize_t cache_units = PyUnicode_CheckExact(pattern_obj) + ? PyUnicode_GET_LENGTH(pattern_obj) + : PyBytes_GET_SIZE(pattern_obj); + if (cache_units > PCRE_PATTERN_CACHE_INPUT_LIMIT) { + use_cache = 0; + } + } PatternObject *result = NULL; flags_obj = PyLong_FromUnsignedLong(flags); diff --git a/pcre_ext/pcre2_module.h b/pcre_ext/pcre2_module.h index 64f486d..5a43e7e 100644 --- a/pcre_ext/pcre2_module.h +++ b/pcre_ext/pcre2_module.h @@ -91,6 +91,8 @@ typedef struct { int subject_is_bytes; uint32_t replay_options; int lastindex_cache; + /* Lazily materialized immutable regs tuple; protected for GIL=0 callers. */ + PyObject *regs_cache; } MatchObject; extern PyTypeObject PatternType; diff --git a/tests/test_c_api_audit.py b/tests/test_c_api_audit.py index a381f4e..8def604 100644 --- a/tests/test_c_api_audit.py +++ b/tests/test_c_api_audit.py @@ -40,6 +40,38 @@ def test_match_repr_uses_character_offsets() -> None: assert "span=(0, 1)" in repr(match) +def test_ascii_match_accessors_preserve_offsets_without_prefix_rescans() -> None: + """Late ASCII matches must retain byte/code-point offset identity. + + This also exercises the C fast path used by span/start/end/regs. ASCII + subjects are common in logs and protocol tokens; their UTF-8 offsets are + already Python character indexes, so no prefix scan is necessary. + """ + subject = "x" * 100_000 + "token" + match = pcre.search("token", subject) + assert match is not None + assert match.span() == (100_000, 100_005) + assert match.start() == 100_000 + assert match.end() == 100_005 + assert match.regs == ((100_000, 100_005),) + + +def test_large_findall_scan_preserves_capture_results() -> None: + pattern = raw.compile(r"(\d+)", flags=TEXT_FLAGS, jit=False) + subject = "x" * 300_000 + "123 x45" + assert pattern.findall(subject) == ["123", "45"] + + +def test_match_expand_literal_template_keeps_replacement_semantics() -> None: + match = pcre.compile(r"(?Pword)").fullmatch("word") + assert match is not None + assert match.expand("literal $1") == "literal $1" + assert match.expand(r"literal\g") == "literalword" + bytes_match = pcre.compile(b"(?Pword)").fullmatch(b"word") + assert bytes_match is not None + assert bytes_match.expand(bytearray(b"literal $1")) == b"literal $1" + + def test_pattern_clamps_start_past_end_for_empty_match() -> None: pattern = raw.compile("", flags=TEXT_FLAGS, jit=False) for method_name in ("match", "search", "fullmatch"): @@ -102,6 +134,23 @@ def test_utf_bytes_empty_matches_advance_by_codepoint() -> None: assert pattern.split(subject) == [b"", subject, b""] +def test_vectorcall_iteration_and_split_entries_match_public_methods() -> None: + pattern = raw.compile(r"(x)", flags=TEXT_FLAGS, jit=False) + subject = "x x" + + public_matches = [match.span() for match in pattern.finditer(subject)] + fast_matches = [ + match.span() for match in pattern._finditer_fast(subject, 0, -1, 0, None) + ] + assert fast_matches == public_matches + assert pattern._split_fast(subject, 0) == pattern.split(subject, 0) + + with pytest.raises(TypeError): + pattern._finditer_fast(subject) # type: ignore[call-arg] + with pytest.raises(TypeError): + pattern._split_fast(subject) # type: ignore[call-arg] + + def test_mutable_utf_buffer_is_snapshotted_before_iteration() -> None: subject = bytearray("é".encode()) pattern = raw.compile(b"", flags=TEXT_FLAGS, jit=False) @@ -135,6 +184,48 @@ def test_duplicate_name_selects_participating_capture( assert match.groupdict() == {"x": expected} +def test_group_name_subclass_does_not_require_hashing() -> None: + class Name(str): + def __hash__(self) -> int: + raise AssertionError("group lookup should use the PCRE name table") + + match = pcre.compile(r"(?Pa)").fullmatch("a") + assert match is not None + assert match.group(Name("name")) == "a" + + +def test_regs_is_cached_as_an_immutable_match_snapshot() -> None: + match = pcre.compile(r"(?Pa)(?Pb)").fullmatch("ab") + assert match is not None + first = match.regs + assert first == ((0, 2), (0, 1), (1, 2)) + assert match.regs is first + + +def test_groups_results_are_fresh_and_do_not_persist_captured_values() -> None: + match = pcre.compile(r"(?Pa)(?Pb)?").fullmatch("a") + assert match is not None + first = match.groups() + assert first == ("a", None) + assert match.groups() is not first + assert match.groups("missing") == ("a", "missing") + assert match.groups("missing") is not match.groups("missing") + + +def test_module_default_fast_dispatch_preserves_public_pattern_and_results() -> None: + compiled = pcre.compile(r"(x)") + match = pcre.match(r"(x)", "x") + assert match is not None + assert match.re is compiled + assert [item.span() for item in pcre.finditer(r"(x)", "xx")] == [ + (0, 1), + (1, 2), + ] + assert pcre.findall(r"(x)", "xx") == ["x", "x"] + assert pcre.split(r"(x)", "xx") == ["", "x", "", "x", ""] + assert pcre.subn(r"(x)", "[X]", "xx") == ("[X][X]", 2) + + def test_global_inline_options_are_exposed() -> None: assert pcre.compile(r"(?i)a").flags & int(pcre.Flag.CASELESS) assert not (pcre.compile(r"(?i:a)").flags & int(pcre.Flag.CASELESS)) diff --git a/tests/test_cache.py b/tests/test_cache.py index 569d537..a22d951 100644 --- a/tests/test_cache.py +++ b/tests/test_cache.py @@ -26,7 +26,9 @@ @pytest.fixture(autouse=True) def _reset_cache_state() -> None: if cache_mod.cache_strategy() != "thread-local": - pytest.skip("global pattern cache enabled via environment; thread-local tests skipped") + pytest.skip( + "global pattern cache enabled via environment; thread-local tests skipped" + ) original_limit = cache_mod.get_cache_limit() cache_mod.clear_cache() try: @@ -42,7 +44,9 @@ def _fresh_thread_cache() -> OrderedDict[Any, Any]: return store -def _run_cache_script(source: str, env_overrides: Dict[str, str] | None = None) -> Dict[str, Any]: +def _run_cache_script( + source: str, env_overrides: Dict[str, str] | None = None +) -> Dict[str, Any]: env = os.environ.copy() pythonpath_entries = [str(PROJECT_ROOT)] existing_pythonpath = env.get("PYTHONPATH") @@ -77,7 +81,9 @@ def _format(row: List[str]) -> str: return lines -def _emit_table(pytestconfig: pytest.Config, title: str, headers: List[str], rows: List[List[str]]) -> None: +def _emit_table( + pytestconfig: pytest.Config, title: str, headers: List[str], rows: List[List[str]] +) -> None: lines = _format_table(headers, rows) reporter = pytestconfig.pluginmanager.get_plugin("terminalreporter") if reporter is None: # pragma: no cover - fallback for unusual runners @@ -101,7 +107,9 @@ def _emit_table(pytestconfig: pytest.Config, title: str, headers: List[str], row writer.hasmarkup = original_hasmarkup -def _benchmark_strategy(strategy: str, iterations: int = 20000, threads: int = 1) -> Dict[str, Any]: +def _benchmark_strategy( + strategy: str, iterations: int = 20000, threads: int = 1 +) -> Dict[str, Any]: script = textwrap.dedent( f""" import json @@ -199,7 +207,9 @@ def test_cached_compile_thread_local_isolation(monkeypatch: pytest.MonkeyPatch) compile_calls: List[str] = [] def fake_compile(pattern: Any, *, flags: int = 0, jit: bool = False) -> str: - compile_calls.append(f"{threading.current_thread().name}:{pattern}:{flags}:{jit}") + compile_calls.append( + f"{threading.current_thread().name}:{pattern}:{flags}:{jit}" + ) return f"compiled:{len(compile_calls)}" monkeypatch.setattr(cache_mod._pcre2, "compile", fake_compile) @@ -250,11 +260,15 @@ def worker() -> None: assert list(main_store.keys()) == [("expr", 0, False)] # untouched by worker -def test_clear_cache_only_resets_current_thread(monkeypatch: pytest.MonkeyPatch) -> None: +def test_clear_cache_invalidates_live_worker_on_next_use( + monkeypatch: pytest.MonkeyPatch, +) -> None: compile_calls: List[str] = [] def fake_compile(pattern: Any, *, flags: int = 0, jit: bool = False) -> str: - compile_calls.append(f"{threading.current_thread().name}:{pattern}:{flags}:{jit}") + compile_calls.append( + f"{threading.current_thread().name}:{pattern}:{flags}:{jit}" + ) return f"compiled:{len(compile_calls)}" monkeypatch.setattr(cache_mod._pcre2, "compile", fake_compile) @@ -307,10 +321,12 @@ def worker() -> None: assert not worker_errors assert worker_state["initial_len"] == 1 assert worker_state["final_len"] == 1 - assert worker_state["first_result"] == worker_state["second_result"] == "compiled:2" + assert worker_state["first_result"] == "compiled:2" + assert worker_state["second_result"] == "compiled:3" assert compile_calls == [ "MainThread:expr:0:False", "cache-worker:expr:0:False", + "cache-worker:expr:0:False", ] @@ -364,8 +380,12 @@ def test_cache_strategy_invalid_value() -> None: cache_mod.cache_strategy("totally-invalid") -def test_cache_strategy_cannot_switch_after_use(monkeypatch: pytest.MonkeyPatch) -> None: - monkeypatch.setattr(cache_mod._pcre2, "compile", lambda pattern, *, flags=0, jit=False: pattern) +def test_cache_strategy_cannot_switch_after_use( + monkeypatch: pytest.MonkeyPatch, +) -> None: + monkeypatch.setattr( + cache_mod._pcre2, "compile", lambda pattern, *, flags=0, jit=False: pattern + ) def wrapper(raw: Any) -> Any: return raw @@ -379,7 +399,9 @@ def wrapper(raw: Any) -> Any: def test_set_cache_limit_zero_clears_cache(monkeypatch: pytest.MonkeyPatch) -> None: - monkeypatch.setattr(cache_mod._pcre2, "compile", lambda pattern, *, flags=0, jit=False: pattern) + monkeypatch.setattr( + cache_mod._pcre2, "compile", lambda pattern, *, flags=0, jit=False: pattern + ) def wrapper(raw: Any) -> Any: return raw @@ -398,7 +420,9 @@ def wrapper(raw: Any) -> Any: def test_set_cache_limit_shrink_evicts_oldest(monkeypatch: pytest.MonkeyPatch) -> None: - monkeypatch.setattr(cache_mod._pcre2, "compile", lambda pattern, *, flags=0, jit=False: pattern) + monkeypatch.setattr( + cache_mod._pcre2, "compile", lambda pattern, *, flags=0, jit=False: pattern + ) def wrapper(raw: Any) -> Any: return raw @@ -414,7 +438,9 @@ def wrapper(raw: Any) -> Any: assert list(store.keys()) == [("3", 0, False), ("4", 0, False)] -def test_cached_compile_bypasses_cache_for_unhashable_key(monkeypatch: pytest.MonkeyPatch) -> None: +def test_cached_compile_bypasses_cache_for_unhashable_key( + monkeypatch: pytest.MonkeyPatch, +) -> None: calls: list[Any] = [] def fake_compile(pattern: Any, *, flags: int = 0, jit: bool = False) -> Any: @@ -478,8 +504,18 @@ def test_shared_pattern_concurrent_execution_produces_expected_results() -> None cases: List[tuple[str, int, int, tuple[str, str] | None]] = [ (subject_one, 0, subject_one.index(" "), ("123", "alpha")), - (subject_two, subject_two.index("456"), subject_two.index(" suffix"), ("456", "beta")), - (subject_three, subject_three.index("789"), subject_three.index(" end"), ("789", "gamma")), + ( + subject_two, + subject_two.index("456"), + subject_two.index(" suffix"), + ("456", "beta"), + ), + ( + subject_three, + subject_three.index("789"), + subject_three.index(" end"), + ("789", "gamma"), + ), (subject_four, 0, len(subject_four), None), ] @@ -499,7 +535,9 @@ def worker(index: int) -> None: match = pattern.search(subject, pos=pos, endpos=endpos) if expected is None: if match is not None: - raise AssertionError(f"expected no match for {subject!r}, got {match.group(0)!r}") + raise AssertionError( + f"expected no match for {subject!r}, got {match.group(0)!r}" + ) local.append(None) continue if match is None: @@ -592,7 +630,9 @@ def worker(): """ ) - result = _run_cache_script(script, env_overrides={"PYPCRE_CACHE_PATTERN_GLOBAL": "1"}) + result = _run_cache_script( + script, env_overrides={"PYPCRE_CACHE_PATTERN_GLOBAL": "1"} + ) assert result["calls"] == 1 assert result["main_result"] == result["worker_result"] == "compiled:1" @@ -604,7 +644,9 @@ def test_cache_strategy_benchmark(pytestconfig: pytest.Config) -> None: max_threads = max(1, cpu_count // 2) # Limit concurrency to half of the visible cores to avoid full-saturation regressions while still # covering progressively larger thread counts when possible. - thread_counts = [count for count in desired_thread_counts if count == 1 or count <= max_threads] + thread_counts = [ + count for count in desired_thread_counts if count == 1 or count <= max_threads + ] scenarios = [ (strategy, thread_count) for thread_count in thread_counts @@ -613,14 +655,23 @@ def test_cache_strategy_benchmark(pytestconfig: pytest.Config) -> None: results: Dict[tuple[str, int], Dict[str, Any]] = {} for strategy, thread_count in scenarios: - stats = _benchmark_strategy(strategy, iterations=iterations, threads=thread_count) + stats = _benchmark_strategy( + strategy, iterations=iterations, threads=thread_count + ) key = (strategy, thread_count) results[key] = stats assert stats["strategy"] == strategy assert stats["threads"] == thread_count assert stats["elapsed"] >= 0.0 - headers = ["strategy", "threads", "total calls", "total ms", "per call ns", "relative"] + headers = [ + "strategy", + "threads", + "total calls", + "total ms", + "per call ns", + "relative", + ] for thread_count in thread_counts: local_stats = results[("thread-local", thread_count)] global_stats = results[("global", thread_count)] @@ -636,8 +687,12 @@ def test_cache_strategy_benchmark(pytestconfig: pytest.Config) -> None: baseline = local_stats["elapsed"] or 1.0 for strategy in ("thread-local", "global"): stats = results[(strategy, thread_count)] - total_calls = stats.get("total_calls", stats["iterations"] * stats.get("threads", 1)) - per_call_ns = (stats["elapsed"] / total_calls) * 1e9 if total_calls else float("nan") + total_calls = stats.get( + "total_calls", stats["iterations"] * stats.get("threads", 1) + ) + per_call_ns = ( + (stats["elapsed"] / total_calls) * 1e9 if total_calls else float("nan") + ) relative = stats["elapsed"] / baseline if baseline else float("nan") rows.append( [ @@ -650,5 +705,9 @@ def test_cache_strategy_benchmark(pytestconfig: pytest.Config) -> None: ] ) - title = "Cache strategy benchmark (single-thread)" if thread_count == 1 else f"Cache strategy benchmark (threads={thread_count})" + title = ( + "Cache strategy benchmark (single-thread)" + if thread_count == 1 + else f"Cache strategy benchmark (threads={thread_count})" + ) _emit_table(pytestconfig, title, headers, rows) diff --git a/tests/test_cache_global.py b/tests/test_cache_global.py index 8572f1f..b53c98f 100644 --- a/tests/test_cache_global.py +++ b/tests/test_cache_global.py @@ -13,13 +13,12 @@ import sys import textwrap from pathlib import Path -from typing import Any, Dict - +from typing import Any PROJECT_ROOT = Path(__file__).resolve().parents[1] -def _run_global_cache_script(source: str) -> Dict[str, Any]: +def _run_global_cache_script(source: str) -> dict[str, Any]: env = os.environ.copy() pythonpath_entries = [str(PROJECT_ROOT)] existing_pythonpath = env.get("PYTHONPATH") @@ -134,3 +133,33 @@ def fake_compile(pattern, *, flags=0, jit=False): ) result = _run_global_cache_script(script) assert result["size"] == 2 + + +def test_global_cache_shares_only_immutable_code_between_thread_policies() -> None: + script = textwrap.dedent( + """ + import concurrent.futures + import json + import threading + import pcre + + barrier = threading.Barrier(2) + def worker(flag): + barrier.wait(timeout=5) + pattern = pcre.compile("shared-policy", flags=int(flag)) + return id(pattern), id(pattern._pattern), pattern.thread_mode + + with concurrent.futures.ThreadPoolExecutor(max_workers=2) as executor: + enabled_future = executor.submit(worker, pcre.Flag.THREADS) + disabled_future = executor.submit(worker, pcre.Flag.NO_THREADS) + enabled = enabled_future.result() + disabled = disabled_future.result() + + print(json.dumps({"enabled": enabled, "disabled": disabled})) + """ + ) + result = _run_global_cache_script(script) + assert result["enabled"][0] != result["disabled"][0] + assert result["enabled"][1] == result["disabled"][1] + assert result["enabled"][2] == "enabled" + assert result["disabled"][2] == "disabled" diff --git a/tests/test_cache_global_inproc.py b/tests/test_cache_global_inproc.py index a9bd714..bc55d1c 100644 --- a/tests/test_cache_global_inproc.py +++ b/tests/test_cache_global_inproc.py @@ -7,11 +7,13 @@ from __future__ import annotations +import threading from typing import Any -import pcre.cache as cache_mod import pytest +import pcre.cache as cache_mod + def test_env_flag_is_true() -> None: assert cache_mod._env_flag_is_true(None) is False @@ -87,16 +89,31 @@ def wrapper(raw: Any) -> Any: assert len(fresh_state.pattern_cache) == 0 -def test_global_set_cache_limit_none(monkeypatch: pytest.MonkeyPatch) -> None: +def test_global_set_cache_limit_none_uses_hard_ceiling( + monkeypatch: pytest.MonkeyPatch, +) -> None: fresh_state = cache_mod._GlobalCacheState() monkeypatch.setattr(cache_mod, "_GLOBAL_STATE", fresh_state) monkeypatch.setattr(cache_mod, "_CACHE_STRATEGY", cache_mod._CacheStrategy.GLOBAL) + monkeypatch.setattr( + cache_mod._pcre2, + "compile", + lambda pattern, *, flags=0, jit=False: pattern, + ) + + def wrapper(raw: Any) -> Any: + return raw cache_mod.set_cache_limit(None) assert cache_mod.get_cache_limit() is None + for index in range(cache_mod._HARD_CACHE_ENTRY_LIMIT + 16): + cache_mod.cached_compile(str(index), 0, wrapper, jit=False) + assert len(fresh_state.pattern_cache) == cache_mod._HARD_CACHE_ENTRY_LIMIT -def test_global_set_cache_limit_shrinks_existing_entries(monkeypatch: pytest.MonkeyPatch) -> None: +def test_global_set_cache_limit_shrinks_existing_entries( + monkeypatch: pytest.MonkeyPatch, +) -> None: def fake_compile(pattern: Any, *, flags: int = 0, jit: bool = False) -> Any: return pattern @@ -117,7 +134,9 @@ def wrapper(raw: Any) -> Any: assert len(fresh_state.pattern_cache) == 1 -def test_global_cached_compile_handles_unhashable_key(monkeypatch: pytest.MonkeyPatch) -> None: +def test_global_cached_compile_handles_unhashable_key( + monkeypatch: pytest.MonkeyPatch, +) -> None: calls: list[Any] = [] def fake_compile(pattern: Any, *, flags: int = 0, jit: bool = False) -> Any: @@ -140,7 +159,9 @@ def wrapper(raw: Any) -> Any: assert len(fresh_state.pattern_cache) == 0 -def test_global_cached_compile_respects_limit_set_to_zero_after_compile(monkeypatch: pytest.MonkeyPatch) -> None: +def test_global_cached_compile_respects_limit_set_to_zero_after_compile( + monkeypatch: pytest.MonkeyPatch, +) -> None: calls: list[Any] = [] def fake_compile(pattern: Any, *, flags: int = 0, jit: bool = False) -> Any: @@ -161,3 +182,35 @@ def wrapper(raw: Any) -> Any: cache_mod.cached_compile("x", 0, wrapper, jit=False) assert len(fresh_state.pattern_cache) == 0 + + +def test_global_clear_does_not_allow_inflight_compile_to_repopulate( + monkeypatch: pytest.MonkeyPatch, +) -> None: + fresh_state = cache_mod._GlobalCacheState() + monkeypatch.setattr(cache_mod, "_GLOBAL_STATE", fresh_state) + monkeypatch.setattr(cache_mod, "_CACHE_STRATEGY", cache_mod._CacheStrategy.GLOBAL) + compile_started = threading.Event() + finish_compile = threading.Event() + + def fake_compile(pattern: Any, *, flags: int = 0, jit: bool = False) -> Any: + compile_started.set() + assert finish_compile.wait(timeout=5) + return pattern + + monkeypatch.setattr(cache_mod._pcre2, "compile", fake_compile) + result: list[Any] = [] + worker = threading.Thread( + target=lambda: result.append( + cache_mod.cached_compile("inflight", 0, lambda raw: raw, jit=False) + ) + ) + worker.start() + assert compile_started.wait(timeout=5) + cache_mod.clear_cache() + finish_compile.set() + worker.join(timeout=5) + + assert not worker.is_alive() + assert result == ["inflight"] + assert fresh_state.pattern_cache == {} diff --git a/tests/test_cache_scope_safety.py b/tests/test_cache_scope_safety.py new file mode 100644 index 0000000..9fd1462 --- /dev/null +++ b/tests/test_cache_scope_safety.py @@ -0,0 +1,276 @@ +# SPDX-FileCopyrightText: 2025 ModelCloud.ai +# SPDX-FileCopyrightText: 2025 qubitium@modelcloud.ai +# SPDX-License-Identifier: Apache-2.0 +# Contact: qubitium@modelcloud.ai, x.com/qubitium + +from __future__ import annotations + +import concurrent.futures +import gc +import importlib +import threading +import tracemalloc +import weakref + +import pcre_ext_c + +import pcre +from pcre import cache as cache_mod +from pcre import pcre as pcre_mod +from pcre import re_compat + + +def test_explicit_thread_policies_have_distinct_stable_wrappers() -> None: + pcre.clear_cache() + enabled = pcre.compile("policy", flags=int(pcre.Flag.THREADS)) + disabled = pcre.compile("policy", flags=int(pcre.Flag.NO_THREADS)) + + assert enabled is not disabled + assert enabled._pattern is disabled._pattern + assert enabled.thread_mode == "enabled" + assert disabled.thread_mode == "disabled" + + assert pcre.compile("policy", flags=int(pcre.Flag.THREADS)) is enabled + assert enabled.thread_mode == "enabled" + assert disabled.thread_mode == "disabled" + + +def test_concurrent_thread_policy_compiles_cannot_clobber_each_other() -> None: + barrier = threading.Barrier(2) + + def compile_with(flag: pcre.Flag) -> pcre.Pattern: + barrier.wait(timeout=5) + result = pcre.compile("concurrent-policy", flags=int(flag)) + for _ in range(1000): + expected = "enabled" if flag == pcre.Flag.THREADS else "disabled" + assert result.thread_mode == expected + return result + + pcre.clear_cache() + with concurrent.futures.ThreadPoolExecutor(max_workers=2) as executor: + enabled_future = executor.submit(compile_with, pcre.Flag.THREADS) + disabled_future = executor.submit(compile_with, pcre.Flag.NO_THREADS) + enabled = enabled_future.result() + disabled = disabled_future.result() + + assert enabled is not disabled + assert enabled.thread_mode == "enabled" + assert disabled.thread_mode == "disabled" + + +def test_module_cache_tracks_thread_default_without_mutating_old_pattern() -> None: + threads_mod = importlib.import_module("pcre.threads") + original = threads_mod.get_thread_default() + pcre.clear_cache() + try: + pcre.configure_threads(enabled=False) + disabled = pcre.search("module-policy", "module-policy").re + pcre.configure_threads(enabled=True) + automatic = pcre.search("module-policy", "module-policy").re + finally: + pcre.configure_threads(enabled=original) + pcre.clear_cache() + + assert automatic is not disabled + assert automatic._pattern is disabled._pattern + assert disabled.thread_mode == "disabled" + assert automatic.thread_mode == "auto" + + +def test_cache_limit_zero_disables_all_high_level_helper_caches() -> None: + original = cache_mod.get_cache_limit() + try: + pcre.set_cache_limit(0) + pcre.clear_cache() + first = pcre.search("uncached-module", "uncached-module").re + second = pcre.search("uncached-module", "uncached-module").re + assert first is not second + assert pcre_mod._local_cache("cache") == {} + assert pcre_mod._module_cache_size() == 0 + assert pcre_mod._replacement_cache_size() == 0 + finally: + pcre.set_cache_limit(original) + pcre.clear_cache() + + +def test_limit_change_during_thread_local_compile_cannot_repopulate_cache( + monkeypatch, +) -> None: + original = cache_mod.get_cache_limit() + try: + cache_mod.set_cache_limit(2) + cache_mod.clear_cache() + monkeypatch.setattr( + cache_mod._pcre2, + "compile", + lambda pattern, *, flags=0, jit=False: pattern, + ) + + def disable_cache(raw): + cache_mod.set_cache_limit(0) + return raw + + assert ( + cache_mod._cached_compile_thread_local( + "reentrant-limit", 0, disable_cache, jit=False + ) + == "reentrant-limit" + ) + assert cache_mod._THREAD_LOCAL.pattern_cache == {} + finally: + cache_mod.set_cache_limit(original) + pcre.clear_cache() + + +def test_none_limit_uses_hard_entry_ceiling() -> None: + original = cache_mod.get_cache_limit() + try: + pcre.set_cache_limit(None) + pcre.clear_cache() + for index in range(cache_mod._HARD_CACHE_ENTRY_LIMIT + 32): + pcre.compile(f"bounded-{index}") + assert len(pcre_mod._local_cache("cache")) == cache_mod._HARD_CACHE_ENTRY_LIMIT + assert ( + len(cache_mod._THREAD_LOCAL.pattern_cache) + == cache_mod._HARD_CACHE_ENTRY_LIMIT + ) + finally: + pcre.set_cache_limit(original) + pcre.clear_cache() + + +def test_oversized_patterns_are_not_retained_by_python_or_c_caches() -> None: + source = "(?#" + "x" * cache_mod._MAX_CACHE_INPUT_UNITS + ")a" + pcre.clear_cache() + first = pcre.compile(source) + second = pcre.compile(source) + assert first is not second + assert first._pattern is not second._pattern + assert pcre_mod._local_cache("cache") == {} + assert cache_mod._THREAD_LOCAL.pattern_cache == {} + + raw_first = pcre_ext_c.compile(source, jit=False) + raw_second = pcre_ext_c.compile(source, jit=False) + assert raw_first is not raw_second + + flagged_first = pcre.compile(source, flags=int(pcre.Flag.CASELESS)) + flagged_second = pcre.compile(source, flags=int(pcre.Flag.CASELESS)) + assert flagged_first is not flagged_second + assert pcre_mod._local_cache("flagged_cache") == {} + + assert pcre.search(source, "a") is not None + assert pcre_mod._module_cache_size() == 0 + + +def test_oversized_templates_are_not_retained() -> None: + pattern = pcre.compile("(x)") + match = pattern.fullmatch("x") + assert match is not None + template = "y" * cache_mod._MAX_CACHE_INPUT_UNITS + r"\1" + + pcre_mod._cached_replacement_parts.cache_clear() + assert pattern.sub(template, "x") == "y" * cache_mod._MAX_CACHE_INPUT_UNITS + "x" + assert pcre_mod._replacement_cache_size() == 0 + + re_compat._cached_expand_template.cache_clear() + assert match.expand(template) == "y" * cache_mod._MAX_CACHE_INPUT_UNITS + "x" + assert re_compat._expand_template_cache_size() == 0 + + +def test_helper_pattern_wrappers_do_not_cross_thread_scope() -> None: + barrier = threading.Barrier(2) + + def worker() -> pcre.Pattern: + barrier.wait(timeout=5) + return pcre.search("thread-scope", "thread-scope").re + + pcre.clear_cache() + with concurrent.futures.ThreadPoolExecutor(max_workers=2) as executor: + first, second = executor.map(lambda _: worker(), range(2)) + assert first is not second + + +def test_clear_cache_invalidates_a_live_workers_direct_cache() -> None: + def worker() -> pcre.Pattern: + return pcre.compile("worker-generation") + + with concurrent.futures.ThreadPoolExecutor(max_workers=1) as executor: + first = executor.submit(worker).result() + pcre.clear_cache() + second = executor.submit(worker).result() + assert first is not second + + +def test_cache_control_registry_does_not_retain_reloaded_callbacks() -> None: + def callback() -> None: + pass + + callback_ref = weakref.ref(callback) + cache_mod.register_cache_control(callback) + assert callback in cache_mod._CACHE_CONTROL_CALLBACKS + del callback + gc.collect() + assert callback_ref() is None + + +def test_module_uncached_compile_uses_its_configuration_snapshot( + monkeypatch, +) -> None: + base = pcre.compile("snapshot-base", flags=int(pcre.Flag.NO_JIT)) + observed: list[tuple[str, int, bool]] = [] + + def fake_cached_compile(pattern, flags, wrapper, *, jit): + observed.append((pattern, flags, jit)) + return base + + monkeypatch.setattr(pcre_mod, "cached_compile", fake_cached_compile) + monkeypatch.setattr( + pcre_mod, + "_apply_regex_compat", + lambda pattern, enabled: f"{pattern}:{enabled}", + ) + compiled = pcre_mod._module_pattern_uncached( + "snapshot", False, True, pcre_mod._THREAD_MODE_ENABLED + ) + + assert observed == [ + ( + "snapshot:True", + pcre_ext_c.PCRE2_UTF | pcre_ext_c.PCRE2_UCP, + False, + ) + ] + assert compiled is not base + assert compiled._pattern is base._pattern + assert compiled.thread_mode == "enabled" + + +def test_groups_does_not_retain_large_materialized_capture() -> None: + captured = "x" * (1024 * 1024) + subject = "[" + captured + "]" + match = pcre.compile(r"\[(.*)\]").fullmatch(subject) + assert match is not None + + tracemalloc.start() + try: + groups = match.groups() + assert groups == (captured,) + del groups + gc.collect() + retained, _ = tracemalloc.get_traced_memory() + finally: + tracemalloc.stop() + + assert retained < 16 * 1024 + + +def test_groups_is_safe_for_concurrent_free_threaded_reads() -> None: + match = pcre.compile("(a)(b)?").fullmatch("ab") + assert match is not None + + def read_groups() -> None: + for _ in range(5000): + assert match.groups() == ("a", "b") + + with concurrent.futures.ThreadPoolExecutor(max_workers=8) as executor: + list(executor.map(lambda _: read_groups(), range(8))) diff --git a/tests/test_coverage_gaps.py b/tests/test_coverage_gaps.py index 45ae97b..87579af 100644 --- a/tests/test_coverage_gaps.py +++ b/tests/test_coverage_gaps.py @@ -158,6 +158,19 @@ def test_parallel_map_empty_subjects() -> None: assert pcre.parallel_map(pattern, []) == [] +def test_parallel_map_single_subject_avoids_executor(monkeypatch: pytest.MonkeyPatch) -> None: + pattern = pcre.compile(r"\d+", flags=pcre.Flag.THREADS) + monkeypatch.setattr( + pcre_mod, + "ensure_thread_pool", + lambda *_args, **_kwargs: pytest.fail("single-item map must not create a pool"), + ) + results = pcre.parallel_map(pattern, ["123"], method="findall") + assert results == [["123"]] + small_batch = pcre.parallel_map(pattern, ["1", "22", "333"], method="findall") + assert small_batch == [["1"], ["22"], ["333"]] + + def test_parallel_map_invalid_method() -> None: pattern = pcre.compile(r"\d+", flags=pcre.Flag.THREADS) with pytest.raises(ValueError, match="parallel_map"): @@ -195,8 +208,9 @@ def test_parallel_map_memoryview_subject(monkeypatch: pytest.MonkeyPatch) -> Non def test_parallel_map_threading_unsupported_fallback(monkeypatch: pytest.MonkeyPatch) -> None: monkeypatch.setattr(pcre_mod, "threading_supported", lambda: False) pattern = pcre.compile(r"\d+", flags=pcre.Flag.THREADS) - results = pcre.parallel_map(pattern, ["1", "22"]) - assert [m.group(0) for m in results] == ["1", "22"] + subjects = ["1", "22", "333", "4444", "5", "66", "777", "8888", "9"] + results = pcre.parallel_map(pattern, subjects) + assert [m.group(0) for m in results] == subjects def test_compile_disabled_default_with_non_thread_flags(monkeypatch: pytest.MonkeyPatch) -> None: diff --git a/tests/test_module.py b/tests/test_module.py index 02059ec..44bd178 100644 --- a/tests/test_module.py +++ b/tests/test_module.py @@ -34,14 +34,19 @@ def test_module_finditer_and_findall_helpers(): groups = pcre.findall(r"(\w)(\d)", "a1 b2 c3") assert groups == [("a", "1"), ("b", "2"), ("c", "3")] + assert pcre.findall("foo", "foo foo") == ["foo", "foo"] def test_module_split_and_substitutions(): assert pcre.split(r"\s*,\s*", "a, b, c") == ["a", "b", "c"] assert pcre.split(r"\s+", "one two", maxsplit=1) == ["one", "two"] + assert pcre.split(", ", "a, b, c") == ["a", "b", "c"] templated = pcre.sub(r"(?P\d+)", r"<\g>", "item1 item2") assert templated == "item<1> item<2>" + assert pcre.sub("foo", "bar", "foo foo") == "bar bar" + assert pcre.subn("foo", "bar", "foo foo", count=1) == ("bar foo", 1) + assert pcre.subn("foo", "bar", "foo foo", count=-1) == ("foo foo", 0) def bump(match): return str(int(match.group(0)) + 1) diff --git a/tests/test_python_coverage_audit.py b/tests/test_python_coverage_audit.py index 8d56596..cbaf817 100644 --- a/tests/test_python_coverage_audit.py +++ b/tests/test_python_coverage_audit.py @@ -27,11 +27,420 @@ def test_stdlib_parser_fallback(monkeypatch: pytest.MonkeyPatch) -> None: - monkeypatch.delattr(re, "_parser") + # Python 3.10 does not expose ``re._parser`` at module scope, while newer + # releases do. Either state should exercise our fallback loader. + monkeypatch.delattr(re, "_parser", raising=False) parser = stdlib_re._load_parser() assert callable(parser.parse) +def test_stdlib_parser_exported_path(monkeypatch: pytest.MonkeyPatch) -> None: + exported = object() + monkeypatch.setattr(stdlib_re._std_re, "_parser", exported, raising=False) + assert stdlib_re._load_parser() is exported + + +def test_flat_replacement_conversion_paths() -> None: + assert pcre_mod._pcre2_replacement_from_parsed([1, b"$"], True) == b"\\g<1>$$" + assert pcre_mod._pcre2_replacement_from_parsed([1, "$"], False) == r"\g<1>$$" + + +def test_replacement_template_cache_reuses_and_clears( + monkeypatch: pytest.MonkeyPatch, +) -> None: + pattern = pcre.compile(r"(x)") + pcre_mod._cached_replacement_parts.cache_clear() + original = pcre_mod._parser.parse_template + calls = 0 + + def counted_parse(template: Any, state: Any) -> Any: + nonlocal calls + calls += 1 + return original(template, state) + + monkeypatch.setattr(pcre_mod._parser, "parse_template", counted_parse) + assert pattern.sub(r"[\1]", "x") == "[x]" + assert pattern.sub(r"[\1]", "x") == "[x]" + assert calls == 1 + + pcre.clear_cache() + assert pattern.sub(r"[\1]", "x") == "[x]" + assert calls == 2 + + with pytest.raises(pcre.PcreError): + pattern.sub(r"\2", "x", count=1) + + +def test_local_cache_lazy_initializers_and_module_lru_clear() -> None: + pcre.clear_cache() + pcre_mod._DEFAULT_COMPILE_LOCAL.flagged_cache = None + assert pcre_mod._local_cache("flagged_cache") == {} + + pcre_mod._DEFAULT_COMPILE_LOCAL.cache = None + pcre_mod._DEFAULT_COMPILE_LOCAL.epoch = cache_mod.get_cache_epoch() + assert pcre.compile("lazy-default-cache").pattern == "lazy-default-cache" + + assert pcre.search("module-lru-clear", "module-lru-clear") is not None + assert pcre_mod._module_cache_size() == 1 + pcre_mod._clear_module_cache() + assert pcre_mod._module_cache_size() == 0 + + +def test_expand_template_cache_reuses_and_clears( + monkeypatch: pytest.MonkeyPatch, +) -> None: + match = pcre.compile(r"(?Px)").fullmatch("x") + assert match is not None + compat._cached_expand_template.cache_clear() + original = compat._parser.parse_template + calls = 0 + + def counted_parse(template: Any, state: Any) -> Any: + nonlocal calls + calls += 1 + return original(template, state) + + monkeypatch.setattr(compat._parser, "parse_template", counted_parse) + assert match.expand(r"[\g]") == "[x]" + assert match.expand(r"[\g]") == "[x]" + assert calls == 1 + + pcre.clear_cache() + assert match.expand(r"[\g]") == "[x]" + assert calls == 2 + + +def test_expand_template_legacy_groupindex_falls_back() -> None: + class LegacyGroupIndex(dict[str, int]): + def items(self) -> Any: + raise AttributeError("legacy mapping has no items snapshot") + + pattern = type( + "LegacyPattern", (), {"groups": 1, "groupindex": LegacyGroupIndex()} + )() + match = compat.Match( + pattern, + type("RawMatch", (), {"group": lambda self, index: "x"})(), + "x", + 0, + 1, + ) + assert match.expand(r"[\1]") == "[x]" + + +def test_expand_render_single_capture_fast_shapes() -> None: + class RawMatch: + def group(self, index: int) -> str: + assert index == 1 + return "x" + + raw = RawMatch() + assert ( + compat.render_template(([(0, 1)], [None]), raw, is_bytes=False, empty="") == "x" + ) + assert ( + compat.render_template( + ([(1, 1)], ["[", None, "]"]), raw, is_bytes=False, empty="" + ) + == "[x]" + ) + assert compat.render_template([1], raw, is_bytes=False, empty="") == "x" + assert compat.render_template(["[", 1, "]"], raw, is_bytes=False, empty="") == "[x]" + + class TwoGroupRawMatch: + def group(self, index: int) -> str: + return "x" if index == 1 else "y" + + assert ( + compat.render_template( + ["[", 1, "-", 2, "]"], + TwoGroupRawMatch(), + is_bytes=False, + empty="", + ) + == "[x-y]" + ) + assert ( + compat.render_template( + ([(1, 1), (3, 2)], ["[", None, "-", None, "]"]), + TwoGroupRawMatch(), + is_bytes=False, + empty="", + ) + == "[x-y]" + ) + + +def test_cache_control_registration_and_oversized_global_bypass( + monkeypatch: pytest.MonkeyPatch, +) -> None: + def callback() -> None: + pass + + try: + cache_mod.register_cache_control(callback) + cache_mod.register_cache_control(callback) + assert callback in cache_mod._CACHE_CONTROL_CALLBACKS + finally: + cache_mod._CACHE_CONTROL_CALLBACKS.discard(callback) + + original_cache = cache_mod._GLOBAL_STATE.pattern_cache + original_limit = cache_mod._GLOBAL_STATE.cache_limit + cache_mod._GLOBAL_STATE.pattern_cache = {} + cache_mod._GLOBAL_STATE.cache_limit = 2 + monkeypatch.setattr( + cache_mod._pcre2, + "compile", + lambda pattern, *, flags=0, jit=False: f"compiled:{len(pattern)}", + ) + oversized = "x" * (cache_mod._MAX_CACHE_INPUT_UNITS + 1) + try: + assert ( + cache_mod._cached_compile_global( + oversized, 0, lambda value: value, jit=False + ) + == f"compiled:{len(oversized)}" + ) + assert cache_mod._GLOBAL_STATE.pattern_cache == {} + finally: + cache_mod._GLOBAL_STATE.pattern_cache = original_cache + cache_mod._GLOBAL_STATE.cache_limit = original_limit + + +def test_expand_template_cache_limit_and_trim_branches() -> None: + original_limit = cache_mod.get_cache_limit() + try: + cache_mod.set_cache_limit(2) + compat._EXPAND_TEMPLATE_LOCAL.epoch = cache_mod.get_cache_epoch() + compat._EXPAND_TEMPLATE_LOCAL.lru = None + compat._EXPAND_TEMPLATE_LOCAL.limit = None + assert compat._cached_expand_template("literal", 0, ()) + assert compat._expand_template_cache_size() == 1 + + cache_mod.set_cache_limit(1) + compat._cached_expand_template(r"\1-a", 1, ()) + compat._cached_expand_template(r"\1-b", 1, ()) + assert compat._expand_template_cache_size() == 1 + + compat._EXPAND_TEMPLATE_LOCAL.epoch = cache_mod.get_cache_epoch() - 1 + compat._cached_expand_template("new-epoch", 0, ()) + assert compat._expand_template_cache_size() == 1 + compat._trim_expand_template_cache() + assert compat._expand_template_cache_size() == 0 + + cache_mod.set_cache_limit(0) + assert compat._cached_expand_template("not-cached", 0, ()) + assert compat._expand_template_cache_size() == 0 + + cache_mod.set_cache_limit(2) + long_name = "n" * (compat._MAX_GROUPINDEX_CACHE_UNITS + 1) + assert compat._cached_expand_template("name-not-cached", 1, ((long_name, 1),)) + assert compat._expand_template_cache_size() == 0 + finally: + cache_mod.set_cache_limit(original_limit) + pcre.clear_cache() + + +def test_default_compile_cache_is_thread_local_and_tracks_thread_mode( + monkeypatch: pytest.MonkeyPatch, +) -> None: + pcre.clear_cache() + monkeypatch.setattr(pcre_mod, "get_thread_default", lambda: False) + first = pcre.compile("compile-cache") + assert pcre.compile("compile-cache") is first + assert first.thread_mode == pcre_mod._THREAD_MODE_DISABLED + + monkeypatch.setattr(pcre_mod, "get_thread_default", lambda: True) + auto = pcre.compile("compile-cache") + assert auto is not first + assert auto._pattern is first._pattern + assert auto.thread_mode == pcre_mod._THREAD_MODE_AUTO + assert first.thread_mode == pcre_mod._THREAD_MODE_DISABLED + + pcre.clear_cache() + assert pcre.compile("compile-cache") is not first + + original_limit = cache_mod.get_cache_limit() + try: + cache_mod.set_cache_limit(0) + pcre.clear_cache() + uncached = pcre.compile("compile-cache-disabled") + assert pcre.compile("compile-cache-disabled") is not uncached + finally: + cache_mod.set_cache_limit(original_limit) + pcre.clear_cache() + + +def test_compile_legacy_default_fallback_and_subject_length( + monkeypatch: pytest.MonkeyPatch, +) -> None: + backend = pcre_mod._pcre2.compile("x", jit=False) + + def fake_cached(*args: Any, **kwargs: Any) -> pcre_mod.Pattern: + return pcre_mod.Pattern(backend) + + monkeypatch.setattr(pcre_mod, "cached_compile", fake_cached) + monkeypatch.setattr(pcre_mod, "get_thread_default", lambda: False) + compiled = pcre_mod.compile(bytearray(b"x")) + assert compiled.thread_mode == pcre_mod._THREAD_MODE_DISABLED + assert ( + pcre_mod.compile(bytearray(b"x"), flags=int(pcre.Flag.THREADS)).thread_mode + == pcre_mod._THREAD_MODE_ENABLED + ) + assert ( + pcre_mod.compile(bytearray(b"x"), flags=int(pcre.Flag.NO_THREADS)).thread_mode + == pcre_mod._THREAD_MODE_DISABLED + ) + assert ( + pcre_mod.compile(bytearray(b"x"), flags=int(pcre.Flag.CASELESS)).thread_mode + == pcre_mod._THREAD_MODE_DISABLED + ) + assert pcre_mod._subject_length(bytearray(b"x")) == 1 + + +def test_flagged_builtin_compile_cache_tracks_mode_and_limits( + monkeypatch: pytest.MonkeyPatch, +) -> None: + pcre.clear_cache() + pcre_mod._DEFAULT_COMPILE_LOCAL.flagged_cache = None + monkeypatch.setattr(pcre_mod, "get_thread_default", lambda: True) + flags = int(pcre.Flag.CASELESS) + first = pcre.compile("flag-cache", flags=flags) + assert pcre.compile("flag-cache", flags=flags) is first + assert first.thread_mode == pcre_mod._THREAD_MODE_AUTO + enum_first = pcre.compile("enum-cache", flags=pcre.Flag.CASELESS) + assert pcre.compile("enum-cache", flags=pcre.Flag.CASELESS) is enum_first + + monkeypatch.setattr(pcre_mod, "get_thread_default", lambda: False) + disabled = pcre.compile("flag-cache", flags=flags) + assert disabled is not first + assert disabled._pattern is first._pattern + assert disabled.thread_mode == pcre_mod._THREAD_MODE_DISABLED + assert first.thread_mode == pcre_mod._THREAD_MODE_AUTO + + enabled = pcre.compile("flag-enabled", flags=int(pcre.Flag.THREADS)) + assert enabled.thread_mode == pcre_mod._THREAD_MODE_ENABLED + explicitly_disabled = pcre.compile("flag-enabled", flags=int(pcre.Flag.NO_THREADS)) + assert explicitly_disabled is not enabled + assert explicitly_disabled._pattern is enabled._pattern + assert explicitly_disabled.thread_mode == pcre_mod._THREAD_MODE_DISABLED + assert enabled.thread_mode == pcre_mod._THREAD_MODE_ENABLED + + original_limit = cache_mod.get_cache_limit() + try: + cache_mod.set_cache_limit(1) + pcre.clear_cache() + pcre.compile("flag-one", flags=flags) + pcre.compile("flag-two", flags=flags) + + cache_mod.set_cache_limit(0) + pcre.clear_cache() + uncached = pcre.compile("flag-cache-disabled", flags=flags) + assert pcre.compile("flag-cache-disabled", flags=flags) is not uncached + finally: + cache_mod.set_cache_limit(original_limit) + pcre.clear_cache() + + +def test_module_helpers_accept_project_flag_enum_zero() -> None: + assert pcre.match("x", "x", pcre.Flag(0)) is not None + assert pcre.findall("x", "xx", pcre.Flag(0)) == ["x", "x"] + assert pcre.split("x", "xx", flags=pcre.Flag(0)) == ["", "", ""] + assert pcre.sub("x", "y", "xx", flags=pcre.Flag(0)) == "yy" + + +def test_replacement_fallbacks_cover_subclasses_and_bounded_templates() -> None: + class DerivedPattern(pcre_mod.Pattern): + pass + + backend = pcre_mod._pcre2.compile("(x)", jit=False) + derived = DerivedPattern(backend) + assert derived.sub(r"[\1]", "x") == "[x]" + assert derived.sub(r"[\1]", "x", count=1) == "[x]" + + +def test_module_fast_dispatch_falls_back_for_legacy_backends( + monkeypatch: pytest.MonkeyPatch, +) -> None: + class LegacyPattern: + _is_c_pattern = True + _pattern = object() + + def match(self, value: Any) -> str: + return f"match:{value}" + + def search(self, value: Any) -> str: + return f"search:{value}" + + def fullmatch(self, value: Any) -> str: + return f"full:{value}" + + def finditer(self, value: Any) -> list[str]: + return [f"iter:{value}"] + + def findall(self, value: Any) -> list[str]: + return [f"all:{value}"] + + def split(self, value: Any, *, maxsplit: int) -> list[Any]: + return [value, maxsplit] + + def subn(self, repl: Any, value: Any, *, count: int) -> tuple[Any, int]: + return (f"sub:{repl}:{value}", count) + + legacy = LegacyPattern() + monkeypatch.setattr(pcre_mod, "_module_compile", lambda *args: legacy) + assert pcre_mod.match("x", "a") == "match:a" + assert pcre_mod.search("x", "a") == "search:a" + assert pcre_mod.fullmatch("x", "a") == "full:a" + assert pcre_mod.finditer("x", "a") == ["iter:a"] + assert pcre_mod.findall("x", "a") == ["all:a"] + assert pcre_mod.split("x", "a") == ["a", 0] + assert pcre_mod.subn("x", "r", "a") == ("sub:r:a", 0) + + +class _LegacyFastDispatchPattern: + pattern = "x" + groupindex: ClassVar[dict[str, int]] = {"g": 1} + flags = 0 + capture_count = 1 + jit = False + + def match(self, *args: Any, **kwargs: Any) -> None: + return None + + search = match + fullmatch = match + + def finditer(self, *args: Any, **kwargs: Any) -> Any: + return iter(()) + + def findall(self, *args: Any, **kwargs: Any) -> list[Any]: + return [] + + def substitute(self, *args: Any, **kwargs: Any) -> tuple[str, int]: + return ("", 0) + + +def test_legacy_c_dispatch_fallbacks(monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setattr(pcre_mod, "_CPattern", _LegacyFastDispatchPattern) + pattern = pcre_mod.Pattern(_LegacyFastDispatchPattern()) + assert pattern.match("x") is None + assert pattern.search("x") is None + assert pattern.fullmatch("x") is None + assert list(pattern.finditer("x")) == [] + assert pattern.findall("x") == [] + assert pattern.subn("literal", "x") == ("", 0) + assert pattern.subn(r"\g", "x") == ("", 0) + + type_error_backend = _LegacyFastDispatchPattern() + + def reject_keyword_call(*args: Any, **kwargs: Any) -> None: + raise TypeError("legacy substitute signature") + + type_error_backend.substitute = reject_keyword_call # type: ignore[method-assign] + assert pcre_mod.Pattern(type_error_backend).subn("literal", "x") == ("x", 0) + + def test_package_import_without_optional_simd_export() -> None: original = pcre_ext_c._cpu_ascii_vector_mode try: @@ -55,41 +464,6 @@ def test_cache_import_honours_global_environment( importlib.reload(cache_mod) -class _ChangingZeroComparison: - """Behave like a concurrently changed internal cache limit.""" - - def __init__(self) -> None: - self.comparisons = 0 - - def __eq__(self, other: object) -> bool: - assert other == 0 - self.comparisons += 1 - return self.comparisons > 1 - - def __ne__(self, other: object) -> bool: - return not self == other - - -def test_thread_cache_limit_rechecked_before_store( - monkeypatch: pytest.MonkeyPatch, -) -> None: - state = cache_mod._THREAD_LOCAL - original_limit = state.cache_limit - original_cache = state.pattern_cache - state.cache_limit = _ChangingZeroComparison() # type: ignore[assignment] - state.pattern_cache = {} - monkeypatch.setattr(cache_mod._pcre2, "compile", lambda *args, **kwargs: "compiled") - try: - result = cache_mod._cached_compile_thread_local( - "pattern", 0, lambda value: value, jit=False - ) - assert result == "compiled" - assert state.pattern_cache == {} - finally: - state.cache_limit = original_limit - state.pattern_cache = original_cache - - class _SecondLookupMapping(dict[Any, Any]): def __init__( self, second_result: Any = None, *, second_raises: bool = False @@ -225,12 +599,14 @@ def split(self, subject: str, maxsplit: int = 0) -> list[str]: def finditer( self, subject: str, - *, + *args: Any, pos: int = 0, endpos: int = -1, options: int = 0, owner: Any = None, ) -> Any: + if args: + pos, endpos, options, owner = args del options, owner resolved_end = len(subject) if endpos < 0 else endpos return re.compile(self.pattern).finditer(subject, pos, resolved_end) @@ -248,11 +624,13 @@ def __getattribute__(self, name: str) -> Any: def test_split_legacy_backend_fallback_and_limit() -> None: pattern = pcre_mod.Pattern(_LegacySplitPattern()) # type: ignore[arg-type] + pattern._is_c_pattern = True assert pattern.split("a,b,c") == ["a", ",", "b", ",", "c"] assert pattern.split("a,b,c", maxsplit=1) == ["a", ",", "b,c"] no_split = pcre_mod.Pattern(_LegacyNoSplitPattern()) # type: ignore[arg-type] assert no_split.split("a,b") == ["a", "b"] + assert no_split.split("a,b", maxsplit=-1) == ["a,b"] @pytest.mark.parametrize( @@ -268,6 +646,70 @@ def test_empty_pattern_split_fast_path_matches_re( subject, maxsplit=1 ) + class Zero(int): + pass + + assert compiled.split(subject, maxsplit=Zero(0)) == re.compile(pattern).split( + subject + ) + + +def test_c_split_default_dispatch_preserves_results() -> None: + pattern = pcre.compile(r"\s+") + assert pattern.split("a b c") == ["a", "b", "c"] + bytes_pattern = pcre.compile(rb"\s+") + assert bytes_pattern.split(b"a b c") == [b"a", b"b", b"c"] + + +def test_c_literal_split_uses_builtin_only_for_safe_shape() -> None: + text_pattern = pcre.compile(" ") + assert text_pattern.split("a b c") == ["a", "b", "c"] + assert text_pattern.split("a b c", maxsplit=1) == ["a", "b c"] + assert text_pattern.split("a b c", maxsplit=-1) == ["a b c"] + + multi_pattern = pcre.compile(", ") + assert multi_pattern.split("a, b, c") == ["a", "b", "c"] + + bytes_pattern = pcre.compile(b",") + assert bytes_pattern.split(b"a,b,c") == [b"a", b"b", b"c"] + + # Regex metacharacters and non-default compile flags must retain PCRE2. + assert pcre.compile(".").split("a.b") == ["", "", "", ""] + assert pcre.compile("x", flags=pcre.Flag.CASELESS).split("Xx") == ["", "", ""] + + +def test_c_literal_subn_default_dispatch_preserves_results() -> None: + pattern = pcre.compile(r"\w+") + assert pattern.subn("X", "a b") == ("X X", 2) + bytes_pattern = pcre.compile(rb"\w+") + assert bytes_pattern.subn(b"X", b"a b") == (b"X X", 2) + + +def test_c_plain_literal_subn_uses_builtin_only_for_safe_shape() -> None: + pattern = pcre.compile("foo") + assert pattern.subn("bar", "foo foo") == ("bar bar", 2) + assert pattern.subn("bar", "foo foo", count=1) == ("bar foo", 1) + assert pattern.subn("bar", "foo foo", count=-1) == ("foo foo", 0) + assert pattern.subn("bar", "no match") == ("no match", 0) + assert pattern.subn("bar", "fxx") == ("fxx", 0) + + bytes_pattern = pcre.compile(b"foo") + assert bytes_pattern.subn(b"bar", b"foo foo") == (b"bar bar", 2) + + # Regex metacharacters and escaped replacement syntax stay on PCRE2. + assert pcre.compile(".").subn("X", "ab") == ("XX", 2) + assert pcre.compile("foo").subn(r"\\g<0>", "foo") == (r"\g<0>", 1) + + +def test_c_plain_literal_findall_uses_builtin_count_only_for_safe_shape() -> None: + pattern = pcre.compile("foo") + assert pattern.findall("foo foo") == ["foo", "foo"] + assert pattern.findall("no match") == [] + assert pattern.findall("foo foo", pos=1) == ["foo"] + + bytes_pattern = pcre.compile(b"foo") + assert bytes_pattern.findall(b"foo foo") == [b"foo", b"foo"] + def test_compile_existing_pattern_slow_path_and_jit_guards( monkeypatch: pytest.MonkeyPatch, diff --git a/tests/test_threaded_backend.py b/tests/test_threaded_backend.py index ae9f2c4..b584f40 100644 --- a/tests/test_threaded_backend.py +++ b/tests/test_threaded_backend.py @@ -46,7 +46,9 @@ def test_parallel_map_auto_default_threads_above_threshold(self): subjects = ["a" * 70_000, "b" * 70_000] with mock.patch("pcre.pcre.ensure_thread_pool") as ensure_mock: - ensure_mock.side_effect = lambda *args, **kwargs: thread_utils.ensure_thread_pool(*args, **kwargs) + ensure_mock.side_effect = lambda *args, **kwargs: ( + thread_utils.ensure_thread_pool(*args, **kwargs) + ) results = pcre.parallel_map(pattern, subjects) self.assertTrue(ensure_mock.called) @@ -65,11 +67,29 @@ def test_parallel_map_with_flag(self): self.assertEqual([match.group(0) for match in results], subjects) self.assertTrue(pattern.use_threads) + def test_parallel_map_batches_work_without_changing_order(self): + pattern = pcre.compile(r"\w+", Flag.THREADS) + subjects = [word * 2_000 for word in ("alpha", "beta", "gamma", "delta", "epsilon")] + executor = thread_utils.ensure_thread_pool(2) + with ( + mock.patch("pcre.pcre.ensure_thread_pool", return_value=executor), + mock.patch("pcre.pcre.get_thread_pool_size", return_value=2), + mock.patch.object(executor, "submit", wraps=executor.submit) as submit, + ): + results = pcre.parallel_map(pattern, subjects, method="search") + + self.assertEqual([match.group(0) for match in results], subjects) + # Five inputs on two workers are split into three ordered batches. + self.assertEqual(submit.call_count, 3) + def test_pattern_parallel_map(self): pattern = pcre.compile(r"\d+", Flag.THREADS) - subjects = ["1", "22", "nope"] + subjects = ["1", "22", "nope", "4444", "5", "66", "nope", "7777", "8"] results = pattern.parallel_map(subjects, method="findall") - self.assertEqual(results, [["1"], ["22"], []]) + self.assertEqual( + results, + [["1"], ["22"], [], ["4444"], ["5"], ["66"], [], ["7777"], ["8"]], + ) def test_compile_existing_pattern_toggle(self): pattern = pcre.compile(r"foo") @@ -93,7 +113,9 @@ def test_benchmark_non_threaded_baseline(self): subjects = ["bench"] * 32 sequential_pattern = pcre.compile(r"bench") start_seq = time.perf_counter() - sequential_results = [sequential_pattern.search(subject) for subject in subjects] + sequential_results = [ + sequential_pattern.search(subject) for subject in subjects + ] seq_elapsed = time.perf_counter() - start_seq threaded_pattern = pcre.compile(r"bench", Flag.THREADS) diff --git a/tests/test_threads.py b/tests/test_threads.py index d808ae5..3d6adb2 100644 --- a/tests/test_threads.py +++ b/tests/test_threads.py @@ -11,7 +11,9 @@ import pytest -def test_threading_supported_false_on_low_core_count(monkeypatch: pytest.MonkeyPatch) -> None: +def test_threading_supported_false_on_low_core_count( + monkeypatch: pytest.MonkeyPatch, +) -> None: monkeypatch.setattr(threads_mod, "_cpu_total", lambda: 4) assert threads_mod.threading_supported() is False @@ -27,10 +29,15 @@ def test_configure_threads_toggles_default(monkeypatch: pytest.MonkeyPatch) -> N threads_mod.configure_threads(enabled=original) -def test_configure_threads_threshold_validation(monkeypatch: pytest.MonkeyPatch) -> None: +def test_configure_threads_threshold_validation( + monkeypatch: pytest.MonkeyPatch, +) -> None: original = threads_mod.get_auto_threshold() try: - assert threads_mod.configure_threads(threshold=1234) == threads_mod.get_thread_default() + assert ( + threads_mod.configure_threads(threshold=1234) + == threads_mod.get_thread_default() + ) assert threads_mod.get_auto_threshold() == 1234 with pytest.raises(ValueError): @@ -88,12 +95,62 @@ def test_shutdown_thread_pool_is_idempotent() -> None: threads_mod.shutdown_thread_pool(wait=False) -def test_max_threads_zero_when_threading_unsupported(monkeypatch: pytest.MonkeyPatch) -> None: +def test_max_threads_zero_when_threading_unsupported( + monkeypatch: pytest.MonkeyPatch, +) -> None: monkeypatch.setattr(threads_mod, "_cpu_total", lambda: 4) assert threads_mod._max_threads() == 0 -def test_determine_worker_count_requires_supported_backend(monkeypatch: pytest.MonkeyPatch) -> None: +def test_max_threads_prefers_performance_cluster( + monkeypatch: pytest.MonkeyPatch, +) -> None: + monkeypatch.setattr(threads_mod, "_cpu_total", lambda: 16) + monkeypatch.setattr(threads_mod, "_performance_cpu_total", lambda: 3) + assert threads_mod._max_threads() == 3 + + +def test_max_threads_falls_back_to_fraction_without_cluster_info( + monkeypatch: pytest.MonkeyPatch, +) -> None: + monkeypatch.setattr(threads_mod, "_cpu_total", lambda: 16) + monkeypatch.setattr(threads_mod, "_performance_cpu_total", lambda: 0) + assert threads_mod._max_threads() == 4 + + +def test_performance_cpu_count_falls_back_when_sysctl_is_unavailable( + monkeypatch: pytest.MonkeyPatch, +) -> None: + original = threads_mod._PERFORMANCE_CPU_TOTAL + try: + threads_mod._PERFORMANCE_CPU_TOTAL = None + monkeypatch.setattr(threads_mod.sys, "platform", "darwin") + monkeypatch.setattr( + threads_mod.subprocess, + "check_output", + lambda *args, **kwargs: (_ for _ in ()).throw(OSError("sysctl")), + ) + assert threads_mod._performance_cpu_total() == 0 + assert threads_mod._performance_cpu_total() == 0 + finally: + threads_mod._PERFORMANCE_CPU_TOTAL = original + + +def test_performance_cpu_count_is_zero_outside_macos( + monkeypatch: pytest.MonkeyPatch, +) -> None: + original = threads_mod._PERFORMANCE_CPU_TOTAL + try: + threads_mod._PERFORMANCE_CPU_TOTAL = None + monkeypatch.setattr(threads_mod.sys, "platform", "linux") + assert threads_mod._performance_cpu_total() == 0 + finally: + threads_mod._PERFORMANCE_CPU_TOTAL = original + + +def test_determine_worker_count_requires_supported_backend( + monkeypatch: pytest.MonkeyPatch, +) -> None: monkeypatch.setattr(threads_mod, "_max_threads", lambda: 0) with pytest.raises(RuntimeError, match="at least 8 CPU cores"): threads_mod._determine_worker_count(None)