Complete Perl regex parity with Joni: source semantics - #999
Merged
Conversation
Replace the Phase 36 design with the forward-looking full-parity plan, add durable baseline comparison and resource-aware regex test scheduling, and add a temporary forced-backend seam for differential testing. Move Perl option parsing and several ordinary-pattern translations to the Joni boundary. Extend the vendored fork with named control-verb operands, MARK bytecode and backtracking state, named SKIP targets, and Perl-visible REGMARK and REGERROR state while retaining all upstream copyright notices. Generated with [Codex](https://openai.com/codex/) Co-Authored-By: Codex <codex@openai.com>
Expose every capture number in a Joni duplicate-name entry through the backend-neutral matcher so Perl's %- arrays retain unmatched placeholders and the selected branch across matches and substitutions. Generated with [Codex](https://openai.com/codex/) Co-Authored-By: Codex <codex@openai.com>
Teach the vendored Joni fork Perl bare property escapes, Unicode octal and hex escapes, delimiter-free numeric g backreferences, and whitespace inside interval quantifiers. This completes the forced-Joni required literal gate. Generated with [Codex](https://openai.com/codex/) Co-Authored-By: Codex <codex@openai.com>
Enable Joni's existing possessive interval capability in the PerlOnJava syntax
so Perl ranges such as {2,4}+ retain their atomic backtracking behavior.
Generated with [Codex](https://openai.com/codex/)
Co-Authored-By: Codex <codex@openai.com>
Mask trusted callback conditions with temporary named-capture conditions while runtime regex source is parsed, then remove the synthetic captures before final Joni compilation. This keeps mixed dynamic source and callback branches intact. Generated with [Codex](https://openai.com/codex/) Co-Authored-By: Codex <codex@openai.com>
Resolve user-defined Unicode properties in PerlOnJava's source-policy layer and pass character classes to Joni. Preserve forward references by carrying the existing first-use recompilation signal through the Joni backend. Generated with [Codex](https://openai.com/codex/) Co-Authored-By: Codex <codex@openai.com>
Resolve Script_Extensions through the bundled Unicode layer and teach the vendored Joni compiler to emit exact alternatives for finite quantified positive and negative lookbehind up to Perl's 255-character bound. Generated with [Codex](https://openai.com/codex/) Co-Authored-By: Codex <codex@openai.com>
Strip ANSI terminal control sequences before classifying TAP lines so lexical re debugcolor output cannot turn a passing top-level assertion into a false incomplete result. Generated with [Codex](https://openai.com/codex/) Co-Authored-By: Codex <codex@openai.com>
Remove blanket mutation snapshots from Joni callouts so ordinary assignments survive whole-match failure and callback exceptions, while dynamic locals, provisional regex state, and callback result state still unwind separately. Generated with [Codex](https://openai.com/codex/) Co-Authored-By: Codex <codex@openai.com>
Compile assertion conditions with capture-preserving stack state and track semantic subpattern-call frames for (R), (R1), and (R&name) conditions. Route recursion conditions directly to Joni and cover both execution backends. Generated with [Codex](https://openai.com/codex/) Co-Authored-By: Codex <codex@openai.com>
Analyze finite compound lookbehind ranges up to Perl's 255-character bound and enforce the original endpoint while trying each candidate length. Route all lookbehind syntax through Joni and cover positive and negative forms on both execution backends. Generated with [Codex](https://openai.com/codex/) Co-Authored-By: Codex <codex@openai.com>
Add fork-level regression coverage for assertion and recursion conditions, capture-preserving branch selection, and finite compound lookbehind endpoints. Retain the original Joni MIT notice on every added or modified test source. Generated with [Codex](https://openai.com/codex/) Co-Authored-By: Codex <codex@openai.com>
Continue at the following character when a named SKIP resolves to the current search start, matching Perl's repeated-search progression. Cover multi-path SKIP, MARK captures, named COMMIT errors, and THEN behavior on both backends. Generated with [Codex](https://openai.com/codex/) Co-Authored-By: Codex <codex@openai.com>
Separate exact Age semantics from cumulative In and Present_In aliases using the Perl 5.44 Unicode release sequence. Support loose and V-prefixed version spellings and verify matching on both execution backends. Generated with [Codex](https://openai.com/codex/) Co-Authored-By: Codex <codex@openai.com>
Preserve Script, Block, and Blk property/value pairs for ICU and translate them, script extensions, and age aliases before Joni parsing. Update the feature matrix and parity plan with the completed condition and Unicode work. Generated with [Codex](https://openai.com/codex/) Co-Authored-By: Codex <codex@openai.com>
Reset capture allocation for each top-level (?|...) branch and retain the maximum branch allocation so matcher regions expose Perl-native numbering. Cover optional and higher captures on JVM and interpreter backends. Generated with [Codex](https://openai.com/codex/) Co-Authored-By: Codex <codex@openai.com>
Use the backend-neutral matcher for split separators, captures, leading whitespace handling, and zero-width consuming retries. This removes the Java placeholder dependency when Joni is selected and preserves scalar/list split semantics on both execution backends. Update the full-parity plan to repeat the Joni-default whole-suite gate after the known routing blockers were closed. Generated with [Codex](https://openai.com/codex/) Co-Authored-By: Codex <codex@openai.com>
Honor match-time FIND_NOT_EMPTY inside the vendored Joni bytecode machine so global substitution can backtrack to a consuming alternative at the same offset. Add an upstream-style regression test with the original Joni license. Keep named-character escapes literal under extended mode and consume internal surrogate scalar markers as one Perl character in negated classes. Generated with [Codex](https://openai.com/codex/) Co-Authored-By: Codex <codex@openai.com>
Teach the vendored Joni Perl option parser to execute (?n), scoped (?n:...), negative toggles, and (?^) using native capture-allocation state. Preserve the original Joni license in focused parser tests. Quote literal opening brackets inside Perl character classes so RFC-style classes with unbraced hex ranges compile identically on the Joni backend. Generated with [Codex](https://openai.com/codex/) Co-Authored-By: Codex <codex@openai.com>
Translate Java-style Is and gc property spellings emitted inside composite Unicode classes to Joni's native aliases. This keeps built-in Perl properties such as IsWord out of the user-property fallback and preserves their behavior on both execution backends. Generated with [Codex](https://openai.com/codex/) Co-Authored-By: Codex <codex@openai.com>
Add a native Joni option for Perl /aa semantics. Reject ASCII/non-ASCII fold crossings in literals and classes, suppress multi-character folding, and keep the original identity of Unicode simple folds that cross into ASCII. Cover the fork behavior with an upstream-style test retaining Joni's original license and verify full-fold and crossing cases on both PerlOnJava backends. Generated with [Codex](https://openai.com/codex/) Co-Authored-By: Codex <codex@openai.com>
Select Joni for ordinary patterns unless explicit Java differential mode is requested. Compile byte and Unicode Joni variants so default /d character classes follow the target scalar, and preserve Perl POSIX-class nesting, leading class syntax, and compile-time malformed-class diagnostics. Update the full-parity design to reflect the permanent default while retaining the temporary Java selector for corpus comparison. Generated with [Codex](https://openai.com/codex/) Co-Authored-By: Codex <codex@openai.com>
Parse nested /aa and Unicode fold policy changes as native Joni options, carry those options through matcher backtracking, and preserve the source identity of Unicode characters whose folds cross into ASCII. Remove the equivalent PerlOnJava pattern rewrite so nested modifier restoration remains correct. Document MARK state and Joni-owned capture/fold modifiers in the feature matrix. Update the forward-looking regex parity plan with the current engine boundary. Generated with [Codex](https://openai.com/codex/) Co-Authored-By: Codex <codex@openai.com>
Keep callback and dynamic-program source in stringified qr objects, preserve literal extended-mode comments, and remove the source-destroying dynamic constant fold. Update the full regex parity plan with incremental retirement of the imported pat.t compatibility patch. Generated with [Codex](https://openai.com/codex/) Co-Authored-By: Codex <codex@openai.com>
fglock
marked this pull request as ready for review
August 17, 2026 14:31
fglock
changed the title
Resolve the perl test runner conflict by retaining the weighted scheduler as the default while preserving the explicitly requested dedicated CPU-heavy lane. Keep both runner regression suites passing. Generated with [Codex](https://openai.com/codex/) Co-Authored-By: Codex <codex@openai.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
/n,/aa, branch-reset groups, split semantics, and match-time retry behavior in Jonidev/design/phase36-regex-parity.mdVerification
makeWIP follow-up
perl5_t/t/redifferential gate against the recorded baseline/aa, andre_evalgapsGenerated with Codex