Merge pull request #970 from fglock/feature/joni-dynamic-patterns

feat(regex): execute dynamic patterns through Joni

Author: fglock

build.gradle

@@ -84,7 +84,7 @@ cyclonedxBom {
     outputFormat = "all"  // Generate both JSON and XML
     componentName = "perlonjava"
     componentVersion = project.version
-    organizationalEntity { oe ->
+    organizationalEntity = { oe ->
         oe.name = "PerlOnJava Project"
         oe.urls = ["https://github.com/fglock/PerlOnJava"]
     }
@@ -260,6 +260,10 @@ tasks.withType(JavaCompile).configureEach {
 // Test execution configuration with native access and adequate heap
 tasks.withType(Test).configureEach {
     jvmArgs += '--enable-native-access=ALL-UNNAMED'
+    // Netty still uses the transitional sun.misc.Unsafe memory API. PerlOnJava
+    // requires Java 24, where explicitly allowing it suppresses the terminal
+    // deprecation banner while retaining the tested Netty allocation path.
+    jvmArgs += '--sun-misc-unsafe-memory-access=allow'
     maxHeapSize = '1g'
 }
 

docs/design/joni-callout-fork.md

@@ -28,10 +28,11 @@ commit messages, not here.
 
 ## Internal callout syntax
 
-The structured Perl regex frontend emits `(?{=CALL:<id>})` only in an
-engine-facing skeleton. `<id>` is a non-negative decimal index into the callback
-table owned by that regex value. The fork parses this representation into a
-dedicated callout node; it never parses Perl source.
+The structured Perl regex frontend emits `(?{=CALL:<id>})` for plain and
+conditional callbacks and `(?{=DYNAMIC:<id>})` for dynamic subprograms only in
+an engine-facing skeleton. `<id>` is a non-negative decimal index into the
+callback table owned by that regex value. The fork parses these representations
+into dedicated callout nodes; it never parses Perl source.
 
 Runtime-interpolated pattern text must not be promoted to trusted skeleton
 syntax. The PerlOnJava frontend remains responsible for preserving source
@@ -44,10 +45,17 @@ The fork exposes a runtime-neutral API in `org.joni`:
 ```java
 interface CalloutHandler {
     CalloutResult execute(int calloutId, MatchView match);
+    default DynamicPatternResult executeDynamic(int calloutId, MatchView match);
     void unwind(Object backtrackToken);
     default void complete(Object successfulToken) { unwind(successfulToken); }
 }
 
+final class DynamicPatternResult {
+    Regex getRegex();
+    CalloutHandler getCalloutHandler();
+    Object getBacktrackToken();
+}
+
 interface MatchView {
     int currentBytePosition();
     int captureCount();
@@ -64,6 +72,11 @@ state.
 token. The engine neither interprets that token nor depends on PerlOnJava
 runtime classes.
 
+`DynamicPatternResult` supplies a compiled nested program, its matcher-local
+handler, and the token for the dynamic expression's provisional Perl state. The
+default `executeDynamic` implementation fails fast, so handlers that only use
+plain callouts remain source compatible.
+
 ## Execution and unwind contract
 
 1. The callout opcode invokes the handler with the callout ID and a read-only
@@ -78,12 +91,20 @@ runtime classes.
    still-active token in reverse execution order.
 6. A `FAIL` result enters normal matcher backtracking after installing the frame,
    so cleanup follows the same path as later-pattern failure.
-7. Patterns without callout nodes allocate no handler state or callout frames.
+7. A dynamic callout resolves its nested program only when execution reaches
+   the opcode. The nested matcher yields one result at a time; an outer failure
+   resumes the nested matcher at its next alternative before the outer engine
+   backtracks past the dynamic frame.
+8. Nested captures remain private to the nested matcher. Its endpoint advances
+   the outer input position, while outer capture numbering and match variables
+   remain unchanged.
+9. Completing or abandoning a nested continuation propagates completion or
+   unwind to all of its active callback tokens exactly once.
+10. Patterns without callout nodes allocate no handler state or callout frames.
 
 Callback conditions use an internal conditional-callout opcode. `CONTINUE`
 selects the yes branch and `FAIL` selects the no branch without treating the
-condition itself as a failed match. Dynamic nested patterns remain outside this
-contract until their alternatives can participate in outer backtracking.
+condition itself as a failed match.
 
 ## Structured Perl frontend
 
@@ -106,10 +127,11 @@ retaining the last successful plain callback result.
 
 Every structured executable callback template selects Joni, including a
 callback whose body happens to return a constant: `(?{ 1 })` still has match-time
-side effects and unwind semantics. Constant dynamic-pattern expressions
-`(??{ ... })` may retain the existing compile-time fold path; a runtime-dependent
-dynamic-pattern closure selects Joni only after the nested-pattern backtracking
-contract is implemented.
+side effects and unwind semantics. Runtime-dependent dynamic-pattern expressions
+also select Joni and execute only when their opcode is reached. Semantically safe
+constant expressions may use the compile-time fold path; constants with captures
+or top-level alternatives remain dynamic so they cannot change outer grouping or
+capture numbers.
 
 Literal match targets have one stable scalar identity per compiled call site so
 `pos()` and `/g` survive repeated loop execution without allowing two identical
@@ -170,8 +192,9 @@ branch reset, subpattern calls, conditions, extended Unicode and graphemes,
 embedded code, regex debugging, runtime eval, lexical default flags, and named
 capture behavior. As each feature becomes executable, patterns containing it
 select Joni. The current selector routes declarative subpattern calls and every
-structured executable callback template; constant `(??{...})` expressions keep
-their established fold-to-pattern path.
+structured executable callback template, including runtime `(??{...})`.
+Semantically safe constant dynamic expressions may keep their established
+fold-to-pattern path.
 
 ## Implementation stages
 
@@ -185,8 +208,8 @@ their established fold-to-pattern path.
 6. Publish provisional match state for plain callbacks and preserve `$^R` across
    successful completion and backtracking.
 7. Add dynamic-local checkpoints and callback conditions.
-8. Add dynamic nested patterns only after focused differential tests establish
-   their outer-backtracking contract.
+8. Resolve dynamic nested programs at match time and preserve their alternatives
+   as resumable matcher continuations on the outer backtracking stack.
 
 ## Verification
 
@@ -199,8 +222,11 @@ their established fold-to-pattern path.
   both relocated trees, and includes all required notices.
 - An embedding smoke test can load stock Joni and PerlOnJava together.
 - Full `make` and the focused direct/thread regex matrix pass.
-- The executable-callback unit test passes under standard Perl and both
-  PerlOnJava execution backends.
+- The executable-callback and dynamic-pattern unit tests pass under standard
+  Perl and both PerlOnJava execution backends.
+- Dynamic-pattern tests cover delayed expression evaluation, returned strings
+  and `qr//` values, private nested captures, outer-suffix backtracking, nested
+  callback cleanup, and recursive `qr//` values.
 - `dev/tools/perl_test_runner.pl perl5_t/t/re/` is compared file-by-file with
   `../PerlOnJava/logs/test_20260815_080000_958.log`; no previously passing regex
   file may regress, and changed pass counts or blocked-test totals are reported.

docs/reference/feature-matrix.md

@@ -388,21 +388,20 @@ my @copy = @{$z};         # ERROR
 ### Missing Regular Expression Features
 
 - ❌  **Dynamically-scoped regex variables**: Regex variables are not dynamically-scoped.
-- 🟡  **Recursive Patterns**: `(?R)` and `(?0)` execute through Joni. Constant `(??{ code })` expressions fold to a pattern; runtime-dependent dynamic patterns remain unsupported.
-- 🟡  **Backtracking Control Verbs and Definitions**: `(?(DEFINE)...)` and named recursive definitions execute through Joni. Control verbs such as `(*PRUNE)`, `(*SKIP)`, `(*THEN)`, and `(*COMMIT)` are not yet complete. Atomic groups `(?>...)` are supported as noted above.
+- 🟡  **Recursive and Dynamic Patterns**: `(?R)`, `(?0)`, and runtime `(??{ code })` execute through Joni. Dynamic expressions may return strings or `qr//` values, and nested alternatives participate in outer backtracking without changing outer grouping or capture numbering. Deep dynamic recursion still needs an engine-owned depth limit that does not depend on the Java call stack.
+- ❌  **Backtracking Control Verbs**: `(*ACCEPT)`, `(*PRUNE)`, `(*SKIP)`, `(*THEN)`, and `(*COMMIT)` are not supported. `(*FAIL)`/`(*F)` and atomic groups `(?>...)` are supported.
+- ✅  **Regex Definitions**: `(?(DEFINE)...)` containers and numbered or named calls to their subpatterns execute through Joni.
 - ❌  **Lookbehind Assertions**: Variable-length negative or positive lookbehind assertions, e.g., `(?<=...)` or `(?<!...)`, are not supported.
-- ❌  **Branch Reset Groups**: Use of `(?|...)` to reset group numbering across branches is not supported.
+- ✅  **Branch Reset Groups**: `(?|...)` resets capture numbering across alternatives and preserves mapped match variables.
 - ✅  **Advanced Subroutine Calls**: Sub-pattern calls with numbered or named references like `(?1)` and `(?&name)` execute through Joni.
 - 🟡  **Conditional Expressions**: Executable callback conditions `(?(?{ code })yes|no)` execute through Joni; other condition forms are not yet complete.
 - ❌  **Extended Unicode Regex Features**: Some extended Unicode regex functionalities are not supported.
 - ❌  **Extended Grapheme Clusters**: Matching with `\X` for extended grapheme clusters is not supported.
-- 🟡  **Embedded Code in Regex**: `(?{ code })` and executable callback conditions run as lexical closures in Joni with provisional captures, `$^R`, and backtracking unwind. Runtime-dependent `(??{ code })` remains unsupported.
-- 🟡  **Regex Debugging**: Lexical `use/no re 'debug'` and `debugcolor` state is runtime-owned and produces a stable compile/execution trace on both backends and in child threads. Exact Perl optimizer opcodes and diagnostic wording are not complete.
-- ❌  **Regex Optimizations**: Using `use re 'eval';` for runtime regex compilation is not supported.
+- 🟡  **Embedded Code in Regex**: `(?{ code })`, executable callback conditions, and `(??{ code })` run as lexical closures in Joni with provisional captures and backtracking unwind. Optimistic callbacks `(*{ code })` are not supported.
+- ✅  **Regex Debugging**: Lexically scoped `use/no re 'debug'` and `debugcolor` are supported, including runtime snapshot ownership.
+- 🟡  **Runtime Regex Evaluation**: `use re 'eval'` controls whether interpolated patterns containing eval groups may compile, but runtime strings cannot manufacture trusted callback closures.
 - ❌  **Regex Compilation Flags**: Setting default regex flags with `use re '/flags';` is not supported.
-- ❌  **Stricter named captures**
-  - ❌  **No underscore in named captures** `(?<test_field>test)` the name in named captures cannot have underscores.
-  - ❌  **No duplicate named capture groups**: In Java regular expression engine, each named capturing group must have a unique name within a regular expression.
+- ✅  **Perl Named Captures**: Names may contain underscores, and duplicate named groups preserve Perl-style `%+`/`%-` and backreference behavior.
 
 
 ## Statements and Special Operators

src/main/java/org/perlonjava/frontend/parser/StringSegmentParser.java

@@ -8,7 +8,6 @@
 import org.perlonjava.frontend.lexer.LexerToken;
 import org.perlonjava.frontend.lexer.LexerTokenType;
 import org.perlonjava.runtime.operators.PerlUtfString;
-import org.perlonjava.runtime.regex.RegexMarkers;
 import org.perlonjava.runtime.regex.UnicodeResolver;
 import org.perlonjava.runtime.runtimetypes.PerlCompilerException;
 import org.perlonjava.runtime.runtimetypes.RuntimeScalar;
@@ -919,15 +918,18 @@ private void parseRegexCodeBlock(boolean isRecursive) {
         TokenUtils.consume(parser, LexerTokenType.OPERATOR, ")");
 
         if (isRecursive) {
-            // Preserve the existing constant (??{...}) support. Runtime-dependent
-            // nested patterns remain deferred to Stage 36.5.
+            // Keep constant folding as the zero-overhead path, but preserve a
+            // runtime-dependent expression as a lexical dynamic-program closure.
             Node folded = ConstantFoldingVisitor.foldConstants(block);
             if (folded instanceof BlockNode blockNode && blockNode.elements.size() == 1) {
                 folded = blockNode.elements.getFirst();
             }
             RuntimeScalar constant = ConstantFoldingVisitor.getConstantValue(folded);
-            segments.add(new StringNode(constant == null
-                    ? RegexMarkers.RECURSIVE_PATTERN : constant.toString(), savedTokenIndex));
+            if (constant == null || !isSafeDynamicConstantFold(constant.toString())) {
+                segments.add(regexCallback(block, "DYNAMIC", savedTokenIndex));
+            } else {
+                segments.add(new StringNode(constant.toString(), savedTokenIndex));
+            }
         } else {
             segments.add(regexCallback(block, "BLOCK", savedTokenIndex));
         }
@@ -941,6 +943,33 @@ private Node regexCallback(Node block, String kind, int index) {
         return callback;
     }
 
+    /**
+     * Folding is safe only when textual insertion preserves the nested program's
+     * grouping and capture isolation. A top-level alternative could absorb the
+     * outer suffix, and a capturing group could consume an outer capture number.
+     */
+    private static boolean isSafeDynamicConstantFold(String pattern) {
+        boolean escaped = false;
+        boolean inClass = false;
+        for (int i = 0; i < pattern.length(); i++) {
+            char ch = pattern.charAt(i);
+            if (escaped) {
+                escaped = false;
+            } else if (ch == '\\') {
+                escaped = true;
+            } else if (ch == '[') {
+                inClass = true;
+            } else if (ch == ']' && inClass) {
+                inClass = false;
+            } else if (ch == '(' && !inClass) {
+                return false;
+            } else if (ch == '|' && !inClass) {
+                return false;
+            }
+        }
+        return true;
+    }
+
     /**
      * Gets a string context around the specified position for error reporting.
      * This shows the actual string content around where the error occurred.