feat(regex): preserve executable callback pseudo-block scope

dev/design/executable-regex-callbacks.md

@@ -2,11 +2,10 @@
 
 ## Status
 
-- **Current phase:** Phase 0 — design and differential baseline
+- **Current phase:** Phase 3 — backtracking and dynamic scope
 - **Started:** 2026-08-09
-- **Implementation status:** Not started
-- **Prerequisite:** PR #895 (`feature/cpan-workaround-cleanup`) or an equivalent
-  backend-neutral `RegexMatcher` integration
+- **Implementation status:** Callout engine and plain callback bridge integrated
+- **Prerequisite:** Satisfied by the namespaced callout-enabled Joni integration
 - **Primary targets:** `(?{ ... })`, `(?(?{ ... })yes|no)`, and `(??{ ... })`
 
 ## Decision Summary
@@ -617,46 +616,36 @@ record the observed output in this document as phases proceed.
 
 ## Progress Tracking
 
-### Current Status: Phase 0 in progress
+### Current Status: Phase 3 in progress
 
 ### Completed Phases
 
-- [ ] Phase 0: Differential semantics and Joni spike
-- [ ] Phase 1: Structured frontend and runtime template
-- [ ] Phase 2: Plain `(?{ ... })`
+- [x] Phase 0: Differential semantics and Joni spike
+- [x] Phase 1: Structured frontend and runtime template
+- [x] Phase 2: Plain `(?{ ... })`
 - [ ] Phase 3: Backtracking and dynamic scope
-- [ ] Phase 4: Callback conditions
+- [x] Phase 4: Callback conditions
 - [ ] Phase 5: `(??{ ... })` dynamic programs
 - [ ] Phase 6: Runtime source, hardening, and policy removal
 
-### Work Completed
-
-- 2026-08-09: Created this design after reviewing the current parser,
-  `RegexPreprocessor`, `RuntimeRegex`, the Joni 2.2.7 API, PR #895's matcher
-  abstraction, and the executable-regex CPAN policies.
-- 2026-08-09: Selected a structured callback template plus generic Joni callout
-  extension as the preferred architecture.
-- 2026-08-09: Corrected the older blanket-side-effect-journaling proposal: Perl
-  dynamic locals require backtracking unwind, but ordinary side effects must not
-  all be reverted.
-
 ### Next Steps
 
-1. Write and validate `regex_executable_callbacks.t` with standard Perl.
-2. Record exact standard-Perl outputs for every open semantic question reachable
-   without implementation.
-3. Create a disposable Joni 2.2.7 callout spike and measure the patch surface.
-4. Decide whether to upstream the generic callout API or publish a namespaced fork.
-5. Begin Phase 1 only after PR #895's matcher abstraction is merged or rebased into
-   the implementation branch.
+1. Preserve lexical regex flags and package metadata for runtime/interpolated
+   executable source.
+2. Close nested callback caller/source-line and interpolated `qr//` `__SUB__`
+   identity gaps.
+3. Add warning-location, interruption, timeout, and nested-exception gates.
+4. Classify tied, magical, shared, and readonly mutation behavior with standard
+   Perl before extending matcher transactions.
+5. Finish dynamic-pattern recursion/caching gates and then remove only the
+   capability policies justified by unchanged-source results.
 
 ### Blockers
 
-- PR #895 is still the integration prerequisite for the planned engine routing.
-- Joni 2.2.7 does not expose an in-match callback extension point; Phase 0 must
-  validate the maintained-fork approach.
-- Several detailed Perl semantics remain intentionally open pending differential
-  tests.
+- Runtime-injected callback source still requires `use re 'eval'` propagation.
+- Recursive callback frames do not yet retain all Perl caller source lines.
+- Tied, magical, shared, and readonly rollback semantics remain intentionally
+  open pending differential tests.
 
 ## Related Documents and Skills
 

dev/design/phase36-regex-parity.md

@@ -242,12 +242,15 @@ timing delta is a regression only after a serialized same-commit reproduction.
 
 The merged Joni dynamic-pattern engine establishes the Stage 36.5 execution
 seam. The current Stage 36.4 core baseline is `rxcode.t` 42/42 and
-`reg_eval_scope.t` 22/49, with no timeout or incomplete file. Matcher-owned
-transactions now restore ordinary scalar, array, and hash mutations when the
-overall match fails, retain mutations from abandoned alternatives when another
-alternative succeeds, and commit successful matches. Callback exceptions also
-restore dynamic locals, provisional match state, and `$^R` on both execution
-backends. Regex stringification no longer exposes private callback IDs.
+`reg_eval_scope.t` 33/49, with no timeout or incomplete file. Matcher-owned
+transactions restore ordinary scalar, array, and hash mutations on total
+failure while retaining Perl's ordinary side effects from attempted paths.
+Callback dynamic locals now transfer from the implementation CV to the matcher:
+they remain visible to later callbacks on the active path, unwind on
+backtracking, and restore after success, failure, or exception. Regex callbacks
+also behave as pseudo-blocks for `caller`, `__SUB__`, and escaping
+`last`/`next`/`goto` on both execution backends. Named unary `scalar` now keeps a
+following match in scalar context, including callback-bearing matches.
 
 ### Completed stages
 
@@ -262,19 +265,25 @@ backends. Regex stringification no longer exposes private callback IDs.
 
 ### Next steps
 
-1. Complete callback lexical pragma, caller-frame, and control-flow isolation
-   exposed by `reg_eval_scope.t`, without changing its thread wrapper.
-2. Extend the callback semantic matrix with interruption, timeout, nested
-   exception paths; require identical JVM/interpreter cleanup.
-3. Define and implement the mutation policy for tied, magical, shared, and
+1. Preserve lexical package and `use re '/flags'` state for `qr//`, interpolated
+   regex objects, and runtime source admitted by `use re 'eval'`.
+2. Extend pseudo-block frame mapping through nested and recursive callbacks;
+   preserve exact caller source lines and enclosing `__SUB__` for interpolated
+   `qr//` values.
+3. Extend the callback semantic matrix with warning locations, interruption,
+   timeout, and nested exception paths; require identical JVM/interpreter cleanup.
+4. Define and implement the mutation policy for tied, magical, shared, and
    readonly values; ordinary values are now transactionally covered.
-4. Complete the merged dynamic-pattern validation gates, then mark Stage 36.5
+5. Complete the merged dynamic-pattern validation gates, then mark Stage 36.5
    complete and proceed to the remaining declarative parity slices.
 
 ### Open blockers
 
-- Callback lexical pragmata, caller frames, and non-local control-flow
-  boundaries still differ from Perl in `reg_eval_scope.t`.
+- Runtime-injected callback source and lexical regex pragmata remain unsupported;
+  these account for tests 4, 5, 8, 10, 11, and 12 in `reg_eval_scope.t`.
+- Recursive/nested callback caller lines, interpolated `qr//` `__SUB__`, warning
+  locations, and the legacy `qr/\(?{` diagnostic account for the remaining
+  Stage 36.4 failures.
 - Tied, magical, shared, and readonly callback mutation rollback remains
   intentionally outside the ordinary-value transaction until its exact Perl
   behavior is established with differential tests.

src/main/java/org/perlonjava/backend/bytecode/BytecodeCompiler.java

@@ -5915,6 +5915,12 @@ private void visitAnonymousSubroutine(SubroutineNode node) {
             subCode.isMapGrepBlock = true;
             subCode.inheritsSelfReference = true;
         }
+        if (node.getBooleanAnnotation("inheritsSelfReference")) {
+            subCode.inheritsSelfReference = true;
+        }
+        if (node.getBooleanAnnotation("regexCallbackPseudoBlock")) {
+            subCode.isRegexCallbackPseudoBlock = true;
+        }
 
         if (RuntimeCode.isDisassemble()) {
             System.out.println(Disassemble.disassemble(subCode));

src/main/java/org/perlonjava/backend/bytecode/BytecodeInterpreter.java

@@ -3003,7 +3003,7 @@ private static RuntimeList execute(SuspendedInterpreterFrame frame) {
                 frame.suspendedDynamicStates =
                         DynamicVariableManager.suspendAbove(savedLocalLevel);
             } else {
-                DynamicVariableManager.popToLocalLevel(savedLocalLevel);
+                DynamicVariableManager.teardownFrameToLocalLevel(savedLocalLevel);
             }
             currentPackageScalar.set(savedPackage);
             if (frame.suspended && !frame.evalCatchStack.isEmpty()) {

src/main/java/org/perlonjava/backend/jvm/EmitBlock.java

@@ -261,7 +261,9 @@ public static void emitBlock(EmitterVisitor emitterVisitor, BlockNode node) {
         }
 
         // Setup 'local' environment if needed
-        Local.localRecord localRecord = Local.localSetup(emitterVisitor.ctx, node, mv, true);
+        Local.localRecord localRecord = node.getBooleanAnnotation("regexCallbackBody")
+                ? new Local.localRecord(false, -1)
+                : Local.localSetup(emitterVisitor.ctx, node, mv, true);
 
         int regexStateLocal = -1;
         if (!node.getBooleanAnnotation("blockIsSubroutine")

src/main/java/org/perlonjava/backend/jvm/EmitSubroutine.java

@@ -488,6 +488,34 @@ public static void emitSubroutine(EmitterContext ctx, SubroutineNode node) {
                     "(Lorg/perlonjava/runtime/runtimetypes/RuntimeScalar;Lorg/perlonjava/runtime/runtimetypes/RuntimeScalar;)V",
                     false);
         }
+        if (node.getBooleanAnnotation("inheritsSelfReference")
+                && !(isMapGrepBlock != null && isMapGrepBlock)) {
+            mv.visitInsn(Opcodes.DUP);
+            mv.visitVarInsn(Opcodes.ALOAD, 0);
+            mv.visitFieldInsn(Opcodes.GETFIELD,
+                    ctx.javaClassInfo.javaClassName,
+                    "__SUB__",
+                    "Lorg/perlonjava/runtime/runtimetypes/RuntimeScalar;");
+            mv.visitMethodInsn(Opcodes.INVOKESTATIC,
+                    "org/perlonjava/runtime/runtimetypes/RuntimeCode",
+                    "inheritSelfReference",
+                    "(Lorg/perlonjava/runtime/runtimetypes/RuntimeScalar;Lorg/perlonjava/runtime/runtimetypes/RuntimeScalar;)V",
+                    false);
+        }
+        if (node.getBooleanAnnotation("regexCallbackPseudoBlock")) {
+            mv.visitInsn(Opcodes.DUP);
+            mv.visitFieldInsn(Opcodes.GETFIELD,
+                    "org/perlonjava/runtime/runtimetypes/RuntimeScalar",
+                    "value",
+                    "Ljava/lang/Object;");
+            mv.visitTypeInsn(Opcodes.CHECKCAST,
+                    "org/perlonjava/runtime/runtimetypes/RuntimeCode");
+            mv.visitInsn(Opcodes.ICONST_1);
+            mv.visitFieldInsn(Opcodes.PUTFIELD,
+                    "org/perlonjava/runtime/runtimetypes/RuntimeCode",
+                    "isRegexCallbackPseudoBlock",
+                    "Z");
+        }
 
         // Set isEvalBlock on the RuntimeCode so RuntimeCode.apply() propagates
         // non-local returns through eval BLOCK boundaries

src/main/java/org/perlonjava/backend/jvm/Local.java

@@ -22,7 +22,7 @@ static void localTeardown(int dynamicIndex, MethodVisitor mv) {
         mv.visitVarInsn(Opcodes.ILOAD, dynamicIndex);
         mv.visitMethodInsn(Opcodes.INVOKESTATIC,
                 "org/perlonjava/runtime/runtimetypes/DynamicVariableManager",
-                "popToLocalLevel",
+                "teardownFrameToLocalLevel",
                 "(I)V",
                 false);
     }

src/main/java/org/perlonjava/frontend/parser/OperatorParser.java

@@ -800,7 +800,15 @@ static OperatorNode parseKeys(Parser parser, LexerToken token, int currentIndex)
         // Named unary operators have precedence between 20 and 21 in Perl
         // This allows expressions like: values $hashref->%* or keys $hashref->%* or scalar((nil) x 3, 1)
         if (operator.equals("scalar") || operator.equals("values") || operator.equals("keys") || operator.equals("each")) {
-            operand = parser.parseExpression(parser.getPrecedence("=~")); // precedence 20
+            // parseExpression stops before an operator whose precedence is
+            // equal to the supplied floor. Named unary scalar binds across a
+            // following =~ / !~ (`scalar $s =~ /(...)/`) and must force that
+            // match into scalar context rather than letting an enclosing print
+            // put it in list context. The other named unary operators retain
+            // their existing match-level boundary.
+            int operandPrecedence = parser.getPrecedence("=~")
+                    - (operator.equals("scalar") ? 1 : 0);
+            operand = parser.parseExpression(operandPrecedence);
             // Check if operand is null (no argument provided)
             if (operand == null) {
                 throw new PerlCompilerException(currentIndex, "Not enough arguments for " + operator, parser.ctx.errorUtil);

src/main/java/org/perlonjava/frontend/parser/StringSegmentParser.java

@@ -962,6 +962,14 @@ private void parseRegexCodeBlock(boolean isRecursive) {
 
     private Node regexCallback(Node block, String kind, int index) {
         SubroutineNode closure = new SubroutineNode(null, null, null, block, false, index);
+        closure.setAnnotation("inheritsSelfReference", true);
+        closure.setAnnotation("regexCallbackPseudoBlock", true);
+        if (block instanceof AbstractNode abstractBlock) {
+            // (?{ ... }) is a regex pseudo-block, not an ordinary anonymous-sub
+            // scope. Its top-level local() frames belong to the matcher path and
+            // must survive the Java callback return until Joni commits/unwinds it.
+            abstractBlock.setAnnotation("regexCallbackBody", true);
+        }
         OperatorNode callback = new OperatorNode("regexCallback", closure, index);
         callback.setAnnotation("regexCallbackKind", kind);
         hasExecutableRegexCallbacks = true;

src/main/java/org/perlonjava/runtime/regex/JoniRegexPattern.java

@@ -377,7 +377,13 @@ public boolean find() {
                 calloutHandler = new PerlCalloutHandler(input, byteToChar, callbacks, flags);
                 matcher.setCalloutHandler(calloutHandler);
             }
-            int result = matcher.search(charToByte[nextStart], charToByte[regionEnd], Option.NONE);
+            int result;
+            try {
+                result = matcher.search(charToByte[nextStart], charToByte[regionEnd], Option.NONE);
+            } catch (RuntimeException | Error failure) {
+                if (calloutHandler != null) calloutHandler.abort();
+                throw failure;
+            }
             matched = result >= 0;
             if (calloutHandler != null) calloutHandler.finish(matched);
             if (!matched) return false;
@@ -496,6 +502,7 @@ private record Token(int localLevel, RegexState regexState, RuntimeScalar previo
         private final List<RuntimeRegexCallback> callbacks;
         private final RegexFlags outerFlags;
         private final RegexCallbackMutationSnapshot mutations;
+        private final int initialLocalLevel;
         private RuntimeScalar completedResult;
 
         PerlCalloutHandler(String input, int[] byteToChar, List<RuntimeRegexCallback> callbacks,
@@ -513,6 +520,7 @@ private PerlCalloutHandler(String input, int[] byteToChar,
             this.callbacks = callbacks;
             this.outerFlags = outerFlags;
             this.mutations = mutations;
+            this.initialLocalLevel = DynamicVariableManager.getLocalLevel();
             for (RuntimeRegexCallback callback : callbacks) mutations.include(callback.code);
         }
 
@@ -574,8 +582,17 @@ private Evaluation evaluate(RuntimeRegexCallback callback, MatchView match) {
             publishProvisional(match);
 
             try {
-                RuntimeScalar result = RuntimeCode.apply(new RuntimeScalar(callback.code),
-                        new RuntimeArray(), RuntimeContextType.SCALAR).scalar();
+                DynamicVariableManager.CapturedFrame<RuntimeList> frame =
+                        DynamicVariableManager.captureFrameLocals(() -> RuntimeCode.apply(
+                                new RuntimeScalar(callback.code), new RuntimeArray(),
+                                RuntimeContextType.SCALAR));
+                // Joni's complete() notification is delayed until the candidate
+                // path commits. Resume now so a later (?{ ... }) on that same
+                // path observes local() values; unwind() still owns the token's
+                // pre-callback level and rolls the frame back on backtracking.
+                DynamicVariableManager.resumeSuspended(frame.states());
+                rejectEscapedControlFlow(frame.result());
+                RuntimeScalar result = frame.result().scalar();
                 boolean block = callback.kind == RuntimeRegexCallback.Kind.BLOCK;
                 if (block) rVariable.set(result);
                 Token token = new Token(localLevel, savedRegex, previousR,
@@ -602,15 +619,32 @@ public void complete(Object value) {
         }
 
         void finish(boolean matched) {
-            if (!matched) mutations.restore();
-            if (matched && completedResult != null) {
-                GlobalVariable.getGlobalVariable(GlobalContext.encodeSpecialVar("R"))
-                        .set(completedResult);
+            try {
+                if (!matched) mutations.restore();
+                if (matched && completedResult != null) {
+                    GlobalVariable.getGlobalVariable(GlobalContext.encodeSpecialVar("R"))
+                            .set(completedResult);
+                }
+            } finally {
+                DynamicVariableManager.popToLocalLevel(initialLocalLevel);
+            }
+        }
+
+        void abort() {
+            try {
+                mutations.restore();
+            } finally {
+                DynamicVariableManager.popToLocalLevel(initialLocalLevel);
             }
         }
 
         private void restore(Token token, boolean completed) {
-            restoreCallbackScope(token.localLevel(), token.regexState(), token.previousR());
+            if (!completed) {
+                DynamicVariableManager.popToLocalLevel(token.localLevel());
+            }
+            token.regexState().restore();
+            GlobalVariable.getGlobalVariable(GlobalContext.encodeSpecialVar("R"))
+                    .set(token.previousR());
             if (completed && token.block() && completedResult == null) {
                 completedResult = token.result();
             }
@@ -627,6 +661,21 @@ private static void restoreCallbackScope(int localLevel, RegexState regexState,
             }
         }
 
+        private static void rejectEscapedControlFlow(RuntimeList result) {
+            if (!(result instanceof RuntimeControlFlowList flow)) return;
+            ControlFlowMarker marker = flow.marker;
+            if (marker.type == ControlFlowType.GOTO
+                    || marker.type == ControlFlowType.TAILCALL) {
+                // The runtime location is the regex pseudo-block boundary (and
+                // can differ from the marker's inner goto location), so let the
+                // exception formatter attach it.
+                throw new PerlCompilerException("Can't \"goto\" out of a pseudo block");
+            }
+            // Preserve the control op's own location. The terminating newline
+            // tells PerlCompilerException this is already fully formatted.
+            throw new PerlCompilerException(marker.buildErrorMessage() + ".\n");
+        }
+
         private void publishProvisional(MatchView match) {
             RuntimeRegexState state = PerlRuntime.current().regexState;
             int count = match.captureCount();

src/main/java/org/perlonjava/runtime/regex/RuntimeRegexCallback.java

@@ -19,6 +19,7 @@ public static RuntimeScalar wrap(RuntimeScalar codeRef, String kindName) {
         if (!(codeRef.value instanceof RuntimeCode code)) {
             throw new IllegalArgumentException("regex callback is not a code reference");
         }
+        code.isRegexCallbackPseudoBlock = true;
         return new RuntimeScalar(new RuntimeRegexCallback(code, Kind.valueOf(kindName)));
     }
 }