diff --git a/dev/design/executable-regex-callbacks.md b/dev/design/executable-regex-callbacks.md index b8d071f708..1b35ede920 100644 --- a/dev/design/executable-regex-callbacks.md +++ b/dev/design/executable-regex-callbacks.md @@ -2,11 +2,10 @@ ## Status -- **Current phase:** Phase 0 — design and differential baseline +- **Current phase:** Phase 3 — backtracking and dynamic scope - **Started:** 2026-08-09 -- **Implementation status:** Not started -- **Prerequisite:** PR #895 (`feature/cpan-workaround-cleanup`) or an equivalent - backend-neutral `RegexMatcher` integration +- **Implementation status:** Callout engine and plain callback bridge integrated +- **Prerequisite:** Satisfied by the namespaced callout-enabled Joni integration - **Primary targets:** `(?{ ... })`, `(?(?{ ... })yes|no)`, and `(??{ ... })` ## Decision Summary @@ -617,46 +616,36 @@ record the observed output in this document as phases proceed. ## Progress Tracking -### Current Status: Phase 0 in progress +### Current Status: Phase 3 in progress ### Completed Phases -- [ ] Phase 0: Differential semantics and Joni spike -- [ ] Phase 1: Structured frontend and runtime template -- [ ] Phase 2: Plain `(?{ ... })` +- [x] Phase 0: Differential semantics and Joni spike +- [x] Phase 1: Structured frontend and runtime template +- [x] Phase 2: Plain `(?{ ... })` - [ ] Phase 3: Backtracking and dynamic scope -- [ ] Phase 4: Callback conditions +- [x] Phase 4: Callback conditions - [ ] Phase 5: `(??{ ... })` dynamic programs - [ ] Phase 6: Runtime source, hardening, and policy removal -### Work Completed - -- 2026-08-09: Created this design after reviewing the current parser, - `RegexPreprocessor`, `RuntimeRegex`, the Joni 2.2.7 API, PR #895's matcher - abstraction, and the executable-regex CPAN policies. -- 2026-08-09: Selected a structured callback template plus generic Joni callout - extension as the preferred architecture. -- 2026-08-09: Corrected the older blanket-side-effect-journaling proposal: Perl - dynamic locals require backtracking unwind, but ordinary side effects must not - all be reverted. - ### Next Steps -1. Write and validate `regex_executable_callbacks.t` with standard Perl. -2. Record exact standard-Perl outputs for every open semantic question reachable - without implementation. -3. Create a disposable Joni 2.2.7 callout spike and measure the patch surface. -4. Decide whether to upstream the generic callout API or publish a namespaced fork. -5. Begin Phase 1 only after PR #895's matcher abstraction is merged or rebased into - the implementation branch. +1. Preserve lexical regex flags and package metadata for runtime/interpolated + executable source. +2. Close nested callback caller/source-line and interpolated `qr//` `__SUB__` + identity gaps. +3. Add warning-location, interruption, timeout, and nested-exception gates. +4. Classify tied, magical, shared, and readonly mutation behavior with standard + Perl before extending matcher transactions. +5. Finish dynamic-pattern recursion/caching gates and then remove only the + capability policies justified by unchanged-source results. ### Blockers -- PR #895 is still the integration prerequisite for the planned engine routing. -- Joni 2.2.7 does not expose an in-match callback extension point; Phase 0 must - validate the maintained-fork approach. -- Several detailed Perl semantics remain intentionally open pending differential - tests. +- Runtime-injected callback source still requires `use re 'eval'` propagation. +- Recursive callback frames do not yet retain all Perl caller source lines. +- Tied, magical, shared, and readonly rollback semantics remain intentionally + open pending differential tests. ## Related Documents and Skills diff --git a/dev/design/phase36-regex-parity.md b/dev/design/phase36-regex-parity.md index 6da84150a2..69ab2e06ef 100644 --- a/dev/design/phase36-regex-parity.md +++ b/dev/design/phase36-regex-parity.md @@ -242,12 +242,15 @@ timing delta is a regression only after a serialized same-commit reproduction. The merged Joni dynamic-pattern engine establishes the Stage 36.5 execution seam. The current Stage 36.4 core baseline is `rxcode.t` 42/42 and -`reg_eval_scope.t` 22/49, with no timeout or incomplete file. Matcher-owned -transactions now restore ordinary scalar, array, and hash mutations when the -overall match fails, retain mutations from abandoned alternatives when another -alternative succeeds, and commit successful matches. Callback exceptions also -restore dynamic locals, provisional match state, and `$^R` on both execution -backends. Regex stringification no longer exposes private callback IDs. +`reg_eval_scope.t` 33/49, with no timeout or incomplete file. Matcher-owned +transactions restore ordinary scalar, array, and hash mutations on total +failure while retaining Perl's ordinary side effects from attempted paths. +Callback dynamic locals now transfer from the implementation CV to the matcher: +they remain visible to later callbacks on the active path, unwind on +backtracking, and restore after success, failure, or exception. Regex callbacks +also behave as pseudo-blocks for `caller`, `__SUB__`, and escaping +`last`/`next`/`goto` on both execution backends. Named unary `scalar` now keeps a +following match in scalar context, including callback-bearing matches. ### Completed stages @@ -262,19 +265,25 @@ backends. Regex stringification no longer exposes private callback IDs. ### Next steps -1. Complete callback lexical pragma, caller-frame, and control-flow isolation - exposed by `reg_eval_scope.t`, without changing its thread wrapper. -2. Extend the callback semantic matrix with interruption, timeout, nested - exception paths; require identical JVM/interpreter cleanup. -3. Define and implement the mutation policy for tied, magical, shared, and +1. Preserve lexical package and `use re '/flags'` state for `qr//`, interpolated + regex objects, and runtime source admitted by `use re 'eval'`. +2. Extend pseudo-block frame mapping through nested and recursive callbacks; + preserve exact caller source lines and enclosing `__SUB__` for interpolated + `qr//` values. +3. Extend the callback semantic matrix with warning locations, interruption, + timeout, and nested exception paths; require identical JVM/interpreter cleanup. +4. Define and implement the mutation policy for tied, magical, shared, and readonly values; ordinary values are now transactionally covered. -4. Complete the merged dynamic-pattern validation gates, then mark Stage 36.5 +5. Complete the merged dynamic-pattern validation gates, then mark Stage 36.5 complete and proceed to the remaining declarative parity slices. ### Open blockers -- Callback lexical pragmata, caller frames, and non-local control-flow - boundaries still differ from Perl in `reg_eval_scope.t`. +- Runtime-injected callback source and lexical regex pragmata remain unsupported; + these account for tests 4, 5, 8, 10, 11, and 12 in `reg_eval_scope.t`. +- Recursive/nested callback caller lines, interpolated `qr//` `__SUB__`, warning + locations, and the legacy `qr/\(?{` diagnostic account for the remaining + Stage 36.4 failures. - Tied, magical, shared, and readonly callback mutation rollback remains intentionally outside the ordinary-value transaction until its exact Perl behavior is established with differential tests. diff --git a/src/main/java/org/perlonjava/backend/bytecode/BytecodeCompiler.java b/src/main/java/org/perlonjava/backend/bytecode/BytecodeCompiler.java index e4d43806bf..f7adb1f745 100644 --- a/src/main/java/org/perlonjava/backend/bytecode/BytecodeCompiler.java +++ b/src/main/java/org/perlonjava/backend/bytecode/BytecodeCompiler.java @@ -5915,6 +5915,12 @@ private void visitAnonymousSubroutine(SubroutineNode node) { subCode.isMapGrepBlock = true; subCode.inheritsSelfReference = true; } + if (node.getBooleanAnnotation("inheritsSelfReference")) { + subCode.inheritsSelfReference = true; + } + if (node.getBooleanAnnotation("regexCallbackPseudoBlock")) { + subCode.isRegexCallbackPseudoBlock = true; + } if (RuntimeCode.isDisassemble()) { System.out.println(Disassemble.disassemble(subCode)); diff --git a/src/main/java/org/perlonjava/backend/bytecode/BytecodeInterpreter.java b/src/main/java/org/perlonjava/backend/bytecode/BytecodeInterpreter.java index 064ba05704..9cd101ea55 100644 --- a/src/main/java/org/perlonjava/backend/bytecode/BytecodeInterpreter.java +++ b/src/main/java/org/perlonjava/backend/bytecode/BytecodeInterpreter.java @@ -3003,7 +3003,7 @@ private static RuntimeList execute(SuspendedInterpreterFrame frame) { frame.suspendedDynamicStates = DynamicVariableManager.suspendAbove(savedLocalLevel); } else { - DynamicVariableManager.popToLocalLevel(savedLocalLevel); + DynamicVariableManager.teardownFrameToLocalLevel(savedLocalLevel); } currentPackageScalar.set(savedPackage); if (frame.suspended && !frame.evalCatchStack.isEmpty()) { diff --git a/src/main/java/org/perlonjava/backend/jvm/EmitBlock.java b/src/main/java/org/perlonjava/backend/jvm/EmitBlock.java index 2f0392c5ab..1359878a9d 100644 --- a/src/main/java/org/perlonjava/backend/jvm/EmitBlock.java +++ b/src/main/java/org/perlonjava/backend/jvm/EmitBlock.java @@ -261,7 +261,9 @@ public static void emitBlock(EmitterVisitor emitterVisitor, BlockNode node) { } // Setup 'local' environment if needed - Local.localRecord localRecord = Local.localSetup(emitterVisitor.ctx, node, mv, true); + Local.localRecord localRecord = node.getBooleanAnnotation("regexCallbackBody") + ? new Local.localRecord(false, -1) + : Local.localSetup(emitterVisitor.ctx, node, mv, true); int regexStateLocal = -1; if (!node.getBooleanAnnotation("blockIsSubroutine") diff --git a/src/main/java/org/perlonjava/backend/jvm/EmitSubroutine.java b/src/main/java/org/perlonjava/backend/jvm/EmitSubroutine.java index f8453901d8..48c6499951 100644 --- a/src/main/java/org/perlonjava/backend/jvm/EmitSubroutine.java +++ b/src/main/java/org/perlonjava/backend/jvm/EmitSubroutine.java @@ -488,6 +488,34 @@ public static void emitSubroutine(EmitterContext ctx, SubroutineNode node) { "(Lorg/perlonjava/runtime/runtimetypes/RuntimeScalar;Lorg/perlonjava/runtime/runtimetypes/RuntimeScalar;)V", false); } + if (node.getBooleanAnnotation("inheritsSelfReference") + && !(isMapGrepBlock != null && isMapGrepBlock)) { + mv.visitInsn(Opcodes.DUP); + mv.visitVarInsn(Opcodes.ALOAD, 0); + mv.visitFieldInsn(Opcodes.GETFIELD, + ctx.javaClassInfo.javaClassName, + "__SUB__", + "Lorg/perlonjava/runtime/runtimetypes/RuntimeScalar;"); + mv.visitMethodInsn(Opcodes.INVOKESTATIC, + "org/perlonjava/runtime/runtimetypes/RuntimeCode", + "inheritSelfReference", + "(Lorg/perlonjava/runtime/runtimetypes/RuntimeScalar;Lorg/perlonjava/runtime/runtimetypes/RuntimeScalar;)V", + false); + } + if (node.getBooleanAnnotation("regexCallbackPseudoBlock")) { + mv.visitInsn(Opcodes.DUP); + mv.visitFieldInsn(Opcodes.GETFIELD, + "org/perlonjava/runtime/runtimetypes/RuntimeScalar", + "value", + "Ljava/lang/Object;"); + mv.visitTypeInsn(Opcodes.CHECKCAST, + "org/perlonjava/runtime/runtimetypes/RuntimeCode"); + mv.visitInsn(Opcodes.ICONST_1); + mv.visitFieldInsn(Opcodes.PUTFIELD, + "org/perlonjava/runtime/runtimetypes/RuntimeCode", + "isRegexCallbackPseudoBlock", + "Z"); + } // Set isEvalBlock on the RuntimeCode so RuntimeCode.apply() propagates // non-local returns through eval BLOCK boundaries diff --git a/src/main/java/org/perlonjava/backend/jvm/Local.java b/src/main/java/org/perlonjava/backend/jvm/Local.java index 79d31a4e3d..1d27f368c3 100644 --- a/src/main/java/org/perlonjava/backend/jvm/Local.java +++ b/src/main/java/org/perlonjava/backend/jvm/Local.java @@ -22,7 +22,7 @@ static void localTeardown(int dynamicIndex, MethodVisitor mv) { mv.visitVarInsn(Opcodes.ILOAD, dynamicIndex); mv.visitMethodInsn(Opcodes.INVOKESTATIC, "org/perlonjava/runtime/runtimetypes/DynamicVariableManager", - "popToLocalLevel", + "teardownFrameToLocalLevel", "(I)V", false); } diff --git a/src/main/java/org/perlonjava/frontend/parser/OperatorParser.java b/src/main/java/org/perlonjava/frontend/parser/OperatorParser.java index 4cf1265577..1d8d1882e2 100644 --- a/src/main/java/org/perlonjava/frontend/parser/OperatorParser.java +++ b/src/main/java/org/perlonjava/frontend/parser/OperatorParser.java @@ -800,7 +800,15 @@ static OperatorNode parseKeys(Parser parser, LexerToken token, int currentIndex) // Named unary operators have precedence between 20 and 21 in Perl // This allows expressions like: values $hashref->%* or keys $hashref->%* or scalar((nil) x 3, 1) if (operator.equals("scalar") || operator.equals("values") || operator.equals("keys") || operator.equals("each")) { - operand = parser.parseExpression(parser.getPrecedence("=~")); // precedence 20 + // parseExpression stops before an operator whose precedence is + // equal to the supplied floor. Named unary scalar binds across a + // following =~ / !~ (`scalar $s =~ /(...)/`) and must force that + // match into scalar context rather than letting an enclosing print + // put it in list context. The other named unary operators retain + // their existing match-level boundary. + int operandPrecedence = parser.getPrecedence("=~") + - (operator.equals("scalar") ? 1 : 0); + operand = parser.parseExpression(operandPrecedence); // Check if operand is null (no argument provided) if (operand == null) { throw new PerlCompilerException(currentIndex, "Not enough arguments for " + operator, parser.ctx.errorUtil); diff --git a/src/main/java/org/perlonjava/frontend/parser/StringSegmentParser.java b/src/main/java/org/perlonjava/frontend/parser/StringSegmentParser.java index d1e32b39e6..5fe21cea53 100644 --- a/src/main/java/org/perlonjava/frontend/parser/StringSegmentParser.java +++ b/src/main/java/org/perlonjava/frontend/parser/StringSegmentParser.java @@ -962,6 +962,14 @@ private void parseRegexCodeBlock(boolean isRecursive) { private Node regexCallback(Node block, String kind, int index) { SubroutineNode closure = new SubroutineNode(null, null, null, block, false, index); + closure.setAnnotation("inheritsSelfReference", true); + closure.setAnnotation("regexCallbackPseudoBlock", true); + if (block instanceof AbstractNode abstractBlock) { + // (?{ ... }) is a regex pseudo-block, not an ordinary anonymous-sub + // scope. Its top-level local() frames belong to the matcher path and + // must survive the Java callback return until Joni commits/unwinds it. + abstractBlock.setAnnotation("regexCallbackBody", true); + } OperatorNode callback = new OperatorNode("regexCallback", closure, index); callback.setAnnotation("regexCallbackKind", kind); hasExecutableRegexCallbacks = true; diff --git a/src/main/java/org/perlonjava/runtime/regex/JoniRegexPattern.java b/src/main/java/org/perlonjava/runtime/regex/JoniRegexPattern.java index 6b3b906a79..a11ba1716a 100644 --- a/src/main/java/org/perlonjava/runtime/regex/JoniRegexPattern.java +++ b/src/main/java/org/perlonjava/runtime/regex/JoniRegexPattern.java @@ -377,7 +377,13 @@ public boolean find() { calloutHandler = new PerlCalloutHandler(input, byteToChar, callbacks, flags); matcher.setCalloutHandler(calloutHandler); } - int result = matcher.search(charToByte[nextStart], charToByte[regionEnd], Option.NONE); + int result; + try { + result = matcher.search(charToByte[nextStart], charToByte[regionEnd], Option.NONE); + } catch (RuntimeException | Error failure) { + if (calloutHandler != null) calloutHandler.abort(); + throw failure; + } matched = result >= 0; if (calloutHandler != null) calloutHandler.finish(matched); if (!matched) return false; @@ -496,6 +502,7 @@ private record Token(int localLevel, RegexState regexState, RuntimeScalar previo private final List callbacks; private final RegexFlags outerFlags; private final RegexCallbackMutationSnapshot mutations; + private final int initialLocalLevel; private RuntimeScalar completedResult; PerlCalloutHandler(String input, int[] byteToChar, List callbacks, @@ -513,6 +520,7 @@ private PerlCalloutHandler(String input, int[] byteToChar, this.callbacks = callbacks; this.outerFlags = outerFlags; this.mutations = mutations; + this.initialLocalLevel = DynamicVariableManager.getLocalLevel(); for (RuntimeRegexCallback callback : callbacks) mutations.include(callback.code); } @@ -574,8 +582,17 @@ private Evaluation evaluate(RuntimeRegexCallback callback, MatchView match) { publishProvisional(match); try { - RuntimeScalar result = RuntimeCode.apply(new RuntimeScalar(callback.code), - new RuntimeArray(), RuntimeContextType.SCALAR).scalar(); + DynamicVariableManager.CapturedFrame frame = + DynamicVariableManager.captureFrameLocals(() -> RuntimeCode.apply( + new RuntimeScalar(callback.code), new RuntimeArray(), + RuntimeContextType.SCALAR)); + // Joni's complete() notification is delayed until the candidate + // path commits. Resume now so a later (?{ ... }) on that same + // path observes local() values; unwind() still owns the token's + // pre-callback level and rolls the frame back on backtracking. + DynamicVariableManager.resumeSuspended(frame.states()); + rejectEscapedControlFlow(frame.result()); + RuntimeScalar result = frame.result().scalar(); boolean block = callback.kind == RuntimeRegexCallback.Kind.BLOCK; if (block) rVariable.set(result); Token token = new Token(localLevel, savedRegex, previousR, @@ -602,15 +619,32 @@ public void complete(Object value) { } void finish(boolean matched) { - if (!matched) mutations.restore(); - if (matched && completedResult != null) { - GlobalVariable.getGlobalVariable(GlobalContext.encodeSpecialVar("R")) - .set(completedResult); + try { + if (!matched) mutations.restore(); + if (matched && completedResult != null) { + GlobalVariable.getGlobalVariable(GlobalContext.encodeSpecialVar("R")) + .set(completedResult); + } + } finally { + DynamicVariableManager.popToLocalLevel(initialLocalLevel); + } + } + + void abort() { + try { + mutations.restore(); + } finally { + DynamicVariableManager.popToLocalLevel(initialLocalLevel); } } private void restore(Token token, boolean completed) { - restoreCallbackScope(token.localLevel(), token.regexState(), token.previousR()); + if (!completed) { + DynamicVariableManager.popToLocalLevel(token.localLevel()); + } + token.regexState().restore(); + GlobalVariable.getGlobalVariable(GlobalContext.encodeSpecialVar("R")) + .set(token.previousR()); if (completed && token.block() && completedResult == null) { completedResult = token.result(); } @@ -627,6 +661,21 @@ private static void restoreCallbackScope(int localLevel, RegexState regexState, } } + private static void rejectEscapedControlFlow(RuntimeList result) { + if (!(result instanceof RuntimeControlFlowList flow)) return; + ControlFlowMarker marker = flow.marker; + if (marker.type == ControlFlowType.GOTO + || marker.type == ControlFlowType.TAILCALL) { + // The runtime location is the regex pseudo-block boundary (and + // can differ from the marker's inner goto location), so let the + // exception formatter attach it. + throw new PerlCompilerException("Can't \"goto\" out of a pseudo block"); + } + // Preserve the control op's own location. The terminating newline + // tells PerlCompilerException this is already fully formatted. + throw new PerlCompilerException(marker.buildErrorMessage() + ".\n"); + } + private void publishProvisional(MatchView match) { RuntimeRegexState state = PerlRuntime.current().regexState; int count = match.captureCount(); diff --git a/src/main/java/org/perlonjava/runtime/regex/RuntimeRegexCallback.java b/src/main/java/org/perlonjava/runtime/regex/RuntimeRegexCallback.java index a4eb3931d4..60696c2761 100644 --- a/src/main/java/org/perlonjava/runtime/regex/RuntimeRegexCallback.java +++ b/src/main/java/org/perlonjava/runtime/regex/RuntimeRegexCallback.java @@ -19,6 +19,7 @@ public static RuntimeScalar wrap(RuntimeScalar codeRef, String kindName) { if (!(codeRef.value instanceof RuntimeCode code)) { throw new IllegalArgumentException("regex callback is not a code reference"); } + code.isRegexCallbackPseudoBlock = true; return new RuntimeScalar(new RuntimeRegexCallback(code, Kind.valueOf(kindName))); } } diff --git a/src/main/java/org/perlonjava/runtime/runtimetypes/DynamicVariableManager.java b/src/main/java/org/perlonjava/runtime/runtimetypes/DynamicVariableManager.java index 54cfb52362..8d64648cfa 100644 --- a/src/main/java/org/perlonjava/runtime/runtimetypes/DynamicVariableManager.java +++ b/src/main/java/org/perlonjava/runtime/runtimetypes/DynamicVariableManager.java @@ -4,6 +4,7 @@ import java.util.ArrayList; import java.util.Deque; import java.util.List; +import java.util.function.Supplier; /** * The DynamicVariableManager class is responsible for managing a stack of dynamic variables. @@ -13,12 +14,26 @@ */ public class DynamicVariableManager { public record SuspendedState(DynamicState state, Object token) {} + public record CapturedFrame(T result, List states) {} + + static final class FrameCapture { + final int localLevel; + List states; + + FrameCapture(int localLevel) { + this.localLevel = localLevel; + } + } // A stack to hold the dynamic states of variables. // Using ArrayDeque instead of Stack for better performance (no synchronization overhead). private static Deque variableStack() { return PerlRuntime.current().executionState().dynamicVariableStack; } + private static Deque frameCaptures() { + return PerlRuntime.current().executionState().dynamicFrameCaptures; + } + /** * Returns the current local level, which is the size of the variable stack. * This indicates how many dynamic states are currently being managed. @@ -121,6 +136,44 @@ public static void popToLocalLevel(int targetLocalLevel) { } } + /** + * Performs an outer subroutine-frame teardown. Regex executable callbacks + * may temporarily take ownership of the dynamic states that survive the + * callback body, so Joni can commit or abandon them with the match path. + * Ordinary calls retain the normal pop-and-restore behavior. + */ + public static void teardownFrameToLocalLevel(int targetLocalLevel) { + FrameCapture capture = frameCaptures().peekLast(); + if (capture != null && capture.localLevel == targetLocalLevel + && capture.states == null) { + capture.states = suspendAbove(targetLocalLevel); + return; + } + popToLocalLevel(targetLocalLevel); + } + + /** Execute one subroutine call while retaining its surviving local() states. */ + public static CapturedFrame captureFrameLocals(Supplier action) { + FrameCapture capture = new FrameCapture(getLocalLevel()); + Deque captures = frameCaptures(); + captures.addLast(capture); + boolean completed = false; + try { + T result = action.get(); + completed = true; + return new CapturedFrame<>(result, + capture.states == null ? List.of() : capture.states); + } finally { + if (captures.peekLast() != capture) { + throw new IllegalStateException("Dynamic frame capture closed out of order"); + } + captures.removeLast(); + if (!completed && getLocalLevel() > capture.localLevel) { + popToLocalLevel(capture.localLevel); + } + } + } + /** * Detach states belonging to a suspended interpreter frame. Unlike * {@link #popToLocalLevel(int)}, this returns the states in their original diff --git a/src/main/java/org/perlonjava/runtime/runtimetypes/ExecutionRuntimeState.java b/src/main/java/org/perlonjava/runtime/runtimetypes/ExecutionRuntimeState.java index 0aa0984f68..f73e7a2f69 100644 --- a/src/main/java/org/perlonjava/runtime/runtimetypes/ExecutionRuntimeState.java +++ b/src/main/java/org/perlonjava/runtime/runtimetypes/ExecutionRuntimeState.java @@ -13,6 +13,7 @@ public final class ExecutionRuntimeState { final List callerStack = new ArrayList<>(); final Deque dynamicVariableStack = new ArrayDeque<>(); + final Deque dynamicFrameCaptures = new ArrayDeque<>(); final Stack scalarDynamicStates = new Stack<>(); final Stack arrayDynamicStates = new Stack<>(); diff --git a/src/main/java/org/perlonjava/runtime/runtimetypes/RuntimeCode.java b/src/main/java/org/perlonjava/runtime/runtimetypes/RuntimeCode.java index cb94085ab8..99e154f221 100644 --- a/src/main/java/org/perlonjava/runtime/runtimetypes/RuntimeCode.java +++ b/src/main/java/org/perlonjava/runtime/runtimetypes/RuntimeCode.java @@ -892,6 +892,9 @@ public static void registerPadConstants(String className, RuntimeBase[] constant public boolean deferredConstAttribute = false; // Flag to indicate this code is a map/grep block - non-local return should propagate through it public boolean isMapGrepBlock = false; + // Executable regex callbacks are Perl pseudo-blocks. They execute through + // an implementation CV but must not add a caller() frame. + public boolean isRegexCallbackPseudoBlock = false; // Implementation callbacks such as map/grep do not introduce a Perl // subroutine scope, so their __SUB__ comes from the enclosing RuntimeCode. public boolean inheritsSelfReference = false; @@ -1853,6 +1856,7 @@ public void adoptDefinitionFrom(RuntimeCode codeFrom) { this.attributesDispatchedAtCompileTime = codeFrom.attributesDispatchedAtCompileTime; this.deferredConstAttribute = codeFrom.deferredConstAttribute; this.isMapGrepBlock = codeFrom.isMapGrepBlock; + this.isRegexCallbackPseudoBlock = codeFrom.isRegexCallbackPseudoBlock; this.inheritsSelfReference = codeFrom.inheritsSelfReference; this.isEvalBlock = codeFrom.isEvalBlock; this.explicitlyRenamed = codeFrom.explicitlyRenamed; @@ -3657,6 +3661,8 @@ public static RuntimeList callerWithSub(RuntimeList args, int ctx, RuntimeScalar frame = args.getFirst().getInt(); } + frame = callerFrameIncludingRegexPseudoBlocks(frame); + // Save the original user-supplied frame before the JVM skip adjustment. // This value maps directly to hasArgsStack depth: caller(0) → depth 0 (current frame), // caller(1) → depth 1 (caller's frame), etc. The hasArgsStack is pushed/popped in the @@ -4158,6 +4164,27 @@ private static RuntimeCode activeCodeAtCallerFrame(int logicalFrame) { return null; } + /** Map a Perl-visible caller depth to the implementation stack depth. */ + private static int callerFrameIncludingRegexPseudoBlocks(int logicalFrame) { + if (logicalFrame < 0) return logicalFrame; + RuntimeCode previous = null; + int physical = 0; + int visible = 0; + for (RuntimeCode active : activeCodeStack()) { + if (active == previous || isCompilerWrapperPair(active, previous)) { + continue; + } + previous = active; + if (active.isRegexCallbackPseudoBlock) { + physical++; + continue; + } + if (visible++ == logicalFrame) return physical; + physical++; + } + return logicalFrame + (physical - visible); + } + public static RuntimeCode getActiveCodeAtCallerFrame(int logicalFrame) { return activeCodeAtCallerFrame(logicalFrame); } diff --git a/src/main/java/org/perlonjava/runtime/runtimetypes/RuntimeGraphCloner.java b/src/main/java/org/perlonjava/runtime/runtimetypes/RuntimeGraphCloner.java index 01b108c51d..f37d89f9cb 100644 --- a/src/main/java/org/perlonjava/runtime/runtimetypes/RuntimeGraphCloner.java +++ b/src/main/java/org/perlonjava/runtime/runtimetypes/RuntimeGraphCloner.java @@ -293,6 +293,7 @@ private void copyCodeMetadata(RuntimeCode source, RuntimeCode target) { }; } target.isMapGrepBlock = source.isMapGrepBlock; + target.isRegexCallbackPseudoBlock = source.isRegexCallbackPseudoBlock; target.isEvalBlock = source.isEvalBlock; target.isTryExpressionWrapper = source.isTryExpressionWrapper; target.inheritsSelfReference = source.inheritsSelfReference; diff --git a/src/test/resources/unit/regex/callback_caller_scope.t b/src/test/resources/unit/regex/callback_caller_scope.t new file mode 100644 index 0000000000..f755ad439d --- /dev/null +++ b/src/test/resources/unit/regex/callback_caller_scope.t @@ -0,0 +1,37 @@ +use strict; +use warnings; +use feature 'current_sub'; + +print "1..4\n"; + +sub enclosing { + 'a' =~ /(?{ + my $name = (caller(0))[3]; + print defined($name) && $name eq 'main::enclosing' + ? "ok 1 - callback caller is its enclosing sub\n" + : "not ok 1 - callback caller is its enclosing sub\n"; + })a/; +} +enclosing(); + +sub called_from_callback { + my $self = (caller(0))[3]; + my $outer = (caller(1))[3]; + print defined($self) && $self eq 'main::called_from_callback' + ? "ok 2 - real sub frame inside callback is preserved\n" + : "not ok 2 - real sub frame inside callback is preserved\n"; + print !defined($outer) + ? "ok 3 - callback frame is hidden from nested sub\n" + : "not ok 3 - callback frame is hidden from nested sub\n"; +} +'a' =~ /(?{ called_from_callback() })a/; + +sub callback_self { + my $enclosing = __SUB__; + 'a' =~ /(?{ + print __SUB__ == $enclosing + ? "ok 4 - callback inherits enclosing __SUB__\n" + : "not ok 4 - callback inherits enclosing __SUB__\n"; + })a/; +} +callback_self(); diff --git a/src/test/resources/unit/regex/callback_control_flow_boundary.t b/src/test/resources/unit/regex/callback_control_flow_boundary.t new file mode 100644 index 0000000000..88522bdf24 --- /dev/null +++ b/src/test/resources/unit/regex/callback_control_flow_boundary.t @@ -0,0 +1,36 @@ +use strict; +use warnings; + +print "1..4\n"; + +sub escaped_error (&) { + my ($code) = @_; + my $error = ''; + eval { $code->() }; + $error = $@; + return $error; +} + +my $last = escaped_error { 'a' =~ /(?{ last })a/ }; +print $last =~ /Can't "last" outside a loop block/ + ? "ok 1 - last cannot escape a regex pseudo block\n" + : "not ok 1 - last cannot escape a regex pseudo block\n"; + +my $next = escaped_error { 'a' =~ /(?{ next })a/ }; +print $next =~ /Can't "next" outside a loop block/ + ? "ok 2 - next cannot escape a regex pseudo block\n" + : "not ok 2 - next cannot escape a regex pseudo block\n"; + +my $goto = ''; +eval q{'a' =~ /(?{ goto OUT })a/; OUT: 1}; +$goto = $@; +print $goto =~ /Can't "goto" out of a pseudo block/ + ? "ok 3 - goto cannot escape a regex pseudo block\n" + : "not ok 3 - goto cannot escape a regex pseudo block\n"; + +my $inner = escaped_error { + 'a' =~ /(?{ for (1) { last } })a/; +}; +print $inner eq '' + ? "ok 4 - callback-local loop consumes its own control flow\n" + : "not ok 4 - callback-local loop consumes its own control flow\n"; diff --git a/src/test/resources/unit/regex/callback_local_scope.t b/src/test/resources/unit/regex/callback_local_scope.t new file mode 100644 index 0000000000..f6833cbfa4 --- /dev/null +++ b/src/test/resources/unit/regex/callback_local_scope.t @@ -0,0 +1,29 @@ +use strict; +use warnings; + +print "1..4\n"; + +our $value = 1; +our @seen; +'ab' =~ /a(?{ + push @seen, $value; + local $value = $value + 1; +})b(?{ + push @seen, $value; +})/; +print join(',', @seen) eq '1,2' + ? "ok 1 - successful callback local is visible to later callbacks\n" + : "not ok 1 - successful callback local is visible to later callbacks\n"; +print $value == 1 + ? "ok 2 - successful match unwinds callback locals\n" + : "not ok 2 - successful match unwinds callback locals\n"; + +$value = 1; +@seen = (); +'ac' =~ /(?:a(?{ local $value = 2 })b|a(?{ push @seen, $value })c)/; +print join(',', @seen) eq '1' + ? "ok 3 - abandoned callback locals do not leak to another alternative\n" + : "not ok 3 - abandoned callback locals do not leak to another alternative\n"; +print $value == 1 + ? "ok 4 - backtracking unwinds callback locals\n" + : "not ok 4 - backtracking unwinds callback locals\n"; diff --git a/src/test/resources/unit/regex/scalar_match_context.t b/src/test/resources/unit/regex/scalar_match_context.t new file mode 100644 index 0000000000..26a48b63f7 --- /dev/null +++ b/src/test/resources/unit/regex/scalar_match_context.t @@ -0,0 +1,32 @@ +use strict; +use warnings; +use re 'eval'; + +print "1..3\n"; + +my $plain = ''; +open my $plain_out, '>', \$plain or die $!; +{ + local *STDOUT = $plain_out; + print scalar "abcabc" =~ /(abc){2}/; +} +print $plain eq '1' + ? "ok 1 - named unary scalar includes the following match\n" + : "not ok 1 - named unary scalar includes the following match\n"; + +our $value = 1; +my $callback = ''; +open my $callback_out, '>', \$callback or die $!; +{ + local *STDOUT = $callback_out; + print scalar "abcabc" =~ + /(a(?{ local $value = $value + 1 }) + b(?{ local $value = $value + 1 }) + c(?{ local $value = $value + 1 })){2}/x; +} +print $callback eq '1' + ? "ok 2 - callback match remains in scalar context\n" + : "not ok 2 - callback match remains in scalar context\n"; +print $value == 1 + ? "ok 3 - callback locals unwind after scalar match\n" + : "not ok 3 - callback locals unwind after scalar match\n";