Skip to content

Commit a0d5ffc

Browse files
authored
Merge pull request #978 from fglock/feature/phase36-callback-scope
feat(regex): preserve executable callback pseudo-block scope
2 parents cd9bba5 + 57d64ae commit a0d5ffc

19 files changed

Lines changed: 374 additions & 58 deletions

dev/design/executable-regex-callbacks.md

Lines changed: 21 additions & 32 deletions
Original file line numberDiff line numberDiff line change
@@ -2,11 +2,10 @@
22

33
## Status
44

5-
- **Current phase:** Phase 0design and differential baseline
5+
- **Current phase:** Phase 3backtracking and dynamic scope
66
- **Started:** 2026-08-09
7-
- **Implementation status:** Not started
8-
- **Prerequisite:** PR #895 (`feature/cpan-workaround-cleanup`) or an equivalent
9-
backend-neutral `RegexMatcher` integration
7+
- **Implementation status:** Callout engine and plain callback bridge integrated
8+
- **Prerequisite:** Satisfied by the namespaced callout-enabled Joni integration
109
- **Primary targets:** `(?{ ... })`, `(?(?{ ... })yes|no)`, and `(??{ ... })`
1110

1211
## Decision Summary
@@ -617,46 +616,36 @@ record the observed output in this document as phases proceed.
617616

618617
## Progress Tracking
619618

620-
### Current Status: Phase 0 in progress
619+
### Current Status: Phase 3 in progress
621620

622621
### Completed Phases
623622

624-
- [ ] Phase 0: Differential semantics and Joni spike
625-
- [ ] Phase 1: Structured frontend and runtime template
626-
- [ ] Phase 2: Plain `(?{ ... })`
623+
- [x] Phase 0: Differential semantics and Joni spike
624+
- [x] Phase 1: Structured frontend and runtime template
625+
- [x] Phase 2: Plain `(?{ ... })`
627626
- [ ] Phase 3: Backtracking and dynamic scope
628-
- [ ] Phase 4: Callback conditions
627+
- [x] Phase 4: Callback conditions
629628
- [ ] Phase 5: `(??{ ... })` dynamic programs
630629
- [ ] Phase 6: Runtime source, hardening, and policy removal
631630

632-
### Work Completed
633-
634-
- 2026-08-09: Created this design after reviewing the current parser,
635-
`RegexPreprocessor`, `RuntimeRegex`, the Joni 2.2.7 API, PR #895's matcher
636-
abstraction, and the executable-regex CPAN policies.
637-
- 2026-08-09: Selected a structured callback template plus generic Joni callout
638-
extension as the preferred architecture.
639-
- 2026-08-09: Corrected the older blanket-side-effect-journaling proposal: Perl
640-
dynamic locals require backtracking unwind, but ordinary side effects must not
641-
all be reverted.
642-
643631
### Next Steps
644632

645-
1. Write and validate `regex_executable_callbacks.t` with standard Perl.
646-
2. Record exact standard-Perl outputs for every open semantic question reachable
647-
without implementation.
648-
3. Create a disposable Joni 2.2.7 callout spike and measure the patch surface.
649-
4. Decide whether to upstream the generic callout API or publish a namespaced fork.
650-
5. Begin Phase 1 only after PR #895's matcher abstraction is merged or rebased into
651-
the implementation branch.
633+
1. Preserve lexical regex flags and package metadata for runtime/interpolated
634+
executable source.
635+
2. Close nested callback caller/source-line and interpolated `qr//` `__SUB__`
636+
identity gaps.
637+
3. Add warning-location, interruption, timeout, and nested-exception gates.
638+
4. Classify tied, magical, shared, and readonly mutation behavior with standard
639+
Perl before extending matcher transactions.
640+
5. Finish dynamic-pattern recursion/caching gates and then remove only the
641+
capability policies justified by unchanged-source results.
652642

653643
### Blockers
654644

655-
- PR #895 is still the integration prerequisite for the planned engine routing.
656-
- Joni 2.2.7 does not expose an in-match callback extension point; Phase 0 must
657-
validate the maintained-fork approach.
658-
- Several detailed Perl semantics remain intentionally open pending differential
659-
tests.
645+
- Runtime-injected callback source still requires `use re 'eval'` propagation.
646+
- Recursive callback frames do not yet retain all Perl caller source lines.
647+
- Tied, magical, shared, and readonly rollback semantics remain intentionally
648+
open pending differential tests.
660649

661650
## Related Documents and Skills
662651

dev/design/phase36-regex-parity.md

Lines changed: 23 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -242,12 +242,15 @@ timing delta is a regression only after a serialized same-commit reproduction.
242242

243243
The merged Joni dynamic-pattern engine establishes the Stage 36.5 execution
244244
seam. The current Stage 36.4 core baseline is `rxcode.t` 42/42 and
245-
`reg_eval_scope.t` 22/49, with no timeout or incomplete file. Matcher-owned
246-
transactions now restore ordinary scalar, array, and hash mutations when the
247-
overall match fails, retain mutations from abandoned alternatives when another
248-
alternative succeeds, and commit successful matches. Callback exceptions also
249-
restore dynamic locals, provisional match state, and `$^R` on both execution
250-
backends. Regex stringification no longer exposes private callback IDs.
245+
`reg_eval_scope.t` 33/49, with no timeout or incomplete file. Matcher-owned
246+
transactions restore ordinary scalar, array, and hash mutations on total
247+
failure while retaining Perl's ordinary side effects from attempted paths.
248+
Callback dynamic locals now transfer from the implementation CV to the matcher:
249+
they remain visible to later callbacks on the active path, unwind on
250+
backtracking, and restore after success, failure, or exception. Regex callbacks
251+
also behave as pseudo-blocks for `caller`, `__SUB__`, and escaping
252+
`last`/`next`/`goto` on both execution backends. Named unary `scalar` now keeps a
253+
following match in scalar context, including callback-bearing matches.
251254

252255
### Completed stages
253256

@@ -262,19 +265,25 @@ backends. Regex stringification no longer exposes private callback IDs.
262265

263266
### Next steps
264267

265-
1. Complete callback lexical pragma, caller-frame, and control-flow isolation
266-
exposed by `reg_eval_scope.t`, without changing its thread wrapper.
267-
2. Extend the callback semantic matrix with interruption, timeout, nested
268-
exception paths; require identical JVM/interpreter cleanup.
269-
3. Define and implement the mutation policy for tied, magical, shared, and
268+
1. Preserve lexical package and `use re '/flags'` state for `qr//`, interpolated
269+
regex objects, and runtime source admitted by `use re 'eval'`.
270+
2. Extend pseudo-block frame mapping through nested and recursive callbacks;
271+
preserve exact caller source lines and enclosing `__SUB__` for interpolated
272+
`qr//` values.
273+
3. Extend the callback semantic matrix with warning locations, interruption,
274+
timeout, and nested exception paths; require identical JVM/interpreter cleanup.
275+
4. Define and implement the mutation policy for tied, magical, shared, and
270276
readonly values; ordinary values are now transactionally covered.
271-
4. Complete the merged dynamic-pattern validation gates, then mark Stage 36.5
277+
5. Complete the merged dynamic-pattern validation gates, then mark Stage 36.5
272278
complete and proceed to the remaining declarative parity slices.
273279

274280
### Open blockers
275281

276-
- Callback lexical pragmata, caller frames, and non-local control-flow
277-
boundaries still differ from Perl in `reg_eval_scope.t`.
282+
- Runtime-injected callback source and lexical regex pragmata remain unsupported;
283+
these account for tests 4, 5, 8, 10, 11, and 12 in `reg_eval_scope.t`.
284+
- Recursive/nested callback caller lines, interpolated `qr//` `__SUB__`, warning
285+
locations, and the legacy `qr/\(?{` diagnostic account for the remaining
286+
Stage 36.4 failures.
278287
- Tied, magical, shared, and readonly callback mutation rollback remains
279288
intentionally outside the ordinary-value transaction until its exact Perl
280289
behavior is established with differential tests.

src/main/java/org/perlonjava/backend/bytecode/BytecodeCompiler.java

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5915,6 +5915,12 @@ private void visitAnonymousSubroutine(SubroutineNode node) {
59155915
subCode.isMapGrepBlock = true;
59165916
subCode.inheritsSelfReference = true;
59175917
}
5918+
if (node.getBooleanAnnotation("inheritsSelfReference")) {
5919+
subCode.inheritsSelfReference = true;
5920+
}
5921+
if (node.getBooleanAnnotation("regexCallbackPseudoBlock")) {
5922+
subCode.isRegexCallbackPseudoBlock = true;
5923+
}
59185924

59195925
if (RuntimeCode.isDisassemble()) {
59205926
System.out.println(Disassemble.disassemble(subCode));

src/main/java/org/perlonjava/backend/bytecode/BytecodeInterpreter.java

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3003,7 +3003,7 @@ private static RuntimeList execute(SuspendedInterpreterFrame frame) {
30033003
frame.suspendedDynamicStates =
30043004
DynamicVariableManager.suspendAbove(savedLocalLevel);
30053005
} else {
3006-
DynamicVariableManager.popToLocalLevel(savedLocalLevel);
3006+
DynamicVariableManager.teardownFrameToLocalLevel(savedLocalLevel);
30073007
}
30083008
currentPackageScalar.set(savedPackage);
30093009
if (frame.suspended && !frame.evalCatchStack.isEmpty()) {

src/main/java/org/perlonjava/backend/jvm/EmitBlock.java

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -261,7 +261,9 @@ public static void emitBlock(EmitterVisitor emitterVisitor, BlockNode node) {
261261
}
262262

263263
// Setup 'local' environment if needed
264-
Local.localRecord localRecord = Local.localSetup(emitterVisitor.ctx, node, mv, true);
264+
Local.localRecord localRecord = node.getBooleanAnnotation("regexCallbackBody")
265+
? new Local.localRecord(false, -1)
266+
: Local.localSetup(emitterVisitor.ctx, node, mv, true);
265267

266268
int regexStateLocal = -1;
267269
if (!node.getBooleanAnnotation("blockIsSubroutine")

src/main/java/org/perlonjava/backend/jvm/EmitSubroutine.java

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -505,6 +505,34 @@ public static void emitSubroutine(EmitterContext ctx, SubroutineNode node) {
505505
"(Lorg/perlonjava/runtime/runtimetypes/RuntimeScalar;Lorg/perlonjava/runtime/runtimetypes/RuntimeScalar;)V",
506506
false);
507507
}
508+
if (node.getBooleanAnnotation("inheritsSelfReference")
509+
&& !(isMapGrepBlock != null && isMapGrepBlock)) {
510+
mv.visitInsn(Opcodes.DUP);
511+
mv.visitVarInsn(Opcodes.ALOAD, 0);
512+
mv.visitFieldInsn(Opcodes.GETFIELD,
513+
ctx.javaClassInfo.javaClassName,
514+
"__SUB__",
515+
"Lorg/perlonjava/runtime/runtimetypes/RuntimeScalar;");
516+
mv.visitMethodInsn(Opcodes.INVOKESTATIC,
517+
"org/perlonjava/runtime/runtimetypes/RuntimeCode",
518+
"inheritSelfReference",
519+
"(Lorg/perlonjava/runtime/runtimetypes/RuntimeScalar;Lorg/perlonjava/runtime/runtimetypes/RuntimeScalar;)V",
520+
false);
521+
}
522+
if (node.getBooleanAnnotation("regexCallbackPseudoBlock")) {
523+
mv.visitInsn(Opcodes.DUP);
524+
mv.visitFieldInsn(Opcodes.GETFIELD,
525+
"org/perlonjava/runtime/runtimetypes/RuntimeScalar",
526+
"value",
527+
"Ljava/lang/Object;");
528+
mv.visitTypeInsn(Opcodes.CHECKCAST,
529+
"org/perlonjava/runtime/runtimetypes/RuntimeCode");
530+
mv.visitInsn(Opcodes.ICONST_1);
531+
mv.visitFieldInsn(Opcodes.PUTFIELD,
532+
"org/perlonjava/runtime/runtimetypes/RuntimeCode",
533+
"isRegexCallbackPseudoBlock",
534+
"Z");
535+
}
508536

509537
// Set isEvalBlock on the RuntimeCode so RuntimeCode.apply() propagates
510538
// non-local returns through eval BLOCK boundaries

src/main/java/org/perlonjava/backend/jvm/Local.java

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,7 @@ static void localTeardown(int dynamicIndex, MethodVisitor mv) {
2222
mv.visitVarInsn(Opcodes.ILOAD, dynamicIndex);
2323
mv.visitMethodInsn(Opcodes.INVOKESTATIC,
2424
"org/perlonjava/runtime/runtimetypes/DynamicVariableManager",
25-
"popToLocalLevel",
25+
"teardownFrameToLocalLevel",
2626
"(I)V",
2727
false);
2828
}

src/main/java/org/perlonjava/frontend/parser/OperatorParser.java

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -800,7 +800,15 @@ static OperatorNode parseKeys(Parser parser, LexerToken token, int currentIndex)
800800
// Named unary operators have precedence between 20 and 21 in Perl
801801
// This allows expressions like: values $hashref->%* or keys $hashref->%* or scalar((nil) x 3, 1)
802802
if (operator.equals("scalar") || operator.equals("values") || operator.equals("keys") || operator.equals("each")) {
803-
operand = parser.parseExpression(parser.getPrecedence("=~")); // precedence 20
803+
// parseExpression stops before an operator whose precedence is
804+
// equal to the supplied floor. Named unary scalar binds across a
805+
// following =~ / !~ (`scalar $s =~ /(...)/`) and must force that
806+
// match into scalar context rather than letting an enclosing print
807+
// put it in list context. The other named unary operators retain
808+
// their existing match-level boundary.
809+
int operandPrecedence = parser.getPrecedence("=~")
810+
- (operator.equals("scalar") ? 1 : 0);
811+
operand = parser.parseExpression(operandPrecedence);
804812
// Check if operand is null (no argument provided)
805813
if (operand == null) {
806814
throw new PerlCompilerException(currentIndex, "Not enough arguments for " + operator, parser.ctx.errorUtil);

src/main/java/org/perlonjava/frontend/parser/StringSegmentParser.java

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -962,6 +962,14 @@ private void parseRegexCodeBlock(boolean isRecursive) {
962962

963963
private Node regexCallback(Node block, String kind, int index) {
964964
SubroutineNode closure = new SubroutineNode(null, null, null, block, false, index);
965+
closure.setAnnotation("inheritsSelfReference", true);
966+
closure.setAnnotation("regexCallbackPseudoBlock", true);
967+
if (block instanceof AbstractNode abstractBlock) {
968+
// (?{ ... }) is a regex pseudo-block, not an ordinary anonymous-sub
969+
// scope. Its top-level local() frames belong to the matcher path and
970+
// must survive the Java callback return until Joni commits/unwinds it.
971+
abstractBlock.setAnnotation("regexCallbackBody", true);
972+
}
965973
OperatorNode callback = new OperatorNode("regexCallback", closure, index);
966974
callback.setAnnotation("regexCallbackKind", kind);
967975
hasExecutableRegexCallbacks = true;

src/main/java/org/perlonjava/runtime/regex/JoniRegexPattern.java

Lines changed: 57 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -392,7 +392,13 @@ public boolean find() {
392392
calloutHandler = new PerlCalloutHandler(input, byteToChar, callbacks, flags);
393393
matcher.setCalloutHandler(calloutHandler);
394394
}
395-
int result = matcher.search(charToByte[nextStart], charToByte[regionEnd], Option.NONE);
395+
int result;
396+
try {
397+
result = matcher.search(charToByte[nextStart], charToByte[regionEnd], Option.NONE);
398+
} catch (RuntimeException | Error failure) {
399+
if (calloutHandler != null) calloutHandler.abort();
400+
throw failure;
401+
}
396402
matched = result >= 0;
397403
if (calloutHandler != null) calloutHandler.finish(matched);
398404
if (!matched) return false;
@@ -511,6 +517,7 @@ private record Token(int localLevel, RegexState regexState, RuntimeScalar previo
511517
private final List<RuntimeRegexCallback> callbacks;
512518
private final RegexFlags outerFlags;
513519
private final RegexCallbackMutationSnapshot mutations;
520+
private final int initialLocalLevel;
514521
private RuntimeScalar completedResult;
515522

516523
PerlCalloutHandler(String input, int[] byteToChar, List<RuntimeRegexCallback> callbacks,
@@ -528,6 +535,7 @@ private PerlCalloutHandler(String input, int[] byteToChar,
528535
this.callbacks = callbacks;
529536
this.outerFlags = outerFlags;
530537
this.mutations = mutations;
538+
this.initialLocalLevel = DynamicVariableManager.getLocalLevel();
531539
for (RuntimeRegexCallback callback : callbacks) mutations.include(callback.code);
532540
}
533541

@@ -589,8 +597,17 @@ private Evaluation evaluate(RuntimeRegexCallback callback, MatchView match) {
589597
publishProvisional(match);
590598

591599
try {
592-
RuntimeScalar result = RuntimeCode.apply(new RuntimeScalar(callback.code),
593-
new RuntimeArray(), RuntimeContextType.SCALAR).scalar();
600+
DynamicVariableManager.CapturedFrame<RuntimeList> frame =
601+
DynamicVariableManager.captureFrameLocals(() -> RuntimeCode.apply(
602+
new RuntimeScalar(callback.code), new RuntimeArray(),
603+
RuntimeContextType.SCALAR));
604+
// Joni's complete() notification is delayed until the candidate
605+
// path commits. Resume now so a later (?{ ... }) on that same
606+
// path observes local() values; unwind() still owns the token's
607+
// pre-callback level and rolls the frame back on backtracking.
608+
DynamicVariableManager.resumeSuspended(frame.states());
609+
rejectEscapedControlFlow(frame.result());
610+
RuntimeScalar result = frame.result().scalar();
594611
boolean block = callback.kind == RuntimeRegexCallback.Kind.BLOCK;
595612
if (block) rVariable.set(result);
596613
Token token = new Token(localLevel, savedRegex, previousR,
@@ -617,15 +634,32 @@ public void complete(Object value) {
617634
}
618635

619636
void finish(boolean matched) {
620-
if (!matched) mutations.restore();
621-
if (matched && completedResult != null) {
622-
GlobalVariable.getGlobalVariable(GlobalContext.encodeSpecialVar("R"))
623-
.set(completedResult);
637+
try {
638+
if (!matched) mutations.restore();
639+
if (matched && completedResult != null) {
640+
GlobalVariable.getGlobalVariable(GlobalContext.encodeSpecialVar("R"))
641+
.set(completedResult);
642+
}
643+
} finally {
644+
DynamicVariableManager.popToLocalLevel(initialLocalLevel);
645+
}
646+
}
647+
648+
void abort() {
649+
try {
650+
mutations.restore();
651+
} finally {
652+
DynamicVariableManager.popToLocalLevel(initialLocalLevel);
624653
}
625654
}
626655

627656
private void restore(Token token, boolean completed) {
628-
restoreCallbackScope(token.localLevel(), token.regexState(), token.previousR());
657+
if (!completed) {
658+
DynamicVariableManager.popToLocalLevel(token.localLevel());
659+
}
660+
token.regexState().restore();
661+
GlobalVariable.getGlobalVariable(GlobalContext.encodeSpecialVar("R"))
662+
.set(token.previousR());
629663
if (completed && token.block() && completedResult == null) {
630664
completedResult = token.result();
631665
}
@@ -642,6 +676,21 @@ private static void restoreCallbackScope(int localLevel, RegexState regexState,
642676
}
643677
}
644678

679+
private static void rejectEscapedControlFlow(RuntimeList result) {
680+
if (!(result instanceof RuntimeControlFlowList flow)) return;
681+
ControlFlowMarker marker = flow.marker;
682+
if (marker.type == ControlFlowType.GOTO
683+
|| marker.type == ControlFlowType.TAILCALL) {
684+
// The runtime location is the regex pseudo-block boundary (and
685+
// can differ from the marker's inner goto location), so let the
686+
// exception formatter attach it.
687+
throw new PerlCompilerException("Can't \"goto\" out of a pseudo block");
688+
}
689+
// Preserve the control op's own location. The terminating newline
690+
// tells PerlCompilerException this is already fully formatted.
691+
throw new PerlCompilerException(marker.buildErrorMessage() + ".\n");
692+
}
693+
645694
private void publishProvisional(MatchView match) {
646695
RuntimeRegexState state = PerlRuntime.current().regexState;
647696
int count = match.captureCount();

0 commit comments

Comments
 (0)