Complete Perl threads phases 31-34

dev/design/concurrency.md

@@ -571,27 +571,33 @@ Acceptance: `ipc_wait_timeout.t` observes Perl-compatible inherited-pipe
 behavior; default Test2 remains green; `AUTHOR_TESTING` and
 `T2_DO_THREAD_TESTS` thread suites form a separately reported stress gate.
 
-### Phase 31 — Native callbacks and handle ownership
+### Phase 31 — Native callbacks and handle ownership (implemented 2026-08-13)
 
-Define clone, child-owned creation, or explicit rejection for every native
-handle class. Bind callbacks to their captured runtime and make provider/handle
-registries safe under concurrent child creation and deterministic cleanup.
+Net::SSLeay verification, info, and password callbacks are bound to the runtime
+that registered them, including invocation from foreign native callback
+threads. SSL session handles are runtime-owned and reset with their runtime.
+Detached children no longer appear in `threads->list`, and abnormal detached
+termination is reported exactly once. Other native handle classes retain their
+documented clone, child-owned creation, or explicit rejection policy.
 
 Acceptance: Net::SSLeay `61_threads-cb-crash.t` and
 `62_threads-ctx_new-deadlock.t` pass without watchdog, deadlock, cross-runtime
 handle leakage, or callback misbinding; applicable thread-emulated server paths
 in the wider Net::SSLeay suite retain their non-thread baseline.
 
-### Phase 32 — Advanced shared values
+### Phase 32 — Advanced shared values (implemented supported tranche 2026-08-13)
 
-Extend `threads::shared` only where identity, magic, tie callbacks, locking, and
-clone behavior are defined. Blessed, tied, or magical graphs remain explicit
-errors until their complete semantics are proven.
+Nested plain scalar/array/hash graphs now have atomic preflight before any node
+is published as shared. A rejected nested node therefore cannot leave a
+partially shared graph behind. Identity, mutation, recursive locking, and
+condition behavior are stress-tested across child threads. Blessed, tied, and
+other magical graphs remain explicit errors because their callback and
+destruction semantics are not yet a supported shared-value category.
 
 Acceptance: each newly supported value category has standard-Perl-validated
 identity, mutation, lock/condition, clone, destruction, and stress coverage.
 
-### Phase 33 — Compatibility completion, documentation, and examples
+### Phase 33 — Compatibility completion, documentation, and examples (implemented 2026-08-13)
 
 Run the complete applicable core, Test2, Storable, and native thread matrix;
 update the feature matrix from raw results; and add a realistic dynamic
@@ -601,9 +607,11 @@ aggregates through `join`.
 Acceptance: platform threads pass all supported tests on JVM and interpreter
 backends; every remaining skip is an explicit platform or unsupported-feature
 decision; virtual mode has no semantic delta; example output is deterministic
-across system Perl and all supported modes.
+across system Perl and all supported modes. The release gate also includes the
+complete `timeout 3600 ./jcpan --jobs 8 -t DBIx::Class` distribution suite; every DBIx
+test must pass before this phase is complete.
 
-### Phase 34 — Optional runtime pooling
+### Phase 34 — Optional runtime pooling (evaluated; deliberately disabled)
 
 Consider reusable runtimes only after the fresh-runtime equivalence contract is
 implemented in full. Pooling is neither a Perl threads requirement nor a reason
@@ -612,6 +620,16 @@ to weaken close/snapshot isolation.
 Acceptance: every item in `runtime-pooling-reset-contract.md` passes and reuse
 has a measured benefit over a fresh snapshot.
 
+The 2026-08-13 evaluation did not meet that activation threshold. The executable
+negative contract proves that `close()` is terminal and retains observable
+package, regex, and execution state. Pooling therefore remains disabled; fresh
+snapshot runtimes remain the correctness boundary.
+
+Phase 33's release gate completed with `./jcpan --jobs 8 -t DBIx::Class`:
+325 files and 42,671 assertions passed. The final compatibility fix ensures
+non-local labeled control flow tears down every abandoned Perl frame before the
+target resumes, preserving scope-guard diagnostics and redirected STDERR.
+
 ## 6. Known Reference Material and Warnings
 
 - `dev/prompts/multiplicity-v2-plan.md` documents the incremental response to
@@ -626,7 +644,7 @@ has a measured benefit over a fresh snapshot.
 
 ## 7. Progress Tracking
 
-### Current Status: Phases 25–30 implemented; integrated validation in progress
+### Current Status: Phases 31–34 complete for the supported tranche
 
 Hints, warnings, filters, and source maps are runtime-owned while compiler-only
 scratch remains protected by the global compile lock. The Phase 11 inventory is
@@ -781,15 +799,17 @@ request history.
    `regexp_qr_embed` direct-language gaps before asserting thread equivalence.
 3. Finish Phase 29 shutdown warnings and detached platform-thread process
    lifecycle. Continue Phase 30 resource classification beyond explicitly
-   inherited internal pipes, then implement Phase 31 native callback/handle
-   ownership and Phase 32 shared value categories. Preserve the green anchors:
+   inherited internal pipes. Preserve the green anchors:
    `class/threads.t`, `threads-dirh.t`, Storable threads, and default Test2 IPC.
-4. Complete Phase 33 with the full platform-thread matrix on both backends,
-   followed by virtual-mode parity. The new
-   `examples/threads/dynamic_map_reduce.pl` demonstrates a small shared
-   scheduler, isolated worker-local hashes, and deterministic join aggregation;
-   it deliberately makes no performance claim.
-5. Keep virtual threads experimental until native callback diagnostics and
+4. Keep the complete platform-thread matrix green on both backends and retain
+   virtual-mode parity. The new `examples/threads/dynamic_map_reduce.pl`
+   demonstrates a small shared scheduler, isolated worker-local hashes, and
+   deterministic join aggregation; it deliberately makes no performance claim.
+5. Keep the DBIx::Class regression gate green with captured output from
+   `timeout 3600 ./jcpan --jobs 8 -t DBIx::Class`. Investigate every future
+   failure against the merged baseline; partial distribution results are not
+   sufficient for release.
+6. Keep virtual threads experimental until native callback diagnostics and
    repeated benchmarks justify promotion. Keep runtime pooling disabled until
    the separate Phase 34 reset contract proves fresh-runtime equivalence.
 

dev/design/runtime-pooling-reset-contract.md

@@ -2,7 +2,8 @@
 
 ## Status
 
-Runtime pooling is deferred. `PerlRuntime.close()` is a terminal resource-release
+Runtime pooling was re-evaluated on 2026-08-13 and remains deliberately
+disabled. `PerlRuntime.close()` is a terminal resource-release
 operation, not a reset operation, and a closed runtime deliberately rejects
 `bind`, `initialize`, and `execute`. Reusing it would currently expose state that
 a newly constructed runtime does not contain.
@@ -11,6 +12,11 @@ This document defines the proof required before a pool may be implemented. It
 does not authorize clearing state opportunistically or enabling pooling behind
 an experimental flag.
 
+This is the Phase 34 outcome, not an untracked implementation shortcut. Runtime
+pooling is optional and is not required for Perl ithread correctness. The
+negative automated contract below passes, while every positive equivalence
+item remains a prerequisite for any future pooling PR.
+
 ## Fresh-runtime equivalence
 
 A reusable runtime must expose the same observable state as a newly constructed

docs/about/changelog.md

@@ -17,8 +17,12 @@ Release history of PerlOnJava. See [Roadmap](roadmap.md) for future plans.
   locks, condition variables, and compatible imports/stringification.
   `Config` now reports `useithreads`, `usethreads`, and `usemultiplicity` as
   `define`. Platform threads remain the default and virtual threads are an
-  experimental opt-in. Thread signals, effective stack sizing, blessed/tied
-  shared values, and some upstream core/native-callback suites remain limited;
+  experimental opt-in. Live attached children support targeted signals,
+  `object`/`wantarray`, and platform-thread stack sizing. Native-style callback
+  registrations retain their owning runtime, internal pipes have an explicit
+  inherited-handle policy, and nested plain shared graphs are validated before
+  publication. Blessed/tied shared values and some upstream core/regex suites
+  remain limited;
   see the [feature matrix](../reference/feature-matrix.md#concurrency-and-perl-threads).
 - CPAN/tooling: expose tested dependency scripts through `PATH`, deduplicate
   repeated `PERL5LIB` setup, and resolve test prerequisites against tested

docs/about/roadmap.md

@@ -288,11 +288,12 @@ threads are the default; virtual threads are experimental.
 
 Remaining work:
 
-- Complete the currently partial applicable core suites and Storable thread test.
-- Implement thread signals and the remaining `object`/`wantarray` surface.
-- Decide whether effective per-thread stack sizing can be exposed safely.
-- Validate native callback/resource behavior and virtual-thread diagnostics on
-  the supported Java 24 baseline.
+- Complete the currently partial applicable core and regex suites.
+- Extend native-resource inheritance beyond the explicitly supported internal
+  pipe policy only when ownership and last-close semantics are defined.
+- Decide which blessed, tied, or magical values can safely join the supported
+  `threads::shared` tranche.
+- Keep runtime pooling disabled until the reset-equivalence contract is proven.
 
 ---
 

docs/reference/feature-matrix.md

@@ -862,11 +862,11 @@ storage as their parent counterparts. Values explicitly shared through
 
 | Limitation | Effect |
 |---|---|
-| Thread signals | `threads->kill` is not implemented. |
-| Effective stack sizing | Standard `stack_size` import syntax is accepted for source compatibility, but JVM stack sizing remains runtime-managed. |
-| Additional introspection | `threads->object` and `wantarray` are not implemented. |
+| Thread signals | `threads->kill` targets live attached children and resolves the handler inside the child runtime. Completed and detached targets are not signalable. |
+| Effective stack sizing | Platform-backed children honor supported `stack_size` create/import requests. Virtual threads reject nonzero stack sizes because their stacks are JVM-managed. |
+| Additional introspection | `threads->object` and creation-context `wantarray` are implemented; process-shutdown warning parity remains incomplete. |
 | Shared object classes | Blessed and tied values are rejected by the supported `share`/`shared_clone` tranche. |
-| Native resources and callbacks | Java I/O/native handles are not portably duplicated into child snapshots; native callback thread isolation remains suite-specific. |
+| Native resources and callbacks | Internal pipes have an inherited lease policy. Net::SSLeay handles are runtime-owned and stored callbacks bind their registering runtime. Ordinary files, sockets, and other native handles are still rejected rather than silently shared. |
 | Upstream suite coverage | Core compatibility remains partial: measured results include `op/threads.t` 29/30, `op/substr_thr.t` 368/400, and `re/stclass_threads.t` 2/6; `class/threads.t`, Storable's thread test, `threads-dirh.t`, and Test2's thread IPC acceptance test complete. The regex suite now executes its child and exposes unsupported thread-local `re 'debug'` trace parity. |
 | PSGI | Availability of ithreads does not make one captured PSGI application runtime concurrently callable. `Plack::Handler::Netty` advertises `psgi.multithread => \0`. |
 

examples/threads/README.md

@@ -39,6 +39,8 @@ JPERL_OPTS=-Djperl.thread.mode=virtual \
   ./jperl examples/threads/isolated_create_join.pl
 ```
 
-Thread signals, effective per-thread stack sizing, and sharing tied or blessed
-values are outside the currently supported tranche. A captured PSGI runtime is
-also not made concurrently callable merely by enabling ithreads.
+Live attached children support targeted thread signals. Platform-backed
+ithreads accept an effective Java stack-size request; virtual mode rejects a
+nonzero request because virtual-thread stacks are JVM-managed. Sharing tied or
+blessed values remains outside the supported tranche. A captured PSGI runtime
+is also not made concurrently callable merely by enabling ithreads.

src/main/java/org/perlonjava/backend/bytecode/BytecodeCompiler.java

@@ -1399,6 +1399,11 @@ public void visit(BlockNode node) {
             loopStack.pop();
         }
 
+        // A surrounding labeled-block dispatcher may need to enter this
+        // block's teardown after a nested sub/eval returns a LAST marker.
+        // Record the first cleanup opcode before regex/local/lexical unwind.
+        node.setAnnotation("interpreterNonLocalExitPc", bytecode.size());
+
         if (regexSaveReg >= 0) {
             emit(Opcodes.RESTORE_REGEX_STATE);
             emitReg(regexSaveReg);
@@ -6395,16 +6400,24 @@ public void visit(For3Node node) {
                     bodyStartPc, true);
             loopStack.push(loopInfo);
 
+            int nonLocalExitPc = -1;
             enterScope();
             try {
                 if (node.body != null) {
                     compileNode(node.body, outerResultReg, currentCallContext);
+                    Object cleanupPc = node.body.getAnnotation("interpreterNonLocalExitPc");
+                    if (cleanupPc instanceof Integer pc) {
+                        nonLocalExitPc = pc;
+                    }
                 }
                 if (outerResultReg >= 0 && lastResultReg >= 0) {
                     emitAliasWithTarget(outerResultReg, lastResultReg);
                 }
             } finally {
                 // Exit scope to clean up lexical variables
+                if (nonLocalExitPc < 0) {
+                    nonLocalExitPc = bytecode.size();
+                }
                 exitScope(true);  // safe to flush — foreach body, not subroutine
             }
 
@@ -6426,8 +6439,10 @@ public void visit(For3Node node) {
 
             if (node.labelName != null) {
                 emit(Opcodes.POP_LABELED_BLOCK);
-                int exitPc = bytecode.size();
-                patchJump(exitPcPlaceholder, exitPc);
+                // A marker returned by a nested sub/eval must enter at the
+                // lexical teardown sequence that ordinary fallthrough runs.
+                // Jumping past it leaks block lexicals and delays DESTROY.
+                patchJump(exitPcPlaceholder, nonLocalExitPc);
             }
 
             // Patch last (break) PCs to jump to local cleanup (or past the block if no locals).

src/main/java/org/perlonjava/backend/jvm/EmitControlFlow.java

@@ -38,7 +38,7 @@ private static boolean containsAggregateReferenceReturn(Node node) {
                 && (aggregateOp.operator.equals("@") || aggregateOp.operator.equals("%"));
     }
 
-    private static void emitSubroutineExitCleanup(EmitterContext ctx) {
+    static void emitSubroutineExitCleanup(EmitterContext ctx) {
         java.util.List<Integer> scalarIndices =
                 EmitStatement.withoutCaptured(ctx, ctx.symbolTable.getMyScalarIndicesInScope(0));
         java.util.List<Integer> hashIndices =
@@ -248,16 +248,36 @@ static void handleNextOperator(EmitterVisitor emitterVisitor, OperatorNode node)
         Label label = operator.equals("next") ? loopLabels.nextLabel
                 : operator.equals("last") ? loopLabels.lastLabel
                 : loopLabels.redoLabel;
-        emitLoopControlScopeCleanup(ctx, loopLabels);
+        emitLoopControlScopeCleanup(ctx, loopLabels, operator.equals("last"));
         emitMortalFlushAboveMark(ctx);
         ctx.mv.visitJumpInsn(Opcodes.GOTO, label);
     }
 
-    private static void emitLoopControlScopeCleanup(EmitterContext ctx, LoopLabels loopLabels) {
-        if (loopLabels.cleanupScopeIndex < 0) {
+    private static void emitLoopControlScopeCleanup(
+            EmitterContext ctx, LoopLabels loopLabels, boolean exitsLoop) {
+        int cleanupScopeIndex = exitsLoop && loopLabels.lastCleanupScopeIndex >= 0
+                ? loopLabels.lastCleanupScopeIndex
+                : loopLabels.cleanupScopeIndex;
+        if (cleanupScopeIndex < 0) {
             return;
         }
-        EmitStatement.emitScopeExitNullStores(ctx, loopLabels.cleanupScopeIndex, true);
+        EmitStatement.emitLoopControlScopeExit(ctx, cleanupScopeIndex);
+    }
+
+    static void emitLoopControlScopeCleanupForDispatcher(
+            EmitterContext ctx, LoopLabels loopLabels, boolean exitsLoop) {
+        if (loopLabels.cleanupMarkSlot >= 0) {
+            ctx.mv.visitVarInsn(Opcodes.ILOAD, loopLabels.cleanupMarkSlot);
+            ctx.mv.visitMethodInsn(Opcodes.INVOKESTATIC,
+                    "org/perlonjava/runtime/runtimetypes/MyVarCleanupStack",
+                    "unwindTo", "(I)V", false);
+        }
+        emitLoopControlScopeCleanup(ctx, loopLabels, exitsLoop);
+        ctx.mv.visitMethodInsn(Opcodes.INVOKESTATIC,
+                "org/perlonjava/runtime/runtimetypes/MortalList",
+                "flush",
+                "()V",
+                false);
     }
 
     private static void emitMortalFlushAboveMark(EmitterContext ctx) {

src/main/java/org/perlonjava/backend/jvm/EmitEval.java

@@ -368,12 +368,18 @@ static void handleEvalOperator(EmitterVisitor emitterVisitor, OperatorNode node)
                 mv.visitJumpInsn(Opcodes.GOTO, nextLabel);
 
                 mv.visitLabel(isLast);
+                EmitControlFlow.emitLoopControlScopeCleanupForDispatcher(
+                        emitterVisitor.ctx, loopLabels, true);
                 mv.visitJumpInsn(Opcodes.GOTO, loopLabels.lastLabel);
 
                 mv.visitLabel(isNext);
+                EmitControlFlow.emitLoopControlScopeCleanupForDispatcher(
+                        emitterVisitor.ctx, loopLabels, false);
                 mv.visitJumpInsn(Opcodes.GOTO, loopLabels.nextLabel);
 
                 mv.visitLabel(isRedo);
+                EmitControlFlow.emitLoopControlScopeCleanupForDispatcher(
+                        emitterVisitor.ctx, loopLabels, false);
                 mv.visitJumpInsn(Opcodes.GOTO, loopLabels.redoLabel);
 
                 mv.visitLabel(nextLabel);