Merge pull request #949 from fglock/fix/cpan-compiler-tooling

Fix CPAN compiler and XS tooling compatibility

Author: fglock

dev/design/cpan-compiler-tooling-batch-20260813.md

@@ -0,0 +1,147 @@
+# CPAN compiler and tooling compatibility batch (2026-08-13)
+
+## Goal
+
+Make the system-Perl-compatible surfaces of these targets work through
+`jcpan`, fixing shared compiler, runtime, and CPAN tooling behavior instead of
+adding distribution preferences:
+
+- `Net::FS::Flickr`
+- `Marlin::X::Clone`
+- `Sys::GetRandom::PP`
+- `App::calendr`
+- `App::upf`
+- `Hades::Realm::OO`
+- `Authen::Simple::Kerberos`
+
+Reuse bundled Java providers where native CPAN dependencies need equivalent
+primitives.
+
+## Baseline
+
+| Target | Initial PerlOnJava result |
+| --- | --- |
+| Net::FS::Flickr | dependency resolution reached Imager/Flickr::Upload, then a sandboxed download was blocked |
+| Marlin::X::Clone | `Class::XSConstructor` has no Java or pure-Perl XS replacement |
+| Sys::GetRandom::PP | rejects both Darwin system Perl and PerlOnJava as unsupported platforms |
+| App::calendr | Moo's ithread type map cannot resolve a raw `B::SV` address |
+| App::upf | long Data::Sah/Perinci dependency resolution reached a sandboxed download |
+| Hades::Realm::OO | Hades compiles a self-recursive `(??{ ... })` pattern |
+| Authen::Simple::Kerberos | `Authen::Krb5::Simple` has no Java XS replacement |
+
+The isolated system-Perl comparison passed `Marlin::X::Clone`,
+`App::calendr`, `Hades::Realm::OO`, and `Authen::Simple::Kerberos`.
+`Net::FS::Flickr` was excluded because its upstream
+`Acme::Steganography::Image::Png` dependency fails its own system-Perl tests.
+`Sys::GetRandom::PP` was excluded because the distribution rejects Darwin.
+The `App::upf` comparison exceeded its 20-minute bounded dependency run and
+was kept in the PerlOnJava validation set.
+
+## Progress tracking
+
+### Current status: Complete, reconciled with current master (2026-08-14)
+
+### Completed phases
+
+- [x] Initial `jcpan -t` classification (2026-08-13)
+  - Captured full per-target logs with hard process timeouts.
+  - Excluded `Sys::GetRandom::PP` because its current distribution rejects
+    Darwin under system Perl as well.
+- [x] System-Perl differential (2026-08-13)
+  - Excluded the two upstream failures described above.
+  - Kept `App::upf` in scope after the bounded comparison timed out.
+- [x] Shared runtime and CPAN tooling fixes (2026-08-13)
+  - Added bounded, process-wide address-to-reference recovery for
+    `B::SV::object_2svref`, including addresses exposed through
+    `Scalar::Util::refaddr` across compiler/runtime threads.
+  - Made `jcpan` load its targeted `Module::Build::Base` compatibility
+    overlay ahead of stale site-installed copies.
+  - Made explicit target failures produce a failing `jcpan` exit status.
+  - Deferred unsupported dynamic-regex diagnostics from `qr//` construction
+    to first match, allowing modules to define unused patterns honestly.
+  - Added pure-Perl XS overlays for `Class::XSConstructor` and
+    `B::Hooks::AtRuntime::OnlyCoreDependencies`.
+  - Applied one-shot source filters immediately after explicit `BEGIN` blocks
+    and preserved localized at-runtime callback arrays across whole-file
+    tokenization.
+  - Added a JDK Kerberos backend for `Authen::Krb5::Simple`.
+  - Files: `B.pm`, `Module/Build/Base.pm`, `App/Cpan.pm`, `XSLoader.java`,
+    `RuntimeRegex.java`, runtime reference types, and new provider overlays.
+- [x] Target and dependency validation except `App::upf` (2026-08-13)
+  - `Marlin::X::Clone`: 3 files, 8 tests passed.
+  - `Class::XSConstructor`: 27 files, 134 tests passed (two optional suites
+    skipped for unavailable dependencies).
+  - `B::Hooks::AtRuntime::OnlyCoreDependencies`: 3 files, 12 tests passed.
+  - `App::calendr`: 2 files, 2 tests passed; its optional
+    `Calendar::Bahai` suite was skipped upstream.
+  - `Hades::Realm::OO`: 5 files, 40 tests passed (three author-only suites
+    skipped upstream).
+  - `Authen::Simple::Kerberos`: functional load test passed; two author-only
+    POD suites skipped for unavailable test dependencies.
+  - Full project `make` passed after the final runtime changes.
+- [x] Bounded `App::upf` validation (2026-08-13)
+  - A clean run progressed through QuickJS configuration and more than 4,000
+    lines of Data::Sah/Perinci dependency resolution without reaching the
+    target distribution or exposing a PerlOnJava compiler/runtime failure.
+  - The equivalent isolated system-Perl run also did not complete its
+    dependency graph within 20 minutes, so no target-test differential was
+    available. The PerlOnJava run was stopped after exceeding the intended
+    bound while still resolving dependencies.
+- [x] Core-suite regression follow-up (2026-08-13)
+  - Rebased the branch onto current master and removed the obsolete whole-file
+    source-filter pre-pass. Explicit `BEGIN` filters now run only through the
+    parser-time path, fixing the double filtering seen in `op/incfilter.t`.
+  - Preserved the existing `JPERL_UNIMPLEMENTED=warn` dynamic-regex fallback
+    while retaining normal-mode deferred errors for unused `(??{...})`
+    patterns. Regex cache entries now distinguish those modes.
+  - Replaced the bounded strong B-address registry with weak entries so
+    address recovery does not extend Perl value lifetimes.
+  - Exact runs against an isolated current-master build matched the repaired
+    branch for the remaining reported core-test counts. `op/incfilter.t`
+    improved from the regressed 14 tests to 158 passing assertions.
+  - Revalidated `Hades::Realm::OO` (40 tests),
+    `B::Hooks::AtRuntime::OnlyCoreDependencies` (12 tests), `App::calendr`
+    (2 tests), and `Authen::Simple::Kerberos` (functional suite); all passed.
+  - Full project `make` passed after the final regression repairs.
+- [x] Upstream reconciliation (2026-08-14)
+  - Rebuilt PR #949 on current `origin/master`, which now contains the shared
+    regex, warning, tied-array, and other core compatibility repairs.
+  - Retained only this batch's CPAN compiler, source-filter, runtime-provider,
+    and tooling changes; dropped the branch-local core regression commit so
+    the upstream implementations remain authoritative.
+  - Preserved the source-filter and weak-reference corrections that are part
+    of the CPAN objectives.
+  - Full `make` passed. The reported core-test set matched an isolated
+    `origin/master` build; `re/speed.t` varied only with its process timeout
+    and reproduced the upstream 25 passing assertions on a quiet repeat.
+  - Revalidated `Marlin::X::Clone` (8 tests), `App::calendr` (2 tests),
+    `Hades::Realm::OO` (40 tests), and `Authen::Simple::Kerberos` (1 functional
+    test); all passed.
+- [x] Contention-sensitive regression tooling (2026-08-14)
+  - Confirmed the reported `re/speed*.t` losses coincided with concurrent
+    PerlOnJava regression runs and CPAN JVMs from other worktrees; the tests'
+    internal watchdogs expired before the runner's outer timeout.
+  - Applied the runner's existing timeout factor to `re/speed*.t`, keeping the
+    watchdog inside the 600-second resource-sensitive allowance.
+  - Increased the `gh7094` Benchmark.pm sample from three to five CPU seconds
+    to stabilize its global-versus-lexical hash ratios under JVM contention.
+  - Verified `re/pat_advanced.t` at 1376/1687 and the benchmark at 6/6 before
+    the tooling change; `run/switches.t` matched current master at 76/142.
+  - Under continued CPAN JVM load, the adjusted runner reported the benchmark
+    at 6/6, `re/speed.t` at the 26/59 baseline, and `re/speed_thr.t` at 25/59
+    versus the reported 18/59 baseline.
+
+### Next steps
+
+1. Update PR #949 and monitor CI.
+
+### Open questions
+
+- `App::upf` has an unusually large dependency graph; neither system Perl nor
+  PerlOnJava reached its target tests in the bounded comparison.
+
+## References
+
+- [Module porting guide](../../docs/guides/module-porting.md)
+- [Executable regex callbacks](executable-regex-callbacks.md)
+- Skills: `debug-perlonjava`, `port-cpan-module`

dev/tools/perl_test_runner.pl

@@ -257,6 +257,7 @@ sub run_single_test {
     if ($test_file =~ m{
               (?:^|/)perl5_t/t/op/gv\.t$
             | (?:^|/)perl5_t/t/re/pat_advanced(?:_thr)?\.t$
+            | (?:^|/)perl5_t/t/re/speed(?:_thr)?\.t$
         }x
             && (!defined($ENV{PERL_TEST_TIMEOUT_FACTOR})
                 || $ENV{PERL_TEST_TIMEOUT_FACTOR} < 2)) {
@@ -364,6 +365,13 @@ sub run_single_test {
     # Use absolute path for jperl
     my $abs_jperl = File::Spec->rel2abs($jperl_path, $old_dir);
     my $test_name = File::Spec->abs2rel($test_file, $local_test_dir || '.');
+    # Benchmark.pm's default three-CPU-second sample is too noisy while other
+    # JVM builds or CPAN testers are active. A five-second sample still fits
+    # the resource-sensitive test's 600-second runner allowance and makes its
+    # relative global/lexical hash comparisons substantially more stable.
+    my $test_args = $test_file =~ m{
+        (?:^|/)perl5_t/t/benchmark/gh7094-speed-up-keys-on-empty-hash\.t$
+    }x ? ' -5' : '';
 
     # Try to use system timeout command if available.
     # Use --kill-after (-k) so a SIGTERM that the JVM ignores is followed
@@ -388,7 +396,7 @@ sub run_single_test {
     # Never inherit an interactive terminal as stdin: a test that reads from
     # it would receive SIGTTIN and stop indefinitely as a background group.
     my $devnull = File::Spec->devnull();
-    my $cmd = "${timeout_cmd}$abs_jperl $test_name < $devnull 2>&1";
+    my $cmd = "${timeout_cmd}$abs_jperl $test_name$test_args < $devnull 2>&1";
 
     # Capture output with timeout
     my $output = '';

jcpan

@@ -113,6 +113,11 @@ fi
 # pkill'd. See AGENTS.md "ALWAYS WRAP jperl/jcpan IN timeout" rule.
 export JPERL_ORPHAN_EXIT=1
 
+# CPAN build tools may install copies of their own implementation modules into
+# the user library.  Keep PerlOnJava's narrow compatibility overlays ahead of
+# those copies while jcpan and its child build processes run.
+export PERLONJAVA_PREFER_BUNDLED_MODULES="Module/Build/Base.pm${PERLONJAVA_PREFER_BUNDLED_MODULES:+,$PERLONJAVA_PREFER_BUNDLED_MODULES}"
+
 # CPAN test suites should run with deterministic semantics. User-interface
 # color preferences such as NO_COLOR can change module behavior under test
 # (for example Color::ANSI::Util), so do not pass them into jcpan runs.

src/main/java/org/perlonjava/app/cli/CompilerOptions.java

@@ -43,7 +43,6 @@ public class CompilerOptions implements Cloneable {
     public boolean tokenizeOnly = false;
     public boolean parseOnly = false;
     public boolean compileOnly = false;
-    public boolean applySourceFilters = false; // Enable BEGIN filter preprocessing
     public boolean processOnly = false; // For -n
     public boolean processAndPrint = false; // For -p
     public boolean inPlaceEdit = false; // New field for in-place editing

src/main/java/org/perlonjava/app/scriptengine/PerlLanguageProvider.java

@@ -21,7 +21,6 @@
 import org.perlonjava.frontend.semantic.ScopedSymbolTable;
 import org.perlonjava.runtime.io.StandardIO;
 import org.perlonjava.runtime.perlmodule.BHooksEndOfScope;
-import org.perlonjava.runtime.perlmodule.FilterUtilCall;
 import org.perlonjava.runtime.perlmodule.Strict;
 import org.perlonjava.runtime.runtimetypes.*;
 import org.perlonjava.runtime.WarningBitsRegistry;
@@ -227,11 +226,6 @@ public static RuntimeList executePerlCode(CompilerOptions compilerOptions,
         if (CompilerOptions.DEBUG_ENABLED) ctx.logDebug("parse code: " + compilerOptions.code);
         if (CompilerOptions.DEBUG_ENABLED) ctx.logDebug("  call context " + ctx.contextType);
 
-        // Apply any BEGIN-block filters before tokenization if requested
-        // This is a workaround for the limitation that our architecture tokenizes all source upfront
-        if (compilerOptions.applySourceFilters) {
-            compilerOptions.code = FilterUtilCall.preprocessWithBeginFilters(compilerOptions.code);
-        }
         compilerOptions.deparseSourceCode = compilerOptions.code;
 
         // Create the LexerToken list

src/main/java/org/perlonjava/frontend/parser/SpecialBlockParser.java

@@ -9,6 +9,7 @@
 import org.perlonjava.frontend.semantic.ScopedSymbolTable;
 import org.perlonjava.frontend.semantic.SymbolTable;
 import org.perlonjava.runtime.HintHashRegistry;
+import org.perlonjava.runtime.perlmodule.FilterUtilCall;
 import org.perlonjava.runtime.runtimetypes.*;
 
 import java.util.ArrayList;
@@ -175,6 +176,12 @@ static Node parseSpecialBlock(Parser parser) {
         // but never emit `local ${^WARNING_SCOPE} = N`, so warnings::warnif would
         // not honor the suppression at runtime.
         if ("BEGIN".equals(blockName)) {
+            // Source filters may be installed by code inside an explicit
+            // BEGIN block (B::Hooks::AtRuntime uses this to inject callbacks
+            // at the precise transition from compile time to runtime).
+            if (FilterUtilCall.wasFilterInstalled()) {
+                StatementParser.applySourceFilterToRemainingTokens(parser);
+            }
             int warningScopeId = WarningFlags.getLastScopeId();
             WarningFlags.clearLastScopeId();
 

src/main/java/org/perlonjava/frontend/parser/StatementParser.java

@@ -1439,7 +1439,7 @@ public static Node parseOptionalPackageVersion(Parser parser) {
      *
      * @param parser The Parser instance with tokens to filter
      */
-    private static void applySourceFilterToRemainingTokens(Parser parser) {
+    static void applySourceFilterToRemainingTokens(Parser parser) {
         int currentPos = parser.tokenIndex;
 
         // Step 1: Rejoin remaining tokens back to source text