Skip to content

Commit 5a37f7e

Browse files
fglockcodex
fglock
and
codex
committed
fix: improve CPAN compiler and tooling compatibility
Fix shared regex, lexical inspection, weak-reference, Unicode, DBI, and POSIX behavior exposed by a broad CPAN compatibility suite. Add a Java Math::Factor::XS implementation and narrowly scoped patches for legacy Char::Windows1258 and MooseX tooling. Document the completed compatibility work and cover the runtime changes with system-Perl-validated unit tests for both compiler backends. Generated with [Codex](https://openai.com/codex) Co-Authored-By: Codex <codex@openai.com>
1 parent 4de2934 commit 5a37f7e

34 files changed

Lines changed: 918 additions & 24 deletions

‎dev/modules/cpan_compiler_tooling_suite_2.md‎

Lines changed: 90 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,90 @@
1+
# CPAN compiler and tooling compatibility suite II
2+
3+
## Goal
4+
5+
Make `jcpan -t` work for Data::Collector, Pod::Query, Char::Windows1258,
6+
Map::Metro::Plugin::Map::Oslo, DBIx::Dictionary,
7+
Date::Holidays::Abstract, Music::Note::Role::Operators, and their
8+
dependencies. Fix reusable compiler and CPAN tooling defects first. A target
9+
that fails under the local system Perl may be left unsupported with the
10+
failure recorded.
11+
12+
## Baseline (2026-08-14)
13+
14+
| Target | First actionable result |
15+
|---|---|
16+
| Data::Collector | Passes: 3 files / 12 tests. |
17+
| Pod::Query | Empty `qr//` values interpolated from hashes incorrectly reuse the previous successful match. |
18+
| Char::Windows1258 | All 210 files abort because upstream deliberately rejects an executable whose `$^X` contains `jperl`; the same suite passes 5,703 tests on system Perl. |
19+
| Map::Metro::Plugin::Map::Oslo | Dependency failures exposed unsupported `PadWalker::var_name`, incomplete low-level `Unicode::Normalize`, missing reverse charnames lookup, and weak-reference loss for `map` temporaries. |
20+
| DBIx::Dictionary | DBI `execute` returned `-1` for a successful `SELECT`, which failed DBI's documented truth test. |
21+
| Date::Holidays::Abstract | Bare `SUPER::can` resolved relative to `UNIVERSAL` instead of the caller package. |
22+
| Music::Note::Role::Operators | Required native `Math::Factor::XS`; after replacing it in Java, a dependency exposed missing `POSIX::log2`. |
23+
24+
Full command output is captured under `/tmp/jcpan-*.log`; every `jcpan`,
25+
`jperl`, and `prove` run is wrapped in `timeout`.
26+
27+
## Progress Tracking
28+
29+
### Current Status: Implementation and requested-target validation complete
30+
31+
### Completed Phases
32+
33+
- [x] Repository pre-flight and feature branch (2026-08-14)
34+
- Confirmed the tree was clean.
35+
- Created `fix/jcpan-compiler-tooling-batch`.
36+
- [x] Initial system-Perl classification (2026-08-14)
37+
- Char::Windows1258 passes 210 files / 5,703 tests.
38+
- No requested target was excluded as a system-Perl failure.
39+
- [x] Compiler and runtime compatibility fixes (2026-08-14)
40+
- Preserved the construction origin of empty `qr//` values through both
41+
bytecode backends so they no longer acquire the previous match pattern.
42+
- Kept `map`/`grep` aliases alive while their temporary values are active,
43+
allowing weak references to those aliases to behave like Perl.
44+
- Resolved bare `SUPER::can` relative to the current caller package.
45+
- Implemented caller-aware `PadWalker::var_name` for live and captured
46+
lexicals, including aliases inside `map` and `grep`.
47+
- Added low-level Unicode normalization decomposition, canonical reordering,
48+
and composition using the ICU library already shipped by PerlOnJava.
49+
- Added ICU-backed reverse Unicode character-name lookup.
50+
- Made successful DBI `SELECT` execution return the true-but-zero `0E0`
51+
value required by DBI semantics.
52+
- Added the standard `POSIX::log2` helper and exports.
53+
- Files: bytecode compiler/interpreter and regex emitter/runtime, list and
54+
scalar runtimes, `RuntimeCode`, `Universal`, `Internals`, `PadWalker`,
55+
`UnicodeNormalize`, `_charnames`, `Charnames`, `DBI`, and `POSIX`.
56+
- [x] Native dependency and CPAN tooling fixes (2026-08-14)
57+
- Replaced `Math::Factor::XS` with a Java module; its upstream suite passes
58+
4 files / 69 tests without loading native code.
59+
- Added and bootstrapped a reusable Char::Windows1258 patch that delegates
60+
its source-generation step to system Perl, removes its obsolete `jperl`
61+
rejection, and avoids regex constructs that the generated compatibility
62+
layer cannot safely transform itself.
63+
- Added and bootstrapped a MooseX BetterAnonClassNames patch that removes the
64+
obsolete `autobox::Core` dependency from both source and build metadata.
65+
- [x] Regression coverage and requested-target verification (2026-08-14)
66+
- New regression tests were first validated with system Perl and then with
67+
both PerlOnJava backends.
68+
- Full `make` passes after the implementation changes.
69+
- Passing `jcpan -t` results: Data::Collector (3 files / 12 tests), Pod::Query
70+
(10 / 246), DBIx::Dictionary (7 / 30), Date::Holidays::Abstract (9 / 3),
71+
Map::Metro::Plugin::Map::Oslo (3 / 4), and
72+
Music::Note::Role::Operators (2 / 6).
73+
- Char::Windows1258 passes all 210 files / 5,703 tests, matching its system
74+
Perl baseline. `HARNESS_OPTIONS=j4` was used to reduce the cost of its
75+
many independent test files while retaining the exact `jcpan -t` path.
76+
77+
### Next Steps
78+
79+
1. Review and commit the final diff.
80+
2. Open the pull request and monitor CI to completion.
81+
82+
### Open Questions
83+
84+
- None.
85+
86+
## Related References
87+
88+
- `dev/modules/cpan_compiler_tooling_suite.md`
89+
- `dev/design/patch-and-cpan-prefs-layout.md`
90+
- `.agents/skills/debug-perlonjava/SKILL.md`

‎src/main/java/org/perlonjava/backend/bytecode/BytecodeInterpreter.java‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3278,12 +3278,16 @@ private static int executeTypeOps(int opcode, int[] bytecode, int pc,
32783278
int flagsReg = bytecode[pc++];
32793279
int implicitU = bytecode[pc++];
32803280
int warningState = bytecode[pc++];
3281+
int quoteConstruction = bytecode[pc++];
32813282
RuntimeScalar flags = registers[flagsReg].scalar();
32823283
if (implicitU != 0) {
32833284
flags = RuntimeRegex.applyUnicodeStringsFeatureToModifiers(flags);
32843285
}
32853286
RegexQuoteMeta.setCallSiteWarningState(warningState);
32863287
registers[rd] = RuntimeRegex.getQuotedRegex(registers[patternReg].scalar(), flags);
3288+
if (quoteConstruction != 0) {
3289+
registers[rd] = RuntimeRegex.markQuoteConstruction(registers[rd].scalar());
3290+
}
32873291
return pc;
32883292
}
32893293
case Opcodes.QUOTE_REGEX_O -> {
@@ -3293,12 +3297,16 @@ private static int executeTypeOps(int opcode, int[] bytecode, int pc,
32933297
int callsiteId = bytecode[pc++];
32943298
int implicitU = bytecode[pc++];
32953299
int warningState = bytecode[pc++];
3300+
int quoteConstruction = bytecode[pc++];
32963301
RuntimeScalar flags = registers[flagsReg].scalar();
32973302
if (implicitU != 0) {
32983303
flags = RuntimeRegex.applyUnicodeStringsFeatureToModifiers(flags);
32993304
}
33003305
RegexQuoteMeta.setCallSiteWarningState(warningState);
33013306
registers[rd] = RuntimeRegex.getQuotedRegex(registers[patternReg].scalar(), flags, callsiteId);
3307+
if (quoteConstruction != 0) {
3308+
registers[rd] = RuntimeRegex.markQuoteConstruction(registers[rd].scalar());
3309+
}
33023310
return pc;
33033311
}
33043312
default -> throw new RuntimeException("Unknown type opcode: " + opcode);

‎src/main/java/org/perlonjava/backend/bytecode/CompileOperator.java‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -310,13 +310,15 @@ private static void visitMatchRegex(BytecodeCompiler bc, OperatorNode node) {
310310
bc.emitReg(callsiteId);
311311
bc.emit(unicodeStringsImplicitUFlag(bc));
312312
bc.emit(regexWarningState(node));
313+
bc.emit(0);
313314
} else {
314315
bc.emit(Opcodes.QUOTE_REGEX);
315316
bc.emitReg(regexReg);
316317
bc.emitReg(patternReg);
317318
bc.emitReg(flagsReg);
318319
bc.emit(unicodeStringsImplicitUFlag(bc));
319320
bc.emit(regexWarningState(node));
321+
bc.emit(0);
320322
}
321323
int stringReg;
322324
if (args.elements.size() > 2) {
@@ -1094,13 +1096,15 @@ public static void visitOperator(BytecodeCompiler bytecodeCompiler, OperatorNode
10941096
bytecodeCompiler.emitReg(callsiteId);
10951097
bytecodeCompiler.emit(unicodeStringsImplicitUFlag(bytecodeCompiler));
10961098
bytecodeCompiler.emit(regexWarningState(node));
1099+
bytecodeCompiler.emit(1);
10971100
} else {
10981101
bytecodeCompiler.emit(Opcodes.QUOTE_REGEX);
10991102
bytecodeCompiler.emitReg(rd);
11001103
bytecodeCompiler.emitReg(patternReg);
11011104
bytecodeCompiler.emitReg(flagsReg);
11021105
bytecodeCompiler.emit(unicodeStringsImplicitUFlag(bytecodeCompiler));
11031106
bytecodeCompiler.emit(regexWarningState(node));
1107+
bytecodeCompiler.emit(1);
11041108
}
11051109
bytecodeCompiler.lastResultReg = rd;
11061110
}

‎src/main/java/org/perlonjava/backend/bytecode/Disassemble.java‎

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1279,9 +1279,11 @@ public static String disassemble(InterpretedCode interpretedCode) {
12791279
int flagsReg = interpretedCode.bytecode[pc++];
12801280
int implicitU = interpretedCode.bytecode[pc++];
12811281
int warningState = interpretedCode.bytecode[pc++];
1282+
int quoteConstruction = interpretedCode.bytecode[pc++];
12821283
sb.append("QUOTE_REGEX r").append(rd).append(" = qr{r").append(patternReg)
12831284
.append("}r").append(flagsReg).append(" implicitU=").append(implicitU)
1284-
.append(" warningState=").append(warningState).append("\n");
1285+
.append(" warningState=").append(warningState)
1286+
.append(" quoteConstruction=").append(quoteConstruction).append("\n");
12851287
break;
12861288
case Opcodes.QUOTE_REGEX_O:
12871289
rd = interpretedCode.bytecode[pc++];
@@ -1290,10 +1292,12 @@ public static String disassemble(InterpretedCode interpretedCode) {
12901292
int callsiteId = interpretedCode.bytecode[pc++];
12911293
implicitU = interpretedCode.bytecode[pc++];
12921294
warningState = interpretedCode.bytecode[pc++];
1295+
quoteConstruction = interpretedCode.bytecode[pc++];
12931296
sb.append("QUOTE_REGEX_O r").append(rd).append(" = qr{r").append(patternReg)
12941297
.append("}r").append(flagsReg).append(" callsite=").append(callsiteId)
12951298
.append(" implicitU=").append(implicitU)
1296-
.append(" warningState=").append(warningState).append("\n");
1299+
.append(" warningState=").append(warningState)
1300+
.append(" quoteConstruction=").append(quoteConstruction).append("\n");
12971301
break;
12981302
case Opcodes.ITERATOR_CREATE:
12991303
rd = interpretedCode.bytecode[pc++];

‎src/main/java/org/perlonjava/backend/bytecode/Opcodes.java‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -935,7 +935,7 @@ public class Opcodes {
935935

936936
/**
937937
* Quote regex operator: rd = RuntimeRegex.getQuotedRegex(pattern_reg, flags_reg)
938-
* Format: QUOTE_REGEX rd pattern_reg flags_reg implicit_unicode_strings_u warning_state
938+
* Format: QUOTE_REGEX rd pattern_reg flags_reg implicit_unicode_strings_u warning_state quote_construction
939939
*/
940940
public static final short QUOTE_REGEX = 159;
941941

@@ -1834,7 +1834,7 @@ public class Opcodes {
18341834

18351835
/**
18361836
* Quote regex with /o modifier support: rd = RuntimeRegex.getQuotedRegex(pattern_reg, flags_reg, callsite_id)
1837-
* Format: QUOTE_REGEX_O rd pattern_reg flags_reg callsite_id implicit_unicode_strings_u warning_state
1837+
* Format: QUOTE_REGEX_O rd pattern_reg flags_reg callsite_id implicit_unicode_strings_u warning_state quote_construction
18381838
*/
18391839
public static final short QUOTE_REGEX_O = 374;
18401840

‎src/main/java/org/perlonjava/backend/jvm/EmitRegex.java‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -288,6 +288,9 @@ static void handleQuoteRegex(EmitterVisitor emitterVisitor, OperatorNode node) {
288288
emitterVisitor.ctx.mv.visitMethodInsn(Opcodes.INVOKESTATIC,
289289
"org/perlonjava/runtime/regex/RuntimeRegex", "getQuotedRegex",
290290
"(Lorg/perlonjava/runtime/runtimetypes/RuntimeScalar;Lorg/perlonjava/runtime/runtimetypes/RuntimeScalar;)Lorg/perlonjava/runtime/runtimetypes/RuntimeScalar;", false);
291+
emitterVisitor.ctx.mv.visitMethodInsn(Opcodes.INVOKESTATIC,
292+
"org/perlonjava/runtime/regex/RuntimeRegex", "markQuoteConstruction",
293+
"(Lorg/perlonjava/runtime/runtimetypes/RuntimeScalar;)Lorg/perlonjava/runtime/runtimetypes/RuntimeScalar;", false);
291294

292295
if (emitterVisitor.ctx.contextType == RuntimeContextType.VOID) {
293296
emitterVisitor.ctx.mv.visitInsn(Opcodes.POP);

‎src/main/java/org/perlonjava/runtime/operators/ListOperators.java‎

Lines changed: 12 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -45,6 +45,7 @@ public static RuntimeList map(RuntimeList runtimeList, RuntimeScalar perlMapClos
4545
List<RuntimeBase> transformedElements = new ArrayList<>();
4646

4747
RuntimeScalar saveValue = getGlobalVariable("main::_");
48+
boolean savedTemporaryAlias = GlobalVariable.isTemporaryGlobalAlias("main::_");
4849
// Map results are captured by the caller after the operator returns;
4950
// flushing between iterations can destroy blessed return values early.
5051
boolean wasFlushing = MortalList.suppressFlush(true);
@@ -57,7 +58,7 @@ public static RuntimeList map(RuntimeList runtimeList, RuntimeScalar perlMapClos
5758
// Iterate over each element in the current RuntimeArray
5859
for (RuntimeScalar element : runtimeList) {
5960
// Create $_ argument for the map subroutine
60-
GlobalVariable.aliasGlobalVariable("main::_", element);
61+
GlobalVariable.aliasTemporaryGlobalVariable("main::_", element);
6162

6263
// Apply the Perl map subroutine with the outer @_ as arguments
6364
RuntimeList result = RuntimeCode.apply(perlMapClosure, mapArgs, RuntimeContextType.LIST);
@@ -91,7 +92,7 @@ public static RuntimeList map(RuntimeList runtimeList, RuntimeScalar perlMapClos
9192
}
9293
} finally {
9394
MortalList.suppressFlush(wasFlushing);
94-
GlobalVariable.aliasGlobalVariable("main::_", saveValue);
95+
GlobalVariable.restoreTemporaryGlobalVariable("main::_", saveValue, savedTemporaryAlias);
9596
releaseEphemeralCaptures(perlMapClosure);
9697
}
9798
}
@@ -237,6 +238,7 @@ public static RuntimeList grep(RuntimeList runtimeList, RuntimeScalar perlFilter
237238
List<RuntimeBase> filteredElements = new ArrayList<>();
238239

239240
RuntimeScalar saveValue = getGlobalVariable("main::_");
241+
boolean savedTemporaryAlias = GlobalVariable.isTemporaryGlobalAlias("main::_");
240242

241243
try {
242244
// Use the outer @_ instead of an empty array
@@ -246,7 +248,7 @@ public static RuntimeList grep(RuntimeList runtimeList, RuntimeScalar perlFilter
246248
for (RuntimeScalar element : runtimeList) {
247249
try {
248250
// Create $_ argument for the filter subroutine
249-
GlobalVariable.aliasGlobalVariable("main::_", element);
251+
GlobalVariable.aliasTemporaryGlobalVariable("main::_", element);
250252

251253
// Apply the Perl filter subroutine with the outer @_ as arguments
252254
RuntimeList result = RuntimeCode.apply(perlFilterClosure, filterArgs, RuntimeContextType.SCALAR);
@@ -287,7 +289,7 @@ public static RuntimeList grep(RuntimeList runtimeList, RuntimeScalar perlFilter
287289
return filteredList;
288290
}
289291
} finally {
290-
GlobalVariable.aliasGlobalVariable("main::_", saveValue);
292+
GlobalVariable.restoreTemporaryGlobalVariable("main::_", saveValue, savedTemporaryAlias);
291293
releaseEphemeralCaptures(perlFilterClosure);
292294
}
293295
}
@@ -313,6 +315,7 @@ public static RuntimeList grep(RuntimeList runtimeList, RuntimeScalar perlFilter
313315
public static RuntimeList all(RuntimeList runtimeList, RuntimeScalar perlFilterClosure, RuntimeArray outerArgs, int ctx) {
314316

315317
RuntimeScalar saveValue = getGlobalVariable("main::_");
318+
boolean savedTemporaryAlias = GlobalVariable.isTemporaryGlobalAlias("main::_");
316319

317320
try {
318321
RuntimeArray filterArgs = outerArgs != null ? outerArgs : new RuntimeArray();
@@ -321,7 +324,7 @@ public static RuntimeList all(RuntimeList runtimeList, RuntimeScalar perlFilterC
321324
for (RuntimeScalar element : runtimeList) {
322325
try {
323326
// Create $_ argument for the filter subroutine
324-
GlobalVariable.aliasGlobalVariable("main::_", element);
327+
GlobalVariable.aliasTemporaryGlobalVariable("main::_", element);
325328

326329
// Apply the Perl filter subroutine with the argument
327330
RuntimeList result = RuntimeCode.apply(perlFilterClosure, filterArgs, RuntimeContextType.SCALAR);
@@ -346,7 +349,7 @@ public static RuntimeList all(RuntimeList runtimeList, RuntimeScalar perlFilterC
346349

347350
return scalarTrue.getList();
348351
} finally {
349-
GlobalVariable.aliasGlobalVariable("main::_", saveValue);
352+
GlobalVariable.restoreTemporaryGlobalVariable("main::_", saveValue, savedTemporaryAlias);
350353
releaseEphemeralCaptures(perlFilterClosure);
351354
}
352355
}
@@ -372,6 +375,7 @@ public static RuntimeList all(RuntimeList runtimeList, RuntimeScalar perlFilterC
372375
public static RuntimeList any(RuntimeList runtimeList, RuntimeScalar perlFilterClosure, RuntimeArray outerArgs, int ctx) {
373376

374377
RuntimeScalar saveValue = getGlobalVariable("main::_");
378+
boolean savedTemporaryAlias = GlobalVariable.isTemporaryGlobalAlias("main::_");
375379

376380
try {
377381
RuntimeArray filterArgs = outerArgs != null ? outerArgs : new RuntimeArray();
@@ -380,7 +384,7 @@ public static RuntimeList any(RuntimeList runtimeList, RuntimeScalar perlFilterC
380384
for (RuntimeScalar element : runtimeList) {
381385
try {
382386
// Create $_ argument for the filter subroutine
383-
GlobalVariable.aliasGlobalVariable("main::_", element);
387+
GlobalVariable.aliasTemporaryGlobalVariable("main::_", element);
384388

385389
// Apply the Perl filter subroutine with the argument
386390
RuntimeList result = RuntimeCode.apply(perlFilterClosure, filterArgs, RuntimeContextType.SCALAR);
@@ -405,7 +409,7 @@ public static RuntimeList any(RuntimeList runtimeList, RuntimeScalar perlFilterC
405409

406410
return scalarFalse.getList();
407411
} finally {
408-
GlobalVariable.aliasGlobalVariable("main::_", saveValue);
412+
GlobalVariable.restoreTemporaryGlobalVariable("main::_", saveValue, savedTemporaryAlias);
409413
releaseEphemeralCaptures(perlFilterClosure);
410414
}
411415
}

‎src/main/java/org/perlonjava/runtime/perlmodule/Charnames.java‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -99,6 +99,7 @@ public static void initialize() {
9999
Charnames charnames = new Charnames();
100100
try {
101101
charnames.registerMethod("_java_viacode", "javaViacode", "$");
102+
charnames.registerMethod("_java_vianame", "javaVianame", "$");
102103
} catch (NoSuchMethodException e) {
103104
System.err.println("Warning: Missing _charnames method: " + e.getMessage());
104105
}
@@ -129,4 +130,11 @@ public static RuntimeList javaViacode(RuntimeArray args, int ctx) {
129130
}
130131
return new RuntimeList(new RuntimeScalar(name));
131132
}
133+
134+
/** Return the code point for an official Unicode character name. */
135+
public static RuntimeList javaVianame(RuntimeArray args, int ctx) {
136+
int codePoint = UCharacter.getCharFromName(args.getFirst().toString());
137+
if (codePoint < 0) return new RuntimeList(scalarUndef);
138+
return new RuntimeScalar(codePoint).getList();
139+
}
132140
}

‎src/main/java/org/perlonjava/runtime/perlmodule/DBI.java‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -520,9 +520,9 @@ public static RuntimeList execute(RuntimeArray args, int ctx) {
520520

521521
// Return value per DBI spec:
522522
// - For DML (INSERT/UPDATE/DELETE): number of rows affected, or "0E0" for 0 rows
523-
// - For SELECT: -1 (unknown number of rows)
523+
// - For SELECT: "0E0" (true zero; row count is not known until fetching)
524524
if (hasResultSet) {
525-
return new RuntimeScalar(-1).getList();
525+
return new RuntimeScalar("0E0").getList();
526526
} else {
527527
int updateCount = stmt.getUpdateCount();
528528
if (updateCount == 0) {

0 commit comments

Comments
 (0)