Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
90 changes: 90 additions & 0 deletions dev/modules/cpan_compiler_tooling_suite_2.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,90 @@
# CPAN compiler and tooling compatibility suite II

## Goal

Make `jcpan -t` work for Data::Collector, Pod::Query, Char::Windows1258,
Map::Metro::Plugin::Map::Oslo, DBIx::Dictionary,
Date::Holidays::Abstract, Music::Note::Role::Operators, and their
dependencies. Fix reusable compiler and CPAN tooling defects first. A target
that fails under the local system Perl may be left unsupported with the
failure recorded.

## Baseline (2026-08-14)

| Target | First actionable result |
|---|---|
| Data::Collector | Passes: 3 files / 12 tests. |
| Pod::Query | Empty `qr//` values interpolated from hashes incorrectly reuse the previous successful match. |
| Char::Windows1258 | All 210 files abort because upstream deliberately rejects an executable whose `$^X` contains `jperl`; the same suite passes 5,703 tests on system Perl. |
| Map::Metro::Plugin::Map::Oslo | Dependency failures exposed unsupported `PadWalker::var_name`, incomplete low-level `Unicode::Normalize`, missing reverse charnames lookup, and weak-reference loss for `map` temporaries. |
| DBIx::Dictionary | DBI `execute` returned `-1` for a successful `SELECT`, which failed DBI's documented truth test. |
| Date::Holidays::Abstract | Bare `SUPER::can` resolved relative to `UNIVERSAL` instead of the caller package. |
| Music::Note::Role::Operators | Required native `Math::Factor::XS`; after replacing it in Java, a dependency exposed missing `POSIX::log2`. |

Full command output is captured under `/tmp/jcpan-*.log`; every `jcpan`,
`jperl`, and `prove` run is wrapped in `timeout`.

## Progress Tracking

### Current Status: Implementation and requested-target validation complete

### Completed Phases

- [x] Repository pre-flight and feature branch (2026-08-14)
- Confirmed the tree was clean.
- Created `fix/jcpan-compiler-tooling-batch`.
- [x] Initial system-Perl classification (2026-08-14)
- Char::Windows1258 passes 210 files / 5,703 tests.
- No requested target was excluded as a system-Perl failure.
- [x] Compiler and runtime compatibility fixes (2026-08-14)
- Preserved the construction origin of empty `qr//` values through both
bytecode backends so they no longer acquire the previous match pattern.
- Kept `map`/`grep` aliases alive while their temporary values are active,
allowing weak references to those aliases to behave like Perl.
- Resolved bare `SUPER::can` relative to the current caller package.
- Implemented caller-aware `PadWalker::var_name` for live and captured
lexicals, including aliases inside `map` and `grep`.
- Added low-level Unicode normalization decomposition, canonical reordering,
and composition using the ICU library already shipped by PerlOnJava.
- Added ICU-backed reverse Unicode character-name lookup.
- Made successful DBI `SELECT` execution return the true-but-zero `0E0`
value required by DBI semantics.
- Added the standard `POSIX::log2` helper and exports.
- Files: bytecode compiler/interpreter and regex emitter/runtime, list and
scalar runtimes, `RuntimeCode`, `Universal`, `Internals`, `PadWalker`,
`UnicodeNormalize`, `_charnames`, `Charnames`, `DBI`, and `POSIX`.
- [x] Native dependency and CPAN tooling fixes (2026-08-14)
- Replaced `Math::Factor::XS` with a Java module; its upstream suite passes
4 files / 69 tests without loading native code.
- Added and bootstrapped a reusable Char::Windows1258 patch that delegates
its source-generation step to system Perl, removes its obsolete `jperl`
rejection, and avoids regex constructs that the generated compatibility
layer cannot safely transform itself.
- Added and bootstrapped a MooseX BetterAnonClassNames patch that removes the
obsolete `autobox::Core` dependency from both source and build metadata.
- [x] Regression coverage and requested-target verification (2026-08-14)
- New regression tests were first validated with system Perl and then with
both PerlOnJava backends.
- Full `make` passes after the implementation changes.
- Passing `jcpan -t` results: Data::Collector (3 files / 12 tests), Pod::Query
(10 / 246), DBIx::Dictionary (7 / 30), Date::Holidays::Abstract (9 / 3),
Map::Metro::Plugin::Map::Oslo (3 / 4), and
Music::Note::Role::Operators (2 / 6).
- Char::Windows1258 passes all 210 files / 5,703 tests, matching its system
Perl baseline. `HARNESS_OPTIONS=j4` was used to reduce the cost of its
many independent test files while retaining the exact `jcpan -t` path.

### Next Steps

1. Review and commit the final diff.
2. Open the pull request and monitor CI to completion.

### Open Questions

- None.

## Related References

- `dev/modules/cpan_compiler_tooling_suite.md`
- `dev/design/patch-and-cpan-prefs-layout.md`
- `.agents/skills/debug-perlonjava/SKILL.md`
Original file line number Diff line number Diff line change
Expand Up @@ -3278,12 +3278,16 @@ private static int executeTypeOps(int opcode, int[] bytecode, int pc,
int flagsReg = bytecode[pc++];
int implicitU = bytecode[pc++];
int warningState = bytecode[pc++];
int quoteConstruction = bytecode[pc++];
RuntimeScalar flags = registers[flagsReg].scalar();
if (implicitU != 0) {
flags = RuntimeRegex.applyUnicodeStringsFeatureToModifiers(flags);
}
RegexQuoteMeta.setCallSiteWarningState(warningState);
registers[rd] = RuntimeRegex.getQuotedRegex(registers[patternReg].scalar(), flags);
if (quoteConstruction != 0) {
registers[rd] = RuntimeRegex.markQuoteConstruction(registers[rd].scalar());
}
return pc;
}
case Opcodes.QUOTE_REGEX_O -> {
Expand All @@ -3293,12 +3297,16 @@ private static int executeTypeOps(int opcode, int[] bytecode, int pc,
int callsiteId = bytecode[pc++];
int implicitU = bytecode[pc++];
int warningState = bytecode[pc++];
int quoteConstruction = bytecode[pc++];
RuntimeScalar flags = registers[flagsReg].scalar();
if (implicitU != 0) {
flags = RuntimeRegex.applyUnicodeStringsFeatureToModifiers(flags);
}
RegexQuoteMeta.setCallSiteWarningState(warningState);
registers[rd] = RuntimeRegex.getQuotedRegex(registers[patternReg].scalar(), flags, callsiteId);
if (quoteConstruction != 0) {
registers[rd] = RuntimeRegex.markQuoteConstruction(registers[rd].scalar());
}
return pc;
}
default -> throw new RuntimeException("Unknown type opcode: " + opcode);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -310,13 +310,15 @@ private static void visitMatchRegex(BytecodeCompiler bc, OperatorNode node) {
bc.emitReg(callsiteId);
bc.emit(unicodeStringsImplicitUFlag(bc));
bc.emit(regexWarningState(node));
bc.emit(0);
} else {
bc.emit(Opcodes.QUOTE_REGEX);
bc.emitReg(regexReg);
bc.emitReg(patternReg);
bc.emitReg(flagsReg);
bc.emit(unicodeStringsImplicitUFlag(bc));
bc.emit(regexWarningState(node));
bc.emit(0);
}
int stringReg;
if (args.elements.size() > 2) {
Expand Down Expand Up @@ -1094,13 +1096,15 @@ public static void visitOperator(BytecodeCompiler bytecodeCompiler, OperatorNode
bytecodeCompiler.emitReg(callsiteId);
bytecodeCompiler.emit(unicodeStringsImplicitUFlag(bytecodeCompiler));
bytecodeCompiler.emit(regexWarningState(node));
bytecodeCompiler.emit(1);
} else {
bytecodeCompiler.emit(Opcodes.QUOTE_REGEX);
bytecodeCompiler.emitReg(rd);
bytecodeCompiler.emitReg(patternReg);
bytecodeCompiler.emitReg(flagsReg);
bytecodeCompiler.emit(unicodeStringsImplicitUFlag(bytecodeCompiler));
bytecodeCompiler.emit(regexWarningState(node));
bytecodeCompiler.emit(1);
}
bytecodeCompiler.lastResultReg = rd;
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -1279,9 +1279,11 @@ public static String disassemble(InterpretedCode interpretedCode) {
int flagsReg = interpretedCode.bytecode[pc++];
int implicitU = interpretedCode.bytecode[pc++];
int warningState = interpretedCode.bytecode[pc++];
int quoteConstruction = interpretedCode.bytecode[pc++];
sb.append("QUOTE_REGEX r").append(rd).append(" = qr{r").append(patternReg)
.append("}r").append(flagsReg).append(" implicitU=").append(implicitU)
.append(" warningState=").append(warningState).append("\n");
.append(" warningState=").append(warningState)
.append(" quoteConstruction=").append(quoteConstruction).append("\n");
break;
case Opcodes.QUOTE_REGEX_O:
rd = interpretedCode.bytecode[pc++];
Expand All @@ -1290,10 +1292,12 @@ public static String disassemble(InterpretedCode interpretedCode) {
int callsiteId = interpretedCode.bytecode[pc++];
implicitU = interpretedCode.bytecode[pc++];
warningState = interpretedCode.bytecode[pc++];
quoteConstruction = interpretedCode.bytecode[pc++];
sb.append("QUOTE_REGEX_O r").append(rd).append(" = qr{r").append(patternReg)
.append("}r").append(flagsReg).append(" callsite=").append(callsiteId)
.append(" implicitU=").append(implicitU)
.append(" warningState=").append(warningState).append("\n");
.append(" warningState=").append(warningState)
.append(" quoteConstruction=").append(quoteConstruction).append("\n");
break;
case Opcodes.ITERATOR_CREATE:
rd = interpretedCode.bytecode[pc++];
Expand Down
4 changes: 2 additions & 2 deletions src/main/java/org/perlonjava/backend/bytecode/Opcodes.java
Original file line number Diff line number Diff line change
Expand Up @@ -935,7 +935,7 @@ public class Opcodes {

/**
* Quote regex operator: rd = RuntimeRegex.getQuotedRegex(pattern_reg, flags_reg)
* Format: QUOTE_REGEX rd pattern_reg flags_reg implicit_unicode_strings_u warning_state
* Format: QUOTE_REGEX rd pattern_reg flags_reg implicit_unicode_strings_u warning_state quote_construction
*/
public static final short QUOTE_REGEX = 159;

Expand Down Expand Up @@ -1834,7 +1834,7 @@ public class Opcodes {

/**
* Quote regex with /o modifier support: rd = RuntimeRegex.getQuotedRegex(pattern_reg, flags_reg, callsite_id)
* Format: QUOTE_REGEX_O rd pattern_reg flags_reg callsite_id implicit_unicode_strings_u warning_state
* Format: QUOTE_REGEX_O rd pattern_reg flags_reg callsite_id implicit_unicode_strings_u warning_state quote_construction
*/
public static final short QUOTE_REGEX_O = 374;

Expand Down
3 changes: 3 additions & 0 deletions src/main/java/org/perlonjava/backend/jvm/EmitRegex.java
Original file line number Diff line number Diff line change
Expand Up @@ -288,6 +288,9 @@ static void handleQuoteRegex(EmitterVisitor emitterVisitor, OperatorNode node) {
emitterVisitor.ctx.mv.visitMethodInsn(Opcodes.INVOKESTATIC,
"org/perlonjava/runtime/regex/RuntimeRegex", "getQuotedRegex",
"(Lorg/perlonjava/runtime/runtimetypes/RuntimeScalar;Lorg/perlonjava/runtime/runtimetypes/RuntimeScalar;)Lorg/perlonjava/runtime/runtimetypes/RuntimeScalar;", false);
emitterVisitor.ctx.mv.visitMethodInsn(Opcodes.INVOKESTATIC,
"org/perlonjava/runtime/regex/RuntimeRegex", "markQuoteConstruction",
"(Lorg/perlonjava/runtime/runtimetypes/RuntimeScalar;)Lorg/perlonjava/runtime/runtimetypes/RuntimeScalar;", false);

if (emitterVisitor.ctx.contextType == RuntimeContextType.VOID) {
emitterVisitor.ctx.mv.visitInsn(Opcodes.POP);
Expand Down
20 changes: 12 additions & 8 deletions src/main/java/org/perlonjava/runtime/operators/ListOperators.java
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,7 @@ public static RuntimeList map(RuntimeList runtimeList, RuntimeScalar perlMapClos
List<RuntimeBase> transformedElements = new ArrayList<>();

RuntimeScalar saveValue = getGlobalVariable("main::_");
boolean savedTemporaryAlias = GlobalVariable.isTemporaryGlobalAlias("main::_");
// Map results are captured by the caller after the operator returns;
// flushing between iterations can destroy blessed return values early.
boolean wasFlushing = MortalList.suppressFlush(true);
Expand All @@ -57,7 +58,7 @@ public static RuntimeList map(RuntimeList runtimeList, RuntimeScalar perlMapClos
// Iterate over each element in the current RuntimeArray
for (RuntimeScalar element : runtimeList) {
// Create $_ argument for the map subroutine
GlobalVariable.aliasGlobalVariable("main::_", element);
GlobalVariable.aliasTemporaryGlobalVariable("main::_", element);

// Apply the Perl map subroutine with the outer @_ as arguments
RuntimeList result = RuntimeCode.apply(perlMapClosure, mapArgs, RuntimeContextType.LIST);
Expand Down Expand Up @@ -91,7 +92,7 @@ public static RuntimeList map(RuntimeList runtimeList, RuntimeScalar perlMapClos
}
} finally {
MortalList.suppressFlush(wasFlushing);
GlobalVariable.aliasGlobalVariable("main::_", saveValue);
GlobalVariable.restoreTemporaryGlobalVariable("main::_", saveValue, savedTemporaryAlias);
releaseEphemeralCaptures(perlMapClosure);
}
}
Expand Down Expand Up @@ -237,6 +238,7 @@ public static RuntimeList grep(RuntimeList runtimeList, RuntimeScalar perlFilter
List<RuntimeBase> filteredElements = new ArrayList<>();

RuntimeScalar saveValue = getGlobalVariable("main::_");
boolean savedTemporaryAlias = GlobalVariable.isTemporaryGlobalAlias("main::_");

try {
// Use the outer @_ instead of an empty array
Expand All @@ -246,7 +248,7 @@ public static RuntimeList grep(RuntimeList runtimeList, RuntimeScalar perlFilter
for (RuntimeScalar element : runtimeList) {
try {
// Create $_ argument for the filter subroutine
GlobalVariable.aliasGlobalVariable("main::_", element);
GlobalVariable.aliasTemporaryGlobalVariable("main::_", element);

// Apply the Perl filter subroutine with the outer @_ as arguments
RuntimeList result = RuntimeCode.apply(perlFilterClosure, filterArgs, RuntimeContextType.SCALAR);
Expand Down Expand Up @@ -287,7 +289,7 @@ public static RuntimeList grep(RuntimeList runtimeList, RuntimeScalar perlFilter
return filteredList;
}
} finally {
GlobalVariable.aliasGlobalVariable("main::_", saveValue);
GlobalVariable.restoreTemporaryGlobalVariable("main::_", saveValue, savedTemporaryAlias);
releaseEphemeralCaptures(perlFilterClosure);
}
}
Expand All @@ -313,6 +315,7 @@ public static RuntimeList grep(RuntimeList runtimeList, RuntimeScalar perlFilter
public static RuntimeList all(RuntimeList runtimeList, RuntimeScalar perlFilterClosure, RuntimeArray outerArgs, int ctx) {

RuntimeScalar saveValue = getGlobalVariable("main::_");
boolean savedTemporaryAlias = GlobalVariable.isTemporaryGlobalAlias("main::_");

try {
RuntimeArray filterArgs = outerArgs != null ? outerArgs : new RuntimeArray();
Expand All @@ -321,7 +324,7 @@ public static RuntimeList all(RuntimeList runtimeList, RuntimeScalar perlFilterC
for (RuntimeScalar element : runtimeList) {
try {
// Create $_ argument for the filter subroutine
GlobalVariable.aliasGlobalVariable("main::_", element);
GlobalVariable.aliasTemporaryGlobalVariable("main::_", element);

// Apply the Perl filter subroutine with the argument
RuntimeList result = RuntimeCode.apply(perlFilterClosure, filterArgs, RuntimeContextType.SCALAR);
Expand All @@ -346,7 +349,7 @@ public static RuntimeList all(RuntimeList runtimeList, RuntimeScalar perlFilterC

return scalarTrue.getList();
} finally {
GlobalVariable.aliasGlobalVariable("main::_", saveValue);
GlobalVariable.restoreTemporaryGlobalVariable("main::_", saveValue, savedTemporaryAlias);
releaseEphemeralCaptures(perlFilterClosure);
}
}
Expand All @@ -372,6 +375,7 @@ public static RuntimeList all(RuntimeList runtimeList, RuntimeScalar perlFilterC
public static RuntimeList any(RuntimeList runtimeList, RuntimeScalar perlFilterClosure, RuntimeArray outerArgs, int ctx) {

RuntimeScalar saveValue = getGlobalVariable("main::_");
boolean savedTemporaryAlias = GlobalVariable.isTemporaryGlobalAlias("main::_");

try {
RuntimeArray filterArgs = outerArgs != null ? outerArgs : new RuntimeArray();
Expand All @@ -380,7 +384,7 @@ public static RuntimeList any(RuntimeList runtimeList, RuntimeScalar perlFilterC
for (RuntimeScalar element : runtimeList) {
try {
// Create $_ argument for the filter subroutine
GlobalVariable.aliasGlobalVariable("main::_", element);
GlobalVariable.aliasTemporaryGlobalVariable("main::_", element);

// Apply the Perl filter subroutine with the argument
RuntimeList result = RuntimeCode.apply(perlFilterClosure, filterArgs, RuntimeContextType.SCALAR);
Expand All @@ -405,7 +409,7 @@ public static RuntimeList any(RuntimeList runtimeList, RuntimeScalar perlFilterC

return scalarFalse.getList();
} finally {
GlobalVariable.aliasGlobalVariable("main::_", saveValue);
GlobalVariable.restoreTemporaryGlobalVariable("main::_", saveValue, savedTemporaryAlias);
releaseEphemeralCaptures(perlFilterClosure);
}
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -99,6 +99,7 @@ public static void initialize() {
Charnames charnames = new Charnames();
try {
charnames.registerMethod("_java_viacode", "javaViacode", "$");
charnames.registerMethod("_java_vianame", "javaVianame", "$");
} catch (NoSuchMethodException e) {
System.err.println("Warning: Missing _charnames method: " + e.getMessage());
}
Expand Down Expand Up @@ -129,4 +130,11 @@ public static RuntimeList javaViacode(RuntimeArray args, int ctx) {
}
return new RuntimeList(new RuntimeScalar(name));
}

/** Return the code point for an official Unicode character name. */
public static RuntimeList javaVianame(RuntimeArray args, int ctx) {
int codePoint = UCharacter.getCharFromName(args.getFirst().toString());
if (codePoint < 0) return new RuntimeList(scalarUndef);
return new RuntimeScalar(codePoint).getList();
}
}
4 changes: 2 additions & 2 deletions src/main/java/org/perlonjava/runtime/perlmodule/DBI.java
Original file line number Diff line number Diff line change
Expand Up @@ -520,9 +520,9 @@ public static RuntimeList execute(RuntimeArray args, int ctx) {

// Return value per DBI spec:
// - For DML (INSERT/UPDATE/DELETE): number of rows affected, or "0E0" for 0 rows
// - For SELECT: -1 (unknown number of rows)
// - For SELECT: "0E0" (true zero; row count is not known until fetching)
if (hasResultSet) {
return new RuntimeScalar(-1).getList();
return new RuntimeScalar("0E0").getList();
} else {
int updateCount = stmt.getUpdateCount();
if (updateCount == 0) {
Expand Down
Loading
Loading