Skip to content

Commit d007b66

Browse files
Document utf8::valid() fix in cpan_client.md (Phase 16)
Added documentation for the fix that resolved CPAN::Meta::YAML parsing errors that were preventing proper test dependency detection. Generated with [Devin](https://cli.devin.ai/docs) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
1 parent 19c4c4f commit d007b66

1 file changed

Lines changed: 52 additions & 0 deletions

File tree

dev/design/cpan_client.md

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -355,6 +355,58 @@ All major DateTime issues have been fixed. The 7 remaining test failures are:
355355

356356
---
357357

358+
## Phase 16: utf8::valid() Fix for CPAN::Meta Parsing (2026-03-20)
359+
360+
### Problem Statement
361+
362+
When installing DateTime with empty caches, CPAN::Meta::YAML parsing would fail with:
363+
```
364+
Read an invalid UTF-8 string (maybe mixed UTF-8 and 8-bit character set).
365+
Did you decode with lax ":utf8" instead of strict ":encoding(UTF-8)"?
366+
```
367+
368+
This error prevented proper parsing of META.yml/MYMETA.yml files, which meant test dependencies like Test::Without::Module and CPAN::Meta::Check were not being properly detected.
369+
370+
### Root Cause
371+
372+
CPAN::Meta::YAML validates strings before parsing:
373+
```perl
374+
if ( utf8::is_utf8($string) && ! utf8::valid($string) ) {
375+
die "Read an invalid UTF-8 string...";
376+
}
377+
```
378+
379+
The `utf8::valid()` function in PerlOnJava was using `CharsetDetector` which was fundamentally wrong:
380+
- It converted the string to bytes using the default charset
381+
- Then tried to detect if those bytes were UTF-8
382+
- This always failed for properly decoded Unicode strings
383+
384+
### Solution
385+
386+
Rewrote `utf8::valid()` in `Utf8.java` to correctly check string validity:
387+
- **For character strings (UTF-8 flag on)**: Validates that surrogate pairs are properly formed
388+
- **For byte strings (UTF-8 flag off)**: Attempts to decode bytes as UTF-8
389+
390+
### Files Changed
391+
392+
- `src/main/java/org/perlonjava/runtime/perlmodule/Utf8.java` - Fixed `valid()` method
393+
394+
### Test Results
395+
396+
The fix allows CPAN::Meta::YAML to properly parse MYMETA.yml files, enabling CPAN.pm to detect and install test dependencies.
397+
398+
---
399+
400+
## Known Remaining CPAN Issues
401+
402+
| Issue | Status | Impact |
403+
|-------|--------|--------|
404+
| File::stat.pm missing | Not implemented | DateTime::Locale installation fails |
405+
| IPC::Open3 read-only error | Bug in IPCOpen3.java | Some module tests fail |
406+
| Test::Harness UTF-8 error | Pre-existing | Some test output parsing fails |
407+
408+
---
409+
358410
## Related Documents
359411

360412
- `dev/design/xsloader.md` - XSLoader/Java integration

0 commit comments

Comments
 (0)