You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Document utf8::valid() fix in cpan_client.md (Phase 16)
Added documentation for the fix that resolved CPAN::Meta::YAML parsing
errors that were preventing proper test dependency detection.
Generated with [Devin](https://cli.devin.ai/docs)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Copy file name to clipboardExpand all lines: dev/design/cpan_client.md
+52Lines changed: 52 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -355,6 +355,58 @@ All major DateTime issues have been fixed. The 7 remaining test failures are:
355
355
356
356
---
357
357
358
+
## Phase 16: utf8::valid() Fix for CPAN::Meta Parsing (2026-03-20)
359
+
360
+
### Problem Statement
361
+
362
+
When installing DateTime with empty caches, CPAN::Meta::YAML parsing would fail with:
363
+
```
364
+
Read an invalid UTF-8 string (maybe mixed UTF-8 and 8-bit character set).
365
+
Did you decode with lax ":utf8" instead of strict ":encoding(UTF-8)"?
366
+
```
367
+
368
+
This error prevented proper parsing of META.yml/MYMETA.yml files, which meant test dependencies like Test::Without::Module and CPAN::Meta::Check were not being properly detected.
369
+
370
+
### Root Cause
371
+
372
+
CPAN::Meta::YAML validates strings before parsing:
373
+
```perl
374
+
if ( utf8::is_utf8($string) && ! utf8::valid($string) ) {
375
+
die"Read an invalid UTF-8 string...";
376
+
}
377
+
```
378
+
379
+
The `utf8::valid()` function in PerlOnJava was using `CharsetDetector` which was fundamentally wrong:
380
+
- It converted the string to bytes using the default charset
381
+
- Then tried to detect if those bytes were UTF-8
382
+
- This always failed for properly decoded Unicode strings
383
+
384
+
### Solution
385
+
386
+
Rewrote `utf8::valid()` in `Utf8.java` to correctly check string validity:
387
+
-**For character strings (UTF-8 flag on)**: Validates that surrogate pairs are properly formed
388
+
-**For byte strings (UTF-8 flag off)**: Attempts to decode bytes as UTF-8
0 commit comments