Can't properly decode ToUnicode CMap

[shibaev]

For TT_EOF cmap.pdf and PDF/A-2a flavour, veraPDF reports "6.2.11.7.2-1 The Font dictionary of all fonts shall define the map of all used character codes to Unicode values, either via a ToUnicode entry, or other mechanisms as defined in ISO 19005-2, 6.2.11.7.2". Internally, the following error happens:

org.verapdf.pd.font.cmap.CMapFactory getCMap
WARNING: Can't parse CMap CMap 13 0 obj, using default
java.io.IOException: CMap contains invalid entry in bfchar. Expected TT_HEXSTRING but got TT_EOF
        at org.verapdf.pd.font.cmap.CMapParser.checkTokenType(CMapParser.java:336)
        at org.verapdf.pd.font.cmap.CMapParser.readSingleToUnicodeMapping(CMapParser.java:241)
        at org.verapdf.pd.font.cmap.CMapParser.processList(CMapParser.java:143)
        at org.verapdf.pd.font.cmap.CMapParser.processObject(CMapParser.java:95)
        at org.verapdf.pd.font.cmap.CMapParser.parse(CMapParser.java:80)
        at org.verapdf.pd.font.cmap.CMapFactory.getCMap(CMapFactory.java:60)
        at org.verapdf.pd.font.cmap.PDCMap.getCMapFile(PDCMap.java:119)
        at org.verapdf.pd.font.cmap.PDCMap.getCMapFile(PDCMap.java:111)
        at org.verapdf.pd.font.cmap.PDCMap.toUnicode(PDCMap.java:262)
        at org.verapdf.pd.font.PDFont.cMapToUnicode(PDFont.java:328)
        at org.verapdf.pd.font.PDFont.toUnicode(PDFont.java:312)
        at org.verapdf.pd.font.PDSimpleFont.toUnicode(PDSimpleFont.java:59)
        at org.verapdf.gf.model.impl.operator.textshow.GFGlyph.<init>(GFGlyph.java:127)
        at org.verapdf.gf.model.impl.operator.textshow.GFGlyph.<init>(GFGlyph.java:70)
        at org.verapdf.gf.model.impl.operator.textshow.GFGlyph.getGlyph(GFGlyph.java:152)
        at org.verapdf.gf.model.impl.operator.textshow.GFOpTextShow.getUsedGlyphs(GFOpTextShow.java:154)
        at org.verapdf.gf.model.impl.operator.textshow.GFOpTextShow.getLinkedObjects(GFOpTextShow.java:110)
        at org.verapdf.gf.model.impl.operator.textshow.GFOpStringTextShow.getLinkedObjects(GFOpStringTextShow.java:58)
        at org.verapdf.pdfa.validation.validators.BaseValidator.addAllLinkedObjects(BaseValidator.java:285)
        at org.verapdf.pdfa.validation.validators.BaseValidator.checkNext(BaseValidator.java:250)
        at org.verapdf.pdfa.validation.validators.BaseValidator.validate(BaseValidator.java:185)
        at org.verapdf.pdfa.validation.validators.BaseValidator.validateAll(BaseValidator.java:149)
        at org.verapdf.processor.ProcessorImpl.validate(ProcessorImpl.java:241)
        at org.verapdf.processor.ProcessorImpl.process(ProcessorImpl.java:119)
        at org.verapdf.processor.BatchFileProcessor.processItem(BatchFileProcessor.java:167)
        at org.verapdf.processor.BatchFileProcessor.processList(BatchFileProcessor.java:85)
        at org.verapdf.processor.AbstractBatchProcessor.process(AbstractBatchProcessor.java:104)
        at org.verapdf.gui.ValidateWorker.doInBackground(ValidateWorker.java:125)
        at org.verapdf.gui.ValidateWorker.doInBackground(ValidateWorker.java:63)
        at javax.swing.SwingWorker$1.call(Unknown Source)
        at java.util.concurrent.FutureTask.run(Unknown Source)
        at javax.swing.SwingWorker.run(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source)

However, the corresponding ToUnicode stream can be decoded properly. veraPDF does not report any issues for the decoded file TT_EOF cmap decoded.pdf.

[bdoubrov]

There is something wrong here with Flate encoding of the ToUnicode stream. We'll check it further, but for example I see that iText RUPS is also not able to fully decode this stream (note the last en token instead of the expected end):

[shibaev]

We use zlib for decoding Flate data. It also reports a Z_DATA_ERROR while decoding this stream. We stop decompression at this point and obtain the correct ToUnicode CMap (the second end is complete):

...
CMapName currentdict /CMap defineresource pop
end
end

This gives me hope that it might be possible to improve your Flate decoding in some way.

I also notice that PDF viewers (Adobe Reader, Foxit Reader, PDF-XChange Editor, Firefox, Chrome) respect the ToUnicode in this document. To confirm this, I added /Encoding /MacRomanEncoding to the font dictionary. The text is still copied as ü. Without the ToUnicode, the text would be copied as ¸ instead of ü.

[mkl-public]

Considering the intent of PDF/A, if we have a relevant PDF part that is broken (some data error apparently) and this causes different PDF processors to interpret the PDF differently, it is good to have validators reporting an issue, isn't it?

Thus, "improving" the Flate decoding of verpdf appears to be the wrong thing to do here.

[shibaev]

I totally agree that it's fine for validators to be strict. However, there are a few things in this case that make me think something might be improved in veraPDF:

1. Popular PDF processors interpret this ToUnicode stream the same way, differently from veraPDF.
2. veraPDF does not report an issue about incorrect Flate data. It accepts corrupted Flate data and gets stuck parsing the decoded data.

Also the error message states "CMap contains invalid entry in bfchar," so it probably does not relate to an incomplete end operator.

[bdoubrov]

We use java.util.zip.Inflater internally, and it seems that its behavior is consistent in reporting the error, but not consistent in the returned byte(s).

This issue did appear before: #671 (comment) , and one of the solutions was to start reading the Flate stream by a single byte till the error point. But as this example shows, this solution is not universal.

So, I see that a better approach would indeed be to report a Flate decode error and avoid interpreting the corrupt stream. This error would be more clear to the end user, even if sometimes we would disagree with other PDF parser implementations that use a different Flate decode library.