test: compare snapshots by what they draw, not how they were encoded - #87
Merged
Conversation
A PNG carries a deflate stream, and the same pixels compress to different bytes across zlib builds. The snapshot suite compared the files byte for byte, so it was testing whichever encoder happened to be installed: the solid-colour snapshot, a 200x150 flat fill, differs from a fresh render in 332 bytes of IDAT while being pixel-for-pixel identical, same dimensions, same mode, same file length. Snapshots now compare as images. Nothing in the tests changes, because the stored value is what carries the comparison.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
A PNG carries a deflate stream, and the same pixels compress to different bytes across zlib builds. The snapshot suite compared the files byte for byte, so it was testing whichever encoder happened to be installed rather than what the renderer drew.
The flat-colour snapshot makes it concrete: a fresh render of
Canvas(200, 150).background(color="#3498db")differs from the stored file in 332 bytes of IDAT while being pixel-for-pixel identical — same dimensions, same mode, same 469-byte file length, same chunk layout.Changes
--inline-snapshot=create/fixstill writes real PNGsWhat this does not fix
tests/test_rendering.pyis excluded from CI, and the corpus has drifted while nobody was watching. On Linux with locked dependencies it reports 68 stale snapshots; this change accounts for 7 of them, and the other 61 differ in actual pixels. The two hard failures are environmental — one needs network for a URL image, one needsonnxruntime, which has no cp310 wheel.Regenerating those 61 would bless whatever the renderer currently draws in 61 cases nobody has looked at, so it wants a person's eye rather than a
--inline-snapshot=fixrun. This change is the prerequisite: once the corpus is current, it stays current across dependency bumps instead of drifting again, and the CI ignore can come off.