Skip to content

Add fpc compression and decompression mutators - #850

Closed
batmac wants to merge 2 commits into
mainfrom
fpc
Closed

Add fpc compression and decompression mutators#850
batmac wants to merge 2 commits into
mainfrom
fpc

Conversation

@batmac

@batmac batmac commented Apr 14, 2025

Copy link
Copy Markdown
Owner

No description provided.

@batmac

batmac commented Aug 22, 2026

Copy link
Copy Markdown
Owner Author

Closing after evaluation — the code works (I merged main in, resolved the 16-month go.mod conflict, and verified byte-exact round-trips including through the stdin pipeline), but the value case does not hold up:

The algorithm is superseded. FPC is from 2007. The lineage since: Gorilla (2015) → Chimp/Chimp128 (2022) → Elf/Elf+ (VLDB 2023) → ALP (SIGMOD 2024), which is ~49% better than Gorilla and ~43% better than Chimp. None of those have a usable Go implementation: ALP is C++ (cgo would break our CGO_ENABLED=0 release builds), and the Gorilla Go ports are either stale since 2018 or archived.

A dependency-free alternative beats it. The classic byte-shuffle filter (transpose the 8 byte-planes of each float64 — what Blosc/HDF5/Parquet do) feeding our existing compressors, on 160 KB samples:

input fpc shuffle+zstd shuffle+xz
random float64 143 KB 134 KB 128 KB
smooth series 112 KB 127 KB 106 KB

Shuffle+xz wins on both, needs no new dependency, is ~20 lines of pure Go, composes with every compressor ccat already has, and generalises to float32/int64/any fixed-width record via a shuffle:N argument — much more in the spirit of the mutator pipeline than a single-purpose codec.

So: not merging github.com/spenczar/fpc (57★, dormant since Oct 2023) for a narrower win. A shuffle/unshuffle mutator pair is the better direction.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant