oxihipo

Pure-Rust reader and writer for the HIPO v6 binary container used at Jefferson Lab CLAS12. Built so that read throughput meaningfully exceeds the C++ hipo4 reader on the same hardware, with an API that fits Rust idioms.

Reads and writes HIPO version 6 files. FFI and XRootD layers are intentionally out of scope; ROOT is reachable from the Python side through RDataFrame, and that side also carries a thin analysis layer (PDG masses, Lorentz vectors, pindex joins). The Rust crate itself stays physics-free.

Features

• Zero-copy columnar reads. bank.col::<T>("name") returns a Cow<[T]> that borrows straight from the decompressed record buffer when the bytes are aligned (always for 4-byte types), with a one-shot copy fallback otherwise. Fixed-length array columns (name/T#N) read as [T; N] (readable by Java, not by C++ hipo4 — see Writing · Array columns).

• Bounded memory, any file size. Records stream in one at a time via pread into a recycled buffer — the file is never mapped or read whole. A sequential scan of a 100 GB file holds ~one record resident (tens of MB), not the file; parallel scans hold one record per worker. Safe under a memory-capped batch allocation.

• One reader: Chain. Chain::open takes a single file, a directory, a glob, or a list of paths, and streams records on demand. chain.events() yields Result<OwnedEvent>, so a corrupt or truncated record surfaces as an Err (propagate with ?) rather than panicking.

  Files in a chain need not declare the same banks — a run period rarely does, since a pass-2 cook adds one and an MC file carries MC::Lund. The chain's dictionary is the union, and a bank absent from a file yields empty entries for that file's events. Only genuine conflicts are refused: one name with two layouts, or one (group, item) claimed by two banks — the latter matters because banks are located by id, so a collision would decode one file's bytes against another's schema.

• Data-parallel scans. Chain::for_each(threads, f) fans the work across cores out of order (threads = 0 ⇒ all cores, 1 ⇒ sequential, n ⇒ exactly n); shared state in f is atomic or locked.

• Random access that doesn't re-inflate. chain.event(i) keeps the last decoded record on the chain, so a run of lookups landing in the same record — what replaying a list of interesting event indices actually looks like — costs a slice rather than a fresh whole-record decompression. Sequential iteration never consults the cache, so scan throughput is unchanged.

  For a list of indices, chain.read_columns_at(sel, entries, threads) groups them by the record that holds them, so each record is read once however the list is ordered and the groups run in parallel. A scattered list used to cost one whole-record decompression per index — 256 of them, 7 ms against 13 µs for the same count ascending; now ~270 µs either way.

• Compression beyond stock LZ4 / Gzip. Two opt-in format extensions that decompress only what an analysis actually reads: Lz4PerBank (per-bank streams) and Lz4PerColumn (per-column streams) — see the benchmarks below.

  Cross-implementation portability. none / lz4 / lz4best are read by every HIPO implementation. gzip is read by the Java reader but not by the reference C++ reader, which has no gzip decode path. The lz4perbank / lz4percolumn extensions are not read by the released C++/Java readers.

• Event tagging. Filter events by their per-event tag with pushdown (Filter::event_tag_any, read without inflating any bank), name the bits with the tag_flags! macro, persist the name registry in the file so tags self-describe, and write tagged DSTs (Chain::skim_tagged, or Python f.filtered(event_tag="dvcs") / f.skim(tags=…)). The pushdown costs nothing on reads that don't use it (benchmarked below). See the event-tagging design & roadmap.

• A pure-Rust build is available (default-features = false, on lz4_flex), though lz4-c is on by default: lz4-c (C LZ4 bindings — faster decode + Lz4Best HC), lz4-apple (Apple libcompression decode), and mimalloc-allocator.

Add to your project

Not yet published to crates.io — depend on it via git:

[dependencies]
oxihipo = { git = "https://github.com/mathieuouillon/oxihipo" }

Built on a lean, pure-Rust dependency set — lz4_flex 0.14 and flate2 1 (zlib-rs backend) for de/compression, rayon 1 for parallel reads, plus bytemuck 1, serde 1, and thiserror 2. The optional lz4-c feature pulls lz4-sys (C LZ4) for Lz4Best HC and faster decode.

Quick start

use oxihipo::{Chain, Filter};

# fn main() -> oxihipo::Result<()> {
// Single file or many — `Chain` is the sole reader entry point.
let chain = Chain::open("rec.hipo")?
    .with_filter(Filter::require(["REC::Particle"]))?;

// Plain `for` loop. Each item is a `Result<OwnedEvent>`; `?` propagates a
// corrupt record. Each `OwnedEvent` is a slice into a shared, ref-counted
// record buffer — no per-event allocation.
for ev in chain.events() {
    let ev = ev?;
    let p = oxihipo::or_continue!(ev.bank("REC::Particle"));
    for r in 0..p.rows() {
        let pid: i32 = p.get("pid", r);
        let px:  f32 = p.get("px",  r);
        let _ = (pid, px);
    }
}
# Ok(()) }

Multi-file chains are first-class:

use oxihipo::Chain;

# fn main() -> oxihipo::Result<()> {
// `Chain::open` takes a single file, a directory, a glob (e.g.
// "data/*.hipo"), or a list of paths.
let chain = Chain::open("/data/cooked/run5042")?;

// Iterate every event of every file, in input order.
let mut total_rows: u64 = 0;
for ev in chain.events() {
    let ev = ev?;
    total_rows += ev.bank("REC::Particle").map_or(0, |b| b.rows() as u64);
}
println!("{total_rows} REC::Particle rows across the chain");
# Ok(()) }

Process every event with for_each — the threads argument is the only difference between a single-threaded and a parallel scan (1 = on the calling thread in order, 0 = one worker per logical CPU, n = exactly n workers). Shared state is held in atomics because the parallel modes visit events out of order:

use std::sync::atomic::{AtomicU64, Ordering};
use oxihipo::Chain;

# fn main() -> oxihipo::Result<()> {
let chain = Chain::open("/data/cooked/run5042")?;

let total_rows = AtomicU64::new(0);
chain.for_each(0, |ev| {                          // `0` → all cores; `1` → single-threaded
    if let Some(b) = ev.bank("REC::Particle") {
        total_rows.fetch_add(b.rows() as u64, Ordering::Relaxed);
    }
})?;
println!("{} REC::Particle rows across the chain", total_rows.into_inner());
# Ok(()) }

Writing is closure-driven:

use oxihipo::{Compression, Writer};

# fn run(dict: &oxihipo::Dict) -> oxihipo::Result<()> {
let mut w = Writer::create("out.hipo")
    .schemas(dict)
    .compression(Compression::Lz4)
    .build()?;
w.event(|ev| {
    ev.bank("REC::Particle", |b| {
        b.row(|r| { r.set("pid", 11_i32)?; r.set("px", 0.5_f32) })?;
        Ok(())
    })?;
    Ok(())
})?;
w.finish()?;
# Ok(()) }

Status

• Single oxihipo library crate. No bundled binary; downstream consumers build whatever frontend they need on top.
• cargo test, cargo clippy --all-targets -- -D warnings, and cargo fmt --check all clean.
• Validated on a 1.7 GB CLAS12 file (rec_clas_022050.evio.00000.hipo): a sequential Chain::events() scan reads all 187,941 events at ~257 kev/s. Chain::for_each (parallel mode) fans the same scan across cores; measure throughput on your hardware with the bench_par example.

Python bindings

A columnar, uproot-shaped Python binding lives in py/: a HIPO bank reads like an Awkward jagged branch, with the per-event loop running in Rust behind a released GIL and column buffers moved into NumPy zero-copy.

import oxihipo as ox

f = ox.open("run5042.hipo")                        # file | dir | glob | list
p = f.arrays("REC::Particle", ["pid", "px"])        # ak.Array: N * var * {pid, px}

for chunk in f.iterate("REC::Particle", step_size="200 MB"):   # bounded memory
    ...                                             # 10-100 GB inputs in ~constant RAM

# I/O-bound filesystem (ifarm /volatile): read with N processes to beat the
# per-process bandwidth ceiling — guard the script with `if __name__ == "__main__":`
p = ox.arrays("/volatile/run5042/*.hipo", "REC::Particle", ["px"], workers=8)

Writing is uproot-shaped and columnar too — create a new file, recreate to replace one, or update to decorate an existing file with a derived bank (an ML score, a computed kinematic) without rewriting the physics banks:

w = ox.update("dst.hipo", "decorated.hipo")     # copies every event verbatim
w.new_bank("ML::pred", {"score": "F"})            # declare a new bank
w.extend({"ML::pred": {"score": scores}})        # one float32 per event
w.close()

Past reading, the Python side turns columns into physics without hardcoded constants or hand-written joins:

p  = f.arrays("REC::Particle", ["pid", "px", "py", "pz"])
v  = ox.to_vector(p, mass="pdg")            # masses from pid; E, pt, eta, mass
(v[:, 0] + v[:, 1]).mass                    # invariant mass

ev = ox.link(f.arrays(["REC::Particle", "REC::Calorimeter"]))
ev["REC::Calorimeter"].particle.px          # follow pindex, both directions
ev["REC::Particle"]["REC::Calorimeter"]     # that particle's detector rows

h = f.map_reduce(analyze, "REC::Particle", workers=8)   # analyze runs IN the worker
arr = f.to_dask("REC::Particle")            # lazy dask-awkward, columns projected

to_vector needs pip install oxihipo[vector] and to_dask needs oxihipo[dask] — those two are real extras, unlike [awkward]/[pandas]/ [arrow], which ship by default.

pdg_mass handles the two CLAS12 codes that break general PDG helpers — pid == 0 (an unidentified track, so nan rather than an exception) and the Geant3 nuclei 45/46/47/49, which are not PDG codes at all. map_reduce is the one that matters for scale: workers= parallelises only the read, leaving the physics on one core, where a CLAS12 selection actually spends its time.

And for ROOT users, rdataframe hands a selection to RDataFrame via Awkward's generated RDataSource — jagged banks become RVec columns, no copy (iterate_rdataframe streams it for bigger-than-RAM inputs). This is the one path needing something pip can't provide — a working ROOT/PyROOT (conda-forge or system):

df = ox.rdataframe("run5042.hipo", "REC::Particle", ["px", "py", "pid"])
df.Define("pt", "sqrt(REC_Particle_px*REC_Particle_px + REC_Particle_py*REC_Particle_py)") \
  .Histo1D("pt")                                   # REC::Particle/px → REC_Particle_px

Install with pip install oxihipo (wheels for Linux / macOS / Windows, CPython ≥ 3.10) — every backend ships by default, so library="ak" / "pd" / "np" / "arrow" all work with no extra to hunt for. Or build from source with maturin (cd py && maturin develop --release); see the Python guide, py/README.md, and py/examples/. Design notes: Python binding design.

Nearly-native speed. Reading REC::Particle px,py,pz,pid from a 9.1 GB CLAS12 file (598,738 events, 4.7 M particles; Apple M4 Pro, all cores, warm cache):

	throughput	vs Rust
Rust read_columns	6.3 GB/s	1.00×
Python read_columns (NumPy)	5.8 GB/s	0.91×
Python arrays (Awkward)	5.6 GB/s	0.89×

The per-event decode runs in Rust behind a released GIL and columns move into NumPy zero-copy, so the binding costs ~10%. Method + reproduction: Python vs Rust benchmark (examples/bench_columns.rs, py/examples/bench_columns.py).

Against the other HIPO implementations. Same file, same work, each using its own documented fast path; every implementation prints a checksum and the run is only valid if they all agree. 200k events, LZ4, Apple Silicon. warm = best of 10 in-process passes, cold = first pass in a fresh process:

per-event read	Rust	C++	Java
iterate only, no bank	6.0	16.3	6.7
one column (pid)	13.0	16.9	10.1
two columns (pid,px)	15.8	17.3	9.7
all 8 columns	33.0	19.9	16.4
cold, two columns	17.2	18.3	40.0

Read that honestly. Java is quick once its JIT is warm and wins several rows, but a cold pass — what a short job or a per-file batch worker actually pays — costs it 2.6–4.3× more. C++ barely moves as columns are added and so wins the all-columns row outright: oxihipo's per-event accessor sets up a column view per call, which never amortises over 1–5 rows.

The answer to that is the columnar API, which is also what the Python binding drives:

columnar read (read_columns)	1 thread	all cores
Rust, all 8 columns	16.9	7.1
Python, all 8 columns	14.8	7.3

All-columns drops 33.0 → 16.9 ms serial and 7.1 ms across cores, and the Python binding costs ~0–10% over native Rust rather than a multiple. If you read many columns, don't loop events. Full matrix, compression sweep, and the two measurement traps that produced wrong numbers before being caught: vs C++ and Java — harness in benches/cross-impl/.

Layout

Single-crate repo (oxihipo — error, wire, compress, schema, event, read, write). Inside src/:

• error.rs, prelude.rs
• wire/ (private) — constants, bytes, headers, record decompression
• compress.rs (private) — LZ4/gzip + reusable ScratchBuf
• schema/ — Schema, Dict, DataType, typed ColumnHandle<T>
• event/ — Event (raw), EventCtx (with &Dict), Bank, Composite, OwnedEvent, internal BankBuilder / EventBuilder
• read/ — Chain (the sole reader, Arc<FileInner>-backed), ChainEventIter, Filter, parallel for_each (ChainStats)
• write/ — Writer builder, BankWriter, RowWriter, Compression

Build

cargo build --release
cargo test

# Examples
cargo run --release --example write     -- /tmp/demo.hipo
cargo run --release --example read      -- /tmp/demo.hipo
cargo run --release --example parallel  -- /path/to/file.hipo 0
cargo run --release --example bench_par -- /path/to/file.hipo 0

Notable design decisions

• Chain is the only reader. A chain of one file is the common case; multi-file chains stream records on demand and present the union of their dictionaries. Files need not declare the same banks — a run period rarely does — but one bank name with two layouts, or one (group, item) claimed by two banks, is still refused: banks are located by id, so a collision would decode one file's bytes against another's schema.
• Event<'a> carries only borrowed bytes; the typical handle is EventCtx<'a> = (Event<'a>, &Dict), which lets ev.bank("REC::Particle") resolve the schema without separate juggling.
• Bank::col::<T>("name") is the single typed getter (generic; replaces six per-type methods). Returns Cow<[T]>: zero-copy when the bank's bytes are aligned to T (always for 4-byte types; usually for 8-byte types), and a one-shot read_unaligned copy otherwise — matching the C++ reader's memcpy semantics.
• Bank::get::<T>("name", row) is the infallible scalar accessor for hot loops. Type is inferred from the binding (let pid: i32 = b.get("pid", r);) and missing/wrong-type columns return T::default().
• ColumnHandle<T> lives on Schema, resolved once via schema.handle::<T>("name"). Inside hot loops, bank.read(h) is a constant-time cast — no per-event name lookup.
• mimalloc is gated behind the mimalloc-allocator feature (off by default), for allocation-heavy workloads where the system allocator underperforms on macOS.

Performance on shared filesystems

When the input lives on a network filesystem — JLab ifarm's /volatile and /cache (Lustre), NFS, etc. — I/O latency dominates wall time. The reader issues one pread per record and relies on the kernel's per-descriptor readahead to fetch the next record while the current one decompresses; parallel mode keeps several records in flight across workers. Resident memory stays bounded (one record per worker) no matter how large the file, so a wide parallel scan won't be OOM-killed by a memory-capped batch allocation.

If you are still I/O-bound, the levers are user-side:

• File striping. A Lustre file on a single OST is bandwidth-capped no matter the thread count. New outputs: lfs setstripe -c 4 outfile.hipo; existing files: lfs migrate -c 4 file.hipo.
• Thread oversubscription. Pass threads = 2 × num_cpus to for_each to hide network page-fault stalls.
• Stage to local scratch. cp /volatile/.../file.hipo /scratch/$USER/ before analysing — local disk easily beats Lustre per single client.

When LZ4 itself is the bottleneck

Once page-fault stalls are masked, LZ4 inflate dominates. The stock HIPO format stores one LZ4 block per record, so reading any bank inflates every bank. Two opt-in format extensions fix that by storing sub-record streams and inflating only what an analysis touches:

Compression is a (codec × layout) pair, not a flat list of formats:

use oxihipo::{Codec, Compression, Layout};

Compression::new(Codec::Zstd, Layout::PerColumn).with_zstd_level(3)

• Codec — what squeezes the bytes: None, Lz4, Lz4Hc, Gzip, Zstd (levels 1–6).
• Layout — what gets squeezed separately: PerChunk (one stream for the whole record — the stock HIPO shape), PerBank (one per bank type, so ev.bank(…) inflates only that bank), PerColumn (one per (bank, column), cross-event contiguous).

All 15 pairs work. The six that predate the matrix keep their names — Compression::Lz4PerColumn is Lz4Hc × PerColumn, and still means that.

The matrix, measured

248 MB of a real CLAS12 DST, 599k events, Apple M4 Pro, single thread, warm cache. Best-of-3, reps interleaved across cells. scan all reads a value from every bank of every event; scan 1 col reads one column across the file through read_columns. Every cell checksums identically — cargo run --release --example bench_compression -- <file>:

codec	layout	size	ratio	write	scan all	scan 1 col
None	PerChunk	248 MB	1.00×	0.14 s	77 ms	36 ms
None	PerBank	228 MB	1.09×	0.18 s	95 ms	38 ms
None	PerColumn	228 MB	1.09×	0.24 s	113 ms	32 ms
Lz4	PerChunk	146 MB	1.70×	0.33 s	120 ms	78 ms
Lz4	PerBank	142 MB	1.75×	0.38 s	125 ms	69 ms
Lz4	PerColumn	136 MB	1.83×	0.43 s	110 ms	28 ms
Lz4Hc	PerChunk	126 MB	1.96×	6.07 s	109 ms	64 ms
Lz4Hc	PerBank	126 MB	1.97×	5.67 s	125 ms	66 ms
Lz4Hc	PerColumn	122 MB	2.03×	7.73 s	116 ms	32 ms
Gzip	PerChunk	117 MB	2.12×	2.37 s	416 ms	370 ms
Gzip	PerBank	116 MB	2.14×	2.35 s	421 ms	349 ms
Gzip	PerColumn	107 MB	2.32×	2.58 s	112 ms	30 ms
Zstd	PerChunk	124 MB	2.00×	0.75 s	228 ms	184 ms
Zstd	PerBank	121 MB	2.05×	0.80 s	231 ms	168 ms
Zstd	PerColumn	112 MB	2.22×	0.72 s	118 ms	35 ms

Three things the matrix makes obvious:

The layout decides the selective read, not the codec. Every PerColumn cell reads one column in 28–35 ms regardless of codec — even Gzip, which takes 349–370 ms in the same codec on the whole-record layouts. A PerChunk codec must inflate everything to reach anything; that is the whole point of the split layouts, and it dwarfs the codec choice.

Zstd × PerColumn is the best default. It is smaller than Lz4Hc × PerColumn (2.22× against 2.03×), reads a column just as fast, and writes in 0.72 s where LZ4-HC takes 7.73 s — 10.7× faster to write for a smaller file. If you were reaching for Lz4PerColumn, reach for this instead.

Gzip × PerColumn still wins on size (2.32×) if bytes are what you are paying for, at 3.6× the write cost of Zstd. Plain Lz4 × PerColumn is the fastest to write of any compressing cell (0.43 s) and still beats every whole-record codec on selective reads.

Cross-implementation compatibility

Only the six pairs that predate the matrix have wire tags hipo-cpp and hipo-java know: the four PerChunk codecs (None/Lz4/Lz4Hc/Gzip) plus Lz4Hc × PerBank and Lz4Hc × PerColumn. The other nine are oxihipo extensions those readers reject as an unknown tag.

For the two split tags, the released C++ and Java readers do not know them, but the feature/bybank-bycolumn-compression branches of hipo-cpp and hipo-java implement them, and all three libraries produce identical checksums on the same files. Their on-disk ext_format_version — 2 for by-bank, 1 for by-column — is therefore a cross-implementation contract. 0.7.0 bumped it and broke both; 0.7.1 puts it back. The split-record directory stays LZ4 whatever the stream codec, so those two tags remain byte-compatible. Composite banks already written to a split-codec file by 0.6.0 or earlier cannot be recovered — re-convert from the original. See Format versions.

On a 29.7 GB ifarm skim (parallel, Lustre) the by-bank format hit 36.6 Mev/s — 25× the Lz4 baseline.

Reproduce (both benchmarks read the same per-format files):

OXIHIPO_BENCH_KEEP=/tmp/fmt \
  cargo run --release --example bench_read_compression -- rec.hipo 3 50000
python py/examples/bench_compression.py /tmp/fmt 20000 3   # Python side
# and re-emit a real file as Lz4PerBank:
cargo run --release --example recook_by_bank -- in.hipo out_by_bank.hipo

Full format guide, trade-offs, and the complete tables (all read scopes, Rust + Python): Compression formats and Benchmarks.

Known gaps

• Event tagging, Phases 4 & 5 — writer-side record-tag routing (binned skims so a whole category of events shares records, for cheap coarse pushdown) and a schema'd TAG::Event bank (for >32 flags or per-tag payloads like a classifier score). Deferred until a workload needs them; the five shipped phases cover the per-event-u32 case. See the event-tagging roadmap.
• Intra-stream parallel inflate for the by-bank / per-column formats. Closed — measured, not worth doing. The premise was wrong: streams do not parallelise across banks. for_each parallelises over records (the work unit is one record), and within a record a worker inflates streams lazily and serially. So the ceiling on a parallel scan is the record count, not the size of any one stream — and record count is already tunable with max_record_bytes. On real CLAS12 data, dropping 32 MB → 4 MB records took a full parallel scan from 73 ms to 59 ms (6.8× → 8.2× on 12 cores) for +1.0% file size; splitting a stream would need a writer-side format change, cost ratio, and chase a bottleneck the existing knob already reaches. For Lz4PerColumn it could never have paid off at all — the largest column stream is only 2–8% of a record. See Record size and parallel scaling (examples/profile_streams.rs, examples/record_size_scaling.rs).

CI gates

Every PR runs, on Linux, macOS and Windows — endianness, alignment and the unsafe decode paths are target-sensitive, so one platform is not enough:

• cargo fmt --check and cargo clippy --all-targets -- -D warnings
• cargo test --all-targets — unit + integration tests, including a broad end-to-end pass over the read/write API — metadata, sequential + random-access
  • parallel reads, every data type, filters, the columnar reader, skim, and multi-file chains (tests/end_to_end.rs) — plus an all-compression-format write → read → cross-format-skim round-trip (tests/all_formats.rs), a golden file from the reference C++ writer (tests/cpp_golden.rs), and property tests (tests/proptests.rs)
• cargo test --doc — --all-targets compiles doctests but never runs them, so a broken /// example would otherwise ship silently

and on Linux:

• the declared MSRV (1.95) builds and tests, so it cannot silently drift
• every feature combination compiles (cargo hack --feature-powerset), plus full test runs under --no-default-features — the pure-Rust lz4_flex decode path nothing else exercises — and --all-features
• cargo-deny — advisories, licences, banned/duplicate crates, sources
• the fuzz targets build on nightly, so an API change cannot rot them unnoticed (running the fuzzer stays a manual job)
• cargo doc --no-deps with RUSTDOCFLAGS=-D warnings
• a smoke-run of the core examples end-to-end (write → read → recook → read; plus a small all-format encode/read sweep) so a runtime break in an example fails CI

The Python wheel workflow additionally runs pytest, mypy and mypy.stubtest against a freshly built wheel, and builds wheels for Linux (x86_64 + aarch64), macOS (x86_64 + arm64) and Windows.

License

Licensed under the MIT License.