perf: hipo2npz memory efficiency - #1373
Open
c-dilks wants to merge 6 commits into
Open
Conversation
Co-Authored-By: Claude <noreply@anthropic.com>
c-dilks
marked this pull request as ready for review
August 4, 2026 22:26
c-dilks
requested review from
baltzell,
drewkenjo and
raffaelladevita
as code owners
August 4, 2026 22:26
c-dilks
commented
Aug 4, 2026
Member
Author
There was a problem hiding this comment.
Oops. I need to revert this when the host is back online.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The current
hipo2npzis not memory efficient, since it effectively reads and stores a full HIPO files' data structures in memory while producing the NPZ file.This PR replaces the memory storage with disk storage, by creating a directory of temporary files in
<output>.tmp/, where<output>is the output file name. Now the memory usage hovers under 500 MB typically for a DST file.I had Claude generate an NPZ
difftool, stored here ashipo2npz-diff; I used it to verify there is no difference in the resulting NPZ file before and after this PR.About Temporary Files
/tmpfile system, to keep the temp files and output NPZ file on the same disk.SIGINTorSIGTERMSIGKILL), since the JVM does not have a chance to remove them in that case; this means if we run on SLURM, it may be possible that OOM or time-limit kills from SLURM will leave temp files.<output>.tmpalready exists,hipo2npzwill immediately fail, preventing accidental deletion of something that was not created byhipo2npzAdditional changes
--num-eventsand--first-eventto convert a range of events