Skip to content

perf: hipo2npz memory efficiency - #1373

Open
c-dilks wants to merge 6 commits into
developmentfrom
heap-npz
Open

perf: hipo2npz memory efficiency#1373
c-dilks wants to merge 6 commits into
developmentfrom
heap-npz

Conversation

@c-dilks

@c-dilks c-dilks commented Aug 3, 2026

Copy link
Copy Markdown
Member

The current hipo2npz is not memory efficient, since it effectively reads and stores a full HIPO files' data structures in memory while producing the NPZ file.

This PR replaces the memory storage with disk storage, by creating a directory of temporary files in <output>.tmp/, where <output> is the output file name. Now the memory usage hovers under 500 MB typically for a DST file.

I had Claude generate an NPZ diff tool, stored here as hipo2npz-diff; I used it to verify there is no difference in the resulting NPZ file before and after this PR.

About Temporary Files

  • We do not use the /tmp file system, to keep the temp files and output NPZ file on the same disk.
  • The temporary files are deleted when:
    • the conversion is done
    • anything that "nicely" kills the job, such as SIGINT or SIGTERM
  • The temporary files remain if the job is not-so-nicely killed (SIGKILL), since the JVM does not have a chance to remove them in that case; this means if we run on SLURM, it may be possible that OOM or time-limit kills from SLURM will leave temp files.
  • If a directory, file, or symlink with the name <output>.tmp already exists, hipo2npz will immediately fail, preventing accidental deletion of something that was not created by hipo2npz

Additional changes

  • new options --num-events and --first-event to convert a range of events

@c-dilks c-dilks added the qcddat label Aug 3, 2026
@c-dilks
c-dilks marked this pull request as ready for review August 4, 2026 22:26
Comment thread etc/data/nnet

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oops. I need to revert this when the host is back online.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant