Is your feature request related to a problem? Please describe.
Dataset loading can be slower than necessary when the same dataset needs to be processed repeatedly. After the first processing step, the dataset could be reused more efficiently if it were stored on disk in the HData representation instead of being rebuilt from the original source every time.
Describe the solution you would like
Add an option to persist processed datasets to disk as HData, and load that representation directly on subsequent runs when available.
The option should allow users to:
- Enable or disable
HData caching explicitly.
- Choose or infer a cache/output path (reuse the
store_on_disk strategy we have already).
- Reuse the cached
HData representation when the source dataset and processing configuration have not changed.
- Fall back to normal processing when no cached representation exists.
Describe alternatives you've considered
An alternative is to keep relying on the existing dataset processing flow and let users manually serialize processed objects themselves. However, this might make repeated experiments slower and pushes caching logic onto every user or example script.
Another option is to cache intermediate preprocessing artifacts, but storing the final HData representation should provide the most direct speedup for repeated loads.
Additional context
This would be useful for workflows that repeatedly train or evaluate models on the same dataset. The first run would pay the processing cost, while later runs could load the already processed HData representation from disk.
Potential implementation considerations:
- Compatibility across HyperTorch versions.
- Clear documentation for the new option and expected cache behavior.
Is your feature request related to a problem? Please describe.
Dataset loading can be slower than necessary when the same dataset needs to be processed repeatedly. After the first processing step, the dataset could be reused more efficiently if it were stored on disk in the
HDatarepresentation instead of being rebuilt from the original source every time.Describe the solution you would like
Add an option to persist processed datasets to disk as
HData, and load that representation directly on subsequent runs when available.The option should allow users to:
HDatacaching explicitly.store_on_diskstrategy we have already).HDatarepresentation when the source dataset and processing configuration have not changed.Describe alternatives you've considered
An alternative is to keep relying on the existing dataset processing flow and let users manually serialize processed objects themselves. However, this might make repeated experiments slower and pushes caching logic onto every user or example script.
Another option is to cache intermediate preprocessing artifacts, but storing the final
HDatarepresentation should provide the most direct speedup for repeated loads.Additional context
This would be useful for workflows that repeatedly train or evaluate models on the same dataset. The first run would pay the processing cost, while later runs could load the already processed
HDatarepresentation from disk.Potential implementation considerations: