Skip to content

Fix range reads on whole-file caches - #2101

Open
nvbkdw wants to merge 1 commit into
fsspec:masterfrom
nvbkdw:fix-cached-negative-offsets
Open

Fix range reads on whole-file caches#2101
nvbkdw wants to merge 1 commit into
fsspec:masterfrom
nvbkdw:fix-cached-negative-offsets

Conversation

@nvbkdw

@nvbkdw nvbkdw commented Aug 15, 2026

Copy link
Copy Markdown

filecache and simplecache serve range reads from the local copy of a file rather than from the remote, and three things go wrong there:

  1. Negative start/end are documented on AbstractFileSystem.cat_file to count backwards from the end of the file, and remote backends implement them as HTTP suffix ranges. _cat_file instead passed the value straight to f.seek(), so a negative offset raised OSError(EINVAL) after the whole file had already been downloaded.

  2. The sync path reached AbstractFileSystem.cat_file, which needs f.size to resolve negative offsets. Both _open implementations return a bare io.BufferedReader, so it raised AttributeError. LocalFileOpener already attaches size to the raw file this way.

  3. WholeFileCacheFileSystem._check_file returns a (detail, path) tuple while the SimpleCacheFileSystem override returns the path alone. _cat_file and _cat_ranges assumed the latter, so on filecache they raised TypeError/AttributeError for a cached path, and _cat_ranges never downloaded an uncached one (the miss sentinel is False, not None).

This surfaced with zarr v3 sharded arrays, whose chunk index lives at the end of each object and is read with a suffix range: every read through simplecache downloaded the full shard and then failed on the seek.

`filecache` and `simplecache` serve range reads from the local copy of a
file rather than from the remote, and three things go wrong there:

1. Negative `start`/`end` are documented on `AbstractFileSystem.cat_file`
   to count backwards from the end of the file, and remote backends
   implement them as HTTP suffix ranges. `_cat_file` instead passed the
   value straight to `f.seek()`, so a negative offset raised
   `OSError(EINVAL)` after the whole file had already been downloaded.

2. The sync path reached `AbstractFileSystem.cat_file`, which needs
   `f.size` to resolve negative offsets. Both `_open` implementations
   return a bare `io.BufferedReader`, so it raised `AttributeError`.
   `LocalFileOpener` already attaches `size` to the raw file this way.

3. `WholeFileCacheFileSystem._check_file` returns a `(detail, path)`
   tuple while the `SimpleCacheFileSystem` override returns the path
   alone. `_cat_file` and `_cat_ranges` assumed the latter, so on
   `filecache` they raised `TypeError`/`AttributeError` for a cached
   path, and `_cat_ranges` never downloaded an uncached one (the miss
   sentinel is `False`, not `None`).

This surfaced with zarr v3 sharded arrays, whose chunk index lives at the
end of each object and is read with a suffix range: every read through
`simplecache` downloaded the full shard and then failed on the seek.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant