Describe the Bug
DirCache uses functools.lru_cache to limit the number of cached paths to max_paths:
if max_paths:
self._q = lru_cache(max_paths + 1)(lambda key: self._cache.pop(key, None))However, functools.lru_cache only calls the wrapped function (self._cache.pop) when a key is a cache miss (computing a value for a new key), NOT when an old key is evicted from lru_cache's internal index.
This causes three major issues in DirCache:
- Memory Leak (No Eviction on Write): Adding entries beyond
max_paths never deletes old entries from self._cache. self._cache grows without bound.
- Deferred Deletion on Read: When an old key that fell out of
lru_cache is read via __getitem__, _q treats it as a miss and calls self._cache.pop(key), deleting key from _cache at read time and raising a KeyError.
- Cache Destruction during Iteration:
__iter__ filters entries using k in self (__getitem__), triggering cascading lru_cache misses. As a result, calling list(dircache) deletes every single item in _cache and returns [].
Minimal Reproducible Example
from fsspec.dircache import DirCache
dc = DirCache(max_paths=2)
# 1. Add 4 items (exceeding max_paths=2)
dc["a"] = 1
dc["b"] = 2
dc["c"] = 3
dc["d"] = 4
# Bug 1: _cache size is 4 instead of 2
print("Actual size:", len(dc._cache)) # Output: 4 (Expected: 2)
# Bug 2: Iterating wipes out the cache and returns []
print("Keys yielded by iter:", list(dc)) # Output: []
print("Cache after iter:", dc._cache) # Output: {}
Expected Behavior
- Adding keys beyond
max_paths should immediately evict the least recently set keys from self._cache. len(dc._cache) should never exceed max_paths.
- Iterating over
dircache (list(dc)) should yield valid, unexpired keys without mutating or clearing the cache.
Environment Information
- Library:
fsspec
- Python Version: 3.8+
Suggested Fix
Replace functools.lru_cache with collections.OrderedDict (or standard dict popping) in DirCache to handle LRU ordering directly upon __setitem__ and __getitem__.
Describe the Bug
DirCacheusesfunctools.lru_cacheto limit the number of cached paths tomax_paths:However,
functools.lru_cacheonly calls the wrapped function (self._cache.pop) when a key is a cache miss (computing a value for a new key), NOT when an old key is evicted fromlru_cache's internal index.This causes three major issues in
DirCache:max_pathsnever deletes old entries fromself._cache.self._cachegrows without bound.lru_cacheis read via__getitem__,_qtreats it as a miss and callsself._cache.pop(key), deletingkeyfrom_cacheat read time and raising aKeyError.__iter__filters entries usingk in self(__getitem__), triggering cascadinglru_cachemisses. As a result, callinglist(dircache)deletes every single item in_cacheand returns[].Minimal Reproducible Example
Expected Behavior
max_pathsshould immediately evict the least recently set keys fromself._cache.len(dc._cache)should never exceedmax_paths.dircache(list(dc)) should yield valid, unexpired keys without mutating or clearing the cache.Environment Information
fsspecSuggested Fix
Replace
functools.lru_cachewithcollections.OrderedDict(or standard dict popping) inDirCacheto handle LRU ordering directly upon__setitem__and__getitem__.