Does smallpond uses 3fs USRBIO API

[yuyuyu101]

From smallpond/io/filesystem.py, I noticed smallpond leverage 3fs directory garage feature to reduce directory tree removing latency. But I don't find any code related to USRBIO API, am I missing it? Why not smallpond implement USRBIO to accelerate load/write process?

[wangrunji0408]

Smallpond initially utilized USRBIO via fsspec, but the performance was suboptimal due to Python GIL. To address this, we developed a DuckDB extension to leverage USRBIO in DuckDB directly.

[yuyuyu101]

@wangrunji0408 thanks, I got it. do you have plan to open source this duckdb extension?