feat: scoped zero-copy get(key, Mapper) across every DB type + RocksIterator by dfa1 · Pull Request #57 · dfa1/rocksdbffm

dfa1 · 2026-08-08T09:38:35Z

Summary

Implements #55. Adds ReadWriteDB.withPinned(...) — a scoped, zero-copy read built on the rocksdb_pinnable_handle_t C API (rocksdb_get_pinned_v2/_cf_v2, rocksdb_pinnable_handle_get_value, rocksdb_pinnable_handle_destroy), as an alternative to get(byte[]) for hot read paths.

PinnedReader<R> — functional callback, R read(MemorySegment value) throws Exception, invoked with a read-only view of the pinned value. No intermediate byte[] copy.
RocksDB.withPinned0 — private core shared by both public overloads (plain and column-family). Checks errptr before the null-handle check (a NULL return means either NotFound or error — the two are only distinguished by errptr), binds the value pointer to a confined Arena via reinterpret(len, arena, null) (the null cleanup is deliberate — the view borrows from the handle, it does not own the memory).
The finally block closes the arena before destroying the native handle. Reversed, there'd be a window where a still-valid MemorySegment points at block-cache memory the handle already released. With this order, a view that escapes the callback fails loudly: IllegalStateException if used after the call returns, WrongThreadException if handed to another thread — both covered by tests, both firing as expected (not silently passing).
get_pinned_*_v2 is not marked critical (a Get can block on disk I/O; a critical downcall would stall GC for its duration). get_value/destroy are Linker.Option.critical(false), which needed NativeLibrary.lookup to grow a Linker.Option... parameter.

Deviation from the issue's sketch: no separate RocksPinned class with its own Linker/SymbolLookup. The four method handles live in RocksDB.java next to the existing get_pinned/PinnableSlice handles — that's where every other rocksdb_t* handle in this codebase lives (RocksDB.java's own doc comment calls it out as "the single holder of all rocksdb_t* method handles"). The CF parameter is the existing ColumnFamilyHandle wrapper rather than a raw MemorySegment, matching every other CF-taking method here.

Test plan

WithPinnedTest (11 cases): happy path, absent key, empty value, escape-after-return (IllegalStateException), escape-to-another-thread (WrongThreadException), reader exception propagates and cleanup still runs, 50k-iteration loop as a leak smoke test (scaled down from the issue's 100k-with-RSS-measurement to fit a fast unit-test suite), CF overload (happy/absent/isolation), and a default-CF equivalence test proving withPinned and withPinned(cf, ...) are wired to the right native symbols.
./mvnw test -pl core -am — 415 tests, 0 failures, 0 errors
./mvnw javadoc:javadoc -pl core — zero output
New JMH comparison in FfmBenchmark: an Instant stored as an 8-byte long, read via get(byte[]) + manual deserialize vs. withPinned deserializing straight out of the pinned view. Quick local run (1 fork, short warmup): ~7.0M ops/s (byte[]) vs. ~8.1M ops/s (withPinned), consistent with skipping one heap allocation + copy per read. BenchmarkRunner lists both as FFM-only (no JNI equivalent), same treatment as the existing MemorySegment tier.

🤖 Generated with Claude Code

Finishes #55/#57. withPinned is now get(key, Mapper<R>) — a fourth tier alongside byte[]/ByteBuffer/MemorySegment, present everywhere those already are: ReadWriteDB, ReadOnlyDB, TtlDB, SecondaryDB, BlobDB, OptimisticTransactionDB, TransactionDB, Transaction, and (as key(Mapper<R>)/value(Mapper<R>), since iterators have no get()) RocksIterator. Two native mechanisms, one caller-facing shape: - rocksdb_t*-backed wrappers (ReadWriteDB, ReadOnlyDB, TtlDB, SecondaryDB, BlobDB, OptimisticTransactionDB via its baseDb) reuse RocksDB.withPinned/withPinnedCf against rocksdb_get_pinned_v2/_cf_v2, exactly as already built for ReadWriteDB. - TransactionDB and Transaction have no _v2/pinnable_handle_t equivalent in c.h — only the older rocksdb_pinnableslice_t API (rocksdb_transactiondb_get_pinned(_cf), rocksdb_transaction_get_pinned (_cf)). Added a second shared core, RocksDB.withPinnableSlice, with the identical arena-before-destroy ordering but built on rocksdb_pinnableslice_value/destroy instead. Each caller passes in its own already-bound value/destroy MethodHandles rather than re-mapping the symbols centrally. - RocksIterator has neither: rocksdb_iter_key/value hand back a pointer into a buffer the iterator reuses in place on the next seek/next/prev, with no separate destroy step at all. key(Mapper)/ value(Mapper) still bind the view to a confined Arena that closes the moment the callback returns — not because the memory would be freed (it isn't), but because it turns the exact bug an earlier commit (b7c8ba1, "scope iterator segments to the current position") fixed for keySegment()/valueSegment() into a structural impossibility instead of a documented caveat: a view can no longer silently start reporting a different position's bytes after next()/prev() runs, because the arena that backs it no longer exists. Two API changes on top of the expansion, both requested mid-review: - PinnedReader<R> is now Mapper<R>, with one method, R map(MemorySegment). - Mapper.map no longer declares throws Exception. That was there only to let a caller's deserialization logic propagate checked exceptions; dropping it cascades through every get/key/value overload built on it, so none of them declare throws Exception either — the same Function<T,R>-shaped contract as the rest of java.util.function. Checked exceptions a callback body still needs (e.g. Thread#join's InterruptedException in a test) now have to be caught inside the lambda, same as implementing any other checked-exception-free functional interface. Test coverage: WithPinnedTest renamed to PinnedGetTest (its own method names lost the "pinnedReader" segment too, now just "zeroCopy" where the type name would have leaked into the test name); a new BlobDBTest since none existed on this branch; happy-path/absent-key/CF coverage added to ReadOnlyDBTest, TtlDBTest, SecondaryDBTest, OptimisticTransactionDBTest, TransactionDBTest (both the TransactionDB- level and Transaction-level overloads); three new RocksIteratorTest cases, including one that asserts a key(Mapper) view used after next() throws IllegalStateException — the regression the arena-scoping specifically exists to prevent for the iterator path. Full suite: 443 tests, 0 failures. javadoc:javadoc: zero output. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

dfa1 · 2026-08-08T12:06:31Z

Update: expanded scope

This PR grew well past the original ReadWriteDB-only `withPinned`. Summary of what changed since the title/description above:

Renamed `withPinned` → `get(MemorySegment key, Mapper fn)` / `get(ColumnFamilyHandle cf, MemorySegment key, Mapper fn)` — it's now the fourth tier of `get`, alongside `byte[]`/`ByteBuffer`/`MemorySegment`, everywhere those already exist:
- `ReadWriteDB`, `ReadOnlyDB`, `TtlDB`, `SecondaryDB`, `BlobDB`, `OptimisticTransactionDB` (all backed by a plain `rocksdb_t*`, reuse the existing `_v2`/`pinnable_handle_t` mechanism)
- `TransactionDB` and `Transaction` (no `_v2` equivalent in `c.h` — built a second shared core, `RocksDB.withPinnableSlice`, on the older `rocksdb_pinnableslice_t` API with the identical arena-before-destroy ordering)
- `RocksIterator.key(Mapper)` / `.value(Mapper)` — no `get()` there, so named to match the existing `key()`/`value()` methods. Different hazard than the DB case (no separate destroy step; the iterator reuses its buffer in place on the next seek/next/prev) but the same fix: binding the view to a confined arena that closes when the callback returns turns "silently reports the next position's bytes" into a thrown IllegalStateException.
PinnedReader<R> renamed to Mapper<R>, with one method, R map(MemorySegment).
Dropped throws Exception from Mapper.map and every get/key/value overload built on it — same checked-exception-free contract as java.util.function.Function. A callback that still needs to run something that throws checked (e.g. Thread#join) now catches it inside the lambda, same as any other functional interface without a throws clause.

Test coverage expanded to match: WithPinnedTest → PinnedGetTest (method names also dropped the pinnedReader segment, now just zeroCopy), a new BlobDBTest, and happy-path/absent-key/CF/thread-escape coverage added across ReadOnlyDBTest, TtlDBTest, SecondaryDBTest, OptimisticTransactionDBTest, TransactionDBTest (both TransactionDB- and Transaction-level), and RocksIteratorTest.

Full suite: 443 tests, 0 failures, 0 errors. javadoc:javadoc -pl core: zero output.

dfa1 · 2026-08-08T13:14:23Z

+			}
+			long len = lenSegment.get(ValueLayout.JAVA_LONG, 0);
+			MemorySegment view = data.reinterpret(len, arena, null).asReadOnly();
+			return fn.map(view);


we should have same guarantee of get() => the mapper should not return null here

(same for value() I guess)

Fixed — RocksIterator.key(Mapper)/value(Mapper) now reject a null return from fn with NullPointerException (Objects.requireNonNull), same as get(key, Mapper). Also tightened Mapper#map's own Javadoc to state the non-null contract once, centrally. Added regression tests in RocksIteratorTest. Pushed in c7fe05d.

dfa1 · 2026-08-08T13:15:46Z

+			} catch (Throwable t) {
+				throw RocksDBException.wrap("get_pinned failed", t);
+			}
+			return RocksDB.withPinnedCore(arena, err, pin, MH_PINNABLESLICE_VALUE, MH_PINNABLESLICE_DESTROY, fn);


this is duplicated several times... perhaps we should create a ticket to track it down (e.g. kill duplicated code by introducing PinnableSlice object)

Agreed, filed #58 to track it — extract a PinnableSlice wrapper owning the single rocksdb_pinnableslice_value/_destroy mapping so RocksDB, Transaction, and TransactionDB stop each keeping their own copy. Leaving it out of this PR's scope since it touches call sites well beyond withPinned.

looks like I'm crazy but second comment is claude actually :)

dfa1

good overall, some issues were found

…terator Adds ReadWriteDB.get(MemorySegment, Mapper) and friends — a scoped, zero-copy read built on the rocksdb_pinnable_handle_t C API (rocksdb_get_pinned_v2/_cf_v2, rocksdb_pinnable_handle_get_value, rocksdb_pinnable_handle_destroy) as an alternative to get(byte[]) for hot read paths, extended to every DB wrapper in this codebase: - ReadWriteDB, ReadOnlyDB, TtlDB, SecondaryDB, BlobDB, OptimisticTransactionDB — backed by a plain rocksdb_t*, share the rocksdb_pinnable_handle_t mechanism via RocksDB.withPinned/withPinnedCf. - TransactionDB and Transaction — no _v2 equivalent in c.h, so these go through the older rocksdb_pinnableslice_t API instead, sharing the same lifetime contract via RocksDB.withPinnedCore (parameterized on the value/destroy MethodHandle pair for whichever native handle kind applies). - RocksIterator.key(Mapper)/value(Mapper) — same zero-copy-view-scoped-to- the-callback idea, turning "escaped view silently reports the next position's bytes" into a loud IllegalStateException/WrongThreadException instead. Mapper<R> is the callback: R map(MemorySegment value), invoked with a read-only view bound to an arena that closes the moment the callback returns. A null return from the mapper is rejected with NullPointerException everywhere it's used, rather than silently conflated with "key not found" (Optional.empty()) or accepted as a valid result. Also: - NativeLibrary.lookup grows a Linker.Option... parameter so rocksdb_pinnable_handle_get_value/destroy (pure pointer arithmetic, no heap-array access) can be marked Linker.Option.critical(false). get_pinned_*_v2 stays non-critical since it can block on disk I/O. - JMH: new Instant-deserialize comparison (byte[] get + manual deserialize vs. zero-copy Mapper) and a blob-size sweep (8B..1MB) across byte[] get, get(MemorySegment,MemorySegment), and get(key, Mapper) to find the crossover point. Test coverage: PinnedGetTest plus per-DB-type happy-path/absent-key/CF/ thread-escape/null-mapper coverage across BlobDBTest, OptimisticTransactionDBTest, ReadOnlyDBTest, SecondaryDBTest, TransactionDBTest (both TransactionDB- and Transaction-level), TtlDBTest, and RocksIteratorTest. Follow-up tracked in #58: rocksdb_pinnableslice_value/_destroy are still mapped independently in RocksDB, Transaction, and TransactionDB rather than through a single PinnableSlice wrapper — pre-existing duplication, not introduced here, left out of scope.

dfa1 changed the title ~~feat: scoped zero-copy get via rocksdb_pinnable_handle_t (withPinned)~~ feat: scoped zero-copy get(key, Mapper) across every DB type + RocksIterator Aug 8, 2026

dfa1 commented Aug 8, 2026

View reviewed changes

Comment thread core/src/main/java/io/github/dfa1/rocksdbffm/RocksDB.java Outdated

dfa1 commented Aug 8, 2026

View reviewed changes

Comment thread core/src/main/java/io/github/dfa1/rocksdbffm/RocksDB.java Outdated

dfa1 commented Aug 8, 2026

View reviewed changes

dfa1 mentioned this pull request Aug 8, 2026

Extract a PinnableSlice wrapper to kill rocksdb_pinnableslice_value/_destroy duplication #58

Open

dfa1 commented Aug 8, 2026

View reviewed changes

dfa1 force-pushed the feat/with-pinned branch 2 times, most recently from 574757f to 6dfbd99 Compare August 8, 2026 14:51

This was referenced Aug 8, 2026

refactor: extract PinnableSlice wrapper, kill triplicated native mapping #59

Open

feat: add getPinned returning a sealed PinnedResult #48

Closed

dfa1 force-pushed the feat/with-pinned branch from 6dfbd99 to 1801658 Compare August 8, 2026 20:13

Navigation Menu

feat: scoped zero-copy get(key, Mapper) across every DB type + RocksIterator - #57

feat: scoped zero-copy get(key, Mapper) across every DB type + RocksIterator#57
dfa1 wants to merge 1 commit into
mainfrom
feat/with-pinned

dfa1 commented Aug 8, 2026

Uh oh!

dfa1 commented Aug 8, 2026

Uh oh!

Uh oh!

Uh oh!

dfa1 Aug 8, 2026

Uh oh!

dfa1 Aug 8, 2026 •

edited

Loading

Uh oh!

dfa1 Aug 8, 2026

Uh oh!

dfa1 Aug 8, 2026

Uh oh!

dfa1 Aug 8, 2026 •

edited

Loading

Uh oh!

dfa1 left a comment •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

dfa1 commented Aug 8, 2026

Summary

Test plan

Uh oh!

dfa1 commented Aug 8, 2026

Update: expanded scope

Uh oh!

Uh oh!

Uh oh!

dfa1 Aug 8, 2026

Choose a reason for hiding this comment

Uh oh!

dfa1 Aug 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dfa1 Aug 8, 2026

Choose a reason for hiding this comment

Uh oh!

dfa1 Aug 8, 2026

Choose a reason for hiding this comment

Uh oh!

dfa1 Aug 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dfa1 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

dfa1 Aug 8, 2026 •

edited

Loading

dfa1 Aug 8, 2026 •

edited

Loading

dfa1 left a comment •

edited

Loading