fix(server-ng): close server-ng SDK-visible behavior gaps by hubcio · Pull Request #3828 · apache/iggy

hubcio · 2026-08-06T12:37:47Z

The Python SDK suite pinned twelve server-ng divergences from the
legacy server as strict xfails; this closes the eleven that were
server bugs (get_topics on a missing stream erroring is intended).

Timestamp polls now filter on the broker append time replies
report; a missing partition id denies with PartitionNotFound
instead of an empty success; an unmaterialized partition reports
its initial one-empty-segment state. Pre-consensus validation
shared by TCP and HTTP denies TooManyPartitions (create_topic,
create_partitions, delete_partitions) and InvalidTopicSize before
a rejected request burns a replicated log entry. Updates store
ServerDefault sentinels verbatim and reads echo the stored size
and expiry instead of freezing the node default into state.

Purge never reached partition data. Purge commits now wake the
reconciler; the applied generation is durable per partition
(purge.gen, always fsynced last, and an I/O error reading it back
fails boot instead of silently re-purging); a purge floor fences
every journal-apply path, surviving the poll-index rebuild of
eviction re-appends; repair serves only above the floor and defers
while a committed purge is not yet locally applied. Repair-vs-purge
ordering across a StartView stays open, documented as TODO.

codecov · 2026-08-06T12:40:31Z

Codecov Report

❌ Patch coverage is 83.77282% with 160 lines in your changes missing coverage. Please review.
✅ Project coverage is 57.27%. Comparing base (039a91c) to head (f61b074).

Files with missing lines	Patch %	Lines
core/server-ng/src/dispatch.rs	50.70%	70 Missing ⚠️
core/partitions/src/iggy_partition.rs	94.86%	16 Missing and 5 partials ⚠️
core/server-ng/src/responses.rs	72.41%	16 Missing ⚠️
core/partitions/src/offset_storage.rs	88.77%	3 Missing and 8 partials ⚠️
core/shard/src/lib.rs	75.67%	8 Missing and 1 partial ⚠️
core/server-ng/src/http/handlers.rs	0.00%	7 Missing ⚠️
core/shard/src/metrics.rs	14.28%	6 Missing ⚠️
core/partitions/src/state_transfer.rs	61.53%	4 Missing and 1 partial ⚠️
core/shard/src/router.rs	0.00%	4 Missing ⚠️
core/metadata/src/impls/metadata.rs	50.00%	3 Missing ⚠️
... and 4 more

Additional details and impacted files

@@              Coverage Diff              @@
##             master    #3828       +/-   ##
=============================================
- Coverage     75.62%   57.27%   -18.36%     
  Complexity     1046     1046               
=============================================
  Files          1337     1335        -2     
  Lines        169125   150790    -18335     
  Branches     141481   123223    -18258     
=============================================
- Hits         127907    86364    -41543     
- Misses        37435    60752    +23317     
+ Partials       3783     3674      -109

Components	Coverage Δ
Rust Core	`52.74% <83.77%> (-22.73%)`	⬇️
Java SDK	`63.67% <ø> (ø)`
C# SDK	`71.14% <ø> (-1.13%)`	⬇️
Python SDK	`88.14% <ø> (ø)`
PHP SDK	`82.97% <ø> (ø)`
Node SDK	`96.36% <ø> (+0.08%)`	⬆️
Go SDK	`43.08% <ø> (ø)`

Files with missing lines	Coverage Δ
core/metadata/src/stm/stream.rs	`74.70% <100.00%> (+3.51%)`	⬆️
core/server-ng/src/partition_reconciler.rs	`92.82% <100.00%> (+0.39%)`	⬆️
core/partitions/src/journal.rs	`83.56% <98.59%> (+3.88%)`	⬆️
core/server-ng/src/partition_helpers.rs	`67.13% <66.66%> (-0.01%)`	⬇️
core/metadata/src/impls/metadata.rs	`69.83% <50.00%> (+0.62%)`	⬆️
core/server-ng/src/bootstrap.rs	`24.89% <78.57%> (+0.47%)`	⬆️
core/server-ng/src/segment_cleaner.rs	`52.63% <90.90%> (+52.63%)`	⬆️
core/shard/src/router.rs	`46.92% <0.00%> (-0.54%)`	⬇️
core/partitions/src/state_transfer.rs	`26.21% <61.53%> (+0.70%)`	⬆️
core/shard/src/metrics.rs	`78.62% <14.28%> (-3.27%)`	⬇️
... and 6 more

... and 365 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

The Python SDK suite pinned twelve server-ng divergences from the legacy server as strict xfails; this closes the eleven that were server bugs (get_topics on a missing stream erroring is intended). Timestamp polls now filter on the broker append time replies report; a missing partition id denies with PartitionNotFound instead of an empty success; an unmaterialized partition reports its initial one-empty-segment state. Pre-consensus validation shared by TCP and HTTP denies TooManyPartitions (create_topic, create_partitions, delete_partitions) and InvalidTopicSize before a rejected request burns a replicated log entry. Updates store ServerDefault sentinels verbatim and reads echo the stored size and expiry instead of freezing the node default into state. Purge never reached partition data. Purge commits now wake the reconciler; the applied generation is durable per partition (purge.gen, always fsynced last, and an I/O error reading it back fails boot instead of silently re-purging); a purge floor fences every journal-apply path, surviving the poll-index rebuild of eviction re-appends; repair serves only above the floor and defers while a committed purge is not yet locally applied. Repair-vs-purge ordering across a StartView stays open, documented as TODO. # Conflicts: # core/partitions/src/iggy_partition.rs # core/server-ng/src/bootstrap.rs # core/server-ng/src/dispatch.rs # core/server-ng/src/partition_helpers.rs # core/server-ng/src/partition_reconciler.rs # core/server-ng/src/responses.rs # core/shard/src/lib.rs

Several purge paths still resurrected or destroyed data. A replica that had not applied a committed purge accepted repair floors from purged peers and silently dropped every post-purge batch behind its stale durable line; repaired appends bypassed the purge floor invariant; a pre-purge truncation watermark survived the purge and could delete post-purge segments; a purge.gen left behind by a failed teardown made a recreated partition swallow its next purge; and fenced journal entries stayed poll-visible and pinned memory on idle purged partitions. Defer repair replies until the committed purge applies locally, guard repaired-append accounting behind the floor, zero the watermark in both purge applies, key purge.gen to the partition's created_revision so a stale incarnation hydrates as zero, filter fenced entries out of resident polls, and evict the fenced prefix up to commit_min (evicting past the walk would wedge it on a hole). Also reject zero-count partition changes pre-consensus (legacy create_topic admits zero, so only the change paths gate), deny get_consumer_offset typed on a missing partition, pass PartitionNotFound through HTTP polls, resolve the ServerDefault size sentinel against the node default at cleaner enforcement time, and count deferred repair serves.

The remaining review findings that were deferred by design had no presence in code: the follower-local get_consumer_offset read, the replica-plane version fence behind the router's silent drop of an unknown consensus operation, the materialization signal the stats registry lacks, and the untestable repair-reply purge gate. Each now carries a TODO at the exact site so the gaps cannot be rediscovered from scratch. The purge's two directory fsyncs also swallowed their errors. Both now log: the partition-dir failure is benign (the generation record has not run, a crash re-purges at boot), but a failed offsets-dir fsync followed by the generation write can resurrect deleted offset files with no re-purge left to clear them, so silence there hid a real hazard.

github-actions Bot added the S-waiting-on-review PR is waiting on a reviewer label Aug 6, 2026