fix(java): Stabilize async consumer group test by ryerraguntla · Pull Request #3697 · apache/iggy

ryerraguntla · 2026-07-18T17:22:35Z

Fixes flakiness in the Java SDK async consumer-group integration test for concurrent join/leave operations. The change adds bounded retry handling for transient not-found errors and waits for the consumer-group membership count to converge before asserting, making the test more stable without changing its intent.

Handle transient IggyResourceNotFoundException during concurrent join/leave operations in AsyncConsumerGroupsTest. This adds bounded retry/backoff helpers and waits for the expected member count before asserting, reducing flakiness from eventually consistent consumer-group state.

Which issue does this PR address?

Closes # 3641
Relates to # pr=3515

Rationale

To address the flaky java tests.

What changed?

This PR stabilizes a flaky Java SDK integration test covering concurrent consumer-group join and leave operations.

The test previously asserted consumer-group state immediately after concurrent operations completed. In some runs, transient IggyResourceNotFoundException errors or short propagation delays caused intermittent failures even though the underlying behavior was correct.

Changes in this PR
add retry handling around concurrent joinConsumerGroup calls
add retry handling around concurrent leaveConsumerGroup calls
retry only for transient IggyResourceNotFoundException cases
wait until the consumer group reaches the expected member count before asserting
extract shared helper methods for retry and polling behavior
add reusable timeout and backoff constants for transient-state handling
Why
Concurrent async operations can briefly observe unstable or not-yet-visible consumer-group state. This PR makes the test robust to those short-lived conditions while preserving the original behavior being verified.

Impact
improves reliability of the Java SDK integration test suite
reduces intermittent failures in consumer-group concurrency coverage
limits scope to test code only
Reviewer-focused summary

Primary goal: eliminate flaky failures in AsyncConsumerGroupsTest
Approach: add bounded retry logic for transient not-found errors and poll for expected membership state
Scope: test-only change in the Java SDK
Risk: low, since production code is unchanged
Behavioral effect: assertions are now based on eventual stable state rather than immediate visibility after concurrent operations

Local Execution

Passed
Pre-commit hooks ran

AI Usage

If AI tools were used, please answer:

Which tools? Claude
Scope of usage? generated functions and implementation)
How did you verify the generated code works correctly? Manually verified
Can you explain every line of the code if asked? yes

Handle transient `IggyResourceNotFoundException` during concurrent join/leave operations in `AsyncConsumerGroupsTest`. This adds bounded retry/backoff helpers and waits for the expected member count before asserting, reducing flakiness from eventually consistent consumer-group state.

github-actions · 2026-07-18T17:22:48Z

Thanks for the PR. It is labeled S-waiting-on-review and queued for review.

Slash commands (own line, regular comment) move it around the queue:

/ready - back to S-waiting-on-review after addressing feedback
/author - flip to S-waiting-on-author while you finish changes
/request-review @user-or-team - request a reviewer

See CONTRIBUTING.md for details.

ryerraguntla · 2026-07-18T18:38:49Z

/ready

…oups

codecov · 2026-07-22T02:21:23Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 75.85%. Comparing base (a684e55) to head (68d9116).

Additional details and impacted files

@@             Coverage Diff              @@
##             master    #3697      +/-   ##
============================================
- Coverage     75.87%   75.85%   -0.02%     
  Complexity      969      969              
============================================
  Files          1324     1323       -1     
  Lines        160623   160101     -522     
  Branches     133499   133497       -2     
============================================
- Hits         121867   121449     -418     
+ Misses        35112    35009     -103     
+ Partials       3644     3643       -1

Components	Coverage Δ
Rust Core	`75.77% <ø> (ø)`
Java SDK	`62.71% <ø> (ø)`
C# SDK	`72.26% <ø> (ø)`
Python SDK	`93.04% <ø> (+3.59%)`	⬆️
PHP SDK	`84.52% <ø> (ø)`
Node SDK	`96.14% <ø> (+0.02%)`	⬆️
Go SDK	`43.08% <ø> (ø)`
see 6 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

slbotbm

other than one comment, lgtm, but let's wait for maciek to approve

slbotbm · 2026-07-22T18:41:08Z

-            CompletableFuture.allOf(
-                            client.consumerGroups().leaveConsumerGroup(STREAM_ID, TOPIC_ID, groupId),
-                            secondClient.consumerGroups().leaveConsumerGroup(STREAM_ID, TOPIC_ID, groupId),
-                            thirdClient.consumerGroups().leaveConsumerGroup(STREAM_ID, TOPIC_ID, groupId))
-                    .get(TIMEOUT_SECONDS * 2, TimeUnit.SECONDS);
-
-            // then
-            ConsumerGroupDetails afterLeave = client.consumerGroups()
-                    .getConsumerGroup(STREAM_ID, TOPIC_ID, groupId)
-                    .get(TIMEOUT_SECONDS, TimeUnit.SECONDS)
-                    .get();
-            assertThat(afterLeave.membersCount()).isEqualTo(0);
+            retryOnTransientNotFound(
+                    () -> CompletableFuture.allOf(
+                                    client.consumerGroups().leaveConsumerGroup(STREAM_ID, TOPIC_ID, groupId),
+                                    secondClient.consumerGroups().leaveConsumerGroup(STREAM_ID, TOPIC_ID, groupId),
+                                    thirdClient.consumerGroups().leaveConsumerGroup(STREAM_ID, TOPIC_ID, groupId))
+                            .get(TIMEOUT_SECONDS * 2, TimeUnit.SECONDS),
+                    TRANSIENT_RETRY_TIMEOUT,
+                    TRANSIENT_RETRY_BACKOFF);
+            awaitMembersCount(groupId, 0);


Here, if one leave succeeds while another returns IggyResourceNotFoundException, retry resubmits all three leaves. Already-successful client now returns “member not found,” so every later attempt fails. Helper eventually times out even if member count reaches zero. This defeats exact flaky case being fixed.

fix: Treat “already left” as success, retry clients individually, or tolerate leave exceptions and use awaitMembersCount(groupId, 0) as final condition.

mmodzelewski

+1 to @slbotbm's comment: since retryOnTransientNotFound re-runs the whole 3-client leave batch, any partial success turns every retry into ConsumerGroupMemberNotFound (5006, mapped to IggyResourceNotFoundException) for the already-left members, so the helper loops until the deadline and throws AssertionError before awaitMembersCount(0) is ever reached. The original CI failure in #3641 was exactly in the leave batch, so that flake isn't fixed, it just fails 5 seconds later with a different message.

There is also a second, more fundamental problem: the flake looks like a server-side bug, not eventual consistency. In the failing run the joins were acked, the membersCount == 3 assertion passed, and only then a leave got "not joined". A durably visible member disappeared between a successful read and the leave, which points at lost state on the server. Candidate mechanism: client_id is a hash of ip:port (client_manager.rs), so ephemeral port reuse across the test suite lets a late disconnect cleanup_connection from a previous connection with the same address delete the new client and force-leave its consumer groups (tcp_listener.rs / shard/system/clients.rs). Retrying in the test only hides that signal from CI.

Suggested path: capture server container logs on failure first. A "Deleted ... client with ID ..." log line between join and leave would confirm the force-removal, and if it does, file a server-side issue for the lost-membership race.

github-actions · 2026-07-31T00:44:34Z

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs.

If you need a review, please ensure CI is green and the PR is rebased on the latest master. Don't hesitate to ping the maintainers - either @core on Discord or by mentioning them directly here on the PR.

Thank you for your contribution!

…oups

Leave is non-idempotent (CONSUMER_GROUP_NOT_JOINED / 5006). Re-running the whole three-client leave allOf after a partial success caused already-left clients to fail forever until the retry deadline. Treat already-left as success per client, keep join-batch retry only for transient stream/topic/group resolution codes, clamp attempt timeouts to the remaining budget, and poll membersCount plus members().size(). Co-authored-by: ryerraguntla <ryerraguntla@users.noreply.github.com>

ryerraguntla · 2026-08-03T22:35:55Z

/ready

…oups

github-actions Bot added the S-waiting-on-review PR is waiting on a reviewer label Jul 18, 2026

ryerraguntla changed the title ~~fix(java-sdk): Stabilize async consumer group test~~ fix(java): Stabilize async consumer group test Jul 18, 2026

Reformat long method signatures in consumer group tests

e80aa58

Merge branch 'master' into fix(java-sdk)/3641-flaky-async-consumer-gr…

9bda9da

…oups

hubcio previously approved these changes Jul 22, 2026

View reviewed changes

slbotbm requested changes Jul 22, 2026

View reviewed changes

github-actions Bot added S-waiting-on-author PR is waiting on author response and removed S-waiting-on-review PR is waiting on a reviewer labels Jul 22, 2026

mmodzelewski requested changes Jul 23, 2026

View reviewed changes

github-actions Bot added the S-stale Inactive issue or pull request label Jul 31, 2026

Merge branch 'master' into fix(java-sdk)/3641-flaky-async-consumer-gr…

1ec2190

…oups

github-actions Bot removed the S-stale Inactive issue or pull request label Aug 3, 2026

cursor Bot dismissed hubcio’s stale review via 958b3ad August 3, 2026 21:42

Fixing few nits

6751526

Merge branch 'master' into fix(java-sdk)/3641-flaky-async-consumer-gr…

68d9116

…oups

github-actions Bot added S-waiting-on-review PR is waiting on a reviewer and removed S-waiting-on-author PR is waiting on author response labels Aug 3, 2026

Navigation Menu

Uh oh!

fix(java): Stabilize async consumer group test - #3697

fix(java): Stabilize async consumer group test#3697
ryerraguntla wants to merge 7 commits into
apache:masterfrom
ryerraguntla:fix(java-sdk)/3641-flaky-async-consumer-groups

ryerraguntla commented Jul 18, 2026

Uh oh!

github-actions Bot commented Jul 18, 2026

Uh oh!

ryerraguntla commented Jul 18, 2026

Uh oh!

codecov Bot commented Jul 22, 2026 •

edited

Loading

Uh oh!

slbotbm left a comment

Uh oh!

slbotbm Jul 22, 2026

Uh oh!

mmodzelewski left a comment

Uh oh!

github-actions Bot commented Jul 31, 2026

Uh oh!

ryerraguntla commented Aug 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

Conversation

ryerraguntla commented Jul 18, 2026

Which issue does this PR address?

Rationale

What changed?

Local Execution

AI Usage

Uh oh!

github-actions Bot commented Jul 18, 2026

Uh oh!

ryerraguntla commented Jul 18, 2026

Uh oh!

codecov Bot commented Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

slbotbm left a comment

Choose a reason for hiding this comment

Uh oh!

slbotbm Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

mmodzelewski left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Jul 31, 2026

Uh oh!

ryerraguntla commented Aug 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

codecov Bot commented Jul 22, 2026 •

edited

Loading