Prefer topology views with full slot coverage during topology refresh… - #3854
Open
wmxl wants to merge 1 commit into
Open
Prefer topology views with full slot coverage during topology refresh…#3854wmxl wants to merge 1 commit into
wmxl wants to merge 1 commit into
Conversation
Collaborator
|
Hey @wmxl |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Make sure that:
mvn formatter:formattarget. Don’t submit any formatting related changes.Summary
Follow-up to the discussion in #2769 (reported by me). When a node joins a cluster (e.g. adding a replica), it briefly acts as a master without slots and has not yet synchronized the cluster topology. If a topology refresh queries such a node, it returns a view in which itself and some actual masters carry no slots.
KnownMajoritycounts only how many known nodes a view contains, so this incorrect view ties with healthy views and can be adopted by the random tie-break — leaving the client throwingPartitionSelectorException: Cannot determine a partition for slot ...for a full refresh period.determinePartitionsnow drops topology views without full slot coverage before consensus, as long as at least one view covers all 16384 slots.Key Decisions & Assumptions
cluster-require-full-coverage no, bootstrap, lost slots), all views are retained and behavior is unchanged, so partial-coverage clusters keep working and the filter can never produce an empty candidate set.PartitionsConsensusis intentionally untouched: it remains solely responsible for split-brain arbitration; view validation happens before consensus indeterminePartitions. Network partitions do not unassign slots, so slot coverage is orthogonal to the split-brain signal thatKNOWN_MAJORITYevaluates.RedisClusterNode#getSlotCount()(backed byBitSet.cardinality()) to compute coverage without materializing a slotListper node, following up onhasNoSlots()(Add hasNoSlots() Method to RedisClusterNode #3014).Behavioral / Conceptual Changes
KNOWN_MAJORITY(periodic/adaptive refresh) andHEALTHY_MAJORITY(initial topology load).CLUSTER DELSLOTS, forgetting a slot-holding node), clients may keep a stale full-coverage view for up to one additional refresh period until all views reflect the new state.determinePartitionsbypass the filter (unchanged extension point).Testing
New unit tests cover
determinePartitionsdirectly: an incomplete view (as reported by a just-joined node) loses against a full-coverage view on both the known-majority and initial-load paths, all views are retained when none has full coverage, and a single partial view is still returned on initial load. Each assertion runs 10 iterations to rule out shuffle luck.getSlotCount()is covered for assigned, empty, andnullslot sets.We have also been running an equivalent filter in production since early 2024 (via a
RedisClusterClientsubclass overridingdeterminePartitions, as suggested in #2769) across a 30-shard cluster with frequent replica-scaling operations, with no recurrence of the exception. The original failure was reproducible by repeatedly re-adding a replica (del-node+add-node --cluster-slave) under a 1s refresh period, as described in #2769.Notes
ClusterClientOptions(as you suggested in PartitionSelectorException in Lettuce Client Triggered by Addition of New Slave Nodes in Redis Cluster #2769); happy to adjust.Fixes #2769.