When running redisques in a Vert.x clustered environment (e.g., with Hazelcast), the EventBus consumer registered by QueueStatisticsCollector is never unregistered during shutdown. This causes stale entries to accumulate in the Hazelcast __vertx.subs multimap, leading to persistent "Not a member of the cluster" errors when other nodes try to send messages to the dead member.
Root Cause
In QueueStatisticsCollector.java (lines 103-112) (https://github.com/swisspost/vertx-redisques/blob/cb72c1e/src/main/java/org/swisspush/redisques/util/QueueStatisticsCollector.java#L103-L112), an EventBus consumer is registered in the constructor:
vertx.eventBus().consumer(keyspaceHelper.getQueueStatisticQueueSizeSyncKey(),
(Handler<Message<QueueSizeInfoMap>>) event ->
event.body().getValue().forEach((key, value) -> {
if (!keyspaceHelper.getVerticleUid().equals(key)) {
approximateQueueSize.put(key, value);
}
}));
Problems:
- The MessageConsumer reference is not stored
- QueueStatisticsCollector has no
stop() or close() method
- The consumer is not included in
QueueRegistryService.stop() or gracefulStop() cleanup
Impact
In a Kubernetes environment with pod scaling/restarts:
- Pod A registers EventBus consumer for address
redisques-addr-queueStateQueueItemCounter
- Pod A terminates (scaled down, crashed, or restarted)
- Subscription remains in Hazelcast
__vertx.subs multimap with Pod A's member UUID
- Other pods continuously try to send messages to the dead UUID
- Logs fill with warnings:
WARN ConnectionHolder - Connecting to server <dead-uuid> failed io.vertx.core.impl.NoStackTraceThrowable: Not a member of the cluster
- The stale entry persists indefinitely until ALL cluster nodes are restarted simultaneously
Proposed Fix
Store the MessageConsumer reference and unregister it during shutdown:
// Add field
private MessageConsumer<QueueSizeInfoMap> queueSizeConsumer;
// In constructor, store reference
this.queueSizeConsumer = vertx.eventBus().consumer(
keyspaceHelper.getQueueStatisticQueueSizeSyncKey(),
(Handler<Message<QueueSizeInfoMap>>) event -> { ... }
);
// Add stop method
public void stop() {
if (queueSizeConsumer != null) {
queueSizeConsumer.unregister();
}
}
Then integrate into QueueRegistryService.stop():
public void stop() {
queueStatisticsCollector.stop(); // Add this line
unregisterConsumers(UnregisterConsumerType.FORCE);
}
When running redisques in a Vert.x clustered environment (e.g., with Hazelcast), the EventBus consumer registered by QueueStatisticsCollector is never unregistered during shutdown. This causes stale entries to accumulate in the Hazelcast __vertx.subs multimap, leading to persistent "Not a member of the cluster" errors when other nodes try to send messages to the dead member.
Root Cause
In QueueStatisticsCollector.java (lines 103-112) (https://github.com/swisspost/vertx-redisques/blob/cb72c1e/src/main/java/org/swisspush/redisques/util/QueueStatisticsCollector.java#L103-L112), an EventBus consumer is registered in the constructor:
Problems:
stop()orclose()methodQueueRegistryService.stop()orgracefulStop()cleanupImpact
In a Kubernetes environment with pod scaling/restarts:
redisques-addr-queueStateQueueItemCounter__vertx.subsmultimap with Pod A's member UUIDWARN ConnectionHolder - Connecting to server <dead-uuid> failed io.vertx.core.impl.NoStackTraceThrowable: Not a member of the clusterProposed Fix
Store the MessageConsumer reference and unregister it during shutdown:
Then integrate into QueueRegistryService.stop():