Skip to content

修复半开探测许可泄漏导致故障转移后无法切回 - #399

Open
wplct wants to merge 1 commit into
SaladDay:mainfrom
wplct:pr/fix-half-open-failback
Open

修复半开探测许可泄漏导致故障转移后无法切回#399
wplct wants to merge 1 commit into
SaladDay:mainfrom
wplct:pr/fix-half-open-failback

Conversation

@wplct

@wplct wplct commented Aug 7, 2026

Copy link
Copy Markdown

问题

自动故障转移的高优先级供应商熔断后,会在冷却期结束时进入 HalfOpen 并尝试恢复探测。旧实现通过计数器占用唯一的 HalfOpen 探测许可;如果请求在记录成功或失败之前被取消、提前返回或 panic,许可不会归还。

之后所有请求都无法再次探测已恢复的高优先级供应商,只能持续使用 fallback,看起来就是“故障转移后永远不会切回”。

修复

  • 使用 ProviderRequestPermit 生命周期守卫持有 HalfOpen 探测许可。
  • 在 guard Drop 时自动释放许可,覆盖正常返回、错误、任务取消和 panic 展开。
  • 为 HalfOpen permit 墕加 generation,避免旧请求释放新一轮探测名额。
  • 转发器只在实际尝试供应商时获取 permit,避免提前占用后续候选供应商的探测名额。

测试

  • 任务取消后可以重新获取 HalfOpen permit。
  • 旧 generation 的 guard 不会释放新 generation 的 permit。
  • 较早供应商成功时不会预占后续 HalfOpen 供应商。
  • 高优先级供应商熔断、fallback 接管后,冷却结束且主供应商恢复时,下一次请求会重新选择主供应商。

行为说明

当前恢复是请求驱动的:冷却期结束后的下一次业务请求触发 HalfOpen 探测,不新增后台健康检查任务。

Problem

After a higher-priority provider trips its circuit breaker, it enters HalfOpen after the cooldown period and should be probed again. The previous implementation reserved the single HalfOpen probe slot with a counter. If the request was cancelled, returned early, or panicked before recording success or failure, the slot was never released.

All later requests were then unable to probe the recovered provider and kept using the fallback indefinitely.

Fix

  • Introduce a lifecycle-bound ProviderRequestPermit guard for HalfOpen probes.
  • Release the permit from Drop, covering successful returns, errors, task cancellation, and panic unwinding.
  • Track HalfOpen generations so stale requests cannot release a permit from a newer recovery cycle.
  • Acquire permits only when a provider is actually attempted, avoiding premature reservation for later candidates.

Tests

  • A cancelled task releases its HalfOpen permit.
  • A stale-generation guard cannot release a current-generation permit.
  • An earlier successful provider does not pre-acquire a later HalfOpen provider's permit.
  • After the primary provider opens and fallback takes over, the recovered primary is selected again on the next request after cooldown.

Behavioral note

Recovery remains request-driven: the first business request after cooldown triggers the HalfOpen probe. This PR does not introduce a background health-check task.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant