Skip to content

scheduler: limit the total number of scatter-range schedulers to avoid PD CPU overload #11124

Description

@bufferflies

Enhancement Task

Background

PD currently allows an unbounded number of scatter-range-scheduler instances to be added via pd-ctl scheduler add scatter-range-scheduler <start-key> <end-key>. Each active scatter-range scheduler runs its own periodic scheduling loop and region-fit computation. #3594 already showed that FitRegion cost is significant per scatter-range scheduler.

Problem

There is no upper bound on how many scatter-range schedulers can be created. When the total count grows too large (e.g. via automation or misconfiguration repeatedly adding key-range schedulers), the cumulative scheduling/FitRegion cost can drive PD CPU usage very high, degrading other latency-sensitive PD work (region heartbeat processing, leader election, TSO) and overall service quality.

Proposal

  • Add a configurable limit on the total number of concurrently active scatter-range schedulers.
  • When scheduler add scatter-range-scheduler is invoked and the current count has already reached the limit, reject the creation with a clear error instead of silently accepting it.
  • Expose the current count and limit (via scheduler config output and/or metrics) so operators can observe usage before hitting the ceiling.

Alternatives considered

  • Leave enforcement to operators manually tracking scheduler counts — no safety net, easy to overshoot.
  • Throttle/rate-limit scatter-range scheduler execution instead of capping count — reduces CPU spikes but doesn't bound per-scheduler bookkeeping overhead, and is a larger change than a simple count check.

Metadata

Metadata

Assignees

No one assigned

    Labels

    type/enhancementThe issue or PR belongs to an enhancement.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions