grpcconf: add opt-in round_robin LB policy and client keepalive

[samlown]

Summary

gRPC clients built via grpcconf.Service used gRPC's default pick_first balancer: each client opens one connection to a single backend pod and pins to it for the connection's lifetime. When that pod is rolled the client keeps dialing the dead endpoint, and with no keepalive a stalled request hangs until the kernel TCP timeout (~15 min). This is the root cause of the 2026-07-13 API 504 storm.

Changes

• New Policy field on Service — pick_first (default / zero value) or round_robin. round_robin is opt-in per service via config; callers targeting a headless Service set policy: round_robin to spread calls across all backend pods and drop failed sub-channels on re-resolution.
• Unconditional client keepalive (Time: 5m, Timeout: 20s, PermitWithoutStream: false) so a black-holed connection is detected in minutes rather than at the TCP timeout. Applies regardless of policy (safe against the servers' default EnforcementPolicy).
• Explicit blank import _ "google.golang.org/grpc/balancer/roundrobin" so an upstream grpc change can't silently drop the transitively-registered balancer and revert us to pick_first.

Caller-supplied dial opts are still applied last and win.

Behaviour / rollout note

Because round_robin is now opt-in and defaults to pick_first, merging this alone does not change runtime behaviour — each consuming service (api → silo/transform/access/sequence) must set policy: round_robin in its config to get the fix. Tracked as a follow-up.

Keepalive 5m floor

The silo/transform/access/sequence servers run gRPC's default EnforcementPolicy (MinTime 5m); a faster client ping would be too_many_pings-GOAWAY'd. 5m is the safe floor for a client-only change. Server-side follow-ups (invopop/silo#231, invopop/transform#83, invopop/access#238, invopop/sequence#25) relax the policy; a companion PR (#8) then shortens the interval.

Testing

go build ./..., go vet, gofmt clean. Confirmed round_robin resolves at runtime (balancer.Get("round_robin") != nil).

[Copilot]

Pull request overview

This PR updates the shared grpcconf.Service client defaults to improve resiliency when dialing internal gRPC backends (particularly across Kubernetes pod rolls and dead/black-holed connections).

Changes:

• Default gRPC client load balancing to round_robin via grpc.WithDefaultServiceConfig.
• Add client keepalive parameters (Time: 5m, Timeout: 20s, PermitWithoutStream: false) to bound stalls on dead connections.
• Clarify that caller-provided dial options are appended last (so they can override defaults).

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[samlown]

Thanks — checked this against the pinned gRPC version (google.golang.org/grpc v1.81.1), and the round_robin balancer is already registered, so no side-effect import is needed here:

• The core google.golang.org/grpc package blank-imports the balancer itself — clientconn.go:55: _ "google.golang.org/grpc/balancer/roundrobin" // To register roundrobin.
• balancer/roundrobin registers itself in init() via balancer.Register(builder{}) (Name = "round_robin").
• go list -deps ./pkg/grpcconf shows google.golang.org/grpc/balancer/roundrobin in the transitive graph, and a minimal program importing only google.golang.org/grpc reports balancer.Get("round_robin") != nil → true.

Since grpcconf imports google.golang.org/grpc, the balancer is registered transitively and WithDefaultServiceConfig applies the policy at runtime. The suggestion is correct in general for balancers that core doesn't pull in (and for some older grpc-go versions), but it's redundant here. Happy to add the explicit _ "google.golang.org/grpc/balancer/roundrobin" as belt-and-suspenders if we'd rather not depend on the transitive import staying put across upgrades.

[Copilot]

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 2 out of 3 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (1)

pkg/grpcconf/grpcconf.go:107

• When PolicyRoundRobin is enabled, the effectiveness of round_robin depends on the name resolver returning multiple backend addresses (e.g., DNS A records for a headless Service) and re-resolving on failures. This code always dials s.URL() (Host+":"+Port) without normalizing/validating the target scheme, so a misconfigured Host value could silently result in no effective load balancing.

func (s *Service) Connection(opts ...grpc.DialOption) (*grpc.ClientConn, error) {
	do := s.DialOptions()
	do = append(do, opts...)
	return grpc.NewClient(s.URL(), do...)
}