Add BBS-connectivity health check to auctioneer

[ivo1116]

Summary

Adds an optional enable_bbs_health_check runner to the auctioneer. When enabled, the auctioneer periodically probes BBS via bbs.Client.Ping() and exits after N consecutive failures. Monit then restarts it, and the normal SIGTERM path releases the auctioneer's Locket lock cleanly so a healthy-AZ standby can acquire leadership.

Disabled by default — no behavior change unless the property is set.

Motivation

The auctioneer's Locket lock is renewed over a persistent gRPC connection to Locket, whose DB path is served from a warm connection pool — DNS-free.

Auction work, in contrast, resolves bbs.service.cf.internal on every call — DNS-dependent.

Under a DNS-only partition, renewals keep succeeding while the leader can't do any actual work. The two paths share no state, so the process does not self-evict and the Locket lock's 15-second TTL is continuously bumped forward. A functionally-dead leader retains the lock indefinitely, blocking failover.

This PR is the auctioneer analog of the BBS enable_db_health_check pattern used in some deployments.

Configuration

Property	Default	Purpose
diego.auctioneer.enable_bbs_health_check	false	Master toggle — off preserves current behavior
diego.auctioneer.bbs_health_check_interval	10s	Time between probes
diego.auctioneer.bbs_health_check_timeout	5s	Per-probe timeout
diego.auctioneer.bbs_health_check_failure_threshold	3	Consecutive failures before exit

Worst-case detection latency = interval × threshold.

Implementation

• New file: src/code.cloudfoundry.org/auctioneer/cmd/auctioneer/bbs_health_check_runner.go — the ifrit runner.
• Config additions in cmd/auctioneer/config/config.go.
• Wired into the ifrit Members group in cmd/auctioneer/main.go behind cfg.EnableBBSHealthCheck.
• BOSH job property additions in jobs/auctioneer/spec and template wiring in jobs/auctioneer/templates/auctioneer.json.erb.

Uses the existing bbs.Client.Ping(logger, traceID) method — same DNS/TLS/HTTP path the auction code uses, so functional degradation manifests identically to the health check.

The probe is wrapped in a context.WithTimeout so a hung Ping cannot stall the health-check goroutine. Recovery is logged as bbs-health-check.recovered when a probe succeeds after prior failures, resetting the counter.

Tests

Two Go unit tests in bbs_health_check_runner_test.go using a fake bbs.Client and fakeclock:

• TestBBSHealthCheckRunner_ExitsOnThreshold — the runner returns a non-nil error after failureThreshold consecutive failed probes.
• TestBBSHealthCheckRunner_RecoversOnSuccess — a successful probe resets the counter and the runner continues.