Search API Gateway

Multi-provider search middleware for failover, credential rotation, caching, and operator-declared egress. The mechanisms below are covered by in-repo tests where cited; this README does not claim deployment maturity.

POST /v1/search  {"query": "...", "lane": "deep"}  ->  unified SearchResponse

---

1. Design position

Three decisions shape everything else.

Blame attribution, not status codes. A failed upstream call is classified as Blame.KEY, Blame.PROVIDER or Blame.CALLER before any retry decision is made. tests/test_failover.py pins these cases. Retrying a malformed query across every credential burns the pool to answer a 400, and quarantining keys during a provider outage leaves no credentials once the provider recovers.

Upstream signal	Blame	Key effect	Failover
429	KEY	cooled for Retry-After	next key, same provider
401 / 403 (auth)	KEY	quarantined, time-boxed	next key, same provider
402 / plan limit	KEY	exhausted until reset	next key, same provider
5xx / timeout / connect	PROVIDER	untouched	trip circuit, next provider
2xx, unparseable	PROVIDER	untouched	trip circuit, next provider
400 / 422 (bad query)	CALLER	untouched	stop; return 400
200, zero results	—	success	next provider; empty is a valid answer

Stale fallback is a cache mechanism. Entries live for ttl_s + stale_ttl_s but are fresh only for ttl_s. When every provider fails, a stale entry is served with stale: true rather than a 503; tests/test_cache.py covers stale-as-last-resort behavior.

Rotation state is cluster state. Key selection uses one Redis Lua script that filters, picks and reserves atomically. Per-process rotation would let N replicas each independently decide the same key is least-used and stampede it. tests/test_redis.py exercises the Redis-backed rotation path when Redis is reachable.

---

2. Topology

graph TD
    A[Research Agent] -->|POST /v1/search| B[FastAPI]
    B --> C[PriorityScheduler<br/>FAST preempts DEEP]
    C --> D[SearchOrchestrator]
    D -->|1. fresh read| E[(Redis Cache)]
    D -->|2. stampede lock| E
    D -->|3. gate| F[CircuitRegistry]
    D -->|4. lease| G[KeyManager]
    G --> H[(SQLite/Postgres<br/>credentials)]
    G --> I[(Redis<br/>rotation counters)]
    D -->|5. route| J[EgressRouter]
    J --> K[Serper]
    J --> L[Tavily]
    J --> M[Exa]
    J --> N[Brave]
    J --> O[Google CSE]
    D -->|6. publish| E

Loading

Request path, in order: admission → fresh cache read → stampede lock → per-provider (circuit → key lease → HTTP) → cache write → stale fallback.

Module map

Module	Responsibility
models.py	Unified wire schema. The only contract the agent sees.
errors.py	Blame-carrying exception hierarchy. Drives failover cases covered by tests/test_failover.py.
providers/base.py	Strategy ABC. Transport + classification once, adapters supply build/parse.
providers/*.py	Five adapters, ~60 lines each.
keys/store.py	Persistent credentials (SQLAlchemy async).
keys/runtime.py	Ephemeral rotation counters. Redis (Lua) or in-memory; tests/test_redis.py exercises the Redis path when Redis is reachable.
keys/manager.py	Lease lifecycle, penalties, health.
cache.py	Fresh/stale entries, stampede lock, request coalescing; covered by tests/test_cache.py.
circuit.py	Per-provider breaker with exponential re-open backoff; covered by tests/test_failover.py.
scheduler.py	Two-lane bulkhead + priority queue + load shedding; covered by tests/test_pool.py.
egress.py	Deliberate proxy routing; owns the httpx client pool; covered by tests/test_egress.py.
orchestrator.py	Failover engine.
api.py / deps.py	HTTP surface and composition root.

---

3. Key pool

Two notions of state, deliberately separated:

• Persistent (KeyState, in the database): ACTIVE / DISABLED. An operator sets this; it survives restarts.
• Runtime (KeyStatus, in Redis): AVAILABLE, COOLING, EXHAUSTED, QUARANTINED, SATURATED. Ephemeral and shared across replicas when Redis is configured.

A 429 must be visible to every configured replica and must expire on its own. Writing that to the database on the hot path would mix a transient throttle with the operator-controlled persistent key state.

Every penalty is time-boxed. An auth failure quarantines for auth_failure_disable_s (default 15 min), it does not permanently disable. This is covered by tests/test_failover.py::test_auth_failure_quarantines_then_rotates and tests/test_pool.py::test_quarantine_is_time_boxed_not_permanent.

Rotation strategies: round_robin (even wear), least_used (default), and weighted (respects per-key quota tiers). tests/test_pool.py covers all three.

Leases are context managers. The in-flight counter is released and the outcome is reported from the exception type when the call exits; the lease lifecycle is covered by tests/test_pool.py.

---

4. Caching

• Fingerprint: SHA-256 over the fields that change the result set — query, lane, count, locale, country, freshness, domain filters, explicit route. trace, session_id and cache are excluded; including them would fragment the keyspace and quietly destroy the hit rate.
• Volatile fields (request_id, elapsed_ms, attempts) are stripped before storage. A cached response must never replay another caller's trace.
• Stampede protection: an SET NX lock whose value is a unique owner token; losers poll for the winner's result instead of duplicating the upstream call. The cache is published before the lock is released.
  • Release is a Lua compare-and-delete, so a holder whose TTL already lapsed cannot delete the lock a later caller now owns. A plain DEL did exactly that.
  • The holder renews its own lock while it works, with a conditional if GET == token then PEXPIRE, so a slow-but-alive provider call does not hand the lock to a second caller mid-flight. Renewal stops on release, exception and cancellation; losing ownership emits lock_lost.
  • Residual window: a process frozen for longer than the lock TTL cannot renew, so its lock can still expire and a second caller can enter. Closing that would need a fencing token checked by the upstream itself.
  • Proven across real OS processes, not coroutines, in tests/test_multiprocess_lock.py: several gateway processes issue the same cold query simultaneously and the fake upstream is called exactly once. The same test fails with four upstream calls when the shared lock is disabled, which is what makes it evidence rather than decoration.
• Cache failures are demoted to a miss, so Redis cache errors do not directly decide the search response.

---

5. Scheduling

Two lanes with independent concurrency budgets (bulkheads), one shared priority queue:

Lane	Providers	Concurrency	Queue timeout	Deadline
fast	Serper, Brave, Google CSE	64	2 s	10 s
deep	Tavily, Exa	16	10 s	45 s

A late-arriving FAST request jumps ahead of queued DEEP work, while separate budgets stop a DEEP flood from starving FAST of execution slots. Beyond max_queued the gateway sheds with 503 immediately — converting a fast failure into a slow one helps nobody.

Every request carries a monotonic deadline. Failover stops when the budget is gone rather than serially exhausting providers past the point the caller cares.

---

6. Network egress — policies and scope

EgressRouter routes every outbound call over endpoints the operator declares and controls, with these policies:

Policy	Selection	Use
DIRECT	no proxy	default
STATIC	stable egress per provider	provider IP allowlisting
GEO	egress in the requested country	locale-correct results
STICKY	stable egress per caller session	result continuity
ROTATING	round-robin across declared endpoints	spreading requests across configured egresses; see tests/test_egress.py

Endpoints are filtered by a liveness health-check (health_check_url): a background monitor probes reachability and drops dead egresses from rotation, falling back to the full pool only if every endpoint looks down (so a flapping probe can't black-hole all routes). This is a liveness probe, not an anonymity check.

What this layer deliberately does not do

It does not integrate dynamically-sourced residential proxy pools, source a "fresh" IP per request, verify anonymity/IP-leak before dispatch, or jitter timing to defeat correlation. Two concrete reasons, not preference:

1. It provides no anonymity for authenticated traffic. Every call carries an API key; the provider correlates by key, not source IP. Rotating egress gives zero anonymity from the authenticated party, and one key arriving from many IPs is itself the fingerprint of credential abuse — more conspicuous in ingress logs, not less.
2. Residential pools route through third parties' devices, typically without consent. Making unwitting civilians the apparent origin of authenticated traffic is out of scope for this system.

If the real goal is origin-anonymity from an on-path network observer (not the provider), the honest transport is operator-owned egress or Tor, supplied as the proxy URL of a declared endpoint — not a commercial residential network. TLS already conceals request payloads.

Provider rate limits are handled where they belong: honour Retry-After, cool the credential, fail over to the next key or provider, serve cache. To increase the configured credential pool, add keys hot via POST /v1/keys.

---

7. API

Method	Path	Purpose
POST	/v1/search	Full search request
GET	/v1/search?q=	Convenience form
GET	/v1/status	Circuits, scheduler, cache, egress snapshot
GET	/v1/keys	Per-key health and runtime status
POST	/v1/keys	Add a credential (hot)
PATCH	/v1/keys/{id}	Enable / disable
GET	/healthz	Liveness
GET	/readyz	Readiness (503 when no usable key)
GET	/metrics	Prometheus exposition

// POST /v1/search
{"query": "post-quantum tls adoption", "lane": "deep", "max_results": 5, "trace": true}

{
  "request_id": "8f2a1c...", "query": "post-quantum tls adoption", "lane": "deep",
  "provider": "tavily", "cached": false, "stale": false, "elapsed_ms": 812.4,
  "answer": "Adoption of hybrid X25519+Kyber ...",
  "results": [{"url": "https://...", "title": "...", "snippet": "...",
               "rank": 1, "provider": "tavily", "site": "example.org",
               "score": 0.94, "published_at": "2025-11-02T00:00:00Z"}],
  "attempts": [{"provider": "tavily", "key_id": "9c2...", "outcome": "rate_limited", "latency_ms": 210.5},
               {"provider": "tavily", "key_id": "1af...", "outcome": "success",      "latency_ms": 601.9}]
}

attempts is the failover audit trail, returned only when trace: true.

---

8. Running

python -m venv .venv && .venv/Scripts/python.exe -m pip install -e ".[dev]"
cp .env.example .env          # add real credentials
.venv/Scripts/python.exe -m gateway.main
# or: uvicorn gateway.main:app --port 8080

Keys are seeded from comma-separated env vars (SERPER_API_KEYS, TAVILY_API_KEYS, EXA_API_KEYS, BRAVE_API_KEYS, GOOGLE_CSE_API_KEYS) into the database on boot, idempotently, then managed through /v1/keys.

Redis and Postgres are both optional for local work: without them the gateway runs on an in-process cache and SQLite, logs the degradation, and keeps serving.

Run the published image

docker run --rm -p 8080:8080 \
  -v search-gateway-data:/data \
  -e GATEWAY_DATABASE_URL=sqlite+aiosqlite:////data/gateway.db \
  -e GATEWAY_REDIS_URL= \
  -e GATEWAY_REDIS_REQUIRED=false \
  -e SERPER_API_KEYS="${SERPER_API_KEYS:-}" \
  -e TAVILY_API_KEYS="${TAVILY_API_KEYS:-}" \
  -e EXA_API_KEYS="${EXA_API_KEYS:-}" \
  -e BRAVE_API_KEYS="${BRAVE_API_KEYS:-}" \
  -e GOOGLE_CSE_API_KEYS="${GOOGLE_CSE_API_KEYS:-}" \
  -e GOOGLE_CSE_ENGINE_ID="${GOOGLE_CSE_ENGINE_ID:-}" \
  ghcr.io/tempoloss/search-gateway:edge

Leaving provider key variables empty still lets the process boot; add real keys to make /v1/search and /readyz healthy.

.venv/Scripts/python.exe -m pytest      # test suite

---

9. Adding a provider

1. Subclass SearchProvider, declare a ProviderSpec, implement build() and parse(). Override classify() only where the provider deviates from HTTP semantics (Tavily's 432/433, Google's quota-flavoured 403).
2. Register in providers/__init__.py:PROVIDER_CLASSES.
3. Add its name to Routing.fast / Routing.deep.

No other module changes. That is the point of the strategy split.

---

10. Scaling notes

• Replica coordination. Rotation counters, cache, and locks live in Redis when Redis is configured, so multiple stateless app instances can share that coordination state. tests/test_redis.py exercises this path when Redis is reachable.
• SQLite → Postgres is a database_url change; the store is SQLAlchemy async throughout.
• Connection reuse. One AsyncClient per distinct egress is kept warm to avoid rebuilding the client and its connections for each search call.
• Watch these metrics: gateway_cache_events_total{event="hit"} (cost), gateway_key_penalised_total (pool health), gateway_scheduler_shed_total (capacity), gateway_provider_latency_ms (route ordering).
• Tune the route order by observed cost and latency: fast is ordered cheapest-first, deep is ordered by answer quality.

Limitations / not yet proven

• No published benchmark, load-test result, throughput number, latency SLO, or uptime target is included in this repository.
• The tests cover failover, stale cache fallback, scheduler behavior, egress selection, and Redis coordination, but they do not prove a multi-node availability deployment.

---

11. License

MIT. See LICENSE.