Python API client architecture and migration plan

[AbhiPrasad]

Status: Implementation in progress

Implementation progress

• PR 1: Extract the existing transport without changing behavior — #641, merged in 625a5e77bcd0192bae4aa8e30c86e753d34187d2 (ref(api): establish transport boundary for API client).
• PR 2: Add structured errors and the policy-aware request engine — merged in 0cbf393e8992c756e3d10e16be4be9eb659c9abe (feat(api): add policy-aware HTTP transport, #653).
• PRs 3–12 and public API stabilization — not started.

Context

The Python SDK does not currently have an API module comparable to the Go SDK's api/ package. HTTP transport, authentication, endpoint calls, response parsing, retries, and product-level fallback behavior are spread across logger.py, functions/invoke.py, framework2.py, sandbox.py, the devserver, and CLI modules.

This became especially visible in issue #639: Experiment.summarize() catches any failure from experiment-comparison2 and returns empty score and metric maps. That makes a failed read look like a successful experiment with no scores.

An audit of the Python SDK's Braintrust-directed HTTP calls found a broader transport problem:

• Before login, connections have no default retries.
• After login, RetryRequestExceptionsAdapter retries request exceptions for every HTTP method, including unsafe POST requests — unless the user installed a custom adapter via set_http_adapter(), in which case make_long_lived() is a no-op and SDK-managed retries are replaced entirely.
• It does not retry completed 429 or 5xx responses.
• Retry behavior cannot be inferred from the HTTP method because several logical reads, including BTQL and legacy lookup endpoints, use POST.
• Some call sites have their own retry or fallback behavior, while others raise immediately.
• Several backend write endpoints are not idempotent, so adding global status retries would create duplicate work or misleading failures.

The SDK should centralize endpoint definitions and make retry and fallback behavior explicit per operation.

Goals

1. Give each Braintrust endpoint one authoritative SDK implementation for request shape, response parsing, errors, and retry safety.
2. Retry transient failures for safe reads, including logical reads implemented with POST.
3. Avoid automatically replaying operations that can create duplicate experiments, invoke functions twice, incur provider costs, or produce duplicate versions.
4. Preserve HTTP status, request ID, attempt count, and exception causes in structured errors.
5. Separate transport failures from product-level decisions such as cache fallback or lenient eval reporting.
6. Keep existing high-level APIs such as braintrust.init(), Experiment.summarize(), and load_prompt() compatible while migrating them internally: signatures, return types, and serialized shapes stay backward compatible (using deprecated bridge fields where a model gains a new authoritative representation). Intentional behavior changes, such as strict-by-default summary failures, ship only in dedicated PRs with release notes.
7. Preserve custom transport injection for tests, VCR, and users of set_http_adapter().
8. Provide a resource-oriented public client under braintrust.api.
9. Prefer realistic integration and VCR-backed coverage over mocked sessions, responses, transports, or service clients.
10. Make the migration incremental and independently testable rather than a large rewrite.

Non-goals

• Replacing the SDK's high-level experiment, dataset, prompt, or logging APIs.
• Moving provider integration HTTP traffic into this client.
• Immediately generating a complete client from the OpenAPI specification.
• Applying one retry policy to every POST request.
• Rewriting background log ingestion in the first phase.
• Providing complete coverage of every OpenAPI endpoint in the first release of braintrust.api.

Current architecture

BraintrustState owns three HTTPConnection instances:

• app/control-plane connection;
• API/data-plane connection;
• proxy connection.

HTTPConnection and RetryRequestExceptionsAdapter are defined in logger.py. Endpoint calls are distributed across the package.

The Go SDK provides a useful organizational model:

api/client.go
api/projects/
api/experiments/
api/datasets/
api/functions/
internal/https/client.go

Its main strengths are the resource-oriented service layer and typed HTTP errors. The Python implementation should adopt those ideas, but add operation-level retry policies rather than copying the Go transport directly.

HTTP call-site audit summary

Safe logical reads

These operations can retry transient network failures, 408, 429, and transient 5xx responses:

• API-key login and organization discovery;
• ping and API version discovery;
• project, experiment, dataset, prompt, and function lookups;
• prompt and parameter loading;
• BTQL queries;
• environment resolution;
• base-experiment lookup;
• experiment comparison and dataset summary;
• attachment metadata and downloads.

Some of these are POST requests even though they are reads, so an HTTP-method-only policy is insufficient.

Some reads also translate specific statuses into domain results today, and the migration must preserve those translations explicitly. In particular, base-experiment lookup (POST /api/base_experiment/get_id on the app connection) treats HTTP 400 as "no base experiment" and returns None. A fresh project's first eval always takes this path, so migrating the endpoint to raise typed errors for all 4xx responses would break it; ExperimentsAPI must keep the 400-to-None rule as a documented status-to-domain translation.

Idempotent or conditionally idempotent writes

• Plain project registration is a get-or-create operation by organization and name. This classification does not automatically extend to registration variants with secondary resources such as agents.
• Named dataset registration with update=False resolves name conflicts to the existing dataset. update=True reapplies the update but may duplicate audit side effects.
• Named experiment registration with update=True returns the existing experiment and is conditionally replay-safe. Registration without an explicit name, or with update=False, is not.
• Attachment status updates overwrite the same status object.
• A code-bundle PUT to an already-issued signed URL allows overwrite.

These operations may opt into retries, but the policy should be explicit and should account for secondary effects such as audit records.

Attachment operations that require reconciliation

The current backend contract does not make the whole attachment workflow transparently replayable:

• POST /attachment first creates <attachment-key>.status.json with allowOverwrite: false. Repeating the request for the same client-generated key returns HTTP 409 instead of another signed upload URL. A lost successful initialization response therefore cannot be recovered by blindly replaying the POST, so this operation cannot use generic status retries under the current contract.
• The initialization response includes If-None-Match: * for the signed upload PUT, and the object-store URL is issued with overwrite disabled. The SDK also adds x-ms-blob-type: BlockBlob for Azure. Replaying an upload after an ambiguous success is non-destructive, but it can return an object-store precondition failure rather than the original success.
• GET /attachment checks both the status object and uploaded object. A specialized attachment workflow can use contentLength/download metadata to establish that the object exists after an ambiguous upload. If the original status object still says uploading, that status currently takes precedence, so the client may need to write done; the backend does not automatically report done merely because the object exists. This is workflow-level reconciliation, not a reason to classify either initialization or upload as an ordinary IDEMPOTENT_WRITE.

Under the current contract, attachment initialization uses RetryMode.NONE. Upload and status handling remain specialized and explicitly interpret precondition failures and metadata/status reads.

Unsafe operations

The following should not be replayed automatically without an idempotency mechanism:

• experiment registration without both an explicit name and update=True, because the backend creates a new generated or uniquely suffixed experiment;
• function invocation, because it can incur cost and execute arbitrary side effects;
• sandbox listing, because it creates a sandbox and runs code;
• requesting a code-bundle upload, because the backend allocates a new bundle ID and random object path;
• inserting or replacing functions, prompts, and parameters, because a replay can fail with an existing slug or create another version;
• organization updates, unless duplicate audit effects are acceptable.

Specialized at-least-once operations

Background /logs3 ingestion already has batching and its own retry loop. Each backend execution gets a new transaction ID, so retries are at-least-once rather than truly idempotent. Overflow upload setup can also leave unused objects after retries.

Logging should retain a dedicated policy and should not accidentally gain a second generic retry loop.

Existing fallback behavior

Transport retries do not resolve ambiguous fallback semantics. Current examples include:

• experiment summary returning empty scores and metrics after any final comparison failure;
• prompt and parameter loading returning cached data after an unpinned request fails;
• pinned prompt requests converting infrastructure failures into "not found" errors;
• API version discovery caching a default payload limit for the process after one failure, which also memoizes can_use_overflow: False and disables overflow uploads for the rest of the process without ever re-probing;
• the explicit comparison-name lookup in Experiment.summarize() swallowing any exception and continuing without a comparison name;
• named base-experiment registration warning and continuing without the requested baseline;
• asynchronous log ingestion dropping a batch after retries.

The API layer should report errors. Higher layers should own and clearly mark any fallback.

Backend contract assumptions

The architecture and endpoint policies must be checked against the backend contract, not inferred from HTTP verbs or endpoint names. The current backend establishes these details:

• GET or POST /api/apikey/login returns org_info entries with id, name, api_url, proxy_url, realtime_url, is_universal_api, and Git metadata settings. The control plane currently fills a missing api_url from the deployment default and a missing proxy_url from the resolved API URL, but the Python router should still tolerate null or omitted values for older and self-hosted deployments.
• The data plane serves native data-plane routes and proxies control-plane GET/POST routes matching /api/:object_type and /api/:object_type/:action to its configured app origin. Bootstrap still has to contact the configured app origin because the organization-specific API URL is not known before login.
• Data-plane authentication expects Authorization: Bearer <token>. Proxy operations such as function invocation also require operation-specific headers such as x-bt-org-name; these belong in the service endpoint definition rather than in a global transport default.
• Data-plane responses expose x-bt-internal-trace-id when tracing is active. Control-plane infrastructure may expose a deployment request ID such as x-vercel-id. Structured errors should capture the recognized response identifier and retain the relevant response headers rather than assuming a single universal request-id header.
• Error bodies are not uniform during this migration: data-plane errors commonly use Code/Message, while control-plane errors may use error. Services must use status codes for control flow and treat the parsed body as diagnostic context, not depend on one error-body schema.
• Backend response objects evolve additively. Request models can reject invalid input, but response parsing must tolerate unknown fields (and preserve them where the public model promises round-tripping) so a newly added backend field does not break an older SDK.
• The /v1/:object_type facade maps writes onto legacy registration semantics. Project POST is get-or-create. Dataset POST sends update=False, while dataset PUT sends update=True. Experiment POST/PUT send update=True unless ensure_new=True; a named experiment is therefore get-existing by default, but an unnamed experiment or ensure_new=True still creates a new experiment. Retry policy must be selected from those effective semantics and payload fields, not from POST versus PUT.
• Legacy base-experiment lookup returns HTTP 400 when no usable baseline exists. That status is a domain-level absence only for this operation; it is not a general not-found convention.
• Attachment initialization and upload have the create-only/precondition behavior described above. The client must not describe them as overwrite operations.

These assumptions were checked against app/pages/api/apikey/login.ts, app/pages/api/{project,dataset,experiment}/register.ts, app/pages/api/base_experiment/get_id.ts, api-ts/src/api.ts, api-ts/src/attachment.ts, api-ts/src/proxy/register-code.ts, and app-schema/schema.sql in the Braintrust backend repository. They should have contract tests against the local backend or VCR recordings. If the backend contract changes, update this section and the corresponding endpoint policy together.

Proposed package structure

Use the public braintrust.api namespace from the beginning. Public client, service, model, error, and policy types live directly in this package; low-level implementation details use underscore-prefixed modules:

src/braintrust/api/
  __init__.py
  client.py
  errors.py
  policies.py
  _routing.py
  _transport.py
  auth.py
  projects.py
  experiments.py
  datasets.py
  prompts.py
  functions.py
  queries.py
  attachments.py
  models/
    projects.py
    experiments.py
    datasets.py
    functions.py

braintrust.api.__init__ should explicitly export the supported public surface. _transport.py remains private in the same way that the Go SDK keeps its HTTP implementation under internal/https.

Transport layer

The policy-driven client's end state should use one configured transport and one underlying requests.Session for Braintrust app, API, and proxy targets. The initial public resource client is synchronous; it does not expose thread-wrapped pseudo-async methods. Existing background workers remain responsible for queued log ingestion and continue running blocking HTTP away from the caller. An endpoint router supplies the destination URL for each request; separate Braintrust destinations do not require separate transport objects. During incremental migration, legacy HTTPConnection sessions may need to coexist until their call sites and mutable adapter behavior have moved; forcing every compatibility shim onto the shared session in PR 3 would not be behavior-preserving.

The transport is responsible only for HTTP mechanics:

• resolving a target through the endpoint router and joining its URL;
• authorization and common headers;
• timeout handling;
• request and response debug logging;
• bounded retries according to an explicit policy;
• status checking;
• JSON decoding;
• structured errors;
• custom requests.Session or adapter injection.

Per-attempt timeout is configured only through RetryPolicy, so timeout and elapsed-budget validation have one source of truth. A transport closes sessions it creates and supports context-manager cleanup; an injected session remains owned by its caller.

It must not implement domain fallbacks such as returning cached prompts or empty experiment summaries.

Session sharing constraints

Consolidating three sessions into one introduces constraints the current architecture satisfies by construction:

• Thread safety. The session is shared between user threads and the background logger's thread pool (HTTP_REQUEST_THREAD_POOL, sized to cpu_count()). requests.Session is not safe for concurrent mutation: session-level headers and the cookie jar must be treated as immutable after bootstrap. Authorization and token refresh must be applied per request (or per target) rather than by mutating session.headers the way set_token() does today, and lazy client construction must be guarded against concurrent initialization.
• Credential scoping. Authorization is currently a session-default header on each Braintrust connection, while object-storage requests deliberately use separate token-less connections. With one session, the Authorization header must be attached only to requests routed to Braintrust targets. Absolute signed object-storage URLs (S3, GCS, Azure) must never receive the Braintrust API key — sending it there is a credential exposure, and stray headers can break Azure signed-URL requests.
• Cookie isolation. Three separate sessions currently prevent cookies set by one host from being replayed to another. Although a normal cookie jar scopes cookies by domain, Braintrust API-key traffic does not require persisted cookies; the shared transport should reject or clear response cookies so concurrent cookie-jar mutation and accidental cross-target state are impossible.
• Adapter scope. A requests.Session selects adapters by URL prefix, not by logical RequestTarget. Legacy connections remain isolated until endpoint policies replace their timing-dependent mutation quirks. In the final client, a custom adapter applies consistently to app, API, proxy, and signed-object requests regardless of installation timing. Injecting a session or adapter disables the SDK retry loop by default; that transport owner assumes responsibility for replay safety. A caller may explicitly enable the SDK loop only when it knows the injected adapters do not retry.
• Streaming. RetryRequestExceptionsAdapter.send() fully downloads response.content before returning, even for stream=True requests, so streamed proxy function invocations are fully buffered in memory today. The new transport must support true streaming; un-buffering invocation responses is a user-visible behavior change and is handled explicitly in PR 9.

A conceptual interface is:

class RetryMode(enum.Enum):
    NONE = "none"
    SAFE_READ = "safe_read"
    IDEMPOTENT_WRITE = "idempotent_write"
    LOG_INGESTION = "log_ingestion"


class Transport:
    def request(
        self,
        target: RequestTarget,
        method: str,
        path: str,
        *,
        params: Mapping[str, Any] | None = None,
        json: Any = None,
        data: Any = None,
        headers: Mapping[str, str] | None = None,
        retry_mode: RetryMode = RetryMode.NONE,
        stream: bool = False,
    ) -> requests.Response:
        ...

    def request_json(self, ...) -> Any:
        ...

Retry mode must be supplied by the endpoint implementation rather than inferred only from the HTTP method. For example, BTQL is a safe logical read even though it uses POST, while function invocation is not retryable even though it also uses POST.

RequestTarget covers Braintrust destinations only. Absolute signed object-storage URLs are deliberately not expressible through Transport.request(); they go through the specialized object-storage path described under "Specialized clients", which guarantees the Braintrust Authorization header is never attached.

Retry behavior

SAFE_READ and explicitly approved IDEMPOTENT_WRITE operations should support:

• connection failures and timeouts;
• HTTP 408;
• HTTP 429 with Retry-After;
• transient 5xx responses;
• bounded exponential backoff when the server does not provide a retry delay;
• a maximum attempt count and elapsed-time budget;
• response cleanup before retry;
• deterministic test hooks for sleep and clocks.

Ordinary 4xx responses, JSON decoding failures, schema validation failures, and deterministic request-construction failures such as invalid URLs, schemas, or headers should not retry by default. Only transient request exceptions such as connection failures and timeouts are eligible. The default retryable 5xx set is exactly 500, 502, 503, and 504; statuses such as 501, 505, 507, 508, and 511 are not retried unless an endpoint declares an explicit, evidence-based override adding them (overrides may also shrink the set). A retryable 503 with Retry-After follows the same server-delay and budget rules as 429.

The SDK retry loop must also be able to reproduce the request body. JSON bodies and immutable str/bytes data may retry. File-like bodies, generators, multipart files, and other potentially consumed data are sent once unless a specialized workflow prepares and rewinds them explicitly.

The default SAFE_READ policy allows four total attempts and 60 seconds of total elapsed time, including request execution and sleeps. Elapsed time uses a monotonic clock, and each request timeout and sleep is clamped to the remaining budget.

The per-attempt timeout must be materially below the elapsed budget, or timeout retries are structurally impossible: today's default request timeout is 60 seconds (BRAINTRUST_HTTP_TIMEOUT, applied in make_long_lived()), and under a 60-second budget a first-attempt read timeout would exhaust the entire budget, so timeouts — first on the retryable list — would never actually retry. Therefore:

• the default SAFE_READ per-attempt timeout is 20 seconds, so a first-attempt timeout leaves budget for at least two more attempts;
• NONE operations keep the current 60-second default timeout; their behavior is unchanged because they never retry;
• endpoints with a legitimate longer runtime, such as BTQL or experiment comparison, declare an explicit override with both a longer per-attempt timeout and a proportionally larger budget (for example, 60 seconds per attempt within a 180-second budget) rather than weakening the global default;
• policy construction should reject or flag a configuration whose elapsed budget is less than or equal to its per-attempt timeout, and tests should assert that a first-attempt timeout still leaves at least one retry.

HTTP 429 and Retry-After

A 429 response may only retry when the operation's retry mode permits replay. NONE operations, such as function invocation, must surface the 429 without replaying the request.

For retryable operations, the client must respect Retry-After:

• support both RFC forms: delay-seconds (for example, Retry-After: 5) and an HTTP date;
• use the server-provided delay directly, without applying client backoff;
• fall back to exponential backoff when the header is absent or malformed;
• use the injected clock in tests when converting an HTTP date into a delay;
• include the parsed Retry-After value in debug logging and final retry-exhaustion context.

Retry-After does not override the operation's maximum attempts or elapsed-time budget. If the server asks the client to wait longer than the remaining budget, the client must stop and raise a typed rate-limit/retry-exhaustion error rather than retry early or sleep past the configured budget. The error should preserve the 429 status and original Retry-After header so the caller can decide whether to retry at a higher level.

NONE should not replay ambiguous request failures. In particular, it should not inherit the current behavior where all POST requests are retried after request exceptions.

LOG_INGESTION performs no generic transport retries; at the transport level it is equivalent to NONE. The variant exists so ingestion requests are explicitly labeled, letting the transport and tests assert that no generic retry loop stacks underneath the background logger's own at-least-once retry loop.

Structured errors

Replace body-only AugmentedHTTPError handling with an error that preserves useful context:

class BraintrustHTTPError(BraintrustAPIError, AugmentedHTTPError):
    method: str
    url: str
    status_code: int
    response_body: str
    request_id: str | None
    request_id_header: str | None
    response_headers: Mapping[str, str]
    attempts: int
    retryable: bool

Braintrust response bodies should be preserved verbatim in structured errors. request_id should prefer x-bt-internal-trace-id and may fall back to recognized deployment headers such as x-vercel-id; the selected header name and an explicitly allowlisted subset of response headers should also be retained. Avoid denylist-based response filtering. The original transport exception should remain available through exception chaining.

BraintrustHTTPError must subclass the existing AugmentedHTTPError (braintrust.util) for the duration of the migration. AugmentedHTTPError is raised by response_raise_for_status() across logger.py, sandbox.py, functions/invoke.py, and the CLI, is caught internally (for example, in Experiment.summarize()), and is importable by user code; if migrated services raised an unrelated type, existing except AugmentedHTTPError: handlers would silently stop catching and handled fallbacks would become uncaught exceptions. Dropping the subclassing relationship is a future major-version decision.

Separate errors should cover transport failure, retry exhaustion, and response decoding. A retry-exhaustion error caused by a final HTTP response should remain a BraintrustHTTPError subtype so status and AugmentedHTTPError compatibility are preserved; exhaustion caused only by request exceptions should remain chained from the final transport exception. Service methods may translate specific statuses into domain outcomes, such as a documented 404 returning None.

Client facade and service layer

One transport, routed destinations

The current SDK has three HTTPConnection objects because requests may be sent to three destinations:

• app/control plane;
• API/data plane;
• function proxy.

Those destinations should be routing configuration, not separate transport instances. A single Transport and underlying requests.Session can send requests to multiple hosts; requests maintains the appropriate connection pools internally.

class RequestTarget(enum.Enum):
    APP = "app"
    API = "api"
    PROXY = "proxy"


@dataclasses.dataclass
class EndpointRouter:
    app_url: str
    api_url: str | None = None
    proxy_url: str | None = None

    def resolve(self, target: RequestTarget, path: str) -> str:
        ...


@dataclasses.dataclass(frozen=True)
class ClientContext:
    org_id: str
    org_name: str


class BraintrustClient:
    def __init__(
        self,
        *,
        transport: Transport,
        org_id: str,
        org_name: str,
    ):
        context = ClientContext(org_id=org_id, org_name=org_name)
        self.projects = ProjectsAPI(transport, context)
        self.experiments = ExperimentsAPI(transport, context)
        self.datasets = DatasetsAPI(transport, context)
        self.prompts = PromptsAPI(transport, context)
        self.functions = FunctionsAPI(transport, context)
        self.queries = QueriesAPI(transport, context)
        self.attachments = AttachmentsAPI(transport, context)

EndpointRouter.resolve() must preserve the current proxy fallback: when proxy_url is unset, RequestTarget.PROXY resolves to the API URL. BraintrustState.proxy_conn() behaves this way today, and function invocation, sandbox listing, the devserver, and push all rely on it for organizations without a configured proxy URL. The router must also keep Universal Proxy URL normalization (stripping the /v1/proxy suffix), which _normalize_proxy_conn_url() already performs centrally inside proxy_conn() — the router inherits an existing centralized rule rather than fixing scattered call sites.

The constructor above is the low-level form, taking an already-bootstrapped transport. The public convenience constructor (BraintrustClient(api_key=..., org_name=...), shown in the follow-up section) performs login, organization selection, and URL discovery internally and then delegates to this form.

Each service identifies the target as part of its endpoint definition:

return self._transport.request_json(
    RequestTarget.API,
    "GET",
    "/experiment-comparison2",
    params=params,
    retry_mode=RetryMode.SAFE_READ,
)

This gives us one timeout implementation, one retry engine, one error model, one debug-logging path, and one custom session/adapter injection point.

Can the destinations also become one URL?

For organizations using the Universal API, many control-plane and proxy routes are available from the API host. After login, the router should use the Universal API URL for every endpoint that the backend contract supports there. The login response already includes is_universal_api, api_url, and proxy_url; the Python SDK currently ignores is_universal_api.

The initial /api/apikey/login request still needs the configured app URL because URL discovery has not happened yet. Legacy and non-universal organizations may also require distinct app, API, or proxy hosts. Therefore, the client can guarantee one transport immediately, but cannot yet guarantee one physical origin for every deployment.

The router provides a migration path:

• bootstrap through app_url;
• record is_universal_api and the discovered URLs so the router can later prefer one Universal API origin — actually switching traffic to it is a user-visible routing change (firewall allowlists, self-hosted proxies, and cassette matching all observe the host) and ships in its own PR with release notes, never inside a "behavior change: none" PR;
• retain explicit per-target URLs for legacy and self-hosted compatibility. Preserve the current override precedence: BRAINTRUST_API_URL/BRAINTRUST_PROXY_URL environment variables win over discovered organization URLs (_check_org_info). Note that no constructor override for api_url/proxy_url exists today — login() accepts only app_url, api_key, and org_name — so if the new client adds explicit URL parameters, their precedence (constructor over environment over discovered) is new behavior to define, not existing behavior to preserve;
• keep Universal Proxy URL normalization centralized, as _normalize_proxy_conn_url() already does inside proxy_conn() today;
• preserve the proxy-to-API fallback when no proxy URL is configured;
• keep absolute unauthenticated URLs out of the router and Transport.request() entirely; they are served by the specialized object-storage path, which never attaches the Braintrust Authorization header.

Bootstrap should:

1. construct one router with the configured app URL;
2. construct one session and transport;
3. call /api/apikey/login through the app target;
4. select the requested organization;
5. update the router with api_url, proxy_url, and is_universal_api, then apply environment URL overrides with the same precedence as the current login path (and any new explicit constructor URLs ahead of both, per the migration-path note above);
6. construct the service facade around the same transport;
7. while the legacy connections exist, propagate the login result to them — set their tokens and call make_long_lived() on the app, API, and proxy connections exactly as login() does today — so un-migrated call sites keep working and keep their post-login request-exception retries until PR 12 removes the legacy path.

BraintrustState can then hold the client. Existing app_conn(), api_conn(), and proxy_conn() methods — and the module-level api_conn()/app_conn()/proxy_conn() functions in logger.py, which framework2.py and cli/install/api.py import directly — remain as temporary isolated legacy connections. Each call site switches to the shared transport when it migrates to a service; the legacy sessions disappear only after their mutable surface is no longer needed.

The shim contract must cover HTTPConnection's mutable surface, not just its request methods: make_long_lived() (which installs RetryRequestExceptionsAdapter only when no custom adapter is set), set_token(), _set_adapter(), and direct .session access. During migration, these mutators must keep applying to isolated legacy connections only, so legacy call sites neither lose their post-login request-exception retries before PRs 6 and 8 assign explicit policies, nor stack the legacy retry loop underneath the new policy engine for migrated endpoints. New services use the policy-aware transport directly; a shim with raw .session access cannot honestly be backed by the same single session while preserving arbitrary mutation behavior.

Service responsibilities

A service owns endpoint semantics:

• request and response models, with forward-compatible response parsing that tolerates additive backend fields;
• validation before sending;
• endpoint path and method;
• target, organization scoping, and required endpoint headers such as x-bt-org-name;
• retry mode;
• response parsing;
• documented status-to-domain translations.

For example, ExperimentsAPI would own registration, lookup, base lookup, and comparison. The comparison operation would use SAFE_READ and either return a valid comparison or raise a typed API error. It would never manufacture empty score maps.

The high-level Experiment.summarize() method would remain responsible for building ExperimentSummary and deciding whether a final comparison failure should raise or produce a clearly marked lenient result.

Specialized clients

Log ingestion

Background logging should use shared low-level request and error primitives but retain a dedicated ingestion implementation for batching, queueing, overflow uploads, at-least-once retries, synchronous flush behavior, and failed-payload persistence.

Function invocation

Function invocation needs a dedicated non-retrying path because it supports streaming, can be long-running, incurs provider cost, and can execute tools or arbitrary user code.

Object storage

Signed object-storage requests should not inherit Braintrust API retry behavior or Braintrust authorization headers. They execute outside Transport.request() and RequestTarget: the object-storage helper owns its own session (or per-request connection), separate from the shared Braintrust session, mirroring today's dedicated token-less HTTPConnection(base_url="") instances. Two properties of the current path must carry over explicitly:

• No Authorization header, ever. Session separation makes the credential-scoping guarantee structural rather than dependent on a per-request suppression branch.
• Adapter/session injection still applies. Today the object-store connections are constructed with the global _http_adapter, so set_http_adapter() (and VCR-style test injection) reaches object-store uploads and downloads. The specialized path must keep an injection point, and PR 12's final target-independent set_http_adapter() contract must state explicitly that it includes object-store requests.

Each workflow must specify whether replay is safe:

• code-bundle upload to the same overwrite-enabled URL can retry;
• attachment upload PUTs use backend-supplied headers, including If-None-Match: *, plus x-ms-blob-type: BlockBlob for Azure. The signed URL is issued with overwrite disabled. After an ambiguous upload result, replay may return an object-store precondition failure; the attachment client must check GET /attachment and translate the result rather than treating the upload as an overwrite or expecting initialization to issue another URL.

Compatibility strategy

• Keep existing high-level public APIs unchanged while migrating them to braintrust.api.
• Preserve lazy login and lazy resource registration behavior.
• Preserve set_http_adapter() opt-out semantics: supplying a custom adapter disables the SDK retry loop rather than composing two potentially retrying layers. An explicitly injected session also owns retries by default; callers may opt the SDK loop back in only when its active adapters do not retry. During migration, retain the characterized timing-dependent legacy scope. After unsafe proxy calls use RetryMode.NONE, normalize the final scope so the custom adapter consistently applies to app, API, proxy, and signed-object requests regardless of installation timing. The adapter owner is responsible for replay safety. The new public client should prefer explicit Session or Transport injection while retaining set_http_adapter() as the global convenience API.
• Keep HTTPConnection as a compatibility wrapper or alias during migration, honoring the mutator contract (make_long_lived(), set_token(), _set_adapter(), raw .session access) described in the bootstrap section.
• Keep BraintrustHTTPError a subclass of AugmentedHTTPError so existing except AugmentedHTTPError: handlers and user imports keep working (see "Structured errors").
• Re-export moved summary/domain types from their existing locations if needed.
• Do not change serialized result shapes in a transport-only PR.
• Make fallback behavior changes in dedicated PRs with release notes.

Incremental PR plan

Each PR should be independently mergeable and should avoid mixing code movement with behavior changes unless explicitly stated.

Planning PR: document the architecture and audit

Scope:

• add this design document;
• link issue Experiment.summarize() silently discards scores/metrics on any fetch failure instead of surfacing or retrying #639;
• record the call-site retry classifications;
• agree on public type names and summary failure semantics.

Behavior change: none.

PR 1: Extract the existing transport without changing behavior (merged in 625a5e77bcd0192bae4aa8e30c86e753d34187d2)

Scope:

• create braintrust/api/_transport.py;
• move HTTPConnection and RetryRequestExceptionsAdapter out of logger.py;
• preserve existing imports through aliases or re-exports;
• preserve current session, adapter, timeout, and retry behavior exactly, including the adapter's known full-body buffering of stream=True responses (see "Session sharing constraints"; changed only in PR 9);
• move focused HTTP tests next to the transport or update their imports.

Behavior change: none.

Validation:

• existing core tests;
• characterization tests proving the extracted transport has identical behavior;
• import compatibility tests for any currently reachable symbols.

PR 2: Add structured errors and the policy-aware request engine (merged in 0cbf393e8992c756e3d10e16be4be9eb659c9abe)

Scope:

• add errors.py and policies.py;
• implement one request loop that handles both transport exceptions and HTTP statuses;
• add RetryMode.NONE, SAFE_READ, IDEMPOTENT_WRITE, and LOG_INGESTION;
• add Retry-After, exponential backoff, elapsed-time limits, and deterministic sleep injection;
• keep existing call sites on a temporary legacy path so endpoint behavior does not change yet;
• ensure custom adapters do not cause accidental nested retries.

Behavior change: none for existing endpoint calls.

Validation:

• exercise the transport over real HTTP sockets rather than mocking requests.Session, send(), or Response objects;
• use a deterministic local HTTP server only for protocol conditions that cannot be reliably produced by the real Braintrust service, such as scripted 429/5xx sequences, disconnects, timeouts, and retry-budget exhaustion;
• add 429 coverage for delay-seconds, HTTP-date, absent and malformed Retry-After, and a delay larger than the remaining retry budget;
• prove valid Retry-After values are used directly without client backoff;
• prove unsafe requests are sent once under NONE;
• prove logical POST reads retry under SAFE_READ;
• use recorded real Braintrust requests for authentication, headers, URL construction, and representative successful service calls.

PR 3: Add client bootstrap and service facade

Scope:

• add the synchronous BraintrustClient, AuthAPI, EndpointRouter, and RequestTarget, without thread-wrapped async resource methods;
• centralize app login, organization selection, and API/proxy URL discovery;
• use one transport and session across the new client's routed app, API, and proxy requests, keeping each request's destination host identical to today's (including the proxy-to-API fallback when no proxy URL is configured);
• record is_universal_api and the discovered URLs in the router without changing which host any request targets; switching traffic to the Universal API origin is a separate PR with release notes;
• let BraintrustState lazily own a client, with thread-safe lazy construction;
• retain the BraintrustState connection methods and the module-level api_conn()/app_conn()/proxy_conn() functions as isolated legacy connections, honoring the mutator contract from the bootstrap section rather than pretending raw .session mutation can safely share the new client's session;
• keep the centralized login hydrating those legacy connections — set their tokens and call make_long_lived() exactly as login() does today (bootstrap step 7) — so un-migrated call sites neither lose authentication nor their post-login request-exception retries;
• support custom sessions/adapters for VCR and user configuration, preserving the set_http_adapter() no-stacked-retries invariant while the legacy path remains.

Behavior change: none.

Validation:

• existing login and state tests;
• VCR-backed login and organization-discovery tests recorded from the real Braintrust service;
• multiple-organization selection tests using recorded real response shapes;
• custom adapter/session integration tests without mocked response objects;
• Universal API routing, legacy multi-origin routing, and explicit/environment URL override precedence tests;
• proxy-target resolution falls back to the API host when no proxy URL is configured;
• legacy mutator coverage: make_long_lived(), set_token(), and set_http_adapter() retain their characterized target- and timing-dependent behavior;
• after a login through the new bootstrap, the legacy connections are authenticated and long-lived: an un-migrated call site can issue a request and still gets request-exception retries;
• no duplicate client/session creation under concurrent lazy access.

PR 4: Add the experiment service with behavior parity

Scope:

• add experiment request/response models;
• migrate experiment lookup, explicit comparison-name lookup, base lookup, and comparison requests into ExperimentsAPI;
• delegate Experiment methods to the service;
• preserve the current high-level fallback behavior temporarily;
• preserve the documented status-to-domain translations: base-experiment lookup returns None on HTTP 400 ("no base experiment") instead of raising a typed error;
• mark reads as SAFE_READ so transient 429/5xx failures receive bounded retries.

Behavior change:

• transient read failures may now recover;
• final summary failure behavior remains unchanged until the next PR.

Validation:

• VCR-backed experiment creation, base lookup, and comparison against the real Braintrust service;
• real backend response shapes with and without a baseline and with an explicit baseline;
• a fresh project's base-experiment lookup (HTTP 400) returns None without raising or retrying;
• end-to-end Experiment.summarize() coverage rather than mocking ExperimentsAPI;
• real-socket transport coverage for transient 429/5xx and network failures that cannot be recorded reliably from Braintrust;
• no retry for 400/401/403 or decoding failures;
• existing experiment summary formatting and serialization tests.

PR 5: Fix experiment summary failure semantics

Scope:

• stop treating failed comparison retrieval as a successful empty comparison;
• stop silently swallowing failures of the explicit comparison-name lookup; surface them in the structured failure description;
• add a nested discriminated comparison result to ExperimentSummary as the authoritative representation: SummarySuccess, SummarySkipped, or SummaryFailed;
• put the real scores and metrics maps only on SummarySuccess, a reason on SummarySkipped, and structured error details only on SummaryFailed; preserve Braintrust response bodies verbatim and include only explicitly allowlisted response headers;
• keep the existing top-level ExperimentSummary.scores and ExperimentSummary.metrics as deprecated read-only bridges delegating to the nested result (the success maps on SummarySuccess, empty maps otherwise) and keep serializing them alongside the nested field — ExperimentSummary is a public serialized dataclass whose fields user code, custom reporters, and CI tooling read directly, so removing them outright would break Goal 6; removal is a future major-version change;
• add strict behavior that raises the typed API error for automation;
• update CLI and JSON reporters to display or serialize "summary unavailable" explicitly;
• document the compatibility and default-mode decision.

Decision:

• Experiment.summarize() is strict by default and raises the typed API error when comparison retrieval fails;
• framework-driven evals are also strict by default: a summary failure fails the eval rather than reporting successful completion;
• callers may explicitly opt into lenient reporting, which returns experiment metadata and a failure-marked summary instead of manufacturing empty score or metric maps.

Behavior change: intentional and user-visible.

Validation:

• genuine empty scores produce SummarySuccess(scores={}, metrics=...);
• the deprecated top-level scores/metrics bridges mirror the SummarySuccess maps, are empty for skipped/failed summaries, and stay in the serialized output;
• summarize_scores=False produces SummarySkipped with a reason;
• retry exhaustion raises in strict mode and produces SummaryFailed only in explicitly lenient mode, never an empty success;
• strict mode raises;
• explicitly lenient eval completion still returns experiment metadata and URL;
• JSON and human-readable output expose the failure.

PR 6: Migrate safe read services

Split this into small PRs if review size warrants it.

Suggested order:

1. QueriesAPI: BTQL and prompt version queries.
2. DatasetsAPI: lookup, environment resolution, fetch, and summary.
3. PromptsAPI: prompt retrieval.
4. FunctionsAPI: parameter/function metadata retrieval only.
5. ProjectsAPI: project lookup and read-only metadata.
6. AttachmentsAPI: metadata and download reads.

Behavior change: transient safe-read failures receive consistent retries.

Validation:

• endpoint-specific response parsing;
• pagination/cursor behavior;
• pinned and environment lookups;
• 4xx errors are not retried;
• fallback layers receive typed underlying errors.

PR 7: Make prompt and parameter fallback explicit

Scope:

• keep cache fallback in the high-level prompt and parameter loaders, not in the API services;
• define a public LoadedResource[T] envelope containing value, source (api, memory_cache, or disk_cache), cached_at, and an optional structured fallback_error;
• do not change existing loader return types: load_prompt() keeps returning Prompt (Goal 6), and the load metadata is attached to the returned object as a load_info attribute carrying the same fields; only new braintrust.api surfaces return the LoadedResource[T] envelope directly;
• stop rewriting infrastructure failures as "not found" for pinned versions or environments;
• preserve the original typed exception as the cause;
• do not expose a misleading stale boolean: for unpinned data the client cannot know whether a cached value is outdated, so expose only source, cache time, and the API failure that caused fallback.

Behavior change: clearer errors and explicit cache-source metadata.

Validation:

• latest unpinned request can use cache after a transient failure and returns source, cache time, and structured fallback-error metadata;
• pinned/environment requests surface infrastructure failures;
• genuine 404/not-found remains distinguishable;
• cache corruption and cache miss preserve the server cause.

PR 8: Migrate registration and deployment writes

Use separate service-level PRs for different retry safety classes.

Scope:

• plain project get-or-create registration: explicitly IDEMPOTENT_WRITE; variants that create secondary resources require their own classification;
• named dataset registration with update=False (including /v1/dataset POST): explicitly IDEMPOTENT_WRITE; updates, including /v1/dataset PUT, require an audit-effect review before opting in;
• named experiment registration with update=True (the /v1/experiment default when ensure_new is not true): conditionally IDEMPOTENT_WRITE; unnamed registration, ensure_new=True, and all other update=False registration use NONE;
• code-bundle allocation: NONE;
• insert-functions: NONE until idempotency exists;
• sandbox creation/listing: NONE;
• organization patching: explicit policy rather than inherited behavior.

Behavior change:

• unsafe POST operations no longer receive implicit request-exception retries;
• idempotent registration operations can recover from transient statuses.

Validation:

• a real-socket ambiguous/lost response test does not create a second experiment or code bundle;
• VCR-backed project and dataset get-or-create operations recover safely;
• function replacement and sandbox operations are attempted once;
• errors retain enough context for reconciliation.

PR 9: Migrate function invocation

Scope:

• add a dedicated invocation client;
• preserve streaming and non-streaming behavior;
• use RetryMode.NONE by default;
• separate connection timeout from potentially long response/read timeout;
• preserve provider and function errors without generic retries.

Behavior change:

• eliminates implicit duplicate invocation after request exceptions;
• streamed responses are no longer fully buffered by the transport (today RetryRequestExceptionsAdapter downloads the entire body even for stream=True, so streamed invocations are buffered in memory); this changes memory and latency behavior for streaming invocations and must be called out in release notes.

Validation:

• streaming is not eagerly consumed by the transport;
• read timeout does not replay the invocation;
• non-streaming response parsing remains compatible;
• 429/5xx responses surface immediately unless a future explicit invocation policy is introduced.

PR 10: Migrate attachment workflows

Scope:

• centralize attachment initialization, upload, status, metadata, and download;
• keep initialization on RetryMode.NONE: the client-generated key prevents duplicate attachment identity, but the backend creates the status object with overwrite disabled and returns 409 on duplicate initialization instead of a fresh signed URL;
• send the backend-provided If-None-Match: * upload header and add x-ms-blob-type: BlockBlob for Azure;
• retry safe metadata reads and explicitly idempotent status updates;
• reconcile an ambiguous upload by checking GET /attachment for contentLength/download metadata and then writing done if the persisted status still says uploading. Interpret object-store precondition failures only after that check;

Behavior change:

• attachment failures are more accurately reported; recoverability remains bounded by the current create-only initialization contract;
• initialization loses today's implicit request-exception retries (attachment upload runs after login() on the long-lived API connection, whose adapter retries connection failures up to 10 times), so a plain connection failure during POST /attachment — where the request never reached the backend and replay would be safe — now surfaces immediately. This must be called out in release notes; a future refinement could retry only failures known to have occurred before the request was sent, without waiting for the backend initialize/resume contract.

Validation:

• initialization request fails before the backend creates the status object;
• initialization succeeds but its response is lost, and the client does not blindly turn that into a misleading successful retry;
• upload succeeds followed by a lost response and is reconciled through GET /attachment;
• duplicate initialization returns and preserves HTTP 409 under the current backend contract;
• status update retries;
• object download retries;
• If-None-Match and Azure header behavior across supported object stores.

PR 11: Isolate background log ingestion

Scope:

• move background log HTTP operations into a dedicated ingestion module;
• make its at-least-once semantics explicit;
• prevent nested transport retries;
• preserve overflow uploads, batching, synchronous flush, and payload persistence;
• allow the memoized /version probe failure to be re-probed instead of permanently disabling overflow uploads for the process;
• separately decide whether 413 and async retry exhaustion should continue dropping data.

Behavior change: ideally none initially; data-loss semantics should be changed only in a dedicated follow-up.

Validation:

• exact attempt counts;
• no adapter-level multiplication of attempts;
• successful replay of stable log payloads;
• overflow upload behavior;
• synchronous versus asynchronous failure behavior;
• failed payload persistence.

PR 12: Remove the legacy transport path

Scope:

• remove temporary legacy retry behavior;
• stop constructing endpoint-specific HTTPConnection objects in logger.py and consolidate Braintrust targets onto the policy-aware session only after all unsafe proxy calls have explicit NONE policies;
• remove compatibility shims that are not publicly supported;
• retain documented adapter/session injection and implement the decided target-independent set_http_adapter() scope, including the intentional removal of the old late-mutation quirk and disabling SDK retries whenever a custom adapter is present;
• update internal imports and architecture documentation.

Behavior change: completes adoption of explicit endpoint policies.

Validation:

• full core test suite;
• lint and type checks;
• wheel sanity check if imports or package discovery changed;
• a search confirming that Braintrust endpoint paths live in service modules or explicitly specialized clients.

Follow-up: stabilize and document the public API

The client is exposed through braintrust.api as services are added:

from braintrust.api import BraintrustClient

client = BraintrustClient(api_key="...", org_name="my-org")
project = client.projects.create(name="my-project")
summary = client.experiments.summarize(experiment_id="...")

This is the convenience constructor: it performs login, organization selection, and URL discovery internally, then builds the transport-based client described in the facade section.

After the migrated services have seen internal and external use, perform a dedicated API review covering naming, pagination, model stability, whether a separate native AsyncBraintrustClient is warranted, and compatibility guarantees. Add public reference documentation and examples for the supported service surface.

Testing strategy

Prefer the highest-fidelity practical test for each behavior:

1. Use end-to-end tests through the existing public SDK API whenever possible. Do not replace BraintrustClient, resource services, or transports with mocks in high-level tests.
2. Use VCR-backed requests recorded from the real Braintrust service for endpoint contracts, authentication, URL routing, request bodies, response parsing, and fallback behavior. Extend existing cassettes rather than constructing canned response dictionaries.
3. Use the local Braintrust backend for integration scenarios that require controlled server state when practical.
4. Use a deterministic local HTTP server over real sockets only for transport failures that cannot be safely or reliably induced against Braintrust, including disconnects, timeouts, malformed responses, exact 429 Retry-After timing, and scripted 5xx recovery. Prefer this over mocking requests.Session.send() or creating fake Response objects.
5. Use ordinary unit tests only for pure parsing, policy selection, delay calculation, and model validation.

Additional rules:

• Add characterization coverage before moving each existing call site.
• Test attempt counts, not only final success or failure.
• For every write operation, include realistic coverage showing whether replay is allowed.
• Do not hand-author VCR cassettes; record them from actual requests.
• Test custom adapter/session injection through real requests to prevent double retries.
• Keep log-ingestion tests separate from ordinary API transport tests.
• If a mock or fake is unavoidable, document why real HTTP, VCR, or the local backend was impractical.

Resolved decisions

1. Experiment.summarize() and framework-driven evals are strict by default. A summary retrieval failure raises the typed API error and fails the eval. Explicit lenient mode returns experiment metadata plus a failure-marked summary; the authoritative nested result never disguises failure as empty score or metric maps (only the deprecated top-level bridge fields stay empty, as they are today).
2. ExperimentSummary contains a nested discriminated comparison result as the authoritative representation: SummarySuccess has score and metric maps, SummarySkipped has a reason, and SummaryFailed has structured error details. Braintrust response bodies are preserved verbatim and response headers use an explicit allowlist. Strict mode raises rather than returning SummaryFailed; that variant exists for explicit lenient mode. The nested result never encodes state through nullable or fake empty maps. The existing top-level scores/metrics fields remain as deprecated read-only bridges (delegating to SummarySuccess, empty otherwise) and stay in the serialized shape until a future major version, so existing consumers of the public dataclass keep working.
3. The default retryable 5xx statuses are 500, 502, 503, and 504, and only replayable operations may retry them. A 503 Retry-After is honored under the same rules as 429. Other 5xx statuses require an explicit endpoint override.
4. The default SAFE_READ policy permits four total attempts within 60 seconds of total monotonic elapsed time, including requests and sleeps, with a 20-second default per-attempt timeout so a first-attempt timeout still leaves room to retry. Timeouts and sleeps are clamped to the remaining budget. NONE operations keep the current 60-second default timeout. Long-running endpoints require explicit policy overrides covering both per-attempt timeout and budget, and a policy whose budget does not exceed its per-attempt timeout is rejected.
5. Custom transports do not compose with SDK retries by default: supplying a session or adapter disables the SDK retry loop, and the transport owner assumes responsibility for replay safety. A caller may explicitly enable SDK retries only when it knows the injected adapters do not retry. The final adapter scope consistently includes app, API, proxy, and signed-object requests regardless of installation timing. Explicit Session or Transport injection is preferred for the new client, while set_http_adapter() remains the global convenience API.
6. Server idempotency design is out of scope for this client architecture. The client classifies writes against the current documented backend contract and keeps unsafe operations on RetryMode.NONE; it may opt into retries later if the backend independently provides a verified idempotency contract.
7. Cache-source metadata is modeled as a public LoadedResource[T] with the value, source (api, memory_cache, or disk_cache), cache timestamp, and optional structured fallback error. Braintrust response bodies are preserved verbatim and response headers use an explicit allowlist. Existing high-level loaders keep their return types — load_prompt() still returns Prompt, with the metadata attached as a load_info attribute — and only new braintrust.api surfaces return the envelope directly. API service methods remain cache-free. No stale boolean is exposed because the client cannot establish whether unpinned cached data is current.
8. Preserve the existing background workers for queued log ingestion, but keep the initial public resource client synchronous. Do not expose asyncio.to_thread() wrappers as async client methods. A future async API, if warranted, should be a separate native AsyncBraintrustClient with dedicated transport, cancellation, streaming, timeout, and test coverage.

Recommended first implementation slice

The smallest useful sequence is:

1. extract the transport without behavior changes;
2. add typed errors and opt-in retry policies;
3. add the client facade;
4. migrate experiment base lookup and comparison reads;
5. fix experiment summary failure semantics.

That sequence addresses issue #639 while establishing the architecture needed to safely migrate the remaining endpoints.