feat: telemetry usage metering - #7
Open
Ali (alifayed02) wants to merge 10 commits into
Open
Conversation
Ali (alifayed02)
changed the title
…rizedFetch Step 1 probe: @opentelemetry/otlp-transformer@0.220.0 exports ProtobufLogsSerializer/ProtobufMetricsSerializer/ProtobufTraceSerializer exactly as expected, no fallback needed.
…112 KiB limit BatchLogRecordProcessor's default maxExportBatchSize (512) can serialize to a protobuf payload larger than workers/telemetry's MAX_BODY_BYTES cap in the elide cloud repo, which 413s the whole batch — dropping every record in it, including any aura.usage.tokens billing records riding along with observability logs. Pin a conservative 64-record cap on the LoggerProvider's batch processor.
emitOtelLog (init.ts) filtered every record — including the aura.usage.tokens billing record — on OTEL_LOG_LEVEL, so an operator setting OTEL_LOG_LEVEL=warn (or lower) to quiet observability noise would silently stop usage metering too. The billing record is metering, not observability, so exempt it from the level gate by eventName. Covered by a new out-of-process probe (mirroring the existing otel-*-probe.ts pattern so the global LoggerProvider singleton never leaks into the test runner): with OTEL_LOG_LEVEL=error, an ordinary info-level bridged log is still suppressed (the gate still works), while a billable chat.usage event's aura.usage.tokens record still reaches the collector (the exemption holds).
… path sendAuthorized did one authorizedFetch and nothing else: a transient 429/5xx permanently dropped the batch (BatchLogRecordProcessor does not re-queue a FAILED export), and no signal meant an export could hang forever. Mirror the stock otlp-exporter-base behavior in spirit: retry on 429/502/503/504 and network rejection with jittered exponential backoff (honoring Retry-After when present) up to 3 attempts total, and bound every attempt with AbortSignal.timeout (10s, the stock default). Non-retryable statuses still fail immediately, with no retry. sleep is injectable via AuthorizedExporterOptions so the new retry tests run instantly instead of on real timers. New tests: retry-then-success on 503 and on a network rejection, exhaustion after persistent 503s -> FAILED, a non-retryable 400 fails immediately without retrying, Retry-After overrides the computed backoff, each attempt carries an AbortSignal, and the previously-missing direct happy-path tests for AuthorizedTraceExporter and AuthorizedMetricExporter through the shared send.
… exists InitTelemetryOptions.cloud has been the documented remaining step for authenticated telemetry: with it, the Aura tier exports through the authorized exporters (fresh bearer per export via TokenManager.authorizedFetch) so the cloud relay can attribute usage to an account. Nothing passed it, so the tier always exported unauthenticated. resolveCloudTelemetryTransport closes that seam. It gates on the pure resolveAuraAuthorizedUrl first, so an install whose telemetry does not go to the Aura tier never opens the token database or pulls cloud auth onto the startup graph; cloud/token-store and cloud/auth stay behind lazy import() per the ownership note in src/cloud/index.ts. It never throws — a missing database or an invalid AURA_DOMAIN degrades to unauthenticated export, not a failed startup. This is dormant until an Elide Cloud login exists: nothing in the CLI constructs an AuraAuthClient today, so status().signedIn is always false and the transport resolves to undefined. When a login lands, authenticated export starts working with no further change here. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Ali (alifayed02)
force-pushed
the
feat/telemetry-usage-metering
branch
from
August 8, 2026 04:34
3e66297 to
689694a
Compare
There was a problem hiding this comment.
Pull request overview
Adds Aura-tier “usage metering” telemetry by emitting a dedicated billing log record per chat call and introducing authorized OTLP exporters that authenticate exports via the cloud token manager, with reliability fixes to avoid silent record drops.
Changes:
- Emit
aura.usage.tokenslog records from the OTLP sink for eachchat.usageevent (only positive token counters, plus provider/model). - Add Aura authorized OTLP log/trace/metric exporters using
otlp-transformer+TokenManager.authorizedFetch, including retry/backoff/timeout. - Harden export/metering reliability (OTEL log-level exemption for billing record, reduced log batch size) and add targeted tests/probes.
Reviewed changes
Copilot reviewed 13 out of 14 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| packages/coding-agent/src/telemetry/sink-otlp.ts | Adds billing event constant + attribute mapping and emits billing log record for chat.usage. |
| packages/coding-agent/src/telemetry/init.ts | Wires authorized exporters for Aura-tier destinations, caps log batch size, and exempts billing from OTEL_LOG_LEVEL. |
| packages/coding-agent/src/telemetry/authorized-exporters.ts | Implements authorized OTLP exporters and shared sendAuthorized with retry/backoff/timeout. |
| packages/coding-agent/src/telemetry/cloud-session.ts | Resolves Aura cloud transport lazily (only when Aura-tier destination would be used and a session exists). |
| packages/coding-agent/src/telemetry/index.ts | Re-exports cloud-session from the telemetry barrel. |
| packages/coding-agent/src/main.ts | Resolves cloud telemetry transport and passes it into initTelemetryExport when available. |
| packages/coding-agent/test/telemetry-sink.test.ts | Adds sink-level tests validating billing attribute mapping and “not billable” behavior. |
| packages/coding-agent/test/telemetry-export.test.ts | Adds an out-of-process probe test for the OTEL log-level billing exemption; adds tests for Aura authorized URL resolution precedence. |
| packages/coding-agent/test/telemetry-cloud-session.test.ts | Adds tests ensuring cloud transport resolution declines cheaply in non-Aura-tier scenarios or without session. |
| packages/coding-agent/test/telemetry-authorized-exporters.test.ts | Adds tests for retry semantics, failure mapping, and happy paths for all three authorized exporters. |
| packages/coding-agent/test/otel-log-level-billing-probe.ts | New subprocess probe that validates billing record bypasses OTEL_LOG_LEVEL while ordinary info logs remain gated. |
| packages/coding-agent/package.json | Adds @opentelemetry/otlp-transformer dependency for authorized exporters. |
| package.json | Adds workspace dependency on @opentelemetry/otlp-transformer. |
| bun.lock | Locks the new dependency and reorders patchedDependencies entries. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+84
to
+86
| function defaultSleep(ms: number): Promise<void> { | ||
| return new Promise(resolve => setTimeout(resolve, ms)); | ||
| } |
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Emit a billing record per chat call and authenticate Aura-tier telemetry exports.
sink-otlp.tsemits oneaura.usage.tokenslog record perchat.usageevent, with one attribute per positive token count (input,output,cache_read,cache_write,reasoning) plus provider/model.Authorized{Log,Trace,Metric}Exporterserialize with the sameotlp-transformerthe stock exporters use, then send throughTokenManager.authorizedFetch— fresh token per export. Only the Aura tier constructs them; env endpoints and an explicittelemetry.endpointboth outrank it and never authenticate.Plus three fixes, each of which silently dropped billing records:
OTEL_LOG_LEVELgated the billing record, so quieting log noise also stopped metering. Now exempt byeventName.sendAuthorizeddid one fetch with no retry, so a transient 503 dropped the batch permanently (BatchLogRecordProcessordoesn't re-queue a FAILED export). Now 3 attempts with jittered backoff honoringRetry-After, each bounded by a 10s timeout.Why
Client half of the Metronome integration. Pairs with elide-dev/cloud#7, which writes these records to
usage_eventsand forwards them to Metronome. Neither half bills anything alone.The event name and attribute keys are consumed verbatim by
workers/telemetry/meter.ts— renaming either silently stops metering, so treat both as frozen. The cloud side pins this with a golden-wire test.Not yet wired end-to-end. Nothing passes
InitTelemetryOptions.cloud, because the CLI has no cloud sign-in command yet (lands withcloud/auth.tsAuraAuthClient.manager). Without a signed-in session there's no token to attach, so Aura-tier exports go out unauthenticated and the edge 401s them — no usage is metered from the CLI until sign-in lands. Built-in and operator tiers are unaffected; Grafana telemetry keeps working.Testing
test/telemetry-authorized-exporters.test.ts(new, 11 tests): retry-then-success on 503 and on network rejection, exhaustion → FAILED, non-retryable 400 without retry,Retry-Afteroverriding backoff, per-attemptAbortSignal, happy path for all three exporters.sleepis injectable so these run instantly.test/otel-log-level-billing-probe.ts(new): out-of-process probe, matching the existingotel-*-probe.tspattern so the globalLoggerProvidersingleton doesn't leak into the runner. WithOTEL_LOG_LEVEL=error, an ordinary info log stays suppressed while the billing record still reaches the collector.Verified locally:
telemetry-authorized-exporters.test.ts11/11. The probe suites need thepi_nativesaddon built (bun --cwd=packages/natives run build); without it, untouched baseline tests fail identically, so that failure is environmental. CI builds natives.bun checkpasses