Repository navigation
feat: Support ingestion keys in initLogger - #2573
Closed
Luca Forstner (lforst) wants to merge 5 commits into
Closed
Luca Forstner (lforst) wants to merge 5 commits into
Luca Forstner (lforst) wants to merge 5 commits into
Conversation
`initLogger({ ingestionKey })` and `BRAINTRUST_INGESTION_KEY` now log through
a project's ingestion URL with the key as a Bearer token. These loggers get
their own state and queue, never log in, don't look up the project and ignore
private credentials. An explicit `apiKey` or `state` still wins over the env
var, and combining one with `ingestionKey` throws.
Attachments and overflow batches go through the chunked `/v1/uploads` flow
before the rows that reference them, and rows are trimmed to the fields the
ingestion endpoint accepts. Span context is shared with the global state, so
nesting and W3C propagation keep working without a project id.
- Batches above 512 KiB overflow through an upload by default, since public clients often sit behind 1 MiB ingress limits and Lambda sees base64 bodies. - Parse the ingestion URL with `URLSearchParams`, require the exact 48 char key and trim all trailing slashes, matching the shared contract. - Drop the optional upload `sha256`, which read the whole upload into memory. - A `state` only counts as a private credential if it is logged in or has an `apiKey`. Otherwise it provides masking and the current logger. - Don't retry auth and validation failures, and honor `Retry-After`. - `flush()` drains every ingestion key queue of the state, not only the one of the current logger. - Document that ingestion keys can update any row whose id is known.
The public server path rejects `external_attachment` references, since they can point at any object store content. Rows that contain an `ExternalAttachment` or a raw external reference are now dropped before they are sent, and the error goes to `onFlushError`. Managed attachments and already committed `braintrust_attachment` references keep working, and the private path is unchanged. The ids of `orgProjectMetadata` are now sent along for the server to check instead of being dropped. `state` only provides context, so its login is ignored and only an explicit `apiKey` conflicts with `ingestionKey`.
Runs the URL, upload response and chunk plan cases of typespecs/src/public-ingestion.fixture.json (braintrustdata/braintrust@74ce022) through the parser and the upload flow.
Ingestion key requests no longer follow redirects, which could forward the
key to another origin, and the key is redacted from error responses and
transport errors. They also skip `keepalive`, since browsers reject keepalive
bodies above 64 KiB and native rows go up to 512 KiB.
Upload grants are anchored on a monotonic clock, and chunk and complete
requests are aborted once the grant expires. A 410 requests a new grant,
while invalid responses are not retried. An empty `BRAINTRUST_INGESTION_KEY`
now fails instead of falling back to private credentials.
`updateSpan({ exported })` updates project log spans through the current
ingestion key logger, and spans exported without a project are rejected
instead of being updated with private credentials.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Ref: https://github.com/braintrustdata/braintrust/pull/21591 (contract), braintrustdata/braintrust-sdk-python#845 (Python SDK)
This adds public trace ingestion keys to the JS SDK.
initLogger({ ingestionKey })orBRAINTRUST_INGESTION_KEYtakes the full ingestion URL (https://<data plane>/ingest?ingestKey=bt-ik-...), and the logger then only talks to that endpoint with the key as a Bearer token. The key is never sent in a URL.What a public logger does differently:
/version, or readsBRAINTRUST_API_KEY.projectIdandorgProjectMetadataids are only sent along for the server to check, andprojectNameis ignored.apiKeywins over the env var, and combining it withingestionKeythrows. Astateonly provides masking and the current logger. Its login is ignored, and an empty env var throws instead of silently falling back to private.flush()drains all public queues, andupdateSpan({ exported })goes through the current public logger.{root}/v1/logs, trimmed to the fields the endpoint accepts. Batches above 512 KiB overflow through the chunked/v1/uploadsflow, and attachments are uploaded the same way before the rows that reference them.ExternalAttachmentis rejected, since it can point at any object store content.keepalive, and the key is redacted from error bodies and transport errors. Upload requests are bounded by the remaining grant lifetime. 410s get a new grant, auth/validation failures and invalid responses aren't retried, and 429s honorRetry-After.The private logging path is unchanged.
This can merge independently of the backend, but working keys only exist once the backend stack (contract, control plane and data plane) is rolled out. I haven't run this against a real server that supports ingestion keys yet. The hosted e2e replay suite doesn't cover this feature, so it only shows that private logging is unaffected.
Validation (all local):
js/src/ingestion-key.test.tsruns the actual logger and batcher against a mocked data plane implementing the frozen contract. It also checks the URL, upload response and chunk plan cases of the shared fixture from https://github.com/braintrustdata/braintrust/pull/21591.logger,background-logger,propagationandtracesuites, the corepnpm testsuite andcheck:typingspass locally.