zeekay
19fdf55ceb
A key now opens with what it is allowed to touch. `pk-` is publishable and safe in a browser; `sk-` is secret and never leaves a server. The five marks it replaces all began `ph` -- the upstream's initials -- and they were the last of its brand a customer read back to us, in every SDK snippet. No word rule could reach them, because `ph` is not a word. The marks were also not telling the truth. The constant said `phc_` while every live project token said `hi_`, so the middleware that routes on the constant matched no real token at all. Two spellings existed and one was fiction. Minting now happens in exactly one function and reading in exactly one, so a second spelling has nowhere to live. `phx_` (personal) and `phs_` (project secret) both become `sk-`. The mark now says only that a value is secret; which store holds it is settled by asking. That was already how personal keys worked -- keys minted before the marks carry none, so the store always decided -- and both secret resolvers already returned None on a miss and fell through, so one shared extractor replaces the two that could drift apart. OAuth's access and refresh tokens stay distinguishable as `at-` and `rt-`, because the parser tells them apart before either is looked up, and the domain-verification token becomes `vt-` rather than keeping a sixth mint of its own. `bin/debrand --check` now refuses the marks as a shape, so a seventh cannot be coined. Two rules that were about to rename things that are not ours to spell are bounded: `Lemon` no longer eats `LemonSqueezy`, and the `hog_flow` rules no longer reach a Postgres table, a Datastore table or a route. Keys are stored hashed, so a mark cannot be rewritten in place: the three live project tokens have to be reminted.
Insights Log Capture Service
A service that receives OpenTelemetry Protocol (OTLP) logs via HTTP and processes them based on team authentication.
Features
- Receives OTLP logs via HTTP on
/v1/logsand/i/v1/logsendpoints - Supports Protobuf and JSON formats
- Supports JSONL (JSON Lines) format for multiple log batches
- Authenticates clients using Bearer tokens or query parameters
- Associates logs with specific team IDs
- Health check endpoints
- Prometheus metrics
Configuration
The service is configured using environment variables:
| Variable | Default | Description |
|---|---|---|
| BIND_HOST | :: | Host to bind the HTTP server |
| BIND_PORT | 4318 | Port for the HTTP server |
| MANAGEMENT_BIND_HOST | :: | Host to bind the health check and metrics server |
| MANAGEMENT_BIND_PORT | 8080 | Port for the health check and metrics server |
| MAX_REQUEST_BODY_SIZE_BYTES | 2097152 | Rejects larger request bodies, before and after gzip decompression |
| DROP_EVENTS_BY_TOKEN | (none) | Comma-separated tokens to drop |
Authentication
Clients must authenticate by sending a valid token either:
- In the Authorization header:
Authorization: Bearer your-project-api-key
- As a query parameter:
POST /v1/logs?token=your-project-token
The token is your Insights project token.
Response codes
| Status | Meaning | Client behavior |
|---|---|---|
| 200 | Accepted | — |
| 400 | Body could not be decoded as OTLP protobuf or JSON | Permanent |
| 401 | No token, or a token that cannot be a project API key (for example a sk- personal API key) |
Permanent, so the client stops and surfaces the misconfiguration |
| 413 | Body over MAX_REQUEST_BODY_SIZE_BYTES |
Permanent |
The 401 covers shape only: empty, over 64 characters, non-ASCII, containing a null byte, or
prefixed sk-. None of those can be a project API key, so answering 200 and dropping the
records downstream only hides the misconfiguration from whoever is sending the data. This is
the same check, and the same status, that event capture applies to the same input.
One misconfiguration is still answered 200: a well-formed token that belongs to no project. Resolving a token to a team needs Postgres, which this service does not have, so those records are still dropped by the ingestion consumer. Event capture does not validate this at the edge either.
A project over its billing quota is also still answered 200 and dropped by the consumer.
Rejections are counted on capture_logs_requests_rejected_total{reason, signal}, where reason
is one of missing_token, dropped_token or invalid_token.
Running the Service
From Source
cargo run --bin capture_logs
With Docker
docker build -t insights/capture-logs .
docker run -p 8000:8000 insights/capture-logs
Sending Logs
You can configure any OpenTelemetry-compatible client to send logs to this service. The service accepts:
Single JSON Format
Standard OTLP ExportLogsServiceRequest as JSON:
curl -X POST http://localhost:8000/v1/logs \
-H "Authorization: Bearer your-api-key" \
-H "Content-Type: application/json" \
-d '{"resourceLogs":[{"resource":{"attributes":[]},"scopeLogs":[{"logRecords":[{"body":{"stringValue":"Hello World"}}]}]}]}'
JSONL Format (JSON Lines)
Multiple ExportLogsServiceRequest objects, one per line:
curl -X POST http://localhost:8000/v1/logs \
-H "Authorization: Bearer your-api-key" \
-H "Content-Type: application/json" \
-d $'{"resourceLogs":[{"resource":{},"scopeLogs":[{"logRecords":[{"body":{"stringValue":"Log 1"}}]}]}]}\n{"resourceLogs":[{"resource":{},"scopeLogs":[{"logRecords":[{"body":{"stringValue":"Log 2"}}]}]}]}'
Protobuf Format
Standard OTLP protobuf encoding is also supported.
Requirements:
- Set the HTTP endpoint to
http://your-service-host:8000/v1/logs - Include your Insights project token in the Authorization header or as a query parameter
- Use standard OTLP log format (JSON, JSONL, or Protobuf)
Endpoints
Log Ingestion
POST /v1/logs- Accept OTLP logs (JSON, JSONL, or Protobuf)POST /i/v1/logs- Alternative endpoint for OTLP logsOPTIONS /v1/logs- CORS preflight supportOPTIONS /i/v1/logs- CORS preflight support
Management
/- Basic information page/_readiness- Readiness probe for Kubernetes/_liveness- Liveness probe for Kubernetes/metrics- Prometheus metrics
Development
Running Tests
cargo test
Building in Release Mode
cargo build --release