mirror of https://github.com/hanzoai/insights.git synced 2026-10-30 16:15:31 +00:00

Files

T

zeekay 19fdf55ceb feat(auth): one key scheme, minted and read in one place

A key now opens with what it is allowed to touch. `pk-` is publishable and
safe in a browser; `sk-` is secret and never leaves a server. The five marks
it replaces all began `ph` -- the upstream's initials -- and they were the
last of its brand a customer read back to us, in every SDK snippet. No word
rule could reach them, because `ph` is not a word.

The marks were also not telling the truth. The constant said `phc_` while
every live project token said `hi_`, so the middleware that routes on the
constant matched no real token at all. Two spellings existed and one was
fiction. Minting now happens in exactly one function and reading in exactly
one, so a second spelling has nowhere to live.

`phx_` (personal) and `phs_` (project secret) both become `sk-`. The mark
now says only that a value is secret; which store holds it is settled by
asking. That was already how personal keys worked -- keys minted before the
marks carry none, so the store always decided -- and both secret resolvers
already returned None on a miss and fell through, so one shared extractor
replaces the two that could drift apart.

OAuth's access and refresh tokens stay distinguishable as `at-` and `rt-`,
because the parser tells them apart before either is looked up, and the
domain-verification token becomes `vt-` rather than keeping a sixth mint of
its own.

`bin/debrand --check` now refuses the marks as a shape, so a seventh cannot
be coined. Two rules that were about to rename things that are not ours to
spell are bounded: `Lemon` no longer eats `LemonSqueezy`, and the `hog_flow`
rules no longer reach a Postgres table, a Datastore table or a route.

Keys are stored hashed, so a mark cannot be rewritten in place: the three
live project tokens have to be reminted.

2026-08-08 19:42:44 -07:00

12 KiB

Raw Permalink Blame History

Choosing an AI gateway

✅ Default to Insights/ai-gateway for new callers and features.

⚠️ Use services/llm-gateway temporarily only when an active caller is blocked by a gap below.

Migration policy

services/llm-gateway is under an unofficial code freeze. Everyone should move to the Go gateway unless they can name a contract it does not support.

A PR that changes the Python gateway must:

Name the active caller and its use case.
Name the exact Go gateway parity gap.
Explain why the change cannot wait for or land in the Go gateway.
Limit the Python change to the blocked caller's needs.
Update this document when it discovers or closes a gap.

Bug, security, and reliability fixes for blocked callers are valid reasons. Convenience, an existing Python client, or an unfamiliar Go API are not parity gaps.

When a gap closes, new work uses the Go gateway and affected callers should migrate. Do not add the same feature to both gateways by default.

Choose by use case

✅ Use the Go gateway

Use case	Why it fits
Customer LLM traffic	The caller uses a project secret or OAuth credential, the team wallet should pay, and standard OpenAI or Anthropic APIs are enough.
Server-to-server Insights call	A `sk-` credential, team wallet billing, and informational event properties provide enough policy and attribution. Internal spend can use a Insights-owned team wallet.
Stock SDK proxy	The caller needs OpenAI Chat Completions or Responses, Anthropic Messages or token counting, streaming, idempotency, or the model catalog.
Gateway-managed routing	The caller accepts Go host selection across OpenAI, Anthropic, Azure OpenAI, Bedrock, or selected Modal models.

Existing Django callers should use build_openai_client, build_async_openai_client, or build_async_anthropic_client from insights/llm/gateway_client.py. These builders keep a temporary Python fallback during rollout.

⛔ Stay on the Python gateway for now

Use case	Blocking contract
Insights Desktop, Insights Code, cloud agents, onboarding, Wizard, and similar first-party products	Trusted product identity, OAuth application allowlists, OAuth user attribution, server-minted credential requirements, per-product model policy, and product billing or unbilled policy are not available together in Go. An `ai_product` event property is not an authorization or billing boundary.
Product or user budget enforcement	Python supports product and user cost limits, plan checks, quota buckets, and requests that debit no wallet. Go currently admits against the credential owner's team wallet and does not branch settlement on credential billing mode.
OpenRouter, Fireworks, Cloudflare Workers AI, or Baseten	These provider paths are not available in Go.
OpenAI audio transcription	Go has no transcription endpoint.
OpenAI models through Anthropic Messages	Go does not provide this reverse translation.
Python product usage status	The Python product usage and quota API is different from Go request usage and wallet APIs.

🔎 Verify before switching

These are compatibility checks, not automatic blockers:

Model: it appears in GET /v1/models and supports the requested API shape and capabilities.
Credential: Go accepts the credential type and projects the correct team, scope, and revocation state.
Billing: identify which team owns the Go credential and should pay, then confirm its wallet is funded for the expected usage. Internal workloads can debit a Insights-owned team wallet so spend stays attributable without charging a customer.
Attribution: X-Insights-Distinct-Id, X-Insights-Trace-Id, and X-Insights-Properties provide enough event context. Go does not derive the event distinct ID from OpenAI user, Anthropic metadata.user_id, or the OAuth user. Caller-supplied properties are not trusted policy.
Provider behavior: health-based routing or strict X-Insights-Provider pinning matches the caller's fallback requirements.
Wire behavior: request fields, streaming chunks, errors, timeouts, and retries match what the caller handles.
Metadata: Python per-key property and feature flag headers are converted to the Go JSON properties header.

Parity map

Contract	Go gateway today	Python-only behavior that can block migration
Authentication	`sk-` project secrets and `at-` OAuth credentials with the literal `llm_gateway:read` scope; credential resolves team and revocation state.	`sk-` personal keys, wildcard OAuth scope handling, and product-specific API key or OAuth application rules.
Trusted first-party identity	No trusted product identity or product-scoped authorization boundary.	Product route selects allowlisted OAuth apps, server credential requirements, and model policy.
Attribution	Distinct ID comes from `X-Insights-Distinct-Id` or falls back to team ID. Supports trace ID, custom JSON properties, team project token, provider, usage, and cost.	OAuth user identity, body-derived OpenAI or Anthropic end-user identity, gateway-owned `$ai_product` and `$ai_billable`, per-key properties, and feature flag headers.
Billing and limits	Wallet reservation and settlement, idempotency, balance read, request usage read, and front-line rate limiting.	Product and user cost windows, plans, quota buckets, free tiers, unbilled products, and product usage status.
OpenAI APIs	Chat Completions, Responses, bare Responses alias, and normalized router chat.	Audio transcription and broader LiteLLM translation.
Anthropic APIs	Messages and token counting, including Bedrock-hosted models.	OpenAI models exposed through the Anthropic shape and Python-specific Bedrock opt-in behavior.
Providers	OpenAI, Anthropic, Azure OpenAI, Bedrock, and selected Modal models.	OpenRouter, Fireworks, Cloudflare Workers AI, and Baseten.
Models	Gateway-owned catalog, canonical IDs and aliases, capability checks, and router categories.	Broader LiteLLM model acceptance and Python product allowlists.
Routing and failure behavior	Health-based host choice, circuit breakers, hosted-provider failover, and strict provider pinning.	Caller opt-in Bedrock fallback and provider-specific Python routing.
Event metadata	One `X-Insights-Properties` JSON object plus dedicated distinct ID, trace ID, and provider headers.	`X-INSIGHTS-PROPERTY-` and `X-INSIGHTS-FLAG-` headers.

Migration checklist

Run /migrating-llm-gateway-callers to inventory and convert a caller.

Describe the caller and identify which identity controls its access and spend.
Check every relevant contract above against the real request and response.
Start with a shared Go-capable builder where one exists.
Test success, streaming if used, provider errors, attribution, and billing.
Keep the Python fallback only for staged rollout or a named blocker. Record the rollout plan or blocker in the PR.

Refreshing this document

Run /auditing-llm-gateway-parity after either gateway changes auth, attribution, billing, endpoints, providers, models, routing, or event metadata. The skill audits implementation sources in both repositories and updates this file without migrating callers.

Last verified on 2026-07-31 against:

Insights/insights master at 75c2b2ff6845097a3eb6dc5214e3f4f660395fb8
Insights/ai-gateway main at ea8230b6dbc4ebf6c4be83d359e561477a13b215

References

Python gateway: services/llm-gateway
Shared clients: insights/llm/gateway_client.py
Go gateway: Insights/ai-gateway, especially docs/product.md

12 KiB Raw Permalink Blame History