mirror of
https://github.com/hanzoai/insights.git
synced 2026-10-30 16:15:31 +00:00
zeekay
19fdf55ceb
A key now opens with what it is allowed to touch. `pk-` is publishable and safe in a browser; `sk-` is secret and never leaves a server. The five marks it replaces all began `ph` -- the upstream's initials -- and they were the last of its brand a customer read back to us, in every SDK snippet. No word rule could reach them, because `ph` is not a word. The marks were also not telling the truth. The constant said `phc_` while every live project token said `hi_`, so the middleware that routes on the constant matched no real token at all. Two spellings existed and one was fiction. Minting now happens in exactly one function and reading in exactly one, so a second spelling has nowhere to live. `phx_` (personal) and `phs_` (project secret) both become `sk-`. The mark now says only that a value is secret; which store holds it is settled by asking. That was already how personal keys worked -- keys minted before the marks carry none, so the store always decided -- and both secret resolvers already returned None on a miss and fell through, so one shared extractor replaces the two that could drift apart. OAuth's access and refresh tokens stay distinguishable as `at-` and `rt-`, because the parser tells them apart before either is looked up, and the domain-verification token becomes `vt-` rather than keeping a sixth mint of its own. `bin/debrand --check` now refuses the marks as a shape, so a seventh cannot be coined. Two rules that were about to rename things that are not ours to spell are bounded: `Lemon` no longer eats `LemonSqueezy`, and the `hog_flow` rules no longer reach a Postgres table, a Datastore table or a route. Keys are stored hashed, so a mark cannot be rewritten in place: the three live project tokens have to be reminted.
LLM Gateway Benchmarks
Measures the overhead the Insights LLM gateway adds compared to calling providers directly.
Metrics
- TTFT — Time to first token (streaming only)
- TPS — Tokens per second (streaming only)
- Total latency — End-to-end request time
Each metric is collected for both direct provider calls and gateway-proxied calls, then compared.
Setup
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
cp .env.example .env
# Edit .env with your API keys
Usage
# Quick connectivity check
python benchmark.py --iterations 1 --warmup 0 --models gpt-4.1-nano
# Full run with defaults (5 iterations, 1 warmup, all models)
python benchmark.py
# More iterations for stable numbers
python benchmark.py --iterations 10
# Test under concurrent load
python benchmark.py --concurrency 5 --iterations 3
# Only streaming or non-streaming
python benchmark.py --streaming-only
python benchmark.py --non-streaming-only
# Specific models
python benchmark.py --models gpt-4.1-mini claude-haiku-4-5-20251001
# Custom prompt and token limit
python benchmark.py --prompt "Explain quantum computing" --max-tokens 500
# Re-print saved results without re-running
python benchmark.py --report results/benchmark_YYYYMMDD_HHMMSS.json
Default models
| Model | Provider |
|---|---|
| gpt-4.1-mini | OpenAI |
| gpt-4.1-nano | OpenAI |
| claude-sonnet-4-5-20250514 | Anthropic |
| claude-haiku-4-5-20251001 | Anthropic |
Output
Results are printed as a comparison table showing mean / median / p95 for each metric, with an overhead row showing the absolute and percentage delta between direct and gateway calls.
Raw results are saved as JSON to results/ (gitignored) after each run for later review with --report.