mirror of https://github.com/hanzoai/insights.git synced 2026-10-30 16:15:31 +00:00

Files

T

History

zeekay 19fdf55ceb feat(auth): one key scheme, minted and read in one place

A key now opens with what it is allowed to touch. `pk-` is publishable and
safe in a browser; `sk-` is secret and never leaves a server. The five marks
it replaces all began `ph` -- the upstream's initials -- and they were the
last of its brand a customer read back to us, in every SDK snippet. No word
rule could reach them, because `ph` is not a word.

The marks were also not telling the truth. The constant said `phc_` while
every live project token said `hi_`, so the middleware that routes on the
constant matched no real token at all. Two spellings existed and one was
fiction. Minting now happens in exactly one function and reading in exactly
one, so a second spelling has nowhere to live.

`phx_` (personal) and `phs_` (project secret) both become `sk-`. The mark
now says only that a value is secret; which store holds it is settled by
asking. That was already how personal keys worked -- keys minted before the
marks carry none, so the store always decided -- and both secret resolvers
already returned None on a miss and fell through, so one shared extractor
replaces the two that could drift apart.

OAuth's access and refresh tokens stay distinguishable as `at-` and `rt-`,
because the parser tells them apart before either is looked up, and the
domain-verification token becomes `vt-` rather than keeping a sixth mint of
its own.

`bin/debrand --check` now refuses the marks as a shape, so a seventh cannot
be coined. Two rules that were about to rename things that are not ours to
spell are bounded: `Lemon` no longer eats `LemonSqueezy`, and the `hog_flow`
rules no longer reach a Postgres table, a Datastore table or a route.

Keys are stored hashed, so a mark cannot be rewritten in place: the three
live project tokens have to be reminted.

2026-08-08 19:42:44 -07:00

benchmark

…

runner

feat(auth): one key scheme, minted and read in one place

2026-08-08 19:42:44 -07:00

README.md

…

README.md

MCP agent-experience evals

The objective function for improving the MCP server: a fixed benchmark of agent tasks, sampled from real usage, that can be scored against a live MCP server. A change to a tool description, schema, or handler "improves the MCP" only if these scores say so.

Two consumers:

Regression protection — run before/after any change to tool descriptions, schemas, or the catalog.
The improve-my-mcp campaign — an autoresearch-style loop that proposes a change, re-runs the affected benchmark slice, and only keeps changes that measurably help. Every campaign PR carries before/after scores from this harness as evidence.

Layout

benchmark/tasks.yaml — the task set (v1). Each task is a realistic agent goal with the tools a competent agent should reach for.
benchmark/schema.ts — zod schema, loader, and types. tests/evals/benchmark.test.ts validates the fixtures against the schema and the live tool catalog, so a tool rename or removal fails CI here instead of silently invalidating the benchmark.

Task format

- id: flags-list-active # kebab-case, unique
  category: feature-flags # see TASK_CATEGORIES in schema.ts
  intent: 'List all our active feature flags.' # what the agent is asked, phrased like a real request
  expected_tools: [feature-flag-get-all] # what a competent agent should call
  acceptable_tools: [execute-sql] # also fine; no tool-selection penalty
  success_criteria: "Returns the project's active flags by key." # pass condition for the agent-mode judge
  probe: # optional: deterministic call, no LLM needed
    tool: feature-flag-get-all
    args: {}
    max_ms: 15000

Modes

Probe mode (deterministic, no LLM): executes each task's probe against a live server and validates schemas, latency, and error responses for every referenced tool. Probes must reference read-only tools — the fixture test enforces readOnlyHint, and the runner refuses anything else, so a bad fixture cannot mutate project data.

Agent mode (LLM): replays each intent through an agent loop against the live server, then scores task success against success_criteria (LLM judge) and tool selection against expected_tools/acceptable_tools.

Scores per run: task success rate, tool-selection accuracy, schema-validation failure count, tool error rate, retries per task, latency p50/p95, tokens per task.

Authoring rules

Sample intents from real usage ($mcp_intent clusters, query-mcp-tool-sample-intents) but paraphrase — never paste customer text verbatim, and never include PII.
Keep the task set stable within a campaign: scores are only comparable across runs of the same benchmark version. Bump version on breaking changes to the set.
Weight new tasks toward observed pain: high-error tools, discoverability misses (intents where agents picked the wrong tool), and multi-step chains (resolve id → act).
A task must be achievable in a seeded demo project — don't write tasks that depend on one specific production dataset.