mirror of
https://github.com/hanzoai/skills.git
synced 2026-11-19 00:47:36 +00:00
hanzoskills.com/llms.txt now opens with the install line, the API reference index and the company index, so an agent that lands on any one of the three can find the rest. robots.txt claimed 537 documents while llms.txt counts 920. The number is gone from robots.txt; llms.txt derives it. public/ is regenerated from scripts/site.py, which it had fallen behind: the copy-prompt menu and agent bar were in the generator and not in the committed pages. tests/site.mjs passes against the result. Co-authored-by: Hanzo Dev <[email protected]>
371 lines
43 KiB
HTML
371 lines
43 KiB
HTML
<!doctype html>
|
|
<html lang="en"><head>
|
|
<meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1">
|
|
<title>hip-0129</title>
|
|
<meta name="color-scheme" content="dark">
|
|
<style>
|
|
/* Committed to one visual world: a dark instrument panel. No light variant —
|
|
the dot field and the hairline rules only read on black, and a naive
|
|
inversion would wash both out. Every colour is painted explicitly so the page
|
|
never borrows a host background. */
|
|
@font-face{font-family:'Zen';src:url('/fonts/Zen-Variable.woff2') format('woff2');font-weight:100 900;font-style:normal;font-display:swap}
|
|
@font-face{font-family:'Zen';src:url('/fonts/Zen-Italic-Variable.woff2') format('woff2');font-weight:100 900;font-style:italic;font-display:swap}
|
|
@font-face{font-family:'Zen Mono';src:url('/fonts/ZenMono-Variable.woff2') format('woff2');font-weight:100 900;font-style:normal;font-display:swap}
|
|
@font-face{font-family:'Zen Mono';src:url('/fonts/ZenMono-Italic-Variable.woff2') format('woff2');font-weight:100 900;font-style:italic;font-display:swap}
|
|
@font-face{font-family:'Zen Pixel Square';src:url('/fonts/ZenPixel-Square.woff2') format('woff2');font-style:normal;font-display:swap}
|
|
|
|
:root{
|
|
--ink:#f4f4f2; --soft:#9c9c98; --faint:#6a6a66; --ground:#000; --panel:#0a0a0a;
|
|
--line:#232323; --line-hi:#3a3a3a; --accent:#f4f4f2;
|
|
--font-zen-sans:'Zen',-apple-system,BlinkMacSystemFont,"Segoe UI",Helvetica,Arial,sans-serif;
|
|
--font-zen-mono:'Zen Mono',ui-monospace,"SF Mono",SFMono-Regular,Menlo,Consolas,monospace;
|
|
--font-zen-pixel-square:'Zen Pixel Square',monospace;
|
|
--mono:var(--font-zen-mono);
|
|
--sans:var(--font-zen-sans);
|
|
/* Controls are pills and surfaces are soft, as on hanzo.ai. */
|
|
--r:999px; --r-soft:14px;
|
|
}
|
|
*{box-sizing:border-box}
|
|
html{-webkit-text-size-adjust:100%}
|
|
body{margin:0;background:var(--ground);color:var(--ink);font-family:var(--sans);
|
|
font-size:16px;line-height:1.65;-webkit-font-smoothing:antialiased}
|
|
.wrap{max-width:60rem;margin:0 auto;padding:0 1.75rem}
|
|
|
|
/* The dot field. A halftone made of one repeating radial-gradient, faded out
|
|
along the diagonal by a mask so it reads as a wave rather than wallpaper. */
|
|
/* No overflow clip here: the dot field is inset to the box already, and a clip
|
|
would cut off the copy-prompt menu. */
|
|
.hero{position:relative;border-bottom:1px solid var(--line)}
|
|
.hero::before{content:"";position:absolute;inset:0;
|
|
background-image:radial-gradient(var(--line-hi) 1px, transparent 1px);
|
|
background-size:7px 7px;
|
|
-webkit-mask-image:linear-gradient(115deg,#000 0%,rgba(0,0,0,.55) 38%,transparent 72%);
|
|
mask-image:linear-gradient(115deg,#000 0%,rgba(0,0,0,.55) 38%,transparent 72%);
|
|
pointer-events:none}
|
|
.hero .wrap{position:relative;padding-top:5.5rem;padding-bottom:4rem}
|
|
|
|
nav{border-bottom:1px solid var(--line)}
|
|
nav .in{max-width:60rem;margin:0 auto;padding:1.1rem 1.75rem;display:flex;
|
|
justify-content:space-between;align-items:center;gap:1rem}
|
|
.brand{font-family:var(--mono);font-size:.85rem;letter-spacing:.14em;
|
|
text-transform:uppercase;color:var(--ink);text-decoration:none;
|
|
display:inline-flex;align-items:center;gap:.6rem}
|
|
.brand i{font-style:normal;color:var(--faint)}
|
|
.mark{width:1.15em;height:1.15em;display:block;flex:0 0 auto}
|
|
nav a.up{font-family:var(--mono);font-size:.72rem;letter-spacing:.14em;
|
|
text-transform:uppercase;color:var(--soft);text-decoration:none}
|
|
nav a.up:hover{color:var(--ink)}
|
|
|
|
h1{font-family:var(--mono);font-size:clamp(2rem,5.4vw,3.4rem);line-height:1.12;
|
|
letter-spacing:-.02em;margin:0 0 1.1rem;font-weight:500;text-wrap:balance}
|
|
h1 em{font-style:normal;color:var(--faint)}
|
|
.lede{color:var(--soft);font-size:1.05rem;max-width:52ch;margin:0 0 2.25rem}
|
|
|
|
.btns{display:flex;flex-wrap:wrap;gap:.7rem;margin:0 0 .5rem;align-items:center}
|
|
.btn{display:inline-flex;align-items:center;gap:.5rem;font-family:var(--mono);
|
|
font-size:.72rem;letter-spacing:.14em;text-transform:uppercase;
|
|
padding:.8rem 1.4rem;border:1px solid var(--line-hi);border-radius:var(--r);
|
|
color:var(--ink);text-decoration:none;background:transparent;cursor:pointer;
|
|
line-height:1.2;font-weight:400}
|
|
.btn:hover{background:#141414;text-decoration:none;border-color:var(--soft)}
|
|
.btn.solid{background:var(--ink);color:#000;border-color:var(--ink)}
|
|
.btn.solid:hover{background:#d8d8d4}
|
|
.btn .chev{width:.7em;height:.7em;flex:0 0 auto;transition:transform .15s ease}
|
|
.btn[aria-expanded="true"] .chev{transform:rotate(180deg)}
|
|
|
|
/* Copy-prompt selector. Each row copies the prompt for its own target. */
|
|
.copy{position:relative;display:inline-flex}
|
|
.menu{position:absolute;top:calc(100% + .5rem);left:0;z-index:20;min-width:21rem;
|
|
background:var(--panel);border:1px solid var(--line-hi);border-radius:var(--r-soft);
|
|
padding:.4rem;display:flex;flex-direction:column;
|
|
box-shadow:0 12px 32px rgba(0,0,0,.75)}
|
|
.menu[hidden]{display:none}
|
|
.mi{display:flex;align-items:center;gap:.7rem;width:100%;text-align:left;
|
|
padding:.62rem .8rem;border:0;border-radius:calc(var(--r-soft) - .35rem);
|
|
background:transparent;color:var(--ink);font-family:var(--mono);font-size:.76rem;
|
|
letter-spacing:.04em;text-decoration:none;cursor:pointer;line-height:1.3;
|
|
white-space:nowrap}
|
|
.mi:hover,.mi:focus-visible{background:#181818;outline:none}
|
|
.mi .mark{width:1.05rem;height:1.05rem;color:var(--soft)}
|
|
.mi:hover .mark{color:var(--ink)}
|
|
.mi u{margin-left:auto;color:var(--faint);text-decoration:none;font-size:.68rem;
|
|
letter-spacing:.1em;text-transform:uppercase}
|
|
|
|
/* Works-with bar: one scrolling row, name and mark per agent. */
|
|
.strip{border-bottom:1px solid var(--line);background:var(--panel)}
|
|
.strip .in{max-width:60rem;margin:0 auto;padding:.9rem 1.75rem;display:flex;
|
|
align-items:center;gap:1.25rem;font-family:var(--mono);font-size:.72rem;
|
|
letter-spacing:.1em;text-transform:uppercase;color:var(--faint)}
|
|
.strip b{color:var(--soft);font-weight:400;flex:0 0 auto;
|
|
border-right:1px solid var(--line);padding-right:1.25rem;align-self:stretch;
|
|
display:flex;align-items:center}
|
|
.bar{display:flex;align-items:center;gap:.5rem;flex:1 1 auto;min-width:0;
|
|
overflow-x:auto;scrollbar-width:none;scroll-snap-type:x proximity;
|
|
-webkit-overflow-scrolling:touch;
|
|
-webkit-mask-image:linear-gradient(90deg,#000 calc(100% - 3rem),transparent);
|
|
mask-image:linear-gradient(90deg,#000 calc(100% - 3rem),transparent)}
|
|
.bar::-webkit-scrollbar{display:none}
|
|
.agent{display:inline-flex;align-items:center;gap:.45rem;flex:0 0 auto;
|
|
padding:.4rem .85rem;border:1px solid var(--line);border-radius:var(--r);
|
|
color:var(--soft);white-space:nowrap;scroll-snap-align:start;
|
|
text-decoration:none;background:var(--ground)}
|
|
.agent:hover{color:var(--ink);border-color:var(--line-hi);text-decoration:none}
|
|
.agent .mark{width:.95rem;height:.95rem}
|
|
|
|
h2{font-family:var(--mono);font-size:1.1rem;letter-spacing:-.01em;margin:2.75rem 0 .75rem;font-weight:500}
|
|
h3{font-size:1rem;margin:2rem 0 .5rem;font-weight:600}
|
|
h4,h5,h6{font-size:.94rem;margin:1.5rem 0 .5rem;font-weight:600}
|
|
p{margin:0 0 1rem;max-width:70ch}
|
|
ul,ol{margin:0 0 1rem;padding-left:1.35rem;max-width:70ch}
|
|
li{margin:.22rem 0}
|
|
a{color:var(--ink);text-decoration:underline;text-underline-offset:3px;
|
|
text-decoration-color:var(--line-hi)}
|
|
a:hover{text-decoration-color:var(--ink)}
|
|
a:focus-visible{outline:1px solid var(--ink);outline-offset:3px}
|
|
code{font-family:var(--mono);font-size:.85em;background:var(--panel);
|
|
border:1px solid var(--line);border-radius:5px;padding:.1em .36em}
|
|
pre{background:var(--panel);border:1px solid var(--line);padding:1rem 1.1rem;
|
|
border-radius:var(--r-soft);overflow-x:auto;margin:0 0 1.25rem}
|
|
pre code{background:none;border:0;padding:0;font-size:.82rem;line-height:1.7}
|
|
blockquote{margin:0 0 1rem;padding-left:1rem;border-left:1px solid var(--line-hi);color:var(--soft)}
|
|
hr{border:0;border-top:1px solid var(--line);margin:2.5rem 0}
|
|
table{border-collapse:collapse;width:100%;margin:0 0 1.25rem;font-size:.92rem;
|
|
display:block;overflow-x:auto}
|
|
th,td{border:1px solid var(--line);padding:.5rem .7rem;text-align:left}
|
|
|
|
.curl{display:block;background:var(--panel);border:1px solid var(--line);
|
|
border-radius:var(--r-soft);padding:.95rem 1.1rem;font-family:var(--mono);
|
|
font-size:.8rem;color:var(--soft);overflow-x:auto;margin:0 0 2.5rem;
|
|
white-space:pre;line-height:1.9}
|
|
.curl b{color:var(--ink);font-weight:400}
|
|
.curl i{font-style:normal;color:var(--faint)}
|
|
|
|
/* index */
|
|
#q{width:100%;padding:.85rem 1.2rem;font-family:var(--mono);font-size:.85rem;color:var(--ink);
|
|
background:var(--panel);border:1px solid var(--line);border-radius:var(--r);margin:0 0 .6rem}
|
|
#q::placeholder{color:var(--faint);text-transform:uppercase;letter-spacing:.1em;font-size:.72rem}
|
|
#q:focus{outline:none;border-color:var(--soft)}
|
|
#count{color:var(--faint);font-family:var(--mono);font-size:.72rem;letter-spacing:.1em;
|
|
text-transform:uppercase;margin:0 0 2rem}
|
|
.cat{margin:2.5rem 0 .5rem;font-family:var(--mono);font-size:.7rem;letter-spacing:.16em;
|
|
text-transform:uppercase;color:var(--faint);border-bottom:1px solid var(--line);
|
|
padding-bottom:.45rem}
|
|
.row{display:block;padding:.6rem .9rem;margin:0 -.9rem;border-bottom:1px solid var(--line);
|
|
border-radius:var(--r-soft);text-decoration:none}
|
|
.row:hover{background:#0a0a0a}
|
|
.nm{font-family:var(--mono);font-size:.84rem;color:var(--ink);display:block}
|
|
.ds{color:var(--soft);font-size:.9rem;display:block;margin-top:.12rem}
|
|
footer{margin-top:4rem;padding:2rem 0 4rem;border-top:1px solid var(--line);
|
|
color:var(--faint);font-size:.85rem}
|
|
footer a{color:var(--soft)}
|
|
.fmap{display:grid;gap:1.75rem;grid-template-columns:repeat(auto-fit,minmax(11rem,1fr));
|
|
margin:0 0 2rem;font-size:.85rem}
|
|
.fmap b{display:block;font-family:var(--mono);font-size:.68rem;letter-spacing:.16em;
|
|
text-transform:uppercase;color:var(--faint);font-weight:400;margin:0 0 .6rem;
|
|
border-bottom:1px solid var(--line);padding-bottom:.4rem}
|
|
.fmap a{display:inline-block;color:var(--ink);text-decoration:none;margin:.3rem .35rem 0 0}
|
|
.fmap a:hover{text-decoration:underline}
|
|
.fmap div{color:var(--faint);line-height:1.5}
|
|
.fend{border-top:1px solid var(--line);padding-top:1.25rem;font-size:.82rem}
|
|
|
|
@media (max-width:640px){
|
|
.hero .wrap{padding-top:3.25rem;padding-bottom:2.75rem}
|
|
.btns{gap:.5rem}
|
|
.btn{padding:.75rem 1.15rem;font-size:.68rem}
|
|
.copy,.copy .btn{width:100%;justify-content:center}
|
|
.menu{left:0;right:0;min-width:0}
|
|
.strip .in{flex-direction:column;align-items:flex-start;gap:.6rem;padding:.9rem 1.75rem}
|
|
.strip b{border-right:0;padding-right:0}
|
|
.bar{width:100%}
|
|
}
|
|
</style>
|
|
<link rel="alternate" type="text/markdown" href="/hip-0129.md">
|
|
|
|
<link rel="icon" href="/favicon.svg">
|
|
<link rel="canonical" href="https://hanzoskills.com/hip-0129">
|
|
</head>
|
|
<!-- Reading this with a machine? Every page here is also plain markdown.
|
|
This one: https://hanzoskills.com/hip-0129.md
|
|
Index: https://hanzoskills.com/llms.txt
|
|
Corpus: https://hanzoskills.com/all.md
|
|
Append .md to any path for the markdown twin. -->
|
|
<body>
|
|
<nav><div class="in">
|
|
<a class="brand" href="/"><svg class="mark" viewBox="0 0 67 67" fill="currentColor" aria-hidden="true"><path d="M22.21 67V44.6369H0V67H22.21Z"/><path d="M66.7038 22.3184H22.2534L0.0878906 44.6367H44.4634L66.7038 22.3184Z"/><path d="M22.21 0H0V22.3184H22.21V0Z"/><path d="M66.7198 0H44.5098V22.3184H66.7198V0Z"/><path d="M66.7198 67V44.6369H44.5098V67H66.7198Z"/></svg> HANZO<i>SKILLS</i></a>
|
|
<a class="up" href="/">All skills</a>
|
|
</div></nav>
|
|
|
|
<div class="wrap">
|
|
<h1>hip-0129</h1><p class="lede">HIP-129: Eval — A Score Over Model Output. Status Final. Hanzo architectural specification.</p><h1>HIP-0129: Eval — A Score Over Model Output</h1>
|
|
<h2>Abstract</h2>
|
|
<p><code>hanzoai/eval</code> is the third plane of the Hanzo AI surface and its own subsystem repo. Three planes, orthogonal:</p>
|
|
<p>| Plane | Question | Direction | Owns | |-------|----------|-----------|------| | <strong>o11y</strong> | what happened | passive, derived from events | spans, logs, metrics — <strong>read only</strong> | | <strong>ai</strong> | do the thing | inference | models, completions, embeddings | | <strong>eval</strong> | was it good | judgment | dataset, judge, rubric, experiment, run, score, queue |</p>
|
|
<p>eval <strong>consumes</strong> o11y and <strong>calls</strong> ai. Neither depends on eval, and neither may ever. That single-direction edge is the whole justification: judgment is a different concern from observation and from inference, so it is a different plane, with its own repo, its own store, its own routes, and its own name. Repo hygiene is not the argument — an acyclic dependency graph is.</p>
|
|
<p>The plane exists today as a cloud subsystem, not yet as its own repo: <code>apps/eval</code> in <code>hanzoai/cloud</code> serves <code>/v1/eval/<em></code> (the manifest row is <code>manifest/apps.go:408</code>, the group <code>apps/eval/eval.go:207</code>); o11y serves a <strong>write</strong> endpoint, <code>POST /v1/o11y/llm/annotation</code> (<code>pkg/apiserver/o11yapiserver/llmobs.go</code>), inside a read plane; <code>apps/o11y</code> serves a </em>second*, unrelated annotation-queue entity at <code>/v1/o11y/reviews</code> on its own SQLite file (<code>annotation_store.go</code>, pool <code>o11y_annotations</code>). One resource, three owners, so none of them kept it in sight of the others — which is why the console's queue module and the span plane's annotation write name the same idea at two addresses, and neither side is wrong. This HIP does not patch that mismatch. It moves the resource to the plane that owns it, after which the mismatch is not expressible.</p>
|
|
<h2>Motivation</h2>
|
|
<p>A score is not an observation. An observation is a fact the system emitted about itself; a score is a <em>verdict</em> a judge — a model or a human — pronounced about that fact. Storing verdicts in the observation plane forces that plane to accept writes, and a plane that accepts writes is no longer derivable from events. That is how o11y grew a <code>POST</code>, a mutable <code>llm_scores</code> table with a <code>DELETE</code>, and a free-text <code>queue</code> column naming a queue entity that lives in a different repo.</p>
|
|
<p>Judgment also costs money — a judge run is inference — so the plane that issues verdicts must be the plane that is billed, on the one payer path, or the cost lands under the wrong owner. And judgment must be observable <em>by the plane it observes</em>: an eval run that emits no spans is a black box asking to be trusted.</p>
|
|
<h2>Specification</h2>
|
|
<p>The key words MUST, MUST NOT, SHOULD, SHOULD NOT, and MAY are to be interpreted as in RFC 2119.</p>
|
|
<h3>§1 The plane, and the direction of the edge</h3>
|
|
<p><code>hanzoai/eval</code> MUST depend on o11y and ai only through their public surfaces, and o11y and ai MUST NOT depend on eval — not by import, not by module requirement, not by a table one of them reads. A build of either that resolves <code>hanzoai/eval</code> in <code>go list -deps</code> is a defect, not a convenience.</p>
|
|
<p>Consequences, stated so they are not re-argued:</p>
|
|
<ul><li>Anything eval takes from o11y or ai is a <strong>read</strong>. eval MUST NOT write to a table</li></ul>
|
|
<p> another plane owns — in particular <code>hanzo.cloud_usage</code>, which is ai's metering warehouse (HIP-0106) and never a second observation store.</p>
|
|
<ul><li>Anything that only <em>aggregates</em> what already happened is observability, not</li></ul>
|
|
<p> judgment, and MUST NOT live on this plane. <code>GET /v1/eval/metrics</code> — which reads <code>hanzo.cloud_usage</code> and the o11y span index and writes nothing — is an o11y board mis-homed on eval; it moves to o11y and does not reappear under <code>/v1/eval</code>. It is still served today (<code>apps/eval/metrics.go</code>, in the published subset), so the move is owed, not done.</p>
|
|
<ul><li>eval is the only plane that stores verdicts. There is exactly one score store.</li></ul>
|
|
<h3>§2 What eval owns</h3>
|
|
<p>| Resource | It is | Store | |----------|-------|-------| | <strong>dataset</strong> | a named set of inputs, with items | metastore | | <strong>item</strong> (of a dataset) | one input, identified within its dataset | metastore | | <strong>judge</strong> | a named evaluation method — <code>kind: model</code> (LLM-as-judge) or <code>kind: code</code> | metastore | | <strong>rubric</strong> | the definition a verdict is produced against and validated against: numeric with bounds, categorical with a label set, or boolean | metastore | | <strong>experiment</strong> | a named comparison that groups runs | metastore | | <strong>run</strong> | one execution of a judge over a dataset, with its per-item results | metastore (rollup) + OLAP (items) | | <strong>score</strong> | one verdict — value, rubric, source (<code>model</code> \| <code>code</code> \| <code>human</code>), optional comment | OLAP, append-only | | <strong>queue</strong> | a queue of items awaiting <strong>human</strong> judgment, with items and assignment | metastore |</p>
|
|
<p>Two renames, both from first principles, both narrowing a name to what it is:</p>
|
|
<ul><li><code>evaluator</code> → <strong>judge</strong>. It judges. <code>ai/object/eval_judge.go</code> already calls the</li></ul>
|
|
<p> type <code>JudgeRubric</code> and the entry point <code>RunJudge</code>; the resource takes the name the code already uses.</p>
|
|
<ul><li><code>score-config</code> → <strong>rubric</strong>. It is the criteria and the admissible range of a</li></ul>
|
|
<p> verdict, which is what a rubric is, and it is one word rather than two.</p>
|
|
<p>One deletion. <strong>There is no <code>annotation</code> resource.</strong> What <code>POST /api/annotation</code> wrote was a verdict with an author and no rubric — a score, missing its definition. A human's output on this plane is a <strong>score with <code>source: human</code></strong>, validated against the same rubric a model judge is validated against. The queue is what routes work to the human; the score is what the human produces. Deleting the noun is what makes the three-way seam unrepresentable rather than merely fixed.</p>
|
|
<h3>§3 Route surface — one prefix, <code>/v1/eval</code>, singular throughout</h3>
|
|
<p>What is served today is the <strong>plural</strong> surface, and any reader of the table below should know the distance: <code>/v1/eval/{datasets, datasets/{name}, datasets/{name}/items, evaluators, rubrics, runs, scores, traces, metrics}</code> — nine paths, in <code>plugin/eval/openapi.json</code>, with the resource still named <code>evaluators</code> rather than <code>judge</code> and no queue routes at all. The table below is the target grammar this HIP proposes; until the migration lands, the capability's name and address are <code>eval</code> (<code>manifest/apps.go:408</code>).</p>
|
|
<p>Every eval endpoint MUST live under <code>/v1/eval</code> and MUST follow the resource-name grammar (HIP-0119 §2): the resource is named <strong>once</strong>, in the singular, and the HTTP method carries collection-versus-item. <code>POST /v1/eval/dataset</code> creates one; <code>GET /v1/eval/dataset</code> lists them; <code>GET /v1/eval/dataset/{name}</code> reads one.</p>
|
|
<p>Nesting rule: a child is nested <strong>only when its identifier is scoped by its parent</strong> (a dataset item's id means nothing without its dataset). A resource with its own id, unique within the org, is flat and filtered by query parameter. Compound segments (<code>dataset-items</code>, <code>score-configs</code>, <code>annotation-queues</code>) are the shape this rule replaces, and MUST NOT appear.</p>
|
|
<p>| Method | Path | Notes | |--------|------|-------| | POST · GET | <code>/v1/eval/dataset</code> | create · list | | GET · DELETE | <code>/v1/eval/dataset/{name}</code> | read (with item count) · delete (with its items, one tx) | | POST · GET | <code>/v1/eval/dataset/{name}/item</code> | create · list | | GET · PATCH · DELETE | <code>/v1/eval/dataset/{name}/item/{id}</code> | an item MUST NOT change dataset | | POST · GET | <code>/v1/eval/judge</code> | create · list | | GET · PATCH | <code>/v1/eval/judge/{name}</code> | | | POST · GET | <code>/v1/eval/rubric</code> | create · list | | GET · PATCH | <code>/v1/eval/rubric/{name}</code> | | | POST · GET | <code>/v1/eval/experiment</code> | create · list | | GET | <code>/v1/eval/experiment/{id}</code> | | | POST · GET | <code>/v1/eval/run</code> | start a run · list, <code>?experiment=</code> <code>?dataset=</code> | | GET | <code>/v1/eval/run/{id}</code> | rollup | | GET | <code>/v1/eval/run/{id}/item</code> | per-item results | | POST · GET | <code>/v1/eval/score</code> | record a verdict · list, <code>?run=</code> <code>?item=</code> <code>?trace=</code> | | GET | <code>/v1/eval/score/{id}</code> | | | POST · GET | <code>/v1/eval/queue</code> | create · list | | GET · PATCH · DELETE | <code>/v1/eval/queue/{id}</code> | | | POST · GET | <code>/v1/eval/queue/{id}/item</code> | enqueue · list | | GET · PATCH | <code>/v1/eval/queue/{id}/item/{itemId}</code> | | | PUT · DELETE | <code>/v1/eval/queue/{id}/item/{itemId}/assignment</code> | claim · release | | GET | <code>/v1/eval/health</code> | eval's own probe, §5 |</p>
|
|
<p>A score MUST NOT be updated or deleted. A verdict is the record of a judgment that was made; a retraction is a <strong>new score</strong>, not an edit of the old one. The OLAP engine enforces this by construction, and no route may work around it — which is precisely why <code>DELETE /v1/o11y/llm/score/{id}</code> does not survive the move.</p>
|
|
<p>Static sub-routes MUST be registered before their <code>{param}</code> siblings, so a real dataset name can never shadow a collection route.</p>
|
|
<h3>§4 Annotation moves off o11y</h3>
|
|
<p>The following are <strong>removed</strong>, not aliased:</p>
|
|
<ul><li>o11y: <code>GET</code>/<code>POST /v1/o11y/llm/annotation</code></li></ul>
|
|
<p> (<code>pkg/apiserver/o11yapiserver/llmobs.go</code>) and the <code>llm_annotations</code> table.</p>
|
|
<ul><li>o11y: <code>GET</code>/<code>POST /v1/o11y/llm/scores</code>, <code>GET</code>/<code>DELETE /v1/o11y/llm/score/{id}</code></li></ul>
|
|
<p> and <code>llm_scores</code>. Score is a verdict; verdicts live on one plane, and it is not the read plane.</p>
|
|
<ul><li>cloud: <code>apps/o11y/annotation_queues.go</code>, its eight</li></ul>
|
|
<p> <code>/v1/o11y/reviews*</code> routes and its <code>o11y_annotations</code> metastore (<code>apps/o11y/annotation_store.go</code>).</p>
|
|
<p>o11y keeps exactly what it is: the passive span, log and metric views (<code>/v1/o11y/observations</code>, <code>/traces</code>, <code>/sessions</code>, <code>/users</code>), which read and never write. A <code>POST</code> in a read plane is the defect; deleting it is the fix. Because the queue and the verdict now have one owner under one prefix, the console's <code>/v1/o11y/reviews</code> call against a server registering <code>/v1/o11y/llm/annotation</code> cannot be written at all — there is no second spelling left to disagree with. A correct boundary does not need two sides kept in sync; it makes the disagreement unrepresentable.</p>
|
|
<h3>§5 The subsystem contract</h3>
|
|
<p>eval is a Hanzo service (HIP-0119) and a cloud subsystem (HIP-0106). It MUST be:</p>
|
|
<p><strong>ZAP-native.</strong> Routes are registered on <code>*zip.App</code> (HIP-0122); transport is HTTP + ZAP (HIP-0114/0120). No gRPC — not on the wire, not in the module graph. No <code>/api/</code> prefix anywhere, per HIP-0119 §2/§9.</p>
|
|
<p><strong>Mounted as data, with no dependency cycle.</strong> <code>hanzoai/eval</code> exports <code>Mount(app <em>zip.App, deps eval.Deps) error</code> and <code>Shutdown(ctx context.Context) error</code>, where <code>eval.Deps</code> is a small set of <strong>interfaces eval declares itself</strong> — logger, data directory, durability, an inference seam, a datastore seam, a meter seam. <code>hanzoai/eval</code> MUST NOT import <code>hanzoai/cloud</code>. This is deliberate, and it is not how o11y does it: o11y's <code>Mount</code> takes <code>cloud.Deps</code>, so <code>o11y/go.mod</code> requires <code>hanzoai/cloud</code> </em>and* carries <code>replace … => ../cloud</code>, which makes the pinned version and the CI checkout ref one fact in two places. A new plane MUST NOT inherit that tax.</p>
|
|
<p>Cloud names eval in exactly two places: a thin adapter in <code>cloud/clients/eval</code> that fills <code>eval.Deps</code> from <code>cloud.Deps</code>, and one line in <code>apps.Wire()</code>:</p>
|
|
<pre><code class="lang-go">{Name: "eval", Mount: eval.Mount, Shutdown: eval.Shutdown, OwnsHealth: true},</code></pre>
|
|
<p>Slice position IS mount order (<code>cloud.MountSpec</code> has no order field); <code>MountAll</code> registers <code>Shutdown</code> via <code>app.OnShutdown</code>, which zip drains LIFO after in-flight requests. Wiring <code>Shutdown</code> is REQUIRED: today's <code>eval.Shutdown</code> exists and is never wired, so the metastore handle is not closed on SIGTERM. <code>OwnsHealth: true</code> suppresses the generic always-ok route; <code>GET /v1/eval/health</code> MUST return <code>200</code> when the metastore is open and report OLAP availability in its body. It MUST NOT return <code>503</code> merely because the OLAP store is absent — that degradation is per-route (§9).</p>
|
|
<p><strong>Org-keyed and fail-closed.</strong> Org is the only tenant key, on every read and every write. Every handler MUST resolve the tenant <em>first</em>, via <code>principal.Validated(c)</code> then <code>principal.Org(c)</code>, and MUST <code>403</code> when either fails. A handler MUST NOT read <code>X-Org-Id</code> (or any identity header) directly: <code>SanitizeIdentity</code> strips and re-mints those headers, but it restores a client-supplied <code>X-Org-Id</code> on the bearer-less path for data scoping — so an org without a validated principal is forgeable, and only the <code>Validated</code> gate closes it.</p>
|
|
<p>The org string MUST be used <strong>verbatim</strong>: never lowercased, never trimmed, never folded. Normalizing collapses distinct owners into one storage bucket and puts the authorization check and the storage key on different values. Metastore tables MUST carry a NOT NULL <code>org</code> column with a composite <code>(org, id)</code> primary key — a global id primary key is a cross-tenant existence oracle. OLAP tables MUST order by <code>org</code> first, so a tenant read is a prefix scan. <code>principal.ProjectScope(c)</code> narrows within an org; it never widens. <code>principal.IsSuperAdmin(c)</code> MAY drop the org predicate only on an explicitly fleet-wide read, and §3 defines none.</p>
|
|
<p><strong>Metered on the one payer path.</strong> Judge inference is inference: it MUST be issued through the ai seam <strong>carrying the caller's identity</strong>, so ai's own path meters it into <code>hanzo.cloud_usage</code> exactly once. eval MUST NOT write <code>cloud_usage</code>, MUST NOT construct a second <code>metering.Client</code>, and MUST NOT re-derive who pays. Data scope reads <code>principal.Org</code>; money reads <code>principal.HomeOrg</code> / <code>principal.WalletOf</code>, whose answer comes from <code>hanzoai/account.Payer</code> and nowhere else — the gate and the debit must address the same wallet, a split that has already shipped wrong twice.</p>
|
|
<p>Where eval charges for its own resources, it uses the one in-handler sequence:</p>
|
|
<pre><code class="lang-go">fee := cloud.ResourceFeeCents("EVAL", kind)
|
|
_, projectValidated := principal.ValidatedProject(c)
|
|
if err := bill.Gate(c.Context(), principal.HomeOrg(c), project, projectValidated, kind, fee); err != nil {
|
|
return cloud.DenyResource(c, err)
|
|
}
|
|
// … do the work …
|
|
bill.Meter(principal.HomeOrg(c), project, kind, fee, c.RequestID(), cloud.ClientIP(c))</code></pre>
|
|
<p><code>bill</code> is the <code>*cloud.ResourceMeter</code> that <code>cloud.NewBase(deps, "eval")</code> supplies, with provider <code>eval</code>. A run MUST be gated <strong>before</strong> the first judge call and MUST NOT write a score when the gate denies.</p>
|
|
<p>What is priced today: the surface itself is <strong>free</strong> — the plugin declares <code>Price: cloud.Free</code> (<code>plugin/eval/main.go:21</code>) and <code>ResourceFeeCents("EVAL", …)</code> is unset, so every eval resource costs nothing in those words. The only money on this plane is the judge's inference, metered by ai on the caller's identity as above. The capability publishes <strong>no events</strong> on the bus, so a customer's webhooks receive nothing from it. Its stage is <strong>ga</strong> (HIP-0139 §8): the manifest row declares no stage. It forks, embeds or mirrors <strong>no OSS upstream</strong> — its OLAP tables ride the shared datastore client another capability owns (<code>apps/eval/telemetry.go</code>), which is that capability's upstream to declare, not this one's.</p>
|
|
<p><strong>Observable by the plane it observes.</strong> eval MUST emit spans from the <strong>global</strong> tracer provider (<code>otel.Tracer("hanzo-eval")</code>), parented from <code>c.Context()</code> — cloud installs the one provider before mounting subsystems and calls <code>aiobject.AdoptHostTracerProvider()</code> so ai's <code>gen_ai</code> spans share it. eval MUST NOT construct a provider or an exporter of its own.</p>
|
|
<p>Today <code>apps/eval</code> constructs no tracer and emits no span of its own (there is no otel import in the package), so what a customer can read back under <code>/v1/o11y</code> from this capability is the request span every route already gets and nothing more. The mandate below is the unimplemented half of this section.</p>
|
|
<p>Run, item and judge spans MUST carry <code>gen_ai.hanzo.org_id</code> set to the caller's org — o11y's llmobs views hard-filter on it, and a span without it is invisible. eval adds <code>eval.run.id</code>, <code>eval.dataset</code>, <code>eval.item.id</code>, <code>eval.judge</code>, <code>eval.rubric</code>, <code>eval.score.name</code>, <code>eval.score.value</code>, <code>eval.score.source</code>. Cost attributes (<code>_o11y.gen_ai.*</code>) belong to ai and MUST NOT be re-emitted by eval — one cost vocabulary, emitted by the plane that knows the price.</p>
|
|
<h3>§6 Storage</h3>
|
|
<p>Two stores, one seam each.</p>
|
|
<p><strong>Metastore</strong> — encrypted SQLite (<code>cek</code>) under <code>DataDir</code>, opened through <code>cloud.OrgDB</code> / <code>cloud.NewOrgStore</code> with <code>cloud.WithDurable</code> so it survives a rolling deploy on the HA plane (HIP-0107). It holds dataset, item, judge, rubric, experiment, run rollup, queue, queue item and assignment. Natural keys are unique <strong>within an org</strong> (<code>(org, name)</code>), never globally. Today's implementation is one shared file with a NOT NULL <code>org</code> column as the trust boundary (<code>apps/eval/store.go</code>), not a file per org — the per-org-file question is named in Open questions and this paragraph is its target state.</p>
|
|
<p><strong>OLAP</strong> — scores and per-item run results, over the <strong>shared</strong> datastore client (<code>aiobject.DatastoreEnabled</code>, <code>DatastoreExec</code>, <code>DatastoreQuery</code>). eval MUST NOT open its own datastore connection: the pool, the retries, and the KMS-injected credentials live in exactly one place. Tables are append-only, partitioned by month, ordered by org first; every predicate is a bound parameter, never interpolated; and every list route is bounded by a server-side limit.</p>
|
|
<p>SQL identifiers are not URL paths. The grammar of §3 governs the route surface; the existing table names (<code>hanzo.eval_traces</code>, <code>hanzo.eval_scores</code>) are data at rest and are NOT renamed by this HIP — renaming them buys nothing and costs a migration.</p>
|
|
<h3>§7 Bounds instead of a queue</h3>
|
|
<p>eval bounds work rather than queueing it: a per-org concurrent-run cap that returns <code>429</code> immediately (runs are never queued — the only queue on this plane is the human one), a wall-clock run deadline, a per-run item cap, and content and comment size caps. Fail fast and say so; a silent backlog is a worse answer than a refusal.</p>
|
|
<h3>§8 Judge</h3>
|
|
<p>A judge is <code>kind: model</code> or <code>kind: code</code>.</p>
|
|
<p>A <strong>model</strong> judge MUST validate its verdict against the named rubric before the score is recorded, and MUST fail the item when the reply cannot be parsed against that rubric. It MUST NOT default an unparseable verdict to a value — a fabricated zero is worse than a recorded failure. Untrusted content (the item, the model's output) MUST be delimiter-fenced and never concatenated into the instruction. <code>ai/object/eval_judge.go</code> (<code>JudgeRubric</code>, <code>RunJudge</code>, <code>parseVerdict</code>) is the implementation of record; eval MUST NOT ship a second, weaker judge parser beside it, which is what <code>apps/eval/runner.go</code> is today.</p>
|
|
<p>A <strong>code</strong> judge is a deterministic function of <code>(input, output, expected)</code> returning a value admissible under its rubric. Where it executes is an open question.</p>
|
|
<h3>§9 Degraded, honestly</h3>
|
|
<p>With no OLAP store configured, eval MUST serve dataset, item, judge, rubric, experiment and queue from the metastore, MUST return <code>503</code> from the score and run routes, and MUST report the degradation on <code>/v1/eval/health</code>. It MUST NOT fabricate an empty result that reads as a real one.</p>
|
|
<h3>§10 Forbidden</h3>
|
|
<ul><li>A dependency edge from o11y or ai to eval.</li><li>A write endpoint in the o11y plane; a second score store; a mutable or deletable</li></ul>
|
|
<p> score.</p>
|
|
<ul><li>An <code>annotation</code> resource, or any second spelling of the queue.</li><li>A plural, compound, or <code>/api/</code>-prefixed path segment; a second prefix aliasing</li></ul>
|
|
<p> <code>/v1/eval</code>.</p>
|
|
<ul><li>Reading an identity header directly; an org normalized before it is used as a key;</li></ul>
|
|
<p> a global-id primary key on a tenant table.</p>
|
|
<ul><li>A second billing path, a second <code>metering.Client</code>, a write to <code>cloud_usage</code>, or a</li></ul>
|
|
<p> fee charged to any org but the payer's.</p>
|
|
<ul><li>A tracer provider, an exporter, or a datastore connection of eval's own.</li><li>An aggregate-only dashboard on this plane.</li></ul>
|
|
<h2>Migration</h2>
|
|
<p>One release; the old spellings are removed, not aliased — two ways to say one thing is the defect this HIP exists to remove.</p>
|
|
<ol><li>Stand up <code>hanzoai/eval</code> with <code>Mount</code>/<code>Shutdown</code> over <code>eval.Deps</code> (no <code>cloud</code></li></ol>
|
|
<p> import), porting <code>apps/eval</code> and its ~40 tests, which are the real specification of the current behavior.</p>
|
|
<ol><li>Add the adapter and the one <code>apps.Wire()</code> line <strong>with <code>Shutdown</code> wired</strong>; delete</li></ol>
|
|
<p> <code>apps/eval</code>.</p>
|
|
<ol><li>Move the annotation queue to <code>/v1/eval/queue</code>; delete</li></ol>
|
|
<p> <code>apps/o11y/annotation_queues.go</code>, the <code>o11y_annotations</code> store, o11y's <code>/v1/o11y/llm/annotation</code> and <code>/v1/o11y/llm/score*</code> routes, <code>llm_annotations</code> and <code>llm_scores</code>.</p>
|
|
<ol><li>Move the metrics board to o11y; <code>/v1/eval/metrics</code> is never created.</li><li>Repoint the console: <code>evals/<em></code> → <code>eval/</em></code> in its API modules, its proxy allow</li></ol>
|
|
<p> list, and the tests that pin the literal paths.</p>
|
|
<ol><li>Resolve <code>clients/admin</code>'s direct read of <code>hanzo.eval_traces</code> and</li></ol>
|
|
<p> <code>hanzo.eval_scores</code> (see Open questions) before eval's tables are touched.</p>
|
|
<h2>Acceptance tests</h2>
|
|
<p>A conformant implementation passes all of these. (1) and (4) are where this breaks silently, so they are not optional.</p>
|
|
<ol><li><strong>Cross-tenant isolation.</strong> For every route in §3: a validated principal of org B</li></ol>
|
|
<p> presenting an id created by org A gets <code>404</code>/<code>403</code> and never <code>200</code>, on reads, writes and deletes alike. A list issued by B never contains a row of A's.</p>
|
|
<ol><li><strong>Fail-closed tenancy.</strong> Every route, called with <code>X-Org-Id: victim</code> and <strong>no</strong></li></ol>
|
|
<p> bearer, returns <code>403</code> — the header alone never scopes anything.</p>
|
|
<ol><li><strong>Org verbatim.</strong> Orgs differing only by case or surrounding whitespace are</li></ol>
|
|
<p> distinct tenants and never share a row, a file or a natural key.</p>
|
|
<ol><li><strong>A judge run bills the correct org.</strong> A run issued by a principal of org A whose</li></ol>
|
|
<p> home org is A produces exactly one <code>cloud_usage</code> attribution per judge call, addressed to <code>account.Payer</code> for A; when the effective org differs from the home org, the debit follows the <strong>home</strong> org and the data follows the effective one. eval itself writes zero <code>cloud_usage</code> rows.</p>
|
|
<ol><li><strong>Gate precedes work.</strong> A spend-capped org gets <code>402</code> from <code>POST /v1/eval/run</code></li></ol>
|
|
<p> before any judge call is issued, and no score, run rollup or item row is written.</p>
|
|
<ol><li><strong>Score immutability.</strong> No route mutates or deletes a score; a retraction is a new</li></ol>
|
|
<p> row and the original remains readable.</p>
|
|
<ol><li><strong>Rubric fail-closed.</strong> A score violating its rubric's type, bounds or label set is</li></ol>
|
|
<p> rejected; an unparseable model verdict fails the item and records no value.</p>
|
|
<ol><li><strong>Assignment is exclusive.</strong> Two annotators claiming one queue item: one <code>200</code>,</li></ol>
|
|
<p> one <code>409</code>; releasing returns it to the pool.</p>
|
|
<ol><li><strong>Degraded honesty.</strong> With no OLAP store: score and run routes <code>503</code>; dataset,</li></ol>
|
|
<p> item, judge, rubric, experiment and queue <code>200</code>; <code>/v1/eval/health</code> <code>200</code>, reporting the degradation.</p>
|
|
<ol><li><strong>Bounds hold.</strong> Exceeding the per-org concurrent-run cap returns <code>429</code></li></ol>
|
|
<p> immediately and enqueues nothing; the run deadline terminates a run and records it as failed.</p>
|
|
<ol><li><strong>Teardown.</strong> SIGTERM closes the metastore handle exactly once, via the wired</li></ol>
|
|
<p> <code>Shutdown</code>.</p>
|
|
<ol><li><strong>Grammar.</strong> A table-driven assertion over the registered route list: no plural</li></ol>
|
|
<p> segment, no compound segment, no <code>/api/</code>, no prefix but <code>/v1/eval</code>.</p>
|
|
<ol><li><strong>Acyclic.</strong> <code>go list -deps</code> for o11y and for ai contains no <code>hanzoai/eval</code>.</li><li><strong>Span shape.</strong> A run emits spans carrying <code>gen_ai.hanzo.org_id</code> equal to the</li></ol>
|
|
<p> caller's org, parented to the request span, and emits no <code>_o11y.gen_ai.*</code> attribute.</p>
|
|
<h2>Open questions</h2>
|
|
<p>Named, not hidden. Each needs a human call before the corresponding code is written.</p>
|
|
<ul><li><strong>Where a code judge executes.</strong> In-process pure expression, the extension runtime</li></ul>
|
|
<p> (HIP-0105/0116), or a sandbox — and whether it may make network calls at all. This is an arbitrary-code-execution decision and is deliberately unresolved here.</p>
|
|
<ul><li><strong>Standalone or embedded.</strong> Whether <code>hanzoai/eval</code> also ships a binary, an image</li></ul>
|
|
<p> and a <code>Service</code> CR (o11y's shape, with the two listeners of HIP-0119 §1), or exists only embedded in cloud. This HIP specifies the embedded contract; the standalone one is additive.</p>
|
|
<ul><li><strong>The existing rows.</strong> <code>llm_scores</code> and <code>llm_annotations</code> — migrate into eval's</li></ul>
|
|
<p> score store, or drop? Migration <code>099</code>'s <code>Down()</code> is a no-op, so whichever is chosen is one-way.</p>
|
|
<ul><li><strong><code>clients/admin</code>'s direct read</strong> of <code>hanzo.eval_traces</code> and <code>hanzo.eval_scores</code> for</li></ul>
|
|
<p> the SuperAdmin fleet board: does admin keep a documented warehouse read (eval owns the schema, admin reads it with no interface between them), or does eval expose a fleet-scoped read the board calls?</p>
|
|
<ul><li><strong>Ordering with the console.</strong> Whether the metrics board lands in o11y before or</li></ul>
|
|
<p> after the eval cut; the console reads it today either way.</p>
|
|
<ul><li><strong>Per-org file or shared file.</strong> eval's metastore is one shared <code>evals.db</code> with a</li></ul>
|
|
<p> <code>WHERE org = ?</code>; <code>cloud.OrgDB</code>/<code>NewOrgStore</code> would make the isolation a file boundary. Stronger, and a data migration.</p>
|
|
<ul><li><strong>Is project authoritative?</strong> Whether the project axis hard-enforces</li></ul>
|
|
<p> (<code>ValidatedProject</code>) or stays advisory on eval reads.</p>
|
|
<ul><li><strong>What eval charges for.</strong> <code>ResourceFeeCents("EVAL", …)</code> is unset, so everything</li></ul>
|
|
<p> but the judge's inference is free today. Whether a run, a stored dataset or a queue seat is billable — and whether eval gets its own spend-cap service axis or inherits ai's — is a pricing decision, not an engineering one.</p>
|
|
<h2>Security Considerations</h2>
|
|
<p>What an attacker gets from the wrong implementation: a score store that can be edited is the ability to rewrite verdicts after the fact — an agent judged dangerous re-judged safe with no record — which is why a score is append-only and a retraction is a new row (§3). A judge that concatenates untrusted item content into its instruction is a prompt-injection path from evaluated data into the evaluator (§8). A tenancy defect here discloses another org's datasets and verdicts and, worse, lets one org's runs bill another's wallet — the acceptance tests (1), (2) and (4) are the checks that hold those three closed.</p>
|
|
<h2>References</h2>
|
|
<ul><li>HIP-0106 Unified Cloud Binary · HIP-0111 IAM</li><li>HIP-0114 ZAP Transport · HIP-0119 Service Conventions · HIP-0120 gRPC Elimination</li><li>HIP-0122 zip — ZAP-Native Application Server · HIP-0132 One Telemetry Plane</li><li>HIP-0105/0116 Extension Runtime</li><li>HIP-0139 Capability</li></ul>
|
|
<h2>Copyright</h2>
|
|
<p>Released under CC0 1.0 Universal Public Domain Dedication.</p>
|
|
<footer>
|
|
<div class="fmap">
|
|
<div><b>Build</b>
|
|
<a href="https://hanzo.app">hanzo.app</a> app builder
|
|
<a href="https://hanzo.codes">hanzo.codes</a> code with an agent
|
|
<a href="https://hanzoskills.com">hanzoskills.com</a> what an agent reads</div>
|
|
<div><b>Run</b>
|
|
<a href="https://console.hanzo.ai">console.hanzo.ai</a> cloud console
|
|
<a href="https://api.hanzo.ai">api.hanzo.ai</a> the API</div>
|
|
<div><b>Account</b>
|
|
<a href="https://hanzo.id">hanzo.id</a> sign in
|
|
<a href="https://pay.hanzo.ai">pay.hanzo.ai</a> top up
|
|
<a href="https://billing.hanzo.ai">billing.hanzo.ai</a> invoices and usage</div>
|
|
<div><b>Company</b>
|
|
<a href="https://hanzo.ai">hanzo.ai</a> canonical
|
|
<a href="https://hanzo.ventures">hanzo.ventures</a> ventures</div>
|
|
</div>
|
|
<div class="fend"><a href="/hip-0129.md">This page as plain text</a> · <a href="/llms.txt">Index for agents</a></div>
|
|
</footer>
|
|
</div></body></html> |