Files
skills/public/schemas.html
zeekayandHanzo Dev 9e10c1c10e Each llms index points at the other two
hanzoskills.com/llms.txt now opens with the install line, the API
reference index and the company index, so an agent that lands on any
one of the three can find the rest.

robots.txt claimed 537 documents while llms.txt counts 920. The number
is gone from robots.txt; llms.txt derives it.

public/ is regenerated from scripts/site.py, which it had fallen
behind: the copy-prompt menu and agent bar were in the generator and
not in the committed pages. tests/site.mjs passes against the result.

Co-authored-by: Hanzo Dev <[email protected]>
2026-09-11 00:27:59 -07:00

554 lines
29 KiB
HTML

<!doctype html>
<html lang="en"><head>
<meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1">
<title>schemas</title>
<meta name="color-scheme" content="dark">
<style>
/* Committed to one visual world: a dark instrument panel. No light variant —
the dot field and the hairline rules only read on black, and a naive
inversion would wash both out. Every colour is painted explicitly so the page
never borrows a host background. */
@font-face{font-family:'Zen';src:url('/fonts/Zen-Variable.woff2') format('woff2');font-weight:100 900;font-style:normal;font-display:swap}
@font-face{font-family:'Zen';src:url('/fonts/Zen-Italic-Variable.woff2') format('woff2');font-weight:100 900;font-style:italic;font-display:swap}
@font-face{font-family:'Zen Mono';src:url('/fonts/ZenMono-Variable.woff2') format('woff2');font-weight:100 900;font-style:normal;font-display:swap}
@font-face{font-family:'Zen Mono';src:url('/fonts/ZenMono-Italic-Variable.woff2') format('woff2');font-weight:100 900;font-style:italic;font-display:swap}
@font-face{font-family:'Zen Pixel Square';src:url('/fonts/ZenPixel-Square.woff2') format('woff2');font-style:normal;font-display:swap}
:root{
--ink:#f4f4f2; --soft:#9c9c98; --faint:#6a6a66; --ground:#000; --panel:#0a0a0a;
--line:#232323; --line-hi:#3a3a3a; --accent:#f4f4f2;
--font-zen-sans:'Zen',-apple-system,BlinkMacSystemFont,"Segoe UI",Helvetica,Arial,sans-serif;
--font-zen-mono:'Zen Mono',ui-monospace,"SF Mono",SFMono-Regular,Menlo,Consolas,monospace;
--font-zen-pixel-square:'Zen Pixel Square',monospace;
--mono:var(--font-zen-mono);
--sans:var(--font-zen-sans);
/* Controls are pills and surfaces are soft, as on hanzo.ai. */
--r:999px; --r-soft:14px;
}
*{box-sizing:border-box}
html{-webkit-text-size-adjust:100%}
body{margin:0;background:var(--ground);color:var(--ink);font-family:var(--sans);
font-size:16px;line-height:1.65;-webkit-font-smoothing:antialiased}
.wrap{max-width:60rem;margin:0 auto;padding:0 1.75rem}
/* The dot field. A halftone made of one repeating radial-gradient, faded out
along the diagonal by a mask so it reads as a wave rather than wallpaper. */
/* No overflow clip here: the dot field is inset to the box already, and a clip
would cut off the copy-prompt menu. */
.hero{position:relative;border-bottom:1px solid var(--line)}
.hero::before{content:"";position:absolute;inset:0;
background-image:radial-gradient(var(--line-hi) 1px, transparent 1px);
background-size:7px 7px;
-webkit-mask-image:linear-gradient(115deg,#000 0%,rgba(0,0,0,.55) 38%,transparent 72%);
mask-image:linear-gradient(115deg,#000 0%,rgba(0,0,0,.55) 38%,transparent 72%);
pointer-events:none}
.hero .wrap{position:relative;padding-top:5.5rem;padding-bottom:4rem}
nav{border-bottom:1px solid var(--line)}
nav .in{max-width:60rem;margin:0 auto;padding:1.1rem 1.75rem;display:flex;
justify-content:space-between;align-items:center;gap:1rem}
.brand{font-family:var(--mono);font-size:.85rem;letter-spacing:.14em;
text-transform:uppercase;color:var(--ink);text-decoration:none;
display:inline-flex;align-items:center;gap:.6rem}
.brand i{font-style:normal;color:var(--faint)}
.mark{width:1.15em;height:1.15em;display:block;flex:0 0 auto}
nav a.up{font-family:var(--mono);font-size:.72rem;letter-spacing:.14em;
text-transform:uppercase;color:var(--soft);text-decoration:none}
nav a.up:hover{color:var(--ink)}
h1{font-family:var(--mono);font-size:clamp(2rem,5.4vw,3.4rem);line-height:1.12;
letter-spacing:-.02em;margin:0 0 1.1rem;font-weight:500;text-wrap:balance}
h1 em{font-style:normal;color:var(--faint)}
.lede{color:var(--soft);font-size:1.05rem;max-width:52ch;margin:0 0 2.25rem}
.btns{display:flex;flex-wrap:wrap;gap:.7rem;margin:0 0 .5rem;align-items:center}
.btn{display:inline-flex;align-items:center;gap:.5rem;font-family:var(--mono);
font-size:.72rem;letter-spacing:.14em;text-transform:uppercase;
padding:.8rem 1.4rem;border:1px solid var(--line-hi);border-radius:var(--r);
color:var(--ink);text-decoration:none;background:transparent;cursor:pointer;
line-height:1.2;font-weight:400}
.btn:hover{background:#141414;text-decoration:none;border-color:var(--soft)}
.btn.solid{background:var(--ink);color:#000;border-color:var(--ink)}
.btn.solid:hover{background:#d8d8d4}
.btn .chev{width:.7em;height:.7em;flex:0 0 auto;transition:transform .15s ease}
.btn[aria-expanded="true"] .chev{transform:rotate(180deg)}
/* Copy-prompt selector. Each row copies the prompt for its own target. */
.copy{position:relative;display:inline-flex}
.menu{position:absolute;top:calc(100% + .5rem);left:0;z-index:20;min-width:21rem;
background:var(--panel);border:1px solid var(--line-hi);border-radius:var(--r-soft);
padding:.4rem;display:flex;flex-direction:column;
box-shadow:0 12px 32px rgba(0,0,0,.75)}
.menu[hidden]{display:none}
.mi{display:flex;align-items:center;gap:.7rem;width:100%;text-align:left;
padding:.62rem .8rem;border:0;border-radius:calc(var(--r-soft) - .35rem);
background:transparent;color:var(--ink);font-family:var(--mono);font-size:.76rem;
letter-spacing:.04em;text-decoration:none;cursor:pointer;line-height:1.3;
white-space:nowrap}
.mi:hover,.mi:focus-visible{background:#181818;outline:none}
.mi .mark{width:1.05rem;height:1.05rem;color:var(--soft)}
.mi:hover .mark{color:var(--ink)}
.mi u{margin-left:auto;color:var(--faint);text-decoration:none;font-size:.68rem;
letter-spacing:.1em;text-transform:uppercase}
/* Works-with bar: one scrolling row, name and mark per agent. */
.strip{border-bottom:1px solid var(--line);background:var(--panel)}
.strip .in{max-width:60rem;margin:0 auto;padding:.9rem 1.75rem;display:flex;
align-items:center;gap:1.25rem;font-family:var(--mono);font-size:.72rem;
letter-spacing:.1em;text-transform:uppercase;color:var(--faint)}
.strip b{color:var(--soft);font-weight:400;flex:0 0 auto;
border-right:1px solid var(--line);padding-right:1.25rem;align-self:stretch;
display:flex;align-items:center}
.bar{display:flex;align-items:center;gap:.5rem;flex:1 1 auto;min-width:0;
overflow-x:auto;scrollbar-width:none;scroll-snap-type:x proximity;
-webkit-overflow-scrolling:touch;
-webkit-mask-image:linear-gradient(90deg,#000 calc(100% - 3rem),transparent);
mask-image:linear-gradient(90deg,#000 calc(100% - 3rem),transparent)}
.bar::-webkit-scrollbar{display:none}
.agent{display:inline-flex;align-items:center;gap:.45rem;flex:0 0 auto;
padding:.4rem .85rem;border:1px solid var(--line);border-radius:var(--r);
color:var(--soft);white-space:nowrap;scroll-snap-align:start;
text-decoration:none;background:var(--ground)}
.agent:hover{color:var(--ink);border-color:var(--line-hi);text-decoration:none}
.agent .mark{width:.95rem;height:.95rem}
h2{font-family:var(--mono);font-size:1.1rem;letter-spacing:-.01em;margin:2.75rem 0 .75rem;font-weight:500}
h3{font-size:1rem;margin:2rem 0 .5rem;font-weight:600}
h4,h5,h6{font-size:.94rem;margin:1.5rem 0 .5rem;font-weight:600}
p{margin:0 0 1rem;max-width:70ch}
ul,ol{margin:0 0 1rem;padding-left:1.35rem;max-width:70ch}
li{margin:.22rem 0}
a{color:var(--ink);text-decoration:underline;text-underline-offset:3px;
text-decoration-color:var(--line-hi)}
a:hover{text-decoration-color:var(--ink)}
a:focus-visible{outline:1px solid var(--ink);outline-offset:3px}
code{font-family:var(--mono);font-size:.85em;background:var(--panel);
border:1px solid var(--line);border-radius:5px;padding:.1em .36em}
pre{background:var(--panel);border:1px solid var(--line);padding:1rem 1.1rem;
border-radius:var(--r-soft);overflow-x:auto;margin:0 0 1.25rem}
pre code{background:none;border:0;padding:0;font-size:.82rem;line-height:1.7}
blockquote{margin:0 0 1rem;padding-left:1rem;border-left:1px solid var(--line-hi);color:var(--soft)}
hr{border:0;border-top:1px solid var(--line);margin:2.5rem 0}
table{border-collapse:collapse;width:100%;margin:0 0 1.25rem;font-size:.92rem;
display:block;overflow-x:auto}
th,td{border:1px solid var(--line);padding:.5rem .7rem;text-align:left}
.curl{display:block;background:var(--panel);border:1px solid var(--line);
border-radius:var(--r-soft);padding:.95rem 1.1rem;font-family:var(--mono);
font-size:.8rem;color:var(--soft);overflow-x:auto;margin:0 0 2.5rem;
white-space:pre;line-height:1.9}
.curl b{color:var(--ink);font-weight:400}
.curl i{font-style:normal;color:var(--faint)}
/* index */
#q{width:100%;padding:.85rem 1.2rem;font-family:var(--mono);font-size:.85rem;color:var(--ink);
background:var(--panel);border:1px solid var(--line);border-radius:var(--r);margin:0 0 .6rem}
#q::placeholder{color:var(--faint);text-transform:uppercase;letter-spacing:.1em;font-size:.72rem}
#q:focus{outline:none;border-color:var(--soft)}
#count{color:var(--faint);font-family:var(--mono);font-size:.72rem;letter-spacing:.1em;
text-transform:uppercase;margin:0 0 2rem}
.cat{margin:2.5rem 0 .5rem;font-family:var(--mono);font-size:.7rem;letter-spacing:.16em;
text-transform:uppercase;color:var(--faint);border-bottom:1px solid var(--line);
padding-bottom:.45rem}
.row{display:block;padding:.6rem .9rem;margin:0 -.9rem;border-bottom:1px solid var(--line);
border-radius:var(--r-soft);text-decoration:none}
.row:hover{background:#0a0a0a}
.nm{font-family:var(--mono);font-size:.84rem;color:var(--ink);display:block}
.ds{color:var(--soft);font-size:.9rem;display:block;margin-top:.12rem}
footer{margin-top:4rem;padding:2rem 0 4rem;border-top:1px solid var(--line);
color:var(--faint);font-size:.85rem}
footer a{color:var(--soft)}
.fmap{display:grid;gap:1.75rem;grid-template-columns:repeat(auto-fit,minmax(11rem,1fr));
margin:0 0 2rem;font-size:.85rem}
.fmap b{display:block;font-family:var(--mono);font-size:.68rem;letter-spacing:.16em;
text-transform:uppercase;color:var(--faint);font-weight:400;margin:0 0 .6rem;
border-bottom:1px solid var(--line);padding-bottom:.4rem}
.fmap a{display:inline-block;color:var(--ink);text-decoration:none;margin:.3rem .35rem 0 0}
.fmap a:hover{text-decoration:underline}
.fmap div{color:var(--faint);line-height:1.5}
.fend{border-top:1px solid var(--line);padding-top:1.25rem;font-size:.82rem}
@media (max-width:640px){
.hero .wrap{padding-top:3.25rem;padding-bottom:2.75rem}
.btns{gap:.5rem}
.btn{padding:.75rem 1.15rem;font-size:.68rem}
.copy,.copy .btn{width:100%;justify-content:center}
.menu{left:0;right:0;min-width:0}
.strip .in{flex-direction:column;align-items:flex-start;gap:.6rem;padding:.9rem 1.75rem}
.strip b{border-right:0;padding-right:0}
.bar{width:100%}
}
</style>
<link rel="alternate" type="text/markdown" href="/schemas.md">
<link rel="icon" href="/favicon.svg">
<link rel="canonical" href="https://hanzoskills.com/schemas">
</head>
<!-- Reading this with a machine? Every page here is also plain markdown.
This one: https://hanzoskills.com/schemas.md
Index: https://hanzoskills.com/llms.txt
Corpus: https://hanzoskills.com/all.md
Append .md to any path for the markdown twin. -->
<body>
<nav><div class="in">
<a class="brand" href="/"><svg class="mark" viewBox="0 0 67 67" fill="currentColor" aria-hidden="true"><path d="M22.21 67V44.6369H0V67H22.21Z"/><path d="M66.7038 22.3184H22.2534L0.0878906 44.6367H44.4634L66.7038 22.3184Z"/><path d="M22.21 0H0V22.3184H22.21V0Z"/><path d="M66.7198 0H44.5098V22.3184H66.7198V0Z"/><path d="M66.7198 67V44.6369H44.5098V67H66.7198Z"/></svg> HANZO<i>SKILLS</i></a>
<a class="up" href="/">All skills</a>
</div></nav>
<div class="wrap">
<h1>schemas</h1><p class="lede">This document defines the JSON schemas used by skill-creator.</p><h1>JSON Schemas</h1>
<p>This document defines the JSON schemas used by skill-creator.</p>
<hr>
<h2>evals.json</h2>
<p>Defines the evals for a skill. Located at <code>evals/evals.json</code> within the skill directory.</p>
<pre><code class="lang-json">{
&quot;skill_name&quot;: &quot;example-skill&quot;,
&quot;evals&quot;: [
{
&quot;id&quot;: 1,
&quot;prompt&quot;: &quot;User&#x27;s example prompt&quot;,
&quot;expected_output&quot;: &quot;Description of expected result&quot;,
&quot;files&quot;: [&quot;evals/files/sample1.pdf&quot;],
&quot;expectations&quot;: [
&quot;The output includes X&quot;,
&quot;The skill used script Y&quot;
]
}
]
}</code></pre>
<p><strong>Fields:</strong></p>
<ul><li><code>skill_name</code>: Name matching the skill&#x27;s frontmatter</li><li><code>evals[].id</code>: Unique integer identifier</li><li><code>evals[].prompt</code>: The task to execute</li><li><code>evals[].expected_output</code>: Human-readable description of success</li><li><code>evals[].files</code>: Optional list of input file paths (relative to skill root)</li><li><code>evals[].expectations</code>: List of verifiable statements</li></ul>
<hr>
<h2>history.json</h2>
<p>Tracks version progression in Improve mode. Located at workspace root.</p>
<pre><code class="lang-json">{
&quot;started_at&quot;: &quot;2026-01-15T10:30:00Z&quot;,
&quot;skill_name&quot;: &quot;pdf&quot;,
&quot;current_best&quot;: &quot;v2&quot;,
&quot;iterations&quot;: [
{
&quot;version&quot;: &quot;v0&quot;,
&quot;parent&quot;: null,
&quot;expectation_pass_rate&quot;: 0.65,
&quot;grading_result&quot;: &quot;baseline&quot;,
&quot;is_current_best&quot;: false
},
{
&quot;version&quot;: &quot;v1&quot;,
&quot;parent&quot;: &quot;v0&quot;,
&quot;expectation_pass_rate&quot;: 0.75,
&quot;grading_result&quot;: &quot;won&quot;,
&quot;is_current_best&quot;: false
},
{
&quot;version&quot;: &quot;v2&quot;,
&quot;parent&quot;: &quot;v1&quot;,
&quot;expectation_pass_rate&quot;: 0.85,
&quot;grading_result&quot;: &quot;won&quot;,
&quot;is_current_best&quot;: true
}
]
}</code></pre>
<p><strong>Fields:</strong></p>
<ul><li><code>started_at</code>: ISO timestamp of when improvement started</li><li><code>skill_name</code>: Name of the skill being improved</li><li><code>current_best</code>: Version identifier of the best performer</li><li><code>iterations[].version</code>: Version identifier (v0, v1, ...)</li><li><code>iterations[].parent</code>: Parent version this was derived from</li><li><code>iterations[].expectation_pass_rate</code>: Pass rate from grading</li><li><code>iterations[].grading_result</code>: &quot;baseline&quot;, &quot;won&quot;, &quot;lost&quot;, or &quot;tie&quot;</li><li><code>iterations[].is_current_best</code>: Whether this is the current best version</li></ul>
<hr>
<h2>grading.json</h2>
<p>Output from the grader agent. Located at <code>&lt;run-dir&gt;/grading.json</code>.</p>
<pre><code class="lang-json">{
&quot;expectations&quot;: [
{
&quot;text&quot;: &quot;The output includes the name &#x27;John Smith&#x27;&quot;,
&quot;passed&quot;: true,
&quot;evidence&quot;: &quot;Found in transcript Step 3: &#x27;Extracted names: John Smith, Sarah Johnson&#x27;&quot;
},
{
&quot;text&quot;: &quot;The spreadsheet has a SUM formula in cell B10&quot;,
&quot;passed&quot;: false,
&quot;evidence&quot;: &quot;No spreadsheet was created. The output was a text file.&quot;
}
],
&quot;summary&quot;: {
&quot;passed&quot;: 2,
&quot;failed&quot;: 1,
&quot;total&quot;: 3,
&quot;pass_rate&quot;: 0.67
},
&quot;execution_metrics&quot;: {
&quot;tool_calls&quot;: {
&quot;Read&quot;: 5,
&quot;Write&quot;: 2,
&quot;Bash&quot;: 8
},
&quot;total_tool_calls&quot;: 15,
&quot;total_steps&quot;: 6,
&quot;errors_encountered&quot;: 0,
&quot;output_chars&quot;: 12450,
&quot;transcript_chars&quot;: 3200
},
&quot;timing&quot;: {
&quot;executor_duration_seconds&quot;: 165.0,
&quot;grader_duration_seconds&quot;: 26.0,
&quot;total_duration_seconds&quot;: 191.0
},
&quot;claims&quot;: [
{
&quot;claim&quot;: &quot;The form has 12 fillable fields&quot;,
&quot;type&quot;: &quot;factual&quot;,
&quot;verified&quot;: true,
&quot;evidence&quot;: &quot;Counted 12 fields in field_info.json&quot;
}
],
&quot;user_notes_summary&quot;: {
&quot;uncertainties&quot;: [&quot;Used 2023 data, may be stale&quot;],
&quot;needs_review&quot;: [],
&quot;workarounds&quot;: [&quot;Fell back to text overlay for non-fillable fields&quot;]
},
&quot;eval_feedback&quot;: {
&quot;suggestions&quot;: [
{
&quot;assertion&quot;: &quot;The output includes the name &#x27;John Smith&#x27;&quot;,
&quot;reason&quot;: &quot;A hallucinated document that mentions the name would also pass&quot;
}
],
&quot;overall&quot;: &quot;Assertions check presence but not correctness.&quot;
}
}</code></pre>
<p><strong>Fields:</strong></p>
<ul><li><code>expectations[]</code>: Graded expectations with evidence</li><li><code>summary</code>: Aggregate pass/fail counts</li><li><code>execution_metrics</code>: Tool usage and output size (from executor&#x27;s metrics.json)</li><li><code>timing</code>: Wall clock timing (from timing.json)</li><li><code>claims</code>: Extracted and verified claims from the output</li><li><code>user_notes_summary</code>: Issues flagged by the executor</li><li><code>eval_feedback</code>: (optional) Improvement suggestions for the evals, only present when the grader identifies issues worth raising</li></ul>
<hr>
<h2>metrics.json</h2>
<p>Output from the executor agent. Located at <code>&lt;run-dir&gt;/outputs/metrics.json</code>.</p>
<pre><code class="lang-json">{
&quot;tool_calls&quot;: {
&quot;Read&quot;: 5,
&quot;Write&quot;: 2,
&quot;Bash&quot;: 8,
&quot;Edit&quot;: 1,
&quot;Glob&quot;: 2,
&quot;Grep&quot;: 0
},
&quot;total_tool_calls&quot;: 18,
&quot;total_steps&quot;: 6,
&quot;files_created&quot;: [&quot;filled_form.pdf&quot;, &quot;field_values.json&quot;],
&quot;errors_encountered&quot;: 0,
&quot;output_chars&quot;: 12450,
&quot;transcript_chars&quot;: 3200
}</code></pre>
<p><strong>Fields:</strong></p>
<ul><li><code>tool_calls</code>: Count per tool type</li><li><code>total_tool_calls</code>: Sum of all tool calls</li><li><code>total_steps</code>: Number of major execution steps</li><li><code>files_created</code>: List of output files created</li><li><code>errors_encountered</code>: Number of errors during execution</li><li><code>output_chars</code>: Total character count of output files</li><li><code>transcript_chars</code>: Character count of transcript</li></ul>
<hr>
<h2>timing.json</h2>
<p>Wall clock timing for a run. Located at <code>&lt;run-dir&gt;/timing.json</code>.</p>
<p><strong>How to capture:</strong> When a subagent task completes, the task notification includes <code>total_tokens</code> and <code>duration_ms</code>. Save these immediately — they are not persisted anywhere else and cannot be recovered after the fact.</p>
<pre><code class="lang-json">{
&quot;total_tokens&quot;: 84852,
&quot;duration_ms&quot;: 23332,
&quot;total_duration_seconds&quot;: 23.3,
&quot;executor_start&quot;: &quot;2026-01-15T10:30:00Z&quot;,
&quot;executor_end&quot;: &quot;2026-01-15T10:32:45Z&quot;,
&quot;executor_duration_seconds&quot;: 165.0,
&quot;grader_start&quot;: &quot;2026-01-15T10:32:46Z&quot;,
&quot;grader_end&quot;: &quot;2026-01-15T10:33:12Z&quot;,
&quot;grader_duration_seconds&quot;: 26.0
}</code></pre>
<hr>
<h2>benchmark.json</h2>
<p>Output from Benchmark mode. Located at <code>benchmarks/&lt;timestamp&gt;/benchmark.json</code>.</p>
<pre><code class="lang-json">{
&quot;metadata&quot;: {
&quot;skill_name&quot;: &quot;pdf&quot;,
&quot;skill_path&quot;: &quot;/path/to/pdf&quot;,
&quot;executor_model&quot;: &quot;claude-sonnet-4-20250514&quot;,
&quot;analyzer_model&quot;: &quot;most-capable-model&quot;,
&quot;timestamp&quot;: &quot;2026-01-15T10:30:00Z&quot;,
&quot;evals_run&quot;: [1, 2, 3],
&quot;runs_per_configuration&quot;: 3
},
&quot;runs&quot;: [
{
&quot;eval_id&quot;: 1,
&quot;eval_name&quot;: &quot;Ocean&quot;,
&quot;configuration&quot;: &quot;with_skill&quot;,
&quot;run_number&quot;: 1,
&quot;result&quot;: {
&quot;pass_rate&quot;: 0.85,
&quot;passed&quot;: 6,
&quot;failed&quot;: 1,
&quot;total&quot;: 7,
&quot;time_seconds&quot;: 42.5,
&quot;tokens&quot;: 3800,
&quot;tool_calls&quot;: 18,
&quot;errors&quot;: 0
},
&quot;expectations&quot;: [
{&quot;text&quot;: &quot;...&quot;, &quot;passed&quot;: true, &quot;evidence&quot;: &quot;...&quot;}
],
&quot;notes&quot;: [
&quot;Used 2023 data, may be stale&quot;,
&quot;Fell back to text overlay for non-fillable fields&quot;
]
}
],
&quot;run_summary&quot;: {
&quot;with_skill&quot;: {
&quot;pass_rate&quot;: {&quot;mean&quot;: 0.85, &quot;stddev&quot;: 0.05, &quot;min&quot;: 0.80, &quot;max&quot;: 0.90},
&quot;time_seconds&quot;: {&quot;mean&quot;: 45.0, &quot;stddev&quot;: 12.0, &quot;min&quot;: 32.0, &quot;max&quot;: 58.0},
&quot;tokens&quot;: {&quot;mean&quot;: 3800, &quot;stddev&quot;: 400, &quot;min&quot;: 3200, &quot;max&quot;: 4100}
},
&quot;without_skill&quot;: {
&quot;pass_rate&quot;: {&quot;mean&quot;: 0.35, &quot;stddev&quot;: 0.08, &quot;min&quot;: 0.28, &quot;max&quot;: 0.45},
&quot;time_seconds&quot;: {&quot;mean&quot;: 32.0, &quot;stddev&quot;: 8.0, &quot;min&quot;: 24.0, &quot;max&quot;: 42.0},
&quot;tokens&quot;: {&quot;mean&quot;: 2100, &quot;stddev&quot;: 300, &quot;min&quot;: 1800, &quot;max&quot;: 2500}
},
&quot;delta&quot;: {
&quot;pass_rate&quot;: &quot;+0.50&quot;,
&quot;time_seconds&quot;: &quot;+13.0&quot;,
&quot;tokens&quot;: &quot;+1700&quot;
}
},
&quot;notes&quot;: [
&quot;Assertion &#x27;Output is a PDF file&#x27; passes 100% in both configurations - may not differentiate skill value&quot;,
&quot;Eval 3 shows high variance (50% ± 40%) - may be flaky or model-dependent&quot;,
&quot;Without-skill runs consistently fail on table extraction expectations&quot;,
&quot;Skill adds 13s average execution time but improves pass rate by 50%&quot;
]
}</code></pre>
<p><strong>Fields:</strong></p>
<ul><li><code>metadata</code>: Information about the benchmark run</li><li><code>skill_name</code>: Name of the skill</li><li><code>timestamp</code>: When the benchmark was run</li><li><code>evals_run</code>: List of eval names or IDs</li><li><code>runs_per_configuration</code>: Number of runs per config (e.g. 3)</li><li><code>runs[]</code>: Individual run results</li><li><code>eval_id</code>: Numeric eval identifier</li><li><code>eval_name</code>: Human-readable eval name (used as section header in the viewer)</li><li><code>configuration</code>: Must be <code>&quot;with_skill&quot;</code> or <code>&quot;without_skill&quot;</code> (the viewer uses this exact string for grouping and color coding)</li><li><code>run_number</code>: Integer run number (1, 2, 3...)</li><li><code>result</code>: Nested object with <code>pass_rate</code>, <code>passed</code>, <code>total</code>, <code>time_seconds</code>, <code>tokens</code>, <code>errors</code></li><li><code>run_summary</code>: Statistical aggregates per configuration</li><li><code>with_skill</code> / <code>without_skill</code>: Each contains <code>pass_rate</code>, <code>time_seconds</code>, <code>tokens</code> objects with <code>mean</code> and <code>stddev</code> fields</li><li><code>delta</code>: Difference strings like <code>&quot;+0.50&quot;</code>, <code>&quot;+13.0&quot;</code>, <code>&quot;+1700&quot;</code></li><li><code>notes</code>: Freeform observations from the analyzer</li></ul>
<p><strong>Important:</strong> The viewer reads these field names exactly. Using <code>config</code> instead of <code>configuration</code>, or putting <code>pass_rate</code> at the top level of a run instead of nested under <code>result</code>, will cause the viewer to show empty/zero values. Always reference this schema when generating benchmark.json manually.</p>
<hr>
<h2>comparison.json</h2>
<p>Output from blind comparator. Located at <code>&lt;grading-dir&gt;/comparison-N.json</code>.</p>
<pre><code class="lang-json">{
&quot;winner&quot;: &quot;A&quot;,
&quot;reasoning&quot;: &quot;Output A provides a complete solution with proper formatting and all required fields. Output B is missing the date field and has formatting inconsistencies.&quot;,
&quot;rubric&quot;: {
&quot;A&quot;: {
&quot;content&quot;: {
&quot;correctness&quot;: 5,
&quot;completeness&quot;: 5,
&quot;accuracy&quot;: 4
},
&quot;structure&quot;: {
&quot;organization&quot;: 4,
&quot;formatting&quot;: 5,
&quot;usability&quot;: 4
},
&quot;content_score&quot;: 4.7,
&quot;structure_score&quot;: 4.3,
&quot;overall_score&quot;: 9.0
},
&quot;B&quot;: {
&quot;content&quot;: {
&quot;correctness&quot;: 3,
&quot;completeness&quot;: 2,
&quot;accuracy&quot;: 3
},
&quot;structure&quot;: {
&quot;organization&quot;: 3,
&quot;formatting&quot;: 2,
&quot;usability&quot;: 3
},
&quot;content_score&quot;: 2.7,
&quot;structure_score&quot;: 2.7,
&quot;overall_score&quot;: 5.4
}
},
&quot;output_quality&quot;: {
&quot;A&quot;: {
&quot;score&quot;: 9,
&quot;strengths&quot;: [&quot;Complete solution&quot;, &quot;Well-formatted&quot;, &quot;All fields present&quot;],
&quot;weaknesses&quot;: [&quot;Minor style inconsistency in header&quot;]
},
&quot;B&quot;: {
&quot;score&quot;: 5,
&quot;strengths&quot;: [&quot;Readable output&quot;, &quot;Correct basic structure&quot;],
&quot;weaknesses&quot;: [&quot;Missing date field&quot;, &quot;Formatting inconsistencies&quot;, &quot;Partial data extraction&quot;]
}
},
&quot;expectation_results&quot;: {
&quot;A&quot;: {
&quot;passed&quot;: 4,
&quot;total&quot;: 5,
&quot;pass_rate&quot;: 0.80,
&quot;details&quot;: [
{&quot;text&quot;: &quot;Output includes name&quot;, &quot;passed&quot;: true}
]
},
&quot;B&quot;: {
&quot;passed&quot;: 3,
&quot;total&quot;: 5,
&quot;pass_rate&quot;: 0.60,
&quot;details&quot;: [
{&quot;text&quot;: &quot;Output includes name&quot;, &quot;passed&quot;: true}
]
}
}
}</code></pre>
<hr>
<h2>analysis.json</h2>
<p>Output from post-hoc analyzer. Located at <code>&lt;grading-dir&gt;/analysis.json</code>.</p>
<pre><code class="lang-json">{
&quot;comparison_summary&quot;: {
&quot;winner&quot;: &quot;A&quot;,
&quot;winner_skill&quot;: &quot;path/to/winner/skill&quot;,
&quot;loser_skill&quot;: &quot;path/to/loser/skill&quot;,
&quot;comparator_reasoning&quot;: &quot;Brief summary of why comparator chose winner&quot;
},
&quot;winner_strengths&quot;: [
&quot;Clear step-by-step instructions for handling multi-page documents&quot;,
&quot;Included validation script that caught formatting errors&quot;
],
&quot;loser_weaknesses&quot;: [
&quot;Vague instruction &#x27;process the document appropriately&#x27; led to inconsistent behavior&quot;,
&quot;No script for validation, agent had to improvise&quot;
],
&quot;instruction_following&quot;: {
&quot;winner&quot;: {
&quot;score&quot;: 9,
&quot;issues&quot;: [&quot;Minor: skipped optional logging step&quot;]
},
&quot;loser&quot;: {
&quot;score&quot;: 6,
&quot;issues&quot;: [
&quot;Did not use the skill&#x27;s formatting template&quot;,
&quot;Invented own approach instead of following step 3&quot;
]
}
},
&quot;improvement_suggestions&quot;: [
{
&quot;priority&quot;: &quot;high&quot;,
&quot;category&quot;: &quot;instructions&quot;,
&quot;suggestion&quot;: &quot;Replace &#x27;process the document appropriately&#x27; with explicit steps&quot;,
&quot;expected_impact&quot;: &quot;Would eliminate ambiguity that caused inconsistent behavior&quot;
}
],
&quot;transcript_insights&quot;: {
&quot;winner_execution_pattern&quot;: &quot;Read skill -&gt; Followed 5-step process -&gt; Used validation script&quot;,
&quot;loser_execution_pattern&quot;: &quot;Read skill -&gt; Unclear on approach -&gt; Tried 3 different methods&quot;
}
}</code></pre>
<footer>
<div class="fmap">
<div><b>Build</b>
<a href="https://hanzo.app">hanzo.app</a> app builder
<a href="https://hanzo.codes">hanzo.codes</a> code with an agent
<a href="https://hanzoskills.com">hanzoskills.com</a> what an agent reads</div>
<div><b>Run</b>
<a href="https://console.hanzo.ai">console.hanzo.ai</a> cloud console
<a href="https://api.hanzo.ai">api.hanzo.ai</a> the API</div>
<div><b>Account</b>
<a href="https://hanzo.id">hanzo.id</a> sign in
<a href="https://pay.hanzo.ai">pay.hanzo.ai</a> top up
<a href="https://billing.hanzo.ai">billing.hanzo.ai</a> invoices and usage</div>
<div><b>Company</b>
<a href="https://hanzo.ai">hanzo.ai</a> canonical
<a href="https://hanzo.ventures">hanzo.ventures</a> ventures</div>
</div>
<div class="fend"><a href="/schemas.md">This page as plain text</a> · <a href="/llms.txt">Index for agents</a></div>
</footer>
</div></body></html>