Files
zeekayandHanzo Dev 6752f71a47 The benchmarks page leads with the numbers that reproduce
/benchmarks opened on LoCoMo retrieval, which made the benchmarks page for a
whole platform read as a page about conversational memory. Four of the ten
benchmarks in the registry were not on it at all, and one paragraph explicitly
sent the runtime measurements away — "they belong beside the platform rather
than here" — when those are the cheapest numbers here to check: one command, one
laptop, a minute.

So the order is the stack now, declared once in lib/benchmarks as LANES and used
by both the index and the directory, so the two cannot disagree about what comes
first. Runtime, then inference, then routing, then memory, then code.

Three paper links answered 404. papers.hanzo.ai does not carry
hanzo-dormant-agents, hanzo-multi-hop-retrieval or locomo-subject-scope, and
every page citing them pointed at the library anyway — on pages whose whole
argument is that a number can be checked. A Paper now says which of the two
addresses serves it, and the three unpublished ones point at the public source
tree that does have them.

Two corrections that matter more than the ordering.

The sandbox lane reported a V8 context at 0.15 ms against a cited isolate at
2.79 ms and called it seventeenfold. Those are different primitives: a context
shares the heap of an isolate already running, which is the one thing an isolate
does not do. The upstream bench measured isolated-vm instead of citing it and
got 0.59 ms and 1.00 MiB. The tables carry a third column for that — what their
library does when we run it here — and the 17x is gone.

The inference page published "decode at parity on all three" from a paper table
with no variance estimate, and a cooled Metal pass whose run directory is in no
repository and on no machine we can find. Six committed run directories on
bench/dossier have carried better evidence the whole time: four backends, two
models, four prompt lengths, seven repetitions, a 95% interval and a verdict on
every cell. They say decode is a loss on every backend measured. The standing is
a surface rather than a scalar — CUDA wins at long prompts, Metal reaches parity
only at the longest, ROCm falls away, Vulkan collapses — and the page is now
arranged so that shape is the first thing visible. The missing pass is withdrawn
and said to be withdrawn.

The router's GPQA-Diamond ladder joins as its own bench and route, with each
score against its tier and its price, the count out of 198, and the column that
separates what we measured from what a publisher reports about itself.

Nothing was added that a reader cannot regenerate. The per-hour hardware ladder
and the on-premise round trip are arithmetic over an assumption and a figure with
no command behind it; both stay off, and the page says why.

(cherry picked from commit 45b3a92187)

Co-authored-by: Hanzo Dev <[email protected]>
2026-09-11 12:16:58 -07:00

133 lines
4.4 KiB
TypeScript

import { BENCHMARKS, benchUrl, paperUrl, type Bench } from '@/lib/benchmarks'
/**
* What a bench page says to a machine.
*
* Three entities, because three different things are true of one page and
* flattening them loses the part that matters:
*
* Dataset the benchmark itself — somebody else's work, under
* somebody else's licence. `url` is this page, which is what
* schema.org means by it (the page DESCRIBING the dataset),
* and `sameAs` is the repo or dataset card that actually
* holds it, so the claim is "we describe this" and never
* "we made this".
* WebPage this page, `about` the dataset, published by the
* Organization the root layout already declares — referenced
* by its `@id` rather than described a second time.
* BreadcrumbList /benchmarks → this one. Two levels, which is all there is.
*
* A ScholarlyArticle joins them where a paper of ours reports the numbers, and
* its `about` points back at the same Dataset node, so the paper, the page and
* the benchmark are one graph rather than three assertions that happen to share
* a subject.
*
* A bench with no runs gets the page and the breadcrumb and no Dataset:
* `variableMeasured` on a benchmark nothing has been run on would describe
* measurements that do not exist.
*/
const SITE = 'https://hanzo.ai'
const ORG = `${SITE}/#organization`
const INDEX = `${SITE}/benchmarks`
/**
* `</script>` inside a JSON string ends the block and the rest of the page
* becomes markup. Nothing here contains one today, which is exactly the
* condition under which it later will.
*/
const embed = (value: unknown) => JSON.stringify(value).replace(/</g, '\\u003c')
export function BenchJsonLd({ b, measures }: { b: Bench; measures?: string[] }) {
const url = benchUrl(b.slug)
const dataset = b.dataset
? {
'@type': 'Dataset',
'@id': `${url}#dataset`,
name: b.dataset.name,
description: b.lede,
url,
sameAs: [b.dataset.url, b.dataset.paper.url],
license: b.dataset.license,
isAccessibleForFree: true,
creator: { '@type': 'Organization', name: b.dataset.paper.authors },
citation: {
'@type': 'ScholarlyArticle',
name: b.dataset.paper.title,
url: b.dataset.paper.url,
},
...(measures?.length ? { variableMeasured: measures } : {}),
measurementTechnique: b.harness.command,
}
: null
const graph = [
...(dataset ? [dataset] : []),
{
'@type': 'WebPage',
'@id': url,
url,
name: `${b.title} — Hanzo AI`,
description: b.lede,
isPartOf: { '@type': 'WebPage', '@id': INDEX, name: 'Benchmarks', url: INDEX },
publisher: { '@id': ORG },
...(dataset ? { about: { '@id': `${url}#dataset` } } : {}),
},
{
'@type': 'BreadcrumbList',
itemListElement: [
{ '@type': 'ListItem', position: 1, name: 'Benchmarks', item: INDEX },
{ '@type': 'ListItem', position: 2, name: b.name, item: url },
],
},
...b.papers.map((p) => ({
'@type': 'ScholarlyArticle',
name: p.title,
url: paperUrl(p),
author: { '@id': ORG },
...(dataset ? { about: { '@id': `${url}#dataset` } } : {}),
})),
]
return (
<script
type="application/ld+json"
dangerouslySetInnerHTML={{ __html: embed({ '@context': 'https://schema.org', '@graph': graph }) }}
/>
)
}
/**
* The index, as the list it is.
*
* An `ItemList` of the bench pages rather than a Dataset of its own: /benchmarks
* measures nothing itself, it says what has been measured and where each one
* lives.
*/
export function IndexJsonLd() {
const graph = [
{
'@type': 'ItemList',
'@id': `${INDEX}#list`,
name: 'Hanzo benchmarks',
numberOfItems: BENCHMARKS.length,
itemListElement: BENCHMARKS.map((b, i) => ({
'@type': 'ListItem',
position: i + 1,
name: b.name,
url: benchUrl(b.slug),
})),
},
{
'@type': 'BreadcrumbList',
itemListElement: [{ '@type': 'ListItem', position: 1, name: 'Benchmarks', item: INDEX }],
},
]
return (
<script
type="application/ld+json"
dangerouslySetInnerHTML={{ __html: embed({ '@context': 'https://schema.org', '@graph': graph }) }}
/>
)
}