Files
hanzo.ai/scripts/audit-price-literals.mjs
zeekay d01992490a Every Enso figure is read from its research runs when a page opens, and the leaderboard snapshot is gone
/enso leads with three lines built from the enso.* runs: the router at
92.9% on GPQA Diamond, level with the pool's best single model, at $2.20,
5x less than Opus 4.8 (86.4%, $10.97); LiveCodeBench 93.7% at $3.06,
12.9x less than Opus 4.8 (72.0%, $39.43); and the pool's ceiling, 98.0%
(194/198) with four arms, said only as the ceiling the router is trained
toward. Under them a note says what that router was: enso-bench's at the
tag its run names, and the arms its notes count, not today's presets.
Every percentage and multiple is computed from the runs.

A board is each system's frontier run, else its v1 run (the same prompts;
v2 and max put others), over the whole set, none degraded. The GPQA and
LiveCodeBench boards draw on /enso, /models/enso and /benchmarks, the
GPQA one with the ceiling under it on /benchmarks/gpqa. /benchmarks' Enso
title, figure, hero tile and frontier (now accuracy against what the run
cost), the /models family card and the home tile read the same runs, and
each says the API is unreachable rather than show an older number.

lib/data/leaderboard.json and lib/leaderboard.ts are deleted: their Enso
tiers (98.0, 96.0, 92.9) were no run's, 98.0 being the pool's oracle.
With them go the vendor-reported field, the accuracy-at-cost scatter, the
reported-versus-measured browser, the shelf's Enso entries and the two
pages they generated, and the logo files only the scatter read. A preset
card keeps its name and rate and states no score. The gates serve
invented runs and hold every surface, the ceiling's figure and the
unreachable state to lib/enso.
2026-10-15 15:18:16 -07:00

126 lines
5.4 KiB
JavaScript

#!/usr/bin/env node
/**
* Fail if a model price is typed into source.
*
* A per-token price has ONE owner — the catalog in commerce.hanzo.ai, edited in
* admin and published through api.hanzo.ai/v1/pricing. Every other copy is a
* copy, and copies go stale silently: this repo advertised Enso at 3/12 while
* the service charged 4/20, another copy said 20/60, and the public catalog
* carried no Enso at all. Nobody noticed, because a wrong number renders exactly
* as convincingly as a right one.
*
* So the rule is not "keep the numbers up to date". It is: don't hold a second
* copy. Read the catalog when the page opens (lib/api); no snapshot of it is
* committed either.
*
* WHAT COUNTS AS A LITERAL: the shapes a per-MTok rate actually takes in this
* codebase — a rate field assigned a number (`input: 4`, `priceOut: 20`), and a
* rendered price band (`'$4 → $20'`). Both are checked, because the first pass
* of this guard only knew the first shape and reported the marketing pages
* clean while they carried six hardcoded bands.
*
* WHAT IT DELIBERATELY DOES NOT COVER: a bare `price:` number. Subscription
* plans, DNS, vector and blockchain pricing all use that field for something
* that is not a token rate — requiring it would flag twelve unrelated files, and
* a guard that cries wolf gets switched off. The shapes above are the ones an Enso rate is actually written in, which is
* what this is defending.
*
* THE LIST BELOW IS A RATCHET. It records the copies that exist TODAY, each with
* the reason it is still allowed. It may only SHRINK. Adding a file to it is
* adding a fourth owner to a number that already has one too many — fix the file
* instead. When a converted file drops off, delete its line.
*
* Usage: node scripts/audit-price-literals.mjs
*/
import { execFileSync } from "child_process";
import { readFileSync } from "fs";
import { resolve, dirname, relative } from "path";
import { fileURLToPath } from "url";
import { code, isCode } from "./source.mjs";
const ROOT = resolve(dirname(fileURLToPath(import.meta.url)), "..");
// This file, and the one file the scan must not read: the shapes a rate is
// written in have to be WRITTEN DOWN somewhere, and the place they are written
// down is here — in the doc comment above and in the patterns below. Scanning
// itself, it found `input: 4` and `'$4 → $20'` in its own explanation of what
// those mean and exited 1 on every run, so the step could never pass and said
// nothing about the repo. Derived from `import.meta.url` rather than typed as a
// path, so renaming the file cannot quietly re-break it.
const SELF = relative(ROOT, fileURLToPath(import.meta.url));
// A per-MTok rate as this codebase writes one: a rate field with a number, or a
// rendered "$in → $out" band.
const RATE_FIELD =
/(?:^|[^\w])"?(input|output|priceIn|priceOut|price_in|price_out|cacheRead|cacheWrite)"?\s*:\s*"?\$?\d+(?:\.\d+)?/;
const PRICE_BAND = /\$\d+(?:\.\d+)?\s*(?:→|->|—>)\s*\$\d+(?:\.\d+)?/;
const PRICE = { test: (s) => RATE_FIELD.test(s) || PRICE_BAND.test(s) };
const EXT = new Set([".ts", ".tsx", ".js", ".jsx", ".mjs", ".json", ".mdx", ".md"]);
// The gates' stand-in for the backend, allowed by kind: e2e serves a page an
// invented answer and checks the page draws that answer and nothing else, so its
// numbers are made up on purpose and never reach a reader. A real rate in one
// would prove nothing a made-up one does not.
const FIXTURE = /^e2e\//;
// The ratchet. Path → why it still holds a literal. SHRINK ONLY.
const ALLOWED = {};
// Only TRACKED files. A guard checks what is committed; ignored paths (build
// output, local agent worktrees) are not this repo's copies of anything.
function tracked() {
return execFileSync("git", ["ls-files", "-z"], { cwd: ROOT, maxBuffer: 64 << 20 })
.toString()
.split("\0")
.filter((p) => p && EXT.has(p.slice(p.lastIndexOf("."))));
}
/** The lines as a reader of the site sees them: comments are not copy. */
const copy = (rel) => {
const text = readFileSync(resolve(ROOT, rel), "utf8");
return (isCode(rel) ? code(text) : text).split("\n");
};
const offenders = [];
for (const rel of tracked()) {
if (rel === SELF || FIXTURE.test(rel) || ALLOWED[rel]) continue;
copy(rel).forEach((line, i) => {
if (PRICE.test(line)) offenders.push(`${rel}:${i + 1}: ${line.trim().slice(0, 100)}`);
});
}
if (offenders.length) {
console.error(
`\nA model price is typed into ${offenders.length} place(s) that should be reading it.\n\n` +
offenders.map((o) => ` ${o}`).join("\n") +
`\n\nA per-token price is owned by the catalog in commerce.hanzo.ai and published\n` +
`at api.hanzo.ai/v1/pricing. Read it when the page opens (lib/api,\n` +
`components/api/rate). Do not add a line to ALLOWED in this script —\n` +
`that list only shrinks.\n`
);
process.exit(1);
}
const stale = Object.keys(ALLOWED).filter((p) => {
try {
return !copy(p).some((l) => PRICE.test(l));
} catch {
return true;
}
});
if (stale.length) {
console.error(
`\nThese are on the ratchet but no longer hold a price literal:\n\n` +
stale.map((p) => ` ${p}`).join("\n") +
`\n\nDelete them from ALLOWED — the list is meant to shrink, and an entry that\n` +
`no longer describes anything hides the ones that still do.\n`
);
process.exit(1);
}
console.log(
`price literals: ${Object.keys(ALLOWED).length} known holder(s) remaining, no snapshot, no new copies.`
);