mirror of
https://github.com/hanzo-apps/hanzo.ai.git
synced 2026-10-29 07:20:43 +00:00
/enso leads with three lines built from the enso.* runs: the router at 92.9% on GPQA Diamond, level with the pool's best single model, at $2.20, 5x less than Opus 4.8 (86.4%, $10.97); LiveCodeBench 93.7% at $3.06, 12.9x less than Opus 4.8 (72.0%, $39.43); and the pool's ceiling, 98.0% (194/198) with four arms, said only as the ceiling the router is trained toward. Under them a note says what that router was: enso-bench's at the tag its run names, and the arms its notes count, not today's presets. Every percentage and multiple is computed from the runs. A board is each system's frontier run, else its v1 run (the same prompts; v2 and max put others), over the whole set, none degraded. The GPQA and LiveCodeBench boards draw on /enso, /models/enso and /benchmarks, the GPQA one with the ceiling under it on /benchmarks/gpqa. /benchmarks' Enso title, figure, hero tile and frontier (now accuracy against what the run cost), the /models family card and the home tile read the same runs, and each says the API is unreachable rather than show an older number. lib/data/leaderboard.json and lib/leaderboard.ts are deleted: their Enso tiers (98.0, 96.0, 92.9) were no run's, 98.0 being the pool's oracle. With them go the vendor-reported field, the accuracy-at-cost scatter, the reported-versus-measured browser, the shelf's Enso entries and the two pages they generated, and the logo files only the scatter read. A preset card keeps its name and rate and states no score. The gates serve invented runs and hold every surface, the ceiling's figure and the unreachable state to lib/enso.
126 lines
5.4 KiB
JavaScript
126 lines
5.4 KiB
JavaScript
#!/usr/bin/env node
|
|
/**
|
|
* Fail if a model price is typed into source.
|
|
*
|
|
* A per-token price has ONE owner — the catalog in commerce.hanzo.ai, edited in
|
|
* admin and published through api.hanzo.ai/v1/pricing. Every other copy is a
|
|
* copy, and copies go stale silently: this repo advertised Enso at 3/12 while
|
|
* the service charged 4/20, another copy said 20/60, and the public catalog
|
|
* carried no Enso at all. Nobody noticed, because a wrong number renders exactly
|
|
* as convincingly as a right one.
|
|
*
|
|
* So the rule is not "keep the numbers up to date". It is: don't hold a second
|
|
* copy. Read the catalog when the page opens (lib/api); no snapshot of it is
|
|
* committed either.
|
|
*
|
|
* WHAT COUNTS AS A LITERAL: the shapes a per-MTok rate actually takes in this
|
|
* codebase — a rate field assigned a number (`input: 4`, `priceOut: 20`), and a
|
|
* rendered price band (`'$4 → $20'`). Both are checked, because the first pass
|
|
* of this guard only knew the first shape and reported the marketing pages
|
|
* clean while they carried six hardcoded bands.
|
|
*
|
|
* WHAT IT DELIBERATELY DOES NOT COVER: a bare `price:` number. Subscription
|
|
* plans, DNS, vector and blockchain pricing all use that field for something
|
|
* that is not a token rate — requiring it would flag twelve unrelated files, and
|
|
* a guard that cries wolf gets switched off. The shapes above are the ones an Enso rate is actually written in, which is
|
|
* what this is defending.
|
|
*
|
|
* THE LIST BELOW IS A RATCHET. It records the copies that exist TODAY, each with
|
|
* the reason it is still allowed. It may only SHRINK. Adding a file to it is
|
|
* adding a fourth owner to a number that already has one too many — fix the file
|
|
* instead. When a converted file drops off, delete its line.
|
|
*
|
|
* Usage: node scripts/audit-price-literals.mjs
|
|
*/
|
|
|
|
import { execFileSync } from "child_process";
|
|
import { readFileSync } from "fs";
|
|
import { resolve, dirname, relative } from "path";
|
|
import { fileURLToPath } from "url";
|
|
import { code, isCode } from "./source.mjs";
|
|
|
|
const ROOT = resolve(dirname(fileURLToPath(import.meta.url)), "..");
|
|
|
|
// This file, and the one file the scan must not read: the shapes a rate is
|
|
// written in have to be WRITTEN DOWN somewhere, and the place they are written
|
|
// down is here — in the doc comment above and in the patterns below. Scanning
|
|
// itself, it found `input: 4` and `'$4 → $20'` in its own explanation of what
|
|
// those mean and exited 1 on every run, so the step could never pass and said
|
|
// nothing about the repo. Derived from `import.meta.url` rather than typed as a
|
|
// path, so renaming the file cannot quietly re-break it.
|
|
const SELF = relative(ROOT, fileURLToPath(import.meta.url));
|
|
|
|
// A per-MTok rate as this codebase writes one: a rate field with a number, or a
|
|
// rendered "$in → $out" band.
|
|
const RATE_FIELD =
|
|
/(?:^|[^\w])"?(input|output|priceIn|priceOut|price_in|price_out|cacheRead|cacheWrite)"?\s*:\s*"?\$?\d+(?:\.\d+)?/;
|
|
const PRICE_BAND = /\$\d+(?:\.\d+)?\s*(?:→|->|—>)\s*\$\d+(?:\.\d+)?/;
|
|
const PRICE = { test: (s) => RATE_FIELD.test(s) || PRICE_BAND.test(s) };
|
|
|
|
const EXT = new Set([".ts", ".tsx", ".js", ".jsx", ".mjs", ".json", ".mdx", ".md"]);
|
|
|
|
// The gates' stand-in for the backend, allowed by kind: e2e serves a page an
|
|
// invented answer and checks the page draws that answer and nothing else, so its
|
|
// numbers are made up on purpose and never reach a reader. A real rate in one
|
|
// would prove nothing a made-up one does not.
|
|
const FIXTURE = /^e2e\//;
|
|
|
|
// The ratchet. Path → why it still holds a literal. SHRINK ONLY.
|
|
const ALLOWED = {};
|
|
|
|
// Only TRACKED files. A guard checks what is committed; ignored paths (build
|
|
// output, local agent worktrees) are not this repo's copies of anything.
|
|
function tracked() {
|
|
return execFileSync("git", ["ls-files", "-z"], { cwd: ROOT, maxBuffer: 64 << 20 })
|
|
.toString()
|
|
.split("\0")
|
|
.filter((p) => p && EXT.has(p.slice(p.lastIndexOf("."))));
|
|
}
|
|
|
|
/** The lines as a reader of the site sees them: comments are not copy. */
|
|
const copy = (rel) => {
|
|
const text = readFileSync(resolve(ROOT, rel), "utf8");
|
|
return (isCode(rel) ? code(text) : text).split("\n");
|
|
};
|
|
|
|
const offenders = [];
|
|
for (const rel of tracked()) {
|
|
if (rel === SELF || FIXTURE.test(rel) || ALLOWED[rel]) continue;
|
|
copy(rel).forEach((line, i) => {
|
|
if (PRICE.test(line)) offenders.push(`${rel}:${i + 1}: ${line.trim().slice(0, 100)}`);
|
|
});
|
|
}
|
|
|
|
if (offenders.length) {
|
|
console.error(
|
|
`\nA model price is typed into ${offenders.length} place(s) that should be reading it.\n\n` +
|
|
offenders.map((o) => ` ${o}`).join("\n") +
|
|
`\n\nA per-token price is owned by the catalog in commerce.hanzo.ai and published\n` +
|
|
`at api.hanzo.ai/v1/pricing. Read it when the page opens (lib/api,\n` +
|
|
`components/api/rate). Do not add a line to ALLOWED in this script —\n` +
|
|
`that list only shrinks.\n`
|
|
);
|
|
process.exit(1);
|
|
}
|
|
|
|
const stale = Object.keys(ALLOWED).filter((p) => {
|
|
try {
|
|
return !copy(p).some((l) => PRICE.test(l));
|
|
} catch {
|
|
return true;
|
|
}
|
|
});
|
|
if (stale.length) {
|
|
console.error(
|
|
`\nThese are on the ratchet but no longer hold a price literal:\n\n` +
|
|
stale.map((p) => ` ${p}`).join("\n") +
|
|
`\n\nDelete them from ALLOWED — the list is meant to shrink, and an entry that\n` +
|
|
`no longer describes anything hides the ones that still do.\n`
|
|
);
|
|
process.exit(1);
|
|
}
|
|
|
|
console.log(
|
|
`price literals: ${Object.keys(ALLOWED).length} known holder(s) remaining, no snapshot, no new copies.`
|
|
);
|