mirror of
https://github.com/hanzo-apps/hanzo.ai.git
synced 2026-10-28 07:32:52 +00:00
zeekay
18222da10c
The agentic coding table quoted 21.2x decision latency, 23.4x wall time and -98.7% cost from benchmarks-coding-agent.json, which agent_coding_bench.py wrote by simulating both loops. No run stood behind it. The board is now the research API's runs of coding.repair: Hanzo Dev (Kai + Enso Flash on api.hanzo.ai) against native Claude Code on 44 held-out repairs, each solved only when its hidden verifier passes. Both solved counts stand beside the paired speed, and every path keeps its interval, resolved or not: 3.83x [2.76, 5.25] over the 40 tasks both solved, 8.22x where a search settled the task with no model call, 1.22x [0.96, 1.53] where a model wrote the edit. scripts/coding.mjs snapshots the runs at prebuild and useCoding refreshes them in the browser, as Kai's boards do. The board is drawn only while the speed over all tasks both solved is a resolved gain.
37 lines
1.6 KiB
JavaScript
37 lines
1.6 KiB
JavaScript
#!/usr/bin/env node
|
|
/**
|
|
* Hanzo Dev against Claude Code, as the research API holds it.
|
|
*
|
|
* node scripts/coding.mjs [api base URL]
|
|
*
|
|
* Saves the public runs of `GET /v1/research/runs?org=hanzo&benchmark=coding.repair`
|
|
* (lib/coding.ts RUNS) as lib/data/benchmarks-coding.json: the snapshot /benchmarks
|
|
* renders before its own fetch of the same address returns. `bench/code/post.py` in
|
|
* hanzo-inc/kai writes those runs. Part of `prebuild`, so every build ships the
|
|
* latest pass. The committed snapshot is kept, and the build goes on, when the API
|
|
* cannot be reached, answers with an error, or holds no pass the page would draw.
|
|
*/
|
|
import { writeFileSync } from 'node:fs'
|
|
|
|
const FILE = new URL('../lib/data/benchmarks-coding.json', import.meta.url)
|
|
const keep = (why) => {
|
|
console.log(`coding: kept the frozen snapshot (${why})`)
|
|
process.exit(0)
|
|
}
|
|
|
|
try {
|
|
const { RUNS, derive } = await import('../lib/coding.ts')
|
|
const base = process.argv[2]?.replace(/\/$/, '')
|
|
const url = base ? RUNS.replace(new URL(RUNS).origin, base) : RUNS
|
|
const res = await fetch(url, { signal: AbortSignal.timeout(20_000) })
|
|
if (!res.ok) keep(`${url}: ${res.status}`)
|
|
const body = await res.json()
|
|
const data = (body.data ?? []).filter((r) => (r.visibility ?? 'public') === 'public')
|
|
const board = derive(data)
|
|
if (!board) keep(`${url}: no pass to draw`)
|
|
writeFileSync(FILE, `{\n"data": [\n${data.map((r) => JSON.stringify(r)).join(',\n')}\n]\n}\n`)
|
|
console.log(`coding: ${data.length} runs; ${board.ours.version} ${board.speed.value.toFixed(2)}x over ${board.speed.n} tasks both solved`)
|
|
} catch (e) {
|
|
keep(e instanceof Error ? e.message : String(e))
|
|
}
|