Files
zeekay c38d21b533 Point Benchmarks and Philosophy CTAs to platform.hanzo.ai
platform.hanzo.ai is the platform console (with cloud.hanzo.ai
redirecting to it at the edge). Point the remaining cloud.hanzo.ai
links on /benchmarks and /philosophy directly at https://platform.hanzo.ai.

Claude-Session: https://claude.ai/code/session_015XYKLix9PSyn6Jw2HWKprk
2026-11-09 12:16:29 -08:00

1 line
5.6 KiB
JSON

{"text":["GPQA-Diamond","Benchmarks","measured","The router on GPQA-Diamond","One set of 198 graduate-level questions, answered by Enso's router and by single models from other labs, every row a research run.","dataset","GPQA","licence","CC BY 4.0","introduced by","Rein, Hou, Stickland, Petty, Pang, Dirani, Michael, Bowman","COLM 2024","GPQA: A Graduate-Level Google-Proof Q&A Benchmark","harness","api.hanzo.ai/v1/research","curl -s \"https://api.hanzo.ai/v1/research/runs?org=hanzo&benchmark=enso.gpqa_diamond\"","prints the table on this page — transcribed from the research API, read when the page opens","What is measured","GPQA-Diamond is the hardest subset of GPQA: 198 multiple-choice questions in biology, physics and chemistry, written by domain PhDs and filtered to the ones that other PhDs get right and skilled non-experts get wrong even with the web open. It is a knowledge benchmark rather than an agentic one — one question, one answer, no tools — which is what makes it useful here: it isolates the choice of model from everything else a system does.","What the router adds is that the choice is per question. Enso reads each one and dispatches it, so a tier is not a model — it is a pool and a willingness to spend. Every row below answers the same 198 questions, which is why a score and what its run cost belong in one table, and why a score quoted without its cost says nothing about the system it came from.","GPQA Diamond · enso.gpqa_diamond, split test · every row a research run, read from api.hanzo.ai when the page opens · sorted by accuracy","System","Accuracy","Correct","Run cost","Tag, date","enso","run","92.9%","184 of 198","$5.58","frontier","2026-07-19","enso-ultra","$20.20","gpt-5.6-sol","91.9%","182 of 198","$4.08","opus-5.5","90.9%","180 of 198","$3.78","fill","2026-10-16","sonnet-5.5","90.4%","179 of 198","$1.85","gpt-5.5","89.9%","178 of 198","$11.95","fable-5.1","87.9%","174 of 198","$8.23","opus-4.8","86.4%","171 of 198","$3.66","haiku-5.5","83.3%","165 of 198","$0.190","fable-5","82.8%","164 of 198","$9.87","claude-5-sonnet","77.8%","154 of 198","$5.66","deepseek-v4-pro","74.2%","147 of 198","$0.203","gpt-4o","54.5%","108 of 198","$1.58","Each system's frontier run, else its v1 run, else its fill run: the three put the same prompts to a model. A run cost is the input and output tokens the run records at the price list's rate for its own model id, the router's included, at list prices, Nov 7, 2026; a dash is a run that records no tokens, a model the list does not price, or a list that could not be read.","What this does not establish","GPQA-Diamond is four-way multiple choice on settled science. It measures what a pool knows and rewards a router that recognises a hard question; it says nothing about whether the answer is reasoned, whether the model holds up over a long autonomous run, or whether it can use a tool. The routing overhead is real work this table does not score either.","A benchmark score is not a capability.","Every row went through enso-bench's harness at the tag and on the date it shows; a system is the router or a single model as that run met it, not necessarily a preset the catalog serves today. What a row cost is the tokens its run records at today's list price for its own model id, the router's included, so two rows are priced by one rule.","A row is a run, as of its date.","Hanzo AI Cloud","Build on Hanzo Cloud","Start on Hanzo Cloud","View Models & Pricing","Previous","Engine prefill on CUDA vs llama.cpp","All Benchmarks Board","Next","LoCoMo","Other benchmarks","MemoryAgentBench","LongMemEval","CronQuestions","RepoBench-R","LoCoMo · subject scope","LoCoMo-Conv","Fleet residency","Live agent footprint","Sandbox cold start","every figure a research run at","· the dataset:","under","api.hanzo.ai"],"links":["/benchmarks","https://huggingface.co/datasets/Idavidrein/gpqa","https://arxiv.org/abs/2311.12022","https://api.hanzo.ai/v1/research/runs?org=hanzo&benchmark=enso.gpqa_diamond","https://api.hanzo.ai/v1/research/runs?org=hanzo&id=enso.gpqa_diamond-test-frontier-enso","https://api.hanzo.ai/v1/research/runs?org=hanzo&id=enso.gpqa_diamond-test-frontier-enso-ultra","https://api.hanzo.ai/v1/research/runs?org=hanzo&id=enso.gpqa_diamond-test-frontier-gpt-5.6-sol","https://api.hanzo.ai/v1/research/runs?org=hanzo&id=enso.gpqa_diamond-test-fill-opus-5.5","https://api.hanzo.ai/v1/research/runs?org=hanzo&id=enso.gpqa_diamond-test-fill-sonnet-5.5","https://api.hanzo.ai/v1/research/runs?org=hanzo&id=enso.gpqa_diamond-test-frontier-gpt-5.5","https://api.hanzo.ai/v1/research/runs?org=hanzo&id=enso.gpqa_diamond-test-fill-fable-5.1","https://api.hanzo.ai/v1/research/runs?org=hanzo&id=enso.gpqa_diamond-test-frontier-opus-4.8","https://api.hanzo.ai/v1/research/runs?org=hanzo&id=enso.gpqa_diamond-test-fill-haiku-5.5","https://api.hanzo.ai/v1/research/runs?org=hanzo&id=enso.gpqa_diamond-test-frontier-fable-5","https://api.hanzo.ai/v1/research/runs?org=hanzo&id=enso.gpqa_diamond-test-frontier-claude-5-sonnet","https://api.hanzo.ai/v1/research/runs?org=hanzo&id=enso.gpqa_diamond-test-frontier-deepseek-v4-pro","https://api.hanzo.ai/v1/research/runs?org=hanzo&id=enso.gpqa_diamond-test-frontier-gpt-4o","https://platform.hanzo.ai","/pricing","https://hanzo.ai/benchmarks/inference","https://hanzo.ai/benchmarks/locomo","https://hanzo.ai/benchmarks/memoryagentbench","https://hanzo.ai/benchmarks/longmemeval","https://hanzo.ai/benchmarks/cronquestions","https://hanzo.ai/benchmarks/repobench-r","https://hanzo.ai/benchmarks/locomo-subject-scope","https://hanzo.ai/benchmarks/locomo-conv","https://hanzo.ai/benchmarks/fleet","https://hanzo.ai/benchmarks/goroutine","https://hanzo.ai/benchmarks/sandbox"]}