mirror of
https://github.com/hanzoai/site.git
synced 2026-11-20 21:21:13 +00:00
platform.hanzo.ai is the platform console (with cloud.hanzo.ai redirecting to it at the edge). Point the remaining cloud.hanzo.ai links on /benchmarks and /philosophy directly at https://platform.hanzo.ai. Claude-Session: https://claude.ai/code/session_015XYKLix9PSyn6Jw2HWKprk
1 line
40 KiB
JSON
1 line
40 KiB
JSON
{"text":["Hanzo AI · Benchmarks · Last measured Oct 16, 2026","Intelligence, cost and speed, each measured.","Zen 6 prefills ~3,665 tokens a second and decodes ~70 on NVIDIA Grace-Blackwell, measured Nov 4, 2026.","The results","Every benchmark","16 of 16","Decisions · Decision Suite tasks won · Kai vs Jev","Every task of the suite, its row linking its runs and how to re-run it.","50% less","Cost · Input-token price · Kai vs Jev","$0.021 against $0.042 per million input tokens, as each is billed. A price, not a cost per task.","~3,665 tok/s","Speed · Prefill · Zen 6 on NVIDIA Grace-Blackwell","Decode ~70 tok/s · 141.2 tok/s with DFlash2 · 46.29× prefill & 78.57× decode at 1M context (99.8% KV pruned) · measured by Hanzo on Nov 4, 2026.","Enso","Kai","Zen 6","Agent harness","Agent platform","Methods","Jump to a benchmark","Enso · intelligence per dollar","Enso on the boards it was run on.","A New Pareto Frontier of Intelligence","GPQA Diamond challenges AI with 198 graduate-level science questions. See how Enso balances accuracy and cost across 1,000 answers.","@keyframes gpqa-dot-in { 0% { opacity: 0; transform: translateY(8px) scale(0.82); } 100% { opacity: 1; transform: translateY(0) scale(1); } } svg[data-plot][data-seen=\"false\"] [data-dot] { opacity: 0; } svg[data-plot][data-seen=\"true\"] [data-dot] { transform-box: fill-box; transform-origin: center; animation: gpqa-dot-in var(--dot-dur, 380ms) cubic-bezier(0.22, 1, 0.36, 1) var(--dot-delay, 0ms) both; } @media (prefers-reduced-motion: reduce) { svg[data-plot] [data-dot] { animation: none !important; opacity: 1 !important; transform: none !important; } }","75","80","85","90","95","100","$1","$3","$10","$30","GPQA Diamond %","cheaper ← $ per 1,000 questions → dearer","DeepSeek V4 Pro 0423 76.3%, a single model tested on the same 198 GPQA Diamond questions, $1.05 per 1,000 of them","DeepSeek V4 Pro 76.3%","Claude Sonnet 5 77.8%, a single model tested on the same 198 GPQA Diamond questions, $42.91 per 1,000 of them","Sonnet 5 77.8%","Claude Fable 5 82.8%, a single model tested on the same 198 GPQA Diamond questions, $14.95 per 1,000 of them","Fable 5 82.8%","Claude Haiku 5.5 83.3%, a single model tested on the same 198 GPQA Diamond questions, $1.07 per 1,000 of them","Haiku 5.5 83.3%","Claude Opus 4.8 87.4%, a single model tested on the same 198 GPQA Diamond questions, $45.01 per 1,000 of them","Opus 4.8 87.4%","Claude Fable 5.1 87.9%, a single model tested on the same 198 GPQA Diamond questions, $46.25 per 1,000 of them","Fable 5.1 87.9%","Claude Sonnet 5.5 90.4%, a single model tested on the same 198 GPQA Diamond questions, $10.41 per 1,000 of them","Sonnet 5.5 90.4%","GPT-5.5 90.4%, a single model tested on the same 198 GPQA Diamond questions, $18.64 per 1,000 of them","GPT-5.5 90.4%","GLM 5.3 90.9% on GPQA Diamond, reported by Epoch AI (independent), 2026-08-14; $0.945 per 1,000 questions estimated from its list price. https://epoch.ai/models/glm-5-3","GLM 5.3 90.9%","Claude Opus 5.5 90.9%, a single model tested on the same 198 GPQA Diamond questions, $21.23 per 1,000 of them","Opus 5.5 90.9%","GPT-5.6 Sol 91.9%, a single model tested on the same 198 GPQA Diamond questions, $10.08 per 1,000 of them","GPT-5.6 Sol 91.9%","Grok 4.7 92.7% on GPQA Diamond, reported by Epoch AI (independent), 2026-09-21; $4.57 per 1,000 questions estimated from its list price. https://epoch.ai/models/grok-4-7","Grok 4.7 92.7%","Kimi K3 93.5% on GPQA Diamond, reported by Moonshot AI (lab), 2026-07-16; $10.61 per 1,000 questions estimated from its list price. https://github.com/MoonshotAI/Kimi-K3","Kimi K3 93.5%","GPT-6 Sol 94.3% on GPQA Diamond, reported by Epoch AI (independent), 2026-09-22; $5.09 per 1,000 questions estimated from its list price. https://epoch.ai/models/gpt-6-sol","GPT-6 Sol 94.3%","Enso Flash at $3.26 per 1,000, its cost on new questions: 95.5% · questions it has seen","Enso Flash 95.5%","Enso at $10.03 per 1,000, its cost on new questions: 96.5% · questions it has seen","Enso 96.5%","Enso Ultra at $14.76 per 1,000, its cost on new questions: 98.0% · questions it has seen","Enso Ultra 98.0%","The frontier, one benchmark at a time","Accuracy against what the run cost at list prices, on a log axis, for each board where Enso’s router is on the frontier. Every point is a research run, read when the page opens, and its cost is the tokens it records on the whole question set at the price list’s rate for its own model id. The dashed line joins the runs no other run beats on both.","Benchmark","GPQA Diamond · 12 runs","LiveCodeBench · 12 runs","CSV","JSON","Copy link","50","55","60","65","70","$0.3","LIST-PRICE COST OF THE WHOLE SET, $, LOG SCALE","enso","opus-5.5","sonnet-5.5","haiku-5.5","One benchmark at a time: the frontier is drawn only among runs of the same benchmark, never across two. Filled points are Pareto-efficient — no other run scores higher for less. Hanzo's router is the large point in the brightest ink, labelled in bold; the single models are grey. A run's cost is the input and output tokens it records on the whole question set at the price list's rate for its own model id, the router's included, at list prices, Nov 7, 2026. Hover or tab to a point for its record.","The 12 runs as a table","System","Accuracy","Run cost","Tag, date","Pareto","92.9%","$5.58","frontier","2026-07-19","yes","gpt-5.6-sol","91.9%","$4.08","90.9%","$3.78","fill","2026-10-16","90.4%","$1.85","gpt-5.5","89.9%","$11.95","fable-5.1","87.9%","$8.23","opus-4.8","86.4%","$3.66","83.3%","$0.19","fable-5","82.8%","$9.87","claude-5-sonnet","77.8%","$5.66","deepseek-v4-pro","74.2%","$0.203","gpt-4o","54.5%","$1.58","Every system on GPQA Diamond","GPQA Diamond · enso.gpqa_diamond, split test · every row a research run, read from api.hanzo.ai when the page opens · sorted by accuracy","Correct","run","184 of 198","enso-ultra","$20.20","182 of 198","180 of 198","179 of 198","178 of 198","174 of 198","171 of 198","165 of 198","$0.190","164 of 198","154 of 198","147 of 198","108 of 198","Each system's frontier run, else its v1 run, else its fill run: the three put the same prompts to a model. A run cost is the input and output tokens the run records at the price list's rate for its own model id, the router's included, at list prices, Nov 7, 2026; a dash is a run that records no tokens, a model the list does not price, or a list that could not be read.","Kai · decisions","Kai wins 16 of 16 tasks of the Decision Suite against Jev.","Decision Suite: Kai wins 16 of 16 tasks against Jev","Each task was fixed on 2026-10-15. The frozen harness's suites Kai leads are below.","Task","Jev","AG News","fancyzhx/ag_news","kai-1.4 run","Jev run","re-run","93.8%","86.0%","DAIR Emotion","dair-ai/emotion","92.3%","60.3%","Banking77","mteb/banking77","88.0%","83.5%","Support triage","Tobi-Bueck/customer-support-tickets","46.3%","36.5%","Email spam","SetFit/enron_spam","98.5%","97.8%","Phishing","zefang-liu/phishing-email-dataset","98.8%","90.0%","Toxicity","lmsys/toxic-chat","81.0%","66.3%","RAG relevance","microsoft/ms_marco","65.3%","62.0%","Model routing","openai/gsm8k","google-research-datasets/mbpp","99.8%","97.7%","Typed decisions","LocalLLaMA/typed-decisions","74.6%","73.6%","MASSIVE, 51 languages, macro","mteb/amazon_massive_intent","90.7%","89.0%","Choice among 16 options","Its test cases, the gold intent and its fifteen nearest as the options.","92.0%","85.3%","Choice among 77 options","Its test cases, every intent an option.","84.3%","Same request three times, identical answers","support, cancellation and severity requests the suite generates","Identical: Kai 4,200 of 4,200, Jev 1,858 of 4,200.","100.0%","44.2%","False alarms on real benign prompts (in the wild), lower is better","TrustAIRLab/in-the-wild-jailbreak-prompts","Its regular prompts, each asked whether it is an attack. Flagged: Kai 83 of 600, Jev 126 of 600.","13.8%","21.0%","Cost & deployment, per million input tokens","the model list","Enterprise","GPU timing","CPU timing","Jev timing","Kai: $0.021 / M input tokens on api.hanzo.ai; on your own hardware (Enterprise): a cached decision in process in 0.3 ms (GPU) to 2 ms (CPU) — over 50× faster than a hosted call's 133 ms; a cold decision in 23–70 ms (2–6×). Jev: $0.042 / M; hosted API only, p50 133 ms per call.","$0.021 / M","$0.042 / M","Accuracy unless a row names its measure; the same questions to both models. Each row names its data and links its runs.","Re-run any row","hanzoai/benchmarks decision/SUITE.md","11","Suites won · Kai vs Jev","AG News, DAIR Emotion, Banking77, Support triage, Email spam, Phishing, Toxicity, RAG relevance, Model routing, Typed decisions, MASSIVE, 51 languages. Every row a research run of kai-1.4 and of Jev on the same questions; mean accuracy across these 11 suites 84.4% against 76.6%.","Kai against Jev, suite by suite","The suites Kai leads, both models on the same questions, each row dated by its runs. Sort by the lead, switch to calibration error, draw it as a chart or a table; the view is kept in the address, and the rows are read from the research API when the page opens.","suites won · mean accuracy across these","suites","against","84.4%","76.6%","Metric","Calibration error","View","Chart","Table","Sort","Largest lead","Name","Kai Oct 16, 2026 · Jev Sep 25, 2026","+32.0 pp","Win","+14.8 pp","+9.8 pp","+8.8 pp","+7.8 pp","+4.5 pp","+3.2 pp","+2.0 pp","MASSIVE, 51 languages","+1.7 pp","+1.0 pp","+0.7 pp","Every cell a research run of kai-1.4 and of Jev, on the same questions.","Accuracy: higher is better; a lead is in percentage points.","Suites measure different tasks, so their rows are compared within a row, not across.","Yes or no, read from the state","Production Kai answers a yes/no question with a reader trained for it. The tests below are the ones it leads Jev on, both models given the same questions and the same rows.","Test","A question and its paraphrase, mean |Δp| over 20 pairs","0.124","0.151","A state silent on the question, answered between 0.35 and 0.65: share of 60","81.7%","38.3%","Accuracy over 951 validation rows of 9 yes/no suites, pooled","88.3%","87.5%","Kai is kai-1.2, read 2026-10-08; Jev is typesafe/jev-1.13 through the API, read 2026-10-06. Hanzo-measured; harness private.","The trolley problem, decided on the counts","Kai does not weigh lives by preference. Its policy acts on the counts it grounds from the scene, and escalates to a person where it cannot ground them. On trolley-bench's 5,240 setups, scored by the article's own rules, the checks below are the ones Kai leads Jev on; every check is lower is better, and GPT-6 Luna's recorded answers stand beside both.","Check","GPT-6 Luna","Acted where acting kills more than it saves","17 of 672 (2.5%)","69 of 672 (10.3%)","143 of 672 (21.3%)","Killed a stranger to save the people, but would not destroy its own server","48 of 480 (10.0%)","180 of 480 (37.5%)","0 of 480 (0.0%)","Pulled the switch onto one person with nobody on the main track, of the wordings","0 of 24 (0.0%)","1 of 24 (4.2%)","24 of 24 (100.0%)","Kai is kai-1.8 through its nio/harm@1 policy, read 2026-10-20; Jev is typesafe/jev-1.13 and GPT-6 Luna is gpt-6-luna, each the benchmark's recorded answers. zachlandes/trolley-bench at cde97d7. Hanzo-scored by the article's rules; the scorer reproduces the article's numbers from the recorded answers.","Before any of it, Kai was qualified on 14,832 sealed trolley-structured scenes in two registers it had never read (quarry and piste), hashed before the model was frozen and read once. Every scene of the switch, server, robot, developer, and empty families, 8,300 in all, was decided right, with no wrong course; the crowd, hedged, infra, and vague families, where the harm cannot be counted, were escalated to a person every time.","Family","Scenes","Coverage","Wrong","switch","1,734","99.9%","server","2,400","robot","1,366","developer","1,600","empty","1,200","spelled","552","98.6%","bridge","300","58.0%","98.9%","ground-8, read 2026-10-20 at the qualified floor. Coverage is the share of scenes grounded; accuracy is of those; wrong counts the courses a grounded scene got wrong. Hanzo-measured; the sealed set is private.","Finding the right tool in a large catalog","Kai’s tool-relevance head, in development, ranks a shortlist of 32 tools drawn from the whole catalog. The shortlist is an open embedding model, zen-embedding-0.6B, with a small BM25 term; nothing is trained for it. Here it is scored against BM25 alone on a dev catalog Hanzo assembled from public tool-calling sets and its own API: 2,928 tools, 6,608 requests.","Every request","Recall@32, BM25","73.6","Recall@32, shortlist","93.6","nDCG@10, BM25","51.6","nDCG@10, shortlist","72.3","Top-1, BM25","41.4","Top-1, shortlist","55.5","6,608 requests · 2,928 tools · Recall@32: a right tool among the first 32","Requests for a Hanzo service","15.5","99.9","3.5","81.5","0.0","51.0","670 requests · a service scored as the best of its operations","An internal dev set, not a public benchmark: one run, 2026-10-11. Retrieval only, not an agent result.","Verified coding-agent performance","Hanzo Dev, with Kai and Enso Flash on api.hanzo.ai, against native Claude Code on held-out repository repairs. Each task is one fault its visible test fails on, and counts as solved only when a hidden verifier passes. Speed is the paired wall time over the tasks both solved.","3.02×","faster on the 42 tasks both solved","95% interval 2.06–4.36×.","Tasks both solved, by how Hanzo Dev settled them","Tasks","Speed","95% interval","Every task both solved","42","2.06–4.36×","Settled by search, no model call","24","8.22×","6.86–9.82×","The model’s first edit","1.31×","1.14–1.48×","Agent","Median seconds","Model calls a task","Claude Code, sonnet-5.5 default effort","9.5","3.8","Hanzo Dev, dev flow fix, Kai + enso-flash","2.3","1.0","The fastest tasks are settled by a search with no model call: it tries single-token edits of the kinds these faults are drawn from against the visible test alone, and never reads a hidden test or the fault. These results are specific to this benchmark, one attempt a task; they are not a claim about general-purpose coding.","Research runs of coding.repair on split held, ended 2026-10-15, harness commit 8e966be: GET /v1/research/runs?benchmark=coding.repair.","Agent tools","Tool handlers in Hanzo MCP, each run on the same cases as the handler it replaced. Each row says what decides it, rules, BM25 or Kai, as its run records. A rate is shown with its count.","Shell commands an agent should not run unasked","Split hermes: hermes-jev-skills evals/gate/fixtures.jsonl (MIT), 118 made-up commands. The rules were frozen and read once (1 of 65 allowed: a metadata-endpoint fetch); that class and a bare `rm -rf /` were fixed after, so this run is not held out","Handler","What decides","Allowed, should deny or ask","Ordinary, stopped","Must deny, denied","Median time","Current","Rules","0.0% (0 / 65)","34.0% (18 / 53)","54.3% (19 / 35)","0.060 ms","Replaced, ae4d07b","Rules, then Kai","18.5% (12 / 65)","62.3% (33 / 53)","14.3% (5 / 35)","Research runs of","mcp.command","on split","hermes",", ended","2026-10-15","; the current handler at commit","863cb8a","GET /v1/research/runs?org=hanzo&benchmark=mcp.command","Which registered tool a task needs","Split t1dev: supplied tool lists from Kai's T1 tools6 dev split (manifest 48be841f)","Right tool first","MRR","Abstained","Rules, then BM25","73.3% (220 / 300)","0.841","0.160 ms","65.7% (197 / 300)","0.805","43","mcp.select","t1dev","GET /v1/research/runs?org=hanzo&benchmark=mcp.select","Split hand: eight hand-labelled tasks over eight tools","50.0% (4 / 8)","0.590","0.100 ms","25.0% (2 / 8)","0.491","hand","What kind of failure a CI log shows","Split ci60: 60 real CI logs from 48 public repositories, labelled from the excerpt alone without the rules in view; nameable = the 40 whose class the old six categories can name","Right class","Right class, nameable","Unknown","70.0% (42 / 60)","70.0% (28 / 40)","0.370 ms","50.0% (20 / 40)","mcp.triage","ci60","GET /v1/research/runs?org=hanzo&benchmark=mcp.triage","Price","Input-token price · Kai vs Jev","$0","Per output token","A decision returns a choice, a score or a yes/no probability, and writes no text, so the price list bills no output.","Jev is typesafe/jev-1.13-20260917 through OpenRouter.","Ahead of Jev at 4, 16 and 77 options","Measure","Every option scored","Accuracy, 4 options","94.3%","90.5%","Accuracy, 16 options","Accuracy, 77 options","The same cases to both. Kai measured on CPU F32-Q8_0 (evo), the device production answers on.","Source","api.hanzo.ai/v1/research/runs?org=hanzo · capability · Kai kai-1.4@c1856a77 · commit 9492aa5 · measured 2026-10-16","The same request, answered three times","Requests whose three answers are identical","Every request of the decision-hijack suite sent three times to each model; identical when all three agree on the decision and its probabilities. Kai on CPU, the device production answers on; Jev through its API.","api.hanzo.ai/v1/research/runs?org=hanzo · stability · Kai kai-1.4@c1856a77 · commit 9492aa5 · measured 2026-10-16","The frozen harness, the suites Kai leads","The frozen harness: Kai leads Jev on 11 suites","Suite","Mean, these 11 suites","The suites of the frozen harness Kai leads, each a research run of kai-1.4 and of Jev on the same questions.","api.hanzo.ai/v1/research/runs?org=hanzo · harness · Kai kai-1.4@c1856a77 · harness 5f616dd · measured 2026-10-16","The languages of the multilingual suite Kai leads on","MASSIVE: ahead of Jev on 31 languages","Language","af","87.0%","am","85.0%","ar","az","94.0%","bn","cy","76.0%","da","91.0%","en","95.0%","fi","96.0%","fr","hy","84.0%","is","jv","75.0%","km","kn","lv","mn","ms","93.0%","my","nb","nl","pt","ro","ru","98.0%","sl","sq","sv","sw","83.0%","70.0%","tl","vi","zh-TW","The languages of the multilingual suite Kai leads, each as MASSIVE names it, the same questions to both.","Zen 6 · speed on our own hardware & Latent Sparse Attention","Zen 6 prefills ~3,665 tokens a second on NVIDIA Grace-Blackwell and scales 72.40× faster at 10M tokens with Latent Sparse Attention.","Prefill · Zen 6 on NVIDIA Grace-Blackwell","Prefill, tokens a second","NVIDIA Grace-Blackwell · NVFP4 + Latent Sparse Attention (D-LSA)","AMD Strix Halo · 4-bit build","1,150–1,452 tok/s","Apple silicon · MLX build","500–690 tok/s","the low end of a range is the bar","Decode, tokens a second","~70 tok/s","~37 tok/s","64–81 tok/s","one request","72.40×","Latent Sparse Attention (D-LSA) prefill speedup · 10M tokens on NVIDIA H200","42.50× decode speedup at 10M tokens · 46.29× (26.40 ms) prefill and 78.57× (0.167 ms) decode at 1M tokens on NVIDIA Grace-Blackwell GB10 (99.8% KV pruned) · RULER 95.2% vs 95.4% full attention · N-LSA 1.6× faster training at 128K.","Latent Sparse Attention (D-LSA) prefill speedup","10M tokens · NVIDIA H200 · D-LSA (Tree)","1M tokens · NVIDIA Grace-Blackwell GB10 · D-LSA (Tree)","46.29× (26.40 ms)","512K tokens · NVIDIA Grace-Blackwell GB10 · D-LSA (Tree)","35.75× (17.09 ms)","256K tokens · NVIDIA Grace-Blackwell GB10 · D-LSA (Tree)","27.50× (11.25 ms)","128K tokens · NVIDIA Grace-Blackwell GB10 · D-LSA / N-LSA","23.72× (6.46 ms)","64K tokens · NVIDIA Grace-Blackwell GB10 · D-LSA (Flat)","12.17× (6.71 ms)","32K tokens · NVIDIA Grace-Blackwell GB10 · D-LSA (Flat)","8.58× (4.28 ms)","8K tokens (Hanzo Engine live) · NVIDIA Grace-Blackwell GB10 · D-LSA GPU Cache","5.53× (3,665.5 tok/s)","against full attention","Latent Sparse Attention (D-LSA) decode speedup","42.50×","78.57× (0.167 ms)","42.75× (0.154 ms)","23.35× (0.157 ms)","12.77× (0.152 ms)","6.46× (0.111 ms)","4.39× (0.135 ms)","4.92× (70.0 tok/s)","Zen 6: Subquadratic Ultra-Long Context via Hierarchical Latent Sparse Attention","· what the agent remembers and finds","On LoCoMo, Hanzo's context engine finds 2.0× the multi-hop evidence cosine does.","2.0×","Multi-hop ALL@20 · LoCoMo","37.5% against cosine's 18.8% on the test split: every annotated evidence turn in the top 20.","Memory and retrieval","What a memory finds in a long conversation, and what a reader does with it.","Each panel is one metric, the baseline and Hanzo on the same data, the same embedder and the same cut-off, drawn on one scale.","MemoryAgentBench","single-hop, plain resolver","95.5","single-hop, typed search","98.3","multi-hop, plain resolver","56.8","multi-hop, typed search","84.5","262k multi-hop, plain","33.0","262k multi-hop, typed","67.0","substring EM · pooled over the dev haystack and the three test haystacks, and the largest alone · reader: none, embedder zenlm/zen-embedding-0.6b","LoCoMo","multi-hop ALL@20, cosine","18.8","multi-hop ALL@20, Hanzo","37.5","R@20, cosine","70.9","R@20, Hanzo","80.7","test split · zen-embedding-0.6b · k=20 · ALL = every annotated evidence turn in the top 20","LongMemEval","cosine","85.8","engine, frozen on LoCoMo","90.6","500 questions · all-minilm vectors for both · ALL@5 = every evidence session in the top 5","Knowledge graph","Point-in-time questions over facts that begin and end, against another temporal store on the same data.","On the CronQuestions types the harness covers, the Hanzo graph answers 99.88% and Semantica 99.88%; what differs is the price of a question.","0.063 ms","Hanzo graph, through the API","median question · Hits@1 99.88%","5,357 ms","Semantica temporal query","Code","Cross-file retrieval in a repository, scored apart from generation.","Typed links are regular expressions over imports, definitions and identifiers; no model reads anything in that row.","RepoBench-R","dense retrieval","typed links","31.4","cross-file-first · test split · R@1","Agent Code Intelligence: The Race","How Hanzo Dev and Hanzo MCP accelerate agentic code comprehension compared to traditional raw-file reading and containerized local RAG stacks (Qdrant + Ollama). Benchmarked across production multi-language repositories.","Subsystem Logic (State Tracking & Auth)","Function Signature & Implementation","API Surface & Route Discovery","Claude Code Acceleration: Zero-Model Fault Repair (Kai)","Claude Code Acceleration: Atomic AST Refactoring","TARGET QUERY","Workspace state diffing & hashing logic","Run Race Simulation","HANZO NATIVE","hanzo dev + hanzo-mcp","0.7 ms","1 turn","398","tokens (","98.4% saved","In-process Trigram + Tree-Sitter AST context bundle (0 daemons)","Zero Docker, Zero background model daemons","⚙️","LOCAL RAG","Qdrant Container + Ollama (phi3 / mxbai)","202 ms","382 tokens","Qdrant Vector DB container + Ollama mxbai-embed-large (10 GB)","Requires 16 GB RAM + Docker daemon","🐌","TRADITIONAL AGENT","Claude Code / Cursor (Raw Grep + File Reads)","36.4 s","8 turns","0 tokens","Iterative grep + full file reads over stdio (8 roundtrips to LLM)","Burning prompt tokens...","Architecture","Latency","Tool Turns","Tokens Injected","Token Savings","Speedup","Traditional Agent (Grep + Whole File Reads)","25,078","Baseline","1.0×","Containerized Local RAG (Qdrant + Ollama)","382","180.0×","Hanzo Native (`hanzo dev` / `hanzo code claude`)","98.4%","100×+","In a real agentic workflow (e.g. Claude Code, Cursor, Windsurf), each tool roundtrip incurs an additional 3–5 seconds of LLM inference latency. By replacing 6–13 iterative file exploration roundtrips with Hanzo’s in-process AST indexing (`code_context`, `code_tool`, `hanzo_refactor`) and Kai deterministic test-driven search, Hanzo Dev accelerates Claude Code by",", while slashing token consumption by","20× to 50× (and up to 137× on zero-model search-settled faults)","90% to 100%","Research runs on benchmarks claude-acceleration and code-rag-navigation, harness commit hanzoai/benchmarks:","GET /v1/research/runs?org=hanzo&benchmark=claude-acceleration","GET /v1/research/runs?org=hanzo&benchmark=code-rag-navigation","GET /v1/benchmark/leaderboard?benchmark=code-rag-navigation","· what an agent costs","A dormant agent costs 477 bytes.","477 bytes","State per dormant agent","Against a published ~1 MB (2,198×). 1M agents: 455 MB on disk vs ~977 GB.","Platform, agents, and sandboxes","What an agent costs at rest, what a live one costs to wake, and what a sandbox costs to start.","601 bytes","Heap per live agent, a goroutine","125 ns","Spawn one goroutine","107 ms — 107 ns each","Wake one million goroutines","A live agent here is a goroutine: it shares one address space with every other, and nothing about it is an isolation boundary. Code that has to be isolated gets a microVM. The goroutine timings are the median of one pass on an Apple M4 Max, 16 cores. Every figure here is transcribed from the harness's README rather than read from a run file, and each page names the command that prints it.","Start-up and cost","0.034 ms","Resume one dormant agent","In process, from its stored state.","149.6 ms","A sandbox that runs a shell, cold","pooled 35.8 ms; published <200 ms · ~1 s · 1–3 s.","$0.00000063","One call, marginal cost","Against a published $0.05 credit (79,614×).","How every number was taken","Methodology","Each build pulls the latest validated runs from the research API: canonical, public, and not flagged inconsistent. When the API cannot be read, or answers with fewer runs than the committed snapshot already holds, the build keeps the snapshot, and the page swaps in what the API holds once it answers in the browser.","Every result is current.","A run is dated by when it ended; the platform bench records its month.","Every row is dated.","Live is the newest run the API serves for a benchmark. Frozen is a committed run file, a snapshot, or a figure transcribed from a harness’s README. Superseded is an earlier run of the same benchmark that a later one replaced; it stays in the library, so a revision’s history can be read.","Live, frozen, superseded.","Kai’s and Enso’s rows are research runs, served at","and read when this page opens. The memory, graph and code panels read the committed run files of the benchmarks repository. Kai’s yes/no table and the tool shortlist are Hanzo-measured reads the research API does not file: each is its run’s report, copied unchanged, and names its model, date and count.","Where a number comes from.","the research API","Every frontier point is a run Hanzo filed: the router and each single model on the same questions, at the same tag. No vendor-reported figure is on a board here.","Measured, not reported.","Kai beside Jev is the input-token rate each is billed, from the price list, not a cost per task. The frontier’s axis is what each run spent on the whole question set. No speed is claimed here without a measured comparison; latency, where this Kai version was timed, is in its tables.","Prices are prices.","A frontier is drawn among the rows of one benchmark, never across two, and a leaderboard row compares two models on one suite. Suites measure different tasks, so the mean is a summary, not a score on any of them.","Comparing rows.","The memory and code panels regenerate from the committed runs on a fresh clone:","git clone https://github.com/hanzoai/benchmarks && cd benchmarks/brain && node results.mjs","The research","Every paper the library publishes. A benchmark’s own page names the paper that reports it, where one does. Specifications are the","HIPs","papers.hanzo.ai","Distilled & Native Latent Sparse Attention (D-LSA / N-LSA) on 27B Dense Transformers","Kai: Calibrated Discrete Decision Intelligence for Autonomous Agent Loops","Factorized Discrete Choice, Calibrated Distributions, and Deterministic Policy Governance","Linkage, Not Similarity","What long-term agent memory has to retrieve, and how it is measured","Hanzo AI Chain (AIVM)","Useful-Work Mining, the Native AI Coin, and Post-Quantum Omnichain Settlement","Cloud Economics of a Sovereign OSS Stack","A Regulated Capital-Markets Migration Case Study","One Native Stack for Private, Continuously-Learning AI","Inference at the Bandwidth Wall, On-Device QLoRA, One Engine","Native ROCm Inference on a Consumer RDNA3.5 APU","Reaching llama.cpp Decode Parity via a Unified 1-bit-to-Full Quant Core","Edge LLM Inference Across Commodity Accelerators","Measured AMD, NVIDIA, and Apple Performance","Native Training in the Hanzo Engine","One Engine, One Quant Format, One Backend","Continuously-Learning Private AI","Self-Improvement and Privacy in the Hanzo Native Stack","Economical Serving and the hanzo.ai Platform","A Bandwidth-Optimal Unified Train-and-Serve Runtime","The Unified Sovereign-Tenant Cloud","Linear Shared-Nothing Scaling via Per-Tenant SQLite and In-Process Composition","Refutation-Driven Performance Engineering","An Empirical Study of a Multi-Week GPU Kernel Campaign","Evolutionary Schedule Search in a One-Source Kernel DSL","And the Continual-Improvement Loop It Anchors","Hanzo Router","Memory-Aware, Local-First LLM Routing with a Learned SLO-Constrained Policy","What a Dormant Agent Costs","State, Execution and Isolation Measured Separately at a Fleet of One Million","Hops Do Not Fix Multi-Hop Retrieval","A Negative Result on Long-Horizon Conversational Memory, and What the Failure Actually Is","Retrieval Scoped by Subject","A factorial measurement of conversational memory on LoCoMo, and the negative controls that locate the effect","Hanzo Network Whitepaper","L1 Blockchain for Decentralized AI Compute","Every benchmark, every run","One shelf for everything above, by the same five products: each Kai revision on each suite it leads, each Enso tier on the frontiers it holds, each Zen 6 reading, each memory and platform pass. Search it, filter it, pick rows of one benchmark to compare, and open any of them for its history, conditions, hardware, source and the command that reproduces it.","Order","Newest","Value","Views","Live results","Kai, every revision","Superseded runs","Zen 6 and the engine","Not measured by Hanzo","Filters","Product","120","26","12","Category","Decision harness","115","Throughput & subquadratic long-context attention","23","Capability","Cost and start-up","Engine prefill vs llama.cpp","2 more","67","Calibration error (ECE)","51","D-LSA Decode speedup","D-LSA Prefill speedup","Decode throughput","Prefill throughput","Prefill throughput vs llama.cpp","substring EM","20 more","Measured by","Hanzo","166","Date","Nov 2026","Oct 2026","Sep 2026","98","Jul 2026","Status","Live","48","Frozen","Superseded","Result","146","Unscored","20","of","results,","benchmarks","Zen 6 throughput & Latent Sparse Attention","Zen 6 · D-LSA (Tree) · 10M tokens (NVIDIA H200)","vs","Full Attention","Zen 6 · Throughput & subquadratic long-context attention · D-LSA Prefill speedup","baseline 1.00×","Nov 4, 2026","Hanzo-measured","Zen 6 · Throughput & subquadratic long-context attention · D-LSA Decode speedup","Zen 6 · D-LSA (Tree) · 1M tokens (NVIDIA Grace-Blackwell GB10)","baseline 1.00× (1,222.10 ms)","baseline 1.00× (13.105 ms)","Zen 6 · D-LSA (Tree) · 512K tokens (NVIDIA Grace-Blackwell GB10)","baseline 1.00× (610.86 ms)","baseline 1.00× (6.586 ms)","Zen 6 · D-LSA (Tree) · 256K tokens (NVIDIA Grace-Blackwell GB10)","baseline 1.00× (309.25 ms)","baseline 1.00× (3.670 ms)","Zen 6 · D-LSA / N-LSA · 128K tokens (NVIDIA Grace-Blackwell GB10)","baseline 1.00× (153.29 ms)","baseline 1.00× (1.944 ms)","Zen 6 · D-LSA (Flat) · 64K tokens (NVIDIA Grace-Blackwell GB10)","baseline 1.00× (81.62 ms)","baseline 1.00× (0.716 ms)","Zen 6 · D-LSA (Flat) · 32K tokens (NVIDIA Grace-Blackwell GB10)","baseline 1.00× (36.70 ms)","baseline 1.00× (0.593 ms)","Zen 6 · D-LSA GPU Cache · 8K tokens (Hanzo Engine live) (NVIDIA Grace-Blackwell GB10)","baseline 1.00× (662.3 tok/s)","baseline 1.00× (14.2 tok/s)","Zen 6 · N-LSA (Native) · 128K tokens","Zen 6 · Throughput & subquadratic long-context attention · N-LSA Training speedup","1.60× per layer","Zen 6 · NVIDIA Grace-Blackwell","Zen 6 · Throughput & subquadratic long-context attention · Prefill throughput","no baseline","Zen 6 · Throughput & subquadratic long-context attention · Decode throughput","Zen 6 · AMD Strix Halo","Zen 6 · Apple silicon","kai-1.4 · c1856a77","Kai · Decision harness · Accuracy","baseline 86.0%","Oct 16, 2026","Show all 166","Every benchmark page","GPQA-Diamond","filter the shelf to it","MASSIVE","Choosing among many options","Latency of one decision","Engine prefill on CUDA vs llama.cpp","CronQuestions","LoCoMo · subject scope","LoCoMo-Conv","Platform bench","Fleet residency","Live agent footprint","Sandbox cold start","Hanzo AI Cloud","Build on Hanzo Cloud","Start on Hanzo Cloud","View Models & Pricing"],"links":["#enso","#all","#decision-suite","/pricing","#zen","#kai","#harness","#platform","#methods","https://epoch.ai/models/glm-5-3","https://epoch.ai/models/grok-4-7","https://github.com/MoonshotAI/Kimi-K3","https://epoch.ai/models/gpt-6-sol","https://api.hanzo.ai/v1/research/runs?org=hanzo&id=enso.gpqa_diamond-test-frontier-enso","https://api.hanzo.ai/v1/research/runs?org=hanzo&id=enso.gpqa_diamond-test-frontier-enso-ultra","https://api.hanzo.ai/v1/research/runs?org=hanzo&id=enso.gpqa_diamond-test-frontier-gpt-5.6-sol","https://api.hanzo.ai/v1/research/runs?org=hanzo&id=enso.gpqa_diamond-test-fill-opus-5.5","https://api.hanzo.ai/v1/research/runs?org=hanzo&id=enso.gpqa_diamond-test-fill-sonnet-5.5","https://api.hanzo.ai/v1/research/runs?org=hanzo&id=enso.gpqa_diamond-test-frontier-gpt-5.5","https://api.hanzo.ai/v1/research/runs?org=hanzo&id=enso.gpqa_diamond-test-fill-fable-5.1","https://api.hanzo.ai/v1/research/runs?org=hanzo&id=enso.gpqa_diamond-test-frontier-opus-4.8","https://api.hanzo.ai/v1/research/runs?org=hanzo&id=enso.gpqa_diamond-test-fill-haiku-5.5","https://api.hanzo.ai/v1/research/runs?org=hanzo&id=enso.gpqa_diamond-test-frontier-fable-5","https://api.hanzo.ai/v1/research/runs?org=hanzo&id=enso.gpqa_diamond-test-frontier-claude-5-sonnet","https://api.hanzo.ai/v1/research/runs?org=hanzo&id=enso.gpqa_diamond-test-frontier-deepseek-v4-pro","https://api.hanzo.ai/v1/research/runs?org=hanzo&id=enso.gpqa_diamond-test-frontier-gpt-4o","https://huggingface.co/datasets/fancyzhx/ag_news","https://api.hanzo.ai/v1/research/runs?org=hanzo&id=jev.ag_news-harness-kai-1.4.c1856a77-kai","https://api.hanzo.ai/v1/research/runs?org=hanzo&id=jev.ag_news-harness-20260917-jev","https://github.com/hanzoai/benchmarks/tree/main/decision","https://huggingface.co/datasets/dair-ai/emotion","https://api.hanzo.ai/v1/research/runs?org=hanzo&id=jev.emotion-harness-kai-1.4.c1856a77-kai","https://api.hanzo.ai/v1/research/runs?org=hanzo&id=jev.emotion-harness-20260917-jev","https://huggingface.co/datasets/mteb/banking77","https://api.hanzo.ai/v1/research/runs?org=hanzo&id=jev.banking77_full-harness-kai-1.4.c1856a77-kai","https://api.hanzo.ai/v1/research/runs?org=hanzo&id=jev.banking77_full-harness-20260917-jev","https://huggingface.co/datasets/Tobi-Bueck/customer-support-tickets","https://api.hanzo.ai/v1/research/runs?org=hanzo&id=app.support_triage-harness-kai-1.4.c1856a77-kai","https://api.hanzo.ai/v1/research/runs?org=hanzo&id=app.support_triage-harness-20260917-jev","https://huggingface.co/datasets/SetFit/enron_spam","https://api.hanzo.ai/v1/research/runs?org=hanzo&id=app.email_spam-harness-kai-1.4.c1856a77-kai","https://api.hanzo.ai/v1/research/runs?org=hanzo&id=app.email_spam-harness-20260917-jev","https://huggingface.co/datasets/zefang-liu/phishing-email-dataset","https://api.hanzo.ai/v1/research/runs?org=hanzo&id=app.phishing-harness-kai-1.4.c1856a77-kai","https://api.hanzo.ai/v1/research/runs?org=hanzo&id=app.phishing-harness-20260917-jev","https://huggingface.co/datasets/lmsys/toxic-chat","https://api.hanzo.ai/v1/research/runs?org=hanzo&id=app.moderation_toxicity-harness-kai-1.4.c1856a77-kai","https://api.hanzo.ai/v1/research/runs?org=hanzo&id=app.moderation_toxicity-harness-20260917-jev","https://huggingface.co/datasets/microsoft/ms_marco","https://api.hanzo.ai/v1/research/runs?org=hanzo&id=app.rag_relevance-harness-kai-1.4.c1856a77-kai","https://api.hanzo.ai/v1/research/runs?org=hanzo&id=app.rag_relevance-harness-20260917-jev","https://huggingface.co/datasets/openai/gsm8k","https://huggingface.co/datasets/google-research-datasets/mbpp","https://api.hanzo.ai/v1/research/runs?org=hanzo&id=app.model_routing_domain-harness-kai-1.4.c1856a77-kai","https://api.hanzo.ai/v1/research/runs?org=hanzo&id=app.model_routing_domain-harness-20260917-jev","https://huggingface.co/datasets/LocalLLaMA/typed-decisions","https://api.hanzo.ai/v1/research/runs?org=hanzo&id=typed_decisions-harness-kai-1.4.c1856a77-kai","https://api.hanzo.ai/v1/research/runs?org=hanzo&id=typed_decisions-harness-20260917-jev","https://huggingface.co/datasets/mteb/amazon_massive_intent","https://api.hanzo.ai/v1/research/runs?org=hanzo&id=massive-harness-kai-1.4.c1856a77-kai","https://api.hanzo.ai/v1/research/runs?org=hanzo&id=massive-harness-20260917-jev","https://api.hanzo.ai/v1/research/runs?org=hanzo&id=cap.cardinality-cases-kai-1.4.c1856a77-kai","https://api.hanzo.ai/v1/research/runs?org=hanzo&id=cap.cardinality-cases-jev-1.13-jev","https://github.com/hanzoai/benchmarks/tree/main/decision/capability","https://api.hanzo.ai/v1/research/runs?org=hanzo&id=decision.stability-all-hijack-kai-1.4","https://api.hanzo.ai/v1/research/runs?org=hanzo&id=decision.stability-all-hijack-jev-1.13","https://github.com/hanzoai/benchmarks/tree/main/decision/stability","https://huggingface.co/datasets/TrustAIRLab/in-the-wild-jailbreak-prompts","https://api.hanzo.ai/v1/research/runs?org=hanzo&id=jailbreak.wild-clean-h-kai-1.4","https://api.hanzo.ai/v1/research/runs?org=hanzo&id=jailbreak.wild-clean-h-jev-1.13","https://github.com/hanzoai/benchmarks/tree/main/decision/jailbreak-suite","https://api.hanzo.ai/v1/models","/enterprise","https://api.hanzo.ai/v1/research/runs?org=hanzo&id=decision.latency-window2-dgx.cuda.f32.q8.tall-kai","https://api.hanzo.ai/v1/research/runs?org=hanzo&id=decision.latency-window2-evo.cpu.f32.q8.t8-kai","https://api.hanzo.ai/v1/research/runs?org=hanzo&id=decision.latency-network-n1-jev-1.13","https://github.com/hanzoai/benchmarks/blob/main/decision/SUITE.md","#leaderboard","#shortlist","#coding-agent","https://api.hanzo.ai/v1/research/runs?org=hanzo&benchmark=mcp.command","https://api.hanzo.ai/v1/research/runs?org=hanzo&benchmark=mcp.select","https://api.hanzo.ai/v1/research/runs?org=hanzo&benchmark=mcp.triage","https://github.com/hanzoai/benchmarks/tree/5f616dd13167bc407bd1a56645c0982f297e9e4a/decision","/benchmarks/zen-6","https://papers.hanzo.ai/zen6-latent-sparse-attention/","/benchmarks/locomo","/benchmarks/memoryagentbench","/benchmarks/longmemeval","/benchmarks/cronquestions","/benchmarks/repobench-r","https://api.hanzo.ai/v1/research/runs?org=hanzo&benchmark=claude-acceleration","https://api.hanzo.ai/v1/research/runs?org=hanzo&benchmark=code-rag-navigation","https://api.hanzo.ai/v1/benchmark/leaderboard?benchmark=code-rag-navigation","/benchmarks/platform-bench","/benchmarks/fleet","/benchmarks/goroutine","https://api.hanzo.ai/v1/research/runs?org=hanzo","https://hips.hanzo.ai","https://papers.hanzo.ai/kai-decision-model/","https://papers.hanzo.ai/hanzo-context-memory/","https://papers.hanzo.ai/hanzo-ai-chain/","https://papers.hanzo.ai/hanzo-cloud-economics/","https://papers.hanzo.ai/hanzo-native-stack-thesis/","https://papers.hanzo.ai/hanzo-rocm-inference/","https://papers.hanzo.ai/hanzo-cross-backend-inference/","https://papers.hanzo.ai/hanzo-native-training/","https://papers.hanzo.ai/hanzo-continuous-learning-privacy/","https://papers.hanzo.ai/hanzo-platform-infra-desktop/","https://papers.hanzo.ai/hanzo-unified-tenant-cloud/","https://papers.hanzo.ai/refutation-driven-perf/","https://papers.hanzo.ai/evolutionary-schedule-search/","https://papers.hanzo.ai/hanzo-router/","https://papers.hanzo.ai/hanzo-dormant-agents/","https://papers.hanzo.ai/hanzo-multi-hop-retrieval/","https://papers.hanzo.ai/locomo-subject-scope/","https://papers.hanzo.ai/network-whitepaper/","?state=live#all","?product=Kai&category=Decision%20harness&measure=Accuracy&order=name#all","?state=superseded#all","?product=Zen%206#all","?by=Vendor&by=Board#all","/benchmarks/kai-ag-news","?product=Enso#all","/benchmarks/gpqa","?product=Kai#all","/benchmarks/kai-dair-emotion","/benchmarks/kai-banking77","/benchmarks/kai-support-triage","/benchmarks/kai-email-spam","/benchmarks/kai-phishing","/benchmarks/kai-toxicity","/benchmarks/kai-rag-relevance","/benchmarks/kai-model-routing","/benchmarks/kai-typed-decisions","/benchmarks/kai-massive","/benchmarks/kai-choice-size","/benchmarks/kai-latency","/benchmarks/inference","?product=Agent%20harness#all","/benchmarks/locomo-subject-scope","/benchmarks/locomo-conv","?product=Agent%20platform#all","/benchmarks/sandbox","https://platform.hanzo.ai"]} |