mirror of
https://github.com/hanzoai/site.git
synced 2026-11-21 00:46:26 +00:00
platform.hanzo.ai is the platform console (with cloud.hanzo.ai redirecting to it at the edge). Point the remaining cloud.hanzo.ai links on /benchmarks and /philosophy directly at https://platform.hanzo.ai. Claude-Session: https://claude.ai/code/session_015XYKLix9PSyn6Jw2HWKprk
206 lines
4.7 KiB
JSON
206 lines
4.7 KiB
JSON
[
|
||
[
|
||
{
|
||
"p": "Benchmarks / MemoryAgentBench"
|
||
},
|
||
{
|
||
"a": "Benchmarks",
|
||
"href": "/benchmarks"
|
||
},
|
||
{
|
||
"h1": "MemoryAgentBench"
|
||
},
|
||
{
|
||
"p": "FactConsolidation: haystacks of 6k to 262k tokens where a later line silently overwrites an earlier one. A typed search over parsed facts resolves the question with no language model reading anything."
|
||
},
|
||
{
|
||
"a": "MemoryAgentBench",
|
||
"href": "https://huggingface.co/datasets/ai-hyz/MemoryAgentBench"
|
||
},
|
||
{
|
||
"a": "Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactions",
|
||
"href": "https://arxiv.org/abs/2507.05257"
|
||
},
|
||
{
|
||
"a": "Linkage, Not Similarity",
|
||
"href": "https://github.com/hanzoai/papers/blob/main/site/public/pdfs/hanzo-context-memory.pdf"
|
||
},
|
||
{
|
||
"code": "node results.mjs"
|
||
},
|
||
{
|
||
"p": "regenerates every table on this page from the raw runs"
|
||
},
|
||
{
|
||
"h2": "What is measured"
|
||
},
|
||
{
|
||
"p": "FactConsolidation. A haystack is a numbered list of templated facts in which a later line silently replaces an earlier one about the same subject and relation, and nothing marks the update. Questions ask for the current value, single-hop and multi-hop, over haystacks of 6k to 262k tokens. One hundred questions per haystack. The metric is substring exact match, the benchmark’s own; an unanswered question scores zero. Dev is the 6k haystacks, test is 32k, 64k and 262k."
|
||
},
|
||
{
|
||
"p": "No language model reads the memory or writes the answer in any row here. The question is compiled into a plan sketch and a deterministic typed search over the parsed fact store resolves it — 45 templates, no line unparsed at any size, the supersession chain keyed by subject and relation. That is why these rows are not comparable to a pipeline that reads the haystack with a frontier model, and it is also the point: the retrieval is doing the work."
|
||
},
|
||
{
|
||
"h2": "Results"
|
||
},
|
||
{
|
||
"p": "substring exact match · 6k from dev, 32k / 64k / 262k from test · no reader · pooled = the mean of the four haystacks"
|
||
},
|
||
{
|
||
"h3": "single-hop"
|
||
},
|
||
{
|
||
"table": [
|
||
[
|
||
"configuration",
|
||
"6k",
|
||
"32k",
|
||
"64k",
|
||
"262k",
|
||
"pooled"
|
||
],
|
||
[
|
||
"plain resolver",
|
||
"96.0",
|
||
"97.0",
|
||
"95.0",
|
||
"94.0",
|
||
"95.5"
|
||
],
|
||
[
|
||
"typed search",
|
||
"100.0",
|
||
"98.0",
|
||
"98.0",
|
||
"97.0",
|
||
"98.3"
|
||
]
|
||
]
|
||
},
|
||
{
|
||
"h3": "multi-hop"
|
||
},
|
||
{
|
||
"table": [
|
||
[
|
||
"configuration",
|
||
"6k",
|
||
"32k",
|
||
"64k",
|
||
"262k",
|
||
"pooled"
|
||
],
|
||
[
|
||
"plain resolver",
|
||
"76.0",
|
||
"59.0",
|
||
"59.0",
|
||
"33.0",
|
||
"56.8"
|
||
],
|
||
[
|
||
"typed search",
|
||
"93.0",
|
||
"89.0",
|
||
"89.0",
|
||
"67.0",
|
||
"84.5"
|
||
]
|
||
]
|
||
},
|
||
{
|
||
"p": "Typed search pools to 98.3 single-hop and 84.5 multi-hop against the plain resolver’s 95.5 and 56.8. The largest single move is 262k multi-hop, from 33.0 to 67.0."
|
||
},
|
||
{
|
||
"h2": "Numbers other systems publish"
|
||
},
|
||
{
|
||
"p": "Quoted with their protocols, because the protocols differ, and none of them re-run here. CAR (arXiv 2606.01435) reports 78.0 single-hop / 30.2 multi-hop pooled over 6k–262k with gpt-4o-mini and 94.8 / 51.5 with gpt-4o, and 82 / 27 and 93 / 41 respectively at 262k. Its own baselines at 262k multi-hop: HippoRAG-v2 5, gpt-4o whole-context 5, BM25 3, Cognee and MemGPT 3, Mem0 2, Zep 3. That is the nearest thing to a like-for-like comparison anywhere on this site — same data, same split, same substring-EM metric — and it is still not one, because CAR reads the haystack with a frontier model and the rows above use no model at all."
|
||
},
|
||
{
|
||
"p": "Pith publishes per-lane multi-hop figures on a vendor page: 83 / 84 at 32k and 64k, where typed search scores 89.0 / 89.0."
|
||
},
|
||
{
|
||
"p": "Hanzo AI Cloud"
|
||
},
|
||
{
|
||
"h3": "Build on Hanzo Cloud"
|
||
},
|
||
{
|
||
"a": "Start on Hanzo Cloud",
|
||
"href": "https://platform.hanzo.ai"
|
||
},
|
||
{
|
||
"a": "View Models & Pricing",
|
||
"href": "/pricing"
|
||
},
|
||
{
|
||
"a": "Previous LoCoMo",
|
||
"href": "/benchmarks/locomo"
|
||
},
|
||
{
|
||
"a": "All Benchmarks Board",
|
||
"href": "/benchmarks"
|
||
},
|
||
{
|
||
"a": "Next LongMemEval",
|
||
"href": "/benchmarks/longmemeval"
|
||
},
|
||
{
|
||
"h3": "Related in memory"
|
||
},
|
||
{
|
||
"a": "LoCoMo",
|
||
"href": "/benchmarks/locomo"
|
||
},
|
||
{
|
||
"a": "LongMemEval",
|
||
"href": "/benchmarks/longmemeval"
|
||
},
|
||
{
|
||
"a": "LoCoMo · subject scope",
|
||
"href": "/benchmarks/locomo-subject-scope"
|
||
},
|
||
{
|
||
"a": "LoCoMo-Conv",
|
||
"href": "/benchmarks/locomo-conv"
|
||
},
|
||
{
|
||
"p": "CronQuestions · RepoBench-R · Fleet residency · Live agent footprint · Sandbox cold start · Engine prefill on CUDA vs llama.cpp · GPQA-Diamond"
|
||
},
|
||
{
|
||
"a": "CronQuestions",
|
||
"href": "/benchmarks/cronquestions"
|
||
},
|
||
{
|
||
"a": "RepoBench-R",
|
||
"href": "/benchmarks/repobench-r"
|
||
},
|
||
{
|
||
"a": "Fleet residency",
|
||
"href": "/benchmarks/fleet"
|
||
},
|
||
{
|
||
"a": "Live agent footprint",
|
||
"href": "/benchmarks/goroutine"
|
||
},
|
||
{
|
||
"a": "Sandbox cold start",
|
||
"href": "/benchmarks/sandbox"
|
||
},
|
||
{
|
||
"a": "Engine prefill on CUDA vs llama.cpp",
|
||
"href": "/benchmarks/inference"
|
||
},
|
||
{
|
||
"a": "GPQA-Diamond",
|
||
"href": "/benchmarks/gpqa"
|
||
},
|
||
{
|
||
"p": "raw runs, plans, traces and scoring: hanzoai/benchmarks · brain · every table above regenerated by node results.mjs"
|
||
},
|
||
{
|
||
"code": "node results.mjs"
|
||
}
|
||
]
|
||
] |