Files
zeekay c38d21b533 Point Benchmarks and Philosophy CTAs to platform.hanzo.ai
platform.hanzo.ai is the platform console (with cloud.hanzo.ai
redirecting to it at the edge). Point the remaining cloud.hanzo.ai
links on /benchmarks and /philosophy directly at https://platform.hanzo.ai.

Claude-Session: https://claude.ai/code/session_015XYKLix9PSyn6Jw2HWKprk
2026-11-09 12:16:29 -08:00

1 line
3.9 KiB
JSON
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
{"text":["MemoryAgentBench","Benchmarks","measured","FactConsolidation: haystacks of 6k to 262k tokens where a later line silently overwrites an earlier one. A typed search over parsed facts resolves the question with no language model reading anything.","dataset","licence","MIT","introduced by","Hu, Wang, McAuley","UC San Diego, 2025","Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactions","harness","hanzoai/benchmarks · brain","our paper","Linkage, Not Similarity","node results.mjs","regenerates every table on this page from the raw runs","What is measured","FactConsolidation. A haystack is a numbered list of templated facts in which a later line silently replaces an earlier one about the same subject and relation, and nothing marks the update. Questions ask for the current value, single-hop and multi-hop, over haystacks of 6k to 262k tokens. One hundred questions per haystack. The metric is substring exact match, the benchmark’s own; an unanswered question scores zero. Dev is the 6k haystacks, test is 32k, 64k and 262k.","No language model reads the memory or writes the answer in any row here. The question is compiled into a plan sketch and a deterministic typed search over the parsed fact store resolves it — 45 templates, no line unparsed at any size, the supersession chain keyed by subject and relation. That is why these rows are not comparable to a pipeline that reads the haystack with a frontier model, and it is also the point: the retrieval is doing the work.","Results","substring exact match · 6k from dev, 32k / 64k / 262k from test · no reader · pooled = the mean of the four haystacks","single-hop","configuration","6k","32k","64k","262k","pooled","plain resolver","96.0","97.0","95.0","94.0","95.5","typed search","100.0","98.0","98.3","multi-hop","76.0","59.0","33.0","56.8","93.0","89.0","67.0","84.5","Typed search pools to 98.3 single-hop and 84.5 multi-hop against the plain resolver’s 95.5 and 56.8. The largest single move is 262k multi-hop, from 33.0 to 67.0.","Numbers other systems publish","Quoted with their protocols, because the protocols differ, and none of them re-run here. CAR (arXiv 2606.01435) reports 78.0 single-hop / 30.2 multi-hop pooled over 6k–262k with gpt-4o-mini and 94.8 / 51.5 with gpt-4o, and 82 / 27 and 93 / 41 respectively at 262k. Its own baselines at 262k multi-hop: HippoRAG-v2 5, gpt-4o whole-context 5, BM25 3, Cognee and MemGPT 3, Mem0 2, Zep 3. That is the nearest thing to a like-for-like comparison anywhere on this site — same data, same split, same substring-EM metric — and it is still not one, because CAR reads the haystack with a frontier model and the rows above use no model at all.","Pith publishes per-lane multi-hop figures on a vendor page: 83 / 84 at 32k and 64k, where typed search scores 89.0 / 89.0.","Hanzo AI Cloud","Build on Hanzo Cloud","Start on Hanzo Cloud","View Models & Pricing","Previous","LoCoMo","All Benchmarks Board","Next","LongMemEval","Related in memory","LoCoMo · subject scope","LoCoMo-Conv","Other benchmarks","CronQuestions","RepoBench-R","Fleet residency","Live agent footprint","Sandbox cold start","Engine prefill on CUDA vs llama.cpp","GPQA-Diamond","raw runs, plans, traces and scoring:","· every table above regenerated by"],"links":["/benchmarks","https://huggingface.co/datasets/ai-hyz/MemoryAgentBench","https://arxiv.org/abs/2507.05257","https://github.com/hanzoai/papers/blob/main/site/public/pdfs/hanzo-context-memory.pdf","https://platform.hanzo.ai","/pricing","https://hanzo.ai/benchmarks/locomo","https://hanzo.ai/benchmarks/longmemeval","https://hanzo.ai/benchmarks/locomo-subject-scope","https://hanzo.ai/benchmarks/locomo-conv","https://hanzo.ai/benchmarks/cronquestions","https://hanzo.ai/benchmarks/repobench-r","https://hanzo.ai/benchmarks/fleet","https://hanzo.ai/benchmarks/goroutine","https://hanzo.ai/benchmarks/sandbox","https://hanzo.ai/benchmarks/inference","https://hanzo.ai/benchmarks/gpqa"]}