Files
zeekay c38d21b533 Point Benchmarks and Philosophy CTAs to platform.hanzo.ai
platform.hanzo.ai is the platform console (with cloud.hanzo.ai
redirecting to it at the edge). Point the remaining cloud.hanzo.ai
links on /benchmarks and /philosophy directly at https://platform.hanzo.ai.

Claude-Session: https://claude.ai/code/session_015XYKLix9PSyn6Jw2HWKprk
2026-11-09 12:16:29 -08:00

240 lines
4.8 KiB
JSON
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
[
[
{
"p": "Benchmarks / LongMemEval"
},
{
"a": "Benchmarks",
"href": "/benchmarks"
},
{
"h1": "LongMemEval"
},
{
"p": "Five hundred questions, each with its own haystack of chat sessions around 115k tokens. The engine, frozen on LoCoMo and run here untuned, against the cosine floor on the same vectors across 4,846 sessions."
},
{
"a": "LongMemEval",
"href": "https://huggingface.co/datasets/xiaowu0162/longmemeval"
},
{
"a": "LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory",
"href": "https://arxiv.org/abs/2410.10813"
},
{
"code": "node longmemeval/engine.mjs --embed=minilm && node results.mjs"
},
{
"p": "regenerates every table on this page from the raw runs"
},
{
"h2": "What is measured"
},
{
"p": "Five hundred questions, each with its own haystack of chat sessions about 115k tokens long. One or more sessions hold the evidence, and retrieval is scored as recall of those sessions against the benchmark’s own answer_session_ids . Every session is embedded turn by turn and ranked by its best turn — 4,846 sessions and 199,641 turns. ALL@5 is the share of questions whose every evidence session is in the top five; ANY@5 is the share with at least one."
},
{
"code": "answer_session_ids"
},
{
"p": "Both columns use the same all-minilm vectors. The cosine column ranks a session by its best turn’s cosine. The engine column ranks it by its best turn under the engine’s score, with the configuration frozen on LoCoMo dev at 33d0f8749bd1 — nothing was tuned on this benchmark, so all 500 questions are the report."
},
{
"code": "33d0f8749bd1"
},
{
"h2": "Results"
},
{
"p": "embedder all-minilm · engine b467901f3950 · run longmemeval-all-context-minilm"
},
{
"code": "b467901f3950"
},
{
"code": "longmemeval-all-context-minilm"
},
{
"table": [
[
"question type",
"n",
"ALL@5 cosine",
"ALL@5 engine",
"ANY@5 cosine",
"ANY@5 engine",
"ALL@10 cosine",
"ALL@10 engine",
"MRR cosine",
"MRR engine"
],
[
"all questions",
"500",
"85.8",
"90.6",
"97.4",
"98.2",
"93.4",
"97.0",
"0.903",
"0.919"
],
[
"single-session-user",
"70",
"95.7",
"98.6",
"95.7",
"98.6",
"97.1",
"100.0",
"0.852",
"0.872"
],
[
"multi-session",
"133",
"75.9",
"85.7",
"98.5",
"99.2",
"89.5",
"94.7",
"0.912",
"0.917"
],
[
"single-session-preference",
"30",
"96.7",
"96.7",
"96.7",
"96.7",
"96.7",
"96.7",
"0.803",
"0.832"
],
[
"temporal-reasoning",
"133",
"75.9",
"81.2",
"94.7",
"95.5",
"88.7",
"94.7",
"0.865",
"0.894"
],
[
"knowledge-update",
"78",
"96.2",
"98.7",
"100.0",
"100.0",
"98.7",
"100.0",
"0.964",
"0.981"
]
]
},
{
"h2": "Where it gains"
},
{
"p": "Every evidence session is in the top five for 90.6% of questions against 85.8% for cosine, and in the top ten for 97.0% against 93.4%. The largest gains are multi-session, where ALL@5 moves from 75.9 to 85.7, and temporal-reasoning, from 75.9 to 81.2. LongMemEval has no fact layer, so the engine’s fact, entity and typed-graph generators propose nothing here; what does the work is lexical match, the session timeline, adjacency and a second hop."
},
{
"h2": "What this table is not"
},
{
"p": "Session recall is not answer accuracy. Systems reporting LongMemEval scores in the high eighties usually report an end-to-end answer metric with a language-model judge, which is a different measurement and not comparable to these columns. The run uses the release the dataset card now marks deprecated in favour of a cleaned one."
},
{
"p": "Hanzo AI Cloud"
},
{
"h3": "Build on Hanzo Cloud"
},
{
"a": "Start on Hanzo Cloud",
"href": "https://platform.hanzo.ai"
},
{
"a": "View Models & Pricing",
"href": "/pricing"
},
{
"a": "Previous MemoryAgentBench",
"href": "/benchmarks/memoryagentbench"
},
{
"a": "All Benchmarks Board",
"href": "/benchmarks"
},
{
"a": "Next CronQuestions",
"href": "/benchmarks/cronquestions"
},
{
"h3": "Related in memory"
},
{
"a": "LoCoMo",
"href": "/benchmarks/locomo"
},
{
"a": "MemoryAgentBench",
"href": "/benchmarks/memoryagentbench"
},
{
"a": "LoCoMo · subject scope",
"href": "/benchmarks/locomo-subject-scope"
},
{
"a": "LoCoMo-Conv",
"href": "/benchmarks/locomo-conv"
},
{
"p": "CronQuestions · RepoBench-R · Fleet residency · Live agent footprint · Sandbox cold start · Engine prefill on CUDA vs llama.cpp · GPQA-Diamond"
},
{
"a": "CronQuestions",
"href": "/benchmarks/cronquestions"
},
{
"a": "RepoBench-R",
"href": "/benchmarks/repobench-r"
},
{
"a": "Fleet residency",
"href": "/benchmarks/fleet"
},
{
"a": "Live agent footprint",
"href": "/benchmarks/goroutine"
},
{
"a": "Sandbox cold start",
"href": "/benchmarks/sandbox"
},
{
"a": "Engine prefill on CUDA vs llama.cpp",
"href": "/benchmarks/inference"
},
{
"a": "GPQA-Diamond",
"href": "/benchmarks/gpqa"
},
{
"p": "raw runs and scoring: hanzoai/benchmarks · brain · regenerated by node longmemeval/engine.mjs --embed=minilm && node results.mjs"
},
{
"code": "node longmemeval/engine.mjs --embed=minilm && node results.mjs"
}
]
]