mirror of
https://github.com/hanzoai/site.git
synced 2026-11-21 01:07:08 +00:00
platform.hanzo.ai is the platform console (with cloud.hanzo.ai redirecting to it at the edge). Point the remaining cloud.hanzo.ai links on /benchmarks and /philosophy directly at https://platform.hanzo.ai. Claude-Session: https://claude.ai/code/session_015XYKLix9PSyn6Jw2HWKprk
1 line
38 KiB
HTML
1 line
38 KiB
HTML
<div class="is_View _col-_platformweb_text-primar121 _fd-column _dsp-block _minH-100vh _bg-background"><main class="is_View _col-_platformweb_text-primar121 _fd-column _maxW-1080px _width-10037 _mr-auto _ml-auto _pt-64px _pr-24px _pl-24px _pb-96px _bg-background"><p class="is_Text _col-muted-foreg108704142 _dsp-block _mb-4px _fs-f-size-1 _lh-f-lineHeigh112922" data-slot="text"><a href="/benchmarks">Benchmarks</a> / <!-- -->LongMemEval</p><span class="is_Text _col-muted-foreg108704142 _dsp-block _fs-f-size-1 _lh-f-lineHeigh112922 _ls-0--88px _tt-uppercase" data-slot="text">measured</span><h1 class="is_Text font_heading _fs-_sm_f-size-11 _fs-_lg_text-6xl _textWrap-_platformweb_balance _col-foreground _dsp-revert-laye114 _ws-inherit _ff-f-family _fs-f-size-10 _fw-497 _lh-1--19 _ls--0--012em _textWrap-balance _mt-8px _mb-16px" data-slot="display">LongMemEval</h1><p class="is_Text _col-muted-foreg108704142 _dsp-revert-laye114 _ws-inherit _fs-f-size-3 _lh-1--65 _ls-0px _textWrap-pretty _maxW-720px" data-slot="lede">Five hundred questions, each with its own haystack of chat sessions around 115k tokens. The engine, frozen on LoCoMo and run here untuned, against the cosine floor on the same vectors across 4,846 sessions.</p><dl class="is_Text _col-text-primar121 _dsp-grid _gridTemplateColumns-minmax120px329255122 _rowGap-8px _columnGap-20px _mt-20px _mr-0px _mb-0px _ml-0px _maxW-760px _fs-f-size-2 _lh-f-lineHeigh112922" data-slot="text"><dt class="is_Text _col-muted-foreg108704142" data-slot="text">dataset</dt><dd class="is_Text _col-inherit _mt-0px _mr-0px _mb-0px _ml-0px" data-slot="text"><a href="https://huggingface.co/datasets/xiaowu0162/longmemeval">LongMemEval</a></dd><dt class="is_Text _col-muted-foreg108704142" data-slot="text">licence</dt><dd class="is_Text _col-inherit _mt-0px _mr-0px _mb-0px _ml-0px" data-slot="text">MIT</dd><dt class="is_Text _col-muted-foreg108704142" data-slot="text">introduced by</dt><dd class="is_Text _col-inherit _mt-0px _mr-0px _mb-0px _ml-0px" data-slot="text"><a href="https://arxiv.org/abs/2410.10813">LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory</a> — <!-- -->Wu et al.<!-- -->, <!-- -->2024</dd><dt class="is_Text _col-muted-foreg108704142" data-slot="text">harness</dt><dd class="is_Text _col-inherit _mt-0px _mr-0px _mb-0px _ml-0px" data-slot="text">hanzoai/benchmarks · brain</dd></dl><code class="is_Text font_mono _col-text-primar121 _dsp-block _mt-10px _pt-10px _pb-10px _pr-14px _pl-14px _btw-1px _brw-1px _borderBottomWidth-1px _borderLeftWidth-1px _btc-edge _brc-edge _borderBottomColor-edge _borderLeftColor-edge _btlr-6px _btrr-6px _bbrr-6px _bblr-6px _ox-auto _bg-card _ff-f-family _fs-f-size-1 _bbs-solid _bts-solid _bls-solid _brs-solid" data-slot="text">node longmemeval/engine.mjs --embed=minilm && node results.mjs</code><p class="is_Text font_mono _col-muted-foreg108704142 _dsp-block _mb-10px _ff-f-family _fs-f-size-1 _lh-f-lineHeigh112922 _mt-8px" data-slot="text">regenerates every table on this page from the raw runs</p><h2 class="is_Text font_heading _col-foreground _dsp-revert-laye114 _ws-inherit _ff-f-family _fs-f-size-7 _fw-497 _lh-1--2 _ls--0--01em _textWrap-balance _mt-56px _mb-8px" data-slot="title">What is measured</h2><p class="is_Text _col-text-second96872 _dsp-block _maxW-720px _lh-1--3563rem" data-slot="text">Five hundred questions, each with its own haystack of chat sessions about 115k tokens long. One or more sessions hold the evidence, and retrieval is scored as recall of those sessions against the benchmark’s own <code>answer_session_ids</code>. Every session is embedded turn by turn and ranked by its best turn — 4,846 sessions and 199,641 turns. ALL@5 is the share of questions whose every evidence session is in the top five; ANY@5 is the share with at least one.</p><p class="is_Text _col-text-second96872 _dsp-block _maxW-720px _lh-1--3563rem" data-slot="text">Both columns use the same <!-- -->all-minilm<!-- --> vectors. The cosine column ranks a session by its best turn’s cosine. The engine column ranks it by its best turn under the engine’s score, with the configuration frozen on LoCoMo dev at <code>33d0f8749bd1</code> — nothing was tuned on this benchmark, so all 500 questions are the report.</p><h2 class="is_Text font_heading _col-foreground _dsp-revert-laye114 _ws-inherit _ff-f-family _fs-f-size-7 _fw-497 _lh-1--2 _ls--0--01em _textWrap-balance _mt-56px _mb-8px" data-slot="title">Results</h2><p class="is_Text font_mono _col-muted-foreg108704142 _dsp-block _mb-10px _ff-f-family _fs-f-size-1 _lh-f-lineHeigh112922" data-slot="text">embedder <!-- -->all-minilm<!-- --> · engine <code>b467901f3950</code> · run <code>longmemeval-all-context-minilm</code></p><div class="is_View _ox-auto _btw-1px _brw-1px _borderBottomWidth-1px _borderLeftWidth-1px _btc-edge _brc-edge _borderBottomColor-edge _borderLeftColor-edge _btlr-6px _btrr-6px _bbrr-6px _bblr-6px _bg-card _bbs-solid _bts-solid _bls-solid _brs-solid"><table class="is_Text _dsp-_platformweb_table _borderCollapse-_platformweb_collapse _fontVariantNumeric-_platformweb_tabular-num115 _col-text-primar121 _width-10037 _fs-f-size-2 _lh-f-lineHeigh112922" data-slot="text"><thead><tr><th class="is_Text _dsp-_platformweb_table-cell _fontVariantNumeric-_platformweb_tabular-num115 _col-muted-foreg108704142 _text-left _pt-8px _pb-8px _pr-12px _pl-12px _borderBottomWidth-1px _btc-edge _brc-edge _borderBottomColor-edge _borderLeftColor-edge _fw-500 _ws-nowrap _bbs-solid" data-slot="text">question type</th><th class="is_Text _dsp-_platformweb_table-cell _fontVariantNumeric-_platformweb_tabular-num115 _col-muted-foreg108704142 _text-right _pt-8px _pb-8px _pr-12px _pl-12px _borderBottomWidth-1px _btc-edge _brc-edge _borderBottomColor-edge _borderLeftColor-edge _fw-500 _ws-nowrap _bbs-solid" data-slot="text">n</th><th class="is_Text _dsp-_platformweb_table-cell _fontVariantNumeric-_platformweb_tabular-num115 _col-muted-foreg108704142 _text-right _pt-8px _pb-8px _pr-12px _pl-12px _borderBottomWidth-1px _btc-edge _brc-edge _borderBottomColor-edge _borderLeftColor-edge _fw-500 _ws-nowrap _bbs-solid" data-slot="text">ALL@5 cosine</th><th class="is_Text _dsp-_platformweb_table-cell _fontVariantNumeric-_platformweb_tabular-num115 _col-muted-foreg108704142 _text-right _pt-8px _pb-8px _pr-12px _pl-12px _borderBottomWidth-1px _btc-edge _brc-edge _borderBottomColor-edge _borderLeftColor-edge _fw-500 _ws-nowrap _bbs-solid" data-slot="text">ALL@5 engine</th><th class="is_Text _dsp-_platformweb_table-cell _fontVariantNumeric-_platformweb_tabular-num115 _col-muted-foreg108704142 _text-right _pt-8px _pb-8px _pr-12px _pl-12px _borderBottomWidth-1px _btc-edge _brc-edge _borderBottomColor-edge _borderLeftColor-edge _fw-500 _ws-nowrap _bbs-solid" data-slot="text">ANY@5 cosine</th><th class="is_Text _dsp-_platformweb_table-cell _fontVariantNumeric-_platformweb_tabular-num115 _col-muted-foreg108704142 _text-right _pt-8px _pb-8px _pr-12px _pl-12px _borderBottomWidth-1px _btc-edge _brc-edge _borderBottomColor-edge _borderLeftColor-edge _fw-500 _ws-nowrap _bbs-solid" data-slot="text">ANY@5 engine</th><th class="is_Text _dsp-_platformweb_table-cell _fontVariantNumeric-_platformweb_tabular-num115 _col-muted-foreg108704142 _text-right _pt-8px _pb-8px _pr-12px _pl-12px _borderBottomWidth-1px _btc-edge _brc-edge _borderBottomColor-edge _borderLeftColor-edge _fw-500 _ws-nowrap _bbs-solid" data-slot="text">ALL@10 cosine</th><th class="is_Text _dsp-_platformweb_table-cell _fontVariantNumeric-_platformweb_tabular-num115 _col-muted-foreg108704142 _text-right _pt-8px _pb-8px _pr-12px _pl-12px _borderBottomWidth-1px _btc-edge _brc-edge _borderBottomColor-edge _borderLeftColor-edge _fw-500 _ws-nowrap _bbs-solid" data-slot="text">ALL@10 engine</th><th class="is_Text _dsp-_platformweb_table-cell _fontVariantNumeric-_platformweb_tabular-num115 _col-muted-foreg108704142 _text-right _pt-8px _pb-8px _pr-12px _pl-12px _borderBottomWidth-1px _btc-edge _brc-edge _borderBottomColor-edge _borderLeftColor-edge _fw-500 _ws-nowrap _bbs-solid" data-slot="text">MRR cosine</th><th class="is_Text _dsp-_platformweb_table-cell _fontVariantNumeric-_platformweb_tabular-num115 _col-muted-foreg108704142 _text-right _pt-8px _pb-8px _pr-12px _pl-12px _borderBottomWidth-1px _btc-edge _brc-edge _borderBottomColor-edge _borderLeftColor-edge _fw-500 _ws-nowrap _bbs-solid" data-slot="text">MRR engine</th></tr></thead><tbody><tr class="is_Text _dsp-_platformweb_table-row _col-inherit" data-slot="text"><td class="is_Text _dsp-_platformweb_table-cell _fontVariantNumeric-_platformweb_tabular-num115 _col-inherit _text-left _pt-7px _pb-7px _pr-12px _pl-12px _borderBottomWidth-1px _btc-edge _brc-edge _borderBottomColor-edge _borderLeftColor-edge _ws-nowrap _bbs-solid" data-slot="text">all questions</td><td class="is_Text _dsp-_platformweb_table-cell _fontVariantNumeric-_platformweb_tabular-num115 _col-inherit _text-right _pt-7px _pb-7px _pr-12px _pl-12px _borderBottomWidth-1px _btc-edge _brc-edge _borderBottomColor-edge _borderLeftColor-edge _ws-nowrap _bbs-solid" data-slot="text">500</td><td class="is_Text _dsp-_platformweb_table-cell _fontVariantNumeric-_platformweb_tabular-num115 _col-inherit _text-right _pt-7px _pb-7px _pr-12px _pl-12px _borderBottomWidth-1px _btc-edge _brc-edge _borderBottomColor-edge _borderLeftColor-edge _ws-nowrap _bbs-solid" data-slot="text">85.8</td><td class="is_Text _dsp-_platformweb_table-cell _fontVariantNumeric-_platformweb_tabular-num115 _col-inherit _text-right _pt-7px _pb-7px _pr-12px _pl-12px _borderBottomWidth-1px _btc-edge _brc-edge _borderBottomColor-edge _borderLeftColor-edge _ws-nowrap _bbs-solid" data-slot="text">90.6</td><td class="is_Text _dsp-_platformweb_table-cell _fontVariantNumeric-_platformweb_tabular-num115 _col-inherit _text-right _pt-7px _pb-7px _pr-12px _pl-12px _borderBottomWidth-1px _btc-edge _brc-edge _borderBottomColor-edge _borderLeftColor-edge _ws-nowrap _bbs-solid" data-slot="text">97.4</td><td class="is_Text _dsp-_platformweb_table-cell _fontVariantNumeric-_platformweb_tabular-num115 _col-inherit _text-right _pt-7px _pb-7px _pr-12px _pl-12px _borderBottomWidth-1px _btc-edge _brc-edge _borderBottomColor-edge _borderLeftColor-edge _ws-nowrap _bbs-solid" data-slot="text">98.2</td><td class="is_Text _dsp-_platformweb_table-cell _fontVariantNumeric-_platformweb_tabular-num115 _col-inherit _text-right _pt-7px _pb-7px _pr-12px _pl-12px _borderBottomWidth-1px _btc-edge _brc-edge _borderBottomColor-edge _borderLeftColor-edge _ws-nowrap _bbs-solid" data-slot="text">93.4</td><td class="is_Text _dsp-_platformweb_table-cell _fontVariantNumeric-_platformweb_tabular-num115 _col-inherit _text-right _pt-7px _pb-7px _pr-12px _pl-12px _borderBottomWidth-1px _btc-edge _brc-edge _borderBottomColor-edge _borderLeftColor-edge _ws-nowrap _bbs-solid" data-slot="text">97.0</td><td class="is_Text _dsp-_platformweb_table-cell _fontVariantNumeric-_platformweb_tabular-num115 _col-inherit _text-right _pt-7px _pb-7px _pr-12px _pl-12px _borderBottomWidth-1px _btc-edge _brc-edge _borderBottomColor-edge _borderLeftColor-edge _ws-nowrap _bbs-solid" data-slot="text">0.903</td><td class="is_Text _dsp-_platformweb_table-cell _fontVariantNumeric-_platformweb_tabular-num115 _col-inherit _text-right _pt-7px _pb-7px _pr-12px _pl-12px _borderBottomWidth-1px _btc-edge _brc-edge _borderBottomColor-edge _borderLeftColor-edge _ws-nowrap _bbs-solid" data-slot="text">0.919</td></tr><tr class="is_Text _dsp-_platformweb_table-row _col-inherit" data-slot="text"><td class="is_Text _dsp-_platformweb_table-cell _fontVariantNumeric-_platformweb_tabular-num115 _col-inherit _text-left _pt-7px _pb-7px _pr-12px _pl-12px _borderBottomWidth-1px _btc-edge _brc-edge _borderBottomColor-edge _borderLeftColor-edge _ws-nowrap _bbs-solid" data-slot="text">single-session-user</td><td class="is_Text _dsp-_platformweb_table-cell _fontVariantNumeric-_platformweb_tabular-num115 _col-inherit _text-right _pt-7px _pb-7px _pr-12px _pl-12px _borderBottomWidth-1px _btc-edge _brc-edge _borderBottomColor-edge _borderLeftColor-edge _ws-nowrap _bbs-solid" data-slot="text">70</td><td class="is_Text _dsp-_platformweb_table-cell _fontVariantNumeric-_platformweb_tabular-num115 _col-inherit _text-right _pt-7px _pb-7px _pr-12px _pl-12px _borderBottomWidth-1px _btc-edge _brc-edge _borderBottomColor-edge _borderLeftColor-edge _ws-nowrap _bbs-solid" data-slot="text">95.7</td><td class="is_Text _dsp-_platformweb_table-cell _fontVariantNumeric-_platformweb_tabular-num115 _col-inherit _text-right _pt-7px _pb-7px _pr-12px _pl-12px _borderBottomWidth-1px _btc-edge _brc-edge _borderBottomColor-edge _borderLeftColor-edge _ws-nowrap _bbs-solid" data-slot="text">98.6</td><td class="is_Text _dsp-_platformweb_table-cell _fontVariantNumeric-_platformweb_tabular-num115 _col-inherit _text-right _pt-7px _pb-7px _pr-12px _pl-12px _borderBottomWidth-1px _btc-edge _brc-edge _borderBottomColor-edge _borderLeftColor-edge _ws-nowrap _bbs-solid" data-slot="text">95.7</td><td class="is_Text _dsp-_platformweb_table-cell _fontVariantNumeric-_platformweb_tabular-num115 _col-inherit _text-right _pt-7px _pb-7px _pr-12px _pl-12px _borderBottomWidth-1px _btc-edge _brc-edge _borderBottomColor-edge _borderLeftColor-edge _ws-nowrap _bbs-solid" data-slot="text">98.6</td><td class="is_Text _dsp-_platformweb_table-cell _fontVariantNumeric-_platformweb_tabular-num115 _col-inherit _text-right _pt-7px _pb-7px _pr-12px _pl-12px _borderBottomWidth-1px _btc-edge _brc-edge _borderBottomColor-edge _borderLeftColor-edge _ws-nowrap _bbs-solid" data-slot="text">97.1</td><td class="is_Text _dsp-_platformweb_table-cell _fontVariantNumeric-_platformweb_tabular-num115 _col-inherit _text-right _pt-7px _pb-7px _pr-12px _pl-12px _borderBottomWidth-1px _btc-edge _brc-edge _borderBottomColor-edge _borderLeftColor-edge _ws-nowrap _bbs-solid" data-slot="text">100.0</td><td class="is_Text _dsp-_platformweb_table-cell _fontVariantNumeric-_platformweb_tabular-num115 _col-inherit _text-right _pt-7px _pb-7px _pr-12px _pl-12px _borderBottomWidth-1px _btc-edge _brc-edge _borderBottomColor-edge _borderLeftColor-edge _ws-nowrap _bbs-solid" data-slot="text">0.852</td><td class="is_Text _dsp-_platformweb_table-cell _fontVariantNumeric-_platformweb_tabular-num115 _col-inherit _text-right _pt-7px _pb-7px _pr-12px _pl-12px _borderBottomWidth-1px _btc-edge _brc-edge _borderBottomColor-edge _borderLeftColor-edge _ws-nowrap _bbs-solid" data-slot="text">0.872</td></tr><tr class="is_Text _dsp-_platformweb_table-row _col-inherit" data-slot="text"><td class="is_Text _dsp-_platformweb_table-cell _fontVariantNumeric-_platformweb_tabular-num115 _col-inherit _text-left _pt-7px _pb-7px _pr-12px _pl-12px _borderBottomWidth-1px _btc-edge _brc-edge _borderBottomColor-edge _borderLeftColor-edge _ws-nowrap _bbs-solid" data-slot="text">multi-session</td><td class="is_Text _dsp-_platformweb_table-cell _fontVariantNumeric-_platformweb_tabular-num115 _col-inherit _text-right _pt-7px _pb-7px _pr-12px _pl-12px _borderBottomWidth-1px _btc-edge _brc-edge _borderBottomColor-edge _borderLeftColor-edge _ws-nowrap _bbs-solid" data-slot="text">133</td><td class="is_Text _dsp-_platformweb_table-cell _fontVariantNumeric-_platformweb_tabular-num115 _col-inherit _text-right _pt-7px _pb-7px _pr-12px _pl-12px _borderBottomWidth-1px _btc-edge _brc-edge _borderBottomColor-edge _borderLeftColor-edge _ws-nowrap _bbs-solid" data-slot="text">75.9</td><td class="is_Text _dsp-_platformweb_table-cell _fontVariantNumeric-_platformweb_tabular-num115 _col-inherit _text-right _pt-7px _pb-7px _pr-12px _pl-12px _borderBottomWidth-1px _btc-edge _brc-edge _borderBottomColor-edge _borderLeftColor-edge _ws-nowrap _bbs-solid" data-slot="text">85.7</td><td class="is_Text _dsp-_platformweb_table-cell _fontVariantNumeric-_platformweb_tabular-num115 _col-inherit _text-right _pt-7px _pb-7px _pr-12px _pl-12px _borderBottomWidth-1px _btc-edge _brc-edge _borderBottomColor-edge _borderLeftColor-edge _ws-nowrap _bbs-solid" data-slot="text">98.5</td><td class="is_Text _dsp-_platformweb_table-cell _fontVariantNumeric-_platformweb_tabular-num115 _col-inherit _text-right _pt-7px _pb-7px _pr-12px _pl-12px _borderBottomWidth-1px _btc-edge _brc-edge _borderBottomColor-edge _borderLeftColor-edge _ws-nowrap _bbs-solid" data-slot="text">99.2</td><td class="is_Text _dsp-_platformweb_table-cell _fontVariantNumeric-_platformweb_tabular-num115 _col-inherit _text-right _pt-7px _pb-7px _pr-12px _pl-12px _borderBottomWidth-1px _btc-edge _brc-edge _borderBottomColor-edge _borderLeftColor-edge _ws-nowrap _bbs-solid" data-slot="text">89.5</td><td class="is_Text _dsp-_platformweb_table-cell _fontVariantNumeric-_platformweb_tabular-num115 _col-inherit _text-right _pt-7px _pb-7px _pr-12px _pl-12px _borderBottomWidth-1px _btc-edge _brc-edge _borderBottomColor-edge _borderLeftColor-edge _ws-nowrap _bbs-solid" data-slot="text">94.7</td><td class="is_Text _dsp-_platformweb_table-cell _fontVariantNumeric-_platformweb_tabular-num115 _col-inherit _text-right _pt-7px _pb-7px _pr-12px _pl-12px _borderBottomWidth-1px _btc-edge _brc-edge _borderBottomColor-edge _borderLeftColor-edge _ws-nowrap _bbs-solid" data-slot="text">0.912</td><td class="is_Text _dsp-_platformweb_table-cell _fontVariantNumeric-_platformweb_tabular-num115 _col-inherit _text-right _pt-7px _pb-7px _pr-12px _pl-12px _borderBottomWidth-1px _btc-edge _brc-edge _borderBottomColor-edge _borderLeftColor-edge _ws-nowrap _bbs-solid" data-slot="text">0.917</td></tr><tr class="is_Text _dsp-_platformweb_table-row _col-inherit" data-slot="text"><td class="is_Text _dsp-_platformweb_table-cell _fontVariantNumeric-_platformweb_tabular-num115 _col-inherit _text-left _pt-7px _pb-7px _pr-12px _pl-12px _borderBottomWidth-1px _btc-edge _brc-edge _borderBottomColor-edge _borderLeftColor-edge _ws-nowrap _bbs-solid" data-slot="text">single-session-preference</td><td class="is_Text _dsp-_platformweb_table-cell _fontVariantNumeric-_platformweb_tabular-num115 _col-inherit _text-right _pt-7px _pb-7px _pr-12px _pl-12px _borderBottomWidth-1px _btc-edge _brc-edge _borderBottomColor-edge _borderLeftColor-edge _ws-nowrap _bbs-solid" data-slot="text">30</td><td class="is_Text _dsp-_platformweb_table-cell _fontVariantNumeric-_platformweb_tabular-num115 _col-inherit _text-right _pt-7px _pb-7px _pr-12px _pl-12px _borderBottomWidth-1px _btc-edge _brc-edge _borderBottomColor-edge _borderLeftColor-edge _ws-nowrap _bbs-solid" data-slot="text">96.7</td><td class="is_Text _dsp-_platformweb_table-cell _fontVariantNumeric-_platformweb_tabular-num115 _col-inherit _text-right _pt-7px _pb-7px _pr-12px _pl-12px _borderBottomWidth-1px _btc-edge _brc-edge _borderBottomColor-edge _borderLeftColor-edge _ws-nowrap _bbs-solid" data-slot="text">96.7</td><td class="is_Text _dsp-_platformweb_table-cell _fontVariantNumeric-_platformweb_tabular-num115 _col-inherit _text-right _pt-7px _pb-7px _pr-12px _pl-12px _borderBottomWidth-1px _btc-edge _brc-edge _borderBottomColor-edge _borderLeftColor-edge _ws-nowrap _bbs-solid" data-slot="text">96.7</td><td class="is_Text _dsp-_platformweb_table-cell _fontVariantNumeric-_platformweb_tabular-num115 _col-inherit _text-right _pt-7px _pb-7px _pr-12px _pl-12px _borderBottomWidth-1px _btc-edge _brc-edge _borderBottomColor-edge _borderLeftColor-edge _ws-nowrap _bbs-solid" data-slot="text">96.7</td><td class="is_Text _dsp-_platformweb_table-cell _fontVariantNumeric-_platformweb_tabular-num115 _col-inherit _text-right _pt-7px _pb-7px _pr-12px _pl-12px _borderBottomWidth-1px _btc-edge _brc-edge _borderBottomColor-edge _borderLeftColor-edge _ws-nowrap _bbs-solid" data-slot="text">96.7</td><td class="is_Text _dsp-_platformweb_table-cell _fontVariantNumeric-_platformweb_tabular-num115 _col-inherit _text-right _pt-7px _pb-7px _pr-12px _pl-12px _borderBottomWidth-1px _btc-edge _brc-edge _borderBottomColor-edge _borderLeftColor-edge _ws-nowrap _bbs-solid" data-slot="text">96.7</td><td class="is_Text _dsp-_platformweb_table-cell _fontVariantNumeric-_platformweb_tabular-num115 _col-inherit _text-right _pt-7px _pb-7px _pr-12px _pl-12px _borderBottomWidth-1px _btc-edge _brc-edge _borderBottomColor-edge _borderLeftColor-edge _ws-nowrap _bbs-solid" data-slot="text">0.803</td><td class="is_Text _dsp-_platformweb_table-cell _fontVariantNumeric-_platformweb_tabular-num115 _col-inherit _text-right _pt-7px _pb-7px _pr-12px _pl-12px _borderBottomWidth-1px _btc-edge _brc-edge _borderBottomColor-edge _borderLeftColor-edge _ws-nowrap _bbs-solid" data-slot="text">0.832</td></tr><tr class="is_Text _dsp-_platformweb_table-row _col-inherit" data-slot="text"><td class="is_Text _dsp-_platformweb_table-cell _fontVariantNumeric-_platformweb_tabular-num115 _col-inherit _text-left _pt-7px _pb-7px _pr-12px _pl-12px _borderBottomWidth-1px _btc-edge _brc-edge _borderBottomColor-edge _borderLeftColor-edge _ws-nowrap _bbs-solid" data-slot="text">temporal-reasoning</td><td class="is_Text _dsp-_platformweb_table-cell _fontVariantNumeric-_platformweb_tabular-num115 _col-inherit _text-right _pt-7px _pb-7px _pr-12px _pl-12px _borderBottomWidth-1px _btc-edge _brc-edge _borderBottomColor-edge _borderLeftColor-edge _ws-nowrap _bbs-solid" data-slot="text">133</td><td class="is_Text _dsp-_platformweb_table-cell _fontVariantNumeric-_platformweb_tabular-num115 _col-inherit _text-right _pt-7px _pb-7px _pr-12px _pl-12px _borderBottomWidth-1px _btc-edge _brc-edge _borderBottomColor-edge _borderLeftColor-edge _ws-nowrap _bbs-solid" data-slot="text">75.9</td><td class="is_Text _dsp-_platformweb_table-cell _fontVariantNumeric-_platformweb_tabular-num115 _col-inherit _text-right _pt-7px _pb-7px _pr-12px _pl-12px _borderBottomWidth-1px _btc-edge _brc-edge _borderBottomColor-edge _borderLeftColor-edge _ws-nowrap _bbs-solid" data-slot="text">81.2</td><td class="is_Text _dsp-_platformweb_table-cell _fontVariantNumeric-_platformweb_tabular-num115 _col-inherit _text-right _pt-7px _pb-7px _pr-12px _pl-12px _borderBottomWidth-1px _btc-edge _brc-edge _borderBottomColor-edge _borderLeftColor-edge _ws-nowrap _bbs-solid" data-slot="text">94.7</td><td class="is_Text _dsp-_platformweb_table-cell _fontVariantNumeric-_platformweb_tabular-num115 _col-inherit _text-right _pt-7px _pb-7px _pr-12px _pl-12px _borderBottomWidth-1px _btc-edge _brc-edge _borderBottomColor-edge _borderLeftColor-edge _ws-nowrap _bbs-solid" data-slot="text">95.5</td><td class="is_Text _dsp-_platformweb_table-cell _fontVariantNumeric-_platformweb_tabular-num115 _col-inherit _text-right _pt-7px _pb-7px _pr-12px _pl-12px _borderBottomWidth-1px _btc-edge _brc-edge _borderBottomColor-edge _borderLeftColor-edge _ws-nowrap _bbs-solid" data-slot="text">88.7</td><td class="is_Text _dsp-_platformweb_table-cell _fontVariantNumeric-_platformweb_tabular-num115 _col-inherit _text-right _pt-7px _pb-7px _pr-12px _pl-12px _borderBottomWidth-1px _btc-edge _brc-edge _borderBottomColor-edge _borderLeftColor-edge _ws-nowrap _bbs-solid" data-slot="text">94.7</td><td class="is_Text _dsp-_platformweb_table-cell _fontVariantNumeric-_platformweb_tabular-num115 _col-inherit _text-right _pt-7px _pb-7px _pr-12px _pl-12px _borderBottomWidth-1px _btc-edge _brc-edge _borderBottomColor-edge _borderLeftColor-edge _ws-nowrap _bbs-solid" data-slot="text">0.865</td><td class="is_Text _dsp-_platformweb_table-cell _fontVariantNumeric-_platformweb_tabular-num115 _col-inherit _text-right _pt-7px _pb-7px _pr-12px _pl-12px _borderBottomWidth-1px _btc-edge _brc-edge _borderBottomColor-edge _borderLeftColor-edge _ws-nowrap _bbs-solid" data-slot="text">0.894</td></tr><tr class="is_Text _dsp-_platformweb_table-row _col-inherit" data-slot="text"><td class="is_Text _dsp-_platformweb_table-cell _fontVariantNumeric-_platformweb_tabular-num115 _col-inherit _text-left _pt-7px _pb-7px _pr-12px _pl-12px _borderBottomWidth-1px _btc-edge _brc-edge _borderBottomColor-edge _borderLeftColor-edge _ws-nowrap _bbs-solid" data-slot="text">knowledge-update</td><td class="is_Text _dsp-_platformweb_table-cell _fontVariantNumeric-_platformweb_tabular-num115 _col-inherit _text-right _pt-7px _pb-7px _pr-12px _pl-12px _borderBottomWidth-1px _btc-edge _brc-edge _borderBottomColor-edge _borderLeftColor-edge _ws-nowrap _bbs-solid" data-slot="text">78</td><td class="is_Text _dsp-_platformweb_table-cell _fontVariantNumeric-_platformweb_tabular-num115 _col-inherit _text-right _pt-7px _pb-7px _pr-12px _pl-12px _borderBottomWidth-1px _btc-edge _brc-edge _borderBottomColor-edge _borderLeftColor-edge _ws-nowrap _bbs-solid" data-slot="text">96.2</td><td class="is_Text _dsp-_platformweb_table-cell _fontVariantNumeric-_platformweb_tabular-num115 _col-inherit _text-right _pt-7px _pb-7px _pr-12px _pl-12px _borderBottomWidth-1px _btc-edge _brc-edge _borderBottomColor-edge _borderLeftColor-edge _ws-nowrap _bbs-solid" data-slot="text">98.7</td><td class="is_Text _dsp-_platformweb_table-cell _fontVariantNumeric-_platformweb_tabular-num115 _col-inherit _text-right _pt-7px _pb-7px _pr-12px _pl-12px _borderBottomWidth-1px _btc-edge _brc-edge _borderBottomColor-edge _borderLeftColor-edge _ws-nowrap _bbs-solid" data-slot="text">100.0</td><td class="is_Text _dsp-_platformweb_table-cell _fontVariantNumeric-_platformweb_tabular-num115 _col-inherit _text-right _pt-7px _pb-7px _pr-12px _pl-12px _borderBottomWidth-1px _btc-edge _brc-edge _borderBottomColor-edge _borderLeftColor-edge _ws-nowrap _bbs-solid" data-slot="text">100.0</td><td class="is_Text _dsp-_platformweb_table-cell _fontVariantNumeric-_platformweb_tabular-num115 _col-inherit _text-right _pt-7px _pb-7px _pr-12px _pl-12px _borderBottomWidth-1px _btc-edge _brc-edge _borderBottomColor-edge _borderLeftColor-edge _ws-nowrap _bbs-solid" data-slot="text">98.7</td><td class="is_Text _dsp-_platformweb_table-cell _fontVariantNumeric-_platformweb_tabular-num115 _col-inherit _text-right _pt-7px _pb-7px _pr-12px _pl-12px _borderBottomWidth-1px _btc-edge _brc-edge _borderBottomColor-edge _borderLeftColor-edge _ws-nowrap _bbs-solid" data-slot="text">100.0</td><td class="is_Text _dsp-_platformweb_table-cell _fontVariantNumeric-_platformweb_tabular-num115 _col-inherit _text-right _pt-7px _pb-7px _pr-12px _pl-12px _borderBottomWidth-1px _btc-edge _brc-edge _borderBottomColor-edge _borderLeftColor-edge _ws-nowrap _bbs-solid" data-slot="text">0.964</td><td class="is_Text _dsp-_platformweb_table-cell _fontVariantNumeric-_platformweb_tabular-num115 _col-inherit _text-right _pt-7px _pb-7px _pr-12px _pl-12px _borderBottomWidth-1px _btc-edge _brc-edge _borderBottomColor-edge _borderLeftColor-edge _ws-nowrap _bbs-solid" data-slot="text">0.981</td></tr></tbody></table></div><h2 class="is_Text font_heading _col-foreground _dsp-revert-laye114 _ws-inherit _ff-f-family _fs-f-size-7 _fw-497 _lh-1--2 _ls--0--01em _textWrap-balance _mt-56px _mb-8px" data-slot="title">Where it gains</h2><p class="is_Text _col-text-second96872 _dsp-block _maxW-720px _lh-1--3563rem" data-slot="text">Every evidence session is in the top five for 90.6% of questions against 85.8% for cosine, and in the top ten for 97.0% against 93.4%. The largest gains are multi-session, where ALL@5 moves from 75.9 to 85.7, and temporal-reasoning, from 75.9 to 81.2. LongMemEval has no fact layer, so the engine’s fact, entity and typed-graph generators propose nothing here; what does the work is lexical match, the session timeline, adjacency and a second hop.</p><h2 class="is_Text font_heading _col-foreground _dsp-revert-laye114 _ws-inherit _ff-f-family _fs-f-size-7 _fw-497 _lh-1--2 _ls--0--01em _textWrap-balance _mt-56px _mb-8px" data-slot="title">What this table is not</h2><p class="is_Text _col-text-second96872 _dsp-block _maxW-720px _lh-1--3563rem" data-slot="text">Session recall is not answer accuracy. Systems reporting LongMemEval scores in the high eighties usually report an end-to-end answer metric with a language-model judge, which is a different measurement and not comparable to these columns. The run uses the release the dataset card now marks deprecated in favour of a cleaned one.</p><div class="is_View _fd-column _mt-48px _pt-16px"><div class="is_View _text-_platformweb_center _fd-column _mt-56px _mb-24px _pt-36px _pb-36px _pr-32px _pl-32px _btlr-12px _btrr-12px _bbrr-12px _bblr-12px _btw-1px _brw-1px _borderBottomWidth-1px _borderLeftWidth-1px _btc-border-stro3513 _brc-border-stro3513 _borderBottomColor-border-stro3513 _borderLeftColor-border-stro3513 _backgroundImage-linear-grad1889052070 _items-center _gap-16px _bbs-solid _bts-solid _bls-solid _brs-solid"><p class="is_Text font_mono _col-muted-foreg108704142 _dsp-block _mb-0px _ff-f-family _fs-f-size-1 _lh-f-lineHeigh112922 _mt-0px _mr-0px _ml-0px _ls-1--32px _tt-uppercase" data-slot="text">Hanzo AI Cloud</p><h3 class="is_Text _fs-_md_f-size-10 _lh-_md_2--5rem _col-text-primar121 _dsp-block _mt-0px _mr-0px _mb-0px _ml-0px _fs-f-size-7 _lh-1--6406rem _fw-600 _ls-f-letterSpa1360334199" data-slot="text">Build on Hanzo Cloud</h3><div class="is_View _fd-row _fwr-wrap _gap-12px _mt-8px _justify-center"><a class="is_Text _col-background _dsp-inline-flex _items-center _gap-8px _pt-12px _pb-12px _pr-24px _pl-24px _btlr-9999px _btrr-9999px _bbrr-9999px _bblr-9999px _btw-1px _brw-1px _borderBottomWidth-1px _borderLeftWidth-1px _btc-transparent _brc-transparent _borderBottomColor-transparent _borderLeftColor-transparent _bg-text-primar121 _fs-f-size-3 _lh-f-lineHeigh112922 _fw-600 _bbs-solid _bts-solid _bls-solid _brs-solid" data-slot="text" href="https://platform.hanzo.ai">Start on Hanzo Cloud<svg aria-hidden="true" data-i18n-mirror="true" fill="none" height="14" stroke="var(--background)" stroke-linecap="round" stroke-linejoin="round" stroke-width="2" style="color:var(--background);flex-shrink:0" viewbox="0 0 24 24" width="14"><path d="M5 12h14" stroke="var(--background)"></path><path d="m12 5 7 7-7 7" stroke="var(--background)"></path></svg></a><a class="is_Text _col-text-primar121 _dsp-inline-flex _items-center _gap-8px _pt-12px _pb-12px _pr-24px _pl-24px _btlr-9999px _btrr-9999px _bbrr-9999px _bblr-9999px _btw-1px _brw-1px _borderBottomWidth-1px _borderLeftWidth-1px _btc-border-stro3513 _brc-border-stro3513 _borderBottomColor-border-stro3513 _borderLeftColor-border-stro3513 _bg-transparent _fs-f-size-3 _lh-f-lineHeigh112922 _fw-500 _bbs-solid _bts-solid _bls-solid _brs-solid" data-slot="text" href="/pricing">View Models & Pricing</a></div></div><div class="is_View _fd-column _btw-1px _btc-edge _brc-edge _borderBottomColor-edge _borderLeftColor-edge _pt-32px _bts-solid"><div class="is_View _fd-row _fwr-wrap _justify-space-betwe3241 _items-center _gap-16px _mb-32px _pt-16px _pb-16px _pr-20px _pl-20px _bg-card _btw-1px _brw-1px _borderBottomWidth-1px _borderLeftWidth-1px _btc-border _brc-border _borderBottomColor-border _borderLeftColor-border _btlr-8px _btrr-8px _bbrr-8px _bblr-8px _bbs-solid _bts-solid _bls-solid _brs-solid"><a class="is_View _fd-column _gap-2px _items-flex-start" href="/benchmarks/memoryagentbench"><div class="is_View _fd-row _items-center _gap-4px"><svg aria-hidden="true" data-i18n-mirror="true" fill="none" height="12" stroke="var(--muted-foreground)" stroke-linecap="round" stroke-linejoin="round" stroke-width="2" style="color:var(--muted-foreground);flex-shrink:0" viewbox="0 0 24 24" width="12"><path d="m12 19-7-7 7-7" stroke="var(--muted-foreground)"></path><path d="M19 12H5" stroke="var(--muted-foreground)"></path></svg><span class="is_Text _col-muted-foreg108704142 _fs-f-size-1 _lh-f-lineHeigh112922 _tt-uppercase _ls-0--44px" data-slot="text">Previous</span></div><span class="is_Text _col-text-primar121 _fs-f-size-3 _lh-f-lineHeigh112922 _fw-500" data-slot="text">MemoryAgentBench</span></a><a class="is_Text _col-muted-foreg108704142 _pt-6px _pb-6px _pr-12px _pl-12px _btlr-4px _btrr-4px _bbrr-4px _bblr-4px _bg-edge _btw-1px _brw-1px _borderBottomWidth-1px _borderLeftWidth-1px _btc-border _brc-border _borderBottomColor-border _borderLeftColor-border _fs-f-size-2 _lh-f-lineHeigh112922 _bbs-solid _bts-solid _bls-solid _brs-solid" data-slot="text" href="/benchmarks">All Benchmarks Board</a><a class="is_View _fd-column _gap-2px _items-flex-end" href="/benchmarks/cronquestions"><div class="is_View _fd-row _items-center _gap-4px"><span class="is_Text _col-muted-foreg108704142 _fs-f-size-1 _lh-f-lineHeigh112922 _tt-uppercase _ls-0--44px" data-slot="text">Next</span><svg aria-hidden="true" data-i18n-mirror="true" fill="none" height="12" stroke="var(--muted-foreground)" stroke-linecap="round" stroke-linejoin="round" stroke-width="2" style="color:var(--muted-foreground);flex-shrink:0" viewbox="0 0 24 24" width="12"><path d="M5 12h14" stroke="var(--muted-foreground)"></path><path d="m12 5 7 7-7 7" stroke="var(--muted-foreground)"></path></svg></div><span class="is_Text _col-text-primar121 _fs-f-size-3 _lh-f-lineHeigh112922 _fw-500" data-slot="text">CronQuestions</span></a></div><div class="is_View _fd-column _mb-24px"><h3 class="is_Text font_mono _col-text-primar121 _dsp-block _mt-0px _mb-12px _fs-f-size-3 _lh-f-lineHeigh112922 _ff-f-family _fw-500" data-slot="text">Related in memory</h3><div class="is_View _fd-row _fwr-wrap _gap-10px"><a class="is_Text _col-text-primar121 _dsp-inline-flex _items-center _gap-6px _pt-6px _pb-6px _pr-12px _pl-12px _btlr-6px _btrr-6px _bbrr-6px _bblr-6px _bg-edge _btw-1px _brw-1px _borderBottomWidth-1px _borderLeftWidth-1px _btc-border _brc-border _borderBottomColor-border _borderLeftColor-border _fs-f-size-2 _lh-f-lineHeigh112922 _bbs-solid _bts-solid _bls-solid _brs-solid" data-slot="text" href="/benchmarks/locomo"><span class="is_Text _col-inherit" data-slot="text">LoCoMo</span><svg aria-hidden="true" data-i18n-mirror="true" fill="none" height="12" stroke="var(--muted-foreground)" stroke-linecap="round" stroke-linejoin="round" stroke-width="2" style="color:var(--muted-foreground);flex-shrink:0" viewbox="0 0 24 24" width="12"><path d="M5 12h14" stroke="var(--muted-foreground)"></path><path d="m12 5 7 7-7 7" stroke="var(--muted-foreground)"></path></svg></a><a class="is_Text _col-text-primar121 _dsp-inline-flex _items-center _gap-6px _pt-6px _pb-6px _pr-12px _pl-12px _btlr-6px _btrr-6px _bbrr-6px _bblr-6px _bg-edge _btw-1px _brw-1px _borderBottomWidth-1px _borderLeftWidth-1px _btc-border _brc-border _borderBottomColor-border _borderLeftColor-border _fs-f-size-2 _lh-f-lineHeigh112922 _bbs-solid _bts-solid _bls-solid _brs-solid" data-slot="text" href="/benchmarks/memoryagentbench"><span class="is_Text _col-inherit" data-slot="text">MemoryAgentBench</span><svg aria-hidden="true" data-i18n-mirror="true" fill="none" height="12" stroke="var(--muted-foreground)" stroke-linecap="round" stroke-linejoin="round" stroke-width="2" style="color:var(--muted-foreground);flex-shrink:0" viewbox="0 0 24 24" width="12"><path d="M5 12h14" stroke="var(--muted-foreground)"></path><path d="m12 5 7 7-7 7" stroke="var(--muted-foreground)"></path></svg></a><a class="is_Text _col-text-primar121 _dsp-inline-flex _items-center _gap-6px _pt-6px _pb-6px _pr-12px _pl-12px _btlr-6px _btrr-6px _bbrr-6px _bblr-6px _bg-edge _btw-1px _brw-1px _borderBottomWidth-1px _borderLeftWidth-1px _btc-border _brc-border _borderBottomColor-border _borderLeftColor-border _fs-f-size-2 _lh-f-lineHeigh112922 _bbs-solid _bts-solid _bls-solid _brs-solid" data-slot="text" href="/benchmarks/locomo-subject-scope"><span class="is_Text _col-inherit" data-slot="text">LoCoMo · subject scope</span><svg aria-hidden="true" data-i18n-mirror="true" fill="none" height="12" stroke="var(--muted-foreground)" stroke-linecap="round" stroke-linejoin="round" stroke-width="2" style="color:var(--muted-foreground);flex-shrink:0" viewbox="0 0 24 24" width="12"><path d="M5 12h14" stroke="var(--muted-foreground)"></path><path d="m12 5 7 7-7 7" stroke="var(--muted-foreground)"></path></svg></a><a class="is_Text _col-text-primar121 _dsp-inline-flex _items-center _gap-6px _pt-6px _pb-6px _pr-12px _pl-12px _btlr-6px _btrr-6px _bbrr-6px _bblr-6px _bg-edge _btw-1px _brw-1px _borderBottomWidth-1px _borderLeftWidth-1px _btc-border _brc-border _borderBottomColor-border _borderLeftColor-border _fs-f-size-2 _lh-f-lineHeigh112922 _bbs-solid _bts-solid _bls-solid _brs-solid" data-slot="text" href="/benchmarks/locomo-conv"><span class="is_Text _col-inherit" data-slot="text">LoCoMo-Conv</span><svg aria-hidden="true" data-i18n-mirror="true" fill="none" height="12" stroke="var(--muted-foreground)" stroke-linecap="round" stroke-linejoin="round" stroke-width="2" style="color:var(--muted-foreground);flex-shrink:0" viewbox="0 0 24 24" width="12"><path d="M5 12h14" stroke="var(--muted-foreground)"></path><path d="m12 5 7 7-7 7" stroke="var(--muted-foreground)"></path></svg></a></div></div><div class="is_View _fd-column"><span class="is_Text font_mono _col-muted-foreg108704142 _dsp-block _mb-8px _ff-f-family _fs-f-size-2 _lh-f-lineHeigh112922 _tt-uppercase _ls-0--52px" data-slot="text">Other benchmarks</span><p class="is_Text _col-text-second96872 _dsp-block _maxW-720px _lh-1--3563rem _mt-0px _mr-0px _mb-0px _ml-0px _fs-f-size-2" data-slot="text"><span class="is_Text _col-inherit" data-slot="text"><a class="is_Text _col-text-second96872" data-slot="text" href="/benchmarks/cronquestions">CronQuestions</a></span><span class="is_Text _col-inherit" data-slot="text"> · <a class="is_Text _col-text-second96872" data-slot="text" href="/benchmarks/repobench-r">RepoBench-R</a></span><span class="is_Text _col-inherit" data-slot="text"> · <a class="is_Text _col-text-second96872" data-slot="text" href="/benchmarks/fleet">Fleet residency</a></span><span class="is_Text _col-inherit" data-slot="text"> · <a class="is_Text _col-text-second96872" data-slot="text" href="/benchmarks/goroutine">Live agent footprint</a></span><span class="is_Text _col-inherit" data-slot="text"> · <a class="is_Text _col-text-second96872" data-slot="text" href="/benchmarks/sandbox">Sandbox cold start</a></span><span class="is_Text _col-inherit" data-slot="text"> · <a class="is_Text _col-text-second96872" data-slot="text" href="/benchmarks/inference">Engine prefill on CUDA vs llama.cpp</a></span><span class="is_Text _col-inherit" data-slot="text"> · <a class="is_Text _col-text-second96872" data-slot="text" href="/benchmarks/gpqa">GPQA-Diamond</a></span></p></div></div></div><p class="is_Text font_mono _col-muted-foreg108704142 _dsp-block _mb-10px _ff-f-family _fs-f-size-1 _lh-f-lineHeigh112922" data-slot="text">raw runs and scoring: <!-- -->hanzoai/benchmarks · brain<!-- --> · regenerated by <code>node longmemeval/engine.mjs --embed=minilm && node results.mjs</code></p></main><!--$--><!--/$--></div> |