mirror of
https://github.com/hanzoai/engine.git
synced 2026-11-21 02:36:58 +00:00
Compressed layers previously had NO memory past the 128-token raw window at decode (the compressor only ran at prefill; decode attention was window-clamped) -> beyond ~128 tokens, 40/43 layers were amnesic: the model lost the question, wandered, never terminated (ds4 concludes+EOS at ~784; ours looped to any cap). Now: CompressorState = chunked input history (O(1) append) + incrementally-emitted compressed rows (per-'ratio' cadence, slice-recompute with overlap window); SELF-HEALING under speculative rollback via sync_to(absolute position) — no plumbing into the cache guards. Decode/verify beyond the window attends [raw-window slice ++ compressed rows] with the per-row window/causal combined mask (eager; s<=window keeps the fused kernel). combined_compressed_mask gains absolute raw offsets + the raw sliding-window clamp (also fixes >128-token PREFILL raw attention, previously unclamped). dense_attn extracted (one fused-path impl for both arms). VALIDATED: step-0 unchanged (We ~certainty); 900-tok sampled run now EXITS thinking and writes the structured final answer (markdown, post-</think>) — first time on this stack; ds4-matching behavior. Honest cost: ~5.7 t/s at depth on this correctness-first eager path (the prior 14.45 'fast' number was the amnesic config skipping memory). Next: s==1 deep-ctx fused path (mask provably all-zeros after pre-slicing -> kernel over [window ++ comp], no eager).