mirror of
https://github.com/hanzoai/laya.git
synced 2026-10-30 19:33:31 +00:00
* feat: optional TileLang GPU fast path (laya.load(..., fast=True)) Adds laya/tl_kernels.py (fused GEMM+epilogue, GEMM+GEGLU, residual+LayerNorm, in-place RoPE, padding-aware sliding-window flash attention) and laya/fast.py, which rebuilds the DecisionModel forward from those kernels with bf16-resident weights and a CUDA graph per (batch, length) bucket. Agent.accelerate() / deaccelerate() and the `fast=` flag on load(); the `laya[fast]` extra pulls tilelang. Nothing is imported or changed unless fast=True is requested; without CUDA or tilelang the stock forward is used (warning, or raise with strict=True). Numerics match the stock bf16 path (AG News / emotion accuracy and ECE identical within noise, 99.5% argmax agreement, same max |dp| vs fp32). 1-question calls go from 14-18 ms to 3-5 ms on an RTX 4070 Ti SUPER; 30-question batches 1.2-1.9x. benchmarks/bench_fast.py reproduces the numbers (--eval N adds the dataset comparison); tests/test_fast.py checks each kernel against torch and the full forward against the stock model (skips without CUDA/tilelang). * fix(fast): match HF's sliding window bound, keep the residual stream in fp32; add parity evidence Two deviations from the stock forward surfaced on a fixed-state parity run (benchmarks/parity_fast.py, new): - The local-attention kernel used ModernBertAttention.sliding_window (config value + 1, flash-attn's inclusive convention). HF's bidirectional mask keeps |i - j| <= config.sliding_window, so every local layer saw one extra key per side. On ModernBERT-large this moved noul probabilities by up to 0.6 on some inputs (mmBERT-base was barely affected). The kernel now takes config.sliding_window. - The residual stream was stored in bf16 between layers; the stock path keeps it in fp32 and ModernBERT-large's residual activations reach ~3e4, where bf16 loses ~100 units per add. add_ln_kernel now updates an fp32 stream and emits the bf16 normalised input; embeddings are gathered from an exact fp16 copy. Result on 60 fixed states x up to 8 questions (both checkpoints): the fast path is at least as close to an fp32 forward as the stock bf16 path (max |dp| vs fp32 <= 0.046 fast vs <= 0.058 stock), argmax agreement >= 47/48 per question type, and on every disagreement the fast path sides with fp32. Per-option probabilities for all three paths are in benchmarks/results/parity_*.json. Latency is unchanged (1-question e2e 3.2 ms multilingual / 5.3 ms English).