Files
cklxxandGitHub 2a2a2b1a3d Optional TileLang GPU fast path: 4-5x lower latency per call, identical answers (#25)
* feat: optional TileLang GPU fast path (laya.load(..., fast=True))

Adds laya/tl_kernels.py (fused GEMM+epilogue, GEMM+GEGLU, residual+LayerNorm,
in-place RoPE, padding-aware sliding-window flash attention) and laya/fast.py,
which rebuilds the DecisionModel forward from those kernels with bf16-resident
weights and a CUDA graph per (batch, length) bucket. Agent.accelerate() /
deaccelerate() and the `fast=` flag on load(); the `laya[fast]` extra pulls
tilelang. Nothing is imported or changed unless fast=True is requested; without
CUDA or tilelang the stock forward is used (warning, or raise with strict=True).

Numerics match the stock bf16 path (AG News / emotion accuracy and ECE identical
within noise, 99.5% argmax agreement, same max |dp| vs fp32). 1-question calls go
from 14-18 ms to 3-5 ms on an RTX 4070 Ti SUPER; 30-question batches 1.2-1.9x.

benchmarks/bench_fast.py reproduces the numbers (--eval N adds the dataset
comparison); tests/test_fast.py checks each kernel against torch and the full
forward against the stock model (skips without CUDA/tilelang).

* fix(fast): match HF's sliding window bound, keep the residual stream in fp32; add parity evidence

Two deviations from the stock forward surfaced on a fixed-state parity run
(benchmarks/parity_fast.py, new):

- The local-attention kernel used ModernBertAttention.sliding_window (config
  value + 1, flash-attn's inclusive convention). HF's bidirectional mask keeps
  |i - j| <= config.sliding_window, so every local layer saw one extra key per
  side. On ModernBERT-large this moved noul probabilities by up to 0.6 on some
  inputs (mmBERT-base was barely affected). The kernel now takes
  config.sliding_window.
- The residual stream was stored in bf16 between layers; the stock path keeps
  it in fp32 and ModernBERT-large's residual activations reach ~3e4, where bf16
  loses ~100 units per add. add_ln_kernel now updates an fp32 stream and emits
  the bf16 normalised input; embeddings are gathered from an exact fp16 copy.

Result on 60 fixed states x up to 8 questions (both checkpoints): the fast path
is at least as close to an fp32 forward as the stock bf16 path (max |dp| vs fp32
<= 0.046 fast vs <= 0.058 stock), argmax agreement >= 47/48 per question type,
and on every disagreement the fast path sides with fp32. Per-option
probabilities for all three paths are in benchmarks/results/parity_*.json.
Latency is unchanged (1-question e2e 3.2 ms multilingual / 5.3 ms English).
2026-09-23 19:01:15 +05:30
..