Files
Eric BuehlerandGitHub dc1b46cb76 feat(cuda): implement cuda graphs and various optimizations (#2180)
* First stab at it

* Integrate flashinfer for decode

* Specialize a but

* Tuning

* Use CUDA graph HtoD cache guard

* Candle

* Better integrate flashdecode

* Ckeanup

* Clean

* Integrate ModelForwardContext

* Progress

* Even better

* Cleanup

* Cleaner

* Some enabled more

* Fix

* Fixes

* UPdate

* perf(cuda): reduce gemma4 prefill overhead

* Add flashinfer prefill chunking kernels

* Tune gqa ratio for flashinfer prefill kernel

* Add proper moe routint kernel

* Optimize prefill overhead, strided rmsnorm fast path

* Chunk perfill default

* Cleanup

* Docs

* Default on

* Refactor

* Add intiail cutile backend for moe bits

* RPogress

* Fix cutile warmup

* Cleaner logging

* Closed form solution for all possible sizes

* MoE: make ENK [E,N,K] the canonical layout, unify loaders, drop transposed kernels

Make [E,N,K] = [E,out,in] (natural HF/vLLM) the single canonical MoE weight
layout. The cuTile fused_moe_kernel now addresses B as ENK directly (K
contiguous, vLLM offs_bn % N for the last-tile bound); loaders normalize every
checkpoint to ENK via StackedExpertWeights, transposing only non-canonical
conv-A checkpoints. The Fused backend consumes ENK via moe_gemm.

Delete the now-dead EKN path: moe_gemm_transposed (Rust wrapper + FFI), the
transposed wmma/gemv CUDA kernels, and the stacked_format flag.

Fix a CUDA decode-graph teardown UAF: captured decode graphs reference the
engine thread's thread-local cuTile JIT modules, but were destroyed on the main
thread after that thread exited and unloaded them. Destroy them in
EngineInstance::drop (via a cleanup_cuda_graphs Pipeline hook) while the engine
thread is still alive.

* Refactor experts backends

* Refactor cutile kernels

* Set features

* DOcs

* Tweak

* Temp fix

* Add cutile attn kernels

* Clean

* Cleanup

* Improve decode perf on b200

* Cleanup

* Fmt

* Fix

* Fix

* Fix

* Fix

* Add report

* Bump

* Add ui gif

* Add ui gif

* version
2026-06-01 10:01:58 -04:00

15 MiB

The file is too large to be shown. View Raw