mirror of
https://github.com/hanzoai/engine.git
synced 2026-11-20 14:17:52 +00:00
* First stab at it * Integrate flashinfer for decode * Specialize a but * Tuning * Use CUDA graph HtoD cache guard * Candle * Better integrate flashdecode * Ckeanup * Clean * Integrate ModelForwardContext * Progress * Even better * Cleanup * Cleaner * Some enabled more * Fix * Fixes * UPdate * perf(cuda): reduce gemma4 prefill overhead * Add flashinfer prefill chunking kernels * Tune gqa ratio for flashinfer prefill kernel * Add proper moe routint kernel * Optimize prefill overhead, strided rmsnorm fast path * Chunk perfill default * Cleanup * Docs * Default on * Refactor * Add intiail cutile backend for moe bits * RPogress * Fix cutile warmup * Cleaner logging * Closed form solution for all possible sizes * MoE: make ENK [E,N,K] the canonical layout, unify loaders, drop transposed kernels Make [E,N,K] = [E,out,in] (natural HF/vLLM) the single canonical MoE weight layout. The cuTile fused_moe_kernel now addresses B as ENK directly (K contiguous, vLLM offs_bn % N for the last-tile bound); loaders normalize every checkpoint to ENK via StackedExpertWeights, transposing only non-canonical conv-A checkpoints. The Fused backend consumes ENK via moe_gemm. Delete the now-dead EKN path: moe_gemm_transposed (Rust wrapper + FFI), the transposed wmma/gemv CUDA kernels, and the stacked_format flag. Fix a CUDA decode-graph teardown UAF: captured decode graphs reference the engine thread's thread-local cuTile JIT modules, but were destroyed on the main thread after that thread exited and unloaded them. Destroy them in EngineInstance::drop (via a cleanup_cuda_graphs Pipeline hook) while the engine thread is still alive. * Refactor experts backends * Refactor cutile kernels * Set features * DOcs * Tweak * Temp fix * Add cutile attn kernels * Clean * Cleanup * Improve decode perf on b200 * Cleanup * Fmt * Fix * Fix * Fix * Fix * Add report * Bump * Add ui gif * Add ui gif * version
15 MiB
15 MiB
The file is too large to be shown.
View Raw