mirror of
https://github.com/zenlm/engine.git
synced 2026-10-30 15:00:05 +00:00
Eric BuehlerandGitHub
c10c22f608
* Flash attn prefill support in paged attn * Add flash attention v3 * Format * Use candle flash attn v3 from candle * Fix build * Update submodules * Fix cargo.lock * Revert flash attn v3 cutlass
690 B
690 B
FlashAttention in mistral.rs
Mistral.rs supports FlashAttention V2 and V3 on CUDA devices (V3 is only supported when CC >= 9.0).
Note: If compiled with FlashAttention and PagedAttention is enabled, then FlashAttention will be used in tandem to accelerate the prefill phase.
Using FlashAttention V2/V3
To use FlashAttention V2/V3, compile with the following feature flags.
| FlashAttention | Feature flag |
|---|---|
| V2 (CC < 9.0) | --features flash-attn |
| V3 (CC >= 9.0) | --features flash-attn-v3 |
Note: FlashAttention V2 and V3 are mutually exclusive Note: To use FlashAttention in the Python API, compile from source.