Every mention of another inference engine is rewritten to say what our code does, or the math it computes. Reference values are "the reference value from the published config" or "the recorded reference output", line citations into another engine's source become the rule itself, comparisons with other engines are dropped, and asides about other engines' bugs go. Identifiers follow: - CUDA: hanzo-paged-attn's `namespace vllm` is `pagedattn`, hanzo-engine's MoE helpers `moe_dtype` and `moe_detail`, the rotary kernels' `hanzo_rotary`; the shuffle/ldg macros are HANZO_*. - GPTQ: the shuffled-layout path (`reconstruct_exllama`, `use_exllama`) is `reconstruct_shuffled` / `is_shuffled`, on the kernel and the FFI side. - Tests: `*_matches_vllm` and friends name what they compare against (the reference fixture, the tile rule). - MiniCPM-o's vision-LLM embedding is `vlm_embedding`; the rotary custom op is `rotary-embedding`. - hanzo-bench's board scores ours against a "rival" whose name comes from the manifest: `rival_<tag>.json`, `--rival*` manifest flags, `rival t/s` columns; dossier.sh and spec_decode.sh take `--rival-bench`. - install.ps1 installs this repository's hanzo-cli, as install.sh does, with HANZO_INSTALL_NCCL / HANZO_INSTALL_NO_NCCL. - bench_http.py reads only the hanzo: metric families. Kept, because they are what they are: LICENSE, LICENSE-APACHE, NOTICE and tools/upstream-map.sh; each ported file's copyright/licence/provenance header; Cargo.toml dependency names; data identifiers (Hugging Face repo IDs, fixture file names and fixture metadata keys the tests read, on-disk paths, executable names the GPU and engine guards match on a host); model and format names. LLM.md says the rule once, under "One name: Hanzo Engine". Co-authored-by: Hanzo Dev <[email protected]>
2.0 KiB
Enhancing ISQ with an imatrix
Hanzo Engine supports enhancing the performance of models quantized with ISQ by collecting an imatrix from calibration data. The following quantizations are supported with an imatrix:
Q2KQ3KQ4KQ5KQ6K
What is an imatrix? An imatrix (importance matrix) is generated from data collected during the execution of the model on calibration data. This data is used to enhance the performance of the model by enabling a weighted RMSE minimization when quantizing the tensor.
Using an imatrix causes the quantization process to take longer as the data must be collected, but there is no inference-time performance decrease.
Note: Hanzo Engine will automatically generate a .cimatrix file which can be used within Hanzo Engine as a replacement for a .imatrix file. The primary advantage is the in-situ generation within Hanzo Engine. The format is Hanzo Engine's own; tools that read
.imatrixfiles cannot read it.
To use this, simply specify the calibration data file in the various APIs as detailed below.
With the CLI
hanzo run --isq 4 -m meta-llama/Llama-3.2-3B-Instruct --calibration-file calibration_data/calibration_datav3_small.txt
With the Rust SDK
You can find this example here.
let model = TextModelBuilder::new("meta-llama/Llama-3.2-3B-Instruct")
.with_isq(IsqType::Q4K)
.with_calibration_file("calibration_data/calibration_datav3_small.txt".into())
.with_logging()
.with_paged_attn(|| PagedAttentionMetaBuilder::default().build())?
.build()
.await?;
With the Python SDK
You can find this example here.
runner = Runner(
which=Which.Plain(
model_id="meta-llama/Llama-3.2-3B-Instruct",
calibration_file="calibration_data/calibration_datav3_small.txt"
),
in_situ_quant="4",
)