mirror of
https://github.com/hanzoai/engine.git
synced 2026-11-20 23:45:50 +00:00
Replace all remaining references to the upstream project/author/framework brands with hanzo / hanzoai across source, comments, docs, examples, and build scripts. - Project spellings mistral.rs / mistral-rs / mistralrs -> hanzo (and the hanzo-* crate/binary names in Dockerfiles, Makefile, install.ps1, .gitignore). - Author EricLBuehler / EricBuehler / EricB / Eric Buehler -> hanzoai / Hanzo AI (incl. Cargo `authors` and LICENSE copyright). - URLs: github.com/EricLBuehler/mistral.rs -> github.com/hanzoai/engine; ericlbuehler.github.io/mistral.rs -> hanzoai.github.io/engine; github.com/huggingface/candle -> github.com/hanzoai/ml; docs.rs/candle-* -> docs.rs/hanzo-ml; ghcr.io image -> ghcr.io/hanzoai/engine. - Env var prefix MISTRALRS_ -> HANZO_ (producers and consumers kept in sync, incl. the build-script-injected HANZO_GIT_REVISION). - ROCm static lib libmistralrsrocm.a / mistralrsrocm -> libhanzorocm.a / hanzorocm. - localStorage keys mistralrs_settings -> hanzo_settings. Deliberately preserved: real model architectures (struct Mistral, models/mistral.rs file, Mixtral/Llama/Gemma/Qwen/Phi), real model-org HF repo-ids (mistralai/, meta-llama/, google/, Qwen/, microsoft/), the `use hanzo_ml as candle` code aliases and `candle::` identifiers, and the upstream-author HF data repo-id suffix in "hanzoai/mistralrs_tests". Both `cargo check` and `cargo check --features vulkan` pass with 0 errors. Co-authored-by: Hanzo Dev <[email protected]>
2.5 KiB
2.5 KiB
Structured model loading with .toml files
Hanzo Engine supports loading models from a .toml file, and the fields are the same as for the CLI. Please find some example toml selectors here.
There are a few cases which add functionality that cannot be found in the CLI.
Speculative decoding
What to specify
Under [speculative]
- Specify the
gammaparameter
Under [speculative.draft_model]
- Choose a draft model, just like under
[model](only requirement is that they have the same tokenizer)
[model]
model_id = "mistralai/Mistral-7B-Instruct-v0.1"
arch = "mistral"
[speculative]
gamma = 32
[speculative.draft_model]
tok_model_id = "mistralai/Mistral-7B-Instruct-v0.1"
quantized_model_id = "TheBloke/Mistral-7B-Instruct-v0.1-GGUF"
quantized_filename = "mistral-7b-instruct-v0.1.Q2_K.gguf"
hanzo from-config -f toml-selectors/speculative-gguf.toml
AnyMoE
What to specify
Under [anymoe], required unless specified
- Specify the dataset
- Find and specify the prefix/mlp values
- Go to
https://huggingface.co/<MODEL ID>/tree/main?show_file_info=model.safetensors.index.json - Look for the mlp layers: For example
model.layers.27.mlp.down_proj.weightmeans that the prefix ismodel.layersand the mlp ismlp.
- Go to
- Specify the expert or LoRA adapter model IDs
- (Optional) Specify layers to apply AnyMoE to.
Under [anymoe.config]
- Hidden size, typically found at
https://huggingface.co/<BASE MODEL ID>/blob/main/config.json
(For LoRA experts) Under [anymoe.config.expert_type.lora_adapter]
- Rank
- Alpha
- Target modules
hanzo from-config -f toml-selectors/anymoe.toml
With fine-tuned experts
[model]
model_id = "mistralai/Mistral-7B-Instruct-v0.1"
arch = "mistral"
[anymoe]
dataset_json = "test.csv"
prefix = "model.layers"
mlp = "mlp"
model_ids = ["HuggingFaceH4/zephyr-7b-beta"]
layers = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]
[anymoe.config]
hidden_size = 4096
expert_type = "fine_tuned"
With LoRA adapter experts
[model]
model_id = "HuggingFaceH4/zephyr-7b-beta"
arch = "mistral"
[anymoe]
dataset_json = "test.csv"
prefix = "model.layers"
mlp = "mlp"
model_ids = ["hanzoai/example_adapter"]
layers = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]
[anymoe.config]
hidden_size = 4096
[anymoe.config.expert_type.lora_adapter]
rank = 16
alpha = 16
target_modules = ["gate_proj"]