mirror of
https://github.com/hanzoai/ml.git
synced 2026-11-20 23:08:29 +00:00
Three-way per file through the rename: base 39355c6c91, theirs the upstream
merge of main with 35d7ae7c (via 7ef6a694d), ours origin/main.
Lands the barrier thread pool, runtime CPU feature dispatch, aarch64 and x86
repacked quantized kernels (indexed_gemv, gemv_fused_shared_lhs), quantized
embedding on CPU/Metal/CUDA, paged flash-attn kernels, cutile fused MoE, the
in-place op aliasing API, parking_lot storage locks, Metal buffer builders and
debug labels, safetensors 0.8, hf-hub 1.0, tokenizers 0.23.
Ours keeps ROCm, Vulkan, wgpu, the IQ/MXFP4/NVFP4/TQ dtypes, and the i-quant
weight/activation block split in matmul. Our backends ride InplaceOpN and the
quantized embedding; Metal get_rows gains the IQ kernels. ug leaves, as upstream.
Co-authored-by: Hanzo Dev <[email protected]>
SmolLM3 Unified Inference
A unified Rust implementation for running SmolLM3 models using the Hanzo ML framework. Supports both quantized (GGUF) and full precision (safetensors) models with a single codebase.
Features
- Dual Model Support: Run either quantized or full precision models
- Multiple Quantization Levels: Q4_K_M (1.9GB), Q8_0 (3.3GB), F16 (6.2GB)
- Chat Template Support: Automatic formatting for instruction-tuned models
- Thinking Mode: Enable reasoning traces with
/thinkmode - NoPE Architecture: Supports SmolLM3's mixed RoPE/NoPE layer configuration
- Auto-download: Automatically fetches models from HuggingFace Hub
Quick Start
Quantized Model (Recommended)
cargo run --release --example smollm3 -- \
--model-type quantized \
--quantization q8_0 \
--prompt "Explain Rust's ownership system"
Full Precision Model
cargo run --release --example smollm3 -- \
--model-type full \
--dtype f16 \
--prompt "Write a sorting algorithm in Rust"
Command Line Options
Model Selection
--model-type <TYPE>: Choosequantizedorfull(default: quantized)--model <VARIANT>: Choose3b(instruct) or3b-base(default: 3b)--quantization <LEVEL>: For quantized models -q4_k_m,q8_0, orf16(default: q8_0)--dtype <TYPE>: For full models -f32,f16,bf16, orauto(default: auto)
Generation Parameters
--prompt <TEXT>: The prompt to generate from-n, --sample-len <NUM>: Number of tokens to generate (default: 1000)--temperature <FLOAT>: Sampling temperature, 0 for greedy (default: 0.8)--top-p <FLOAT>: Nucleus sampling probability cutoff--top-k <NUM>: Only sample among top K tokens--repeat-penalty <FLOAT>: Penalty for repeating tokens (default: 1.1)--repeat-last-n <NUM>: Context size for repeat penalty (default: 64)
Advanced Options
--no-chat-template: Disable chat template formatting (use for base models)--thinking: Enable thinking/reasoning mode with/thinktags--split-prompt: Process prompt tokens individually (for debugging)--tracing: Enable performance tracing (generates trace JSON)--model-path <PATH>: Use local model file instead of auto-download--tokenizer <PATH>: Use local tokenizer instead of auto-download
Quantization Comparison
| Level | Size | Quality | Use Case |
|---|---|---|---|
| Q4_K_M | 1.9GB | Good | Fast inference, constrained environments |
| Q8_0 | 3.3GB | Better | Balanced quality and speed |
| F16 | 6.2GB | Best | Maximum quality in GGUF format |
Examples
Creative Writing with Thinking Mode
cargo run --release --example smollm3 -- \
--thinking \
--temperature 0.9 \
--prompt "Write a short sci-fi story about AI"
Code Generation (Base Model)
cargo run --release --example smollm3 -- \
--model 3b-base \
--no-chat-template \
--temperature 0.2 \
--prompt "def fibonacci(n):"
High Quality Output
cargo run --release --example smollm3 -- \
--model-type full \
--dtype f16 \
--temperature 0.7 \
--prompt "Explain quantum entanglement"
Model Architecture
SmolLM3 uses a hybrid RoPE/NoPE architecture:
- RoPE layers: Standard rotary position embeddings (75% of layers)
- NoPE layers: No position embeddings (25% of layers - every 4th layer)
This configuration is automatically detected and handled by the implementation.
Hardware Requirements
- Quantized Q4_K_M: ~2.5GB RAM
- Quantized Q8_0: ~4GB RAM
- Full F16: ~7GB RAM
- Full F32: ~13GB RAM
GPU acceleration supported via CUDA (with cuda feature) or Metal (macOS).
Troubleshooting
Model download fails: Check internet connection and HuggingFace Hub access
Out of memory: Try a smaller quantization level or use --sample-len to reduce generation length
Compilation errors: Ensure you're using the latest version of the Hanzo crate
License
This implementation follows the Hanzo framework license. SmolLM3 models are available under Apache 2.0.