Unbrand the pre-existing HANZO_* env vars (multinode MN_*, SANDBOX, VK_*, NO_MMAP/NO_MLA/NO_NCCL, IGPU_MEMORY_FRACTION, ...) to bare functional names, keeping their meaning. No HANZO_ env var branding remains.
9.2 KiB
Troubleshooting
Common issues and solutions for Hanzo Engine.
Debug Mode
Enable debug mode for more information:
DEBUG=1 hanzo run -m <model>
Debug mode causes:
- If loading a GGUF or GGML model, outputs a file containing the names, shapes, and types of each tensor:
hanzo_gguf_tensors.txtorhanzo_ggml_tensors.txt
- Increased logging verbosity
System Diagnostics
Run the built-in diagnostics tool:
hanzo doctor
This checks your system configuration and reports any issues.
Common Issues
CUDA Issues
Setting the CUDA compiler path:
- Set the
NVCC_CCBINenvironment variable during build
Error: recompile with -fPIE:
- Some Linux distributions require compiling with
-fPIE - Set during build:
CUDA_NVCC_FLAGS=-fPIE cargo build --release --features cuda
Error: CUDA_ERROR_NOT_FOUND or symbol not found:
- For non-quantized models, specify the data type to load and run in
- Use one of
f32,f16,bf16orauto(auto chooses based on device) - Example:
hanzo run -m <model> -d auto
Minimum CUDA compute capability:
- The minimum supported CUDA compute cap is 5.3
- Set a specific compute cap with:
CUDA_COMPUTE_CAP=80 cargo build --release --features cuda
Metal Issues (macOS)
Metal not found (error: unable to find utility "metal"):
-
Install Xcode:
xcode-select --install -
Set the active developer directory:
sudo xcode-select --switch /Applications/Xcode.app/Contents/Developer
error: cannot execute tool 'metal' due to missing Metal toolchain
- Install Metal Toolchain:
xcodebuild -downloadComponent MetalToolchain
Disabling Metal kernel precompilation:
- By default, Metal kernels are precompiled during build time for better performance
- To skip precompilation (useful for CI or when Metal is not needed):
METAL_PRECOMPILE=0 cargo build --release --features metal
Memory Issues
Disabling mmap loading:
- Set
NO_MMAP=1to disable memory-mapped file loading - Forces all tensor data into memory
- Useful if you're seeing mmap-related errors
Out of memory errors:
- Try using quantization:
--isq q4kor--isq q8_0 - Use device mapping to offload layers:
-n 0:16;cpu:16 - Reduce context length with PagedAttention:
--pa-context-len 4096
Model Loading Issues
Model type not auto-detected:
- If auto-detection fails, please raise an issue
- You can manually specify the architecture if needed
Chat template issues:
- Templates are usually auto-detected
- Override with:
-c /path/to/template.jinja - See Chat Templates for details
Python SDK Installation Issues
maturin build failures (missing system libraries):
Building the Python SDK from source requires system libraries. If the build fails with errors about missing headers or libraries, install the prerequisites:
Ubuntu/Debian:
sudo apt install libssl-dev pkg-config
macOS:
brew install openssl pkg-config
You also need a working Rust toolchain. Install it from https://rustup.rs/ if you have not already.
Wheel not found for platform:
Pre-built wheels are only available for supported platform combinations. Check the Python Installation page for the full support matrix. If no wheel exists for your platform, build from source:
git clone https://github.com/hanzoai/engine.git
cd Hanzo Engine/hanzo-pyo3
python -m venv .venv && source .venv/bin/activate
pip install maturin[patchelf]
maturin develop -r --features <your-features>
CUDA version mismatch between pip package and system:
The hanzo-cuda pip package is compiled against a specific CUDA toolkit version. If your system has a different CUDA version, you may see errors like libcudart.so.XX: cannot open shared object file or symbol-not-found crashes at runtime. Solutions:
- Install the matching CUDA toolkit version, or
- Build from source against your local CUDA installation:
maturin develop -r --features "cuda flash-attn cudnn"
Out of Memory (OOM)
OOM during model loading:
The model itself may not fit into GPU memory at full precision. Reduce memory usage by quantizing on the fly with In-Situ Quantization (ISQ):
hanzo run -m <model> --isq q4k
Alternatively, load a pre-quantized model in GGUF or UQFF format, which are already compressed on disk:
# GGUF
hanzo run --format gguf -m <repo> -f <file.gguf>
# UQFF (generate first with `hanzo quantize`, then load)
hanzo run --format uqff -m <model> -f <file.uqff>
You can also offload some layers to CPU with device mapping:
hanzo run -m <model> -n "0:16;cpu:16"
OOM during inference:
If the model loads successfully but you hit OOM while generating, the KV cache is likely consuming too much memory. When PagedAttention is enabled (the default on CUDA), it pre-allocates GPU memory for the KV cache at startup. Reduce this allocation:
# Limit KV cache to a specific context length
hanzo serve -m <model> --pa-context-len 4096
# Or set a fixed memory budget in MB
hanzo serve -m <model> --pa-memory-mb 2048
# Or reduce the fraction of free GPU memory used (default is 0.90)
hanzo serve -m <model> --pa-memory-fraction 0.5
Reducing the maximum number of concurrent sequences also lowers peak memory usage:
hanzo serve -m <model> --max-seqs 8
If none of these help, disable PagedAttention entirely to use a dynamically allocated cache:
hanzo serve -m <model> --no-paged-attn
OOM with multiple models:
When running multiple models via the multi-model configuration, each loaded model consumes GPU memory. Unload models you are not actively using, or configure auto-reload so that only one model is resident at a time.
Model Output Issues
Garbage or repetitive output:
This is usually a chat template mismatch. The model expects a specific prompt format and produces nonsense when it receives the wrong one. Try providing an explicit Jinja template:
hanzo run -m <model> --jinja-explicit chat_templates/<template>.jinja
Browse the chat_templates/ directory for templates matching your model family. See Chat Templates for details.
Model produces tool calls when not expected:
Some models are fine-tuned for tool use and may emit tool-call JSON even when no tools are provided. To suppress this, set tool_choice to "none" in your request:
{
"tool_choice": "none",
"messages": [...]
}
Thinking tags appearing in output:
Models that support thinking mode (such as Qwen3 and DeepSeek) emit <think>...</think> blocks containing chain-of-thought reasoning. This is expected behavior. To disable thinking entirely, pass --thinking false:
hanzo run -m Qwen/Qwen3-4B --thinking false
If you omit the --thinking flag, the chat template's default behavior applies. See CLI Reference for details.
Docker Issues
GPU not available inside container:
Docker does not expose GPUs by default. You must pass the --gpus all flag (requires the NVIDIA Container Toolkit):
docker run --gpus all -p 1234:1234 ghcr.io/hanzoai/engine:latest \
serve -m Qwen/Qwen3-4B
If you see could not select device driver "" with capabilities: [[gpu]], install the NVIDIA Container Toolkit and restart Docker.
Permission denied on Hugging Face cache:
The container needs a writable directory for downloading model files. Mount a host volume for the HF cache:
docker run --gpus all \
-v ~/.cache/huggingface:/root/.cache/huggingface \
-p 1234:1234 \
ghcr.io/hanzoai/engine:latest \
serve -m Qwen/Qwen3-4B
This also avoids re-downloading models every time you start a new container.
Authentication Issues
"Gated model" or "access denied" errors:
Some models on Hugging Face (such as Llama and Mistral) require you to accept a license agreement and authenticate. First, accept the license on the model's Hugging Face page, then authenticate:
# Interactive login (saves token to ~/.cache/huggingface/token)
hanzo login
# Or provide the token directly
hanzo login --token hf_xxxxxxxxxxxxx
# Or pass a token via environment variable at runtime
hanzo run -m <model> --token-source env:HF_TOKEN
"Model not found" errors:
- Double-check the model ID for typos (e.g.,
mistralai/Mistral-7B-Instruct-v0.1, notmistral/Mistral-7B) - Verify Hugging Face connectivity by running
hanzo doctor-- it tests the connection and token validity - If you are behind a corporate proxy or firewall, ensure
https://huggingface.cois reachable - Check whether the model has been renamed or moved on Hugging Face
Getting Help
If you're still stuck:
- Discord: Community support
- Matrix: Alternative chat
- GitHub Issues: Bug reports and feature requests
When reporting issues, please include:
- Output of
hanzo doctor - Full error message
- Command you ran
- Hardware (GPU model, OS)