Files
engine/docs/TRAINING.md

4.5 KiB

Training (LoRA fine-tuning)

The engine trains natively in Rust. hanzo-train implements the four Tinker-shaped primitives on the engine's ML stack (hanzo-ml autograd + hanzo-nn AdamW), and the same loop is exposed three ways — CLI, HTTP, and the Python SDK. The base model stays frozen; LoRA factors on the attention + MLP projections are the trainable parameters. Saved adapters are standard PEFT (adapter_model.safetensors + adapter_config.json) and load straight back into the engine for inference.

Supported today: Llama / Qwen2-family decoders (SmolLM2, Qwen2.5, Llama), CPU F32. The primitives:

Primitive What it does
create_lora_training_client load base frozen, inject LoRA Vars
forward_backward forward with grad → masked next-token CE → backward (grads accumulate)
optim_step apply AdamW to the accumulated gradients
sample decode from the current base+LoRA weights
save_weights_and_get_sampling_client write the PEFT adapter for inference

CLI

Datasets are JSONL, one {"prompt": ..., "completion": ...} per line. The loss mask covers exactly the completion (+ EOS), never the prompt.

hanzo train -m HuggingFaceTB/SmolLM2-135M --data sft.jsonl \
    --lora-rank 16 --lr 1e-4 --steps 100 --out ./adapter \
    --sample-prompt "2+2="

--lora-alpha defaults to 2 * rank. The run logs per-step loss and prints the adapter path, trainable-parameter count, and final loss.

Distillation

hanzo distill does sequence-level knowledge distillation: a teacher model samples completions for a {"prompt"} JSONL set, and the student LoRA-fine-tunes on the teacher's outputs. The generated dataset is kept at out/distill.jsonl for inspection; the student adapter lands in out/adapter.

hanzo distill --teacher Qwen/Qwen2.5-1.5B --student HuggingFaceTB/SmolLM2-135M \
    --prompts prompts.jsonl --max-tokens 128 --temperature 0.7 \
    --steps 200 --out ./distilled

HTTP API

hanzo serve exposes the same primitives under /v1/training — the server that does inference also trains. Model load is async: create returns immediately with status: "loading"; poll the client until ready.

Route Method Purpose
/v1/training/clients POST create: {"base_model", "lora_config": {"rank", "alpha", "target_modules"}}
/v1/training/clients GET list clients
/v1/training/clients/{id} GET status, counters, loss_history
/v1/training/clients/{id} DELETE drop the client, free its memory
/v1/training/clients/{id}/forward_backward POST {"data": [...]} → {"loss", "num_tokens", "metrics"}
/v1/training/clients/{id}/optim_step POST {"adam_params": {"lr", ...}}
/v1/training/clients/{id}/sample POST {"prompt" or "tokens", "sampling_params", "num_samples"}
/v1/training/clients/{id}/save_weights POST {"name"} → {"path", "format": "peft"}, written under ~/.cache/hanzo/adapters/{id}/

data entries are either raw text — {"prompt": "...", "completion": "..."}, tokenized server-side with the client's tokenizer — or a pre-tokenized datum: {"model_input": {"tokens": [...]}, "target_tokens": [...], "weights": [...]} (weights is the per-position loss mask). Errors: 400 bad input, 404 unknown client, 409 while loading or after a failed load.

The routes are in the OpenAPI doc at /docs on a running server.

Python SDK

pip install hanzo-train — the Tinker-shaped client over the HTTP API:

from hanzo_train import ServiceClient, LoraConfig, AdamParams, SamplingParams

sc = ServiceClient(base_url="http://localhost:1234")
tc = sc.create_lora_training_client("HuggingFaceTB/SmolLM2-135M",
                                    lora_config=LoraConfig(rank=16))
for batch in batches:  # [{"prompt": ..., "completion": ...}, ...]
    out = tc.forward_backward(batch).result()
    tc.optim_step(AdamParams(lr=1e-4)).result()
print(tc.sample(prompt="2+2=", sampling_params=SamplingParams(max_tokens=8),
                num_samples=1).result().sequences[0].text)
adapter = tc.save_weights_and_get_sampling_client(name="my-adapter").result().path

Rust

Use the crate directly for in-process training — hanzo_train::run_sft is the one-shot loop behind hanzo train, and hanzo_train::TrainingClient is the step-by-step API (see the crate docs).

Using the adapter

The saved directory is a normal PEFT adapter — pass it to the engine's LoRA inference loading (see Adapter Models) to serve the fine-tuned model.