12 KiB
LLM.md — hanzoai/ml
Fast, multi-backend tensor & ML framework for Rust (CPU · CUDA · Metal · ROCm · Vulkan) with quantization (GGUF/GGML/AFQ/GPTQ/AWQ). The compute core beneath Hanzo inference.
Canonical role
- This repo is the canonical implementation of the Hanzo Rust ML/tensor core. One impl, one place — discovery/wrapper repos link here, never copy the code.
- Rust is the 2nd-most-complete ecosystem (Python → Rust → C++ → Go → …).
- Crate
hanzo-ml(crates.io) · docs at docs.rs/hanzo-ml.
Install / run
- Core:
cargo add hanzo-ml(Tensor/Device); addhanzo-nnto build models. - GPU:
--features cuda(+cudnn), ormetal/rocm/vulkan. - Examples:
cargo run --example quantized --release(seehanzo-ml-examples/).
Key entry points
hanzo-ml/— core ops, devices,Tensor.hanzo-nn/— layers & model building.hanzo-transformers/— model implementations.hanzo-kernels/,hanzo-flash-attn/— CUDA kernels & FlashAttention v2.hanzo-onnx/,hanzo-datasets/,hanzo-ml-wasm-examples/.hanzo-research/— the/v1/researchclient, the Rust peer of the Pythonhanzo-research. A record batch goes to its kind's route under its kind's key ({"runs": [...]}), the body hanzo-inc/cloudapps/researchdecodes;tests/record.rsis a stand-in that answers as those Go handlers do. Canonical JSON is RFC 8785 (numbers as ECMAScript writes the double, Ryu's digits), held to the Python package's bytes and WAL bytests/vectors/python.jsonandwal.jsonl(tests/vectors/gen.pyregenerates them). serde_json'sfloat_roundtripis on so a WAL line reads back as the double that was hashed; it unifies into every consumer's build.hanzo-train/— training on hanzo-ml.cluster: one model across heterogeneous machines (Metal, CUDA, ROCm, CPU) by local SGD with an outer Nesterov step (DiLoCo). A coordinator owns the plan, θ_global in F32, the outer step and checkpoints; workers join any time, take batches one at a time, send θ_local − θ_global in bf16 with error feedback; deltas sum in join order, so a run equals its rounds replayed in one process (cluster::replay) bit for bit, and a checkpoint (outer momentum and carry, merged batches, round, every worker's AdamW state) resumes bit-identically. A worker that drops has its batches requeued; a round waits for no one past its deadline plusgrace(a silent member is dropped, its link shut), a join under a member's name replaces it, links have keepalive and bounded reads/writes (net::link), and a worker whose link fails joins again on its own with backoff. A model plugs in behindcluster::Model(varsF32 masters,rateper parameter —Nonefrozen,stepover a batch of row indices the caller planned); the coordinator's owner behindcluster::Keep(round line, validation, save).adam: AdamW with per-parameter rates and open moments (hanzo-nn'soptim::adamwupdate, fused on Metal).gpu: free-memory floor.dspark: the DSpark draft on the runtime (hanzo-train fit/join). Tests:cargo test -p hanzo-train(tiny classifier and synthetic DSpark cache, all CPU). Kai (hanzoai/decisiontrain) is one user.
Backends and models
- Vulkan dtypes. Storage is F32 or U32 (BF16 is held as F32, I64 as U32). U32 reductions
run their own kernel (
reduce_u32.comp; argmax/argmin take the first index on ties, as the CPU does) and U32 binary ops take the CPU fallback; any other storage type in a reduction is an error, never an F32 answer.cmp,affine,powfand unary kernels are F32 only.hanzo-ml/tests/vulkan_dtype_tests.rsholds each against the CPU. - Masked softmax on Vulkan. A row's running max starts at
-inf(softmax_rows_blk.spv), so a row masked throughout is uniform, not NaN (softmax_masked_row_is_uniform). The same-3.4e38start is still in the DSL (hanzo-kerneldag.rsRed::Max,attn.rs,flash.rs,quant.rs) and insdpa_blk.spv,flash_attn_dsl_d128.spvandmoe_route.spv: fix it in the DSL and regenerate, with a fully masked row in each test. - ModernBert past its trained positions.
Config::yarn(s)gives a longer sequence YaRN rotary tables (NTK-by-parts, beta 32 and 1, attention factor 0.1 ln s + 1) on the global layers and extended frequencies on the local ones; withinmax_position_embeddingsevery pass reads the trained tables bit for bit. - ModernBert's trained rotary tables take their angles from the host. Each angle
p · fis one F32 product on the host, its sine and cosine taken on the device in F32, then cast: the bits an exact F32 GEMMt · inv_freqgave on the CPU, ROCm, Metal and CUDA (f32_tables_are_the_exact_gemm_bits). The device GEMM it replaces ran on Vulkan's fp16 matrix cores, rounding positions past 2,048 (rows of 4,096 and 8,192 tokens read 25% and 37% from the CPU's states), and in BF16 rounded positions past 256 (to multiples of 2 from 257, more above) and the angles themselves well below 256, so every BF16 pass on every device read other positions than an F32 one: a CUDA BF16 pass read 5.7%, 25% and 36% from the CPU's F32 states at 256, 512 and 1,000 tokens with the old tables, 1.35%, 1.2% and 3.4% with these.trained_tables_hold_every_position_on_every_deviceholds every device's tables to the exact angles (--features rocm|vulkan|metal|cuda): F32 within 1e-3 (2e-3 on CUDA, whose kernels build with--use_fast_math), BF16 within 1e-2. - Past 2^31 elements on CUDA. A tensor of more than 2^31 elements miscomputes on CUDA with no error: the whole padded F32 pass at 16,384 tokens (12 heads, 3.2e9 scores) read 56% from the CPU's states. Tiles keep every score block far below it; a follow-up should refuse such a tensor rather than compute it.
- Vulkan's pool trims.
Device::trimgives Vulkan's idle buffers back too (VulkanDevice::trim, keeping 1 GiB, as ROCm's pool keeps): it flushes and waits, so between forwards, never in one. Its pool otherwise keeps up to 3/5 of the heap idle (30 GiB after a 32,768-token pass on evo). - ModernBert's long padded pass, a tile at a time. A padded pass whose row's scores pass
LOGITS(2^26, 256 MiB in F32: a row past 2,364 tokens over 12 heads) attends a tile of queries at a time (Tiles): a global tile reads every key, at most 2^26 scores over its rows; a local tile holds 512 queries and the keys within their window. The keys' mask is[B, 1, 1, L]and each local tile's window[t, t + 2w](tiles alike share one, built on the pass's device), so no[L, L]tensor exists; a shorter pass is the whole pass, bit for bit. Kᵀ and V are laid out once a layer (Vulkan's matmul copies a strided operand per call), and the raw scores drop once masked: two blocks live at once. (Dropping the masked scores too once the softmax is queued grew Metal's pool past 13 GiB at 16,384 tokens.) Each stage is a TRACE span (layerwithat,qkv,rope,tileoverscores,mask,softmax,values, thenjoin,proj,mlp), which a subscriber can meter.tests/modernbert_memory.rscounts a 20,480-token pass's heap (no block of L x L bytes, less than one[L, L]F32 tensor held) and holds it to the packed pass within 5e-5 of the scale. - ModernBert's residual stream.
ModernBert::stream(xs, mask, at)stops the passforwardruns at the input of layerat(Stream: packed or padded, as that pass lays it out), andModernBert::finishruns it on:finish(stream(.., at))isforwardbit for bit at everyat, packed and padded.Stack::new(vb, config, at)loads layersat..and the final norm apart (a trainable copy of a top, for hanzo-inc/kai's routed yes/no reader), reading another pass's stream;Stream::rowsandStream::packedtake a packed stream apart by sequence and put it back, the same bits.
Next: synthesis as a training loop
Capability training today is dataset → batches → optimizer → checkpoint, with curricula generated and scored by scripts outside it (hanzo-inc/kai's jailbreak round, 2026-10-03). Its pieces belong here, in this order:
hanzo-trainkeeps per-example signals (loss, margin, prediction, gradient norm) instead of folding them into the batch loss, and reports balanced accuracy and MCC per slice.- A dataset rows can join and leave mid-run, each row carrying its provenance and its factors (what a generated row varies), with weights a sampler reads.
- Contrast pairs as a training item: two rows and a margin on their score gap.
- Micro-fits with successive halving (
hanzo-train sweep): many curricula at a tenth of a run, the best continued. - Gradient embeddings over a capability's parameters only, to select the generated rows whose updates align with real errors. Generation itself (propose, mutate, verify against more than its own model) is a separate crate on top of these; a generator that only checks itself learns nothing new.
Releasing
- Registry is crates.io, owner
zeekay. There is no Hanzo cargo registry: no[registries]in.cargo/config.toml, nopublish = [...]allow-list, and the sibling Rust repo (hanzoai/engine) publishes the same way. - Each crate names everything itself: no workspace inheritance. A crate's manifest holds
its own version, package fields, lints and every dependency spec (a sibling as
pathplus the exactversionit was published at); the root holds onlymembers,exclude,resolverand profiles. What a crate ships is what its own manifest says. - Each crate carries its own version and moves by a patch bump from the version it last released. Crates change at different rates, so their numbers differ — that is information, not drift. Never renumber a crate to match another.
scripts/publish-orderis the release set: every crate that does not saypublish = false, topologically sorted.publish = falseis the one way to keep a crate off the registry (examples, demos, the book, the PyPI extension module, andtensor-tools, whose crates.io name belongs to the upstream candle author).- Publishing is CI's job:
.github/workflows/publish.yml, on av*tag or a manual dispatch (gh workflow run publish.yml -R hanzoai/ml --ref main), walks that order withcargo publish --no-verify(GPU build scripts can't run on crates.io builders). It publishes a crate only when its manifest version is ahead of crates.io, so re-running is safe. The token isCARGO_REGISTRY_TOKENat/@prodin org hanzo's KMS, read as thehanzo-ciidentity (universekms-grants.yaml). - A crate whose source changes takes a patch bump in the same commit. The ratchet only
sees versions: a crate edited at an already-published version never ships, and its
consumers pinning crates.io read older code than main (six crates had drifted, among them
hanzo-rocm-kernels, before027c26d9e). Consumers pin published versions, never a git rev (hanzo-inc/kai since 9b382b4). - Run a crate's own tests before its version moves. ROCm needs
LD_LIBRARY_PATH=/opt/rocm/core-7.13/lib—libhiprtc.so.7lives there, not in/opt/rocm/lib, socargo test -p hanzo-kernel --features rocmotherwise dies at load time with the test binary already built.
Brand rules (enforce in all docs)
- Hanzo is the Open AI Cloud / full AI SDK — never an "LLM gateway", never positioned vs LiteLLM, never an "OpenAI-compatible proxy". Purge that framing.
- Paths are
/v1/, never/api/. - Zen models are our own family — don't present upstream model names as ours.
Spec: ~/work/hanzo/SDK-ARCHITECTURE.md — the canonical one-way SDK model.
Local active inference
hanzo-active-inference/ owns finite-state Bayesian filtering, variational free
energy, exact risk/ambiguity and information-gain EFE, finite open-loop policy
posteriors and Dirichlet A/B learning. CPU f64 values, no tensor/backend dependency.
cargo test -p hanzo-active-inference holds the mathematical identities, refusal
boundaries and learning conservation. Engine hosts consume this crate, never copy
its math. Its README states the limits: no contingent trees, parameter epistemic
value or global loopy graph inference. It has no execution interface.