mirror of
https://github.com/hanzoai/ml.git
synced 2026-11-19 23:50:51 +00:00
hanzo-inc/cloud apps/research (record.go) decodes POST /v1/research/runs
as {"runs": [...]} and /benchmarks as {"benchmarks": [...]}, and reads
nothing else.
post_run and post_benchmark sent the bare record, which the server
reads as an empty batch and refuses 400, "nothing to record". Each now
sends its record as a batch of one.
post_batch sent {"runs", "benchmarks"} to /runs, where the benchmarks
were dropped. It now files the benchmarks at /benchmarks, then the runs
at /runs, sends nothing for a kind given nothing, and answers both
batches.
Comparison read `differences`, which the server never sends. It sends
`config`, and the two runs as `a` and `b`, all of which Comparison now
reads.
The response cap was 8 MiB, which the hanzo org's public run listing
(1,489,873 bytes at 356 runs today, 4.2 KB a run) passes near 2,000
runs, and the server pages no listing. It is 64 MiB, the Python
package's, and every read goes through one function that refuses a
body past it: Api::call and get_artifact cut a body at the cap instead,
so a long listing failed as bad JSON and a long artifact came back
short.
tests/record.rs runs the client against a stand-in that answers as
those handlers do: encoding/json v1 decoding (unknown keys ignored,
names matched without case), the 400, 413 and 422 refusals as problem
documents, versions appended by content and read back canonical,
completion and key derived, compare's blocks, config and deltas, nil
slices as null. The old bodies are refused or dropped there, the new
ones filed; a 2,100-run listing past 8 MiB reads, and a body one byte
past the cap is refused. token() hands the error variant to map_err,
which clippy asked for.
hanzo-research 0.2.2.
12 KiB
12 KiB
LLM.md — hanzoai/ml
Fast, multi-backend tensor & ML framework for Rust (CPU · CUDA · Metal · ROCm · Vulkan) with quantization (GGUF/GGML/AFQ/GPTQ/AWQ). The compute core beneath Hanzo inference.
Canonical role
- This repo is the canonical implementation of the Hanzo Rust ML/tensor core. One impl, one place — discovery/wrapper repos link here, never copy the code.
- Rust is the 2nd-most-complete ecosystem (Python → Rust → C++ → Go → …).
- Crate
hanzo-ml(crates.io) · docs at docs.rs/hanzo-ml.
Install / run
- Core:
cargo add hanzo-ml(Tensor/Device); addhanzo-nnto build models. - GPU:
--features cuda(+cudnn), ormetal/rocm/vulkan. - Examples:
cargo run --example quantized --release(seehanzo-ml-examples/).
Key entry points
hanzo-ml/— core ops, devices,Tensor.hanzo-nn/— layers & model building.hanzo-transformers/— model implementations.hanzo-kernels/,hanzo-flash-attn/— CUDA kernels & FlashAttention v2.hanzo-onnx/,hanzo-datasets/,hanzo-ml-wasm-examples/.hanzo-research/— the/v1/researchclient, the Rust peer of the Pythonhanzo-research. A record batch goes to its kind's route under its kind's key ({"runs": [...]}), the body hanzo-inc/cloudapps/researchdecodes;tests/record.rsis a stand-in that answers as those Go handlers do. Canonical JSON is RFC 8785 (numbers as ECMAScript writes the double, Ryu's digits), held to the Python package's bytes and WAL bytests/vectors/python.jsonandwal.jsonl(tests/vectors/gen.pyregenerates them). serde_json'sfloat_roundtripis on so a WAL line reads back as the double that was hashed; it unifies into every consumer's build.hanzo-train/— training on hanzo-ml.cluster: one model across heterogeneous machines (Metal, CUDA, ROCm, CPU) by local SGD with an outer Nesterov step (DiLoCo). A coordinator owns the plan, θ_global in F32, the outer step and checkpoints; workers join any time, take batches one at a time, send θ_local − θ_global in bf16 with error feedback; deltas sum in join order, so a run equals its rounds replayed in one process (cluster::replay) bit for bit, and a checkpoint (outer momentum and carry, merged batches, round, every worker's AdamW state) resumes bit-identically. A worker that drops has its batches requeued; a round waits for no one past its deadline plusgrace(a silent member is dropped, its link shut), a join under a member's name replaces it, links have keepalive and bounded reads/writes (net::link), and a worker whose link fails joins again on its own with backoff. A model plugs in behindcluster::Model(varsF32 masters,rateper parameter —Nonefrozen,stepover a batch of row indices the caller planned); the coordinator's owner behindcluster::Keep(round line, validation, save).adam: AdamW with per-parameter rates and open moments (hanzo-nn'soptim::adamwupdate, fused on Metal).gpu: free-memory floor.dspark: the DSpark draft on the runtime (hanzo-train fit/join). Tests:cargo test -p hanzo-train(tiny classifier and synthetic DSpark cache, all CPU). Kai (hanzoai/decisiontrain) is one user.
Backends and models
- Vulkan dtypes. Storage is F32 or U32 (BF16 is held as F32, I64 as U32). U32 reductions
run their own kernel (
reduce_u32.comp; argmax/argmin take the first index on ties, as the CPU does) and U32 binary ops take the CPU fallback; any other storage type in a reduction is an error, never an F32 answer.cmp,affine,powfand unary kernels are F32 only.hanzo-ml/tests/vulkan_dtype_tests.rsholds each against the CPU. - Masked softmax on Vulkan. A row's running max starts at
-inf(softmax_rows_blk.spv), so a row masked throughout is uniform, not NaN (softmax_masked_row_is_uniform). The same-3.4e38start is still in the DSL (hanzo-kerneldag.rsRed::Max,attn.rs,flash.rs,quant.rs) and insdpa_blk.spv,flash_attn_dsl_d128.spvandmoe_route.spv: fix it in the DSL and regenerate, with a fully masked row in each test. - ModernBert past its trained positions.
Config::yarn(s)gives a longer sequence YaRN rotary tables (NTK-by-parts, beta 32 and 1, attention factor 0.1 ln s + 1) on the global layers and extended frequencies on the local ones; withinmax_position_embeddingsevery pass reads the trained tables bit for bit. - ModernBert's trained rotary tables take their angles from the host. Each angle
p · fis one F32 product on the host, its sine and cosine taken on the device in F32, then cast: the bits an exact F32 GEMMt · inv_freqgave on the CPU, ROCm, Metal and CUDA (f32_tables_are_the_exact_gemm_bits). The device GEMM it replaces ran on Vulkan's fp16 matrix cores, rounding positions past 2,048 (rows of 4,096 and 8,192 tokens read 25% and 37% from the CPU's states), and in BF16 rounded positions past 256 (to multiples of 2 from 257, more above) and the angles themselves well below 256, so every BF16 pass on every device read other positions than an F32 one: a CUDA BF16 pass read 5.7%, 25% and 36% from the CPU's F32 states at 256, 512 and 1,000 tokens with the old tables, 1.35%, 1.2% and 3.4% with these.trained_tables_hold_every_position_on_every_deviceholds every device's tables to the exact angles (--features rocm|vulkan|metal|cuda): F32 within 1e-3 (2e-3 on CUDA, whose kernels build with--use_fast_math), BF16 within 1e-2. - Past 2^31 elements on CUDA. A tensor of more than 2^31 elements miscomputes on CUDA with no error: the whole padded F32 pass at 16,384 tokens (12 heads, 3.2e9 scores) read 56% from the CPU's states. Tiles keep every score block far below it; a follow-up should refuse such a tensor rather than compute it.
- Vulkan's pool trims.
Device::trimgives Vulkan's idle buffers back too (VulkanDevice::trim, keeping 1 GiB, as ROCm's pool keeps): it flushes and waits, so between forwards, never in one. Its pool otherwise keeps up to 3/5 of the heap idle (30 GiB after a 32,768-token pass on evo). - ModernBert's long padded pass, a tile at a time. A padded pass whose row's scores pass
LOGITS(2^26, 256 MiB in F32: a row past 2,364 tokens over 12 heads) attends a tile of queries at a time (Tiles): a global tile reads every key, at most 2^26 scores over its rows; a local tile holds 512 queries and the keys within their window. The keys' mask is[B, 1, 1, L]and each local tile's window[t, t + 2w](tiles alike share one, built on the pass's device), so no[L, L]tensor exists; a shorter pass is the whole pass, bit for bit. Kᵀ and V are laid out once a layer (Vulkan's matmul copies a strided operand per call), and the raw scores drop once masked: two blocks live at once. (Dropping the masked scores too once the softmax is queued grew Metal's pool past 13 GiB at 16,384 tokens.) Each stage is a TRACE span (layerwithat,qkv,rope,tileoverscores,mask,softmax,values, thenjoin,proj,mlp), which a subscriber can meter.tests/modernbert_memory.rscounts a 20,480-token pass's heap (no block of L x L bytes, less than one[L, L]F32 tensor held) and holds it to the packed pass within 5e-5 of the scale. - ModernBert's residual stream.
ModernBert::stream(xs, mask, at)stops the passforwardruns at the input of layerat(Stream: packed or padded, as that pass lays it out), andModernBert::finishruns it on:finish(stream(.., at))isforwardbit for bit at everyat, packed and padded.Stack::new(vb, config, at)loads layersat..and the final norm apart (a trainable copy of a top, for hanzo-inc/kai's routed yes/no reader), reading another pass's stream;Stream::rowsandStream::packedtake a packed stream apart by sequence and put it back, the same bits.
Next: synthesis as a training loop
Capability training today is dataset → batches → optimizer → checkpoint, with curricula generated and scored by scripts outside it (hanzo-inc/kai's jailbreak round, 2026-10-03). Its pieces belong here, in this order:
hanzo-trainkeeps per-example signals (loss, margin, prediction, gradient norm) instead of folding them into the batch loss, and reports balanced accuracy and MCC per slice.- A dataset rows can join and leave mid-run, each row carrying its provenance and its factors (what a generated row varies), with weights a sampler reads.
- Contrast pairs as a training item: two rows and a margin on their score gap.
- Micro-fits with successive halving (
hanzo-train sweep): many curricula at a tenth of a run, the best continued. - Gradient embeddings over a capability's parameters only, to select the generated rows whose updates align with real errors. Generation itself (propose, mutate, verify against more than its own model) is a separate crate on top of these; a generator that only checks itself learns nothing new.
Releasing
- Registry is crates.io, owner
zeekay. There is no Hanzo cargo registry: no[registries]in.cargo/config.toml, nopublish = [...]allow-list, and the sibling Rust repo (hanzoai/engine) publishes the same way. - Each crate names everything itself: no workspace inheritance. A crate's manifest holds
its own version, package fields, lints and every dependency spec (a sibling as
pathplus the exactversionit was published at); the root holds onlymembers,exclude,resolverand profiles. What a crate ships is what its own manifest says. - Each crate carries its own version and moves by a patch bump from the version it last released. Crates change at different rates, so their numbers differ — that is information, not drift. Never renumber a crate to match another.
scripts/publish-orderis the release set: every crate that does not saypublish = false, topologically sorted.publish = falseis the one way to keep a crate off the registry (examples, demos, the book, the PyPI extension module, andtensor-tools, whose crates.io name belongs to the upstream candle author).- Publishing is CI's job:
.github/workflows/publish.yml, on av*tag or a manual dispatch (gh workflow run publish.yml -R hanzoai/ml --ref main), walks that order withcargo publish --no-verify(GPU build scripts can't run on crates.io builders). It publishes a crate only when its manifest version is ahead of crates.io, so re-running is safe. The token isCARGO_REGISTRY_TOKENat/@prodin org hanzo's KMS, read as thehanzo-ciidentity (universekms-grants.yaml). - A crate whose source changes takes a patch bump in the same commit. The ratchet only
sees versions: a crate edited at an already-published version never ships, and its
consumers pinning crates.io read older code than main (six crates had drifted, among them
hanzo-rocm-kernels, before027c26d9e). Consumers pin published versions, never a git rev (hanzo-inc/kai since 9b382b4). - Run a crate's own tests before its version moves. ROCm needs
LD_LIBRARY_PATH=/opt/rocm/core-7.13/lib—libhiprtc.so.7lives there, not in/opt/rocm/lib, socargo test -p hanzo-kernel --features rocmotherwise dies at load time with the test binary already built.
Brand rules (enforce in all docs)
- Hanzo is the Open AI Cloud / full AI SDK — never an "LLM gateway", never positioned vs LiteLLM, never an "OpenAI-compatible proxy". Purge that framing.
- Paths are
/v1/, never/api/. - Zen models are our own family — don't present upstream model names as ours.
Spec: ~/work/hanzo/SDK-ARCHITECTURE.md — the canonical one-way SDK model.