mirror of
https://github.com/hanzoai/ml.git
synced 2026-11-20 23:08:29 +00:00
cluster is Kai's DiLoCo runtime from hanzoai/decision, model-agnostic: a coordinator (plan, θ_global in F32, outer Nesterov step, checkpoints of outer and every worker's AdamW state, resume bit for bit, requeue on drop), workers that join any time on any device, bf16 deltas with error feedback summed in join order, and the hang fix of decision 2f436df (keepalive and bounded links, a round closed at its deadline plus a grace, same-name rejoin, a worker that joins again on its own). A model is Model: vars (F32 masters), rate per parameter (None: frozen, off the wire), step over a batch of row indices the caller planned; the coordinator's owner is Keep: round line, validation, save. replay reruns a run's log in one process and lands on its θ bit for bit at any outer step. adam is AdamW with per-parameter rates and open moments over hanzo-nn's update; gpu is the free-memory floor. The DSpark trainer is one user: `hanzo-train fit` coordinates and trains, `join` adds a machine; its init is seeded, so a frozen Markov head is the same in every process. Tests (CPU): a tiny classifier against its rounds replayed for 1 and 3 workers, micro-batches, requeue, a silent worker, one lost mid-delta, a cut link, frozen parameters, kill and resume; DSpark on a synthetic cache learning on two workers as its rounds replay, its checkpoint in the engine's layout, another cache refused. Claude-Session: https://claude.ai/code/session_018Gt5TvrE2rxXuZikWSauv3