Files
zeekay f23d8ad6b4 hanzo-train trains one model across heterogeneous machines behind a small Model trait, and DSpark trains on it.
cluster is Kai's DiLoCo runtime from hanzoai/decision, model-agnostic: a
coordinator (plan, θ_global in F32, outer Nesterov step, checkpoints of
outer and every worker's AdamW state, resume bit for bit, requeue on drop),
workers that join any time on any device, bf16 deltas with error feedback
summed in join order, and the hang fix of decision 2f436df (keepalive and
bounded links, a round closed at its deadline plus a grace, same-name
rejoin, a worker that joins again on its own). A model is Model: vars (F32
masters), rate per parameter (None: frozen, off the wire), step over a
batch of row indices the caller planned; the coordinator's owner is Keep:
round line, validation, save. replay reruns a run's log in one process
and lands on its θ bit for bit at any outer step. adam is AdamW with
per-parameter rates and open moments over hanzo-nn's update; gpu is the
free-memory floor.

The DSpark trainer is one user: `hanzo-train fit` coordinates and trains,
`join` adds a machine; its init is seeded, so a frozen Markov head is the
same in every process. Tests (CPU): a tiny classifier against its rounds
replayed for 1 and 3 workers, micro-batches, requeue, a silent worker, one
lost mid-delta, a cut link, frozen parameters, kill and resume; DSpark on a
synthetic cache learning on two workers as its rounds replay, its
checkpoint in the engine's layout, another cache refused.

Claude-Session: https://claude.ai/code/session_018Gt5TvrE2rxXuZikWSauv3
2026-09-27 02:07:18 -07:00
..