New hanzo-router-retrain crate + pipeline that refits the Enso routing heads from the ai routing ledger nightly, gates the candidate on an 80/20 holdout, and publishes only a non-regressing head per scope. - hanzo-router-retrain/: enso-fit binary on the `fit --events --out` contract (drop-in for enso's own fit via $FIT_BIN) + tested core: transform_reward (ai reward tuple -> enso EvalSample), fit_policy (proof.rs (x,p,quality) join -> fit_base), holdout_reward, decide_gate (fail-closed), save_w/load_w (enso-base safetensors: f64 tensor "w", [D*K]). 9 unit tests green. - scripts/router-retrain.sh: one scope (global -> heads-base.safetensors owner "*"; org=<slug> -> heads-<slug>.safetensors owner "<slug>"); pull -> fit -> gate -> promote-or-hold -> log.jsonl -> POST /v1/router/publish-artifact-meta. router-retrain-all.sh sequences global + per-opted-in-org. - deploy/router-retrain/: systemd user timer (04:20 America/Los_Angeles, Persistent) = the live spark scheduler; hanzoai/tasks cron ConfigMap as the in-cluster convergence reference; install.sh; LLM.md with the blocker list. Gate is holdout (RouterBench absent in-repo). DO_RELOAD=0 by default: publish writes the versioned artifact + current pointer but does not restart the engine until it mounts heads (unwired). Rebased onto origin/main; the crate builds against the committed enso primitives (fit_base/ingest/parse_jsonl) and writes the load_learner-compatible safetensors itself. Co-authored-by: Hanzo Dev <[email protected]>
4.7 KiB
Router-heads nightly retrain (Enso router)
Spark-local job that refits the Enso routing heads from the ai routing ledger each
night, gates the candidate on a holdout, and publishes only a non-regressing head.
Enso is the learned policy (the enso crate, W = D*K bilinear utility); this job
produces the heads-{scope}.safetensors it loads.
Pieces (one job, orthogonal parts)
hanzo-router-retrain/crate -- the tested core.src/lib.rs:transform_reward(ai reward tuple -> ensoEvalSample),fit_policy(proof.rs's exact (x,p,quality) join ->fit_base),holdout_reward(score aPolicyon held-out rows),decide_gate(fail-closed publish decision),split_indices(seeded),save_w/load_w(enso-base safetensors: one f64 tensorw, shape[D*K]=192, metadataformat=enso-base). 9 unit tests.src/main.rs:enso-fitbinary. Contract, a drop-in for enso's ownfit:enso-fit --events <in.jsonl> --out <heads.safetensors>(+--from-rewards,--holdout,--incumbent,--min-rows). Emits a gate JSON report to stdout.
scripts/router-retrain.sh-- one scope: pull last-24h rewards ->enso-fit-> holdout gate -> promote-or-hold -> local log record -> POST publish-artifact-meta.scripts/router-retrain-all.sh-- the nightly composition:--scope globalthen a--scope org=<slug>per opted-in org.deploy/router-retrain/-- the scheduler (systemd user timer) + the canonicalhanzoai/taskscron ConfigMap (convergence reference).
Scope model (one mechanism, N scopes)
--scope global->heads-base.safetensors, publishowner:"*"(shared base). Rows come from opted-in orgs only (CONTRIB_ORGS=object.ListTrainingContributorOrgs); the reward export carries noorg, so global fetches per opted-in org and concats.--scope org=<slug>->heads-<slug>.safetensors, publishowner:"<slug>", usingGET /v1/export-routing-rewards?org=<slug>. The gate holds any org underMIN_ROWS.
Filenames align with the engine adapter-cache fallback chain (org -> base -> rules).
Gate
Holdout (RouterBench replication is absent in-repo). Split 80/20, fit the candidate on
80%, score candidate + incumbent on the held-out 20% with holdout_reward
(mean clamp(1-|predicted-quality - reward|, 0, 1)), publish iff
candidate >= incumbent AND holdout_rows >= MIN_ROWS. gateKind:"holdout",
gateMetric:"holdout_reward". No incumbent -> bootstrap publish. Thin ledger -> hold.
Scheduler decision (systemd timer, with evidence)
The job MUST run on spark: it builds + runs the enso Rust fit natively on the GB10
box. The in-cluster hanzoai/tasks durable cron is LIVE but its RunJobActivity
creates k8s Jobs IN-cluster; it cannot place work on spark without an unbuilt
spark<->tasks bridge. By the "exists AND fits a nightly spark job" test the systemd
user timer wins: present on the box, Persistent=true catches missed runs on wake,
OnCalendar=... America/Los_Angeles tracks DST. History = the local log.jsonl plus
the ai router_artifact_meta panel. tasks-cron.configmap.yaml is the documented
in-cluster convergence for when a spark receiver/worker exists.
Install: deploy/router-retrain/install.sh (builds fit, enables linger, installs the
--user timer). Env/secrets: ~/.config/router-retrain/env (ROUTER_ADMIN_TOKEN
from KMS hanzo/prod/router-admin-token, CONTRIB_ORGS, DO_RELOAD).
Blockers (wiring this job depends on, none owned here)
fitbinary: enso ships onlyfit_base/ingest/parse_jsonlas lib fns; thefitCLI +persist.rsare uncommitted WIP in the learning-loop worktree. This job ships a referenceenso-fiton the SAME--events/--outcontract, reusing committed primitives; pointFIT_BINat enso'sfitwhen it merges. Reconcile the fit INPUT schema: enso re-featurizes from the request bucket, so the ledger's stored backbonefeaturesvector is carried but not consumed yet.- Export not on prod:
GET /v1/export-routing-rewards404s on api.hanzo.ai (deployed cloud image predates the routes; they exist on ai main).publish-artifact-metais on ai branchfeat/router-stats-observability(PR #99), not yet deployed. - Super-admin auth: exports need IAM
admin-org identity; no in-cluster/service principal mints one.ROUTER_ADMIN_TOKENmust be provisioned in KMS. Documented fallback: read the cluster store directly (kubectldo-sfo3-hanzo-k8s). - Engine heads-mount: the engine App CR mounts only
hf-cache emptyDir; nothing ingestsheads-*.safetensorsand CI RBAC does not whitelistenginefor restart.DO_RELOAD=0until an initContainer/PVC/S3 pull + reload path lands. Publishing writes the versioned artifact +currentpointer; it does not restart prod.