Files
engine/deploy/router-retrain/tasks-cron.configmap.yaml
zeekayandHanzo Dev 4ffbc0b067 feat(router-retrain): spark-local nightly Enso router-heads retrain
New hanzo-router-retrain crate + pipeline that refits the Enso routing heads
from the ai routing ledger nightly, gates the candidate on an 80/20 holdout,
and publishes only a non-regressing head per scope.

- hanzo-router-retrain/: enso-fit binary on the `fit --events --out` contract
  (drop-in for enso's own fit via $FIT_BIN) + tested core: transform_reward
  (ai reward tuple -> enso EvalSample), fit_policy (proof.rs (x,p,quality)
  join -> fit_base), holdout_reward, decide_gate (fail-closed), save_w/load_w
  (enso-base safetensors: f64 tensor "w", [D*K]). 9 unit tests green.
- scripts/router-retrain.sh: one scope (global -> heads-base.safetensors owner
  "*"; org=<slug> -> heads-<slug>.safetensors owner "<slug>"); pull -> fit ->
  gate -> promote-or-hold -> log.jsonl -> POST /v1/router/publish-artifact-meta.
  router-retrain-all.sh sequences global + per-opted-in-org.
- deploy/router-retrain/: systemd user timer (04:20 America/Los_Angeles,
  Persistent) = the live spark scheduler; hanzoai/tasks cron ConfigMap as the
  in-cluster convergence reference; install.sh; LLM.md with the blocker list.

Gate is holdout (RouterBench absent in-repo). DO_RELOAD=0 by default: publish
writes the versioned artifact + current pointer but does not restart the engine
until it mounts heads (unwired). Rebased onto origin/main; the crate builds
against the committed enso primitives (fit_base/ingest/parse_jsonl) and writes
the load_learner-compatible safetensors itself.

Co-authored-by: Hanzo Dev <[email protected]>
2026-07-16 00:46:52 -07:00

31 lines
1.3 KiB
YAML

# Canonical hanzoai/tasks durable-cron entry for the router-heads retrain.
#
# STATUS: convergence reference, NOT auto-applied. The retrain job is spark-native
# (it builds + runs the enso Rust `fit` on the GB10 box), and the in-cluster tasks
# engine's RunJobActivity places k8s Jobs IN-cluster -- it cannot run work on spark
# without a spark<->tasks bridge (a spark tasks-worker on a `router-retrain` queue,
# or a spark HTTP receiver the poke below can reach). Until that bridge exists the
# LIVE scheduler is the spark systemd user timer (router-retrain.timer). Ship this
# in a kustomization only after a spark receiver/worker is wired.
#
# Contract (clients/cron): a ConfigMap in ns `hanzo` labeled cron.hanzo.ai/enabled
# with data.schedule (UTC cron) + exactly one of data["job.yaml"]|data["poke.json"].
# 11:20 UTC ~= 04:20 America/Los_Angeles during PDT (a fixed UTC cron does not track
# DST; the systemd timer does, which is the other reason it stays the live path).
apiVersion: v1
kind: ConfigMap
metadata:
name: router-retrain
namespace: hanzo
labels:
cron.hanzo.ai/enabled: 'true'
data:
schedule: "20 11 * * *"
poke.json: |
{
"url": "http://router-retrain-bridge.hanzo.svc:8080/v1/router/retrain",
"method": "POST",
"bearerEnv": "ROUTER_RETRAIN_TOKEN",
"timeout": "30s"
}