mirror of
https://github.com/hanzoai/engine.git
synced 2026-11-21 02:36:58 +00:00
New hanzo-router-retrain crate + pipeline that refits the Enso routing heads from the ai routing ledger nightly, gates the candidate on an 80/20 holdout, and publishes only a non-regressing head per scope. - hanzo-router-retrain/: enso-fit binary on the `fit --events --out` contract (drop-in for enso's own fit via $FIT_BIN) + tested core: transform_reward (ai reward tuple -> enso EvalSample), fit_policy (proof.rs (x,p,quality) join -> fit_base), holdout_reward, decide_gate (fail-closed), save_w/load_w (enso-base safetensors: f64 tensor "w", [D*K]). 9 unit tests green. - scripts/router-retrain.sh: one scope (global -> heads-base.safetensors owner "*"; org=<slug> -> heads-<slug>.safetensors owner "<slug>"); pull -> fit -> gate -> promote-or-hold -> log.jsonl -> POST /v1/router/publish-artifact-meta. router-retrain-all.sh sequences global + per-opted-in-org. - deploy/router-retrain/: systemd user timer (04:20 America/Los_Angeles, Persistent) = the live spark scheduler; hanzoai/tasks cron ConfigMap as the in-cluster convergence reference; install.sh; LLM.md with the blocker list. Gate is holdout (RouterBench absent in-repo). DO_RELOAD=0 by default: publish writes the versioned artifact + current pointer but does not restart the engine until it mounts heads (unwired). Rebased onto origin/main; the crate builds against the committed enso primitives (fit_base/ingest/parse_jsonl) and writes the load_learner-compatible safetensors itself. Co-authored-by: Hanzo Dev <[email protected]>
31 lines
1.3 KiB
YAML
31 lines
1.3 KiB
YAML
# Canonical hanzoai/tasks durable-cron entry for the router-heads retrain.
|
|
#
|
|
# STATUS: convergence reference, NOT auto-applied. The retrain job is spark-native
|
|
# (it builds + runs the enso Rust `fit` on the GB10 box), and the in-cluster tasks
|
|
# engine's RunJobActivity places k8s Jobs IN-cluster -- it cannot run work on spark
|
|
# without a spark<->tasks bridge (a spark tasks-worker on a `router-retrain` queue,
|
|
# or a spark HTTP receiver the poke below can reach). Until that bridge exists the
|
|
# LIVE scheduler is the spark systemd user timer (router-retrain.timer). Ship this
|
|
# in a kustomization only after a spark receiver/worker is wired.
|
|
#
|
|
# Contract (clients/cron): a ConfigMap in ns `hanzo` labeled cron.hanzo.ai/enabled
|
|
# with data.schedule (UTC cron) + exactly one of data["job.yaml"]|data["poke.json"].
|
|
# 11:20 UTC ~= 04:20 America/Los_Angeles during PDT (a fixed UTC cron does not track
|
|
# DST; the systemd timer does, which is the other reason it stays the live path).
|
|
apiVersion: v1
|
|
kind: ConfigMap
|
|
metadata:
|
|
name: router-retrain
|
|
namespace: hanzo
|
|
labels:
|
|
cron.hanzo.ai/enabled: 'true'
|
|
data:
|
|
schedule: "20 11 * * *"
|
|
poke.json: |
|
|
{
|
|
"url": "http://router-retrain-bridge.hanzo.svc:8080/v1/router/retrain",
|
|
"method": "POST",
|
|
"bearerEnv": "ROUTER_RETRAIN_TOKEN",
|
|
"timeout": "30s"
|
|
}
|