Files
skills/public/hip-0109.html
zeekayandHanzo Dev 9e10c1c10e Each llms index points at the other two
hanzoskills.com/llms.txt now opens with the install line, the API
reference index and the company index, so an agent that lands on any
one of the three can find the rest.

robots.txt claimed 537 documents while llms.txt counts 920. The number
is gone from robots.txt; llms.txt derives it.

public/ is regenerated from scripts/site.py, which it had fallen
behind: the copy-prompt menu and agent bar were in the generator and
not in the committed pages. tests/site.mjs passes against the result.

Co-authored-by: Hanzo Dev <[email protected]>
2026-09-11 00:27:59 -07:00

555 lines
40 KiB
HTML

<!doctype html>
<html lang="en"><head>
<meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1">
<title>hip-0109</title>
<meta name="color-scheme" content="dark">
<style>
/* Committed to one visual world: a dark instrument panel. No light variant —
the dot field and the hairline rules only read on black, and a naive
inversion would wash both out. Every colour is painted explicitly so the page
never borrows a host background. */
@font-face{font-family:'Zen';src:url('/fonts/Zen-Variable.woff2') format('woff2');font-weight:100 900;font-style:normal;font-display:swap}
@font-face{font-family:'Zen';src:url('/fonts/Zen-Italic-Variable.woff2') format('woff2');font-weight:100 900;font-style:italic;font-display:swap}
@font-face{font-family:'Zen Mono';src:url('/fonts/ZenMono-Variable.woff2') format('woff2');font-weight:100 900;font-style:normal;font-display:swap}
@font-face{font-family:'Zen Mono';src:url('/fonts/ZenMono-Italic-Variable.woff2') format('woff2');font-weight:100 900;font-style:italic;font-display:swap}
@font-face{font-family:'Zen Pixel Square';src:url('/fonts/ZenPixel-Square.woff2') format('woff2');font-style:normal;font-display:swap}
:root{
--ink:#f4f4f2; --soft:#9c9c98; --faint:#6a6a66; --ground:#000; --panel:#0a0a0a;
--line:#232323; --line-hi:#3a3a3a; --accent:#f4f4f2;
--font-zen-sans:'Zen',-apple-system,BlinkMacSystemFont,"Segoe UI",Helvetica,Arial,sans-serif;
--font-zen-mono:'Zen Mono',ui-monospace,"SF Mono",SFMono-Regular,Menlo,Consolas,monospace;
--font-zen-pixel-square:'Zen Pixel Square',monospace;
--mono:var(--font-zen-mono);
--sans:var(--font-zen-sans);
/* Controls are pills and surfaces are soft, as on hanzo.ai. */
--r:999px; --r-soft:14px;
}
*{box-sizing:border-box}
html{-webkit-text-size-adjust:100%}
body{margin:0;background:var(--ground);color:var(--ink);font-family:var(--sans);
font-size:16px;line-height:1.65;-webkit-font-smoothing:antialiased}
.wrap{max-width:60rem;margin:0 auto;padding:0 1.75rem}
/* The dot field. A halftone made of one repeating radial-gradient, faded out
along the diagonal by a mask so it reads as a wave rather than wallpaper. */
/* No overflow clip here: the dot field is inset to the box already, and a clip
would cut off the copy-prompt menu. */
.hero{position:relative;border-bottom:1px solid var(--line)}
.hero::before{content:"";position:absolute;inset:0;
background-image:radial-gradient(var(--line-hi) 1px, transparent 1px);
background-size:7px 7px;
-webkit-mask-image:linear-gradient(115deg,#000 0%,rgba(0,0,0,.55) 38%,transparent 72%);
mask-image:linear-gradient(115deg,#000 0%,rgba(0,0,0,.55) 38%,transparent 72%);
pointer-events:none}
.hero .wrap{position:relative;padding-top:5.5rem;padding-bottom:4rem}
nav{border-bottom:1px solid var(--line)}
nav .in{max-width:60rem;margin:0 auto;padding:1.1rem 1.75rem;display:flex;
justify-content:space-between;align-items:center;gap:1rem}
.brand{font-family:var(--mono);font-size:.85rem;letter-spacing:.14em;
text-transform:uppercase;color:var(--ink);text-decoration:none;
display:inline-flex;align-items:center;gap:.6rem}
.brand i{font-style:normal;color:var(--faint)}
.mark{width:1.15em;height:1.15em;display:block;flex:0 0 auto}
nav a.up{font-family:var(--mono);font-size:.72rem;letter-spacing:.14em;
text-transform:uppercase;color:var(--soft);text-decoration:none}
nav a.up:hover{color:var(--ink)}
h1{font-family:var(--mono);font-size:clamp(2rem,5.4vw,3.4rem);line-height:1.12;
letter-spacing:-.02em;margin:0 0 1.1rem;font-weight:500;text-wrap:balance}
h1 em{font-style:normal;color:var(--faint)}
.lede{color:var(--soft);font-size:1.05rem;max-width:52ch;margin:0 0 2.25rem}
.btns{display:flex;flex-wrap:wrap;gap:.7rem;margin:0 0 .5rem;align-items:center}
.btn{display:inline-flex;align-items:center;gap:.5rem;font-family:var(--mono);
font-size:.72rem;letter-spacing:.14em;text-transform:uppercase;
padding:.8rem 1.4rem;border:1px solid var(--line-hi);border-radius:var(--r);
color:var(--ink);text-decoration:none;background:transparent;cursor:pointer;
line-height:1.2;font-weight:400}
.btn:hover{background:#141414;text-decoration:none;border-color:var(--soft)}
.btn.solid{background:var(--ink);color:#000;border-color:var(--ink)}
.btn.solid:hover{background:#d8d8d4}
.btn .chev{width:.7em;height:.7em;flex:0 0 auto;transition:transform .15s ease}
.btn[aria-expanded="true"] .chev{transform:rotate(180deg)}
/* Copy-prompt selector. Each row copies the prompt for its own target. */
.copy{position:relative;display:inline-flex}
.menu{position:absolute;top:calc(100% + .5rem);left:0;z-index:20;min-width:21rem;
background:var(--panel);border:1px solid var(--line-hi);border-radius:var(--r-soft);
padding:.4rem;display:flex;flex-direction:column;
box-shadow:0 12px 32px rgba(0,0,0,.75)}
.menu[hidden]{display:none}
.mi{display:flex;align-items:center;gap:.7rem;width:100%;text-align:left;
padding:.62rem .8rem;border:0;border-radius:calc(var(--r-soft) - .35rem);
background:transparent;color:var(--ink);font-family:var(--mono);font-size:.76rem;
letter-spacing:.04em;text-decoration:none;cursor:pointer;line-height:1.3;
white-space:nowrap}
.mi:hover,.mi:focus-visible{background:#181818;outline:none}
.mi .mark{width:1.05rem;height:1.05rem;color:var(--soft)}
.mi:hover .mark{color:var(--ink)}
.mi u{margin-left:auto;color:var(--faint);text-decoration:none;font-size:.68rem;
letter-spacing:.1em;text-transform:uppercase}
/* Works-with bar: one scrolling row, name and mark per agent. */
.strip{border-bottom:1px solid var(--line);background:var(--panel)}
.strip .in{max-width:60rem;margin:0 auto;padding:.9rem 1.75rem;display:flex;
align-items:center;gap:1.25rem;font-family:var(--mono);font-size:.72rem;
letter-spacing:.1em;text-transform:uppercase;color:var(--faint)}
.strip b{color:var(--soft);font-weight:400;flex:0 0 auto;
border-right:1px solid var(--line);padding-right:1.25rem;align-self:stretch;
display:flex;align-items:center}
.bar{display:flex;align-items:center;gap:.5rem;flex:1 1 auto;min-width:0;
overflow-x:auto;scrollbar-width:none;scroll-snap-type:x proximity;
-webkit-overflow-scrolling:touch;
-webkit-mask-image:linear-gradient(90deg,#000 calc(100% - 3rem),transparent);
mask-image:linear-gradient(90deg,#000 calc(100% - 3rem),transparent)}
.bar::-webkit-scrollbar{display:none}
.agent{display:inline-flex;align-items:center;gap:.45rem;flex:0 0 auto;
padding:.4rem .85rem;border:1px solid var(--line);border-radius:var(--r);
color:var(--soft);white-space:nowrap;scroll-snap-align:start;
text-decoration:none;background:var(--ground)}
.agent:hover{color:var(--ink);border-color:var(--line-hi);text-decoration:none}
.agent .mark{width:.95rem;height:.95rem}
h2{font-family:var(--mono);font-size:1.1rem;letter-spacing:-.01em;margin:2.75rem 0 .75rem;font-weight:500}
h3{font-size:1rem;margin:2rem 0 .5rem;font-weight:600}
h4,h5,h6{font-size:.94rem;margin:1.5rem 0 .5rem;font-weight:600}
p{margin:0 0 1rem;max-width:70ch}
ul,ol{margin:0 0 1rem;padding-left:1.35rem;max-width:70ch}
li{margin:.22rem 0}
a{color:var(--ink);text-decoration:underline;text-underline-offset:3px;
text-decoration-color:var(--line-hi)}
a:hover{text-decoration-color:var(--ink)}
a:focus-visible{outline:1px solid var(--ink);outline-offset:3px}
code{font-family:var(--mono);font-size:.85em;background:var(--panel);
border:1px solid var(--line);border-radius:5px;padding:.1em .36em}
pre{background:var(--panel);border:1px solid var(--line);padding:1rem 1.1rem;
border-radius:var(--r-soft);overflow-x:auto;margin:0 0 1.25rem}
pre code{background:none;border:0;padding:0;font-size:.82rem;line-height:1.7}
blockquote{margin:0 0 1rem;padding-left:1rem;border-left:1px solid var(--line-hi);color:var(--soft)}
hr{border:0;border-top:1px solid var(--line);margin:2.5rem 0}
table{border-collapse:collapse;width:100%;margin:0 0 1.25rem;font-size:.92rem;
display:block;overflow-x:auto}
th,td{border:1px solid var(--line);padding:.5rem .7rem;text-align:left}
.curl{display:block;background:var(--panel);border:1px solid var(--line);
border-radius:var(--r-soft);padding:.95rem 1.1rem;font-family:var(--mono);
font-size:.8rem;color:var(--soft);overflow-x:auto;margin:0 0 2.5rem;
white-space:pre;line-height:1.9}
.curl b{color:var(--ink);font-weight:400}
.curl i{font-style:normal;color:var(--faint)}
/* index */
#q{width:100%;padding:.85rem 1.2rem;font-family:var(--mono);font-size:.85rem;color:var(--ink);
background:var(--panel);border:1px solid var(--line);border-radius:var(--r);margin:0 0 .6rem}
#q::placeholder{color:var(--faint);text-transform:uppercase;letter-spacing:.1em;font-size:.72rem}
#q:focus{outline:none;border-color:var(--soft)}
#count{color:var(--faint);font-family:var(--mono);font-size:.72rem;letter-spacing:.1em;
text-transform:uppercase;margin:0 0 2rem}
.cat{margin:2.5rem 0 .5rem;font-family:var(--mono);font-size:.7rem;letter-spacing:.16em;
text-transform:uppercase;color:var(--faint);border-bottom:1px solid var(--line);
padding-bottom:.45rem}
.row{display:block;padding:.6rem .9rem;margin:0 -.9rem;border-bottom:1px solid var(--line);
border-radius:var(--r-soft);text-decoration:none}
.row:hover{background:#0a0a0a}
.nm{font-family:var(--mono);font-size:.84rem;color:var(--ink);display:block}
.ds{color:var(--soft);font-size:.9rem;display:block;margin-top:.12rem}
footer{margin-top:4rem;padding:2rem 0 4rem;border-top:1px solid var(--line);
color:var(--faint);font-size:.85rem}
footer a{color:var(--soft)}
.fmap{display:grid;gap:1.75rem;grid-template-columns:repeat(auto-fit,minmax(11rem,1fr));
margin:0 0 2rem;font-size:.85rem}
.fmap b{display:block;font-family:var(--mono);font-size:.68rem;letter-spacing:.16em;
text-transform:uppercase;color:var(--faint);font-weight:400;margin:0 0 .6rem;
border-bottom:1px solid var(--line);padding-bottom:.4rem}
.fmap a{display:inline-block;color:var(--ink);text-decoration:none;margin:.3rem .35rem 0 0}
.fmap a:hover{text-decoration:underline}
.fmap div{color:var(--faint);line-height:1.5}
.fend{border-top:1px solid var(--line);padding-top:1.25rem;font-size:.82rem}
@media (max-width:640px){
.hero .wrap{padding-top:3.25rem;padding-bottom:2.75rem}
.btns{gap:.5rem}
.btn{padding:.75rem 1.15rem;font-size:.68rem}
.copy,.copy .btn{width:100%;justify-content:center}
.menu{left:0;right:0;min-width:0}
.strip .in{flex-direction:column;align-items:flex-start;gap:.6rem;padding:.9rem 1.75rem}
.strip b{border-right:0;padding-right:0}
.bar{width:100%}
}
</style>
<link rel="alternate" type="text/markdown" href="/hip-0109.md">
<link rel="icon" href="/favicon.svg">
<link rel="canonical" href="https://hanzoskills.com/hip-0109">
</head>
<!-- Reading this with a machine? Every page here is also plain markdown.
This one: https://hanzoskills.com/hip-0109.md
Index: https://hanzoskills.com/llms.txt
Corpus: https://hanzoskills.com/all.md
Append .md to any path for the markdown twin. -->
<body>
<nav><div class="in">
<a class="brand" href="/"><svg class="mark" viewBox="0 0 67 67" fill="currentColor" aria-hidden="true"><path d="M22.21 67V44.6369H0V67H22.21Z"/><path d="M66.7038 22.3184H22.2534L0.0878906 44.6367H44.4634L66.7038 22.3184Z"/><path d="M22.21 0H0V22.3184H22.21V0Z"/><path d="M66.7198 0H44.5098V22.3184H66.7198V0Z"/><path d="M66.7198 67V44.6369H44.5098V67H66.7198Z"/></svg> HANZO<i>SKILLS</i></a>
<a class="up" href="/">All skills</a>
</div></nav>
<div class="wrap">
<h1>hip-0109</h1><p class="lede">HIP-109: Hanzo ML Cloud Toolkit. Status Final. Hanzo architectural specification.</p><h1>HIP-0109: Hanzo ML Cloud Toolkit</h1>
<h2>Abstract</h2>
<p>The Hanzo ML Cloud Toolkit is the Hanzo-native ML operator and control plane that lives in <code>~/work/hanzo/operator</code> (Rust, BSD-3). It replaces the functional surface of the prior open-source ML operator estate (training, experiment tracking, hyperparameter sweeps, pipelines, notebooks, model registry, inference serving) <strong>without taking that estate as a dependency</strong>. Seven <code>hanzo.ai/v1</code> Custom Resource Definitions reconciled by one Rust binary, mounted under the unified <code>cloud</code> HTTP surface from HIP-0106, talking to Hanzo primitives: <code>hanzoai/iam</code> (auth), <code>hanzoai/vfs</code> (artifacts), <code>hanzoai/datastore</code> (metrics), <code>hanzoai/base</code> (per-tenant SQLite), <code>hanzoai/tasks</code> (DAG runner), <code>hanzoai/o11y</code> (general observability), <code>hanzoai/insights</code> (LLM-specific observability), <code>hanzoai/engine</code> (inference), <code>hanzoai/ml</code> (Candle-based ML primitives), <code>hanzoai/cloud</code> (control plane mux).</p>
<p><strong>Brand and scope policy.</strong> <code>hanzoai/ml</code> STAYS the primitive Rust ML toolkit (Candle core + tensor ops + model primitives). It does NOT absorb control-plane responsibilities; those land in <code>hanzoai/operator</code>. <code>hanzoai/ml</code> is consumed by training-container images that the operator schedules — it is not the scheduler. One way to do everything: <code>ml/</code> is primitives, <code>operator/</code> is control plane, <code>cloud/</code> is the HTTP mux, and the protocols at the edges are MCP for tools, A2A for federation, and ZAP between Hanzo subsystems.</p>
<h2>Protocol boundaries (decomplect)</h2>
<p>Protocols sit at distinct layers. Each does one thing.</p>
<p>| Protocol | Layer | Role | |---|---|---| | <strong>ZAP</strong> | inter-subsystem (Hanzo internal) | The spine. <code>cloud</code> ↔ <code>operator</code> ↔ <code>iam</code> ↔ <code>tasks</code> etc. — all over ZAP. Same generated bindings serve in-process and over-wire per HIP-0106. | | <strong>MCP</strong> | external tool/resource exposure | The ML control plane exposes its actions (create_training_job, query_experiment, list_models, deploy_inference) as MCP tools so MCP-aware models and IDEs can drive ML workloads. ML is a <strong>server</strong> of MCP tools, not a client. | | <strong>A2A</strong> | external agent-to-agent federation | Hanzo&#x27;s ML-aware agents accept task delegations from other agents via A2A. The ML operator itself does not speak A2A; the agents subsystem does, and the agents call into the operator via the cloud subsystem&#x27;s <code>/v1/ml/<em></code> HTTP routes. | | <strong>ACP compat adapter</strong> | optional IDE / coding-agent client compat | Where an external IDE or coding-agent client speaks ACP and wants to issue ML commands, the cloud subsystem exposes an ACP-compatible adapter that translates to the canonical <code>/v1/ml/</em></code> HTTP routes. NOT canonical core. | | <strong>HTTP/REST</strong> | public management API | The operator&#x27;s CRDs are mirrored under <code>/v1/ml/jobs</code>, <code>/v1/ml/experiments</code>, <code>/v1/ml/sweeps</code>, <code>/v1/ml/pipelines</code>, <code>/v1/ml/notebooks</code>, <code>/v1/ml/models</code>, <code>/v1/ml/inference</code> by the cloud subsystem. JWT-authenticated via IAM, per-org scoped. | | <strong>K8s CRDs</strong> | operator-facing declarative API | Seven Kinds at <code>hanzo.ai/v1</code>: <code>TrainingJob</code>, <code>Experiment</code>, <code>Sweep</code>, <code>Pipeline</code>, <code>Notebook</code>, <code>Model</code>, <code>Inference</code>. The Rust operator reconciles. |</p>
<p><strong>MCP is not the control plane.</strong> MCP exposes ML actions as tools that external models/IDEs can call. The control plane API is HTTP/REST on top of <code>/v1/ml/*</code>, backed by K8s CRDs. Same for A2A — A2A delegates tasks between agents; agents call into the control plane like any other client.</p>
<h2>Specification</h2>
<h3>Lifecycle CRDs at <code>hanzo.ai/v1</code></h3>
<p>The operator owns these Kinds:</p>
<p>| Kind | Plural | Short | What it owns | |---|---|---|---| | <code>TrainingJob</code> | <code>trainingjobs</code> | <code>mlj</code> | Distributed training pods. One unified type with <code>spec.framework: tensorflow | pytorch | mpi | xgboost | jax | candle | hanzo-ml</code>. Backs all training-job variants from the open-source operator family. | | <code>Experiment</code> | <code>experiment</code> | <code>mlx</code> | Tracking experiment. Status surfaces metric streams from <code>hanzoai/datastore</code>; artifacts via <code>hanzoai/vfs</code>; run config in per-tenant <code>hanzoai/base</code>. | | <code>Sweep</code> | <code>sweeps</code> | <code>mls</code> | Hyperparameter sweep. Owns N child <code>TrainingJob</code>s with parameter assignments; runs Bayesian / grid / random search per <code>spec.algorithm</code>. | | <code>Pipeline</code> | <code>pipelines</code> | <code>mlp</code> | DAG of steps. Each step is a <code>TrainingJob</code>, <code>Inference</code>, or generic container task. Durable execution by delegating to <code>hanzoai/tasks</code> (HIP-0108). | | <code>Notebook</code> | <code>notebooks</code> | <code>mln</code> | Per-tenant JupyterLab pod with IAM SSO sidecar. Storage is a PVC backed by <code>hanzoai/vfs</code>. Idle eviction per HIP-0108 tier-2 supervisor. | | <code>Model</code> | <code>models</code> | <code>mlm</code> | Model registry entry. Metadata (versions, lineage, signatures, framework, tags) in per-tenant <code>hanzoai/base</code>; blobs in <code>hanzoai/vfs</code>. | | <code>Inference</code> | <code>inferences</code> | <code>mli</code> | Inference service. Routes to <code>hanzoai/engine</code> for Hanzo-native serving, or to an external runtime container when <code>spec.runtime: external</code>. |</p>
<p>All seven are namespaced. All seven follow the operator&#x27;s existing phase lifecycle: <code>Pending → Creating → Running → Degraded → Deleting</code>. All seven mint standard <code>status.conditions</code> (<code>Ready</code>, <code>Synced</code>, <code>Available</code>).</p>
<h3>Replacement table</h3>
<p>The operator estate this displaces, and the Hanzo CRD that subsumes each:</p>
<p>| Subsumed responsibility | Hanzo CRD | Notes | |---|---|---| | TFJob, PyTorchJob, MPIJob, XGBoostJob, JAXJob | <code>TrainingJob</code> | One Kind. <code>spec.framework</code> selects pod template + worker/parameter-server topology. | | MLflow Tracking + tracking server | <code>Experiment</code> | Metric writes go to <code>hanzoai/datastore</code>; artifact writes go to <code>hanzoai/vfs</code>; UI lives in Hanzo Console. | | Katib Experiment | <code>Sweep</code> | Bayesian/grid/random default. BYO algorithm via in-process extension (HIP-0105). | | KFP Pipeline + Argo backend | <code>Pipeline</code> | DAG durable execution on <code>hanzoai/tasks</code>. No Argo. | | Notebook Operator | <code>Notebook</code> | One JupyterLab image with IAM SSO sidecar. Per-tenant. | | MLflow Model Registry | <code>Model</code> | Metadata in <code>hanzoai/base</code>; blobs in <code>hanzoai/vfs</code>. | | KServe / Seldon Core | <code>Inference</code> | Routes to <code>hanzoai/engine</code> (Rust, Candle) or external runtime. | | Profile Controller (open-source) | <code>hanzoai/iam</code> | Per-org isolation already handled. No new controller. | | Dex | <code>hanzoai/iam</code> | Single IDP across all subsystems. | | Central Dashboard | Hanzo Console <code>/ml/*</code> views | Out of scope for this HIP. |</p>
<p>The replacement is <strong>functional, not source-level</strong>. Zero lines of open-source operator code are imported, vendored, or forked. The Hanzo operator is a clean-room Rust implementation.</p>
<h3>Architecture</h3>
<pre><code>External clients
├── MCP-aware models/IDEs → ml.create_training_job, query_experiment, ...
├── A2A federation → ml agents accept task delegation
└── ACP compat (optional) → IDE/coding-agent clients
K8s API
└── ml CRDs (TrainingJob, Experiment, Sweep, Pipeline, Notebook, Model, Inference)
└── hanzoai/operator (Rust) reconciles
├── pods/services/PVCs
└── GPU/CPU/memory scheduling
cloud subsystem mounts (HTTP routes):
/v1/ml/jobs, /v1/ml/experiments, /v1/ml/sweeps, /v1/ml/pipelines,
/v1/ml/notebooks, /v1/ml/models, /v1/ml/inference
Hanzo primitives consumed:
hanzoai/iam — auth via JWT, per-org isolation
hanzoai/vfs — artifact storage (S3-compat)
hanzoai/datastore — experiment metrics (OLAP)
hanzoai/base — per-tenant SQLite (model registry metadata, run config)
hanzoai/tasks — pipeline DAG durable runner (HIP-0108)
hanzoai/o11y — infrastructure observability
hanzoai/insights — AI-specific observability (LLM traces, eval)
hanzoai/engine — Rust inference serving
hanzoai/ml — Candle-based ML primitives (used INSIDE training containers)</code></pre>
<h3><code>TrainingJob</code> shape</h3>
<pre><code class="lang-yaml">apiVersion: hanzo.ai/v1
kind: TrainingJob
metadata:
name: imagenet-finetune
namespace: tenant-acme
spec:
framework: pytorch # tensorflow|pytorch|mpi|xgboost|jax|candle|hanzo-ml
image: ghcr.io/acme/trainer:v1.2.0
command: [&quot;python&quot;, &quot;train.py&quot;]
workers:
replicas: 4
resources:
requests: { nvidia.com/gpu: &quot;1&quot;, cpu: &quot;8&quot;, memory: &quot;32Gi&quot; }
parameterServer: # optional, framework-specific
replicas: 1
master: # optional, MPI/PyTorch DDP master
replicas: 1
artifacts: # writes go to hanzoai/vfs
output: vfs://tenant-acme/runs/imagenet-finetune/v1
experiment: # optional binding to an Experiment CR
name: acme-finetune-2026q2
env:
- name: WANDB_DISABLED
value: &quot;true&quot;
ttlSecondsAfterFinished: 86400
status:
phase: Running
conditions: [...]
workersReady: 4
workersTotal: 4
startTime: &quot;2026-05-18T...Z&quot;
artifactURI: vfs://tenant-acme/runs/imagenet-finetune/v1</code></pre>
<p>The framework field selects which pod-topology builder runs. The operator ships exactly one image-agnostic worker/master/parameter- server template per framework. Customer code is in <code>spec.image</code> + <code>spec.command</code>; the operator never injects framework sidecars beyond what the canonical template requires.</p>
<h3><code>Experiment</code> shape</h3>
<pre><code class="lang-yaml">apiVersion: hanzo.ai/v1
kind: Experiment
metadata:
name: acme-finetune-2026q2
namespace: tenant-acme
spec:
description: &quot;Q2 fine-tune sweep across LR / batch size&quot;
tags: { team: vision, project: imagenet }
metricsBackend: datastore # hanzoai/datastore (default)
artifactsBackend: vfs # hanzoai/vfs (default)
metadataBackend: base # hanzoai/base per-tenant SQLite (default)
status:
phase: Running
runs: 17
bestRun:
name: imagenet-finetune-r12
metric: { name: val_acc, value: 0.823 }</code></pre>
<p>Experiments are durable handles. <code>TrainingJob.spec.experiment.name</code> binds a job to an experiment; the operator wires the binding into the container env so the customer&#x27;s training script can write metrics to the right experiment without knowing the metric backend URL.</p>
<h3><code>Sweep</code> shape</h3>
<pre><code class="lang-yaml">apiVersion: hanzo.ai/v1
kind: Sweep
metadata:
name: lr-sweep-q2
namespace: tenant-acme
spec:
experiment: acme-finetune-2026q2
algorithm: bayesian # bayesian|grid|random
maxTrials: 50
parallelism: 4
objective:
metric: val_acc
goal: maximize
parameters:
- name: learning_rate
type: log_uniform
min: 1e-5
max: 1e-2
- name: batch_size
type: choice
values: [32, 64, 128]
template: # TrainingJob template
framework: pytorch
image: ghcr.io/acme/trainer:v1.2.0
workers: { replicas: 2 }
status:
phase: Running
trialsRunning: 4
trialsCompleted: 12
bestTrial: { name: lr-sweep-q2-t7, value: 0.811 }</code></pre>
<p>Sweep spawns child <code>TrainingJob</code>s with parameter assignments injected via env. Default algorithm is Bayesian (Tree-Parzen Estimator); grid and random are also built in; BYO algorithm via HIP-0105 extension module.</p>
<h3><code>Pipeline</code> shape</h3>
<pre><code class="lang-yaml">apiVersion: hanzo.ai/v1
kind: Pipeline
metadata:
name: train-eval-deploy
namespace: tenant-acme
spec:
steps:
- name: prep
type: container
image: ghcr.io/acme/prep:v1
- name: train
type: trainingjob
dependsOn: [prep]
template:
framework: pytorch
image: ghcr.io/acme/trainer:v1
workers: { replicas: 4 }
- name: eval
type: container
dependsOn: [train]
image: ghcr.io/acme/eval:v1
- name: deploy
type: inference
dependsOn: [eval]
template:
runtime: engine
model: { name: acme-resnet50, version: latest }
taskQueue: hanzo-tasks # delegates to hanzoai/tasks (HIP-0108)
status:
phase: Running
currentStep: train
completedSteps: [prep]</code></pre>
<p>Pipelines do <strong>not</strong> own their own scheduler. They submit each step as a durable task on <code>hanzoai/tasks</code> and reconcile the resulting status into <code>Pipeline.status</code>. One DAG runner, one place — no parallel scheduler stack.</p>
<h3><code>Notebook</code> shape</h3>
<pre><code class="lang-yaml">apiVersion: hanzo.ai/v1
kind: Notebook
metadata:
name: alice-dev
namespace: tenant-acme
spec:
image: ghcr.io/hanzoai/notebook:v1.0.0
resources:
requests: { nvidia.com/gpu: &quot;0&quot;, cpu: &quot;2&quot;, memory: &quot;4Gi&quot; }
storage:
size: 20Gi
storageClass: &quot;&quot; # defaults to cluster default
idleTimeoutSeconds: 1800 # HIP-0108 tier-2 idle eviction
user: { email: [email protected] } # IAM SSO sidecar gates by email
status:
phase: Running
url: https://nb-alice-dev.tenant-acme.cloud.hanzo.ai</code></pre>
<p>The notebook base image (<code>ghcr.io/hanzoai/notebook:v1.0.0</code>) is a JupyterLab image with the IAM SSO sidecar pre-baked. Per-user, per- tenant isolation. Idle eviction follows HIP-0108. The actual image build lives at <code>~/work/hanzo/notebook</code> (separate repo, separate release cadence).</p>
<h3><code>Model</code> shape</h3>
<pre><code class="lang-yaml">apiVersion: hanzo.ai/v1
kind: Model
metadata:
name: acme-resnet50
namespace: tenant-acme
spec:
framework: pytorch
artifact: vfs://tenant-acme/models/acme-resnet50/v1.bin
signature: # input/output tensor specs
inputs:
- { name: image, shape: [1, 3, 224, 224], dtype: float32 }
outputs:
- { name: logits, shape: [1, 1000], dtype: float32 }
tags: { team: vision }
versions:
- name: v1
artifact: vfs://tenant-acme/models/acme-resnet50/v1.bin
createdAt: &quot;2026-05-01T...Z&quot;
- name: v2
artifact: vfs://tenant-acme/models/acme-resnet50/v2.bin
createdAt: &quot;2026-05-18T...Z&quot;
metadataBackend: base
status:
phase: Available
currentVersion: v2
totalVersions: 2</code></pre>
<p>Metadata in per-tenant <code>hanzoai/base</code> SQLite (one row per Model CR, versions as JSON; or normalized — implementation detail). Blobs in <code>hanzoai/vfs</code>. There is no separate &quot;model registry server&quot;; the operator&#x27;s <code>Model</code> CR is the registry.</p>
<h3><code>Inference</code> shape</h3>
<pre><code class="lang-yaml">apiVersion: hanzo.ai/v1
kind: Inference
metadata:
name: resnet50-prod
namespace: tenant-acme
spec:
runtime: engine # engine|external
model:
name: acme-resnet50
version: v2
replicas: 3
resources:
requests: { nvidia.com/gpu: &quot;1&quot;, cpu: &quot;2&quot;, memory: &quot;8Gi&quot; }
autoscaling:
enabled: true
minReplicas: 1
maxReplicas: 10
targetGPUUtilization: 70
ingress:
enabled: true
hosts: [resnet50.tenant-acme.cloud.hanzo.ai]
status:
phase: Running
url: https://resnet50.tenant-acme.cloud.hanzo.ai
readyReplicas: 3
desiredReplicas: 3</code></pre>
<p><code>runtime: engine</code> materializes pods that pull <code>ghcr.io/hanzoai/engine</code> and load the model from <code>vfs</code>. <code>runtime: external</code> lets the customer ship their own serving image (Triton-style or any other) — the operator just provides the K8s plumbing (Deployment + Service + Ingress + HPA + PDB + NetworkPolicy + KMSSecret).</p>
<h3>Auth, multi-tenancy, and per-org scoping</h3>
<p>All seven CRDs live in tenant namespaces (<code>tenant-&lt;org-slug&gt;</code>). The operator runs cluster-wide and watches CRs across namespaces, but every action enforces:</p>
<ul><li>The namespace&#x27;s <code>hanzo.ai/org</code> label matches the tenant the request</li></ul>
<p> comes from (when the request is over HTTP/REST or MCP).</p>
<ul><li><code>spec.image</code> is pull-allowed from the tenant&#x27;s image-pull-secret</li></ul>
<p> list (KMSSecret-managed per the existing operator pattern).</p>
<ul><li>Artifacts written to <code>vfs://&lt;tenant-org&gt;/...</code> paths.</li><li>Experiment metrics scoped by <code>org_id</code> in <code>hanzoai/datastore</code>.</li><li>Notebook IAM sidecar gates by the tenant&#x27;s IAM application.</li></ul>
<p>No cross-tenant CR visibility. No cross-tenant artifact reads. No cross-tenant metric reads. The pattern matches every other Hanzo subsystem.</p>
<h3>HTTP surface mounted by the <code>cloud</code> subsystem</h3>
<p>The operator is the source of truth for CRD state. The HTTP layer is a thin facade the <code>cloud</code> subsystem mounts when the <code>ml</code> flag is enabled:</p>
<pre><code>POST /v1/ml/jobs → create TrainingJob
GET /v1/ml/jobs → list TrainingJob
GET /v1/ml/jobs/{name} → read TrainingJob
DELETE /v1/ml/jobs/{name} → delete TrainingJob
POST /v1/ml/jobs/{name}/cancel → cancel TrainingJob
POST /v1/ml/experiments → create Experiment
GET /v1/ml/experiments → list Experiment
GET /v1/ml/experiments/{name} → read Experiment
GET /v1/ml/experiments/{name}/runs → list runs (from datastore)
GET /v1/ml/experiments/{name}/metrics → metric stream (from datastore)
POST /v1/ml/sweeps → create Sweep
GET /v1/ml/sweeps → list Sweep
GET /v1/ml/sweeps/{name} → read Sweep
GET /v1/ml/sweeps/{name}/trials → list child TrainingJobs
POST /v1/ml/pipelines → create Pipeline
GET /v1/ml/pipelines → list Pipeline
GET /v1/ml/pipelines/{name} → read Pipeline
POST /v1/ml/pipelines/{name}/run → trigger an execution
POST /v1/ml/notebooks → create Notebook
GET /v1/ml/notebooks → list Notebook
GET /v1/ml/notebooks/{name} → read Notebook
POST /v1/ml/notebooks/{name}/stop → stop (sets idleTimeoutSeconds=0)
POST /v1/ml/models → create Model (registry entry)
GET /v1/ml/models → list Model
GET /v1/ml/models/{name} → read Model
POST /v1/ml/models/{name}/versions → add a version (metadata)
POST /v1/ml/inference → create Inference
GET /v1/ml/inference → list Inference
GET /v1/ml/inference/{name} → read Inference
POST /v1/ml/inference/{name}:predict → forward predict call to engine</code></pre>
<p>Same JWT auth as every other <code>/v1/*</code> endpoint. Same <code>X-Org-Id</code> header contract. Same in-process ZAP call from the cloud subsystem to the operator-exposed ZAP service. No <code>/api/</code> prefix.</p>
<h3>MCP tools surface</h3>
<p>The operator&#x27;s actions are exposed as MCP tools when an MCP server is mounted alongside (typical for Hanzo Cloud deployments):</p>
<ul><li><code>ml.create_training_job(spec) → { name, namespace, phase }</code></li><li><code>ml.query_training_job(name) → TrainingJobStatus</code></li><li><code>ml.cancel_training_job(name) → bool</code></li><li><code>ml.create_experiment(spec) → Experiment</code></li><li><code>ml.query_experiment(name) → ExperimentStatus</code></li><li><code>ml.list_experiments(filter?) → [Experiment]</code></li><li><code>ml.create_sweep(spec) → Sweep</code></li><li><code>ml.query_sweep(name) → SweepStatus</code></li><li><code>ml.create_pipeline(spec) → Pipeline</code></li><li><code>ml.run_pipeline(name) → PipelineRun</code></li><li><code>ml.create_notebook(spec) → Notebook</code></li><li><code>ml.stop_notebook(name) → bool</code></li><li><code>ml.register_model(spec) → Model</code></li><li><code>ml.list_models(filter?) → [Model]</code></li><li><code>ml.deploy_inference(spec) → Inference</code></li><li><code>ml.predict(name, payload) → ndarray</code></li></ul>
<p>The MCP tool definitions are co-located with the Rust types and generated at build time. <strong>MCP is a surface; it is not the API.</strong> The canonical API is the K8s CRDs and the HTTP routes.</p>
<h2>Non-goals</h2>
<ul><li><strong>Bringing in the open-source ML operator estate as a dependency.</strong></li></ul>
<p> Zero lines vendored. The Hanzo operator is a clean-room Rust implementation that consumes only Hanzo primitives.</p>
<ul><li><strong>Making MCP the ML control plane.</strong> MCP exposes ML actions as</li></ul>
<p> tools. The control plane is the CRD API plus the HTTP mux.</p>
<ul><li><strong>Making A2A the ML job API.</strong> A2A delegates tasks between agents.</li></ul>
<p> Agents call the ML control plane via HTTP/REST like any other client.</p>
<ul><li><strong>Folding Python training runtimes into the Go binary.</strong> Per</li></ul>
<p> HIP-0106, Python services with native ML deps stay out of the unified binary. Training containers run as their own pods, scheduled by the operator.</p>
<ul><li><strong>Replacing Hanzo Flow.</strong> Flow is a visual ML pipeline / agent-</li></ul>
<p> building tool. Different concern. Flow can submit Pipelines via the HTTP surface like any other client.</p>
<ul><li><strong>Replacing Hanzo Auto.</strong> Auto is an IFTT-style trigger framework</li></ul>
<p> on top of <code>hanzoai/tasks</code>. Different concern. Auto can call <code>/v1/ml/*</code> like any other client.</p>
<ul><li><strong><code>hanzoai/ml</code> (Rust) becoming the control plane.</strong> It stays as</li></ul>
<p> Candle-based ML primitives consumed inside training containers, not as the scheduler.</p>
<ul><li><strong>Multi-cluster scheduling.</strong> Out of scope — see Open Questions.</li></ul>
<h2>Open questions</h2>
<ol><li><strong>Multi-cluster scheduling.</strong> Defer to a future HIP. For now, the</li></ol>
<p> operator is a single-cluster reconciler per Hanzo deployment. Cross-cluster federation can be layered later via A2A delegation between cluster-local ML agents.</p>
<ol><li><strong>Notebook base image story.</strong> Where does</li></ol>
<p> <code>ghcr.io/hanzoai/notebook:v1.0.0</code> live? Likely a <code>hanzoai/notebook</code> repo with the JupyterLab + IAM SSO sidecar image. Tracked separately; the operator only consumes the image.</p>
<ol><li><strong>Sweep optimization library.</strong> Default is Bayesian (Tree-Parzen</li></ol>
<p> Estimator). Pure Rust implementation in <code>operator/crates/ml-controller/src/sweep_opt.rs</code>. Grid and random are trivial. BYO algorithm via HIP-0105 extension module — open question is whether a stable plugin API graduates from &quot;experimental&quot; in v0.2.</p>
<ol><li><strong>Console ML view.</strong> Whether to fork the open-source ML pipeline</li></ol>
<p> frontend or write a clean Hanzo Console ML view from scratch. The sibling work tracks this; the operator is decoupled from the answer (the HTTP/REST surface is stable regardless).</p>
<ol><li><strong>In-cluster GPU scheduling fairness.</strong> Cross-tenant GPU fairness</li></ol>
<p> on a shared cluster is a real production concern. The operator&#x27;s <code>TrainingJob</code> reconciler respects K8s resource requests but does not yet implement gang scheduling or fair-share. Either land that in v0.2 or document that gang scheduling is out of scope and require workloads that need it to use dedicated nodepools.</p>
<ol><li><strong>Pipeline retry semantics on partial step failure.</strong> Default is</li></ol>
<p> <code>hanzoai/tasks</code>&#x27;s retry policy. Whether to expose <code>spec.steps[].retryPolicy</code> at the Pipeline level for per-step overrides — yes, but tracked for v0.2.</p>
<ol><li><strong>Experiment metric write rate limiting.</strong> A misbehaving training</li></ol>
<p> loop can flood <code>hanzoai/datastore</code>. Need a per-experiment metric write quota; suggested default is 1000 writes/sec/experiment. Tracked for v0.2.</p>
<h2>Pending additions (v0.1.x roadmap)</h2>
<p>The v0.1.0 CRD set covers TrainingJob / Experiment / Sweep / Pipeline / Notebook / Model / Inference. The upstream Kubeflow project family also ships a Spark Operator + Hub (Model Registry); to cover the full lifecycle Hanzo ML will add in subsequent v0.1.x releases:</p>
<p>| Object | Replaces / competes with | Purpose | Target | |---|---|---|---| | <strong>SparkJob</strong> | Kubeflow Spark Operator | Spark + data workloads on K8s | v0.1.1 | | <strong>Run</strong> | KFP / MLflow run record | One execution of a Pipeline / Sweep / Training | v0.1.1 | | <strong>Metric</strong> | MLflow / Katib metric stream | Time-series + scalar metrics per Run | v0.1.1 | | <strong>ModelVersion</strong> | MLflow Model Registry version | Versioned record of a Model | v0.1.1 | | <strong>Artifact</strong> | KFP / MLflow artifact metadata | VFS-backed artifact pointer | v0.1.1 |</p>
<p>After these, Hanzo ML provides Kubeflow&#x27;s full project surface (Spark Operator, Notebooks, Trainer, Katib, KServe, Hub/Model Registry, Pipelines, Dashboard) on Hanzo primitives, with one operator binary and no Kubeflow estate dependency.</p>
<h2>Positioning</h2>
<p>Hanzo ML is <strong>not</strong> a Kubeflow distribution. It is the Hanzo-native integrated ML control plane for teams that want the same lifecycle coverage — training, data jobs, notebooks, experiments, sweeps, pipelines, model registry, and inference — wired directly into Hanzo IAM, VFS, datastore, observability, MCP, billing, and white-label cloud.</p>
<p>Kubeflow&#x27;s strength is community modularity across many CNCF projects. Hanzo ML&#x27;s strength is integration with fewer moving parts. Both are valid choices for different organizations.</p>
<h2>References</h2>
<ul><li>HIP-0026 — IAM Standard (auth for ML control plane)</li><li>HIP-0027 — Secrets Management Standard (KMSSecret CRs for image</li></ul>
<p> pull credentials, model registry credentials)</p>
<ul><li>HIP-0106 — Cloud / Unified Hanzo Binary (the HTTP mux that mounts</li></ul>
<p> <code>/v1/ml/*</code>)</p>
<ul><li>HIP-0108 — On-Demand Supervisor + Warm Pool (tier-2 idle eviction</li></ul>
<p> for notebooks and inference replicas)</p>
<ul><li>HIP-0010 — MCP Integration Standards (the protocol the ML tools</li></ul>
<p> surface speaks)</p>
<ul><li>Upstream references (do NOT vendor or fork): Kubeflow project</li></ul>
<p> (https://www.kubeflow.org), MLflow (https://mlflow.org), Katib (https://www.kubeflow.org/docs/components/katib/), KServe (https://kserve.github.io), Seldon Core (https://github.com/SeldonIO/seldon-core).</p>
<footer>
<div class="fmap">
<div><b>Build</b>
<a href="https://hanzo.app">hanzo.app</a> app builder
<a href="https://hanzo.codes">hanzo.codes</a> code with an agent
<a href="https://hanzoskills.com">hanzoskills.com</a> what an agent reads</div>
<div><b>Run</b>
<a href="https://console.hanzo.ai">console.hanzo.ai</a> cloud console
<a href="https://api.hanzo.ai">api.hanzo.ai</a> the API</div>
<div><b>Account</b>
<a href="https://hanzo.id">hanzo.id</a> sign in
<a href="https://pay.hanzo.ai">pay.hanzo.ai</a> top up
<a href="https://billing.hanzo.ai">billing.hanzo.ai</a> invoices and usage</div>
<div><b>Company</b>
<a href="https://hanzo.ai">hanzo.ai</a> canonical
<a href="https://hanzo.ventures">hanzo.ventures</a> ventures</div>
</div>
<div class="fend"><a href="/hip-0109.md">This page as plain text</a> · <a href="/llms.txt">Index for agents</a></div>
</footer>
</div></body></html>