mirror of
https://github.com/hanzoai/ai.git
synced 2026-11-20 14:13:23 +00:00
Both paths that hand a workload to a Kubernetes scheduler named a vendor in
the contract: the TrainJob's resourcesPerNode.limits."nvidia.com/gpu" and the
KServe InferenceService's limits."nvidia.com/gpu". That is not a requirement,
it is one vendor's name for a requirement, and it cost two things.
A job became unroutable to hardware that could have run it. And -- the
dangerous half -- a cluster whose nodes advertise no accelerator at all still
ACCEPTED the workload, because a resource name a scheduler has never heard of
is not a syntax error. It is a pod that pends forever. The InferenceService
CRD is installed and its controller is running, so that create SUCCEEDS: it
leaves a predictor that can never be scheduled and a model registered on
api.hanzo.ai whose every inference call fails. Nothing says so.
So the requirement stays a COUNT and the NAME is discovered: intersect what
the cluster's nodes advertise as allocatable with the names accelerators are
advertised under. The emitted name is therefore always one some node really
provides, and an empty intersection is refused out loud at submit time --
before a namespace, a secret or a CR exists -- instead of being deferred to a
workload that never schedules. do-sfo3-hanzo-k8s advertises cpu,
ephemeral-storage, memory and pods across all of its nodes and nothing else,
so today both paths refuse with the reason and the remedy.
Vendor names now appear in exactly one table, and only to RECOGNISE an
advertised resource as an accelerator -- never to choose one. Which vendor
clears the requirement is the cluster's business, answered by advertising it,
so adding a vendor to that table can never send a workload somewhere no node
advertises. The vocabulary is the one a linked machine already advertises in
cloud/apps/agents (Spec.GPUs{Vendor,Model,Memory}, and Need with no vendor
field at all): matching a Need against a machine's advertised Spec is
agents.Spec.Satisfies; matching the same requirement against a cluster's
advertised node resources is selectAccelerator. Two substrates, one
vocabulary, and neither asks a caller to name a vendor.
Selection and rendering are pure functions of advertised capability, so the
cluster is stated as data in the tests -- including the measured allocatable
of the live cluster, and the rendered CRs asserted to carry only the
accelerator the cluster supplied. Reintroducing either literal fails that
test.
job.GpuType keeps its nvidia-* SKU names on purpose: it is read only by
MeterFinetuneGpuHours, so it is a price-list key naming a rentable product,
not a scheduling contract.
Co-authored-by: Hanzo Dev <[email protected]>