hanzo-build-linux-amd64, lux-build-linux-amd64, zoo-build-linux-amd64 and
every other org-prefixed spelling all meant the same machine. Which org a
job belongs to does not change which host can build it, so the org bought
one pool name per org per arch and each could go stale on its own -- the
evo host alone answered to fourteen names for one label.
One vocabulary now: linux-amd64, linux-arm64, macos-arm64, win-amd64. The
generic labels every plain workflow uses (ubuntu-latest, self-hosted) are
untouched, so naive CI keeps working.
Co-authored-by: Hanzo Dev <[email protected]>
The lab runs amd64 (evo) and arm64 (spark), and this image was cut
linux/amd64 only — so it deploys on half the fleet and an arm64 node
cannot pull it at all.
`docker build` + `docker push` cannot carry two platforms: the daemon's
built-in builder builds one, and a manifest list cannot be loaded into a
local daemon at all. So the publish runs on a docker-container builder and
pushes straight to the registry, with binfmt for the emulated arm64 leg.
Every native dependency in uv.lock has an aarch64 wheel, so the leg
resolves rather than compiling from source.
The two gates keep the daemon they had: both setups sit BELOW them, so the
suites still build and run this arch natively instead of testing a QEMU
translation of the binary.
The service claimed a throughput without naming the instrument that produced
it. bench.py is that instrument, and the note now records what it measures —
a corpus of real speech with a word-to-time map, the cost of one decode pass,
and N concurrent talkers — so the number can be reproduced or argued with
rather than taken on faith.
Co-authored-by: Hanzo Dev <[email protected]>
The field was read from the raw form, so it worked but openapi.json
still described an endpoint that takes four fields and cannot time
anything. A capability the contract does not mention is one that gets
lost again — which is how this one was lost the first time. Declaring
it as a form field puts it in the schema and removes the second read of
a body FastAPI had already parsed.
The bracketed name is how a multipart array is spelled: it is what the
OpenAI SDKs send and what the ai plane sends. Accepting the bare
spelling too was leniency belonging at the public edge, not on an
in-cluster hop, and the ai plane keeps it.
/v1/audio/transcriptions read four form fields and answered with the
joined text. The decoder knew where every segment started and stopped,
and with one flag knows where every word does, and all of it was dropped
on the floor — so nothing downstream could time a caption to a word, and
consumers that needed word boundaries divided a line's span by its
letters, which drifts.
verbose_json now carries segments, and carries words when the caller asks
for them. timestamp_granularities is read in either spelling: OpenAI's
SDKs bracket the name, hand-rolled clients do not, and reading one leaves
half the callers silently untimed. Segments are the default because the
decode already produced them; word alignment is a second pass over the
cross-attention, so it runs only when asked. Asking for timings without
verbose_json is refused rather than billed and discarded.
0.1.10 built, tested and pushed green, and then stranded itself. The pin step
asks KMS for UNIVERSE_PIN_TOKEN, took a single 503, and exited — so the newest
image sat in the registry unreferenced while the cluster kept serving 0.1.6.
That is precisely the failure pin.sh was written to end, arriving one step
earlier than the step it guards.
The 503 was not a fault. api.hanzo.ai serves KMS from ONE replica on a
ReadWriteOnce volume — only one pod can mount it — so every deploy of that
service is a minute or so with nothing answering, and it deploys often. Any
release whose pin lands in that window loses, at random, for no reason of its
own.
Reading a secret is idempotent, so the fix is to ask again: six tries over ~100s,
which outlasts a rollout. A KMS that is genuinely down still fails the build and
still says which endpoint and org it asked.
Proven three ways before shipping, because an untested retry is worse than none
— it turns a loud failure into a slow one. A URL that answers passes straight
through with no retries; a URL that always 503s gives up after six and reports
it; and a stand-in that refuses twice then answers recovers on the third call.
Co-authored-by: Hanzo Dev <[email protected]>
0.1.9 never built. The live stage died at collection with
test_speech.py:15: import bench
E ModuleNotFoundError: No module named bench
and every car after it stops, so nothing was published — 0.1.6 stayed the newest
image that exists while three versions sat merged.
The cause is that both stages ran a bare `pytest -q`, which collects whatever
files happen to be in the image, so each stage needed its file list maintained by
hand and the two lists had to agree. bench.py arrived with 0.1.9 and went into
the test stage list only. The live stage then collected a test_speech.py whose
import was not there.
Naming the suite in each stage removes the class, not just this instance: a stage
now runs exactly what it declares, so a new test file cannot be picked up by a
stage that was never given what it needs. It also stops the live stage re-running
the stubbed suite — test_speech.py monkeypatches stt.transcribe and tts.speak, so
against real weights it proved nothing the test stage had not already proven, and
test_live.py never imported bench at all.
Proven rather than reasoned: hiding bench.py makes `pytest -q test_speech.py`
exit 2 with that same ModuleNotFoundError, and restoring it passes 50 tests;
test_live.py collects its 8 standalone.
Co-authored-by: Hanzo Dev <[email protected]>
The measurements behind 0.1.7 and 0.1.8 were taken on a box carrying nine runaway
processes at ~81% CPU each, three and a half days old, so every absolute figure in
those two commits and in the comments they shipped was taken at roughly a quarter
of real capacity. Re-run on four idle cores, interleaved over three rounds:
num_workers=1 (the default) 12.59 12.98 12.95 at 1, 2, 4 concurrent
num_workers=2 12.72 12.71 12.81
num_workers=4 12.41 12.40 15.77
cpu_threads=2 + workers=2 8.45 14.68 15.08
Three claims are withdrawn. The default does NOT serialize concurrent decodes —
it already scales flat from one session to four, so there was no serialization
defect and the comment saying otherwise was wrong. num_workers=2 buys nothing.
And cpu_threads=2 was not the worst of six: it trades a third of single-stream
speed for 13-16% at concurrency, which is a real trade rather than a pure loss.
What holds is the conclusion, for a better reason than the one given: PARALLEL=4
is the only setting that keeps full single-stream speed AND takes the best
aggregate, and cpu_threads is still worth dropping.
The FAIRNESS fix is unaffected and is confirmed on the clean box: three concurrent
sessions ran 18, 0 and 19 passes before it and 46, 46, 46 after, committing within
a few percent of the same text. Sustainable concurrency is 4 — median trail 6.33 s,
p95 8.81 s, slope +0.0114 — measured by backlog, never by ack latency.
Co-authored-by: Hanzo Dev <[email protected]>
0.1.7 set cpu_threads=2 alongside num_workers=2, reasoning that 2 x 2 fills a
4-core pod exactly. Measured, that is the WORST of six configurations and worse
than the code it replaced: 1.55 audio-seconds per wall-second at two concurrent
against 3.19 for the original, while burning twice the CPU for less work — the
signature of threads spin-waiting on each other rather than decoding.
num_workers is the half that works. Left to size its own pool, CTranslate2 gives
5.55 a/s at two concurrent and 7.52 at four, against 3.03 for one worker. So the
knob goes, and PARALLEL rises to 4.
Each config measured in its own process, pinned to 4 cores, on the same 8 s
input. The first attempt put all four in ONE process and the models contaminated
each other — the sign was right and the magnitude was not, which is the whole
reason this is a second commit and not the first one.
Co-authored-by: Hanzo Dev <[email protected]>
Carries the fairness fix (a pass hands the worker back, so the pool's queue
decides the order) and real parallelism (num_workers, so two decodes are two
decodes). 0.1.6 has neither.
Co-authored-by: Hanzo Dev <[email protected]>
Two defects, both measured, both about capacity rather than correctness of the
text.
STARVATION. _work looped until its OWN window stopped growing, and under
continuous audio that is never — new bytes always arrive mid-decode — so a
worker that started never returned. The pool's slots were held by whichever
sessions reached them first, for as long as those sessions kept receiving audio,
which in a meeting is the whole meeting. Measured on three concurrent sessions,
one ran ZERO passes across its entire 70 s: it returned empty text and was
billed for every second of it, and its window grew to 69.9 s because the WINDOW
cap lives in the pass that never ran. A pass now stands down and re-submits
itself, so the pool's FIFO queue decides the order.
ILLUSORY CONCURRENCY. load() is cached, so every pool worker shared one
WhisperModel — and CTranslate2 serializes concurrent calls on it. Measured on
one 8 s input: one decode 2.10 s, two concurrent 4.46 s (2.13x one), four
concurrent 9.95 s (4.74x one). The pod ran one decode at a time and left three
of its four cores idle. num_workers is what makes them parallel and cpu_threads
is what each gets; the pool now takes its width from the model rather than
naming its own.
Also: open() names the pod that took the session. A growing transcript is a
window in one replica's memory and this Service address round-robins per
connection, so at two replicas half of a session's pushes reached a pod that had
never heard of it.
What was NOT changed, and why: the window is not decoded in smaller pieces. A
pass costs 1.535 s fixed + 0.0756 s per audio-second, so per audio-second a
bigger window is strictly cheaper — six passes over a 2 s tail cost 10.1 s where
one pass over 12 s costs 2.48 s.
Mutation-proved: restoring the loop reproduces the starvation exactly (session 2
runs 0 passes while 0 and 1 run 34 and 33); making HERE never name the pod fails
the pinning test.
Co-authored-by: Hanzo Dev <[email protected]>
A decoder costs the same for silence as for speech, so an attendant in a
quiet room occupied a worker and billed a tenant for text that came back
empty. Silero already ships inside faster-whisper and already runs inside
every decode; running it in FRONT of the decoder — 0.67 ms per 250 ms push,
measured — turns that second into no decode and no charge at all.
It is a squelch rather than a per-window classifier, because the classic
way to lose more than you save is to start at the trigger: the voice is
not detected until it is under way. Sliding a word across a push shows the
size of it — with no lead-in, 8 of 32 alignments clip up to 71.8 ms off
the front of the word, and with LEAD they keep 428 ms of margin.
`duration` on the ack becomes the seconds that reached a decoder, so the
plane meters what we listened to. It can exceed one push's own length, on
the push that hands over the lead-in, and can never exceed the audio a
session was sent.
Co-authored-by: Hanzo Dev <[email protected]>
The docstring still credited ffmpeg for rendering the test audio after that
step became a numpy resample, which is the kind of comment that sends the next
reader looking for a dependency the code no longer has.
Co-authored-by: Hanzo Dev <[email protected]>
Moving decodes into a pool put their exceptions inside a Future that nothing
reads, so a failing decoder produced exactly the symptoms of a working one:
every push still acked 200 with the right duration, and the transcript was
simply always empty. Measured, not assumed — a raw thread prints such an
exception to stderr and a discarded Future does not.
Logged before standing down, so whatever watches the flag sees the reason
first, and the next push retries.
Co-authored-by: Hanzo Dev <[email protected]>
The live gate caught this on its first run: 40 tests passed and the build was
still red, exit 134. A raw daemon thread is not joined at interpreter shutdown,
so a decode still inside ctranslate2's C++ when the runtime tore down aborted —
`terminate called without an active exception`. The failure is in the exit code,
not in the report, which is why the fast suite and the summary line both looked
fine. In a pod it is a container that cannot drain on SIGTERM.
A pool fixes it because its workers ARE joined, and it is the better shape
anyway: decoding is CPU-bound and the pod is capped at 4 cores, so bounding
concurrent decodes stops many sessions from thrashing each other. Per-session
single flight is unchanged.
The test shuts a child down mid-decode and asserts it exits 0 — with the model
loaded first and a pause after pushing, so shutdown lands inside the decode by
construction rather than by luck.
Co-authored-by: Hanzo Dev <[email protected]>
The three things a reader would otherwise have to rediscover: that a WebSocket
cannot reach this service (ZAP builds a request as one frame and zip has no
hijack, so the wall is inside the pod, not at the edge), that `duration` on an
ack is the seconds that push carried and is what the plane meters, and that
sessions are pod-local while speech.yaml runs two replicas — so whoever routes
must pin a session to its pod.
Co-authored-by: Hanzo Dev <[email protected]>
Transcription only existed as a batch call, so a microphone had to stop before
anything could be decoded. This adds the streaming sibling: audio arrives in
250 ms pushes and the response to each push carries the transcript so far.
Half duplex over ordinary HTTP, because that is the transport this service
actually has. `ai` is a child process reached over ZAP, which builds a request
as ONE frame — there is no request-head frame and no upgrade path, so neither a
WebSocket nor a streamed request body can reach here. A chunk that fits in one
frame is the shape the wire already carries, and it makes backpressure
structural: chunk n+1 cannot be sent until n is acked.
Whisper decodes a span, not a sample, so the window is re-decoded as it fills
and segments that ended a guard-length before the audio did are committed and
their bytes freed. That is what keeps the window bounded and the committed text
stable. The ack never waits for a decode; one decode runs per session at a time.
`duration` on each ack is the seconds THAT push carried, named as the batch
route names it, so a push and a batch transcription are metered by one rule and
every second is billed once. It is arithmetic on the bytes received, so it does
not depend on the decoder returning anything.
The suite gains a second gate: kokoro speaks a sentence and whisper has to say
it back, against the real weights. The fast suite replaces faster-whisper with a
stand-in — which is what makes the window arithmetic checkable exactly, and also
what would let a broken decoder pass unnoticed.
Co-authored-by: Hanzo Dev <[email protected]>
The Ship it step asked for /v1/kms/orgs/<org>/secrets/deploy/UNIVERSE_PIN_TOKEN
and read .secret.value. KMS serves /v1/kms/secrets/deploy/... and answers .value:
the org is carried by the caller credential, not by the path. Measured, both
hosts agree — the served path answers 403 unauthenticated, the /orgs/ form 404s.
So 0.1.2 built, pushed, and then died at the pin with curl exit 22 — the image
is in the registry and the fleet still serves 0.1.1, which is the exact failure
the step was added to prevent, reproduced by the step itself.
0.1.3 carries no service change; the version moves because the guard refuses to
rebuild a published tag, and because a pin step is only proven by a release that
pins itself.
Co-authored-by: Hanzo Dev <[email protected]>
This lane ended at "push". cd.hanzo.ai does not watch the registry — it watches
universe's values files — so an image that is built and never pinned is
published and unreachable. 0.1.1 sat built while the fleet served 0.1.0 for
exactly that reason, and the only thing that moved it was a human noticing.
pin.sh is the fleet's one way to do it, and it resolves tag AND digest from a
single registry lookup. Two lookups at two moments drift, and drift here does
not fail — it succeeds at doing nothing: a reference carrying both resolves by
DIGEST, so a pod reports the new tag and runs the old bytes.
0.1.2 carries no service change. The version moves because the guard above
refuses to rebuild a published tag, and because this is the release that proves
the new step: it is the first speech image whose pin is moved by the build
rather than by hand.
Co-authored-by: Hanzo Dev <[email protected]>
Neither earlier form published 0.1.1. A bare `uv` depends on how the runner
image happens to be built; `docker run -v "$PWD":/w` depends on the runner
sharing a filesystem with the docker daemon, and under docker-in-docker that
path resolves on the HOST — the container mounts an empty directory and the
suite cannot even find pyproject.toml.
`docker build --target test` has neither dependency, and building an image is
the one thing this lane is already known to do.
The stage sits ABOVE the weight bake: the suite stubs both model calls, so it
needs no weights, and a contract break is now caught before 350MB is downloaded
rather than after. Verified locally — 18 passed in 1.83s, 4.5s of build.
Co-authored-by: Hanzo Dev <[email protected]>
`uv run` assumed a uv on the build runner. The runner is guaranteed to have
docker — it builds the image two steps down — and nothing guarantees uv, so the
gate depended on how the runner image happens to be built rather than on
anything this repo controls. 0.1.1 never published.
The pinned uv image also fixes the interpreter at 3.12, the version the
Dockerfile installs, so the suite runs where the service will.
Co-authored-by: Hanzo Dev <[email protected]>
Two values were computed here and thrown away, and both were the answer to a
question the caller had to ask.
DURATION. `segments, _ = load(model).transcribe(...)` discarded faster-whisper's
info, which carries the audio's duration. Transcription is billed per MINUTE by
everyone who sells it, so a response without a duration cannot be billed at all
— the ai plane metered every transcription at zero seconds no matter how much
audio was sent. transcribe returns it now, and verbose_json carries it exactly
as OpenAI's does (plain json stays {"text"}, so a client reading the standard
shape is untouched). It is `duration`, the audio SUBMITTED, never
`duration_after_vad`: a caller is charged for what they asked us to listen to,
not for how much of it turned out to be talking.
CONTAINER. `if response_format == "wav" ... else mp3` answered opus, aac, flac
and pcm with an MP3. The gateway then labelled the response from the REQUEST, so
a caller asking for opus received `Content-Type: audio/opus` wrapping an MP3 —
the API lying to whatever had to play it. ffmpeg in this image has libopus,
libmp3lame, aac, flac and pcm_s16le with the matching muxers, verified against
`ffmpeg -encoders` in the running pod, so every OpenAI format is really encoded
and anything else is a 400 that names what IS available. Substitution is gone,
not relabelled.
A suite arrives with them, and gates the build before the registry login so a
red run cannot publish. It stubs the two model calls, needing no weights and no
GPU, and answers in seconds.
The mutation that matters: stubbing `stt.transcribe` to test the ROUTE let the
duration fix be reverted with the suite still green. The tests that catch it
stub the MODEL and exercise the real function — testing the seam you fixed, not
the one above it.
Co-authored-by: Hanzo Dev <[email protected]>
Three files are the service: routes, the whisper half, the kokoro half. No auth
of its own — in-cluster only; the ai plane (api.hanzo.ai) authenticates and
meters every caller, including the anon grant tier, before a request lands
here. Weights bake into the image at build so the container boots offline.
Phase 2 is /v1/realtime: full-duplex websocket speech with barge-in, plus
camera-frame relay into the multimodal models.
Co-authored-by: Hanzo Dev <[email protected]>
zeekay
