speech: decode the talking, not the room

A decoder costs the same for silence as for speech, so an attendant in a
quiet room occupied a worker and billed a tenant for text that came back
empty. Silero already ships inside faster-whisper and already runs inside
every decode; running it in FRONT of the decoder — 0.67 ms per 250 ms push,
measured — turns that second into no decode and no charge at all.

It is a squelch rather than a per-window classifier, because the classic
way to lose more than you save is to start at the trigger: the voice is
not detected until it is under way. Sliding a word across a push shows the
size of it — with no lead-in, 8 of 32 alignments clip up to 71.8 ms off
the front of the word, and with LEAD they keep 428 ms of margin.

`duration` on the ack becomes the seconds that reached a decoder, so the
plane meters what we listened to. It can exceed one push's own length, on
the push that hands over the lead-in, and can never exceed the audio a
session was sent.

Co-authored-by: Hanzo Dev <[email protected]>
This commit is contained in:
2026-08-09 11:33:49 -07:00
co-authored by Hanzo Dev
parent a48646613c
commit 6d9aca2b3a
7 changed files with 427 additions and 21 deletions
+40 -7
View File
@@ -1,7 +1,8 @@
# speech — CPU STT + TTS, one OpenAI-shaped surface
Four files are the service: `main.py` (routes), `stt.py` (faster-whisper),
`tts.py` (kokoro + ffmpeg for mp3), `transcript.py` (the growing transcript).
Five files are the service: `main.py` (routes), `stt.py` (faster-whisper),
`tts.py` (kokoro + ffmpeg for mp3), `transcript.py` (the growing transcript),
`vad.py` (the squelch in front of it).
Nothing here authenticates: the ai plane (hanzoai/ai, behind api.hanzo.ai)
fronts this service and meters callers — including the anon grant tier (~3 uses
per anon) which lives THERE, not here.
@@ -20,11 +21,33 @@ that grows: `POST` to open, `POST` chunks to it, `DELETE` to close and flush.
ingress alone answers the wrong question — the wall is inside the pod.
- Audio is raw little-endian int16, mono, 16 kHz, in the body. No base64 (that
is a WebSocket text-frame tax we do not pay), no container to demux.
- **Metering: `duration` on each ack is the seconds THAT push carried**, named as
the batch route names it, so `ai` reads it into the same `AudioDurationSeconds`
it already meters (`stt/openai.go:133`) with no new billing code. Summing the
pushes yields the audio submitted once. It is arithmetic on bytes received, so
it does not depend on the decoder returning anything.
- **Metering: `duration` on each ack is the seconds THAT push put in front of a
decoder**, named as the batch route names it, so `ai` reads it into the same
`AudioDurationSeconds` it already meters (`stt/openai.go:133`) with no new
billing code. It is arithmetic on the bytes that got past the squelch, so it
does not depend on the decoder returning anything, and summing the pushes
yields the audio decoded — never more than the audio submitted. A push the
squelch drops reports 0.0; the push that OPENS it reports more than its own
length, because it carries the lead-in that earlier pushes withheld.
- **Silence is not decoded and not billed** (`vad.py`). Silero already ships
inside faster-whisper and already runs inside every decode, so running it in
FRONT of the decoder costs no new weight and no new bake step — 0.67 ms per
250 ms push, measured on the pod, against a decode of seconds. It is a squelch,
not a per-window classifier: two thresholds to open and close, `HANG` of quiet
before it closes, and `LEAD` of audio held back and handed over by the push
that opens. Measured on the pod: an idle room bills 0.0 up to about -45 dBFS
RMS of room tone (it starts to open on tone above that); a keyboard click and
full-scale white noise do not open it; a meeting bills its speech plus about
0.9 s per utterance and nothing for the lulls, so the share saved is 0% when
someone is talking continuously and 86% at 30 s between turns.
**The lead-in is the part that is easy to get wrong and expensive to lose.**
Sliding a word across a push shows why: with `LEAD = 0`, 8 of 32 alignments
clip up to 71.8 ms off the front of the word, because a push is admitted whole
and a word beginning near the end of one has nothing in hand. At 0.5 s, none
clip and the worst case still keeps 428 ms of margin. Hold the lead by slicing
from the FRONT — `[-int(LEAD * SECOND):]` reads as the last N bytes and is the
whole buffer when N is 0, which keeps every second of silence and hands the lot
to the decoder on the next word.
- Whisper decodes a span, not a sample. The window is re-decoded as it fills;
segments ending a guard-length before the audio are committed and their bytes
freed, which is what bounds the window and stops committed text being revised.
@@ -38,6 +61,16 @@ the plane, and `ai` has no route for it yet either.
- Weights bake at image BUILD (deterministic image, boots offline). Adding a
model = extend `MODELS` in stt.py/tts.py AND the bake step in the Dockerfile.
- **There is no fast tier to add: `whisper` already IS the fast one.** Measured
in the pod, both models on the same kokoro fixture, three runs each, median,
no HTTP in the loop: `whisper` (distil-small.en) runs at 1.35–2.75× realtime
and `whisper-small` (small) at 1.03–1.73×, so the default is 1.25–1.64× the
faster of the two, and both transcribe the fixture at WER 0.00 — clean, at
20 dB SNR and at 10 dB SNR alike. Splitting live captions from a final pass
would mean a bigger model for the final one, and `small` at 1.03× is already
the largest that keeps up with a meeting on four cores with no GPU in the
cluster. `whisper-small` earns its place on LANGUAGE, not on accuracy or
speed: distil-small.en is English-only.
- `whisper` / `kokoro` are the ROUTED names the ai plane's provider rows send;
renaming them is a cross-repo change, not a local one.
- CI is `.hanzo/workflows/deploy.yml` (git.hanzo.ai act_runner; GitHub Actions
+1 -1
View File
@@ -4,7 +4,7 @@ COPY --from=ghcr.io/astral-sh/uv:latest /uv /usr/local/bin/uv
WORKDIR /app
COPY pyproject.toml uv.lock ./
RUN uv sync --frozen --no-dev
COPY main.py stt.py tts.py transcript.py ./
COPY main.py stt.py tts.py transcript.py vad.py ./
# The gate, as a build stage. `docker build` is the ONE thing this lane is
# already known to do, so the test needs no uv on the runner and no bind mount
+1 -1
View File
@@ -1,6 +1,6 @@
[project]
name = "speech"
version = "0.1.5"
version = "0.1.6"
description = "CPU speech: STT (faster-whisper) + TTS (kokoro) behind one OpenAI-shaped surface"
requires-python = ">=3.12"
dependencies = [
+87
View File
@@ -171,3 +171,90 @@ def test_the_batch_route_still_reports_a_real_duration(said):
assert r.status_code == 200, r.text
assert r.json()["duration"] == pytest.approx(len(said) / stt.SECOND, abs=0.1)
assert "fox" in r.json()["text"].lower()
# ── voice activity, against the real detector ───────────────────────────────
#
# test_speech.py replaces silero, because the squelch's arithmetic is what those
# tests are about. What it cannot tell is whether the real model agrees with the
# real room: whether a fan is silence and a word is not.
TURNS = ("Right, so the migration landed yesterday and the error rate is flat.",
"Kubernetes rescheduled the workers twice overnight.")
def tone(seconds: float, dbfs: float = -45.0) -> bytes:
"""Room tone: a fan, a laptop, an open microphone in an empty room. Digital
silence is a fixture no detector can fail, so this is not that — it is noise
at a level a meeting really carries."""
rng = numpy.random.default_rng(5)
n = int(seconds * stt.RATE)
warm = numpy.convolve(rng.standard_normal(n), numpy.ones(16) / 16, mode="same")
return pcm(warm / (numpy.abs(warm).max() or 1.0) * 10 ** (dbfs / 20.0))
def pcm(x: numpy.ndarray) -> bytes:
return (numpy.clip(x, -1.0, 1.0) * 32767).astype("<i2").tobytes()
def feed(live, audio: bytes) -> float:
return sum(live.push(audio[at:at + transcript.CHUNK])
for at in range(0, len(audio), transcript.CHUNK))
@pytest.fixture(scope="module")
def meeting() -> bytes:
"""A meeting: a lull, a turn, a pause inside it, another turn, a lull."""
parts = [tone(2.5), speech(TURNS[0]), tone(0.5), speech(TURNS[1]), tone(2.5)]
return b"".join(parts)
def test_an_empty_room_is_never_decoded(said):
"""The whole point, against the real detector. Six seconds of a room with
nobody in it must cost nothing: no bytes to the decoder, no seconds to the
bill. Then the same session hears a voice and pays for it — without which
this test is also passed by a service that has stopped working."""
live = transcript.begin("whisper", "en")
assert feed(live, tone(6.0)) == 0.0, "a room with nobody in it is free"
assert live.seconds == 0.0
assert live._got == 0, "and reaches the decoder not at all"
assert feed(live, said) > 0.0, "and a voice in the same room does not"
def test_a_push_carrying_no_audio_is_answered_not_refused():
"""The route takes whatever body arrives, and an empty one is whole int16
frames and under the ceiling — so it reaches the squelch, and silero cannot
be asked about nothing: it indexes the last window of what it is given, and
of nothing there is none. A push with no audio was a no-op before there was
anything in front of the decoder and it stays one, which is also how a client
reads the newest text without sending more.
It lives here rather than with the fast tests because only the real model
has that edge — a stand-in answers an empty array quite happily, and a suite
that only ever sees the stand-in passes either way."""
live = client.post("/v1/audio/transcript", json={"model": "whisper"})
tid = live.json()["id"]
r = client.post(f"/v1/audio/transcript/{tid}", content=b"")
assert r.status_code == 200, r.text
assert r.json()["duration"] == 0.0
def test_the_word_after_a_pause_is_still_there(meeting):
"""The failure that would make this a bad trade: a squelch that opens on the
trigger has already eaten the syllable that triggered it. Every word must
survive a meeting made of turns, pauses and lulls — read back through the
real decoder, which is the only judge of whether a syllable was lost."""
live = transcript.begin("whisper", "en")
billed = feed(live, meeting)
live.close()
heard = live.text.lower()
said = " ".join(TURNS).lower()
missing = [w.strip(".,") for w in said.split() if w.strip(".,") not in heard]
assert not missing, f"the squelch lost {missing}; heard {heard!r}"
sent = len(meeting) / stt.SECOND
assert billed < sent, "and the lulls were not charged for"
assert billed == pytest.approx(live.seconds)
+167
View File
@@ -16,6 +16,7 @@ import main
import stt
import transcript
import tts
import vad
client = TestClient(main.app)
@@ -474,3 +475,169 @@ def test_a_failed_decode_says_so(monkeypatch, caplog):
settle(live)
assert "decoder is unhappy" in caplog.text
assert live.text == "" and live.pending == ""
# ── voice activity: the decode that never runs ──────────────────────────────
#
# Silero is a weight like every other, so it is replaced here and what runs is
# the squelch's own arithmetic — the two thresholds, the hangover, the lead-in.
# Everything above this line is about the decode window, so everything above
# this line gets a room that hears a voice in all of it and is untouched.
@pytest.fixture(autouse=True)
def talking(monkeypatch):
monkeypatch.setattr(
vad, "get_vad_model",
lambda: lambda heard: numpy.ones((len(heard) // vad.STEP, 1)),
)
class _Room:
"""silero's shape: one probability per window, read from how loud that window
is — so a test writes speech and silence as loud and quiet audio, and the
machine that decides between them is the real one."""
def __call__(self, heard):
return numpy.abs(heard.reshape(-1, vad.STEP)).max(axis=1).reshape(-1, 1)
@pytest.fixture
def room(monkeypatch):
monkeypatch.setattr(vad, "get_vad_model", _Room)
def loud(seconds: float) -> bytes:
return tone(30000, seconds) # 0.92 of full scale
def quiet(seconds: float) -> bytes:
return tone(3, seconds) # 0.0001 — a room with nobody in it
def test_a_quiet_room_costs_nothing(ear, room):
"""The whole point. Silence decodes to "" for the same CPU and the same money
as speech, so an attendant left in an empty room bills a tenant continuously
for nothing. Dropped audio must reach neither the decoder nor the meter."""
live = transcript.begin("whisper", None)
billed = sum(live.push(quiet(0.25)) for _ in range(12)) # 3 s of nobody
assert billed == 0.0, "silence must not be billed"
assert live.seconds == 0.0
assert ear.spans == [], "and must not reach the decoder at all"
assert len(live._window) == 0
def test_a_voice_is_decoded(ear, room):
"""The other direction, which is what makes the test above mean anything: a
squelch that dropped everything would satisfy that one perfectly."""
live = transcript.begin("whisper", None)
billed = sum(live.push(loud(0.25)) for _ in range(8))
settle(live)
assert billed == pytest.approx(2.0)
assert ear.spans, "speech must reach the decoder"
def test_the_onset_that_opened_the_squelch_is_decoded_with_it(ear, room):
"""The classic failure, and the one that costs more than it saves: a voice is
not detected until it is already under way, so a squelch that begins at the
trigger has already eaten the first syllable. The audio from before the
trigger is kept, and handed over by the push that opens."""
live = transcript.begin("whisper", None)
for _ in range(8):
assert live.push(quiet(0.25)) == 0.0
opened = live.push(loud(0.25))
assert opened > 0.25, "the push that opens carries more than itself"
assert bytes(live._window[-len(loud(0.25)):]) == loud(0.25)
before = bytes(live._window[:-len(loud(0.25))])
assert before == quiet(len(before) / stt.SECOND), "and what precedes it is what preceded"
assert len(before) == int(vad.LEAD * stt.SECOND), "as much of it as LEAD says"
assert opened == pytest.approx(len(live._window) / stt.SECOND), "all of it billed once"
def test_a_pause_inside_a_sentence_does_not_end_it(ear, room):
"""Speech is not continuous: there is a gap between words and a breath
between clauses. Closing on the first quiet window would chop a sentence into
fragments and clip the word after every pause."""
live = transcript.begin("whisper", None)
live.push(loud(0.25))
gap = [live.push(quiet(0.25)) for _ in range(2)] # half a second
after = live.push(loud(0.25))
assert gap == [0.25, 0.25], "a pause this short is inside the sentence"
assert after == 0.25, "so the word after it needs no lead-in — it never closed"
def test_the_squelch_closes_once_the_room_stays_quiet(ear, room):
"""And it must actually close, or the saving is only ever deferred. The
hangover is spent — HANG seconds of quiet decode — and then nothing does."""
live = transcript.begin("whisper", None)
live.push(loud(0.25))
after = [live.push(quiet(0.25)) for _ in range(8)] # 2 s of nobody
spent = sum(after)
assert spent == pytest.approx(0.75), "the hangover, and no more than it"
assert spent <= vad.HANG
assert after[3:] == [0.0] * 5, "quiet from there on is free"
def test_a_session_never_bills_a_second_it_did_not_decode(ear, room):
"""The invariant that makes the lead-in safe to charge for: the push that
opens reports more than its own length, so the arithmetic has to hold over
the session rather than over one call. Every second billed is a second a
decoder read, and no second is billed twice."""
live = transcript.begin("whisper", None)
sent = billed = 0.0
for piece in (quiet(1.0), loud(1.0), quiet(1.0), loud(1.0), quiet(2.0)):
for at in range(0, len(piece), transcript.CHUNK):
billed += live.push(piece[at:at + transcript.CHUNK])
sent += len(piece[at:at + transcript.CHUNK]) / stt.SECOND
assert billed == pytest.approx(live.seconds), "the acks and the session agree"
assert live.seconds == pytest.approx(live._got / stt.SECOND), "billed is decoded"
assert billed < sent, "and the quiet room was not charged for"
class _Level:
"""A room at one stated probability — it reads the level, not the audio — so
a test can write the window silero is least sure about."""
def __init__(self):
self.p = 0.0
def __call__(self, heard):
return numpy.full((len(heard) // vad.STEP, 1), self.p)
def test_a_window_the_model_is_unsure_of_holds_its_ground(monkeypatch):
"""Between the two thresholds silero is not sure, and one threshold cannot
express that: the same probability must continue speech without being able to
begin it. With one threshold, a syllable fading through it ends the sentence
and a room's own hum starts one."""
level = _Level()
monkeypatch.setattr(vad, "get_vad_model", lambda: level)
squelch = vad.Squelch()
level.p = (vad.OPEN + vad.SHUT) / 2
assert squelch.admit(tone(1, 0.25)) == b"", "unsure does not begin speech"
level.p = vad.OPEN
assert squelch.admit(tone(1, 0.25)), "certain does"
level.p = (vad.OPEN + vad.SHUT) / 2
assert squelch.admit(tone(1, 2.0)), "and unsure does not end it, however long"
def test_the_ack_reports_zero_for_silence(ear, room):
"""`duration` on the ack is the field the ai plane meters. Whatever the
session believes, this is the number that becomes money."""
live = start()
r = client.post(f"/v1/audio/transcript/{live['id']}", content=quiet(0.25))
assert r.status_code == 200, r.text
assert r.json()["duration"] == 0.0
assert r.json()["seconds"] == 0.0
+29 -12
View File
@@ -13,9 +13,13 @@ latency lands on a later ack instead of on this one. One decode runs per session
at a time, which is also what stops a fast pusher from queueing work faster than
the CPU retires it.
`seconds` is arithmetic on the bytes received, so it is the audio submitted,
exactly, and it cannot silently become zero: it is the same quantity `transcribe`
reports as `duration`, measured the same way, and it is what the call is billed on.
Audio reaches the window through a squelch, so a quiet room reaches nothing at
all: no bytes, no decode, no charge. `seconds` is arithmetic on the bytes that
got through, and those are exactly the bytes a decoder read — so the bill is the
audio we listened to. It is the same quantity `transcribe` reports as `duration`,
measured the same way, it can never exceed the audio submitted, and it cannot
silently become zero for audio that WAS decoded, because it is the length of the
decoded bytes and nothing else.
"""
import logging
@@ -25,6 +29,7 @@ import time
from concurrent.futures import ThreadPoolExecutor
import stt
import vad
log = logging.getLogger(__name__)
@@ -64,9 +69,10 @@ class Transcript:
self.language = language
self.text = "" # committed: settled, never revised
self.pending = "" # the open tail: newest decode, may still change
self.seconds = 0.0 # audio received — the billable quantity
self.seconds = 0.0 # audio decoded — the billable quantity
self._window = bytearray()
self._got = 0 # bytes ever received; marks progress for the worker
self._voice = vad.Squelch()
self._lock = threading.Lock()
self._decoding = False
self._closed = False
@@ -80,23 +86,34 @@ class Transcript:
return now - self._touched > IDLE
def push(self, pcm: bytes) -> float:
"""Accept audio; return the seconds THIS push carried.
"""Accept audio; return the seconds THIS push put in front of a decoder.
The return value is the metered quantity for this call — per push, not
cumulative, so summing the pushes of a session yields the audio submitted
once and only once.
That is the metered quantity for this call — per push, not cumulative, so
summing the pushes of a session yields the audio decoded once and only
once. A push the squelch drops returns 0.0: it was never decoded, so
there is nothing to charge for. A push that OPENS the squelch carries the
lead-in with it and so reports more than its own length; those seconds
were withheld from every earlier push, so the session still bills each
second at most once and never bills one it did not decode.
The squelch runs under the lock because its memory of the room is part of
the session's state, and it is a millisecond of arithmetic — no decode
ever holds this lock.
"""
with self._lock:
self._window += pcm
self._got += len(pcm)
self.seconds += len(pcm) / stt.SECOND
self._touched = time.monotonic()
heard = self._voice.admit(pcm)
if not heard:
return 0.0
self._window += heard
self._got += len(heard)
self.seconds += len(heard) / stt.SECOND
start = not self._decoding and len(self._window) >= FLOOR * stt.SECOND
if start:
self._decoding = True
if start:
_pool.submit(self._work)
return len(pcm) / stt.SECOND
return len(heard) / stt.SECOND
def close(self) -> None:
"""Decode what is left and commit all of it — nothing follows, so nothing
+102
View File
@@ -0,0 +1,102 @@
"""Voice activity: the audio worth decoding, and nothing else.
Silence costs a decoder exactly what speech costs it. A second of an empty room
occupies a worker for a second and bills a second, and returns "". A meeting is
mostly that — the pauses inside one person's turn, and the longer stretches
where nobody has the floor at all — so the cheapest decode is the one that never
runs.
Silero says how much each 32 ms of audio sounds like a voice. faster-whisper
already carries that model and already runs it inside every decode, so this
costs no new weight and no new bake step; what is new is running it BEFORE the
decoder instead of inside it, where the answer can save the decode rather than
merely trim its input.
A probability is not yet a decision. Deciding per window clips speech at both
ends, and a clipped first syllable costs more than the CPU it saved: a word that
begins softly is not detected until it is already under way, and a breath inside
a sentence is not the end of the sentence. So the decision is a squelch, in the
sense radio has always meant — it opens on voice, closes only after a run of
quiet, hands back the audio from just BEFORE it opened, and keeps passing for a
moment after the voice stops.
"""
import numpy
from faster_whisper.vad import get_vad_model
import stt
# Samples silero scores at once: 32 ms at 16 kHz. Its own window, not a choice.
STEP = 512
# Open loud, close quiet — the two thresholds are what stops the decision
# chattering on a syllable's own envelope. Silero's own filter reads its
# probabilities the same way, and this is the same distance apart.
OPEN = 0.5
SHUT = 0.35
# Quiet this long ends the utterance. Longer than a breath or the gap between
# words, shorter than the pause between one person finishing and the next
# starting — that is the whole span in which this number is the right one.
HANG = 0.8
# Audio held back while quiet, and handed over on the push that opens. Whisper
# reads a span, and the span it is given must contain the onset that triggered
# the decision — the trigger necessarily lags the onset, so the onset is in the
# past by the time anything knows to keep it.
LEAD = 0.5
class Squelch:
"""Passes voice, drops silence. One per session: it remembers.
A push is admitted whole or not at all. Cutting on the sample would splice
the window at word boundaries and hand the decoder audio with steps in it;
admitting whole pushes keeps what the decoder reads continuous, and LEAD
covers the onset far better than a sample-exact edge would.
"""
def __init__(self):
self._open = False
self._quiet = 0.0
self._lead = b""
def admit(self, pcm: bytes) -> bytes:
"""Return the audio worth decoding: b"" while the room is quiet, the push
when it is not, and the lead-in as well on the push that opens.
A push carrying no audio carries no decision either, and silero cannot be
asked about nothing — it indexes the last window of what it is given.
"""
if not pcm:
return b""
heard = False
for voiced in self._voiced(pcm):
if voiced >= OPEN:
self._open, self._quiet = True, 0.0
elif self._open:
self._quiet = self._quiet + STEP / stt.RATE if voiced < SHUT else 0.0
self._open = self._quiet < HANG
heard = heard or self._open
if not heard:
# Sliced from the front, because the end of a buffer is counted from
# the end and `[-0:]` is the whole of it: written the other way, a
# LEAD of zero keeps every second of silence instead of none, and
# hands the lot to the decoder on the next word.
held = self._lead + pcm
self._lead = held[max(0, len(held) - int(LEAD * stt.SECOND)):]
return b""
was, self._lead = self._lead, b""
return was + pcm
def _voiced(self, pcm: bytes) -> numpy.ndarray:
"""One probability per STEP samples. The model wants whole windows, so a
push that is not a whole number of them is padded — a 250 ms push at this
rate is exactly eight, and pads not at all."""
heard = stt.samples(pcm)
short = -len(heard) % STEP
if short:
heard = numpy.pad(heard, (0, short))
return get_vad_model()(heard).reshape(-1)