mirror of
https://github.com/hanzoai/speech.git
synced 2026-10-04 07:37:19 +00:00
speech: decode the talking, not the room
A decoder costs the same for silence as for speech, so an attendant in a quiet room occupied a worker and billed a tenant for text that came back empty. Silero already ships inside faster-whisper and already runs inside every decode; running it in FRONT of the decoder — 0.67 ms per 250 ms push, measured — turns that second into no decode and no charge at all. It is a squelch rather than a per-window classifier, because the classic way to lose more than you save is to start at the trigger: the voice is not detected until it is under way. Sliding a word across a push shows the size of it — with no lead-in, 8 of 32 alignments clip up to 71.8 ms off the front of the word, and with LEAD they keep 428 ms of margin. `duration` on the ack becomes the seconds that reached a decoder, so the plane meters what we listened to. It can exceed one push's own length, on the push that hands over the lead-in, and can never exceed the audio a session was sent. Co-authored-by: Hanzo Dev <[email protected]>
This commit is contained in:
@@ -1,7 +1,8 @@
|
||||
# speech — CPU STT + TTS, one OpenAI-shaped surface
|
||||
|
||||
Four files are the service: `main.py` (routes), `stt.py` (faster-whisper),
|
||||
`tts.py` (kokoro + ffmpeg for mp3), `transcript.py` (the growing transcript).
|
||||
Five files are the service: `main.py` (routes), `stt.py` (faster-whisper),
|
||||
`tts.py` (kokoro + ffmpeg for mp3), `transcript.py` (the growing transcript),
|
||||
`vad.py` (the squelch in front of it).
|
||||
Nothing here authenticates: the ai plane (hanzoai/ai, behind api.hanzo.ai)
|
||||
fronts this service and meters callers — including the anon grant tier (~3 uses
|
||||
per anon) which lives THERE, not here.
|
||||
@@ -20,11 +21,33 @@ that grows: `POST` to open, `POST` chunks to it, `DELETE` to close and flush.
|
||||
ingress alone answers the wrong question — the wall is inside the pod.
|
||||
- Audio is raw little-endian int16, mono, 16 kHz, in the body. No base64 (that
|
||||
is a WebSocket text-frame tax we do not pay), no container to demux.
|
||||
- **Metering: `duration` on each ack is the seconds THAT push carried**, named as
|
||||
the batch route names it, so `ai` reads it into the same `AudioDurationSeconds`
|
||||
it already meters (`stt/openai.go:133`) with no new billing code. Summing the
|
||||
pushes yields the audio submitted once. It is arithmetic on bytes received, so
|
||||
it does not depend on the decoder returning anything.
|
||||
- **Metering: `duration` on each ack is the seconds THAT push put in front of a
|
||||
decoder**, named as the batch route names it, so `ai` reads it into the same
|
||||
`AudioDurationSeconds` it already meters (`stt/openai.go:133`) with no new
|
||||
billing code. It is arithmetic on the bytes that got past the squelch, so it
|
||||
does not depend on the decoder returning anything, and summing the pushes
|
||||
yields the audio decoded — never more than the audio submitted. A push the
|
||||
squelch drops reports 0.0; the push that OPENS it reports more than its own
|
||||
length, because it carries the lead-in that earlier pushes withheld.
|
||||
- **Silence is not decoded and not billed** (`vad.py`). Silero already ships
|
||||
inside faster-whisper and already runs inside every decode, so running it in
|
||||
FRONT of the decoder costs no new weight and no new bake step — 0.67 ms per
|
||||
250 ms push, measured on the pod, against a decode of seconds. It is a squelch,
|
||||
not a per-window classifier: two thresholds to open and close, `HANG` of quiet
|
||||
before it closes, and `LEAD` of audio held back and handed over by the push
|
||||
that opens. Measured on the pod: an idle room bills 0.0 up to about -45 dBFS
|
||||
RMS of room tone (it starts to open on tone above that); a keyboard click and
|
||||
full-scale white noise do not open it; a meeting bills its speech plus about
|
||||
0.9 s per utterance and nothing for the lulls, so the share saved is 0% when
|
||||
someone is talking continuously and 86% at 30 s between turns.
|
||||
**The lead-in is the part that is easy to get wrong and expensive to lose.**
|
||||
Sliding a word across a push shows why: with `LEAD = 0`, 8 of 32 alignments
|
||||
clip up to 71.8 ms off the front of the word, because a push is admitted whole
|
||||
and a word beginning near the end of one has nothing in hand. At 0.5 s, none
|
||||
clip and the worst case still keeps 428 ms of margin. Hold the lead by slicing
|
||||
from the FRONT — `[-int(LEAD * SECOND):]` reads as the last N bytes and is the
|
||||
whole buffer when N is 0, which keeps every second of silence and hands the lot
|
||||
to the decoder on the next word.
|
||||
- Whisper decodes a span, not a sample. The window is re-decoded as it fills;
|
||||
segments ending a guard-length before the audio are committed and their bytes
|
||||
freed, which is what bounds the window and stops committed text being revised.
|
||||
@@ -38,6 +61,16 @@ the plane, and `ai` has no route for it yet either.
|
||||
|
||||
- Weights bake at image BUILD (deterministic image, boots offline). Adding a
|
||||
model = extend `MODELS` in stt.py/tts.py AND the bake step in the Dockerfile.
|
||||
- **There is no fast tier to add: `whisper` already IS the fast one.** Measured
|
||||
in the pod, both models on the same kokoro fixture, three runs each, median,
|
||||
no HTTP in the loop: `whisper` (distil-small.en) runs at 1.35–2.75× realtime
|
||||
and `whisper-small` (small) at 1.03–1.73×, so the default is 1.25–1.64× the
|
||||
faster of the two, and both transcribe the fixture at WER 0.00 — clean, at
|
||||
20 dB SNR and at 10 dB SNR alike. Splitting live captions from a final pass
|
||||
would mean a bigger model for the final one, and `small` at 1.03× is already
|
||||
the largest that keeps up with a meeting on four cores with no GPU in the
|
||||
cluster. `whisper-small` earns its place on LANGUAGE, not on accuracy or
|
||||
speed: distil-small.en is English-only.
|
||||
- `whisper` / `kokoro` are the ROUTED names the ai plane's provider rows send;
|
||||
renaming them is a cross-repo change, not a local one.
|
||||
- CI is `.hanzo/workflows/deploy.yml` (git.hanzo.ai act_runner; GitHub Actions
|
||||
|
||||
+1
-1
@@ -4,7 +4,7 @@ COPY --from=ghcr.io/astral-sh/uv:latest /uv /usr/local/bin/uv
|
||||
WORKDIR /app
|
||||
COPY pyproject.toml uv.lock ./
|
||||
RUN uv sync --frozen --no-dev
|
||||
COPY main.py stt.py tts.py transcript.py ./
|
||||
COPY main.py stt.py tts.py transcript.py vad.py ./
|
||||
|
||||
# The gate, as a build stage. `docker build` is the ONE thing this lane is
|
||||
# already known to do, so the test needs no uv on the runner and no bind mount
|
||||
|
||||
+1
-1
@@ -1,6 +1,6 @@
|
||||
[project]
|
||||
name = "speech"
|
||||
version = "0.1.5"
|
||||
version = "0.1.6"
|
||||
description = "CPU speech: STT (faster-whisper) + TTS (kokoro) behind one OpenAI-shaped surface"
|
||||
requires-python = ">=3.12"
|
||||
dependencies = [
|
||||
|
||||
@@ -171,3 +171,90 @@ def test_the_batch_route_still_reports_a_real_duration(said):
|
||||
assert r.status_code == 200, r.text
|
||||
assert r.json()["duration"] == pytest.approx(len(said) / stt.SECOND, abs=0.1)
|
||||
assert "fox" in r.json()["text"].lower()
|
||||
|
||||
|
||||
# ── voice activity, against the real detector ───────────────────────────────
|
||||
#
|
||||
# test_speech.py replaces silero, because the squelch's arithmetic is what those
|
||||
# tests are about. What it cannot tell is whether the real model agrees with the
|
||||
# real room: whether a fan is silence and a word is not.
|
||||
|
||||
TURNS = ("Right, so the migration landed yesterday and the error rate is flat.",
|
||||
"Kubernetes rescheduled the workers twice overnight.")
|
||||
|
||||
|
||||
def tone(seconds: float, dbfs: float = -45.0) -> bytes:
|
||||
"""Room tone: a fan, a laptop, an open microphone in an empty room. Digital
|
||||
silence is a fixture no detector can fail, so this is not that — it is noise
|
||||
at a level a meeting really carries."""
|
||||
rng = numpy.random.default_rng(5)
|
||||
n = int(seconds * stt.RATE)
|
||||
warm = numpy.convolve(rng.standard_normal(n), numpy.ones(16) / 16, mode="same")
|
||||
return pcm(warm / (numpy.abs(warm).max() or 1.0) * 10 ** (dbfs / 20.0))
|
||||
|
||||
|
||||
def pcm(x: numpy.ndarray) -> bytes:
|
||||
return (numpy.clip(x, -1.0, 1.0) * 32767).astype("<i2").tobytes()
|
||||
|
||||
|
||||
def feed(live, audio: bytes) -> float:
|
||||
return sum(live.push(audio[at:at + transcript.CHUNK])
|
||||
for at in range(0, len(audio), transcript.CHUNK))
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def meeting() -> bytes:
|
||||
"""A meeting: a lull, a turn, a pause inside it, another turn, a lull."""
|
||||
parts = [tone(2.5), speech(TURNS[0]), tone(0.5), speech(TURNS[1]), tone(2.5)]
|
||||
return b"".join(parts)
|
||||
|
||||
|
||||
def test_an_empty_room_is_never_decoded(said):
|
||||
"""The whole point, against the real detector. Six seconds of a room with
|
||||
nobody in it must cost nothing: no bytes to the decoder, no seconds to the
|
||||
bill. Then the same session hears a voice and pays for it — without which
|
||||
this test is also passed by a service that has stopped working."""
|
||||
live = transcript.begin("whisper", "en")
|
||||
|
||||
assert feed(live, tone(6.0)) == 0.0, "a room with nobody in it is free"
|
||||
assert live.seconds == 0.0
|
||||
assert live._got == 0, "and reaches the decoder not at all"
|
||||
|
||||
assert feed(live, said) > 0.0, "and a voice in the same room does not"
|
||||
|
||||
|
||||
def test_a_push_carrying_no_audio_is_answered_not_refused():
|
||||
"""The route takes whatever body arrives, and an empty one is whole int16
|
||||
frames and under the ceiling — so it reaches the squelch, and silero cannot
|
||||
be asked about nothing: it indexes the last window of what it is given, and
|
||||
of nothing there is none. A push with no audio was a no-op before there was
|
||||
anything in front of the decoder and it stays one, which is also how a client
|
||||
reads the newest text without sending more.
|
||||
|
||||
It lives here rather than with the fast tests because only the real model
|
||||
has that edge — a stand-in answers an empty array quite happily, and a suite
|
||||
that only ever sees the stand-in passes either way."""
|
||||
live = client.post("/v1/audio/transcript", json={"model": "whisper"})
|
||||
tid = live.json()["id"]
|
||||
r = client.post(f"/v1/audio/transcript/{tid}", content=b"")
|
||||
assert r.status_code == 200, r.text
|
||||
assert r.json()["duration"] == 0.0
|
||||
|
||||
|
||||
def test_the_word_after_a_pause_is_still_there(meeting):
|
||||
"""The failure that would make this a bad trade: a squelch that opens on the
|
||||
trigger has already eaten the syllable that triggered it. Every word must
|
||||
survive a meeting made of turns, pauses and lulls — read back through the
|
||||
real decoder, which is the only judge of whether a syllable was lost."""
|
||||
live = transcript.begin("whisper", "en")
|
||||
billed = feed(live, meeting)
|
||||
live.close()
|
||||
|
||||
heard = live.text.lower()
|
||||
said = " ".join(TURNS).lower()
|
||||
missing = [w.strip(".,") for w in said.split() if w.strip(".,") not in heard]
|
||||
assert not missing, f"the squelch lost {missing}; heard {heard!r}"
|
||||
|
||||
sent = len(meeting) / stt.SECOND
|
||||
assert billed < sent, "and the lulls were not charged for"
|
||||
assert billed == pytest.approx(live.seconds)
|
||||
|
||||
+167
@@ -16,6 +16,7 @@ import main
|
||||
import stt
|
||||
import transcript
|
||||
import tts
|
||||
import vad
|
||||
|
||||
client = TestClient(main.app)
|
||||
|
||||
@@ -474,3 +475,169 @@ def test_a_failed_decode_says_so(monkeypatch, caplog):
|
||||
settle(live)
|
||||
assert "decoder is unhappy" in caplog.text
|
||||
assert live.text == "" and live.pending == ""
|
||||
|
||||
|
||||
# ── voice activity: the decode that never runs ──────────────────────────────
|
||||
#
|
||||
# Silero is a weight like every other, so it is replaced here and what runs is
|
||||
# the squelch's own arithmetic — the two thresholds, the hangover, the lead-in.
|
||||
# Everything above this line is about the decode window, so everything above
|
||||
# this line gets a room that hears a voice in all of it and is untouched.
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def talking(monkeypatch):
|
||||
monkeypatch.setattr(
|
||||
vad, "get_vad_model",
|
||||
lambda: lambda heard: numpy.ones((len(heard) // vad.STEP, 1)),
|
||||
)
|
||||
|
||||
|
||||
class _Room:
|
||||
"""silero's shape: one probability per window, read from how loud that window
|
||||
is — so a test writes speech and silence as loud and quiet audio, and the
|
||||
machine that decides between them is the real one."""
|
||||
|
||||
def __call__(self, heard):
|
||||
return numpy.abs(heard.reshape(-1, vad.STEP)).max(axis=1).reshape(-1, 1)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def room(monkeypatch):
|
||||
monkeypatch.setattr(vad, "get_vad_model", _Room)
|
||||
|
||||
|
||||
def loud(seconds: float) -> bytes:
|
||||
return tone(30000, seconds) # 0.92 of full scale
|
||||
|
||||
|
||||
def quiet(seconds: float) -> bytes:
|
||||
return tone(3, seconds) # 0.0001 — a room with nobody in it
|
||||
|
||||
|
||||
def test_a_quiet_room_costs_nothing(ear, room):
|
||||
"""The whole point. Silence decodes to "" for the same CPU and the same money
|
||||
as speech, so an attendant left in an empty room bills a tenant continuously
|
||||
for nothing. Dropped audio must reach neither the decoder nor the meter."""
|
||||
live = transcript.begin("whisper", None)
|
||||
billed = sum(live.push(quiet(0.25)) for _ in range(12)) # 3 s of nobody
|
||||
|
||||
assert billed == 0.0, "silence must not be billed"
|
||||
assert live.seconds == 0.0
|
||||
assert ear.spans == [], "and must not reach the decoder at all"
|
||||
assert len(live._window) == 0
|
||||
|
||||
|
||||
def test_a_voice_is_decoded(ear, room):
|
||||
"""The other direction, which is what makes the test above mean anything: a
|
||||
squelch that dropped everything would satisfy that one perfectly."""
|
||||
live = transcript.begin("whisper", None)
|
||||
billed = sum(live.push(loud(0.25)) for _ in range(8))
|
||||
settle(live)
|
||||
|
||||
assert billed == pytest.approx(2.0)
|
||||
assert ear.spans, "speech must reach the decoder"
|
||||
|
||||
|
||||
def test_the_onset_that_opened_the_squelch_is_decoded_with_it(ear, room):
|
||||
"""The classic failure, and the one that costs more than it saves: a voice is
|
||||
not detected until it is already under way, so a squelch that begins at the
|
||||
trigger has already eaten the first syllable. The audio from before the
|
||||
trigger is kept, and handed over by the push that opens."""
|
||||
live = transcript.begin("whisper", None)
|
||||
for _ in range(8):
|
||||
assert live.push(quiet(0.25)) == 0.0
|
||||
|
||||
opened = live.push(loud(0.25))
|
||||
|
||||
assert opened > 0.25, "the push that opens carries more than itself"
|
||||
assert bytes(live._window[-len(loud(0.25)):]) == loud(0.25)
|
||||
before = bytes(live._window[:-len(loud(0.25))])
|
||||
assert before == quiet(len(before) / stt.SECOND), "and what precedes it is what preceded"
|
||||
assert len(before) == int(vad.LEAD * stt.SECOND), "as much of it as LEAD says"
|
||||
assert opened == pytest.approx(len(live._window) / stt.SECOND), "all of it billed once"
|
||||
|
||||
|
||||
def test_a_pause_inside_a_sentence_does_not_end_it(ear, room):
|
||||
"""Speech is not continuous: there is a gap between words and a breath
|
||||
between clauses. Closing on the first quiet window would chop a sentence into
|
||||
fragments and clip the word after every pause."""
|
||||
live = transcript.begin("whisper", None)
|
||||
live.push(loud(0.25))
|
||||
gap = [live.push(quiet(0.25)) for _ in range(2)] # half a second
|
||||
after = live.push(loud(0.25))
|
||||
|
||||
assert gap == [0.25, 0.25], "a pause this short is inside the sentence"
|
||||
assert after == 0.25, "so the word after it needs no lead-in — it never closed"
|
||||
|
||||
|
||||
def test_the_squelch_closes_once_the_room_stays_quiet(ear, room):
|
||||
"""And it must actually close, or the saving is only ever deferred. The
|
||||
hangover is spent — HANG seconds of quiet decode — and then nothing does."""
|
||||
live = transcript.begin("whisper", None)
|
||||
live.push(loud(0.25))
|
||||
after = [live.push(quiet(0.25)) for _ in range(8)] # 2 s of nobody
|
||||
|
||||
spent = sum(after)
|
||||
assert spent == pytest.approx(0.75), "the hangover, and no more than it"
|
||||
assert spent <= vad.HANG
|
||||
assert after[3:] == [0.0] * 5, "quiet from there on is free"
|
||||
|
||||
|
||||
def test_a_session_never_bills_a_second_it_did_not_decode(ear, room):
|
||||
"""The invariant that makes the lead-in safe to charge for: the push that
|
||||
opens reports more than its own length, so the arithmetic has to hold over
|
||||
the session rather than over one call. Every second billed is a second a
|
||||
decoder read, and no second is billed twice."""
|
||||
live = transcript.begin("whisper", None)
|
||||
sent = billed = 0.0
|
||||
for piece in (quiet(1.0), loud(1.0), quiet(1.0), loud(1.0), quiet(2.0)):
|
||||
for at in range(0, len(piece), transcript.CHUNK):
|
||||
billed += live.push(piece[at:at + transcript.CHUNK])
|
||||
sent += len(piece[at:at + transcript.CHUNK]) / stt.SECOND
|
||||
|
||||
assert billed == pytest.approx(live.seconds), "the acks and the session agree"
|
||||
assert live.seconds == pytest.approx(live._got / stt.SECOND), "billed is decoded"
|
||||
assert billed < sent, "and the quiet room was not charged for"
|
||||
|
||||
|
||||
class _Level:
|
||||
"""A room at one stated probability — it reads the level, not the audio — so
|
||||
a test can write the window silero is least sure about."""
|
||||
|
||||
def __init__(self):
|
||||
self.p = 0.0
|
||||
|
||||
def __call__(self, heard):
|
||||
return numpy.full((len(heard) // vad.STEP, 1), self.p)
|
||||
|
||||
|
||||
def test_a_window_the_model_is_unsure_of_holds_its_ground(monkeypatch):
|
||||
"""Between the two thresholds silero is not sure, and one threshold cannot
|
||||
express that: the same probability must continue speech without being able to
|
||||
begin it. With one threshold, a syllable fading through it ends the sentence
|
||||
and a room's own hum starts one."""
|
||||
level = _Level()
|
||||
monkeypatch.setattr(vad, "get_vad_model", lambda: level)
|
||||
squelch = vad.Squelch()
|
||||
|
||||
level.p = (vad.OPEN + vad.SHUT) / 2
|
||||
assert squelch.admit(tone(1, 0.25)) == b"", "unsure does not begin speech"
|
||||
|
||||
level.p = vad.OPEN
|
||||
assert squelch.admit(tone(1, 0.25)), "certain does"
|
||||
|
||||
level.p = (vad.OPEN + vad.SHUT) / 2
|
||||
assert squelch.admit(tone(1, 2.0)), "and unsure does not end it, however long"
|
||||
|
||||
|
||||
def test_the_ack_reports_zero_for_silence(ear, room):
|
||||
"""`duration` on the ack is the field the ai plane meters. Whatever the
|
||||
session believes, this is the number that becomes money."""
|
||||
live = start()
|
||||
r = client.post(f"/v1/audio/transcript/{live['id']}", content=quiet(0.25))
|
||||
assert r.status_code == 200, r.text
|
||||
assert r.json()["duration"] == 0.0
|
||||
assert r.json()["seconds"] == 0.0
|
||||
|
||||
|
||||
|
||||
+29
-12
@@ -13,9 +13,13 @@ latency lands on a later ack instead of on this one. One decode runs per session
|
||||
at a time, which is also what stops a fast pusher from queueing work faster than
|
||||
the CPU retires it.
|
||||
|
||||
`seconds` is arithmetic on the bytes received, so it is the audio submitted,
|
||||
exactly, and it cannot silently become zero: it is the same quantity `transcribe`
|
||||
reports as `duration`, measured the same way, and it is what the call is billed on.
|
||||
Audio reaches the window through a squelch, so a quiet room reaches nothing at
|
||||
all: no bytes, no decode, no charge. `seconds` is arithmetic on the bytes that
|
||||
got through, and those are exactly the bytes a decoder read — so the bill is the
|
||||
audio we listened to. It is the same quantity `transcribe` reports as `duration`,
|
||||
measured the same way, it can never exceed the audio submitted, and it cannot
|
||||
silently become zero for audio that WAS decoded, because it is the length of the
|
||||
decoded bytes and nothing else.
|
||||
"""
|
||||
|
||||
import logging
|
||||
@@ -25,6 +29,7 @@ import time
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
|
||||
import stt
|
||||
import vad
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
@@ -64,9 +69,10 @@ class Transcript:
|
||||
self.language = language
|
||||
self.text = "" # committed: settled, never revised
|
||||
self.pending = "" # the open tail: newest decode, may still change
|
||||
self.seconds = 0.0 # audio received — the billable quantity
|
||||
self.seconds = 0.0 # audio decoded — the billable quantity
|
||||
self._window = bytearray()
|
||||
self._got = 0 # bytes ever received; marks progress for the worker
|
||||
self._voice = vad.Squelch()
|
||||
self._lock = threading.Lock()
|
||||
self._decoding = False
|
||||
self._closed = False
|
||||
@@ -80,23 +86,34 @@ class Transcript:
|
||||
return now - self._touched > IDLE
|
||||
|
||||
def push(self, pcm: bytes) -> float:
|
||||
"""Accept audio; return the seconds THIS push carried.
|
||||
"""Accept audio; return the seconds THIS push put in front of a decoder.
|
||||
|
||||
The return value is the metered quantity for this call — per push, not
|
||||
cumulative, so summing the pushes of a session yields the audio submitted
|
||||
once and only once.
|
||||
That is the metered quantity for this call — per push, not cumulative, so
|
||||
summing the pushes of a session yields the audio decoded once and only
|
||||
once. A push the squelch drops returns 0.0: it was never decoded, so
|
||||
there is nothing to charge for. A push that OPENS the squelch carries the
|
||||
lead-in with it and so reports more than its own length; those seconds
|
||||
were withheld from every earlier push, so the session still bills each
|
||||
second at most once and never bills one it did not decode.
|
||||
|
||||
The squelch runs under the lock because its memory of the room is part of
|
||||
the session's state, and it is a millisecond of arithmetic — no decode
|
||||
ever holds this lock.
|
||||
"""
|
||||
with self._lock:
|
||||
self._window += pcm
|
||||
self._got += len(pcm)
|
||||
self.seconds += len(pcm) / stt.SECOND
|
||||
self._touched = time.monotonic()
|
||||
heard = self._voice.admit(pcm)
|
||||
if not heard:
|
||||
return 0.0
|
||||
self._window += heard
|
||||
self._got += len(heard)
|
||||
self.seconds += len(heard) / stt.SECOND
|
||||
start = not self._decoding and len(self._window) >= FLOOR * stt.SECOND
|
||||
if start:
|
||||
self._decoding = True
|
||||
if start:
|
||||
_pool.submit(self._work)
|
||||
return len(pcm) / stt.SECOND
|
||||
return len(heard) / stt.SECOND
|
||||
|
||||
def close(self) -> None:
|
||||
"""Decode what is left and commit all of it — nothing follows, so nothing
|
||||
|
||||
@@ -0,0 +1,102 @@
|
||||
"""Voice activity: the audio worth decoding, and nothing else.
|
||||
|
||||
Silence costs a decoder exactly what speech costs it. A second of an empty room
|
||||
occupies a worker for a second and bills a second, and returns "". A meeting is
|
||||
mostly that — the pauses inside one person's turn, and the longer stretches
|
||||
where nobody has the floor at all — so the cheapest decode is the one that never
|
||||
runs.
|
||||
|
||||
Silero says how much each 32 ms of audio sounds like a voice. faster-whisper
|
||||
already carries that model and already runs it inside every decode, so this
|
||||
costs no new weight and no new bake step; what is new is running it BEFORE the
|
||||
decoder instead of inside it, where the answer can save the decode rather than
|
||||
merely trim its input.
|
||||
|
||||
A probability is not yet a decision. Deciding per window clips speech at both
|
||||
ends, and a clipped first syllable costs more than the CPU it saved: a word that
|
||||
begins softly is not detected until it is already under way, and a breath inside
|
||||
a sentence is not the end of the sentence. So the decision is a squelch, in the
|
||||
sense radio has always meant — it opens on voice, closes only after a run of
|
||||
quiet, hands back the audio from just BEFORE it opened, and keeps passing for a
|
||||
moment after the voice stops.
|
||||
"""
|
||||
|
||||
import numpy
|
||||
from faster_whisper.vad import get_vad_model
|
||||
|
||||
import stt
|
||||
|
||||
# Samples silero scores at once: 32 ms at 16 kHz. Its own window, not a choice.
|
||||
STEP = 512
|
||||
|
||||
# Open loud, close quiet — the two thresholds are what stops the decision
|
||||
# chattering on a syllable's own envelope. Silero's own filter reads its
|
||||
# probabilities the same way, and this is the same distance apart.
|
||||
OPEN = 0.5
|
||||
SHUT = 0.35
|
||||
|
||||
# Quiet this long ends the utterance. Longer than a breath or the gap between
|
||||
# words, shorter than the pause between one person finishing and the next
|
||||
# starting — that is the whole span in which this number is the right one.
|
||||
HANG = 0.8
|
||||
|
||||
# Audio held back while quiet, and handed over on the push that opens. Whisper
|
||||
# reads a span, and the span it is given must contain the onset that triggered
|
||||
# the decision — the trigger necessarily lags the onset, so the onset is in the
|
||||
# past by the time anything knows to keep it.
|
||||
LEAD = 0.5
|
||||
|
||||
|
||||
class Squelch:
|
||||
"""Passes voice, drops silence. One per session: it remembers.
|
||||
|
||||
A push is admitted whole or not at all. Cutting on the sample would splice
|
||||
the window at word boundaries and hand the decoder audio with steps in it;
|
||||
admitting whole pushes keeps what the decoder reads continuous, and LEAD
|
||||
covers the onset far better than a sample-exact edge would.
|
||||
"""
|
||||
|
||||
def __init__(self):
|
||||
self._open = False
|
||||
self._quiet = 0.0
|
||||
self._lead = b""
|
||||
|
||||
def admit(self, pcm: bytes) -> bytes:
|
||||
"""Return the audio worth decoding: b"" while the room is quiet, the push
|
||||
when it is not, and the lead-in as well on the push that opens.
|
||||
|
||||
A push carrying no audio carries no decision either, and silero cannot be
|
||||
asked about nothing — it indexes the last window of what it is given.
|
||||
"""
|
||||
if not pcm:
|
||||
return b""
|
||||
|
||||
heard = False
|
||||
for voiced in self._voiced(pcm):
|
||||
if voiced >= OPEN:
|
||||
self._open, self._quiet = True, 0.0
|
||||
elif self._open:
|
||||
self._quiet = self._quiet + STEP / stt.RATE if voiced < SHUT else 0.0
|
||||
self._open = self._quiet < HANG
|
||||
heard = heard or self._open
|
||||
|
||||
if not heard:
|
||||
# Sliced from the front, because the end of a buffer is counted from
|
||||
# the end and `[-0:]` is the whole of it: written the other way, a
|
||||
# LEAD of zero keeps every second of silence instead of none, and
|
||||
# hands the lot to the decoder on the next word.
|
||||
held = self._lead + pcm
|
||||
self._lead = held[max(0, len(held) - int(LEAD * stt.SECOND)):]
|
||||
return b""
|
||||
was, self._lead = self._lead, b""
|
||||
return was + pcm
|
||||
|
||||
def _voiced(self, pcm: bytes) -> numpy.ndarray:
|
||||
"""One probability per STEP samples. The model wants whole windows, so a
|
||||
push that is not a whole number of them is padded — a 250 ms push at this
|
||||
rate is exactly eight, and pads not at all."""
|
||||
heard = stt.samples(pcm)
|
||||
short = -len(heard) % STEP
|
||||
if short:
|
||||
heard = numpy.pad(heard, (0, short))
|
||||
return get_vad_model()(heard).reshape(-1)
|
||||
Reference in New Issue
Block a user