Say where the sustained-rate number comes from

The service claimed a throughput without naming the instrument that produced
it. bench.py is that instrument, and the note now records what it measures —
a corpus of real speech with a word-to-time map, the cost of one decode pass,
and N concurrent talkers — so the number can be reproduced or argued with
rather than taken on faith.

Co-authored-by: Hanzo Dev <[email protected]>

CLAUDE.md

@@ -59,6 +59,101 @@ belongs in `ai` (which already does provider routing), not here — this service
 decodes, it does not route. Until that exists the endpoint is not usable through
 the plane, and `ai` has no route for it yet either.
 
+## What it sustains, and where that number comes from
+
+`bench.py` is the instrument: `say` once for a corpus of real speech carrying a
+word-to-time map, then `cost` for what a decode pass costs and `talk` to drive N
+sessions in real time. Everything below was measured with it **on a four vCPU
+DigitalOcean worker** — the shape a speech pod runs in. That qualifier is the
+whole point: the same code and the same run on four threads of a desktop Zen 5
+retires 13.7 audio-seconds per wall-second where the pod retires 5.7. A capacity
+number measured on the box is 2.4x the truth.
+
+**Ack latency cannot see any of this and none of these numbers is one.** A push
+returns whatever text the session has, so it costs a memcpy whether the decoder
+is idle or minutes behind — 3.8 ms while the backlog grew fourfold. The metric is
+backlog, as lag: wall seconds elapsed minus seconds of audio already transcribed.
+The verdict is its SLOPE over the second half of a run, in seconds of lag added
+per minute of meeting. Runs must be MINUTES long: a stalling session looks fine
+for its first four minutes, which is how seconds-long runs came to predict eight
+concurrent meetings and were wrong by an order of magnitude.
+
+**A pass costs what it costs, almost regardless of what it covers.** Whisper
+always encodes a thirty second mel window, so on the pod one pass takes 2.7 s over
+a one second window and 3.2 s over an eight second one — about 8.4 CPU-seconds
+either way. Capacity is a budget of PASSES, not of audio, and the way to buy it is
+to pass less often, never to pass over less.
+
+**Past the mel window that cost multiplies.** 24 s -> 5.0 s per pass, 30 s -> 7.5,
+45 s -> 23.5. It is a cliff, not a slope.
+
+**So the pod has a throughput PEAK, and the peak is the instability.** In steady
+state a session gets one pass per interval and that pass covers the audio that
+arrived during it, so audio retired per wall second IS the number of real-time
+streams served, and lag is roughly the window. Measured on the pod:
+
+    window            8 s   12 s   16 s   24 s   30 s
+    streams served    3.1    4.4    5.0    3.9    2.8
+
+Below the peak the loop is self-correcting: a longer window is cheaper per second,
+so lag comes back. Past it, a longer window is DEARER per second, so lag grows,
+which lengthens the window — runaway. That is why the failure is a cliff and not a
+gentle degradation, and why the working point has to sit on the rising branch
+rather than at the top of the curve.
+
+**Measured, in eight minute runs at 250 ms chunks**, and the two agree to within a
+few percent — a session settles where its window buys exactly its own concurrency:
+
+    sessions   lag       slope         verdict
+    3          7.7 s     -2.36 s/min   settles
+    4          12.7 s    -0.45 s/min   settles
+
+So: **four concurrent streams of continuous speech per four vCPU pod**, with the
+transcript about thirteen seconds behind, or three at eight seconds. Two replicas
+is eight. The curve says five sits ON the peak and six has no fixed point at all.
+
+The failure past that edge is not a gentle one. The same runs against the earlier
+two-worker build, which peaked lower, settled at one and two sessions and at FOUR
+ran away to 326 s of lag at +44 s/min — and closing those sessions then blocked for
+400 s each, because `DELETE` decodes everything that piled up.
+
+**A session's ceiling is `LIMIT` = 600 s, so a 45 minute meeting is five sessions,
+not one.** Closing decodes the whole remaining window and blocks, and the transport
+admits one outstanding chunk, so the roll costs the microphone that whole pause.
+At three streams that is a couple of seconds every ten minutes; on a session that
+has fallen behind it is minutes.
+
+### What more of it costs
+
+Per pod, three streams. Per stream of continuous speech that is 1.33 vCPU. So:
+
+    concurrent streams     pods (4 vCPU)     vCPU
+    10                     4                 16
+    100                    34                136
+    1000                   334               1336
+
+Per-participant streams are nearly free next to that, because the squelch means a
+quiet participant is never decoded: about 0.27% of a core for the Silero pass on
+250 ms of quiet. A thousand five-person meetings is ~1000 voiced streams plus
+~13 cores of squelch, not 5000 streams.
+
+Buying 334 four-core nodes is the wrong answer, so state the alternatives with the
+number each has to beat.
+
+**Batching is not the lever, and that is measured, not assumed.** The fixed thirty
+second encode looks exactly like the shape that amortizes, and faster-whisper
+carries `BatchedInferencePipeline`, but on CPU it does not amortize: eight 12 s
+windows took 11.64 s sequentially and 6.79 s batched — 1.71x the wall clock for
+**the same CPU** (24.1 vs 23.6 CPU-seconds). Batching overlaps the work, it does
+not reduce it, so it helps a pod whose cores are idle and does nothing for one
+that is already saturated. Where it would pay is a GPU, where the fixed cost is
+underused hardware rather than arithmetic.
+
+A GPU has to retire 1000 audio-seconds per wall second of distil-small.en over
+~16 s windows to replace those 334 pods. **No GPU figure here is measured, the
+cluster has no GPU node, and every large node is tainted for CI**, so that is a
+procurement decision to be measured on the card and not believed before it is.
+
 - Weights bake at image BUILD (deterministic image, boots offline). Adding a
   model = extend `MODELS` in stt.py/tts.py AND the bake step in the Dockerfile.
 - **There is no fast tier to add: `whisper` already IS the fast one.** Measured