mirror of
https://github.com/hanzoai/ui.git
synced 2026-11-20 22:36:27 +00:00
8.3 KiB
8.3 KiB
@hanzo/voice
One spoken conversation, shared by every Hanzo composer. See README.md for the
API; this file carries only what the source cannot say for itself.
src/voice.tsis the machine,src/control.tsxis the button. The caller owns the machine because it needssay(); the button only draws it. There is no second copy of this state anywhere.onUtteranceMUST be wired to the surface's existing submit path, or the composer dictates throughuseDictation(src/dictation.ts) and the person sends. Voice is a way of typing, not a second way of sending. Dictation that sets the field to each utterance loses every sentence but the last.- Listening has two legs (
src/listen.ts).listen()ranks them and opens the first that exists: the platform's (record +/v1/audio/transcriptions) unless the caller passesprefer: "browser". Until 0.1.6 the rank wascap.recognition ? recogniser : recorder, which meant Chrome, Edge and Safari — almost everybody — never reached Hanzo's transcriber no matter how the surface was configured.preferis a property of the caller's configuration, not an env flag; there is exactly one place the choice is made. - The platform legs give up on first refusal and the browser takes over from
that moment (
goneinsrc/listen.tsandsrc/speak.ts): every later turn would refuse identically, one round trip at a time.listen()decides whether a standby exists and publishesRefusal.covered; the legs only report the transport error upward. - A refusal is REPORTED —
onRefusal,voice.refusal, the button's label anddata-refusal. Silently standing in is the failure shape this package is most exposed to: a browser voice reading a reply sounds exactly like a funded key, and the only other symptom is a bill that reads zero. - Both legs hold an echo-cancelled
getUserMediastream while open. That is the only thing keeping barge-in from firing on the reply's own audio. liveinsrc/voice.tsis module state ON PURPOSE: an open microphone is a fact about the page, and a composer remount (the/c/new->/c/<id>swap a chat surface performs on its FIRST turn) must not end the conversation. Measured on hanzo.chat: that navigation replaces the composer's DOM node and aborts the recogniser. Tests thereforevi.resetModules()per case — a fresh module is a fresh page.- The platform mouth keeps the browser mouth BEHIND it (
src/speak.ts): a refused or unfunded speech service degrades the VOICE, never the conversation. Measured 2026-07-26: api.hanzo.ai/v1/audio/* resolves its caller through IAM — an hk- accessKey passes (402 on balance), an sk- gateway key and a hanzo.idaud: hanzo-appbearer are both 401. hanzo.app's proxy is wired and correct; it upgrades to the platform voice the day the gateway accepts that audience. speech()defaults name Hanzo's own models:zen-scribelistens,zen-voice-minireads, inaf_heart. Voices are the service's own ids, and our clients send those ids, never the OpenAI names the server aliases.- The recorder leg sends a recording only if somebody spoke in it, and starts a
silent one over after
LEAD. A transcriber handed silence returns invented words ("Thank you."), and a closing mic always flushes a tail. - The platform mouth makes and
load()s its one<audio>whenmouth()is called.useVoicecalls it inside the mic click; a surface reading a reply on a click calls it inside that click. Safari refuses sound from an element first touched after the gesture, and a refusedplay()is otherwise silent. - Consumers:
hanzoai/chat(client/src/components/Chat/Input/Mic.tsx) andhanzoai/app(components/editor/ask-ai/mic.ts+ the console bar and the landing composer). All four local mics were deleted on adoption. - Published from
hanzoai/ui(pkg/voice) by the monorepo's publish lane, which readsNPM_TOKENfrom KMS like every other@hanzo/*package here. Bump the patch inpackage.jsonand pushmain; nothing is published by hand. - Talk mode is
useTalk, on/v1/voice(src/talk.ts): the OpenAI realtime wire served by the ai plane. The bearer goes onPOST /v1/voice/sessionand the socket URL carries only the one-use ticket it answers; the socket speaks subprotocolrealtime. Mic up as 16 kHz pcm16 in 100 msinput_audio_buffer .appendmessages, reply down as 24 kHz pcm16response.audio.deltawithresponse.audio_transcript.deltabeside it. Turn-taking is the SERVER's:input_audio_buffer.speech_startedis the barge-in, and the client only stops playback on it. Anerrorevent is one turn's, not the conversation's; a socket the server closes is a refusal. - The live transcript is
useTranscript, on/v1/audio/transcript(src/transcript.ts,speech().stream): POST opens, POST to the id carries raw pcm16, DELETE settles. The endpoint admits ONE push at a time, so audio heard during a push queues and rides the next (up tomax_bytes). A transcript takesmax_secondsof audio; at five seconds short it is closed and the next opened, and what it settled stays in front.settledis text that will not change;partialis everything, tail included. - A failed push is not a refusal (
backoffinsrc/transcript.ts). A 429 (one that names no spent allowance), a 5xx or a dropped connection is a wait: the same push goes again afterRetry-After(SpeechError.retry, ms) or a doubling backoff, while new audio queues behind it, so nothing is dropped or reordered. 402, 403 and every other 4xx refuse at once; so does a failure streak, or a queue, older thanRETRY_WINDOW(60 s, the gateway's rate window). Closing delivers the queue before the DELETE. No single wait outlasts the session'sidle_seconds(less five): a transcript untouched that long is dropped, so a longRetry-Afteris taken in steps, each retry keeping the session alive (measured: a 502 naming 60 s against a 30 s idle life came back to a 409, the session gone). - A session the platform loses is replaced, not refused (
gone: 404, or 409 for an id an earlier process minted — a cloud deploy restarts the one replica that holds every window). Its settled text and last tail are kept as settled, the push it never answered goes to a new session first, then the queue. More than three lost inRETRY_WINDOWis the platform gone and refuses. - A 429 the client gives up on is worded as a limit, never an outage:
refused()reads "Today's Hanzo dictation limit is reached. It resets at - The streaming legs make their AudioContext (
audioContext) at the top ofopen(), before the first await, so it is made inside the click: Safari runs only a context made or resumed in a gesture, and one made after the permission prompt taps and plays nothing. src/mic.tsis the one microphone:capture(the permission ask),tap(an inlined AudioWorklet averaging the context rate down to 16 kHz pcm16 — the average is the low-pass a plain decimation lacks) and the base64 codec. The tap feeds a zero gain into the destination because an unpulled node is not processed.<Voice/>draws any machine withstate/open/level/blocked/reason/refusal/ toggle(Machineinsrc/control.tsx), andsaysrelabels it, so a dictation mic, a talk button and a transcript toggle are one control.- The visitor's lane is
speech({ public: true }):transcribealone, POSTed to/v1/audio/transcriptions/publicwith no credential (credentials: "omit", no Authorization or X-Org-Id even when a token is given). The ai plane holds it to 60 s of audio and a day per visitor; the recorder leg sends a recording atLONGEST(55 s) even mid-sentence, so no upload passes the minute. A spent day isSpeechErrorcodepublic_allowance_spent, andrefused()names it (SPENT) instead of the genericREFUSEDnote. - The mic that is pressed takes the conversation (
togglemakes its composer the owner): a page composer and the shell footer bar mount together, and the words go where the click was, not to whichever mounted last.