mirror of
https://github.com/zap-proto/go.git
synced 2026-10-30 15:05:00 +00:00
zooqueenandHanzo Dev
a5354a5b54
Inbound stream frames were handed to a fixed 16-slot channel with a BLOCKING
send, executed on muxConn.readLoop. That read loop carries every frame on the
connection — other streams' frames and every unary response — so a consumer
slower than its producer filled the buffer, blocked the loop, and froze
everything else sharing that connection.
It is worse than throughput loss: it deadlocks. A stream consumer whose own
next call rides the same connection can never receive that reply, so it can
never drain, so the loop never resumes.
transport.Pool hands out one connection per peer address, so this reached
production as a total outage rather than a slow path: in hanzoai/s3 a single
large ListEntries on the shared filer connection hung every S3 request in the
process. It presented as an auth failure, because unauthenticated requests are
rejected at the front door and never touch the filer, so only signed requests
appeared to hang.
Frames are now appended to a per-stream queue and the reader is woken; delivery
never blocks. Ordering is preserved, and maxStreamQueue (8192) bounds what one
runaway producer can pin — overflowing fails THAT stream with
ErrStreamOverflow instead of the connection everyone shares. Recv drains what
is already queued before reporting overflow or EOF, so no delivered message is
lost to the end condition.
Regression tests verified failing before this change ("unary call starved by
the slow stream after 3.0s") and passing after, under -race.
Co-authored-by: Hanzo Dev <[email protected]>