mirror of
https://github.com/hanzoai/insights.git
synced 2026-10-30 16:15:31 +00:00
zeekay
04aac6bad0
A streamed reply is two scarce things at once — money at the gateway, and a request-serving worker held for its whole duration — and neither was bounded. Spend. The global throttle cannot cover this: rate_limit.py returns True for any authenticated request without a personal API key, which is every request a browser session makes, so this endpoint was never throttled at all. The path was cheap to walk: open a thread, append twenty 40 000-character messages for free (append_message calls no model), then one short message replays the lot into a priced context window, repeatably. Now: per-user burst and daily throttles that actually apply, a cap on messages per thread, a character budget on what is replayed (a forty-turn window is not a bound when a turn may be 40 000 characters), and max_tokens on the reply, which was unlimited. Client-written assistant turns are no longer replayed as the model's own words. append_message stores CLIENT-sourced messages; only MODEL-sourced replies go back into context. That was the free-to-write primitive behind the spend path, and separately it let a caller put words in the assistant's mouth and have them treated as its prior reasoning. Occupancy. Unit gives Python one thread per process unless told otherwise and the deployed CR sets one process, so the whole of Django was two concurrent requests across two replicas — one chat could freeze login and the SPA shell. unit.json.tpl now takes NGINX_UNIT_APP_THREADS (default 4); these handlers wait on the network and release the GIL while they do. Alongside that, replies are capped per process and answer 429 past the cap, a reply has a wall-clock deadline, and Stop now works: cancel marks the thread CANCELING and the generator notices between deltas, keeps what it has and settles. Previously cancel only wrote idle and left the upstream call running. Also: bound the history response, which returned every message of every thread; validate trace_id against its column, so an over-long one is a 400 rather than a DataError raised after the headers are on the wire; and re-resolve rather than 500 when two requests race to open the same client-minted id.
76 lines
3.1 KiB
Bash
Executable File
76 lines
3.1 KiB
Bash
Executable File
#!/bin/bash
|
|
set -e
|
|
|
|
./bin/migrate-check
|
|
|
|
# To ensure we are able to expose metrics from multiple processes, we need to
|
|
# provide a directory for `prometheus_client` to store a shared registry.
|
|
export PROMETHEUS_MULTIPROC_DIR=$(mktemp -d)
|
|
chmod -R 777 $PROMETHEUS_MULTIPROC_DIR
|
|
|
|
export PROMETHEUS_METRICS_EXPORT_PORT=8001
|
|
export STATSD_PORT=${STATSD_PORT:-8125}
|
|
|
|
# Dual-mode support: USE_GRANIAN env var switches between Granian and Unit (default)
|
|
if [ "${USE_GRANIAN:-false}" = "true" ]; then
|
|
echo "🚀 Starting with Granian ASGI server (opt-in via USE_GRANIAN=true)..."
|
|
|
|
# Granian configuration
|
|
# Match Unit's 4 workers for equivalent sync view concurrency.
|
|
# With Django ASGI + sync views (thread_sensitive=True), each worker
|
|
# handles 1 sync request at a time - only workers give concurrency.
|
|
export GRANIAN_WORKERS=${GRANIAN_WORKERS:-4}
|
|
export GRANIAN_THREADS=${GRANIAN_THREADS:-1}
|
|
|
|
# HTTP/1 keep-alive: disable to improve SO_REUSEPORT load distribution
|
|
KEEPALIVE_FLAG=""
|
|
if [ "${GRANIAN_HTTP1_KEEP_ALIVE:-true}" = "false" ]; then
|
|
KEEPALIVE_FLAG="--no-http1-keep-alive"
|
|
fi
|
|
|
|
# Configurable log-level for Granian
|
|
export GRANIAN_LOG_LEVEL=${GRANIAN_LOG_LEVEL:-warning}
|
|
|
|
# Start metrics HTTP server in background on port 8001
|
|
python ./bin/granian_metrics.py &
|
|
METRICS_PID=$!
|
|
|
|
# Combined trap: kill metrics server and cleanup temp directory
|
|
trap 'kill $METRICS_PID 2>/dev/null; rm -rf "$PROMETHEUS_MULTIPROC_DIR"' EXIT
|
|
|
|
exec granian \
|
|
--interface asgi \
|
|
insights.asgi:application \
|
|
--workers $GRANIAN_WORKERS \
|
|
--runtime-threads $GRANIAN_THREADS \
|
|
$KEEPALIVE_FLAG \
|
|
--loop uvloop \
|
|
--host 0.0.0.0 \
|
|
--port 8000 \
|
|
--log-level $GRANIAN_LOG_LEVEL \
|
|
--access-log \
|
|
--respawn-failed-workers
|
|
else
|
|
echo "🔧 Starting with Nginx Unit server (default, stable)..."
|
|
|
|
# Cleanup temp directory on exit
|
|
trap 'rm -rf "$PROMETHEUS_MULTIPROC_DIR"' EXIT
|
|
|
|
export NGINX_UNIT_PYTHON_PROTOCOL=${NGINX_UNIT_PYTHON_PROTOCOL:-wsgi}
|
|
export NGINX_UNIT_APP_PROCESSES=${NGINX_UNIT_APP_PROCESSES:-4}
|
|
# Unit defaults Python to ONE thread per process, and the deployed CR sets
|
|
# processes to 1 — which made the whole of Django two concurrent requests
|
|
# across two replicas. Any slow request then blocks login and the SPA shell,
|
|
# not just itself; a streamed assistant reply is merely the longest one.
|
|
# Threads are the right unit here because these handlers wait on the network
|
|
# and the database, and release the GIL while they do.
|
|
export NGINX_UNIT_APP_THREADS=${NGINX_UNIT_APP_THREADS:-4}
|
|
export NGINX_UNIT_REQUEST_LIMIT=${NGINX_UNIT_REQUEST_LIMIT:-7500}
|
|
envsubst < /docker-entrypoint.d/unit.json.tpl > /docker-entrypoint.d/unit.json
|
|
|
|
# We need to run as root so that nginx unit can proxy the control socket for stats
|
|
# However each application is run as "nobody"
|
|
# nosemgrep: trailofbits.generic.container-user-root.container-user-root
|
|
exec /usr/local/bin/docker-entrypoint.sh unitd --no-daemon --user root
|
|
fi
|