mirror of
https://github.com/hanzoai/papers.git
synced 2026-10-29 07:34:31 +00:00
101 lines
3.2 KiB
TeX
101 lines
3.2 KiB
TeX
\section{Production Deployment}
|
|
|
|
\label{sec:production}
|
|
|
|
\subsection{Deployment Infrastructure}
|
|
|
|
Hanzo Chat has been deployed in production since August 2024, serving users through \texttt{chat.hanzo.ai}. The infrastructure comprises:
|
|
|
|
\begin{itemize}[leftmargin=1.1em]
|
|
\item \textbf{Frontend}: Next.js application deployed on Hanzo Platform with CDN edge caching for static assets.
|
|
\item \textbf{Backend}: Node.js API server with WebSocket support for streaming, running on Kubernetes (3 replicas, auto-scaling to 12).
|
|
\item \textbf{Memory store}: PostgreSQL with pgvector extension, 500GB SSD, WAL replication.
|
|
\item \textbf{LLM Gateway}: Hanzo LLM Gateway proxying to 12 providers with automatic failover.
|
|
\item \textbf{Tool servers}: 8 MCP server processes handling tool execution in Docker containers.
|
|
\end{itemize}
|
|
|
|
\subsection{Usage Statistics}
|
|
|
|
Over 18 months of production operation (August 2024 -- February 2026):
|
|
|
|
\begin{table}[H]
|
|
\centering
|
|
\begin{tabular}{lr}
|
|
\toprule
|
|
\textbf{Metric} & \textbf{Value} \\
|
|
\midrule
|
|
Total conversations & 523,847 \\
|
|
Total messages & 4,891,203 \\
|
|
Avg. turns per conversation & 9.3 \\
|
|
Avg. tokens per message & 847 \\
|
|
Tool calls executed & 1,247,891 \\
|
|
Memory entries stored & 3,214,567 \\
|
|
Unique users & 42,318 \\
|
|
P99 response latency & 4.8s \\
|
|
Uptime & 99.94\% \\
|
|
\bottomrule
|
|
\end{tabular}
|
|
\caption{Production usage statistics (Aug 2024 -- Feb 2026).}
|
|
\label{tab:production}
|
|
\end{table}
|
|
|
|
\subsection{Model Selection Patterns}
|
|
|
|
Analysis of production model routing reveals interesting patterns:
|
|
|
|
\begin{table}[H]
|
|
\centering
|
|
\small
|
|
\begin{tabular}{lccc}
|
|
\toprule
|
|
\textbf{Model} & \textbf{Selection \%} & \textbf{Satisfaction} & \textbf{Cost/q} \\
|
|
\midrule
|
|
Claude 3.5 Sonnet & 34.2\% & 8.9/10 & \$0.021 \\
|
|
GPT-4o & 22.1\% & 8.7/10 & \$0.028 \\
|
|
Claude Opus 4 & 12.8\% & 9.2/10 & \$0.067 \\
|
|
Gemini 2 Flash & 11.4\% & 8.1/10 & \$0.004 \\
|
|
Zen 30B & 8.9\% & 8.4/10 & \$0.008 \\
|
|
Other (10+ models) & 10.6\% & 7.9/10 & \$0.012 \\
|
|
\bottomrule
|
|
\end{tabular}
|
|
\caption{Model selection distribution in production.}
|
|
\label{tab:model_selection}
|
|
\end{table}
|
|
|
|
\subsection{Memory Utilization}
|
|
|
|
Memory retrieval significantly improves response quality in production:
|
|
|
|
\begin{itemize}[leftmargin=1.1em]
|
|
\item 78\% of conversations beyond 5 turns benefit from memory retrieval (at least one retrieved memory used in response).
|
|
\item Average retrieval latency: 23ms (pgvector HNSW index).
|
|
\item Memory hit rate: 62\% (fraction of queries where at least one memory exceeds the relevance threshold).
|
|
\item User-reported improvement: 31\% of users cite ``it remembers what I told it'' as a key differentiator.
|
|
\end{itemize}
|
|
|
|
\subsection{Tool Usage Analysis}
|
|
|
|
The most frequently invoked tools in production:
|
|
|
|
\begin{table}[H]
|
|
\centering
|
|
\small
|
|
\begin{tabular}{lcc}
|
|
\toprule
|
|
\textbf{Tool} & \textbf{Calls/day} & \textbf{Success \%} \\
|
|
\midrule
|
|
Code execution (Python) & 3,247 & 94.1\% \\
|
|
Web search & 2,891 & 97.3\% \\
|
|
File read & 2,456 & 99.8\% \\
|
|
File write & 1,823 & 98.9\% \\
|
|
Git operations & 1,234 & 96.2\% \\
|
|
API calls & 987 & 91.4\% \\
|
|
Database queries & 654 & 93.7\% \\
|
|
Image generation & 432 & 88.9\% \\
|
|
\bottomrule
|
|
\end{tabular}
|
|
\caption{Top tool usage in production (daily averages).}
|
|
\label{tab:tool_usage}
|
|
\end{table}
|
|
|