mirror of
https://github.com/hanzoai/papers.git
synced 2026-10-22 17:20:17 +00:00
127 lines
3.1 KiB
TeX
127 lines
3.1 KiB
TeX
\section{Evaluation}
|
|
|
|
|
|
\subsection{Experimental Setup}
|
|
|
|
We evaluated Verus on production workloads from three merchants:
|
|
|
|
\begin{itemize}
|
|
\item \textbf{Small}: 10K events/day, 5 materialized views.
|
|
\item \textbf{Medium}: 1M events/day, 15 materialized views.
|
|
\item \textbf{Large}: 100M events/day, 30 materialized views.
|
|
\end{itemize}
|
|
|
|
Infrastructure:
|
|
\begin{itemize}
|
|
\item Stream processing: 3-node Flink cluster.
|
|
\item State store: 3-node Redis cluster.
|
|
\item Columnar storage: S3 + 10-node Presto cluster.
|
|
\end{itemize}
|
|
|
|
\subsection{Ingestion Throughput}
|
|
|
|
\begin{table}[h]
|
|
\centering
|
|
\caption{Sustained ingestion throughput}
|
|
\label{tab:throughput}
|
|
\begin{tabular}{lr}
|
|
\hline
|
|
Configuration & Events/second \\
|
|
\hline
|
|
Single partition & 8,500 \\
|
|
10 partitions & 52,000 \\
|
|
100 partitions & 485,000 \\
|
|
\hline
|
|
\end{tabular}
|
|
\end{table}
|
|
|
|
Throughput scales linearly with partitions. The large merchant's peak load (50K events/second) is well within capacity.
|
|
|
|
\subsection{Query Latency}
|
|
|
|
\begin{table}[h]
|
|
\centering
|
|
\caption{Query latency by type (ms)}
|
|
\label{tab:query_latency}
|
|
\begin{tabular}{lrrr}
|
|
\hline
|
|
Query Type & p50 & p95 & p99 \\
|
|
\hline
|
|
Dashboard (Redis) & 18 & 45 & 82 \\
|
|
Dashboard (merged) & 125 & 280 & 520 \\
|
|
Ad-hoc (simple) & 850 & 2,100 & 4,500 \\
|
|
Ad-hoc (complex) & 3,200 & 8,500 & 15,000 \\
|
|
Cohort analysis & 5,500 & 12,000 & 25,000 \\
|
|
\hline
|
|
\end{tabular}
|
|
\end{table}
|
|
|
|
Dashboard queries served from Redis meet interactive requirements (< 100ms p99). Ad-hoc queries trade latency for flexibility.
|
|
|
|
\subsection{Freshness}
|
|
|
|
We measured end-to-end latency from event occurrence to query visibility:
|
|
|
|
\begin{table}[h]
|
|
\centering
|
|
\caption{Data freshness (seconds)}
|
|
\label{tab:freshness}
|
|
\begin{tabular}{lrrr}
|
|
\hline
|
|
Metric & p50 & p95 & p99 \\
|
|
\hline
|
|
Materialized views & 1.2 & 2.8 & 5.1 \\
|
|
Columnar storage & 45 & 120 & 180 \\
|
|
\hline
|
|
\end{tabular}
|
|
\end{table}
|
|
|
|
Materialized views update within seconds. Columnar storage has higher latency due to micro-batching for write efficiency.
|
|
|
|
\subsection{Cost Analysis}
|
|
|
|
\begin{table}[h]
|
|
\centering
|
|
\caption{Monthly infrastructure cost (USD)}
|
|
\label{tab:cost}
|
|
\begin{tabular}{lrrr}
|
|
\hline
|
|
Component & Small & Medium & Large \\
|
|
\hline
|
|
Kafka & 50 & 150 & 800 \\
|
|
Stream processing & 100 & 400 & 2,500 \\
|
|
Redis & 75 & 250 & 1,200 \\
|
|
Storage (S3) & 5 & 50 & 500 \\
|
|
Query engine & 200 & 600 & 3,000 \\
|
|
\hline
|
|
Total & 430 & 1,450 & 8,000 \\
|
|
\hline
|
|
\end{tabular}
|
|
\end{table}
|
|
|
|
Cost scales sub-linearly with volume due to shared infrastructure and compression benefits at scale.
|
|
|
|
\subsection{Comparison with Alternatives}
|
|
|
|
We compared Verus against pure-batch (BigQuery) and pure-stream (custom Flink) approaches:
|
|
|
|
\begin{table}[h]
|
|
\centering
|
|
\caption{Approach comparison}
|
|
\label{tab:comparison}
|
|
\begin{tabular}{lccc}
|
|
\hline
|
|
Metric & BigQuery & Flink-only & Verus \\
|
|
\hline
|
|
Dashboard latency & 2--5s & 20ms & 20ms \\
|
|
Ad-hoc latency & 2--10s & N/A & 1--15s \\
|
|
Freshness & 1--4 hours & < 1s & < 5s \\
|
|
Query flexibility & High & Low & High \\
|
|
Operational complexity & Low & High & Medium \\
|
|
\hline
|
|
\end{tabular}
|
|
\end{table}
|
|
|
|
Verus achieves the freshness of stream processing with the flexibility of batch analytics.
|
|
|