mirror of
https://github.com/hanzoai/papers.git
synced 2026-10-22 17:20:17 +00:00
feat(vs-claude): develop the productivity comparison into a solid case study
- Full case-study arc: Background (regulated platform, 200+ microservices, $35K/mo, regulatory bar) -> Intervention (same Claude, different stack) -> Measured comparison (audit table, 250-900x) -> Outcome (before/after table: 200+->3 binaries, 3.07M->1.06M LOC, ~90% cost, 93 findings->0, FINRA/SEC+SOC2) -> mechanism -> switch case. - Proves the gap was delivered into a regulated, audited production result, not a demo.
This commit is contained in:
Binary file not shown.
Binary file not shown.
+62
-14
@@ -85,26 +85,37 @@ are mandatory for regulated, critical-infrastructure, and government workloads
|
||||
do not hold and cannot run yourself.
|
||||
|
||||
% ============================================================================
|
||||
\section{One Engineer on the Hanzo Stack vs.\ a Team on Claude}
|
||||
\section{Case Study: One Engineer on the Hanzo Stack vs.\ a Team on Claude}
|
||||
\label{sec:oneeng}
|
||||
% ============================================================================
|
||||
|
||||
The cleanest evidence for this paper's thesis is not a benchmark---it is a
|
||||
controlled natural experiment that already happened in production. On a regulated
|
||||
capital-markets platform, every engineer on the program had access to the same
|
||||
frontier chat model (Anthropic's Claude, via \texttt{claude.ai}). The variable
|
||||
that differed was the \emph{tooling around} that model: one engineer worked
|
||||
through Hanzo's owned, agentic stack (\texttt{hanzo.ai})---an orchestration layer
|
||||
that turns the model into an autonomous build-test-ship loop across hundreds of
|
||||
repositories---while the contractor teams used the model the conventional way, as
|
||||
an interactive assistant alongside hand-driven development.
|
||||
controlled natural experiment that already ran in production, on a real regulated
|
||||
system, and was measured by that system's own engineering audit.
|
||||
|
||||
Crucially, this is \emph{not} a claim that one group had a smarter model. They did
|
||||
not. Everyone held the same frontier weights. What separated the outcomes was
|
||||
whether the model was wrapped in an agentic stack or merely chatted with. The
|
||||
platform's own engineering audit measured the result.
|
||||
\subsection{Background: a regulated platform, a sprawling estate}
|
||||
The setting (anonymized) is a U.S.\ regulated securities venue---an SEC-registered
|
||||
alternative trading system with broker-dealer and transfer-agent functions, among
|
||||
the most compliance-heavy software domains that exists. By the time of the
|
||||
engagement it had grown into a sprawling legacy estate: \textbf{200+ microservices}
|
||||
across \textbf{334 repositories}, roughly \textbf{\$35{,}000/month} of public-cloud
|
||||
spend, accumulated security debt, and a hard regulatory bar to clear. The
|
||||
engineering organization was a set of contractor teams, and every engineer on the
|
||||
program had access to the same frontier chat model (Anthropic's Claude, via
|
||||
\texttt{claude.ai}).
|
||||
|
||||
\paragraph{The measured comparison.} The audit covered 334 repositories and
|
||||
\subsection{The intervention: same model, different stack}
|
||||
The variable that differed was the \emph{tooling around} the model. One engineer
|
||||
worked through Hanzo's owned, agentic stack (\texttt{hanzo.ai})---an orchestration
|
||||
layer that turns the model into an autonomous build--test--ship loop across hundreds
|
||||
of repositories---while the contractor teams used the model the conventional way, as
|
||||
an interactive assistant alongside hand-driven development. This is \emph{not} a
|
||||
claim that one group had a smarter model: everyone held the same frontier weights.
|
||||
What separated the outcomes was whether the model was wrapped in an agentic stack or
|
||||
merely chatted with.
|
||||
|
||||
\subsection{The measured comparison}
|
||||
The platform's own engineering audit covered 334 repositories and
|
||||
31{,}058 commits over an 81-day window, attributing every commit from raw version-control
|
||||
history and normalizing cost against each contributor's monthly spend. We report
|
||||
the three metrics the audit treats as defensible: total commits shipped, unit cost
|
||||
@@ -146,6 +157,43 @@ $250\times$--$900\times$ unit-cost gap survives it. Even read at its most
|
||||
conservative, one engineer plus an agentic stack delivered a team's worth of
|
||||
audited, shipped output at a fraction of the cost per line.
|
||||
|
||||
\subsection{The outcome: it shipped}
|
||||
The comparison is not about commit velocity in the abstract---the same engagement
|
||||
\emph{shipped the platform}, and cleared the bar that matters. Over the window the
|
||||
estate was consolidated from \textbf{200+ microservices into 3 compiled binaries};
|
||||
the codebase went from \textbf{3.07M lines to 1.06M active} ($\sim$3.5\,M lines of
|
||||
dead code deleted); a source audit of \textbf{93 findings} (22 critical) plus
|
||||
\textbf{$\sim$780 dependency vulnerabilities} was driven to \textbf{zero actionable};
|
||||
and cloud spend fell \textbf{$\sim$90\%} ($\sim$\$35K/mo $\rightarrow$
|
||||
$\sim$\$3.5K/mo, $\sim$\$380K/yr). The system cleared \textbf{FINRA/SEC approval} and
|
||||
\textbf{SOC~2}, in record time---with the bulk of that work carried by the single
|
||||
agentic-stack engineer; a focused \textbf{10-day remediation sprint} alone produced
|
||||
\textbf{3{,}694 commits} and \textbf{100\% of the critical- and high-severity
|
||||
security fixes}. The full cost-and-consolidation breakdown is its own companion case
|
||||
study; the point here is narrower but decisive: the productivity gap above was
|
||||
\emph{not} bought by cutting corners---it cleared a regulated, externally audited
|
||||
production bar that the contractor teams, on the same model, did not.
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\small
|
||||
\begin{tabular}{l r r}
|
||||
\toprule
|
||||
\textbf{Dimension} & \textbf{Before} & \textbf{After} \\
|
||||
\midrule
|
||||
Microservices & 200+ & 3 compiled binaries \\
|
||||
Codebase (active) & 3.07M lines & 1.06M lines ($-$70\%) \\
|
||||
Cloud spend & $\sim$\$35K/mo & $\sim$\$3.5K/mo ($\sim$90\%) \\
|
||||
Open critical/high findings & 93 (22 critical) & 0 actionable \\
|
||||
Dependency vulnerabilities & $\sim$780 & 0 actionable \\
|
||||
Regulatory status & legacy, unapproved & FINRA/SEC approved, SOC~2 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{The engagement outcome, anonymized. The productivity comparison was
|
||||
delivered \emph{into} this result---a regulated, audited production system---not
|
||||
alongside a prototype.}
|
||||
\end{table}
|
||||
|
||||
\paragraph{Why the stack wins (the mechanism).} The audit is one measured data
|
||||
point; the reason to expect it to reproduce is structural, not anecdotal.
|
||||
Interactive chat is human-paced and serial: a prompt is written, an answer is read,
|
||||
|
||||
Binary file not shown.
Reference in New Issue
Block a user