feat(vs-claude): develop the productivity comparison into a solid case study

- Full case-study arc: Background (regulated platform, 200+ microservices, $35K/mo,
  regulatory bar) -> Intervention (same Claude, different stack) -> Measured comparison
  (audit table, 250-900x) -> Outcome (before/after table: 200+->3 binaries, 3.07M->1.06M
  LOC, ~90% cost, 93 findings->0, FINRA/SEC+SOC2) -> mechanism -> switch case.
- Proves the gap was delivered into a regulated, audited production result, not a demo.
This commit is contained in:
2026-06-17 17:11:14 -07:00
parent 3a00be1de6
commit 680c559633
4 changed files with 62 additions and 14 deletions
Binary file not shown.
Binary file not shown.
+62 -14
View File
@@ -85,26 +85,37 @@ are mandatory for regulated, critical-infrastructure, and government workloads
do not hold and cannot run yourself.
% ============================================================================
\section{One Engineer on the Hanzo Stack vs.\ a Team on Claude}
\section{Case Study: One Engineer on the Hanzo Stack vs.\ a Team on Claude}
\label{sec:oneeng}
% ============================================================================
The cleanest evidence for this paper's thesis is not a benchmark---it is a
controlled natural experiment that already happened in production. On a regulated
capital-markets platform, every engineer on the program had access to the same
frontier chat model (Anthropic's Claude, via \texttt{claude.ai}). The variable
that differed was the \emph{tooling around} that model: one engineer worked
through Hanzo's owned, agentic stack (\texttt{hanzo.ai})---an orchestration layer
that turns the model into an autonomous build-test-ship loop across hundreds of
repositories---while the contractor teams used the model the conventional way, as
an interactive assistant alongside hand-driven development.
controlled natural experiment that already ran in production, on a real regulated
system, and was measured by that system's own engineering audit.
Crucially, this is \emph{not} a claim that one group had a smarter model. They did
not. Everyone held the same frontier weights. What separated the outcomes was
whether the model was wrapped in an agentic stack or merely chatted with. The
platform's own engineering audit measured the result.
\subsection{Background: a regulated platform, a sprawling estate}
The setting (anonymized) is a U.S.\ regulated securities venue---an SEC-registered
alternative trading system with broker-dealer and transfer-agent functions, among
the most compliance-heavy software domains that exists. By the time of the
engagement it had grown into a sprawling legacy estate: \textbf{200+ microservices}
across \textbf{334 repositories}, roughly \textbf{\$35{,}000/month} of public-cloud
spend, accumulated security debt, and a hard regulatory bar to clear. The
engineering organization was a set of contractor teams, and every engineer on the
program had access to the same frontier chat model (Anthropic's Claude, via
\texttt{claude.ai}).
\paragraph{The measured comparison.} The audit covered 334 repositories and
\subsection{The intervention: same model, different stack}
The variable that differed was the \emph{tooling around} the model. One engineer
worked through Hanzo's owned, agentic stack (\texttt{hanzo.ai})---an orchestration
layer that turns the model into an autonomous build--test--ship loop across hundreds
of repositories---while the contractor teams used the model the conventional way, as
an interactive assistant alongside hand-driven development. This is \emph{not} a
claim that one group had a smarter model: everyone held the same frontier weights.
What separated the outcomes was whether the model was wrapped in an agentic stack or
merely chatted with.
\subsection{The measured comparison}
The platform's own engineering audit covered 334 repositories and
31{,}058 commits over an 81-day window, attributing every commit from raw version-control
history and normalizing cost against each contributor's monthly spend. We report
the three metrics the audit treats as defensible: total commits shipped, unit cost
@@ -146,6 +157,43 @@ $250\times$--$900\times$ unit-cost gap survives it. Even read at its most
conservative, one engineer plus an agentic stack delivered a team's worth of
audited, shipped output at a fraction of the cost per line.
\subsection{The outcome: it shipped}
The comparison is not about commit velocity in the abstract---the same engagement
\emph{shipped the platform}, and cleared the bar that matters. Over the window the
estate was consolidated from \textbf{200+ microservices into 3 compiled binaries};
the codebase went from \textbf{3.07M lines to 1.06M active} ($\sim$3.5\,M lines of
dead code deleted); a source audit of \textbf{93 findings} (22 critical) plus
\textbf{$\sim$780 dependency vulnerabilities} was driven to \textbf{zero actionable};
and cloud spend fell \textbf{$\sim$90\%} ($\sim$\$35K/mo $\rightarrow$
$\sim$\$3.5K/mo, $\sim$\$380K/yr). The system cleared \textbf{FINRA/SEC approval} and
\textbf{SOC~2}, in record time---with the bulk of that work carried by the single
agentic-stack engineer; a focused \textbf{10-day remediation sprint} alone produced
\textbf{3{,}694 commits} and \textbf{100\% of the critical- and high-severity
security fixes}. The full cost-and-consolidation breakdown is its own companion case
study; the point here is narrower but decisive: the productivity gap above was
\emph{not} bought by cutting corners---it cleared a regulated, externally audited
production bar that the contractor teams, on the same model, did not.
\begin{table}[H]
\centering
\small
\begin{tabular}{l r r}
\toprule
\textbf{Dimension} & \textbf{Before} & \textbf{After} \\
\midrule
Microservices & 200+ & 3 compiled binaries \\
Codebase (active) & 3.07M lines & 1.06M lines ($-$70\%) \\
Cloud spend & $\sim$\$35K/mo & $\sim$\$3.5K/mo ($\sim$90\%) \\
Open critical/high findings & 93 (22 critical) & 0 actionable \\
Dependency vulnerabilities & $\sim$780 & 0 actionable \\
Regulatory status & legacy, unapproved & FINRA/SEC approved, SOC~2 \\
\bottomrule
\end{tabular}
\caption{The engagement outcome, anonymized. The productivity comparison was
delivered \emph{into} this result---a regulated, audited production system---not
alongside a prototype.}
\end{table}
\paragraph{Why the stack wins (the mechanism).} The audit is one measured data
point; the reason to expect it to reproduce is structural, not anecdotal.
Interactive chat is human-paced and serial: a prompt is written, an answer is read,
Binary file not shown.