papers/hanzo-self-improving-agents/sections/03-the-4-loop-architecture.tex

\section{The 4-Loop Architecture}

\label{sec:architecture}

\subsection{Overview}

We define a \emph{self-improving coding agent} as an agent that operates in
a software development environment and exhibits monotonically non-decreasing
expected task success rate as the number of completed sessions increases. The
4-Loop Architecture provides a concrete instantiation of this definition via
four feedback loops operating at distinct timescales.

\begin{definition}[4-Loop Self-Improvement System]
\label{def:4loop}
A 4-Loop Self-Improvement System $\mathcal{S} = (\mathcal{L}_0, \mathcal{L}_1,
\mathcal{L}_2, \mathcal{L}_3, \mathcal{L}_4)$ consists of five loops:
\begin{itemize}[leftmargin=1.5em]
    \item $\mathcal{L}_0$: The \emph{Telemetry Pipeline}, operating continuously
          with zero latency overhead.
    \item $\mathcal{L}_1$: The \emph{Build-It-Now Loop}, event-driven with
          latency $\leq 5$ minutes.
    \item $\mathcal{L}_2$: The \emph{Active Learning Loop}, hook-driven with
          latency $\leq 30$ seconds.
    \item $\mathcal{L}_3$: The \emph{Session Reflection Loop}, end-of-session
          with latency $\leq 2$ minutes.
    \item $\mathcal{L}_4$: The \emph{Maintenance Pass}, every $N$ sessions,
          human-gated with no latency bound.
\end{itemize}
\end{definition}

Figure~\ref{fig:architecture} illustrates the loop structure and information
flows. The critical invariant is that no loop writes to the agent's active
context window. Each loop reads from and writes to persistent stores (telemetry
log, learned facts database, tool library, proposal queue) that are loaded at
session start, not accumulated during execution.

\begin{figure}[H]
\centering
\fbox{%
\begin{minipage}{0.9\textwidth}
\centering
\vspace{1em}
\textbf{4-Loop Self-Improvement Architecture}\\[1em]
\begin{tabular}{lllll}
\toprule
Loop & Trigger & Latency & Reads & Writes \\
\midrule
$\mathcal{L}_0$ & Every tool call & 0\,ms & -- & Telemetry log \\
$\mathcal{L}_1$ & $\geq$3 identical failures & $\leq$5\,min & Telemetry log & Tool library \\
$\mathcal{L}_2$ & Correction signal & $\leq$30\,sec & Context diff & Facts DB \\
$\mathcal{L}_3$ & Session end & $\leq$2\,min & Telemetry log & Reflection store \\
$\mathcal{L}_4$ & Every $N$ sessions & Unbounded & All stores & Proposal queue \\
\bottomrule
\end{tabular}
\vspace{1em}
\end{minipage}%
}
\caption{Summary of the 4-Loop Architecture. Each loop operates independently
with a well-defined trigger condition, latency bound, and storage contract.
No loop pollutes the active context window.}
\label{fig:architecture}
\end{figure}

\subsection{State Space}

The agent state at session $s$ is a tuple:

\begin{equation}
\label{eq:state}
\Sigma^{(s)} = \bigl( \mathcal{T}^{(s)},\; \mathcal{F}^{(s)},\; \mathcal{K}^{(s)},\;
               \mathcal{R}^{(s)},\; \mathcal{P}^{(s)} \bigr),
\end{equation}

where $\mathcal{T}^{(s)}$ is the telemetry log (all sessions up to $s$),
$\mathcal{F}^{(s)}$ is the learned facts database, $\mathcal{K}^{(s)}$ is the
tool library (built-in plus agent-constructed), $\mathcal{R}^{(s)}$ is the
reflection store, and $\mathcal{P}^{(s)}$ is the human-pending proposal queue.

The self-improvement objective is to maximize expected task success rate
$\mu^{(s)} = \E[\text{success}(t) \mid \Sigma^{(s)}]$ over the task distribution.
Each loop updates a subset of $\Sigma^{(s)}$, and the state is loaded fresh at
the start of each session. This design ensures that improvements from one session
are available in subsequent sessions without contaminating within-session context.

% ============================================================================
% 4. LOOP 0: TELEMETRY PIPELINE
% ============================================================================