papers/hanzo-self-improving-agents/sections/02-related-work.tex

\section{Related Work}

\label{sec:related}

\subsection{bmo: ngrok's Self-Improving Agent}

The most directly related work is bmo~\cite{bmo2024}, ngrok's autonomous coding
agent with a self-improvement loop. bmo maintains a library of tools and a
memory of past interactions, and uses LLM-based reflection to identify
opportunities to build new tools. Over 100 sessions, bmo built 11 tools that
measurably reduced friction in repeated tasks. bmo's key insight---that agents
should build tools to address their own limitations---directly inspires our
Loop~1 design.

However, bmo's improvement mechanism has two limitations our architecture
addresses. First, bmo uses LLM judgment to decide when to build new tools,
which suffers from the self-evaluation problem described in Section~\ref{sec:intro}.
Second, bmo's tool-building is session-agnostic; there is no structured
mechanism to aggregate evidence across sessions before investing in a new tool.
Our architecture addresses both: deterministic triggers replace LLM judgment,
and Loop~4's maintenance pass aggregates cross-session evidence.

\subsection{Reflexion}

\cite{shinn2023reflexion} introduced Reflexion, a framework in which agents
maintain a ``self-reflection'' string that is prepended to the context on
subsequent attempts at a failed task. Reflexion demonstrates that verbal
reflection improves task performance across a range of benchmarks. Our work
is complementary but takes a different design stance: where Reflexion is
optimized for single-task retry scenarios (try, reflect, retry), our
architecture targets persistent skill accumulation across different tasks over
many sessions.

More importantly, Reflexion's verbal reflection is stored as LLM-generated text,
which inherits the entropy limitations we formalize in
Section~\ref{sec:info_theory}. We show that structured telemetry records
preserve significantly more decision-relevant information per byte.

\subsection{Voyager}

\cite{wang2023voyager} presented Voyager, an LLM-powered agent in Minecraft
that builds a skill library via automated curriculum learning. Voyager's
``skill library'' is the closest precedent to our Loop~1 tool-building mechanism.
Key differences: Voyager operates in a static simulated environment where the
space of skills is well-defined, while our architecture operates in open-ended
software development environments where friction modes are not known in advance.
Voyager uses an LLM to decide when to codify a skill; we use deterministic
pattern matching over telemetry streams.

\subsection{OpenHands}

\cite{wang2024openhands} describes OpenHands (formerly OpenDevin), a platform
for AI software engineers that emphasizes action space design and sandboxed
execution. OpenHands provides the operational substrate on which our architecture
can be layered---it handles low-level execution, while we handle the
meta-level improvement loop. OpenHands does not itself provide a self-improvement
mechanism beyond standard context management.

\subsection{AutoGen}

\cite{wu2023autogen} introduced AutoGen, a multi-agent conversation framework
that enables complex agent orchestration via structured dialogue. AutoGen's
conversational model allows agents to request help from specialist agents,
which is one mechanism for ad-hoc improvement. However, AutoGen is designed
for multi-agent coordination rather than within-agent self-improvement, and
does not address the telemetry or anti-deferral problems we study here.

\subsection{Self-Refine}

\cite{madaan2023selfrefine} demonstrated that LLMs can improve their outputs
through iterative self-critique and refinement within a single inference
pass. Self-Refine is the most prominent instantiation of the ``LLM as
self-evaluator'' paradigm that we critically examine. While Self-Refine works
well for short tasks with clear quality signals (code compilation, math
verification), it degrades on long-horizon tasks where quality signals are
delayed and the LLM's self-critique is uncalibrated. Our formal analysis in
Section~\ref{sec:triggers} quantifies this degradation.

\subsection{Tool-Use and Agent Skill Acquisition}

More broadly, the problem of skill acquisition in agents has been studied in
the context of tool-use~\cite{schick2024toolformer, qin2024toolllm},
program synthesis~\cite{chen2021evaluating}, and reinforcement
learning~\cite{sutton2018reinforcement}. Our work is distinguished by its
focus on the \emph{trigger mechanism}---not just how to build new skills, but
when to build them---and its insistence on deterministic triggers over learned
policies for production reliability.

% ============================================================================
% 3. THE 4-LOOP ARCHITECTURE
% ============================================================================