papers/hanzo-agent-grpo/sections/01-introduction.tex

\section{Introduction}

\label{sec:intro}

Tool-using agents---systems that interact with external tools such as code
interpreters, file systems, search engines, and APIs---have emerged as a
primary deployment paradigm for large language models
\cite{schick2023toolformer, yang2024intercode, yao2023react}. Unlike pure
text-generation tasks, agent behavior is evaluated not merely on the quality of
prose but on the correctness of tool invocations, the appropriateness of tool
selection, and the agent's ability to recover from errors encountered during
execution. This richer evaluation surface creates both new challenges and new
opportunities for adaptation.

\paragraph{The Adaptation Gap for Agents.}
Existing continuous learning works for LLMs focus on text-quality
metrics---pass rates on coding benchmarks, BLEU scores, preference rankings.
The Active Semantic Optimization (ASO) framework~\cite{hanzo2026aso} takes a
training-free approach, extracting semantic advantages from grouped rollouts and
applying them as Product-of-Experts decoding factors. Yet ASO and its
predecessors treat the model output as a monolithic text sequence. An agent
trajectory is structurally different: it is a \emph{sequence of decisions}
interleaved with environmental feedback, where each decision (which tool to
call, with what parameters) has an observable outcome that serves as a rich
local reward signal.

This structural difference is consequential. Consider an agent that repeatedly
chooses \texttt{bash} to parse JSON when a dedicated \texttt{read\_json} tool
would be faster, more reliable, and incur lower token cost. This failure to
prefer the specialized tool will not manifest as a text-quality degradation
that text-level GRPO would catch; the output may look correct even when the
tool choice was suboptimal. Conversely, an agent that learns to prefer
\texttt{read\_json} for JSON parsing can apply this lesson to future tasks
without any gradient update---provided the lesson is properly encoded and
retrieved.

\paragraph{The Semantic Experience Library.}
Our central contribution is the \textbf{semantic experience library}
$\mathcal{E}$, a persistent, embedding-indexed store of natural-language
experiences extracted from agent trajectories via an adapted GRPO pipeline.
Each entry $e \in \mathcal{E}$ encodes a reusable lesson: a statement about
when to use which tool, how to structure parameters for a particular API, or
which failure patterns to anticipate in a given domain. At the start of each
new session, the top-$K$ experiences most semantically similar to the incoming
task are injected into the agent's system prompt, providing explicit behavioral
guidance without modifying model weights.

\paragraph{Agent-GRPO.}
We formalize this approach as \textbf{Agent-GRPO}, which extends
Training-Free GRPO (TF-GRPO)~\cite{hanzo2026aso} to agent trajectories through
three technical innovations:

\begin{enumerate}[leftmargin=1.5em]
    \item \textbf{Trajectory-level rewards:} A composite reward function
          combining binary tool success signals and time-delayed user acceptance
          signals, aggregated over the full trajectory.
    \item \textbf{Three-stage extraction pipeline:} Adapted from
          \cite{tencent2025youtoagent}, the pipeline summarizes individual
          trajectories (Stage 1), extracts group advantages by comparing
          successful and unsuccessful groups (Stage 2), and consolidates batch
          operations to produce experience library updates (Stage 3).
    \item \textbf{Memory management:} Four compression strategies---diversity,
          importance, temporal decay, and hybrid---that maintain library quality
          as $|\mathcal{E}|$ grows, preventing staleness and redundancy while
          preserving high-utility experiences.
\end{enumerate}

\paragraph{Contributions.}
\begin{enumerate}[leftmargin=1.5em]
    \item A formal model of agent tool-use trajectories as typed tuples with
          observable success signals (Section~\ref{sec:trajectories}).
    \item A composite reward function and group-relative advantage computation
          adapted for agent trajectories (Section~\ref{sec:reward}).
    \item The complete Agent-GRPO algorithm with formal pseudocode and
          correctness guarantees (Section~\ref{sec:algorithm}).
    \item A semantic memory manager with four compression strategies and
          provably monotone quality improvement (Sections~\ref{sec:memory}
          and~\ref{sec:theory}).
    \item Integration with ASO's four-loop self-improvement architecture, closing
          the feedback loop from telemetry to library update
          (Section~\ref{sec:integration}).
    \item Empirical evaluation demonstrating state-of-the-art results on
          HB-Bench at sub-\$20 adaptation cost (Section~\ref{sec:experiments}).
\end{enumerate}

\paragraph{Paper Outline.}
Section~\ref{sec:background} reviews GRPO, TF-GRPO, and the YouTou-Agent
pipeline. Section~\ref{sec:agentgrpo} formalizes Agent-GRPO.
Section~\ref{sec:memory} presents the semantic memory manager.
Section~\ref{sec:adaptations} describes agent-specific adaptations.
Section~\ref{sec:integration} describes integration with self-improvement loops.
Section~\ref{sec:theory} provides theoretical analysis.
Section~\ref{sec:experiments} presents empirical evaluation.
Section~\ref{sec:related} discusses related work.
Section~\ref{sec:conclusion} concludes.

% ============================================================================
% 2. BACKGROUND
% ============================================================================