papers/hanzo-agent-grpo/sections/09-related-work.tex

\section{Related Work}

\label{sec:related}

\paragraph{Continuous Learning for LLMs.}
\cite{mitchell2022memory} and \cite{de2021continual} survey continual
learning approaches for neural networks, most requiring gradient updates.
Agent-GRPO achieves continuous improvement without parameter changes.

\paragraph{Reflexion and Verbal Reinforcement.}
\cite{shinn2023reflexion} propose Reflexion, which prompts agents to verbally
reflect on failures and store reflections in memory. Agent-GRPO extends this
with group-relative advantage extraction across multiple trajectories,
producing more nuanced experiences than single-trajectory reflection.

\paragraph{Voyager and Tool Learning.}
\cite{wang2023voyager} propose Voyager, a lifelong learning agent that
creates new tools as skills. Agent-GRPO complements this by learning
\emph{when and how to use} existing tools, while Voyager focuses on
\emph{creating} new tools. Integration is described in Section~\ref{sec:integration}.

\paragraph{GRPO and Its Extensions.}
GRPO~\cite{shao2024deepseekmath} underpins the reward computation and
advantage formulation in Agent-GRPO. TF-GRPO~\cite{hanzo2026aso} introduced
the training-free adaptation of GRPO for text outputs. Agent-GRPO extends
both to the structured agent setting.

\paragraph{YouTou-Agent and Experience Libraries.}
\cite{tencent2025youtoagent} propose the three-stage pipeline that
Agent-GRPO adapts. The key difference is that YouTou-Agent was designed for
text-only trajectories with a fixed output space, while Agent-GRPO handles
variable-length tool-call sequences with rich intermediate signals.

\paragraph{Tool-Augmented Language Models.}
\cite{schick2023toolformer} train models to use tools via self-supervised
fine-tuning. \cite{yang2024intercode} evaluate models on interactive coding
tasks requiring tool use. Agent-GRPO complements these lines by providing a
training-free adaptation pathway for frozen tool-using models.

\paragraph{Memory-Augmented Agents.}
\cite{park2023generative} use natural-language memory streams for generative
agents. \cite{zhong2024memorybank} introduce MemoryBank for long-term memory
in LLMs. Agent-GRPO is distinguished by its GRPO-based extraction mechanism
that assigns principled confidence scores and supports four compression
strategies, whereas prior systems rely on recency and frequency alone.

\paragraph{Decentralized Semantic Optimization.}
The multi-agent extension of ASO, DSO~\cite{hanzo2026dso}, provides a
Byzantine-fault-tolerant protocol for sharing experiences across agents. Agent-GRPO
is the single-agent foundation on which DSO builds; experiences extracted by
Agent-GRPO can be submitted to the DSO registry for cross-agent benefit.

% ============================================================================
% 10. DISCUSSION AND FUTURE WORK
% ============================================================================