papers/hanzo-self-improving-agents/sections/13-discussion-and-future-work.tex

\section{Discussion and Future Work}

\label{sec:discussion}

\subsection{Integration with GRPO Continuous Learning}

The 4-Loop Architecture focuses on behavioral self-improvement: building new
tools, encoding user preferences, and refining session patterns. A separate
complementary direction is \emph{parametric} self-improvement: updating
the agent's base model weights using online learning signals.

Our companion paper~\cite{hanzo2026aso} introduces Active Semantic Optimization
(ASO), which uses training-free Group-Relative Policy Optimization (TF-GRPO)
to adapt frozen model weights via decode-time Product-of-Experts decoding.
The 4-Loop Architecture and ASO are architecturally complementary: the
4-Loop system provides the behavioral substrate (what tools are available,
what preferences are encoded), while ASO provides the parametric adaptation
(how the base model responds to prompts within those tools).

In the combined system, telemetry records from Loop~0 provide the reward
signals for TF-GRPO rollouts in ASO, creating a closed loop between behavioral
and parametric improvement. We leave the formal analysis of this combined
system to future work.

\subsection{Federated Self-Improvement Across Agent Fleets}

The current architecture treats self-improvement as a single-agent problem.
In practice, many deployments involve fleets of agents working on related
tasks (e.g., all agents in an organization working on the same codebase).

The DSO protocol~\cite{hanzo2026dso} provides a mechanism for sharing
experiential priors across agents in a Byzantine-fault-tolerant way. Extending
the 4-Loop Architecture to federated settings would allow Loop~4 to aggregate
evidence across all agents in the fleet, not just one agent's sessions. A
\textsc{MaintenanceProposal} that is supported by 50 agent-sessions across
20 agents is a much stronger signal than one supported by 5 sessions of a
single agent.

Technical challenges include: (1) privacy-preserving telemetry aggregation
(telemetry records may contain sensitive code snippets), (2) Byzantine-robust
failure pattern matching across agents with different task distributions, and
(3) conflict resolution when agents in the fleet have contradicting learned
facts (e.g., different users with different style preferences).

\subsection{Automated Proposal Implementation via Harness-Hacker}

The human gate in Loop~4 is a safety mechanism, not an architectural necessity.
As the build pipeline matures and the track record of the automated system
grows, it becomes possible to automate the implementation of low-risk proposals
(e.g., those with $\text{risk} = \text{low}$ and evidence from $\geq 10$
sessions).

We are developing a ``harness-hacker'' system that can automatically generate
comprehensive test suites for proposed tool changes, run them against the
existing tool library, and provide quantitative risk estimates. When the
harness-hacker's confidence in safety is sufficiently high, Loop~4 could
auto-implement low-risk proposals subject to automatic rollback if the
improvement trajectory does not improve within $M$ sessions.

\subsection{Tiered Runtime Integration}

The Hanzo bot architecture supports tiered runtimes: shared gateway, terminal
pod, Linux desktop, EC2 Linux, EC2 Mac, and local desktop. Telemetry collected
from one tier should persist across tier transitions. The current JSONL +
NATS JetStream design supports this: the JSONL file is tier-portable and
NATS can replay from any offset, making Loop~0 resilient to tier transitions
mid-session.

Loops~1 and~2 present a challenge: a BIN trigger fired in a terminal pod
should produce a tool that is immediately available in the local desktop tier
when the user switches. We plan to address this via the PaaS platform's
artifact registry, where built tools are stored as versioned artifacts
accessible from all tiers.

\subsection{Limitations}

\begin{enumerate}[leftmargin=1.5em]
    \item \textbf{Stationarity assumption.} Theorem~\ref{thm:convergence}
          assumes a stationary task distribution. In practice, the task
          distribution shifts as the codebase evolves and new features are
          developed. The architecture's response to non-stationarity is
          loop-level: Loop~4 can retire tools that are no longer relevant,
          and the probationary status mechanism prevents stale tools from
          accumulating without evidence.

    \item \textbf{Build pipeline success rate.} The convergence rate depends
          on $p_b$, the probability that the BIN build pipeline produces a
          useful tool. For difficult failure modes (e.g., \texttt{TOOL\_GAP}
          cases requiring access to external APIs), $p_b$ may be low.
          Failure modes that cannot be addressed by tool building should
          be explicitly excluded from the BIN trigger and handled via
          Loop~4 proposals.

    \item \textbf{Single-agent evaluation.} Our evaluation is primarily
          analytical and model-based, drawing on bmo's empirical results
          as a calibration point. Large-scale empirical evaluation of the
          full 4-Loop system is ongoing work.
\end{enumerate}

% ============================================================================
% 14. CONCLUSION
% ============================================================================