papers/hanzo-operative/sections/02-background-and-related-work.tex

\section{Background and Related Work}

\label{sec:background}

\subsection{Computer Use Agents}

The concept of software agents that interact with GUIs dates to the SIKULI system~\cite{yeh2009sikuli}, which used screenshot-based visual pattern matching for UI automation. Modern computer use agents leverage vision-language models (VLMs) to understand screen content and generate interaction commands.

CogAgent~\cite{hong2024cogagent} demonstrated that VLMs fine-tuned on GUI screenshots can accurately locate and interact with interface elements. SeeClick~\cite{cheng2024seeclick} introduced visual grounding pre-training for improved element localization. WebVoyager~\cite{he2024webvoyager} showed that GPT-4V could complete web tasks by reasoning over screenshots. AgentStudio~\cite{zheng2024agentstudio} provided a benchmark infrastructure for evaluating computer use agents.

\subsection{GUI Understanding}

GUI understanding combines techniques from document understanding, visual grounding, and accessibility research. Key approaches include:

\begin{itemize}[leftmargin=*]
    \item \textbf{Pixel-level}: Direct visual processing of screenshots using CNNs or vision transformers~\cite{dosovitskiy2020image}.
    \item \textbf{DOM/Accessibility tree}: Structured parsing of web page DOM or OS accessibility APIs~\cite{he2024webvoyager}.
    \item \textbf{Hybrid}: Combining visual and structural signals for robust element detection~\cite{hong2024cogagent}.
\end{itemize}

Operative adopts the hybrid approach, fusing pixel-level visual understanding with accessibility tree data for maximum coverage across native desktop and web applications.

\subsection{Action Spaces for Computer Interaction}

Prior work has used various action space formulations:

\begin{itemize}[leftmargin=*]
    \item \textbf{Coordinate-based}: Raw $(x, y)$ click and type actions~\cite{anthropic2024computer}.
    \item \textbf{Element-based}: Actions reference specific UI elements by ID or selector~\cite{he2024webvoyager}.
    \item \textbf{API-based}: Direct invocation of application APIs bypassing the GUI~\cite{wang2024mobile}.
\end{itemize}

Operative supports all three modes with a unified type system, enabling agents to choose the most appropriate interaction level for each task.