mirror of
https://github.com/hanzoai/papers.git
synced 2026-10-29 07:34:31 +00:00
zeekay
0f01f8b555
Resolves high-severity review findings: anachronistic dates aligned to citation floors; arithmetic corrected to match formulas; internal contradictions collapsed to the body-supported value; LP self-references aligned to filenames; liquidity-surface repo paths scrubbed; AI-slop trimmed. PDFs rebuilt separately.
26 lines
2.6 KiB
TeX
26 lines
2.6 KiB
TeX
\section{Introduction}
|
|
|
|
|
|
The ability to operate computers through graphical user interfaces has been a longstanding goal of AI research~\cite{shi2017world}. While large language models have demonstrated remarkable capabilities in text-based tool use~\cite{schick2024toolformer}, extending these capabilities to visual, spatially-grounded computer interaction introduces fundamental challenges in perception, planning, and action execution.
|
|
|
|
Recent work on computer use agents---including Anthropic's computer use mode~\cite{anthropic2024computer}, OpenAI's Operator~\cite{openai2025operator}, and Google's Project Mariner~\cite{google2024mariner}---has demonstrated the feasibility of LLM-driven GUI interaction. However, these systems are tightly coupled to specific model providers, limiting reproducibility, customization, and deployment flexibility.
|
|
|
|
Operative addresses this gap as a model-agnostic, open-source framework for computer use that:
|
|
|
|
\begin{enumerate}[leftmargin=*]
|
|
\item \textbf{Decouples perception from action}: The screen understanding pipeline produces structured representations consumable by any LLM, enabling model-agnostic planning.
|
|
\item \textbf{Formalizes the action space}: All 47 primitive operations have typed signatures with pre-conditions, post-conditions, and reversibility annotations, enabling formal verification of action sequences.
|
|
\item \textbf{Enforces safety boundaries}: A configurable policy engine restricts agent capabilities based on the principle of least privilege, preventing unintended system modifications.
|
|
\item \textbf{Supports multi-step planning}: A hierarchical planner decomposes goals into sub-tasks with checkpointing and rollback, enabling recovery from errors without full task restart.
|
|
\end{enumerate}
|
|
|
|
\paragraph{Contributions.} The specific contributions of this paper are:
|
|
\begin{itemize}[leftmargin=*]
|
|
\item A screen understanding pipeline that fuses OCR, visual grounding, and accessibility tree parsing to produce structured element maps with 94.7\% element detection accuracy (Section~\ref{sec:perception}).
|
|
\item A formally specified action space with pre/post-conditions enabling static verification of action sequences (Section~\ref{sec:actions}).
|
|
\item A safety boundary system with filesystem, network, and system sandboxing (Section~\ref{sec:safety}).
|
|
\item A hierarchical planning architecture with checkpoint-based rollback achieving 38.1\% task completion on OSWorld (Section~\ref{sec:planning}).
|
|
\item Comprehensive evaluation across desktop and browser tasks (Section~\ref{sec:evaluation}).
|
|
\end{itemize}
|
|
|