papers/hanzo-flow/sections/03-visual-debugger.tex

\section{Visual Debugger}

\label{sec:debugger}

\subsection{Execution Traces}

Every execution of a flow graph produces a trace $T = [(v_i, t_i, x_i, y_i, \delta_i)]$ where $v_i$ is the node, $t_i$ is the timestamp, $x_i$ is the input, $y_i$ is the output, and $\delta_i$ is the duration. Traces are stored in a structured log and indexed for retrieval.

\subsection{Replay Mode}

The visual debugger supports full execution replay:

\begin{enumerate}[leftmargin=1.4em]
    \item \textbf{Step-through}: Execute one node at a time, inspecting inputs/outputs at each step.
    \item \textbf{Breakpoints}: Pause execution at specific nodes or when conditions are met.
    \item \textbf{Time travel}: Rewind to any previous step and re-execute from that point with modified inputs.
    \item \textbf{Branch exploration}: At conditional nodes, explore all branches (not just the one taken) to understand alternative paths.
\end{enumerate}

\subsection{Token-Level Attribution}

For LLM Call nodes, the debugger provides token-level attribution showing which input tokens influenced which output tokens, computed via gradient-based attention analysis:

\begin{equation}
\text{Attr}(x_i, y_j) = \sum_{h=1}^H \sum_{l=1}^L \alpha_{h,l}(x_i, y_j),
\end{equation}

where $\alpha_{h,l}$ is the attention weight from head $h$ in layer $l$. This enables developers to understand why a model produced a particular output.

\subsection{Diff View}

When modifying a workflow, the debugger provides a diff view comparing execution traces before and after the change:

\begin{algorithm}[H]
\caption{Execution Diff Analysis}
\label{alg:diff}
\begin{algorithmic}[1]
\Require Traces $T_{\text{old}}$, $T_{\text{new}}$, test inputs $\mathcal{X}$
\For{each test input $x \in \mathcal{X}$}
    \State $y_{\text{old}} \gets \text{Replay}(T_{\text{old}}, x)$
    \State $y_{\text{new}} \gets \text{Execute}(G_{\text{new}}, x)$
    \State $\text{diff} \gets \text{StructuredDiff}(y_{\text{old}}, y_{\text{new}})$
    \State Highlight changed nodes and data paths
\EndFor
\State \Return aggregate diff summary
\end{algorithmic}
\end{algorithm}

\subsection{Debugging Effectiveness}

In a user study with 24 developers debugging intentionally broken workflows (see \S\ref{sec:evaluation} for details):

\begin{table}[H]
\centering
\small
\begin{tabular}{lcc}
\toprule
\textbf{Debugging Method} & \textbf{Time (min)} & \textbf{Fix Rate} \\
\midrule
Log-based (print/grep) & 18.4 & 71\% \\
IDE debugger (breakpoints) & 12.7 & 83\% \\
LangSmith traces & 9.2 & 87\% \\
\textbf{Hanzo Flow debugger} & \textbf{6.1} & \textbf{96\%} \\
\bottomrule
\end{tabular}
\caption{Debugging time and fix rate comparison.}
\label{tab:debug}
\end{table}