papers/zoo-satellite-ecology/sections/06-evaluation.tex

\section{Evaluation}

\label{sec:evaluation}

\subsection{Change Detection Accuracy}

We evaluate TSN on a held-out test set of 2,341 Sentinel-2 tile sequences with expert-annotated change labels:

\begin{table}[H]
\centering
\caption{Change detection accuracy comparison.}
\label{tab:cd_accuracy}
\begin{tabular}{@{}lccccc@{}}
\toprule
\textbf{Method} & \textbf{Precision} & \textbf{Recall} & \textbf{F1} & \textbf{OA (\%)} & \textbf{IoU (\%)} \\
\midrule
NDVI Differencing & 0.624 & 0.781 & 0.694 & 82.3 & 53.1 \\
Random Forest & 0.734 & 0.812 & 0.771 & 87.4 & 62.7 \\
FC-Siam-Diff \citep{daudt2018fully} & 0.847 & 0.863 & 0.855 & 91.2 & 74.6 \\
BIT \citep{chen2022remote} & 0.871 & 0.889 & 0.880 & 92.7 & 78.5 \\
ChangeFormer \citep{bandara2022transformer} & 0.884 & 0.901 & 0.892 & 93.4 & 80.6 \\
\textbf{TSN (ours)} & \textbf{0.921} & \textbf{0.934} & \textbf{0.927} & \textbf{94.2} & \textbf{86.4} \\
TSN (bi-temporal only) & 0.873 & 0.891 & 0.882 & 92.1 & 78.9 \\
\bottomrule
\end{tabular}
\end{table}

TSN achieves 94.2\% overall accuracy and 86.4\% IoU, outperforming all baselines.
The comparison between full TSN and bi-temporal TSN (using only the first and last images) demonstrates that temporal sequence modeling contributes 2.1\% accuracy and 7.5\% IoU improvement by suppressing seasonal and atmospheric false positives.

\subsection{Change Type Classification}

\begin{table}[H]
\centering
\caption{Change type classification accuracy (F1 score by category).}
\label{tab:classification}
\begin{tabular}{@{}lcccccccc@{}}
\toprule
\textbf{Method} & \textbf{DEF} & \textbf{DEG} & \textbf{AGR} & \textbf{URB} & \textbf{MIN} & \textbf{FIR} & \textbf{FLD} & \textbf{REF} \\
\midrule
RF Classifier & 0.78 & 0.54 & 0.71 & 0.82 & 0.76 & 0.84 & 0.79 & 0.61 \\
FC-Siam + MLP & 0.84 & 0.62 & 0.78 & 0.87 & 0.81 & 0.88 & 0.83 & 0.69 \\
\textbf{TSN (ours)} & \textbf{0.93} & \textbf{0.78} & \textbf{0.88} & \textbf{0.94} & \textbf{0.89} & \textbf{0.95} & \textbf{0.91} & \textbf{0.82} \\
\bottomrule
\end{tabular}
\end{table}

TSN achieves 87.8\% macro-averaged F1 across all change types.
Degradation (DEG) is the most challenging category (F1 = 0.78) because subtle canopy thinning produces weak spectral signals.
Fire detection achieves the highest F1 (0.95) due to the strong spectral signature of burned areas.

\subsection{Accuracy by Biome}

\begin{table}[H]
\centering
\caption{Change detection accuracy (F1) by biome.}
\label{tab:biome_accuracy}
\begin{tabular}{@{}lcccc@{}}
\toprule
\textbf{Biome} & \textbf{TSN} & \textbf{ChangeFormer} & \textbf{RF} & \textbf{NDVI Diff} \\
\midrule
Tropical Forest & 0.94 & 0.90 & 0.79 & 0.71 \\
Temperate Forest & 0.93 & 0.89 & 0.78 & 0.69 \\
Savanna & 0.91 & 0.86 & 0.72 & 0.62 \\
Mangrove & 0.92 & 0.87 & 0.74 & 0.64 \\
Wetland & 0.90 & 0.85 & 0.70 & 0.58 \\
Agricultural Frontier & 0.95 & 0.91 & 0.81 & 0.74 \\
\bottomrule
\end{tabular}
\end{table}

TSN achieves consistently high performance across biomes, with the largest advantage over baselines in savanna (+29 F1 points over NDVI Differencing) and wetland (+32 F1 points) ecosystems where phenological variation is most pronounced.

\subsection{Timeliness Evaluation}

We compare alert timeliness against Global Forest Watch GLAD alerts for 234 deforestation events detected by both systems:

\begin{table}[H]
\centering
\caption{Alert timeliness comparison for deforestation events.}
\label{tab:timeliness}
\begin{tabular}{@{}lccc@{}}
\toprule
\textbf{Metric} & \textbf{ZooSat} & \textbf{GLAD} & \textbf{Advantage} \\
\midrule
Median detection latency (days) & 4.7 & 15.9 & 11.2 days faster \\
Events detected within 7 days & 78.2\% & 23.1\% & +55.1\% \\
Events detected within 14 days & 94.4\% & 58.1\% & +36.3\% \\
False positive rate & 10.7\% & 18.4\% & $-$7.7\% \\
\bottomrule
\end{tabular}
\end{table}

ZooSat detects deforestation a median of 11.2 days earlier than GLAD, with a lower false positive rate.
The timeliness advantage stems from three factors: (1) Sentinel-2's 5-day revisit vs.\ Landsat-8's 16-day revisit, (2) temporal sequence processing that reduces cloud-related detection delays, and (3) the weekly processing pipeline vs.\ GLAD's monthly aggregation.

\subsection{Operational Deployment}

Over 18 months of operation (July 2024 -- January 2026), ZooSat monitored 2.4 million km$^2$ across 47 protected areas:

\begin{table}[H]
\centering
\caption{Operational deployment statistics.}
\label{tab:operational}
\begin{tabular}{@{}lr@{}}
\toprule
\textbf{Metric} & \textbf{Value} \\
\midrule
Monitored area & 2.4 million km$^2$ \\
Protected areas & 47 \\
Countries & 14 \\
Sentinel-2 tiles processed & 487,000 \\
Change events detected & 34,218 \\
Conservation alerts generated & 12,847 \\
Alerts validated (true positive) & 11,472 (89.3\%) \\
Critical alerts (Tier 1) & 1,847 \\
Deforestation events detected & 2,341 \\
Estimated area saved by early response & 4,200 hectares \\
\bottomrule
\end{tabular}
\end{table}

\subsection{Conservation Impact}

The early detection enabled measurable conservation outcomes:

\begin{itemize}[leftmargin=*]
    \item \textbf{Rapid response interventions:} 47 enforcement actions were triggered by Critical tier alerts, including 12 cases where illegal logging operations were interrupted within 48 hours of satellite detection.
    \item \textbf{Area protection:} Based on observed deforestation rates in adjacent unmonitored areas, we estimate that early detection and response prevented approximately 4,200 hectares of additional forest loss.
    \item \textbf{Policy influence:} HFI trajectory reports for 8 protected areas were submitted as evidence in national environmental review processes.
    \item \textbf{Community engagement:} Alert data was shared with 23 indigenous community monitoring programs, supporting community-based forest governance.
\end{itemize}

% ============================================================
% 7. System Architecture
% ============================================================