papers/zoo-educational-ai/sections/07-experimental-evaluation.tex

\section{Experimental Evaluation}

\label{sec:evaluation}

\subsection{Deployment Overview}

We deployed ZooEd across 23 conservation sites in 11 countries over 14 months (October 2024 -- December 2025).
Sites span tropical forests (7), marine/coastal (5), savanna (4), montane (4), and wetland (3) ecosystems.
Total enrollment reached 3,847 learners across four learner categories:

\begin{table}[H]
\centering
\caption{Learner demographics across deployment sites.}
\label{tab:demographics}
\begin{tabularx}{\textwidth}{@{}lccX@{}}
\toprule
\textbf{Category} & \textbf{Count} & \textbf{Languages} & \textbf{Description} \\
\midrule
Park Rangers & 1,247 & 23 & Government and community rangers \\
Field Biologists & 892 & 14 & Research staff and graduate students \\
Community Members & 1,203 & 31 & Local residents in buffer zones \\
Policy Staff & 505 & 12 & Government and NGO policy personnel \\
\bottomrule
\end{tabularx}
\end{table}

\subsection{Experimental Design}

We employed a quasi-experimental design with matched comparison groups.
At each site, learners were assigned to one of three conditions:

\begin{enumerate}[leftmargin=*]
    \item \textbf{ZooEd-Full:} Complete adaptive system with ecological grounding and multi-modal assessment ($n = 1,284$).
    \item \textbf{ZooEd-Basic:} Adaptive content selection without ecological grounding or sensor integration ($n = 1,271$).
    \item \textbf{Control:} Standard conservation training curricula used at each site ($n = 1,292$).
\end{enumerate}

Assignment used stratified matching on pre-test scores, job role, education level, and site to minimize confounding.

\subsection{Knowledge Retention}

We measured knowledge retention using standardized assessments administered at post-training (immediate), 3-month, 6-month, and 12-month intervals:

\begin{table}[H]
\centering
\caption{Knowledge retention scores (mean $\pm$ SD) across conditions and time points.}
\label{tab:retention}
\begin{tabular}{@{}lcccc@{}}
\toprule
\textbf{Condition} & \textbf{Post} & \textbf{3-Month} & \textbf{6-Month} & \textbf{12-Month} \\
\midrule
ZooEd-Full & $78.3 \pm 11.2$ & $74.1 \pm 12.8$ & $71.6 \pm 13.4$ & $68.9 \pm 14.1$ \\
ZooEd-Basic & $75.1 \pm 12.4$ & $67.3 \pm 14.1$ & $61.8 \pm 15.2$ & $55.4 \pm 16.0$ \\
Control & $71.2 \pm 14.8$ & $58.7 \pm 16.3$ & $52.1 \pm 17.1$ & $44.3 \pm 17.8$ \\
\bottomrule
\end{tabular}
\end{table}

At 12 months, ZooEd-Full learners retained 34.0\% more knowledge than Control ($p < 0.001$, Cohen's $d = 1.54$).
The SREC ecological grounding component contributed significantly: ZooEd-Full outperformed ZooEd-Basic by 19.6\% at 12 months ($p < 0.001$, $d = 0.90$), indicating that contextual reinforcement through sensor-triggered reviews substantially reduces forgetting.

\subsection{Transfer to Field Tasks}

We evaluated transfer through blind expert assessment of field task performance at 6 months:

\begin{table}[H]
\centering
\caption{Field task performance scores (expert-rated, 0--100 scale).}
\label{tab:transfer}
\begin{tabular}{@{}lccc@{}}
\toprule
\textbf{Task} & \textbf{ZooEd-Full} & \textbf{ZooEd-Basic} & \textbf{Control} \\
\midrule
Species Survey Accuracy & $82.4 \pm 9.1$ & $74.2 \pm 11.3$ & $68.7 \pm 13.4$ \\
Threat Assessment Quality & $76.8 \pm 10.7$ & $70.1 \pm 12.0$ & $63.2 \pm 14.1$ \\
Community Engagement & $71.3 \pm 12.4$ & $67.8 \pm 13.1$ & $62.5 \pm 14.8$ \\
Data Collection Protocol & $84.1 \pm 8.3$ & $77.5 \pm 10.2$ & $71.9 \pm 12.6$ \\
\midrule
\textbf{Overall} & $\mathbf{78.7 \pm 8.9}$ & $72.4 \pm 10.1$ & $66.6 \pm 12.0$ \\
\bottomrule
\end{tabular}
\end{table}

ZooEd-Full learners demonstrated 28.1\% higher field task performance than Control, with the largest gains in species survey accuracy ($+20.0\%$) and data collection protocol adherence ($+17.0\%$).

\subsection{Engagement Metrics}

\begin{table}[H]
\centering
\caption{Engagement metrics across conditions (14-month deployment).}
\label{tab:engagement}
\begin{tabular}{@{}lccc@{}}
\toprule
\textbf{Metric} & \textbf{ZooEd-Full} & \textbf{ZooEd-Basic} & \textbf{Control} \\
\midrule
Completion Rate & 73.2\% & 61.4\% & 48.7\% \\
Weekly Active Sessions & $4.7 \pm 2.1$ & $3.2 \pm 1.8$ & $1.9 \pm 1.4$ \\
Median Session Length (min) & 18.3 & 15.7 & 22.1 \\
Voluntary Re-engagement & 64.8\% & 42.3\% & 21.6\% \\
\bottomrule
\end{tabular}
\end{table}

ZooEd-Full achieved 41.0\% higher engagement (composite of completion rate, session frequency, and re-engagement) than Control.
Notably, ZooEd-Full sessions were shorter on average (18.3 vs.\ 22.1 minutes) but more frequent, consistent with the mobile-first, bite-sized design philosophy.

\subsection{Cross-Language Assessment Reliability}

We evaluated assessment reliability across language groups using inter-rater agreement between LLM evaluation and expert human grading on a stratified sample of 2,400 free-response answers (50 per language for the 12 most-represented languages):

\begin{table}[H]
\centering
\caption{Free-response assessment reliability by language group (Cohen's $\kappa$).}
\label{tab:language_reliability}
\begin{tabular}{@{}lcc@{}}
\toprule
\textbf{Language Group} & \textbf{Raw LLM} & \textbf{Calibrated} \\
\midrule
High-resource (EN, ES, FR, PT) & 0.82 & 0.89 \\
Medium-resource (SW, ID, TH, VI) & 0.71 & 0.83 \\
Low-resource (AM, QU, MG, TN) & 0.58 & 0.76 \\
\midrule
\textbf{Overall} & 0.72 & 0.84 \\
\bottomrule
\end{tabular}
\end{table}

Calibration improved reliability substantially for low-resource languages (from $\kappa = 0.58$ to $\kappa = 0.76$), though a gap remains.
We recommend supplementary human review for high-stakes assessments in low-resource languages.

% ============================================================
% 8. Ablation Studies
% ============================================================