Files
zeekay e9f4c73c3b Add trainer-free GRPO loop to hanzo-training
Implement Group Relative Policy Optimization (GRPO) in native Rust using
pure candle autograd + hanzo_nn::AdamW (no external trainer framework).
Algorithm ported from zooai-gym's GRPO trainer (trainer.py/args.py).

New module hanzo-training/src/grpo/:
- config.rs    GrpoConfig (group_size, temperature, scale_rewards,
               advantage_eps, optional kl_beta, lr, ...) + SamplingMode.
- reward.rs    Reward trait + Completion; example rewards: LengthTargetReward,
               TokenMatchReward, ExactMatchReward.
- policy.rs    Policy trait (trainable_vars / sample_group / sequence_logprob)
               — the seam where a real transformer plugs in — plus Sampler.
- sampler.rs   LogitsSampler wrapping the real LogitsProcessor (argmax/temp).
- math.rs      Real GRPO math: group-relative advantages
               A_i = (r_i - mean)/(std + eps) and PG loss
               -mean(A_i * mean_logprob_i), with an optional sequence-level
               KL (k3 estimator) penalty against a frozen reference policy.
- metrics.rs   GrpoMetrics (loss/pg/kl, reward, reward_std, lengths).
- trainer.rs   GrpoTrainer::step(prompts) -> GrpoMetrics: rollout -> reward ->
               advantage -> differentiable loss -> AdamW step.

Smoke test (grpo::toy_tests): a tiny contextual-bandit policy over a small
vocab learns a TokenMatchReward end to end through the real GRPO/autograd/
optimizer path; mean reward rises from ~0.7 to the ceiling (4.0). A second
test exercises the KL branch and asserts finite, non-negative KL. Plus unit
tests for the advantage/loss/KL math. examples/grpo_toy.rs is a runnable demo.

Workspace: hanzo-training was previously excluded and did not build; moved it
into workspace members and added the missing `chrono` workspace dependency so
`cargo build -p hanzo-training` resolves. No other crates' sources changed.
2026-06-01 17:20:12 -07:00
..