mirror of
https://github.com/hanzoai/ml.git
synced 2026-11-19 23:50:51 +00:00
Implement Group Relative Policy Optimization (GRPO) in native Rust using
pure candle autograd + hanzo_nn::AdamW (no external trainer framework).
Algorithm ported from zooai-gym's GRPO trainer (trainer.py/args.py).
New module hanzo-training/src/grpo/:
- config.rs GrpoConfig (group_size, temperature, scale_rewards,
advantage_eps, optional kl_beta, lr, ...) + SamplingMode.
- reward.rs Reward trait + Completion; example rewards: LengthTargetReward,
TokenMatchReward, ExactMatchReward.
- policy.rs Policy trait (trainable_vars / sample_group / sequence_logprob)
— the seam where a real transformer plugs in — plus Sampler.
- sampler.rs LogitsSampler wrapping the real LogitsProcessor (argmax/temp).
- math.rs Real GRPO math: group-relative advantages
A_i = (r_i - mean)/(std + eps) and PG loss
-mean(A_i * mean_logprob_i), with an optional sequence-level
KL (k3 estimator) penalty against a frozen reference policy.
- metrics.rs GrpoMetrics (loss/pg/kl, reward, reward_std, lengths).
- trainer.rs GrpoTrainer::step(prompts) -> GrpoMetrics: rollout -> reward ->
advantage -> differentiable loss -> AdamW step.
Smoke test (grpo::toy_tests): a tiny contextual-bandit policy over a small
vocab learns a TokenMatchReward end to end through the real GRPO/autograd/
optimizer path; mean reward rises from ~0.7 to the ceiling (4.0). A second
test exercises the KL branch and asserts finite, non-negative KL. Plus unit
tests for the advantage/loss/KL math. examples/grpo_toy.rs is a runnable demo.
Workspace: hanzo-training was previously excluded and did not build; moved it
into workspace members and added the missing `chrono` workspace dependency so
`cargo build -p hanzo-training` resolves. No other crates' sources changed.