mirror of
https://github.com/hanzoai/open-instruct.git
synced 2026-10-30 18:26:32 +00:00
Costa HuangandGitHub
6c6193b7cd
* Add optional r1-style thinking reward * quick change * test * quick change * push latest change * fix * tested it to work * Push changes * push
756 B
756 B
Grouped Relative Policy Optimization (GRPO)
GRPO is an online RL method used in DeepSeek R1 paper and its first appearance is in DeepSeekMath
open_instruct/grpo_vllm_thread_ray_gtrl.py contains an implementation of GRPO.
Get started
Here is a command to run GRPO on the Llama3.1-8b on ai2-adapt-dev/rlvr_gsm8k_zs, which is simply a zero-shot version of the RLVR GSM8K dataset.
bash scripts/train/rlvr/grpo_llama3.1-8b.sh
The results look quite reasonable: with format score, score all going up, KL not exploding, and sequence length seems stable (at least at first)
