mirror of
https://github.com/hanzoai/gym.git
synced 2026-10-30 14:53:18 +00:00
zeekayandHanzo Dev
df832f2104
Rollouts go through a KV cache (prompt prefilled once per group, one decode step per token); cached logits match the full forward to 1e-4. GRPO runs end to end on Metal — a mixed group at step 5 gives reward 0.75, loss −0.032, gradient norm 1.19 — after the clipping reduction stopped going through f64, which Metal does not have. gym synth ports Data Designer's core: sampler, expression and LLM columns (text, structured, judge) generated per row in dependency order against any OpenAI-compatible endpoint, validators that drop or regenerate, jsonl or parquet out. gym trainjob is the Kubeflow TrainJob entry for the hosted fine-tuning API: the broker's env contract becomes a Config; METHOD=lora is served, qlora and full are refused by name. Dockerfile.cuda and image.yml build ghcr.io/hanzoai/gym on the forge runners. 84 tests. Co-authored-by: Hanzo Dev <[email protected]>
26 lines
503 B
YAML
26 lines
503 B
YAML
base_model: Qwen/Qwen3-0.6B
|
|
chat_template: qwen3
|
|
datasets:
|
|
- path: examples/mul.jsonl
|
|
type: chat_template
|
|
output_dir: outputs/qwen3-grpo
|
|
sequence_len: 64
|
|
adapter: lora
|
|
lora_r: 8
|
|
lora_alpha: 16
|
|
lora_target_modules: [q_proj, v_proj]
|
|
rl: grpo
|
|
trl:
|
|
num_generations: 4
|
|
max_completion_length: 48
|
|
reward_funcs: [numeric_match]
|
|
temperature: 1.0
|
|
micro_batch_size: 2
|
|
gradient_accumulation_steps: 1
|
|
num_epochs: 1
|
|
max_steps: 6
|
|
learning_rate: 1.0e-4
|
|
lr_scheduler: constant
|
|
logging_steps: 1
|
|
bf16: false
|