mirror of
https://github.com/hanzoai/open-instruct.git
synced 2026-11-03 02:13:07 +00:00
Jacob MorrisonandGitHub
30e67fd278
Adding the ability to submit multinode jobs, plus connecting the data mixer to finetune.py.
After this is submitted and the image is rebuilt, you can run multinode experiments like this:
`python scripts/submit_finetune_job.py --default_beaker_config configs/beaker_configs/default_finetune_multinode.yaml --config configs/train_configs/sft/tulu3_8b_preview_mix_v3.1.yaml --cluster ai2/pluto-cirrascale --workspace ai2/tulu-3-dev --num_nodes 4`
This will launch a 4-node experiment on `ai2/pluto-cirrascale`, with the config defined in `configs/train_configs/sft/tulu3_8b_preview_mix_v3.1.yaml`. The config and `submit_finetune_job.py` will mostly take care of things for you, but you need to make sure the value for `gradient_accumulation_steps` is set to the value you want given your intended batch size and number of GPUs requested (8 * num_nodes). Also, in `default_finetune_multinode.yaml`, `WANDB_API_KEY` and `HF_TOKEN` must be set to your personal secret values; the values in the config are placeholders.
Additionally, this now supports mixing datasets on the fly. Inside of your SFT config, you can define a data mixture like so:
```
dataset_mixer:
HuggingFaceH4/no_robots: 9500
ai2-adapt-dev/metamath-qa-reformat: 100000
ai2-adapt-dev/codefeedback-single-turn-reformat: 156526
```
This will take the requested number of instances from each dataset and create a training file before launching your finetuning job. This training file will also be saved to the beaker dataset created by your experiment, so you can inspect it later if you want. If you want to add a dataset that isn't in a supported format, you need to add a function here to handle it: https://github.com/allenai/open-instruct/blob/main/open_instruct/utils.py#L215.
35 lines
1.5 KiB
YAML
35 lines
1.5 KiB
YAML
# Note, the exact model was trained on TPUs in a different repo
|
|
# Example file for mixing, use exact tulu 2 mixture for replication allenai/tulu-v2-sft-mixture
|
|
model_name_or_path: meta-llama/Llama-2-7b-hf
|
|
model_revision: main
|
|
use_flash_attn: true
|
|
tokenizer_name: meta-llama/Llama-2-7b-hf
|
|
use_slow_tokenizer: true
|
|
dataset_mixer:
|
|
natolambert/tulu-v2-sft-mixture-flan: 50000
|
|
natolambert/tulu-v2-sft-mixture-cot: 49747
|
|
allenai/openassistant-guanaco-reformatted: 7708 # not exact subset
|
|
Vtuber-plan/sharegpt-cleaned: 114046 # original https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered
|
|
vicgalle/alpaca-gpt4: 20000
|
|
HuggingFaceH4/CodeAlpaca_20K: 18000 # original uses https://github.com/sahil280114/codealpaca
|
|
natolambert/tulu-v2-sft-mixture-lima: 1018 # original has 1030
|
|
WizardLMTeam/WizardLM_evol_instruct_V2_196k: 30000
|
|
Open-Orca/OpenOrca: 30000
|
|
natolambert/tulu-v2-sft-mixture-science: 7468 # original data slightly different
|
|
max_seq_length: 4096 # Note, reduced from 8192 to fit on one GPU with DeepSpeed Stage3
|
|
preprocessing_num_workers: 128
|
|
per_device_train_batch_size: 1 # note, this is set up for 8 GPUs
|
|
gradient_accumulation_steps: 16 # effective batch size 128 for tulu 2
|
|
learning_rate: 2.0e-05
|
|
lr_scheduler_type: linear
|
|
warmup_ratio: 0.03
|
|
weight_decay: 0.0
|
|
num_train_epochs: 3
|
|
output_dir: output/tulu_v2_7b/
|
|
with_tracking: true
|
|
report_to:
|
|
- wandb
|
|
logging_steps: 1
|
|
checkpointing_steps: epoch
|
|
dataset_mix_dir: /output/
|