mirror of
https://github.com/hanzoai/open-instruct.git
synced 2026-10-30 18:26:32 +00:00
74069db498
* Reorganize the data preparation scripts for tulu v1 and v2. * Minor improvement * Remove open_platypus_commercial subset from Daring-Anteater * Use hard-coded examples repo. * Fix some bugs. * Add OpenMathInstruct. * Add a few more v3.5.x SFT mix ablations for the cleaner datasets. * More experiments on mixes. * help merge * prep for merge * reapply changes * fix naming --------- Co-authored-by: Nathan Lambert <[email protected]>
8 lines
663 B
Bash
Executable File
8 lines
663 B
Bash
Executable File
#!/bin/bash
|
|
# ["super_ni", "cot", "flan_v2", "self_instruct", "unnatural_instructions", "stanford_alpaca", "dolly", "sharegpt", "code_alpaca", "gpt4_alpaca", "baize", "oasst1"]
|
|
# Run with scripts/data/sft_v1_v2/get_statistics_tulu_v2.sh
|
|
# for every dataset, get the statistics
|
|
for dataset in super_ni cot flan_v2 self_instruct unnatural_instructions stanford_alpaca dolly sharegpt code_alpaca gpt4_alpaca baize oasst1 lima wizardlm open_orca; do
|
|
echo "Getting statistics for $dataset..."
|
|
python scripts/data/get_statistics.py --data_path data/processed/${dataset}/${dataset}_data.jsonl --save_path data/processed/${dataset}/${dataset}_statistics.json
|
|
done |