hanzoai/open-instruct
2
0
Fork 0
mirror of https://github.com/hanzoai/open-instruct.git synced 2026-10-30 18:26:32 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
main
open-instruct/scripts/data/sft_v1_v2
T
History
Yizhong WangGitHubNathan Lambert
 74069db498 More systematic and reproducible conversion of SFT datasets (#387) 
* Reorganize the data preparation scripts for tulu v1 and v2.

* Minor improvement

* Remove open_platypus_commercial subset from Daring-Anteater

* Use hard-coded examples repo.

* Fix some bugs.

* Add OpenMathInstruct.

* Add a few more v3.5.x SFT mix ablations for the cleaner datasets.

* More experiments on mixes.

* help merge

* prep for merge

* reapply changes

* fix naming

---------

Co-authored-by: Nathan Lambert <[email protected]>
2024-11-11 14:02:18 -08:00
..
get_statistics.sh
More systematic and reproducible conversion of SFT datasets (#387)
2024-11-11 14:02:18 -08:00
prepare_science_data.py
More systematic and reproducible conversion of SFT datasets (#387)
2024-11-11 14:02:18 -08:00
prepare_tulu_v1_v2.sh
More systematic and reproducible conversion of SFT datasets (#387)
2024-11-11 14:02:18 -08:00
reformat_datasets.py
More systematic and reproducible conversion of SFT datasets (#387)
2024-11-11 14:02:18 -08:00
resample_flan_v2.py
More systematic and reproducible conversion of SFT datasets (#387)
2024-11-11 14:02:18 -08:00
split_sharegpt_conversations.py
More systematic and reproducible conversion of SFT datasets (#387)
2024-11-11 14:02:18 -08:00