hanzoai/open-instruct
2
0
Fork 0
mirror of https://github.com/hanzoai/open-instruct.git synced 2026-10-30 18:26:32 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
main
open-instruct/docs/data/preference-data.md
T
Nathan LambertandGitHub 4a95122db0 More preference datasets (#337) 
* add helpsteer2

* add initial nectar

* add nectar

* docs and style
2024-09-09 10:12:47 -07:00

11 KiB
Raw Permalink Blame History

Current preference datasets

To build all the datasets at once (use this carefully), run:

sh scripts/data/preferences/prepare_all.sh

Chat

Maintained here

First, older popular datasets. Build these datasets (a subset only) with:

python scripts/data/preferences/webgpt.py --push_to_hub --hf_entity=ai2-adapt-dev
python scripts/data/preferences/hh-harmless.py --push_to_hub --hf_entity=ai2-adapt-dev
python scripts/data/preferences/hh-helpful.py --push_to_hub --hf_entity=ai2-adapt-dev

Next, Nvidia's recent HelpSteer 2. They are created with:

python scripts/data/preferences/helpsteer2.py --push_to_hub --min_score 2.5 --hf_entity=ai2-adapt-dev
python scripts/data/preferences/helpsteer2.py --push_to_hub --min_score 2 --hf_entity=ai2-adapt-dev
python scripts/data/preferences/helpsteer2.py --push_to_hub --min_score 2 --hf_entity=ai2-adapt-dev --aspects_to_ignore verbosity
python scripts/data/preferences/helpsteer2.py --push_to_hub --hf_entity=ai2-adapt-dev --aspects_to_ignore verbosity

The binarization weighting that Nvidia recommends can be used with:

python scripts/data/preferences/helpsteer2_nvidia.py --push_to_hub --hf_entity ai2-adapt-dev

Some examples include:

Also, specific splits of Nectar (randomly binarized from top 3 completions and a bottom completion) are included with:

python scripts/data/preferences/nectar.py --push_to_hub --hf_entity ai2-adapt-dev
python scripts/data/preferences/nectar.py --push_to_hub --subset anthropic-hh --hf_entity ai2-adapt-dev
python scripts/data/preferences/nectar.py --push_to_hub --deduplication --hf_entity ai2-adapt-dev

The default split is lmsys-chat-1m. The last example is called "deduplication" due to potential overlap with UltraFeedback, given they source from the same underlying dataset. Basic tests showed they did not use the same prompts, but slight modifications could've occured.

Stored on HF

UltraFeedback Replication

The current replications have fewer prompts than the original. These are built by splitting the original and recreating completions. We are working on merging them.

Build these datasets with:

python scripts/data/preferences/ultrafeedback.py --push_to_hub --hf_entity=ai2-adapt-dev

The master version of the UltraFeedback pipeline replication can be found here: ai2-adapt-dev/ultrafeedback-pipeline-replication

UltraFeedback variants explore different combinations of prompt sources, model diversity, sampling methods, and prompt templates:

UltraInteract Variants

Build these datasets with:

python scripts/data/preferences/ultrainteract.py --push_to_hub --hf_entity=ai2-adapt-dev

Split by category and by selecting the longest conversations per prompt or a random length per prompt. From UltraInteract_pair.

Tulu 2.5 Data

Build these datasets with:

python scripts/data/preferences/split_tulu2.5_prefs.py --push_to_hub --hf_entity=ai2-adapt-dev

Split from this dataset for easier mixing:

Reference in New Issue View Git Blame Copy Permalink