synth/games/tasks/dyck_language_errors/README.md

Dyck Language Errors (Bracket Closure Error Detection)

Task Description

The Dyck language consists of correctly paired brackets. In this task, we need to identify the position of the first error in a bracket sequence, or determine if the sequence is valid.

Rules

• The task involves 4 different types of brackets: (), [], {}, <>
• Each game data contains a sequence composed of n_types kinds of brackets (1 ≤ n_types ≤ 4)
• Brackets must be correctly paired: an opening bracket must match the corresponding closing bracket, and the order must be correct
• If the sequence is valid (all brackets are correctly paired), the answer is an empty string ""
• If the sequence is invalid, the answer is the position of the first error (index starts from 1)

Valid Sequence Examples

()            - Answer: "" (empty string)
(())          - Answer: "" (empty string)
()()          - Answer: "" (empty string)
(()())        - Answer: "" (empty string)
({[]})        - Answer: "" (empty string)
<[]()>        - Answer: "" (empty string)

Invalid Sequence Examples

)(          - Error position: 1 (closing bracket at the start)
(()         - Error position: 4 (missing closing bracket)
([)]        - Error position: 3 (incorrect closing order)
{[}>        - Error position: 3 (incorrect closing type)

Code Logic

Generation Logic

The DyckLanguageErrors class implements the generation of bracket sequences:

1. Randomly select n_types kinds of bracket pairs
2. Randomly decide to generate a valid or invalid sequence
3. For valid sequences, use a stack structure to ensure all brackets are correctly paired
4. For invalid sequences, start from a valid sequence and introduce errors in one of the following ways:
   • Replace a closing bracket (does not match the opening bracket)
   • Remove a closing bracket (leaving an unmatched opening bracket)
   • Add an extra closing bracket (at an incorrect position)
5. Use prompt templates to generate problem descriptions (randomly in Chinese or English)

Validation Logic

The validation process determines whether the model's answer is correct:

1. For valid sequences, the correct answer is an empty string "" (backward compatible with "合法" and "valid")
2. For invalid sequences, the correct answer is the error position number
3. Extract the core part of the answer from the model's response (ignore explanatory text)

Answer Extraction Logic

• For valid sequences, the model should output an empty string ""
• For invalid sequences, the model should directly output the error position number
• The system is backward compatible with various answer formats (Chinese/English descriptions, "合法"/"valid", etc.)
• The extraction logic prioritizes the simplest answer format but still supports more complex responses

Usage

# Generate game data (single set of parameters)
python -m games.tasks.dyck_language_errors.scripts.dyck_language_errors --num_of_data 100 --n_types 3 --total_length 20

# Use the run.sh script to generate multiple sets of data with preset parameters
./games/tasks/dyck_language_errors/run.sh --generate_all

Parameter Description

Data Generation Parameters

• num_of_data: Number of game data to generate
• n_types: Number of types of bracket pairs (1-4)
• total_length: Total length of the bracket sequence
• max_attempts: Maximum number of attempts to generate each data item

run.sh Script Parameters

• --generate_all: Generate multiple sets of preset parameter data, 1000 items per set, including:
  • n_types=3, total_length=10
  • n_types=3, total_length=15
  • n_types=4, total_length=20
  • n_types=4, total_length=30
• Single parameter usage: same as directly calling the python script with parameters