14 KiB
name, description
| name | description |
|---|---|
| dspy-optimizers | Optimizing DSPy programs with teleprompters, BootstrapFewShot, MIPROv2, and compilation |
DSPy Optimizers
Scope: Teleprompters, BootstrapFewShot, MIPROv2, COPRO, compilation, metrics Lines: ~440 Last Updated: 2025-10-25
When to Use This Skill
Activate this skill when:
- Improving DSPy program accuracy with automatic optimization
- Compiling modules with training examples
- Generating optimized few-shot demonstrations
- Tuning prompts and LM weights programmatically
- Bootstrapping training data from small seed sets
- Evaluating and iterating on DSPy programs
Core Concepts
What are Optimizers (Teleprompters)?
Definition: Optimizers automatically improve DSPy programs by finding better prompts, demonstrations, and parameters
Original term: "Teleprompters" (legacy name, now called optimizers)
Purpose:
- Auto-tune: Find optimal prompts without manual engineering
- Few-shot learning: Generate effective demonstrations
- Weight tuning: Fine-tune model weights (if supported)
- Metric optimization: Maximize custom metrics
Key insight: Instead of manually tweaking prompts, define a metric and let DSPy optimize
Optimization Process
Compilation workflow:
- Define metric: Function that scores predictions
- Prepare dataset: Training/validation examples
- Choose optimizer: BootstrapFewShot, MIPROv2, etc.
- Compile:
optimizer.compile(module, trainset=data, metric=metric) - Evaluate: Test on held-out data
Common Optimizers
BootstrapFewShot: Generate few-shot examples
- Simplest optimizer
- Creates demonstrations from successful predictions
- Works with any module
- Fast, effective
MIPROv2: Multi-prompt instruction optimization
- Optimizes instructions AND demonstrations
- More sophisticated than BootstrapFewShot
- Better for complex tasks
- Slower but higher quality
COPRO: Coordinate prompt optimization
- Optimizes prompts across multiple modules
- Good for multi-stage pipelines
- Considers module interactions
BootstrapFinetune: Generate data for fine-tuning
- Creates training data for model fine-tuning
- Requires fine-tunable model
- Best accuracy but most expensive
Patterns
Pattern 1: Basic BootstrapFewShot
import dspy
# Configure LM
lm = dspy.LM("openai/gpt-4o-mini")
dspy.configure(lm=lm)
# Define module
qa = dspy.ChainOfThought("question -> answer")
# Prepare training data
trainset = [
dspy.Example(question="What is the capital of France?", answer="Paris").with_inputs("question"),
dspy.Example(question="What is 2+2?", answer="4").with_inputs("question"),
dspy.Example(question="Who wrote Hamlet?", answer="William Shakespeare").with_inputs("question"),
]
# Define metric (validation function)
def validate_answer(example, pred, trace=None):
"""Return True if prediction is correct."""
return example.answer.lower() == pred.answer.lower()
# Create optimizer
optimizer = dspy.BootstrapFewShot(metric=validate_answer)
# Compile (optimize) the module
compiled_qa = optimizer.compile(
student=qa,
trainset=trainset,
)
# Use optimized module
result = compiled_qa(question="What is the capital of Germany?")
print(result.answer)
What happens:
- Runs module on training examples
- Keeps successful predictions as demonstrations
- Optimized module uses these as few-shot examples
- Better accuracy on similar tasks
Pattern 2: MIPROv2 Optimization
import dspy
# More sophisticated optimization
qa = dspy.ChainOfThought("question, context -> answer")
trainset = [
dspy.Example(
question="What is DSPy?",
context="DSPy is a framework for programming language models.",
answer="A framework for programming language models"
).with_inputs("question", "context"),
# ... more examples
]
devset = [
# Validation examples (separate from training)
]
# Define metric with partial credit
def qa_metric(example, pred, trace=None):
"""Score prediction quality (0.0 to 1.0)."""
answer = pred.answer.lower()
gold = example.answer.lower()
# Exact match
if answer == gold:
return 1.0
# Partial match (contains key words)
if gold in answer or answer in gold:
return 0.5
return 0.0
# MIPROv2 optimizer
optimizer = dspy.MIPROv2(
metric=qa_metric,
auto="light", # or "medium", "heavy" for more optimization
num_candidates=10, # Number of prompt variations to try
)
# Compile with training and validation sets
compiled_qa = optimizer.compile(
student=qa,
trainset=trainset,
valset=devset, # Used for selecting best prompts
max_bootstrapped_demos=4, # Max few-shot examples
max_labeled_demos=2, # Max examples from trainset
)
# Optimized module now has better prompts and demonstrations
result = compiled_qa(question="What is DSPy?", context="...")
Benefits over BootstrapFewShot:
- Optimizes instruction text, not just examples
- Tries multiple prompt variations
- Uses validation set to pick best prompts
- Higher quality but slower
Pattern 3: Custom Metric Functions
import dspy
# Metric with detailed scoring
def complex_metric(example, pred, trace=None):
"""Multi-factor metric for comprehensive evaluation."""
score = 0.0
# Factor 1: Answer correctness (0.5 weight)
if example.answer.lower() in pred.answer.lower():
score += 0.5
# Factor 2: Confidence (0.2 weight)
if hasattr(pred, 'confidence'):
try:
conf = float(pred.confidence)
if 0.7 <= conf <= 1.0:
score += 0.2
except:
pass
# Factor 3: Reasoning quality (0.3 weight)
if hasattr(pred, 'reasoning'):
if len(pred.reasoning.split()) > 10: # Detailed reasoning
score += 0.15
if "because" in pred.reasoning.lower(): # Causal reasoning
score += 0.15
return score
# Use metric with optimizer
optimizer = dspy.BootstrapFewShot(
metric=complex_metric,
metric_threshold=0.7, # Only keep examples scoring >= 0.7
)
When to use:
- Multi-factor quality assessment
- Domain-specific requirements
- Nuanced evaluation beyond exact match
Pattern 4: Optimizing Multi-Module Pipelines
import dspy
class RAGPipeline(dspy.Module):
"""Retrieval-Augmented Generation pipeline."""
def __init__(self):
super().__init__()
self.retrieve = dspy.Retrieve(k=3)
self.generate = dspy.ChainOfThought("context, question -> answer")
def forward(self, question):
passages = self.retrieve(question).passages
context = "\n".join(passages)
return self.generate(context=context, question=question)
# Prepare data
trainset = [
dspy.Example(
question="What is DSPy?",
answer="A framework for programming language models"
).with_inputs("question"),
]
# Metric for RAG
def rag_metric(example, pred, trace=None):
# Check if answer is in prediction
return example.answer.lower() in pred.answer.lower()
# COPRO optimizer (coordinates across modules)
optimizer = dspy.COPRO(
metric=rag_metric,
breadth=10, # Number of prompt variations per module
depth=3, # Optimization iterations
)
# Compile entire pipeline
pipeline = RAGPipeline()
compiled_pipeline = optimizer.compile(
student=pipeline,
trainset=trainset,
)
When to use:
- Multi-stage pipelines (RAG, agents, etc.)
- Need to optimize module interactions
- Complex workflows
Pattern 5: BootstrapFinetune (Model Fine-tuning)
import dspy
# Requires fine-tunable model
teacher = dspy.ChainOfThought("question -> answer")
# Large training dataset
trainset = [
# Hundreds or thousands of examples
]
# Generate fine-tuning data
optimizer = dspy.BootstrapFinetune(
metric=lambda ex, pred, trace: ex.answer == pred.answer,
)
# Creates fine-tuning dataset
compiled_teacher = optimizer.compile(
student=teacher,
trainset=trainset,
target="openai/gpt-4o-mini-finetuned", # Your fine-tuned model
)
# Now use fine-tuned model
finetuned_lm = dspy.LM("openai/gpt-4o-mini-finetuned")
dspy.configure(lm=finetuned_lm)
When to use:
- Have large training dataset (1000+ examples)
- Need maximum accuracy
- Can afford fine-tuning cost
- Task is well-defined and stable
Pattern 6: Evaluation-Driven Optimization
import dspy
from dspy.evaluate import Evaluate
# Define program
program = dspy.ChainOfThought("question -> answer")
# Prepare datasets
trainset = [...] # Training examples
valset = [...] # Validation examples
testset = [...] # Test examples (never use for optimization!)
# Define metric
def accuracy(example, pred, trace=None):
return example.answer.lower() == pred.answer.lower()
# Baseline evaluation
evaluator = Evaluate(
devset=testset,
metric=accuracy,
num_threads=4, # Parallel evaluation
)
baseline_score = evaluator(program)
print(f"Baseline accuracy: {baseline_score:.2%}")
# Optimize
optimizer = dspy.MIPROv2(metric=accuracy, auto="medium")
optimized = optimizer.compile(
student=program,
trainset=trainset,
valset=valset,
)
# Evaluate optimized version
optimized_score = evaluator(optimized)
print(f"Optimized accuracy: {optimized_score:.2%}")
print(f"Improvement: {(optimized_score - baseline_score):.2%}")
Best practice workflow:
- Baseline evaluation
- Optimize on train/val sets
- Final evaluation on test set
- Compare before/after
Pattern 7: Saving and Loading Compiled Programs
import dspy
# Compile program
optimizer = dspy.MIPROv2(metric=my_metric)
compiled = optimizer.compile(student=program, trainset=trainset)
# Save compiled program
compiled.save("optimized_qa.json")
# Later, load compiled program
loaded_program = dspy.ChainOfThought("question -> answer")
loaded_program.load("optimized_qa.json")
# Use loaded program (has optimized prompts/demos)
result = loaded_program(question="What is DSPy?")
When to use:
- Production deployment
- Avoid re-optimizing every run
- Share optimized programs with team
- Version control for prompts
Pattern 8: Iterative Optimization
import dspy
program = dspy.ChainOfThought("question -> answer")
# Start with simple optimizer
optimizer1 = dspy.BootstrapFewShot(metric=my_metric)
v1 = optimizer1.compile(student=program, trainset=trainset)
score_v1 = evaluate(v1)
# If not good enough, try more sophisticated optimizer
if score_v1 < 0.8:
optimizer2 = dspy.MIPROv2(metric=my_metric, auto="medium")
v2 = optimizer2.compile(student=v1, trainset=trainset, valset=valset)
score_v2 = evaluate(v2)
# If still not good enough, use heavy optimization
if score_v2 < 0.9:
optimizer3 = dspy.MIPROv2(metric=my_metric, auto="heavy")
v3 = optimizer3.compile(student=v2, trainset=trainset, valset=valset)
score_v3 = evaluate(v3)
# Use best version
best = max([(score_v1, v1), (score_v2, v2), (score_v3, v3)], key=lambda x: x[0])
print(f"Best score: {best[0]:.2%}")
final_program = best[1]
Strategy: Progressive optimization
- Start fast and simple
- Increase sophistication if needed
- Compare results at each stage
Quick Reference
Optimizer Comparison
| Optimizer | Speed | Quality | Use Case |
|---|---|---|---|
| BootstrapFewShot | Fast | Good | Simple tasks, quick iteration |
| MIPROv2 | Medium | Better | Complex tasks, production |
| COPRO | Slow | Best (pipelines) | Multi-module workflows |
| BootstrapFinetune | Slowest | Best (single module) | Large datasets, max accuracy |
Optimization Workflow
1. Define metric function
2. Prepare trainset (and valset)
3. Choose optimizer
4. Compile: optimizer.compile(student, trainset)
5. Evaluate on testset
6. Save optimized program
Metric Function Template
def my_metric(example, pred, trace=None):
"""
Args:
example: Ground truth example
pred: Model prediction
trace: Execution trace (optional)
Returns:
float: Score between 0.0 and 1.0
OR
bool: True if correct, False otherwise
"""
# Implement scoring logic
return score
Best Practices
✅ DO: Start with BootstrapFewShot for quick iteration
✅ DO: Use separate train/val/test sets
✅ DO: Define clear, measurable metrics
✅ DO: Save optimized programs for production
✅ DO: Evaluate baseline before optimizing
❌ DON'T: Optimize on test set (data leakage!)
❌ DON'T: Use tiny training sets (<10 examples)
❌ DON'T: Ignore metric design (affects optimization)
❌ DON'T: Re-optimize in production (pre-compile)
Anti-Patterns
❌ Optimizing on test set: Data leakage, inflated scores
# Bad
optimizer.compile(student=program, trainset=testset) # WRONG!
✅ Use separate train/val/test splits:
# Good
compiled = optimizer.compile(student=program, trainset=trainset, valset=valset)
final_score = evaluate(compiled, testset) # Honest evaluation
❌ Vague metrics: Optimizer can't improve what it can't measure
# Bad
def bad_metric(example, pred, trace=None):
return True # Always returns True, useless!
✅ Define clear, discriminative metrics:
# Good
def good_metric(example, pred, trace=None):
return example.answer.lower() == pred.answer.lower()
❌ Not evaluating baseline: Don't know if optimization helped
# Bad - skip straight to optimization
optimized = optimizer.compile(...) # Did it improve? Unknown!
✅ Always evaluate baseline first:
# Good
baseline_score = evaluate(program, testset)
optimized = optimizer.compile(program, trainset)
optimized_score = evaluate(optimized, testset)
print(f"Improvement: {optimized_score - baseline_score}")
❌ Tiny training sets: Not enough signal for optimization
# Bad
trainset = [example1, example2] # Only 2 examples
✅ Use adequate training data:
# Good
trainset = [...] # 50+ examples for BootstrapFewShot, 500+ for MIPROv2
Related Skills
dspy-evaluation.md- Evaluating DSPy programs with metricsdspy-modules.md- Modules to optimizedspy-signatures.md- Signatures affected by optimizationllm-dataset-preparation.md- Preparing training datasets
Last Updated: 2025-10-25 Format Version: 1.0 (Atomic)