17 KiB
DeltaSoup - Byzantine-Robust Community Learning
Category: Zoo Gym Training Methods Skill Level: Advanced Prerequisites: Understanding of federated learning, Byzantine fault tolerance, differential privacy Related Skills: bitdelta.md, training-free-grpo.md, ../hanzo/hanzo-gym.md
Overview
DeltaSoup enables community-driven model improvement through Byzantine-robust aggregation of personalized model deltas. It allows thousands of users to contribute their fine-tuned improvements while filtering malicious contributions and protecting individual privacy through differential privacy.
Core Innovation: Byzantine-robust averaging + differential privacy + quality-based rewards for community-driven AI model evolution.
Why DeltaSoup?
The Problem: Community AI Improvement
- Centralized Training: Single organization controls model evolution
- No User Contribution: Users can't contribute improvements back
- Malicious Actors: Bad actors can poison the model with adversarial fine-tuning
- Privacy Concerns: Individual contributions can be reverse-engineered
The DeltaSoup Solution
- Decentralized: Community-driven model improvement
- Byzantine-Robust: Filters malicious contributions automatically
- Privacy-Preserving: Differential privacy protects individual data
- Incentive-Aligned: Contributors earn rewards based on quality
How DeltaSoup Works
1. User Contributions
Users fine-tune the base model on their own data:
# User Alice fine-tunes model on her medical diagnosis data
alice_model = fine_tune(base_model, alice_data)
# User Bob fine-tunes model on his legal case analysis
bob_model = fine_tune(base_model, bob_data)
# User Charlie fine-tunes model on his financial reports
charlie_model = fine_tune(base_model, charlie_data)
2. Delta Extraction
Extract the delta (weight difference) from each user:
# Calculate deltas
alice_delta = alice_model.weights - base_model.weights
bob_delta = bob_model.weights - base_model.weights
charlie_delta = charlie_model.weights - base_model.weights
3. Byzantine-Robust Aggregation
Filter malicious contributions using Byzantine-robust averaging:
# Assume Charlie is malicious (extreme delta values)
deltas = [alice_delta, bob_delta, charlie_delta]
# Byzantine-robust averaging (Krum, Multi-Krum, or Trimmed Mean)
aggregated_delta = byzantine_robust_average(deltas)
# Charlie's malicious delta is filtered out
# Only Alice and Bob's deltas are used
4. Differential Privacy
Add noise to protect individual contributions:
# Add calibrated noise to aggregated delta
noise = gaussian_noise(epsilon=1.0, delta=1e-5)
private_delta = aggregated_delta + noise
# Individual contributions cannot be reverse-engineered
5. Model Update
Apply aggregated delta to base model:
# Create new community-improved model
improved_model = base_model + private_delta
# This becomes the new base for next round
Aggregation Methods
DeltaSoup supports multiple Byzantine-robust aggregation methods:
1. Byzantine-Robust (Krum) - Default
Best for: Filtering malicious contributions Assumes: < 30% Byzantine actors
from zoo.gym import DeltaSoup, AggregationMethod, DeltaSoupConfig
config = DeltaSoupConfig(
method=AggregationMethod.BYZANTINE_ROBUST,
byzantine_threshold=0.3 # Max 30% malicious
)
soup = DeltaSoup(config)
How it works:
- Calculate pairwise distances between all deltas
- For each delta, sum distances to k nearest neighbors
- Select delta with smallest sum (most "central")
- Repeat for multiple deltas (Multi-Krum)
2. Trimmed Mean
Best for: Robust averaging with fewer assumptions Assumes: Outliers exist but are minority
config = DeltaSoupConfig(
method=AggregationMethod.TRIMMED_MEAN,
trim_percent=0.2 # Trim 20% from each tail
)
How it works:
- Sort delta values for each weight
- Remove top 20% and bottom 20%
- Average remaining 60%
3. Weighted Mean
Best for: Reputation-based aggregation Assumes: Users have established reputation scores
config = DeltaSoupConfig(
method=AggregationMethod.WEIGHTED_MEAN,
validate_contributions=True # Validate quality
)
How it works:
- Calculate reputation score for each user
- Weight deltas by reputation
- Compute weighted average
Differential Privacy
DeltaSoup protects individual contributions through differential privacy:
Privacy Budget (ε, δ)
config = DeltaSoupConfig(
differential_privacy=True,
privacy_epsilon=1.0, # Privacy budget
privacy_delta=1e-5, # Failure probability
noise_scale=0.01 # Noise scale
)
Privacy guarantees:
- ε = 0.1: Very strong privacy (high noise)
- ε = 1.0: Strong privacy (moderate noise, default)
- ε = 10.0: Weak privacy (low noise)
Noise Calibration
# Noise scale is calibrated to privacy budget
noise_scale = sensitivity / (epsilon * sqrt(2 * log(1.25 / delta)))
# Example:
# sensitivity = 1.0 (max change from one user)
# epsilon = 1.0
# delta = 1e-5
# noise_scale ≈ 0.01
Contributor Profiles and Reputation
DeltaSoup tracks contributor quality and reputation:
Profile Tracking
from zoo.gym import ContributorProfile
# Create profile for user
profile = ContributorProfile(user_id="alice")
# Update reputation after contribution
profile.update_reputation(quality_score=0.85)
# Get aggregation weight
weight = profile.get_weight() # Returns reputation score
Reputation Scoring
# Exponential moving average (EMA) of quality scores
alpha = 0.1 # Learning rate
new_reputation = (1 - alpha) * old_reputation + alpha * quality_score
# Example:
# old_reputation = 0.8
# quality_score = 0.9
# new_reputation = 0.9 * 0.8 + 0.1 * 0.9 = 0.81
Quality Validation
config = DeltaSoupConfig(
validate_contributions=True,
quality_threshold=0.8, # Min quality score
diversity_bonus=0.1 # Bonus for diverse contributions
)
# Contributions below quality threshold are rejected
Reward System
Contributors earn rewards based on contribution quality:
Reward Configuration
config = DeltaSoupConfig(
enable_rewards=True,
reward_pool=1000.0, # Total reward pool
quality_weight=0.7, # 70% weight for quality
participation_weight=0.3 # 30% weight for participation
)
Reward Calculation
# Quality score (0-1): How much the contribution improves the model
quality_score = evaluate_contribution(delta, validation_set)
# Participation score (0-1): Consistency of contributions
participation_score = contribution_count / max_contributions
# Total score
total_score = (
quality_weight * quality_score +
participation_weight * participation_score
)
# Reward proportional to total score
reward = (total_score / sum(all_scores)) * reward_pool
Example Rewards
# 3 contributors:
# Alice: quality=0.9, participation=0.8
# Bob: quality=0.7, participation=1.0
# Charlie: quality=0.5, participation=0.6
# Total scores:
# Alice: 0.7 * 0.9 + 0.3 * 0.8 = 0.87
# Bob: 0.7 * 0.7 + 0.3 * 1.0 = 0.79
# Charlie: 0.7 * 0.5 + 0.3 * 0.6 = 0.53
# Rewards (reward_pool = 1000):
# Alice: (0.87 / 2.19) * 1000 = 397
# Bob: (0.79 / 2.19) * 1000 = 361
# Charlie: (0.53 / 2.19) * 1000 = 242
Installation
# Zoo Gym includes DeltaSoup by default
git clone https://github.com/zooai/gym.git
cd gym
pip install -e .
# Or via pip
pip install zoo-gym
Usage Examples
Basic DeltaSoup Workflow
from zoo.gym import DeltaSoup, DeltaSoupConfig, AggregationMethod
# Configure DeltaSoup
config = DeltaSoupConfig(
method=AggregationMethod.BYZANTINE_ROBUST,
byzantine_threshold=0.3,
differential_privacy=True,
privacy_epsilon=1.0,
enable_rewards=True,
min_contributors=3,
quality_threshold=0.8
)
# Create soup
soup = DeltaSoup(config)
# Users contribute their improvements
soup.contribute(user_id="alice", model=model_alice)
soup.contribute(user_id="bob", model=model_bob)
soup.contribute(user_id="charlie", model=model_charlie)
# Aggregate improvements (Byzantine-robust + differential privacy)
aggregated_model = soup.aggregate()
# Distribute rewards based on quality
rewards = soup.calculate_rewards()
print(rewards) # {'alice': 397, 'bob': 361, 'charlie': 242}
# Save improved model
soup.save_aggregated_model("./community_improved_model")
Integration with BitDelta
from zoo.gym import BitDeltaConfig, DeltaSoupConfig
# Enable BitDelta compression for DeltaSoup
deltasoup_config = DeltaSoupConfig(
method=AggregationMethod.BYZANTINE_ROBUST,
use_bitdelta=True, # Compress deltas with BitDelta
compression_threshold=0.9 # 90% compression
)
# Users contribute compressed deltas (10× smaller)
soup = DeltaSoup(deltasoup_config)
soup.contribute_bitdelta(user_id="alice", delta=alice_bitdelta)
soup.contribute_bitdelta(user_id="bob", delta=bob_bitdelta)
# Aggregation works on compressed deltas
aggregated_model = soup.aggregate()
Production Community Learning
from zoo.gym import CommunityLearningServer
import asyncio
# Initialize community learning server
server = CommunityLearningServer(
base_model="Qwen/Qwen3-4B",
config=DeltaSoupConfig(
method=AggregationMethod.BYZANTINE_ROBUST,
differential_privacy=True,
enable_rewards=True,
min_contributors=10,
quality_threshold=0.8,
contribution_window=86400 # 24 hour window
)
)
# Start server
await server.start()
# Users submit contributions via API
# Server aggregates contributions every 24 hours
# Improved model is published to community
Custom Quality Validation
def custom_quality_validator(delta, base_model, validation_set):
"""Custom quality validation function"""
# Apply delta to base model
test_model = base_model + delta
# Evaluate on validation set
accuracy = evaluate(test_model, validation_set)
# Check for safety violations
jailbreak_rate = test_jailbreaks(test_model)
# Calculate quality score
quality = accuracy * (1 - jailbreak_rate)
return quality
# Use custom validator
config = DeltaSoupConfig(
validate_contributions=True,
quality_validator=custom_quality_validator
)
CLI Usage
# Create DeltaSoup aggregation
llamafactory-cli deltasoup \
--base_model Qwen/Qwen3-4B \
--contributions ./contributions/*.delta \
--method byzantine_robust \
--byzantine_threshold 0.3 \
--differential_privacy \
--privacy_epsilon 1.0 \
--output_dir ./community_improved \
--enable_rewards \
--reward_pool 1000.0
# Serve community-improved model
llamafactory-cli serve \
--model_name_or_path ./community_improved \
--template qwen3 \
--port 8080
Performance Benchmarks
Aggregation Speed (1000 Contributors, Qwen3-4B)
| Method | Aggregation Time | Throughput |
|---|---|---|
| Mean | 0.5s | 2000 contributions/s |
| Trimmed Mean | 1.2s | 833 contributions/s |
| Byzantine-Robust | 8.5s | 118 contributions/s |
| Weighted Mean | 0.8s | 1250 contributions/s |
Byzantine Attack Resistance (30% Malicious)
| Method | Attack Success Rate | Model Quality |
|---|---|---|
| Mean | 100% (poisoned) | 0% |
| Trimmed Mean (20%) | 15% | 82% |
| Byzantine-Robust | 2% | 97% |
| Weighted Mean (reputation) | 5% | 93% |
Privacy vs Quality Trade-off
| Privacy (ε) | Noise Level | Model Quality |
|---|---|---|
| 0.1 | High | 65% |
| 1.0 | Moderate | 92% |
| 10.0 | Low | 98% |
| ∞ (no privacy) | None | 100% |
Byzantine Attack Examples
Attack 1: Large Magnitude Delta
# Malicious user submits large magnitude delta
malicious_delta = 1000 * torch.randn_like(base_model.weights)
# Byzantine-robust aggregation detects and filters
# Distance to other deltas is very large
# Malicious delta is rejected
Attack 2: Adversarial Fine-Tuning
# Malicious user fine-tunes to maximize jailbreak success
malicious_model = adversarial_fine_tune(
base_model,
objective="maximize_jailbreak"
)
# Quality validation catches this
quality_score = evaluate_contribution(malicious_delta, validation_set)
# quality_score < quality_threshold → rejected
Attack 3: Coordinated Attack
# Multiple malicious users coordinate their deltas
malicious_users = ["eve", "mallory", "trudy"]
coordinated_delta = ...
# Byzantine-robust aggregation (Krum) still works if < 30%
# If malicious users > 30%, aggregation may be compromised
# Solution: Use reputation system to limit new users
Integration with Hanzo Ecosystem
DeltaSoup is fully supported across the Hanzo ecosystem:
Python SDK
from hanzo import Hanzo
from zoo.gym import DeltaSoupConfig, AggregationMethod
hanzo = Hanzo(inference_mode='local')
# Contribute user improvement
hanzo.contribute_improvement(
user_id="alice",
base_model="qwen3-4b",
improvement_data=alice_data,
config=DeltaSoupConfig(
method=AggregationMethod.BYZANTINE_ROBUST,
differential_privacy=True
)
)
# Aggregate community improvements
improved_model = hanzo.aggregate_community_improvements(
base_model="qwen3-4b",
min_contributors=10
)
Go SDK
package main
import (
"context"
"github.com/hanzoai/go-sdk"
"github.com/hanzoai/go-sdk/option"
)
func main() {
client := hanzoai.NewClient(
option.WithInferenceMode("local"),
)
// Contribute improvement
contribution, _ := client.CommunityLearning.Contribute(
context.Background(),
hanzoai.CommunityContributionParams{
UserID: hanzoai.F("alice"),
BaseModel: hanzoai.F("qwen3-4b"),
Delta: hanzoai.F(alice_delta),
},
)
// Check reward
println("Reward:", contribution.Reward)
}
Hanzo Node
# Start Hanzo Node with DeltaSoup support
hanzo-node start --enable-deltasoup --community-dir ./community
# Accept user contributions
hanzo-node community accept --user alice --delta ./alice.delta
# Aggregate improvements (daily)
hanzo-node community aggregate \
--method byzantine_robust \
--min-contributors 10 \
--output ./improved_model
# Distribute rewards
hanzo-node community rewards distribute --pool 1000.0
Security Considerations
1. Byzantine Threshold
# Byzantine-robust methods assume < 30% malicious actors
# If malicious actors > 30%, aggregation may be compromised
config = DeltaSoupConfig(
byzantine_threshold=0.3, # Assume max 30% malicious
validate_contributions=True # Additional validation
)
2. Reputation System
# Use reputation to limit impact of new/untrusted users
config = DeltaSoupConfig(
method=AggregationMethod.WEIGHTED_MEAN,
min_reputation=0.5, # Require reputation > 0.5
reputation_decay=0.9 # Reputation decays over time
)
3. Quality Validation
# Always validate contributions before aggregation
config = DeltaSoupConfig(
validate_contributions=True,
quality_threshold=0.8, # Reject contributions < 80%
safety_checks=True # Check for jailbreak attempts
)
Best Practices
1. Start with High Privacy
# Production: Start with strong privacy, relax if needed
config = DeltaSoupConfig(
differential_privacy=True,
privacy_epsilon=1.0, # Strong privacy
privacy_delta=1e-5
)
2. Use Quality Thresholds
# Always validate contribution quality
config = DeltaSoupConfig(
validate_contributions=True,
quality_threshold=0.8, # Reject low-quality contributions
diversity_bonus=0.1 # Encourage diverse improvements
)
3. Incremental Aggregation
# Aggregate incrementally (daily/weekly) rather than all at once
# This allows early detection of attacks
config = DeltaSoupConfig(
contribution_window=86400, # 24 hour window
min_contributors=10 # Minimum 10 contributors per round
)
Limitations and Considerations
1. Byzantine Threshold
- Byzantine-robust methods assume < 30% malicious actors
- If malicious > 30%, aggregation may be compromised
- Use reputation system to mitigate this
2. Privacy-Quality Trade-off
- Strong privacy (ε = 0.1) adds significant noise
- Weak privacy (ε = 10) provides little protection
- Recommended: ε = 1.0 for balance
3. Cold Start Problem
- New users have low reputation scores
- Early contributions may be underweighted
- Solution: Bootstrap with trusted contributors
Related Skills
- bitdelta.md - Per-user personalization (10× compression, integrated with DeltaSoup)
- training-free-grpo.md - RLHF without value networks (default Zoo Gym training)
- ../hanzo/hanzo-gym.md - Comprehensive Zoo Gym training guide
- ../hanzo/python-sdk.md - Python SDK integration
- ../hanzo/go-sdk.md - Go SDK integration
- ../hanzo/hanzo-node.md - Local inference infrastructure
Additional Resources
- GitHub: https://github.com/zooai/gym
- Paper: https://arxiv.org/abs/1802.07927 (Byzantine-Robust Distributed Learning)
- Differential Privacy: https://arxiv.org/abs/1607.00133 (DP-SGD)
- Zoo Labs Foundation: https://zoo.ngo (501(c)(3) wildlife conservation)
Remember: DeltaSoup enables community-driven AI model evolution with Byzantine-robust aggregation, differential privacy, and quality-based rewards - perfect for decentralized AI improvement while filtering malicious contributions and protecting user privacy.