16 KiB
name, description
| name | description |
|---|---|
| engineering-deployment-strategies | Production deployment strategies including blue-green, canary, rolling updates, zero-downtime deployments, and database migrations |
Deployment Strategies
Scope: Comprehensive deployment patterns, progressive delivery, feature flags, rollback procedures, and database migration strategies across Kubernetes, cloud platforms, and containerized environments
Lines: ~650
Last Updated: 2025-10-27
Format Version: 1.0 (Atomic)
When to Use This Skill
Activate this skill when:
- Designing deployment pipelines for production systems
- Implementing zero-downtime deployments
- Setting up progressive delivery (canary, blue-green)
- Planning database schema migrations
- Configuring rollback procedures
- Implementing feature flags for gradual rollouts
- Designing traffic shifting strategies
- Setting up deployment validation and health checks
Don't use this skill for:
- Basic CI/CD pipeline setup (see
continuous-integration.md) - Container orchestration basics (see
kubernetes-deployment) - Infrastructure provisioning (see
terraform-best-practices)
Core Concepts
Concept 1: Deployment Strategy Selection
Definition: Choose deployment approach based on risk tolerance, downtime requirements, and infrastructure capabilities
Decision Matrix:
Requirement | Recommended Strategy
---------------------|--------------------
Zero downtime | Blue-Green, Canary, Rolling
Instant rollback | Blue-Green
Gradual validation | Canary
Limited resources | Rolling, Recreate
Complex migrations | Blue-Green + Feature Flags
High risk changes | Canary + A/B Testing
Key Principles:
- Understand blast radius of failures
- Plan for rollback scenarios
- Validate before full deployment
- Monitor metrics during rollout
- Maintain backward compatibility
Benefits:
- Reduced deployment risk
- Faster time to production
- Improved reliability
- Better user experience
Concept 2: Progressive Delivery
Definition: Gradually release changes with continuous validation and control
Components:
Code → Deploy → Feature Flag → Observe → Decide
↓ ↓ ↓ ↓
Containers Control Metrics Expand/
Exposure Rollback
Traffic Management Patterns:
- Percentage-based: Route X% of traffic to new version
- User-based: Route specific users/groups to canary
- Geography-based: Route by region for localized testing
- Header-based: Route based on request headers (internal testing)
Validation Strategy:
- Deploy to canary environment
- Route small traffic percentage
- Monitor error rates, latency, business metrics
- Gradually increase traffic
- Promote or rollback based on metrics
Concept 3: Database Migration Strategies
Definition: Evolve database schema without breaking application deployments
Expand-Contract Pattern:
Phase 1: EXPAND
├─ Add new schema alongside old
├─ Deploy app with dual writes
└─ Both versions work
Phase 2: MIGRATE
├─ Backfill new schema
└─ Verify data integrity
Phase 3: CONTRACT
├─ Deploy app using only new schema
├─ Remove old schema
└─ Cleanup
Backward Compatibility Rules:
- Add columns as nullable first
- Never drop columns during deployment
- Use views for compatibility layers
- Dual write during transitions
- Batch backfill operations
Rollback Considerations:
- Keep old schema during rollback window
- Test rollback procedures
- Document migration dependencies
- Use transactions where possible
Patterns
Pattern 1: Blue-Green Deployment
Problem: Need instant rollback capability with zero downtime
Kubernetes Implementation:
# Two identical deployments
apiVersion: apps/v1
kind: Deployment
metadata:
name: myapp-blue
spec:
replicas: 3
selector:
matchLabels:
app: myapp
version: blue
template:
metadata:
labels:
app: myapp
version: blue
spec:
containers:
- name: myapp
image: myapp:v1.0.0
readinessProbe:
httpGet:
path: /ready
port: 8080
---
# Service switches between blue and green
apiVersion: v1
kind: Service
metadata:
name: myapp
spec:
selector:
app: myapp
version: blue # Change to 'green' to switch
ports:
- port: 80
targetPort: 8080
Switch Procedure:
# 1. Deploy green environment
kubectl apply -f deployment-green.yaml
# 2. Wait for green to be ready
kubectl wait --for=condition=available deployment/myapp-green
# 3. Test green environment
curl http://myapp-green-preview/health
# 4. Switch traffic to green
kubectl patch service myapp -p '{"spec":{"selector":{"version":"green"}}}'
# 5. Monitor for issues (keep blue running)
sleep 600
# 6. Scale down blue (after validation)
kubectl scale deployment myapp-blue --replicas=0
Benefits: Instant rollback, full testing before switch Trade-offs: Requires 2x infrastructure, database complexity
Pattern 2: Canary Deployment with Flagger
Problem: Need gradual validation with automatic rollback
Flagger Canary Resource:
apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
name: myapp
spec:
targetRef:
apiVersion: apps/v1
kind: Deployment
name: myapp
provider: istio
analysis:
interval: 1m
threshold: 5
maxWeight: 50
stepWeight: 10
metrics:
- name: request-success-rate
thresholdRange:
min: 99
interval: 1m
- name: request-duration
thresholdRange:
max: 500
interval: 1m
Progressive Rollout:
5% → Monitor → 15% → Monitor → 25% → Monitor → 50% → Complete
↓ ↓ ↓ ↓
Healthy? Healthy? Healthy? Healthy?
↓ ↓ ↓ ↓
Continue Continue Continue Promote
or or or or
Rollback Rollback Rollback Rollback
Benefits: Minimal blast radius, data-driven decisions Trade-offs: Slower deployment, complex traffic management
Pattern 3: Rolling Update with Health Checks
Problem: Need zero downtime with limited resources
Kubernetes Configuration:
apiVersion: apps/v1
kind: Deployment
metadata:
name: myapp
spec:
replicas: 10
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1 # Max 1 pod down at a time
maxSurge: 2 # Allow 2 extra pods during update
template:
spec:
terminationGracePeriodSeconds: 60
containers:
- name: myapp
image: myapp:v2.0.0
readinessProbe:
httpGet:
path: /ready
port: 8080
failureThreshold: 2
livenessProbe:
httpGet:
path: /health
port: 8080
failureThreshold: 3
lifecycle:
preStop:
exec:
command: ["/bin/sh", "-c", "sleep 15"]
Update Process:
- Create 1 new pod (within maxSurge)
- Wait for readiness probe to pass
- Terminate 1 old pod
- Wait for graceful shutdown
- Repeat until all pods updated
Benefits: No extra infrastructure, gradual rollout Trade-offs: Slower than blue-green, mixed versions during deployment
Pattern 4: Database Expand-Contract Migration
Problem: Need to change database schema without breaking deployments
Implementation:
-- PHASE 1: EXPAND (v1.1 app deployment)
ALTER TABLE orders ADD COLUMN status_name VARCHAR(50) NULL;
CREATE TRIGGER sync_status
BEFORE INSERT OR UPDATE ON orders
FOR EACH ROW
EXECUTE FUNCTION sync_order_status();
-- PHASE 2: MIGRATE (background job)
UPDATE orders
SET status_name = CASE status
WHEN 0 THEN 'pending'
WHEN 1 THEN 'processing'
WHEN 2 THEN 'completed'
END
WHERE status_name IS NULL;
-- PHASE 3: CONTRACT (v1.2 app deployment)
ALTER TABLE orders ALTER COLUMN status_name SET NOT NULL;
ALTER TABLE orders DROP COLUMN status;
Application Evolution:
v1.0: Reads/writes old column
v1.1: Reads old, writes both (dual write)
v1.2: Reads/writes new column only
v1.3: Old column removed
Benefits: Zero downtime, rollback-safe Trade-offs: Multi-phase deployment, temporary overhead
Pattern 5: Feature Flag Controlled Rollout
Problem: Decouple deployment from release
Implementation:
from launchdarkly import LDClient
ld_client = LDClient("sdk-key")
def get_pricing_page(user_id):
user = {
"key": user_id,
"custom": {
"deployment_version": os.getenv("APP_VERSION"),
"canary_group": os.getenv("CANARY_GROUP")
}
}
# Feature flag controls which version user sees
use_new_pricing = ld_client.variation("new-pricing", user, False)
if use_new_pricing:
return render_template("pricing_v2.html")
else:
return render_template("pricing_v1.html")
Targeting Rules:
{
"flagKey": "new-pricing",
"targeting": [
{
"variation": 1,
"values": ["canary"],
"attribute": "canary_group"
},
{
"variation": 1,
"percentage": {
"variations": [
{"variation": 0, "weight": 95000},
{"variation": 1, "weight": 5000}
]
}
}
]
}
Benefits: Deploy code safely, control feature exposure, instant rollback Trade-offs: Code complexity, flag management overhead
Anti-Patterns
Anti-Pattern 1: No Rollback Plan
Problem: Deploying without tested rollback procedures
Why It's Bad:
- Panic during incidents
- Extended downtime
- Data loss risks
- Manual recovery errors
Solution:
# Document and test rollback
# Blue-Green: Switch back
kubectl patch service myapp -p '{"spec":{"selector":{"version":"blue"}}}'
# Rolling: Rollback deployment
kubectl rollout undo deployment/myapp
# Database: Keep old schema during rollback window
# Feature Flags: Kill switch
Anti-Pattern 2: Testing in Production Only
Problem: No pre-deployment validation
Why It's Bad:
- Discover issues after user impact
- No safety net
- Higher risk
Solution:
# Smoke tests before full rollout
- name: Test green environment
run: |
GREEN_URL=$(get_green_url)
curl -f $GREEN_URL/health
curl -f $GREEN_URL/api/test
./load-test.sh $GREEN_URL
Anti-Pattern 3: Incompatible Schema Changes
Problem: Breaking old application version with schema changes
Why It's Bad:
- Forces all-or-nothing deployment
- Can't rollback
- Downtime required
Solution: Use expand-contract pattern, maintain backward compatibility
Anti-Pattern 4: No Health Checks
Problem: Routing traffic to unhealthy instances
Why It's Bad:
- User-facing errors
- Failed deployments go undetected
- No automatic recovery
Solution:
readinessProbe:
httpGet:
path: /ready
port: 8080
failureThreshold: 2
livenessProbe:
httpGet:
path: /health
port: 8080
failureThreshold: 3
Anti-Pattern 5: Manual Deployment Steps
Problem: Undocumented manual interventions
Why It's Bad:
- Not reproducible
- Error-prone
- Knowledge silos
- Can't automate
Solution: Automate everything, document runbooks, use GitOps
Quick Reference
Deployment Strategy Comparison
| Strategy | Downtime | Rollback | Cost | Complexity |
|---|---|---|---|---|
| Recreate | High | Slow | Low | Low |
| Rolling | None | Medium | Low | Medium |
| Blue-Green | None | Instant | High (2x) | Medium |
| Canary | None | Fast | Medium | High |
Health Check Best Practices
# Liveness: Is app alive?
livenessProbe:
httpGet:
path: /health
port: 8080
initialDelaySeconds: 30
periodSeconds: 10
failureThreshold: 3
# Readiness: Can app serve traffic?
readinessProbe:
httpGet:
path: /ready
port: 8080
initialDelaySeconds: 10
periodSeconds: 5
failureThreshold: 2
Graceful Shutdown
import signal
import sys
def handle_shutdown(signum, frame):
print("Shutting down gracefully...")
# Stop accepting new requests
server.close()
# Wait for existing requests
time.sleep(10)
sys.exit(0)
signal.signal(signal.SIGTERM, handle_shutdown)
Database Migration Checklist
- Schema changes are backward compatible
- Dual write implemented during transition
- Backfill plan for existing data
- Rollback procedure documented
- Migration tested in staging
- Monitoring for data consistency
Level 3: Resources
Extended documentation, production-ready examples, and automation tools
REFERENCE.md
Location: resources/REFERENCE.md (4,033 lines)
Comprehensive reference covering:
- Deployment strategy deep-dive (blue-green, canary, rolling, recreate)
- Progressive delivery patterns and feature flags
- Rollback procedures and automation
- Zero-downtime deployment techniques
- Database migration strategies (expand-contract, shadow tables)
- Kubernetes deployment patterns with Istio/Flagger
- Cloud provider deployment strategies (AWS, GCP, Azure)
- Traffic management and load balancing
- Monitoring and observability during deployments
- Testing strategies and validation
- Complete code examples for all patterns
Scripts
validate_deployment.py (600+ lines) Validates deployment configuration files across multiple platforms:
# Validate Kubernetes manifests
./validate_deployment.py --file deployment.yaml
# Check entire directory
./validate_deployment.py --directory k8s/ --type kubernetes
# JSON output for CI/CD
./validate_deployment.py --file deployment.yaml --json
execute_canary.py (550+ lines) Executes automated canary deployments with progressive traffic shifting:
# Kubernetes canary deployment
./execute_canary.py --platform kubernetes --service myapp --version v2.0
# AWS ALB canary
./execute_canary.py --platform aws-alb --service myapp --version v2.0
# Custom configuration
./execute_canary.py --config canary-config.yaml --json
test_deployment.sh (400+ lines) Tests deployment processes and measures downtime:
# Monitor deployment and measure downtime
./test_deployment.sh --url https://myapp.com --duration 300
# Test specific strategy
./test_deployment.sh --url https://myapp.com --strategy blue-green --json
Production Examples
Blue-Green Deployments:
examples/blue-green/kubernetes-blue-green.yaml: Complete K8s blue-green setup with health checks, PDB, and HPAexamples/docker/docker-compose-blue-green.yml: Docker Compose blue-green with NGINX load balancer
Canary Deployments:
examples/canary/flagger-canary.yaml: Flagger + Istio progressive canary with Prometheus metricsexamples/github-actions/canary-deployment-workflow.yml: GitHub Actions automated canary pipeline
Rolling Updates:
examples/rolling/kubernetes-rolling-update.yaml: Production-ready rolling update with proper health checks and resource limits
Database Migrations:
examples/database-migration/expand-contract-migration.sql: Complete expand-contract pattern with PostgreSQL
All examples include:
- Production-ready configurations
- Comprehensive comments
- Health check implementations
- Rollback procedures
- Monitoring integration
- Security best practices
Related Skills
continuous-integration.md- CI pipeline designtest-driven-development.md- Testing strategiestechnical-debt.md- Managing deployment complexitycloud-kubernetes-deployment- Kubernetes specificsterraform-best-practices- Infrastructure provisioning
Last Updated: 2025-10-27 Maintainer: Skills Team Validation: CI-validated, production-tested patterns