Files
skills/public/engineering-deployment-strategies.md

16 KiB

name, description
name description
engineering-deployment-strategies Production deployment strategies including blue-green, canary, rolling updates, zero-downtime deployments, and database migrations

Deployment Strategies

Scope: Comprehensive deployment patterns, progressive delivery, feature flags, rollback procedures, and database migration strategies across Kubernetes, cloud platforms, and containerized environments

Lines: ~650

Last Updated: 2025-10-27

Format Version: 1.0 (Atomic)


When to Use This Skill

Activate this skill when:

  • Designing deployment pipelines for production systems
  • Implementing zero-downtime deployments
  • Setting up progressive delivery (canary, blue-green)
  • Planning database schema migrations
  • Configuring rollback procedures
  • Implementing feature flags for gradual rollouts
  • Designing traffic shifting strategies
  • Setting up deployment validation and health checks

Don't use this skill for:

  • Basic CI/CD pipeline setup (see continuous-integration.md)
  • Container orchestration basics (see kubernetes-deployment)
  • Infrastructure provisioning (see terraform-best-practices)

Core Concepts

Concept 1: Deployment Strategy Selection

Definition: Choose deployment approach based on risk tolerance, downtime requirements, and infrastructure capabilities

Decision Matrix:

Requirement          | Recommended Strategy
---------------------|--------------------
Zero downtime        | Blue-Green, Canary, Rolling
Instant rollback     | Blue-Green
Gradual validation   | Canary
Limited resources    | Rolling, Recreate
Complex migrations   | Blue-Green + Feature Flags
High risk changes    | Canary + A/B Testing

Key Principles:

  1. Understand blast radius of failures
  2. Plan for rollback scenarios
  3. Validate before full deployment
  4. Monitor metrics during rollout
  5. Maintain backward compatibility

Benefits:

  • Reduced deployment risk
  • Faster time to production
  • Improved reliability
  • Better user experience

Concept 2: Progressive Delivery

Definition: Gradually release changes with continuous validation and control

Components:

Code → Deploy → Feature Flag → Observe → Decide
         ↓           ↓            ↓         ↓
    Containers   Control      Metrics   Expand/
                 Exposure               Rollback

Traffic Management Patterns:

  • Percentage-based: Route X% of traffic to new version
  • User-based: Route specific users/groups to canary
  • Geography-based: Route by region for localized testing
  • Header-based: Route based on request headers (internal testing)

Validation Strategy:

  1. Deploy to canary environment
  2. Route small traffic percentage
  3. Monitor error rates, latency, business metrics
  4. Gradually increase traffic
  5. Promote or rollback based on metrics

Concept 3: Database Migration Strategies

Definition: Evolve database schema without breaking application deployments

Expand-Contract Pattern:

Phase 1: EXPAND
  ├─ Add new schema alongside old
  ├─ Deploy app with dual writes
  └─ Both versions work

Phase 2: MIGRATE
  ├─ Backfill new schema
  └─ Verify data integrity

Phase 3: CONTRACT
  ├─ Deploy app using only new schema
  ├─ Remove old schema
  └─ Cleanup

Backward Compatibility Rules:

  • Add columns as nullable first
  • Never drop columns during deployment
  • Use views for compatibility layers
  • Dual write during transitions
  • Batch backfill operations

Rollback Considerations:

  • Keep old schema during rollback window
  • Test rollback procedures
  • Document migration dependencies
  • Use transactions where possible

Patterns

Pattern 1: Blue-Green Deployment

Problem: Need instant rollback capability with zero downtime

Kubernetes Implementation:

# Two identical deployments
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp-blue
spec:
  replicas: 3
  selector:
    matchLabels:
      app: myapp
      version: blue
  template:
    metadata:
      labels:
        app: myapp
        version: blue
    spec:
      containers:
      - name: myapp
        image: myapp:v1.0.0
        readinessProbe:
          httpGet:
            path: /ready
            port: 8080
---
# Service switches between blue and green
apiVersion: v1
kind: Service
metadata:
  name: myapp
spec:
  selector:
    app: myapp
    version: blue  # Change to 'green' to switch
  ports:
  - port: 80
    targetPort: 8080

Switch Procedure:

# 1. Deploy green environment
kubectl apply -f deployment-green.yaml

# 2. Wait for green to be ready
kubectl wait --for=condition=available deployment/myapp-green

# 3. Test green environment
curl http://myapp-green-preview/health

# 4. Switch traffic to green
kubectl patch service myapp -p '{"spec":{"selector":{"version":"green"}}}'

# 5. Monitor for issues (keep blue running)
sleep 600

# 6. Scale down blue (after validation)
kubectl scale deployment myapp-blue --replicas=0

Benefits: Instant rollback, full testing before switch Trade-offs: Requires 2x infrastructure, database complexity


Pattern 2: Canary Deployment with Flagger

Problem: Need gradual validation with automatic rollback

Flagger Canary Resource:

apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
  name: myapp
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: myapp
  provider: istio
  analysis:
    interval: 1m
    threshold: 5
    maxWeight: 50
    stepWeight: 10
    metrics:
    - name: request-success-rate
      thresholdRange:
        min: 99
      interval: 1m
    - name: request-duration
      thresholdRange:
        max: 500
      interval: 1m

Progressive Rollout:

5% → Monitor → 15% → Monitor → 25% → Monitor → 50% → Complete
     ↓              ↓              ↓              ↓
   Healthy?      Healthy?       Healthy?       Healthy?
     ↓              ↓              ↓              ↓
   Continue      Continue       Continue       Promote
     or             or             or             or
   Rollback      Rollback       Rollback       Rollback

Benefits: Minimal blast radius, data-driven decisions Trade-offs: Slower deployment, complex traffic management


Pattern 3: Rolling Update with Health Checks

Problem: Need zero downtime with limited resources

Kubernetes Configuration:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
spec:
  replicas: 10
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1  # Max 1 pod down at a time
      maxSurge: 2        # Allow 2 extra pods during update
  template:
    spec:
      terminationGracePeriodSeconds: 60
      containers:
      - name: myapp
        image: myapp:v2.0.0
        readinessProbe:
          httpGet:
            path: /ready
            port: 8080
          failureThreshold: 2
        livenessProbe:
          httpGet:
            path: /health
            port: 8080
          failureThreshold: 3
        lifecycle:
          preStop:
            exec:
              command: ["/bin/sh", "-c", "sleep 15"]

Update Process:

  1. Create 1 new pod (within maxSurge)
  2. Wait for readiness probe to pass
  3. Terminate 1 old pod
  4. Wait for graceful shutdown
  5. Repeat until all pods updated

Benefits: No extra infrastructure, gradual rollout Trade-offs: Slower than blue-green, mixed versions during deployment


Pattern 4: Database Expand-Contract Migration

Problem: Need to change database schema without breaking deployments

Implementation:

-- PHASE 1: EXPAND (v1.1 app deployment)
ALTER TABLE orders ADD COLUMN status_name VARCHAR(50) NULL;

CREATE TRIGGER sync_status
BEFORE INSERT OR UPDATE ON orders
FOR EACH ROW
EXECUTE FUNCTION sync_order_status();

-- PHASE 2: MIGRATE (background job)
UPDATE orders
SET status_name = CASE status
  WHEN 0 THEN 'pending'
  WHEN 1 THEN 'processing'
  WHEN 2 THEN 'completed'
END
WHERE status_name IS NULL;

-- PHASE 3: CONTRACT (v1.2 app deployment)
ALTER TABLE orders ALTER COLUMN status_name SET NOT NULL;
ALTER TABLE orders DROP COLUMN status;

Application Evolution:

v1.0: Reads/writes old column
v1.1: Reads old, writes both (dual write)
v1.2: Reads/writes new column only
v1.3: Old column removed

Benefits: Zero downtime, rollback-safe Trade-offs: Multi-phase deployment, temporary overhead


Pattern 5: Feature Flag Controlled Rollout

Problem: Decouple deployment from release

Implementation:

from launchdarkly import LDClient

ld_client = LDClient("sdk-key")

def get_pricing_page(user_id):
    user = {
        "key": user_id,
        "custom": {
            "deployment_version": os.getenv("APP_VERSION"),
            "canary_group": os.getenv("CANARY_GROUP")
        }
    }

    # Feature flag controls which version user sees
    use_new_pricing = ld_client.variation("new-pricing", user, False)

    if use_new_pricing:
        return render_template("pricing_v2.html")
    else:
        return render_template("pricing_v1.html")

Targeting Rules:

{
  "flagKey": "new-pricing",
  "targeting": [
    {
      "variation": 1,
      "values": ["canary"],
      "attribute": "canary_group"
    },
    {
      "variation": 1,
      "percentage": {
        "variations": [
          {"variation": 0, "weight": 95000},
          {"variation": 1, "weight": 5000}
        ]
      }
    }
  ]
}

Benefits: Deploy code safely, control feature exposure, instant rollback Trade-offs: Code complexity, flag management overhead


Anti-Patterns

Anti-Pattern 1: No Rollback Plan

Problem: Deploying without tested rollback procedures

Why It's Bad:

  • Panic during incidents
  • Extended downtime
  • Data loss risks
  • Manual recovery errors

Solution:

# Document and test rollback
# Blue-Green: Switch back
kubectl patch service myapp -p '{"spec":{"selector":{"version":"blue"}}}'

# Rolling: Rollback deployment
kubectl rollout undo deployment/myapp

# Database: Keep old schema during rollback window
# Feature Flags: Kill switch

Anti-Pattern 2: Testing in Production Only

Problem: No pre-deployment validation

Why It's Bad:

  • Discover issues after user impact
  • No safety net
  • Higher risk

Solution:

# Smoke tests before full rollout
- name: Test green environment
  run: |
    GREEN_URL=$(get_green_url)
    curl -f $GREEN_URL/health
    curl -f $GREEN_URL/api/test
    ./load-test.sh $GREEN_URL

Anti-Pattern 3: Incompatible Schema Changes

Problem: Breaking old application version with schema changes

Why It's Bad:

  • Forces all-or-nothing deployment
  • Can't rollback
  • Downtime required

Solution: Use expand-contract pattern, maintain backward compatibility


Anti-Pattern 4: No Health Checks

Problem: Routing traffic to unhealthy instances

Why It's Bad:

  • User-facing errors
  • Failed deployments go undetected
  • No automatic recovery

Solution:

readinessProbe:
  httpGet:
    path: /ready
    port: 8080
  failureThreshold: 2
livenessProbe:
  httpGet:
    path: /health
    port: 8080
  failureThreshold: 3

Anti-Pattern 5: Manual Deployment Steps

Problem: Undocumented manual interventions

Why It's Bad:

  • Not reproducible
  • Error-prone
  • Knowledge silos
  • Can't automate

Solution: Automate everything, document runbooks, use GitOps


Quick Reference

Deployment Strategy Comparison

Strategy Downtime Rollback Cost Complexity
Recreate High Slow Low Low
Rolling None Medium Low Medium
Blue-Green None Instant High (2x) Medium
Canary None Fast Medium High

Health Check Best Practices

# Liveness: Is app alive?
livenessProbe:
  httpGet:
    path: /health
    port: 8080
  initialDelaySeconds: 30
  periodSeconds: 10
  failureThreshold: 3

# Readiness: Can app serve traffic?
readinessProbe:
  httpGet:
    path: /ready
    port: 8080
  initialDelaySeconds: 10
  periodSeconds: 5
  failureThreshold: 2

Graceful Shutdown

import signal
import sys

def handle_shutdown(signum, frame):
    print("Shutting down gracefully...")
    # Stop accepting new requests
    server.close()
    # Wait for existing requests
    time.sleep(10)
    sys.exit(0)

signal.signal(signal.SIGTERM, handle_shutdown)

Database Migration Checklist

  • Schema changes are backward compatible
  • Dual write implemented during transition
  • Backfill plan for existing data
  • Rollback procedure documented
  • Migration tested in staging
  • Monitoring for data consistency

Level 3: Resources

Extended documentation, production-ready examples, and automation tools

REFERENCE.md

Location: resources/REFERENCE.md (4,033 lines)

Comprehensive reference covering:

  • Deployment strategy deep-dive (blue-green, canary, rolling, recreate)
  • Progressive delivery patterns and feature flags
  • Rollback procedures and automation
  • Zero-downtime deployment techniques
  • Database migration strategies (expand-contract, shadow tables)
  • Kubernetes deployment patterns with Istio/Flagger
  • Cloud provider deployment strategies (AWS, GCP, Azure)
  • Traffic management and load balancing
  • Monitoring and observability during deployments
  • Testing strategies and validation
  • Complete code examples for all patterns

Scripts

validate_deployment.py (600+ lines) Validates deployment configuration files across multiple platforms:

# Validate Kubernetes manifests
./validate_deployment.py --file deployment.yaml

# Check entire directory
./validate_deployment.py --directory k8s/ --type kubernetes

# JSON output for CI/CD
./validate_deployment.py --file deployment.yaml --json

execute_canary.py (550+ lines) Executes automated canary deployments with progressive traffic shifting:

# Kubernetes canary deployment
./execute_canary.py --platform kubernetes --service myapp --version v2.0

# AWS ALB canary
./execute_canary.py --platform aws-alb --service myapp --version v2.0

# Custom configuration
./execute_canary.py --config canary-config.yaml --json

test_deployment.sh (400+ lines) Tests deployment processes and measures downtime:

# Monitor deployment and measure downtime
./test_deployment.sh --url https://myapp.com --duration 300

# Test specific strategy
./test_deployment.sh --url https://myapp.com --strategy blue-green --json

Production Examples

Blue-Green Deployments:

  • examples/blue-green/kubernetes-blue-green.yaml: Complete K8s blue-green setup with health checks, PDB, and HPA
  • examples/docker/docker-compose-blue-green.yml: Docker Compose blue-green with NGINX load balancer

Canary Deployments:

  • examples/canary/flagger-canary.yaml: Flagger + Istio progressive canary with Prometheus metrics
  • examples/github-actions/canary-deployment-workflow.yml: GitHub Actions automated canary pipeline

Rolling Updates:

  • examples/rolling/kubernetes-rolling-update.yaml: Production-ready rolling update with proper health checks and resource limits

Database Migrations:

  • examples/database-migration/expand-contract-migration.sql: Complete expand-contract pattern with PostgreSQL

All examples include:

  • Production-ready configurations
  • Comprehensive comments
  • Health check implementations
  • Rollback procedures
  • Monitoring integration
  • Security best practices

  • continuous-integration.md - CI pipeline design
  • test-driven-development.md - Testing strategies
  • technical-debt.md - Managing deployment complexity
  • cloud-kubernetes-deployment - Kubernetes specifics
  • terraform-best-practices - Infrastructure provisioning

Last Updated: 2025-10-27 Maintainer: Skills Team Validation: CI-validated, production-tested patterns