Files
skills/public/cloud-kubernetes-deployment.md

16 KiB

name, category, description, tags, version
name category description tags version
cloud-kubernetes-deployment cloud Deploy and manage applications on Kubernetes with production best practices
kubernetes
k8s
deployment
cloud
containers
orchestration
1.0.0

Kubernetes Deployment

Deploy and manage containerized applications on Kubernetes with production best practices for high availability, security, and scalability.

Hanzo Production Reality (read first)

For Hanzo/Lux services this generic skill is background knowledge only — deployment is prescribed by HIP-0117 (one binary, three modes) and the operator. See skills/hanzo-cloud-architecture/SKILL.md.

  1. cloud serve — single process, NO Kubernetes. The whole cloud (IAM, KMS, o11y, tasks, console, ...) in one binary with per-tenant SQLite. This is local dev and single-node/edge production.
  2. cloud cluster init — the binary fetches k3s, bootstraps the cluster, installs the operator, and hands reconciliation to services.hanzo.ai CRs. Devs do NOT hand-write Deployments; the operator reconciles CRs. (Decided per HIP-0117; staged.)
  3. helm install cloud ./helm/cloud — BYO Kubernetes, same image.

Rules that override the generic advice below:

  • You deploy by editing a services.hanzo.ai CR (or universe manifests), never by kubectl create deployment.
  • CI/CD is Gitea Actions on git.hanzo.ai (GH-Actions-compatible workflows on our runners); GitHub is the OSS mirror. Never build images locally.
  • Images: ghcr.io/hanzoai/* (Hanzo) / ghcr.io/luxfi/* (Lux) — never mix.
  • Secrets come from KMS (KMSSecret CRDs), never plain K8s Secrets in manifests.
  • Ingress is hanzoai/ingress — no nginx, no caddy.
  • 8 Kubernetes clusters are operated; visor autoscales fleets across providers (BYO provider).

Overview

This skill covers Kubernetes deployment strategies, including:

  • Pod and Deployment Management: Creating and managing workloads
  • Service Discovery: Exposing and accessing applications
  • Rolling Updates: Zero-downtime deployments and rollbacks
  • Resource Management: CPU, memory, and storage configuration
  • Health Checks: Liveness, readiness, and startup probes
  • Configuration: ConfigMaps, Secrets, and environment management
  • Networking: Services, Ingress, and NetworkPolicies
  • Security: RBAC, Pod Security Standards, security contexts
  • Autoscaling: HPA, VPA, and cluster autoscaling
  • GitOps: ArgoCD and Flux workflows
  • Helm: Package management and templating

When to Use

  • Deploying containerized applications to Kubernetes
  • Managing application lifecycle in Kubernetes
  • Implementing CI/CD for Kubernetes deployments
  • Setting up production-ready Kubernetes workloads
  • Troubleshooting Kubernetes deployments
  • Migrating applications to Kubernetes

Prerequisites

  • Docker and containerization basics
  • YAML syntax
  • Basic networking concepts
  • Command-line proficiency
  • Access to a Kubernetes cluster (or local testing with kind/minikube)

Key Concepts

Kubernetes Resources

Workloads:

  • Pod: Smallest deployable unit (one or more containers)
  • Deployment: Manages ReplicaSets for stateless apps
  • StatefulSet: Manages stateful workloads with stable identities
  • DaemonSet: Runs pods on all (or selected) nodes
  • Job/CronJob: Batch processing and scheduled tasks

Services:

  • ClusterIP: Internal cluster communication
  • NodePort: Expose on node IPs
  • LoadBalancer: Cloud provider load balancer
  • Ingress: HTTP/HTTPS routing

Configuration:

  • ConfigMap: Non-sensitive configuration data
  • Secret: Sensitive data (base64 encoded)
  • PersistentVolume/PVC: Storage management

Security:

  • ServiceAccount: Pod identity
  • Role/ClusterRole: RBAC permissions
  • NetworkPolicy: Pod network isolation
  • PodSecurityPolicy/Standards: Pod security controls

Deployment Strategies

Rolling Update (Default):

  • Gradual replacement of old pods with new ones
  • Zero downtime
  • Configurable with maxSurge and maxUnavailable

Recreate:

  • Delete all old pods before creating new ones
  • Causes downtime
  • Useful when incompatible versions can't coexist

Canary:

  • Deploy new version to small subset
  • Monitor metrics before full rollout
  • Manual traffic splitting

Blue-Green:

  • Deploy new version alongside old
  • Switch traffic all at once
  • Easy rollback

Basic Usage

Create Deployment

# From command line
kubectl create deployment web --image=nginx:1.25 --replicas=3

# From YAML
kubectl apply -f deployment.yaml

deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:
  replicas: 3
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
      - name: nginx
        image: nginx:1.25
        ports:
        - containerPort: 80
        resources:
          requests:
            cpu: 100m
            memory: 128Mi
          limits:
            cpu: 200m
            memory: 256Mi

Expose with Service

# Create service
kubectl expose deployment web --port=80 --type=LoadBalancer

# From YAML
kubectl apply -f service.yaml

service.yaml:

apiVersion: v1
kind: Service
metadata:
  name: web
spec:
  selector:
    app: web
  ports:
  - port: 80
    targetPort: 80
  type: LoadBalancer

Update Deployment

# Update image
kubectl set image deployment/web nginx=nginx:1.26

# Edit deployment
kubectl edit deployment/web

# Apply changes from file
kubectl apply -f deployment.yaml

# Monitor rollout
kubectl rollout status deployment/web

Rollback Deployment

# View history
kubectl rollout history deployment/web

# Rollback to previous version
kubectl rollout undo deployment/web

# Rollback to specific revision
kubectl rollout undo deployment/web --to-revision=2

Advanced Patterns

Production-Ready Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
  labels:
    app: api
    version: v1
spec:
  replicas: 3
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  selector:
    matchLabels:
      app: api
  template:
    metadata:
      labels:
        app: api
        version: v1
    spec:
      serviceAccountName: api
      securityContext:
        runAsNonRoot: true
        runAsUser: 1000
        fsGroup: 2000

      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: app
                  operator: In
                  values:
                  - api
              topologyKey: kubernetes.io/hostname

      containers:
      - name: api
        image: myapp/api:v1.0.0
        ports:
        - name: http
          containerPort: 8080

        env:
        - name: DB_HOST
          valueFrom:
            configMapKeyRef:
              name: api-config
              key: db.host
        - name: DB_PASSWORD
          valueFrom:
            secretKeyRef:
              name: api-secrets
              key: db.password

        resources:
          requests:
            cpu: 500m
            memory: 512Mi
          limits:
            cpu: 2
            memory: 2Gi

        startupProbe:
          httpGet:
            path: /healthz/startup
            port: http
          periodSeconds: 10
          failureThreshold: 30

        livenessProbe:
          httpGet:
            path: /healthz/live
            port: http
          periodSeconds: 10
          failureThreshold: 3

        readinessProbe:
          httpGet:
            path: /healthz/ready
            port: http
          periodSeconds: 5
          failureThreshold: 3

        securityContext:
          allowPrivilegeEscalation: false
          readOnlyRootFilesystem: true
          runAsNonRoot: true
          capabilities:
            drop:
            - ALL

        volumeMounts:
        - name: tmp
          mountPath: /tmp

      volumes:
      - name: tmp
        emptyDir: {}

Horizontal Pod Autoscaler

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: api-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api
  minReplicas: 3
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
  - type: Resource
    resource:
      name: memory
      target:
        type: Utilization
        averageUtilization: 80

ConfigMap and Secret

# ConfigMap
apiVersion: v1
kind: ConfigMap
metadata:
  name: api-config
data:
  db.host: "postgres.default.svc.cluster.local"
  db.port: "5432"
  app.yaml: |
    server:
      port: 8080
      timeout: 30s
---
# Secret
# ❌ BAD: Hardcoded credentials - example only, never do this in production
# In production, use sealed-secrets, external-secrets-operator, or your cloud provider's secret management
apiVersion: v1
kind: Secret
metadata:
  name: api-secrets
type: Opaque
stringData:
  db.password: "changeme123"  # Example only - use secret management in production
  api.key: "secret-api-key"  # Example only - use secret management in production

Common Commands

# Get resources
kubectl get deployments
kubectl get pods
kubectl get services
kubectl get all

# Describe resource
kubectl describe deployment web
kubectl describe pod web-abc123

# View logs
kubectl logs deployment/web
kubectl logs -f pod/web-abc123  # Follow logs
kubectl logs pod/web-abc123 --previous  # Previous container

# Execute commands in pod
kubectl exec -it pod/web-abc123 -- /bin/sh

# Port forwarding
kubectl port-forward deployment/web 8080:80

# Scale deployment
kubectl scale deployment/web --replicas=5

# Delete resources
kubectl delete deployment web
kubectl delete -f deployment.yaml

Troubleshooting

Pod Not Starting

# Check pod status
kubectl get pods
kubectl describe pod <pod-name>

# Common issues:
# - ImagePullBackOff: Check image name and registry credentials
# - CrashLoopBackOff: Check logs for application errors
# - Pending: Check resource availability and node constraints

# View events
kubectl get events --sort-by='.lastTimestamp'

# Check logs
kubectl logs <pod-name>

Service Not Accessible

# Check service and endpoints
kubectl get service web
kubectl get endpoints web

# Test DNS
kubectl run -it --rm debug --image=busybox --restart=Never -- nslookup web

# Test connectivity
kubectl run -it --rm debug --image=curlimages/curl --restart=Never -- curl http://web

Deployment Not Rolling Out

# Check rollout status
kubectl rollout status deployment/web

# View deployment events
kubectl describe deployment web

# Check replica sets
kubectl get rs -l app=web

# Pause and resume
kubectl rollout pause deployment/web
kubectl rollout resume deployment/web

Best Practices

Resource Management

  • Always set resource requests and limits
  • Use appropriate QoS class (Guaranteed for critical workloads)
  • Monitor actual usage and adjust accordingly

Health Checks

  • Always define readiness probes
  • Use startup probes for slow-starting applications
  • Keep probe endpoints lightweight

Security

  • Run as non-root user
  • Use read-only root filesystem
  • Drop all capabilities unless needed
  • Implement NetworkPolicies
  • Use RBAC with least privilege
  • Scan images for vulnerabilities

High Availability

  • Run multiple replicas (minimum 3 for critical services)
  • Use pod anti-affinity to spread across nodes/zones
  • Define PodDisruptionBudgets
  • Use topology spread constraints

Configuration

  • Externalize configuration with ConfigMaps/Secrets
  • Use immutable ConfigMaps/Secrets in production
  • Never commit secrets to Git
  • Version configuration with application

Deployment

  • Use specific image tags (never :latest)
  • Implement gradual rollouts (canary/blue-green)
  • Monitor during rollout
  • Have rollback plan
  • Test in staging first
  • containers-docker: Container fundamentals
  • cloud-aws-eks: AWS Kubernetes service
  • cloud-gcp-gke: Google Kubernetes Engine
  • observability-prometheus: Metrics collection
  • cicd-github-actions: CI/CD automation
  • infrastructure-terraform: Infrastructure as Code

Level 3: Resources

Comprehensive resources for Kubernetes deployment are available in the resources/ directory:

Reference Material

resources/REFERENCE.md (1,800+ lines) Complete reference covering:

  • Kubernetes architecture and fundamentals
  • Pod lifecycle and multi-container patterns
  • Deployment strategies and rollout management
  • Service networking and discovery
  • Resource management and QoS
  • Health checks configuration
  • ConfigMaps and Secrets management
  • Persistent storage (PVs, PVCs, StorageClasses)
  • Networking (Services, Ingress, NetworkPolicies)
  • Security (RBAC, Pod Security Standards, security contexts)
  • Autoscaling (HPA, VPA, Cluster Autoscaler)
  • GitOps patterns (ArgoCD, Flux)
  • Helm charts and package management
  • Production best practices
  • Troubleshooting guide
  • Common pitfalls and anti-patterns

Executable Scripts

resources/scripts/validate_manifests.py Validate Kubernetes YAML manifests for correctness and best practices:

  • YAML syntax validation
  • Kubernetes API schema compliance
  • Security best practices checks
  • Resource configuration validation
  • Common misconfiguration detection
  • kubectl dry-run validation
  • JSON and colored output options
# Validate single file
./validate_manifests.py deployment.yaml

# Validate directory
./validate_manifests.py --strict manifests/

# JSON output
./validate_manifests.py --json deployment.yaml

resources/scripts/analyze_deployment.py Analyze deployments for issues and optimization opportunities:

  • Configuration issue detection
  • Security vulnerability scanning
  • Resource optimization recommendations
  • High availability concerns
  • Production readiness assessment
  • Live cluster analysis support
# Analyze file
./analyze_deployment.py deployment.yaml

# Analyze live cluster
./analyze_deployment.py --cluster --namespace production

# JSON output
./analyze_deployment.py --json --cluster

resources/scripts/test_deployment.sh Test deployments in local Kubernetes cluster:

  • Creates test cluster (kind/minikube)
  • Deploys manifests
  • Validates rollout
  • Runs health checks
  • Executes smoke tests
  • Automatic cleanup
# Test with kind
./test_deployment.sh deployment.yaml

# Test with minikube
./test_deployment.sh --provider minikube manifests/

# Keep cluster for debugging
./test_deployment.sh --no-cleanup deployment.yaml

Examples

resources/examples/manifests/basic-deployment/ Simple deployment example:

  • Basic Deployment with nginx
  • ClusterIP Service
  • Minimal configuration for learning

resources/examples/manifests/production-deployment/ Production-ready deployment:

  • Full Deployment with all best practices
  • LoadBalancer Service with annotations
  • ConfigMap and Secret management
  • HorizontalPodAutoscaler
  • PodDisruptionBudget
  • Security contexts and RBAC
  • Multi-container patterns
  • Health checks and lifecycle hooks

resources/examples/helm/sample-chart/ Complete Helm chart structure:

  • Chart.yaml with metadata and dependencies
  • values.yaml with sensible defaults
  • Templates for all resources
  • Helper templates (_helpers.tpl)
  • Configurable security and resources

resources/examples/kustomize/ Kustomize overlays for environments:

  • base/: Base manifests
  • overlays/dev/: Development environment customizations
  • overlays/prod/: Production environment customizations
  • Demonstrates patching and configuration management

resources/examples/cicd/github-actions-deploy.yml Complete GitHub Actions workflow:

  • Build and push Docker images
  • Validate manifests
  • Deploy to staging (on PR)
  • Deploy to production (on main)
  • Rollback on failure
  • Smoke tests

Learning Path

  1. Start: Basic deployment and service creation
  2. Intermediate: Add health checks, resource limits, ConfigMaps
  3. Advanced: Implement autoscaling, GitOps, production patterns
  4. Expert: Multi-cluster, service mesh, advanced networking

See Also