Files
skills/public/high-memory-usage.md

11 KiB

Alert: HighMemoryUsage

Status: Active Last Updated: 2025-10-27 Owner: Platform Team Severity: Critical


Overview

This alert fires when memory usage exceeds 90% for more than 5 minutes on any node.

What is being measured: node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes

Why it matters: High memory usage can lead to:

  • OOM (Out of Memory) killer terminating processes
  • System instability and crashes
  • Service degradation or unavailability
  • Swap thrashing (severe performance impact)

Symptoms

What you'll observe:

  • Memory usage > 90% in monitoring
  • Pods being evicted (Kubernetes)
  • Swap usage increasing
  • System slowness
  • Application crashes or restarts
  • "Out of memory" errors in logs

Impact

User Impact

  • Performance degraded (slow responses)
  • Service unavailable (if OOM occurs)
  • Feature disabled
  • No user impact

Business Impact

  • Revenue impact: Medium (degraded performance)
  • Customer impact: Varies by service affected
  • SLA impact: Yes (if availability drops below threshold)

Severity

Current Severity: Critical (if > 95%), Warning (if > 90%)

Escalation Criteria

Escalate to Critical if:

  • Memory usage > 95%
  • OOM events detected
  • Multiple pods evicted
  • Service impact confirmed

Escalate to Manager if:

  • No improvement after 30 minutes
  • Cluster-wide memory pressure
  • Customer complaints received

Diagnosis

1. Verify the Alert

# SSH to affected node
ssh [node-ip]

# Check current memory usage
free -h

# Example output:
#               total        used        free      shared  buff/cache   available
# Mem:           15Gi        14Gi       100Mi        50Mi       900Mi       800Mi
# Swap:         2.0Gi       1.5Gi       500Mi

# Check memory usage percentage
echo "scale=2; $(grep MemAvailable /proc/meminfo | awk '{print $2}') / $(grep MemTotal /proc/meminfo | awk '{print $2}') * 100" | bc

# View top memory consumers
top -o %MEM

# Or use htop (better visualization)
htop

Expected: Memory available > 10% (> 1.5GB on 16GB node) Actual: Memory available < 10%

2. Identify Memory Consumers

# Top 10 processes by memory (Linux)
ps aux --sort=-%mem | head -11

# Detailed per-process memory breakdown
ps aux | awk '{print $6/1024 " MB\t" $11}' | sort -n

# Container memory usage (Docker)
docker stats --no-stream --format "table {{.Container}}\t{{.MemUsage}}"

# Kubernetes pod memory usage
kubectl top pods --all-namespaces --sort-by=memory

# Check for memory leaks in specific process
pmap -x [PID] | tail -1

3. Check for Memory Leaks

# Monitor process memory over time
watch -n 5 'ps aux | grep [process-name]'

# Check for zombie processes
ps aux | grep defunct

# View kernel memory slab usage
sudo slabtop

# Check page cache usage
cat /proc/meminfo | grep -E '(Cached|Buffers|Slab)'

# OOM killer logs
dmesg | grep -i "out of memory"
sudo journalctl -xe | grep -i oom

4. Review Recent Changes

# Recent deployments (Kubernetes)
kubectl rollout history deployment/[name]

# Recent pod restarts
kubectl get pods --all-namespaces --field-selector=status.phase=Running --sort-by=.status.startTime

# Check resource limits
kubectl describe pod [pod-name] | grep -A 5 "Limits"

# Git commits (if applicable)
git log --since="6 hours ago" --oneline

Common Causes

  1. Memory leak: Application not releasing memory

    • Confirm: Memory usage steadily increasing over time
    • Check: Application logs for errors, heap dumps
  2. Increased load: More traffic than capacity

    • Confirm: Request rate increased, more pods running
    • Check: Request metrics, pod count
  3. Large data processing: Batch job or analytics

    • Confirm: Scheduled job running, large dataset being processed
    • Check: Cron jobs, data pipeline status
  4. Page cache: Linux caching disk I/O

    • Confirm: "Cached" high in /proc/meminfo, "Available" still reasonable
    • Action: Usually safe, kernel will free if needed
  5. Resource limit mismatch: Pod limits too high

    • Confirm: Sum of pod limits > node capacity
    • Check: kubectl describe node [node]

Remediation

Immediate Actions (< 5 minutes)

Objective: Prevent OOM, keep system stable

  1. Clear page cache (temporary relief):

    # This is safe - kernel will rebuild cache as needed
    sync && echo 3 > /proc/sys/vm/drop_caches
    

    Expected outcome: Memory usage drops 10-20% Verification: Run free -h again

  2. Identify and restart leaking service (if known):

    # Kubernetes
    kubectl rollout restart deployment/[name]
    
    # Systemd service
    sudo systemctl restart [service]
    
    # Docker container
    docker restart [container-id]
    

    Expected outcome: Memory released, service recovers Verification: Check memory usage of new pods/processes

  3. Scale horizontally (if auto-scaling available):

    # Kubernetes - add more replicas
    kubectl scale deployment [name] --replicas=[N+2]
    
    # Cloud - add instances
    aws autoscaling set-desired-capacity --auto-scaling-group-name [name] --desired-capacity [N+1]
    

    Expected outcome: Load distributed, per-instance memory reduced Verification: kubectl top pods shows lower memory per pod

  4. Evict low-priority workloads (emergency only):

    # Kubernetes - delete non-critical pods
    kubectl delete pod [pod-name] -n [namespace]
    
    # Stop batch jobs
    kubectl delete job [job-name]
    

If above don't work: Escalate to platform team lead immediately

Short-term Fix (< 1 hour)

Objective: Stable state while investigating root cause

  1. Adjust resource limits (if over-committed):

    # Edit deployment
    kubectl edit deployment [name]
    
    # Set appropriate limits
    resources:
      requests:
        memory: "256Mi"
      limits:
        memory: "512Mi"
    
  2. Enable swap (temporary, not recommended long-term):

    # Check current swap
    swapon -s
    
    # Create swap file (if none exists)
    sudo fallocate -l 4G /swapfile
    sudo chmod 600 /swapfile
    sudo mkswap /swapfile
    sudo swapon /swapfile
    

    Note: This buys time but doesn't fix the problem

  3. Tune application memory settings:

    # Java applications - reduce heap size
    export JAVA_OPTS="-Xmx2g -Xms512m"
    
    # Node.js - set max old space
    export NODE_OPTIONS="--max-old-space-size=2048"
    
    # Python - limit worker memory
    export WORKER_MEMORY_LIMIT=512M
    
  4. Implement pod disruption budget (prevent cascading failures):

    apiVersion: policy/v1
    kind: PodDisruptionBudget
    metadata:
      name: [app]-pdb
    spec:
      minAvailable: 2
      selector:
        matchLabels:
          app: [app]
    

Long-term Fix

  1. Increase node resources:

    • Vertical scaling: Upgrade to larger instances
    • Horizontal scaling: Add more nodes
    • Auto-scaling: Configure memory-based scaling
  2. Optimize application memory:

    • Profile and fix memory leaks
    • Optimize data structures
    • Implement caching strategies
    • Add memory limits and monitoring
  3. Improve resource management:

    • Set accurate resource requests/limits
    • Use pod priority classes
    • Enable vertical pod autoscaler
    • Implement memory quotas per namespace

Rollback

If remediation makes things worse:

# Rollback deployment
kubectl rollout undo deployment/[name]

# Restore previous auto-scaling settings
kubectl apply -f autoscaling-backup.yaml

# Disable swap (if enabled as temp fix)
sudo swapoff /swapfile
sudo rm /swapfile

Communication

Who to Notify

Stakeholder When How
Platform Team Immediately Slack #platform-oncall
Engineering Manager If not resolved in 30min Slack + Phone
Application Owners If specific app identified Slack team channel
Customers If service degraded > 15min Status page

Communication Templates

Initial notification:

[INCIDENT] High Memory Usage on [node/cluster]

Impact: [service] may experience slow responses
Started: [time]
Status: Investigating memory consumption
Next update: [+15min]

Incident lead: [oncall engineer]
Slack: #incident-[ID]

Prevention

Monitoring Improvements

  • Alert on memory > 90% (existing)
  • Alert on OOM events
  • Monitor memory growth rate
  • Dashboard for memory breakdown

Process Improvements

  • Memory load testing before deployment
  • Automatic heap dumps on high memory
  • Regular memory profiling of applications

Code/Infrastructure Changes

  • Implement memory limits on all pods
  • Add memory leak detection
  • Configure horizontal pod autoscaling
  • Optimize memory-intensive operations

Validation

Post-Remediation Checks

# 1. Memory usage normal
free -h | grep Mem
# Available should be > 20%

# 2. No OOM events in last 10 minutes
dmesg | grep -i oom | tail -5

# 3. All pods running
kubectl get pods --all-namespaces | grep -v Running

# 4. Application metrics normal
# Check dashboard: [URL]

# 5. No pending pod evictions
kubectl get events --all-namespaces | grep Evicted

All checks passing? ✓ Incident resolved


Dashboards

  • OOMKiller: OOM events occurred
  • HighSwapUsage: Swapping indicates memory pressure
  • PodEvicted: Pods evicted due to resource pressure

Documentation

Previous Incidents

  • INC-234: 2025-08-15 - Java heap leak in API service
  • INC-567: 2025-09-22 - Redis cache grew unbounded
  • INC-890: 2025-10-10 - Pod limits too high, node over-committed

Appendix

Memory Metrics Explained

MemTotal     : Total physical RAM
MemFree      : Unused RAM
MemAvailable : RAM available for starting new applications (includes reclaimable cache)
Buffers      : Temporary storage for block devices
Cached       : Page cache from files
Slab         : Kernel data structures
SwapTotal    : Total swap space
SwapFree     : Unused swap space

Key metric: MemAvailable (not MemFree)

Kubernetes Memory Metrics

# Pod memory requests vs limits
kubectl describe node [node] | grep -A 5 "Allocated resources"

# Memory pressure eviction thresholds
kubectl get nodes -o json | jq '.items[].status.allocatable.memory'

# Top memory-consuming namespaces
kubectl top pods -A | awk '{print $1, $4}' | sort -k2 -rh | head -20

Change Log

Date Author Change
2025-10-27 Platform Team Initial version
2025-10-27 SRE Added Kubernetes-specific steps

Feedback

Found this runbook helpful? Found an issue?