Files
skills/public/template.md

6.2 KiB

Alert: [AlertName]

Status: [Draft | Active | Archived] Last Updated: YYYY-MM-DD Owner: [Team/Person] Severity: [Critical | Warning | Info]


Overview

Brief description of what this alert means and why it exists.

What is being measured: Describe the metric or condition

Why it matters: Explain the business or technical impact


Symptoms

What the oncall engineer will observe:

  • System behavior
  • User-facing symptoms
  • Related metrics

Impact

User Impact

  • Service unavailable
  • Performance degraded
  • Feature disabled
  • No user impact

Business Impact

  • Revenue impact: [None | Low | Medium | High]
  • Customer impact: [Number of affected users]
  • SLA impact: [Yes/No]

Severity

Current Severity: [Critical | Warning | Info]

Escalation Criteria

Escalate to Critical if:

  • Condition persists > [duration]
  • Metric exceeds [threshold]
  • User impact confirmed

Escalate to Manager if:

  • No progress after [duration]
  • Customer escalation received
  • Multiple related incidents

Diagnosis

1. Verify the Alert

Check if alert is accurate:

# Query Prometheus to verify metric
curl 'http://prometheus:9090/api/v1/query?query=[METRIC_NAME]'

# Check Grafana dashboard
# URL: [Dashboard URL]

Expected: [What should you see?] Actual: [What are you seeing?]

# Check service health
kubectl get pods -n [namespace]
kubectl logs -f [pod-name] --tail=100

# Check dependencies
curl -I https://[dependency]/health

3. Review Recent Changes

# Check recent deployments
kubectl rollout history deployment/[name]

# Check git commits
git log --since="1 hour ago" --oneline

# Check recent alerts
# Alertmanager URL: [URL]

4. Identify Root Cause

Common causes:

  1. [Cause 1]: How to identify, how to confirm
  2. [Cause 2]: How to identify, how to confirm
  3. [Cause 3]: How to identify, how to confirm

Remediation

Immediate Actions (< 5 minutes)

Objective: Stop the bleeding, restore service

  1. Action 1:

    # Command with explanation
    kubectl scale deployment/[name] --replicas=[N]
    

    Expected outcome: [What should happen] Verification: [How to verify it worked]

  2. Action 2:

    # Command
    
  3. If above don't work: Escalate to [team/person]

Short-term Fix (< 1 hour)

Objective: Stable workaround while investigating root cause

  1. Action 1:

    • Step-by-step instructions
    • Expected outcomes
    • Rollback procedure if needed
  2. Action 2:

    • Instructions

Long-term Fix

Objective: Permanent resolution

  1. Create ticket: [Ticket system link template]
  2. Investigate root cause
  3. Implement permanent fix
  4. Update monitoring/alerts if needed

Rollback

If remediation makes things worse:

# Rollback deployment
kubectl rollout undo deployment/[name]

# Restore from backup
./restore_backup.sh [backup-id]

# Disable feature flag
curl -X POST https://[feature-flags]/disable/[flag]

Communication

Who to Notify

Stakeholder When How
Team Lead Immediately Slack #[channel]
Manager If not resolved in 30min Phone/Slack
Customers If impact confirmed Status page
Exec Team If revenue impact Email + Slack

Communication Templates

Initial notification (within 5 minutes):

[INCIDENT] [Service] - [Brief description]

Impact: [User-facing impact]
Started: [Time]
Status: Investigating
Next update: [Time]

Incident lead: [Name]
Slack channel: #incident-[ID]

Status update (every 30 minutes):

[UPDATE] [Service] - [Brief description]

Actions taken:
- [Action 1]
- [Action 2]

Current status: [Status]
Next update: [Time]

Resolution:

[RESOLVED] [Service] - [Brief description]

Resolution: [What fixed it]
Root cause: [Brief description]
Duration: [Total time]
Postmortem: [Link] (to be completed within 48h)

Prevention

Monitoring Improvements

  • Add monitoring for [metric]
  • Adjust alert threshold to [value]
  • Create dashboard for [view]

Process Improvements

  • Add deployment check for [condition]
  • Update documentation
  • Add automated test

Code/Infrastructure Changes

  • Increase resource limits
  • Implement circuit breaker
  • Add caching layer

Validation

Post-Remediation Checks

Verify service is healthy:

# Check 1: Service responding
curl https://[service]/health

# Check 2: Metrics recovered
# [Dashboard URL]

# Check 3: Error rate normal
# [Query URL]

# Check 4: Latency acceptable
# [Dashboard URL]

All checks passing? ✓ Incident resolved


Dashboards

  • [AlertName]: [Brief description]
  • [AlertName]: [Brief description]

Documentation

  • Architecture docs: [URL]
  • API docs: [URL]
  • Infrastructure docs: [URL]

Previous Incidents

  • [INC-123]: [Date] - [Brief description]
  • [INC-456]: [Date] - [Brief description]

Team Contacts

  • Oncall: Check PagerDuty schedule
  • Team Slack: #[channel]
  • Team email: [email]

Appendix

Useful Commands

# View logs
kubectl logs -f [pod] --namespace=[ns] --tail=100

# Check resource usage
kubectl top pods -n [namespace]

# Describe pod
kubectl describe pod [pod] -n [namespace]

# Execute command in pod
kubectl exec -it [pod] -n [namespace] -- bash

# Port forward for debugging
kubectl port-forward [pod] 8080:8080 -n [namespace]

Metrics Queries

# Request rate
sum(rate(http_requests_total[5m])) by (service)

# Error rate
sum(rate(http_requests_total{status=~"5.."}[5m])) by (service)
/
sum(rate(http_requests_total[5m])) by (service)

# Latency p95
histogram_quantile(0.95,
  sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service)
)

Change Log

Date Author Change
YYYY-MM-DD [Name] Initial version
YYYY-MM-DD [Name] Updated remediation steps

Feedback

Found an issue with this runbook? Incident didn't match the runbook?

  • Create issue: [Link to issue tracker]
  • Slack: #[channel]
  • Update directly: [Link to this file in git]