mirror of
https://github.com/hanzoai/skills.git
synced 2026-11-19 00:47:36 +00:00
Every page now loads /webmcp.js, scripts/webmcp.ts with its types stripped (npm run webmcp). Where the browser has document.modelContext it registers two read-only tools: search_skills over the published skills index, and get_skill, which returns a skill's markdown checked against the index digest. The markdown alternate link moves from build.py to site.py, so CI holds every page to it. The API catalog gains the site itself (llms.txt and the skills index) and hanzo.ai/agents; the live discovery check now fetches every catalog link. tests/site.mjs drives the tools through a recording modelContext, and checks a browser without one loads clean.
451 lines
26 KiB
HTML
451 lines
26 KiB
HTML
<!doctype html>
|
|
<html lang="en"><head>
|
|
<meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1">
|
|
<title>high-memory-usage</title>
|
|
<meta name="color-scheme" content="dark">
|
|
<style>
|
|
/* Committed to one visual world: a dark instrument panel. No light variant —
|
|
the dot field and the hairline rules only read on black, and a naive
|
|
inversion would wash both out. Every colour is painted explicitly so the page
|
|
never borrows a host background. */
|
|
@font-face{font-family:'Zen';src:url('/fonts/Zen-Variable.woff2') format('woff2');font-weight:100 900;font-style:normal;font-display:swap}
|
|
@font-face{font-family:'Zen';src:url('/fonts/Zen-Italic-Variable.woff2') format('woff2');font-weight:100 900;font-style:italic;font-display:swap}
|
|
@font-face{font-family:'Zen Mono';src:url('/fonts/ZenMono-Variable.woff2') format('woff2');font-weight:100 900;font-style:normal;font-display:swap}
|
|
@font-face{font-family:'Zen Mono';src:url('/fonts/ZenMono-Italic-Variable.woff2') format('woff2');font-weight:100 900;font-style:italic;font-display:swap}
|
|
@font-face{font-family:'Zen Pixel Square';src:url('/fonts/ZenPixel-Square.woff2') format('woff2');font-style:normal;font-display:swap}
|
|
|
|
:root{
|
|
--ink:#f4f4f2; --soft:#9c9c98; --faint:#6a6a66; --ground:#000; --panel:#0a0a0a;
|
|
--line:#232323; --line-hi:#3a3a3a; --accent:#f4f4f2;
|
|
--font-zen-sans:'Zen',-apple-system,BlinkMacSystemFont,"Segoe UI",Helvetica,Arial,sans-serif;
|
|
--font-zen-mono:'Zen Mono',ui-monospace,"SF Mono",SFMono-Regular,Menlo,Consolas,monospace;
|
|
--font-zen-pixel-square:'Zen Pixel Square',monospace;
|
|
--mono:var(--font-zen-mono);
|
|
--sans:var(--font-zen-sans);
|
|
/* Controls are pills and surfaces are soft, as on hanzo.ai. */
|
|
--r:999px; --r-soft:14px;
|
|
}
|
|
*{box-sizing:border-box}
|
|
html{-webkit-text-size-adjust:100%}
|
|
body{margin:0;background:var(--ground);color:var(--ink);font-family:var(--sans);
|
|
font-size:16px;line-height:1.65;-webkit-font-smoothing:antialiased}
|
|
.wrap{max-width:60rem;margin:0 auto;padding:0 1.75rem}
|
|
|
|
/* The dot field. A halftone made of one repeating radial-gradient, faded out
|
|
along the diagonal by a mask so it reads as a wave rather than wallpaper. */
|
|
/* No overflow clip here: the dot field is inset to the box already, and a clip
|
|
would cut off the copy-prompt menu. */
|
|
.hero{position:relative;border-bottom:1px solid var(--line)}
|
|
.hero::before{content:"";position:absolute;inset:0;
|
|
background-image:radial-gradient(var(--line-hi) 1px, transparent 1px);
|
|
background-size:7px 7px;
|
|
-webkit-mask-image:linear-gradient(115deg,#000 0%,rgba(0,0,0,.55) 38%,transparent 72%);
|
|
mask-image:linear-gradient(115deg,#000 0%,rgba(0,0,0,.55) 38%,transparent 72%);
|
|
pointer-events:none}
|
|
.hero .wrap{position:relative;padding-top:5.5rem;padding-bottom:4rem}
|
|
|
|
nav{border-bottom:1px solid var(--line)}
|
|
nav .in{max-width:60rem;margin:0 auto;padding:1.1rem 1.75rem;display:flex;
|
|
justify-content:space-between;align-items:center;gap:1rem}
|
|
.brand{font-family:var(--mono);font-size:.85rem;letter-spacing:.14em;
|
|
text-transform:uppercase;color:var(--ink);text-decoration:none;
|
|
display:inline-flex;align-items:center;gap:.6rem}
|
|
.brand i{font-style:normal;color:var(--faint)}
|
|
.mark{width:1.15em;height:1.15em;display:block;flex:0 0 auto}
|
|
nav a.up{font-family:var(--mono);font-size:.72rem;letter-spacing:.14em;
|
|
text-transform:uppercase;color:var(--soft);text-decoration:none}
|
|
nav a.up:hover{color:var(--ink)}
|
|
|
|
h1{font-family:var(--mono);font-size:clamp(2rem,5.4vw,3.4rem);line-height:1.12;
|
|
letter-spacing:-.02em;margin:0 0 1.1rem;font-weight:500;text-wrap:balance}
|
|
h1 em{font-style:normal;color:var(--faint)}
|
|
.lede{color:var(--soft);font-size:1.05rem;max-width:52ch;margin:0 0 2.25rem}
|
|
|
|
.btns{display:flex;flex-wrap:wrap;gap:.7rem;margin:0 0 .5rem;align-items:center}
|
|
.btn{display:inline-flex;align-items:center;gap:.5rem;font-family:var(--mono);
|
|
font-size:.72rem;letter-spacing:.14em;text-transform:uppercase;
|
|
padding:.8rem 1.4rem;border:1px solid var(--line-hi);border-radius:var(--r);
|
|
color:var(--ink);text-decoration:none;background:transparent;cursor:pointer;
|
|
line-height:1.2;font-weight:400}
|
|
.btn:hover{background:#141414;text-decoration:none;border-color:var(--soft)}
|
|
.btn.solid{background:var(--ink);color:#000;border-color:var(--ink)}
|
|
.btn.solid:hover{background:#d8d8d4}
|
|
.btn .chev{width:.7em;height:.7em;flex:0 0 auto;transition:transform .15s ease}
|
|
.btn[aria-expanded="true"] .chev{transform:rotate(180deg)}
|
|
|
|
/* Copy-prompt selector. Each row copies the prompt for its own target. */
|
|
.copy{position:relative;display:inline-flex}
|
|
.menu{position:absolute;top:calc(100% + .5rem);left:0;z-index:20;min-width:21rem;
|
|
background:var(--panel);border:1px solid var(--line-hi);border-radius:var(--r-soft);
|
|
padding:.4rem;display:flex;flex-direction:column;
|
|
box-shadow:0 12px 32px rgba(0,0,0,.75)}
|
|
.menu[hidden]{display:none}
|
|
.mi{display:flex;align-items:center;gap:.7rem;width:100%;text-align:left;
|
|
padding:.62rem .8rem;border:0;border-radius:calc(var(--r-soft) - .35rem);
|
|
background:transparent;color:var(--ink);font-family:var(--mono);font-size:.76rem;
|
|
letter-spacing:.04em;text-decoration:none;cursor:pointer;line-height:1.3;
|
|
white-space:nowrap}
|
|
.mi:hover,.mi:focus-visible{background:#181818;outline:none}
|
|
.mi .mark{width:1.05rem;height:1.05rem;color:var(--soft)}
|
|
.mi:hover .mark{color:var(--ink)}
|
|
.mi u{margin-left:auto;color:var(--faint);text-decoration:none;font-size:.68rem;
|
|
letter-spacing:.1em;text-transform:uppercase}
|
|
|
|
/* Works-with bar: one scrolling row, name and mark per agent. */
|
|
.strip{border-bottom:1px solid var(--line);background:var(--panel)}
|
|
.strip .in{max-width:60rem;margin:0 auto;padding:.9rem 1.75rem;display:flex;
|
|
align-items:center;gap:1.25rem;font-family:var(--mono);font-size:.72rem;
|
|
letter-spacing:.1em;text-transform:uppercase;color:var(--faint)}
|
|
.strip b{color:var(--soft);font-weight:400;flex:0 0 auto;
|
|
border-right:1px solid var(--line);padding-right:1.25rem;align-self:stretch;
|
|
display:flex;align-items:center}
|
|
.bar{display:flex;align-items:center;gap:.5rem;flex:1 1 auto;min-width:0;
|
|
overflow-x:auto;scrollbar-width:none;scroll-snap-type:x proximity;
|
|
-webkit-overflow-scrolling:touch;
|
|
-webkit-mask-image:linear-gradient(90deg,#000 calc(100% - 3rem),transparent);
|
|
mask-image:linear-gradient(90deg,#000 calc(100% - 3rem),transparent)}
|
|
.bar::-webkit-scrollbar{display:none}
|
|
.agent{display:inline-flex;align-items:center;gap:.45rem;flex:0 0 auto;
|
|
padding:.4rem .85rem;border:1px solid var(--line);border-radius:var(--r);
|
|
color:var(--soft);white-space:nowrap;scroll-snap-align:start;
|
|
text-decoration:none;background:var(--ground)}
|
|
.agent:hover{color:var(--ink);border-color:var(--line-hi);text-decoration:none}
|
|
.agent .mark{width:.95rem;height:.95rem}
|
|
|
|
h2{font-family:var(--mono);font-size:1.1rem;letter-spacing:-.01em;margin:2.75rem 0 .75rem;font-weight:500}
|
|
h3{font-size:1rem;margin:2rem 0 .5rem;font-weight:600}
|
|
h4,h5,h6{font-size:.94rem;margin:1.5rem 0 .5rem;font-weight:600}
|
|
p{margin:0 0 1rem;max-width:70ch}
|
|
ul,ol{margin:0 0 1rem;padding-left:1.35rem;max-width:70ch}
|
|
li{margin:.22rem 0}
|
|
a{color:var(--ink);text-decoration:underline;text-underline-offset:3px;
|
|
text-decoration-color:var(--line-hi)}
|
|
a:hover{text-decoration-color:var(--ink)}
|
|
a:focus-visible{outline:1px solid var(--ink);outline-offset:3px}
|
|
code{font-family:var(--mono);font-size:.85em;background:var(--panel);
|
|
border:1px solid var(--line);border-radius:5px;padding:.1em .36em}
|
|
pre{background:var(--panel);border:1px solid var(--line);padding:1rem 1.1rem;
|
|
border-radius:var(--r-soft);overflow-x:auto;margin:0 0 1.25rem}
|
|
pre code{background:none;border:0;padding:0;font-size:.82rem;line-height:1.7}
|
|
blockquote{margin:0 0 1rem;padding-left:1rem;border-left:1px solid var(--line-hi);color:var(--soft)}
|
|
hr{border:0;border-top:1px solid var(--line);margin:2.5rem 0}
|
|
table{border-collapse:collapse;width:100%;margin:0 0 1.25rem;font-size:.92rem;
|
|
display:block;overflow-x:auto}
|
|
th,td{border:1px solid var(--line);padding:.5rem .7rem;text-align:left}
|
|
|
|
.curl{display:block;background:var(--panel);border:1px solid var(--line);
|
|
border-radius:var(--r-soft);padding:.95rem 1.1rem;font-family:var(--mono);
|
|
font-size:.8rem;color:var(--soft);overflow-x:auto;margin:0 0 2.5rem;
|
|
white-space:pre;line-height:1.9}
|
|
.curl b{color:var(--ink);font-weight:400}
|
|
.curl i{font-style:normal;color:var(--faint)}
|
|
|
|
/* index */
|
|
#q{width:100%;padding:.85rem 1.2rem;font-family:var(--mono);font-size:.85rem;color:var(--ink);
|
|
background:var(--panel);border:1px solid var(--line);border-radius:var(--r);margin:0 0 .6rem}
|
|
#q::placeholder{color:var(--faint);text-transform:uppercase;letter-spacing:.1em;font-size:.72rem}
|
|
#q:focus{outline:none;border-color:var(--soft)}
|
|
#count{color:var(--faint);font-family:var(--mono);font-size:.72rem;letter-spacing:.1em;
|
|
text-transform:uppercase;margin:0 0 2rem}
|
|
.cat{margin:2.5rem 0 .5rem;font-family:var(--mono);font-size:.7rem;letter-spacing:.16em;
|
|
text-transform:uppercase;color:var(--faint);border-bottom:1px solid var(--line);
|
|
padding-bottom:.45rem}
|
|
.row{display:block;padding:.6rem .9rem;margin:0 -.9rem;border-bottom:1px solid var(--line);
|
|
border-radius:var(--r-soft);text-decoration:none}
|
|
.row:hover{background:#0a0a0a}
|
|
.nm{font-family:var(--mono);font-size:.84rem;color:var(--ink);display:block}
|
|
.ds{color:var(--soft);font-size:.9rem;display:block;margin-top:.12rem}
|
|
footer{margin-top:4rem;padding:2rem 0 4rem;border-top:1px solid var(--line);
|
|
color:var(--faint);font-size:.85rem}
|
|
footer a{color:var(--soft)}
|
|
.fmap{display:grid;gap:1.75rem;grid-template-columns:repeat(auto-fit,minmax(11rem,1fr));
|
|
margin:0 0 2rem;font-size:.85rem}
|
|
.fmap b{display:block;font-family:var(--mono);font-size:.68rem;letter-spacing:.16em;
|
|
text-transform:uppercase;color:var(--faint);font-weight:400;margin:0 0 .6rem;
|
|
border-bottom:1px solid var(--line);padding-bottom:.4rem}
|
|
.fmap a{display:inline-block;color:var(--ink);text-decoration:none;margin:.3rem .35rem 0 0}
|
|
.fmap a:hover{text-decoration:underline}
|
|
.fmap div{color:var(--faint);line-height:1.5}
|
|
.fend{border-top:1px solid var(--line);padding-top:1.25rem;font-size:.82rem}
|
|
|
|
@media (max-width:640px){
|
|
.hero .wrap{padding-top:3.25rem;padding-bottom:2.75rem}
|
|
.btns{gap:.5rem}
|
|
.btn{padding:.75rem 1.15rem;font-size:.68rem}
|
|
.copy,.copy .btn{width:100%;justify-content:center}
|
|
.menu{left:0;right:0;min-width:0}
|
|
.strip .in{flex-direction:column;align-items:flex-start;gap:.6rem;padding:.9rem 1.75rem}
|
|
.strip b{border-right:0;padding-right:0}
|
|
.bar{width:100%}
|
|
}
|
|
</style>
|
|
<link rel="icon" href="/favicon.svg">
|
|
<link rel="canonical" href="https://hanzoskills.com/high-memory-usage">
|
|
<link rel="alternate" type="text/markdown" href="/high-memory-usage.md">
|
|
<script type="module" src="/webmcp.js"></script>
|
|
</head>
|
|
<!-- Reading this with a machine? Every page here is also plain markdown.
|
|
This one: https://hanzoskills.com/high-memory-usage.md
|
|
Index: https://hanzoskills.com/llms.txt
|
|
Corpus: https://hanzoskills.com/all.md
|
|
Append .md to any path for the markdown twin. -->
|
|
<body>
|
|
<nav><div class="in">
|
|
<a class="brand" href="/"><svg class="mark" viewBox="0 0 67 67" fill="currentColor" aria-hidden="true"><path d="M22.21 67V44.6369H0V67H22.21Z"/><path d="M66.7038 22.3184H22.2534L0.0878906 44.6367H44.4634L66.7038 22.3184Z"/><path d="M22.21 0H0V22.3184H22.21V0Z"/><path d="M66.7198 0H44.5098V22.3184H66.7198V0Z"/><path d="M66.7198 67V44.6369H44.5098V67H66.7198Z"/></svg> HANZO<i>SKILLS</i></a>
|
|
<a class="up" href="/">All skills</a>
|
|
</div></nav>
|
|
|
|
<div class="wrap">
|
|
<h1>high-memory-usage</h1><p class="lede">This alert fires when memory usage exceeds 90% for more than 5 minutes on any node.</p><h1>Alert: HighMemoryUsage</h1>
|
|
<blockquote><p><strong>Status</strong>: Active <strong>Last Updated</strong>: 2025-10-27 <strong>Owner</strong>: Platform Team <strong>Severity</strong>: Critical</p></blockquote>
|
|
<hr>
|
|
<h2>Overview</h2>
|
|
<p>This alert fires when memory usage exceeds 90% for more than 5 minutes on any node.</p>
|
|
<p><strong>What is being measured</strong>: <code>node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes</code></p>
|
|
<p><strong>Why it matters</strong>: High memory usage can lead to:</p>
|
|
<ul><li>OOM (Out of Memory) killer terminating processes</li><li>System instability and crashes</li><li>Service degradation or unavailability</li><li>Swap thrashing (severe performance impact)</li></ul>
|
|
<hr>
|
|
<h2>Symptoms</h2>
|
|
<p>What you'll observe:</p>
|
|
<ul><li>Memory usage > 90% in monitoring</li><li>Pods being evicted (Kubernetes)</li><li>Swap usage increasing</li><li>System slowness</li><li>Application crashes or restarts</li><li>"Out of memory" errors in logs</li></ul>
|
|
<hr>
|
|
<h2>Impact</h2>
|
|
<h3>User Impact</h3>
|
|
<ul><li>[x] Performance degraded (slow responses)</li><li>[ ] Service unavailable (if OOM occurs)</li><li>[ ] Feature disabled</li><li>[ ] No user impact</li></ul>
|
|
<h3>Business Impact</h3>
|
|
<ul><li>Revenue impact: Medium (degraded performance)</li><li>Customer impact: Varies by service affected</li><li>SLA impact: Yes (if availability drops below threshold)</li></ul>
|
|
<hr>
|
|
<h2>Severity</h2>
|
|
<p><strong>Current Severity</strong>: Critical (if > 95%), Warning (if > 90%)</p>
|
|
<h3>Escalation Criteria</h3>
|
|
<p>Escalate to <strong>Critical</strong> if:</p>
|
|
<ul><li>Memory usage > 95%</li><li>OOM events detected</li><li>Multiple pods evicted</li><li>Service impact confirmed</li></ul>
|
|
<p>Escalate to <strong>Manager</strong> if:</p>
|
|
<ul><li>No improvement after 30 minutes</li><li>Cluster-wide memory pressure</li><li>Customer complaints received</li></ul>
|
|
<hr>
|
|
<h2>Diagnosis</h2>
|
|
<h3>1. Verify the Alert</h3>
|
|
<pre><code class="lang-bash"># SSH to affected node
|
|
ssh [node-ip]
|
|
|
|
# Check current memory usage
|
|
free -h
|
|
|
|
# Example output:
|
|
# total used free shared buff/cache available
|
|
# Mem: 15Gi 14Gi 100Mi 50Mi 900Mi 800Mi
|
|
# Swap: 2.0Gi 1.5Gi 500Mi
|
|
|
|
# Check memory usage percentage
|
|
echo "scale=2; $(grep MemAvailable /proc/meminfo | awk '{print $2}') / $(grep MemTotal /proc/meminfo | awk '{print $2}') * 100" | bc
|
|
|
|
# View top memory consumers
|
|
top -o %MEM
|
|
|
|
# Or use htop (better visualization)
|
|
htop</code></pre>
|
|
<p><strong>Expected</strong>: Memory available > 10% (> 1.5GB on 16GB node) <strong>Actual</strong>: Memory available < 10%</p>
|
|
<h3>2. Identify Memory Consumers</h3>
|
|
<pre><code class="lang-bash"># Top 10 processes by memory (Linux)
|
|
ps aux --sort=-%mem | head -11
|
|
|
|
# Detailed per-process memory breakdown
|
|
ps aux | awk '{print $6/1024 " MB\t" $11}' | sort -n
|
|
|
|
# Container memory usage (Docker)
|
|
docker stats --no-stream --format "table {{.Container}}\t{{.MemUsage}}"
|
|
|
|
# Kubernetes pod memory usage
|
|
kubectl top pods --all-namespaces --sort-by=memory
|
|
|
|
# Check for memory leaks in specific process
|
|
pmap -x [PID] | tail -1</code></pre>
|
|
<h3>3. Check for Memory Leaks</h3>
|
|
<pre><code class="lang-bash"># Monitor process memory over time
|
|
watch -n 5 'ps aux | grep [process-name]'
|
|
|
|
# Check for zombie processes
|
|
ps aux | grep defunct
|
|
|
|
# View kernel memory slab usage
|
|
sudo slabtop
|
|
|
|
# Check page cache usage
|
|
cat /proc/meminfo | grep -E '(Cached|Buffers|Slab)'
|
|
|
|
# OOM killer logs
|
|
dmesg | grep -i "out of memory"
|
|
sudo journalctl -xe | grep -i oom</code></pre>
|
|
<h3>4. Review Recent Changes</h3>
|
|
<pre><code class="lang-bash"># Recent deployments (Kubernetes)
|
|
kubectl rollout history deployment/[name]
|
|
|
|
# Recent pod restarts
|
|
kubectl get pods --all-namespaces --field-selector=status.phase=Running --sort-by=.status.startTime
|
|
|
|
# Check resource limits
|
|
kubectl describe pod [pod-name] | grep -A 5 "Limits"
|
|
|
|
# Git commits (if applicable)
|
|
git log --since="6 hours ago" --oneline</code></pre>
|
|
<h3>Common Causes</h3>
|
|
<ol><li><strong>Memory leak</strong>: Application not releasing memory</li></ol>
|
|
<ul><li>Confirm: Memory usage steadily increasing over time</li><li>Check: Application logs for errors, heap dumps</li></ul>
|
|
<ol><li><strong>Increased load</strong>: More traffic than capacity</li></ol>
|
|
<ul><li>Confirm: Request rate increased, more pods running</li><li>Check: Request metrics, pod count</li></ul>
|
|
<ol><li><strong>Large data processing</strong>: Batch job or analytics</li></ol>
|
|
<ul><li>Confirm: Scheduled job running, large dataset being processed</li><li>Check: Cron jobs, data pipeline status</li></ul>
|
|
<ol><li><strong>Page cache</strong>: Linux caching disk I/O</li></ol>
|
|
<ul><li>Confirm: "Cached" high in <code>/proc/meminfo</code>, "Available" still reasonable</li><li>Action: Usually safe, kernel will free if needed</li></ul>
|
|
<ol><li><strong>Resource limit mismatch</strong>: Pod limits too high</li></ol>
|
|
<ul><li>Confirm: Sum of pod limits > node capacity</li><li>Check: <code>kubectl describe node [node]</code></li></ul>
|
|
<hr>
|
|
<h2>Remediation</h2>
|
|
<h3>Immediate Actions (< 5 minutes)</h3>
|
|
<p><strong>Objective</strong>: Prevent OOM, keep system stable</p>
|
|
<ol><li><strong>Clear page cache</strong> (temporary relief):</li></ol>
|
|
<p> ``<code>bash # This is safe - kernel will rebuild cache as needed sync && echo 3 > /proc/sys/vm/drop_caches </code>`<code> <strong>Expected outcome</strong>: Memory usage drops 10-20% <strong>Verification</strong>: Run </code>free -h` again</p>
|
|
<ol><li><strong>Identify and restart leaking service</strong> (if known):</li></ol>
|
|
<p> ```bash # Kubernetes kubectl rollout restart deployment/[name]</p>
|
|
<p> # Systemd service sudo systemctl restart [service]</p>
|
|
<p> # Docker container docker restart [container-id] ``` <strong>Expected outcome</strong>: Memory released, service recovers <strong>Verification</strong>: Check memory usage of new pods/processes</p>
|
|
<ol><li><strong>Scale horizontally</strong> (if auto-scaling available):</li></ol>
|
|
<p> ```bash # Kubernetes - add more replicas kubectl scale deployment [name] --replicas=[N+2]</p>
|
|
<p> # Cloud - add instances aws autoscaling set-desired-capacity --auto-scaling-group-name [name] --desired-capacity [N+1] ``<code> <strong>Expected outcome</strong>: Load distributed, per-instance memory reduced <strong>Verification</strong>: </code>kubectl top pods` shows lower memory per pod</p>
|
|
<ol><li><strong>Evict low-priority workloads</strong> (emergency only):</li></ol>
|
|
<p> ```bash # Kubernetes - delete non-critical pods kubectl delete pod [pod-name] -n [namespace]</p>
|
|
<p> # Stop batch jobs kubectl delete job [job-name] ```</p>
|
|
<p><strong>If above don't work</strong>: Escalate to platform team lead immediately</p>
|
|
<h3>Short-term Fix (< 1 hour)</h3>
|
|
<p><strong>Objective</strong>: Stable state while investigating root cause</p>
|
|
<ol><li><strong>Adjust resource limits</strong> (if over-committed):</li></ol>
|
|
<p> ```yaml # Edit deployment kubectl edit deployment [name]</p>
|
|
<p> # Set appropriate limits resources: requests: memory: "256Mi" limits: memory: "512Mi" ```</p>
|
|
<ol><li><strong>Enable swap</strong> (temporary, not recommended long-term):</li></ol>
|
|
<p> ```bash # Check current swap swapon -s</p>
|
|
<p> # Create swap file (if none exists) sudo fallocate -l 4G /swapfile sudo chmod 600 /swapfile sudo mkswap /swapfile sudo swapon /swapfile ``` <strong>Note</strong>: This buys time but doesn't fix the problem</p>
|
|
<ol><li><strong>Tune application memory settings</strong>:</li></ol>
|
|
<p> ```bash # Java applications - reduce heap size export JAVA_OPTS="-Xmx2g -Xms512m"</p>
|
|
<p> # Node.js - set max old space export NODE_OPTIONS="--max-old-space-size=2048"</p>
|
|
<p> # Python - limit worker memory export WORKER_MEMORY_LIMIT=512M ```</p>
|
|
<ol><li><strong>Implement pod disruption budget</strong> (prevent cascading failures):</li></ol>
|
|
<p> ``<code>yaml apiVersion: policy/v1 kind: PodDisruptionBudget metadata: name: [app]-pdb spec: minAvailable: 2 selector: matchLabels: app: [app] </code>``</p>
|
|
<h3>Long-term Fix</h3>
|
|
<ol><li><strong>Increase node resources</strong>:</li></ol>
|
|
<ul><li>Vertical scaling: Upgrade to larger instances</li><li>Horizontal scaling: Add more nodes</li><li>Auto-scaling: Configure memory-based scaling</li></ul>
|
|
<ol><li><strong>Optimize application memory</strong>:</li></ol>
|
|
<ul><li>Profile and fix memory leaks</li><li>Optimize data structures</li><li>Implement caching strategies</li><li>Add memory limits and monitoring</li></ul>
|
|
<ol><li><strong>Improve resource management</strong>:</li></ol>
|
|
<ul><li>Set accurate resource requests/limits</li><li>Use pod priority classes</li><li>Enable vertical pod autoscaler</li><li>Implement memory quotas per namespace</li></ul>
|
|
<hr>
|
|
<h2>Rollback</h2>
|
|
<p>If remediation makes things worse:</p>
|
|
<pre><code class="lang-bash"># Rollback deployment
|
|
kubectl rollout undo deployment/[name]
|
|
|
|
# Restore previous auto-scaling settings
|
|
kubectl apply -f autoscaling-backup.yaml
|
|
|
|
# Disable swap (if enabled as temp fix)
|
|
sudo swapoff /swapfile
|
|
sudo rm /swapfile</code></pre>
|
|
<hr>
|
|
<h2>Communication</h2>
|
|
<h3>Who to Notify</h3>
|
|
<p>| Stakeholder | When | How | |-------------|------|-----| | Platform Team | Immediately | Slack #platform-oncall | | Engineering Manager | If not resolved in 30min | Slack + Phone | | Application Owners | If specific app identified | Slack team channel | | Customers | If service degraded > 15min | Status page |</p>
|
|
<h3>Communication Templates</h3>
|
|
<p><strong>Initial notification</strong>:</p>
|
|
<pre><code>[INCIDENT] High Memory Usage on [node/cluster]
|
|
|
|
Impact: [service] may experience slow responses
|
|
Started: [time]
|
|
Status: Investigating memory consumption
|
|
Next update: [+15min]
|
|
|
|
Incident lead: [oncall engineer]
|
|
Slack: #incident-[ID]</code></pre>
|
|
<hr>
|
|
<h2>Prevention</h2>
|
|
<h3>Monitoring Improvements</h3>
|
|
<ul><li>[x] Alert on memory > 90% (existing)</li><li>[ ] Alert on OOM events</li><li>[ ] Monitor memory growth rate</li><li>[ ] Dashboard for memory breakdown</li></ul>
|
|
<h3>Process Improvements</h3>
|
|
<ul><li>[ ] Memory load testing before deployment</li><li>[ ] Automatic heap dumps on high memory</li><li>[ ] Regular memory profiling of applications</li></ul>
|
|
<h3>Code/Infrastructure Changes</h3>
|
|
<ul><li>[ ] Implement memory limits on all pods</li><li>[ ] Add memory leak detection</li><li>[ ] Configure horizontal pod autoscaling</li><li>[ ] Optimize memory-intensive operations</li></ul>
|
|
<hr>
|
|
<h2>Validation</h2>
|
|
<h3>Post-Remediation Checks</h3>
|
|
<pre><code class="lang-bash"># 1. Memory usage normal
|
|
free -h | grep Mem
|
|
# Available should be > 20%
|
|
|
|
# 2. No OOM events in last 10 minutes
|
|
dmesg | grep -i oom | tail -5
|
|
|
|
# 3. All pods running
|
|
kubectl get pods --all-namespaces | grep -v Running
|
|
|
|
# 4. Application metrics normal
|
|
# Check dashboard: [URL]
|
|
|
|
# 5. No pending pod evictions
|
|
kubectl get events --all-namespaces | grep Evicted</code></pre>
|
|
<p>All checks passing? ✓ Incident resolved</p>
|
|
<hr>
|
|
<h2>Related Information</h2>
|
|
<h3>Dashboards</h3>
|
|
<ul><li>Node Resources: https://grafana.example.com/d/node-resources</li><li>Pod Memory: https://grafana.example.com/d/pod-memory</li><li>OOM Events: https://grafana.example.com/d/oom-events</li></ul>
|
|
<h3>Related Alerts</h3>
|
|
<ul><li><code>OOMKiller</code>: OOM events occurred</li><li><code>HighSwapUsage</code>: Swapping indicates memory pressure</li><li><code>PodEvicted</code>: Pods evicted due to resource pressure</li></ul>
|
|
<h3>Documentation</h3>
|
|
<ul><li>Memory Troubleshooting: https://docs.example.com/memory</li><li>Resource Limits Guide: https://docs.example.com/resources</li><li>Kubernetes Best Practices: https://docs.example.com/k8s</li></ul>
|
|
<h3>Previous Incidents</h3>
|
|
<ul><li>INC-234: 2025-08-15 - Java heap leak in API service</li><li>INC-567: 2025-09-22 - Redis cache grew unbounded</li><li>INC-890: 2025-10-10 - Pod limits too high, node over-committed</li></ul>
|
|
<hr>
|
|
<h2>Appendix</h2>
|
|
<h3>Memory Metrics Explained</h3>
|
|
<pre><code>MemTotal : Total physical RAM
|
|
MemFree : Unused RAM
|
|
MemAvailable : RAM available for starting new applications (includes reclaimable cache)
|
|
Buffers : Temporary storage for block devices
|
|
Cached : Page cache from files
|
|
Slab : Kernel data structures
|
|
SwapTotal : Total swap space
|
|
SwapFree : Unused swap space</code></pre>
|
|
<p><strong>Key metric</strong>: <code>MemAvailable</code> (not <code>MemFree</code>)</p>
|
|
<h3>Kubernetes Memory Metrics</h3>
|
|
<pre><code class="lang-bash"># Pod memory requests vs limits
|
|
kubectl describe node [node] | grep -A 5 "Allocated resources"
|
|
|
|
# Memory pressure eviction thresholds
|
|
kubectl get nodes -o json | jq '.items[].status.allocatable.memory'
|
|
|
|
# Top memory-consuming namespaces
|
|
kubectl top pods -A | awk '{print $1, $4}' | sort -k2 -rh | head -20</code></pre>
|
|
<hr>
|
|
<h2>Change Log</h2>
|
|
<p>| Date | Author | Change | |------|--------|--------| | 2025-10-27 | Platform Team | Initial version | | 2025-10-27 | SRE | Added Kubernetes-specific steps |</p>
|
|
<hr>
|
|
<h2>Feedback</h2>
|
|
<p>Found this runbook helpful? Found an issue?</p>
|
|
<ul><li>GitHub: https://github.com/example/runbooks/issues</li><li>Slack: #platform-runbooks</li><li>Email: [email protected]</li></ul>
|
|
<footer>
|
|
<div class="fmap">
|
|
<div><b>Build</b>
|
|
<a href="https://hanzo.app">hanzo.app</a> app builder
|
|
<a href="https://hanzo.codes">hanzo.codes</a> code with an agent
|
|
<a href="https://hanzoskills.com">hanzoskills.com</a> what an agent reads</div>
|
|
<div><b>Run</b>
|
|
<a href="https://console.hanzo.ai">console.hanzo.ai</a> cloud console
|
|
<a href="https://api.hanzo.ai">api.hanzo.ai</a> the API</div>
|
|
<div><b>Account</b>
|
|
<a href="https://hanzo.id">hanzo.id</a> sign in
|
|
<a href="https://pay.hanzo.ai">pay.hanzo.ai</a> top up
|
|
<a href="https://billing.hanzo.ai">billing.hanzo.ai</a> invoices and usage</div>
|
|
<div><b>Company</b>
|
|
<a href="https://hanzo.ai">hanzo.ai</a> canonical
|
|
<a href="https://hanzo.ventures">hanzo.ventures</a> ventures</div>
|
|
</div>
|
|
<div class="fend"><a href="/high-memory-usage.md">This page as plain text</a> · <a href="/llms.txt">Index for agents</a></div>
|
|
</footer>
|
|
</div></body></html> |