Files
skills/public/debugging-performance-profiling.html
zeekayandHanzo Dev 9e10c1c10e Each llms index points at the other two
hanzoskills.com/llms.txt now opens with the install line, the API
reference index and the company index, so an agent that lands on any
one of the three can find the rest.

robots.txt claimed 537 documents while llms.txt counts 920. The number
is gone from robots.txt; llms.txt derives it.

public/ is regenerated from scripts/site.py, which it had fallen
behind: the copy-prompt menu and agent bar were in the generator and
not in the committed pages. tests/site.mjs passes against the result.

Co-authored-by: Hanzo Dev <[email protected]>
2026-09-11 00:27:59 -07:00

702 lines
29 KiB
HTML

<!doctype html>
<html lang="en"><head>
<meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1">
<title>debugging-performance-profiling</title>
<meta name="color-scheme" content="dark">
<style>
/* Committed to one visual world: a dark instrument panel. No light variant —
the dot field and the hairline rules only read on black, and a naive
inversion would wash both out. Every colour is painted explicitly so the page
never borrows a host background. */
@font-face{font-family:'Zen';src:url('/fonts/Zen-Variable.woff2') format('woff2');font-weight:100 900;font-style:normal;font-display:swap}
@font-face{font-family:'Zen';src:url('/fonts/Zen-Italic-Variable.woff2') format('woff2');font-weight:100 900;font-style:italic;font-display:swap}
@font-face{font-family:'Zen Mono';src:url('/fonts/ZenMono-Variable.woff2') format('woff2');font-weight:100 900;font-style:normal;font-display:swap}
@font-face{font-family:'Zen Mono';src:url('/fonts/ZenMono-Italic-Variable.woff2') format('woff2');font-weight:100 900;font-style:italic;font-display:swap}
@font-face{font-family:'Zen Pixel Square';src:url('/fonts/ZenPixel-Square.woff2') format('woff2');font-style:normal;font-display:swap}
:root{
--ink:#f4f4f2; --soft:#9c9c98; --faint:#6a6a66; --ground:#000; --panel:#0a0a0a;
--line:#232323; --line-hi:#3a3a3a; --accent:#f4f4f2;
--font-zen-sans:'Zen',-apple-system,BlinkMacSystemFont,"Segoe UI",Helvetica,Arial,sans-serif;
--font-zen-mono:'Zen Mono',ui-monospace,"SF Mono",SFMono-Regular,Menlo,Consolas,monospace;
--font-zen-pixel-square:'Zen Pixel Square',monospace;
--mono:var(--font-zen-mono);
--sans:var(--font-zen-sans);
/* Controls are pills and surfaces are soft, as on hanzo.ai. */
--r:999px; --r-soft:14px;
}
*{box-sizing:border-box}
html{-webkit-text-size-adjust:100%}
body{margin:0;background:var(--ground);color:var(--ink);font-family:var(--sans);
font-size:16px;line-height:1.65;-webkit-font-smoothing:antialiased}
.wrap{max-width:60rem;margin:0 auto;padding:0 1.75rem}
/* The dot field. A halftone made of one repeating radial-gradient, faded out
along the diagonal by a mask so it reads as a wave rather than wallpaper. */
/* No overflow clip here: the dot field is inset to the box already, and a clip
would cut off the copy-prompt menu. */
.hero{position:relative;border-bottom:1px solid var(--line)}
.hero::before{content:"";position:absolute;inset:0;
background-image:radial-gradient(var(--line-hi) 1px, transparent 1px);
background-size:7px 7px;
-webkit-mask-image:linear-gradient(115deg,#000 0%,rgba(0,0,0,.55) 38%,transparent 72%);
mask-image:linear-gradient(115deg,#000 0%,rgba(0,0,0,.55) 38%,transparent 72%);
pointer-events:none}
.hero .wrap{position:relative;padding-top:5.5rem;padding-bottom:4rem}
nav{border-bottom:1px solid var(--line)}
nav .in{max-width:60rem;margin:0 auto;padding:1.1rem 1.75rem;display:flex;
justify-content:space-between;align-items:center;gap:1rem}
.brand{font-family:var(--mono);font-size:.85rem;letter-spacing:.14em;
text-transform:uppercase;color:var(--ink);text-decoration:none;
display:inline-flex;align-items:center;gap:.6rem}
.brand i{font-style:normal;color:var(--faint)}
.mark{width:1.15em;height:1.15em;display:block;flex:0 0 auto}
nav a.up{font-family:var(--mono);font-size:.72rem;letter-spacing:.14em;
text-transform:uppercase;color:var(--soft);text-decoration:none}
nav a.up:hover{color:var(--ink)}
h1{font-family:var(--mono);font-size:clamp(2rem,5.4vw,3.4rem);line-height:1.12;
letter-spacing:-.02em;margin:0 0 1.1rem;font-weight:500;text-wrap:balance}
h1 em{font-style:normal;color:var(--faint)}
.lede{color:var(--soft);font-size:1.05rem;max-width:52ch;margin:0 0 2.25rem}
.btns{display:flex;flex-wrap:wrap;gap:.7rem;margin:0 0 .5rem;align-items:center}
.btn{display:inline-flex;align-items:center;gap:.5rem;font-family:var(--mono);
font-size:.72rem;letter-spacing:.14em;text-transform:uppercase;
padding:.8rem 1.4rem;border:1px solid var(--line-hi);border-radius:var(--r);
color:var(--ink);text-decoration:none;background:transparent;cursor:pointer;
line-height:1.2;font-weight:400}
.btn:hover{background:#141414;text-decoration:none;border-color:var(--soft)}
.btn.solid{background:var(--ink);color:#000;border-color:var(--ink)}
.btn.solid:hover{background:#d8d8d4}
.btn .chev{width:.7em;height:.7em;flex:0 0 auto;transition:transform .15s ease}
.btn[aria-expanded="true"] .chev{transform:rotate(180deg)}
/* Copy-prompt selector. Each row copies the prompt for its own target. */
.copy{position:relative;display:inline-flex}
.menu{position:absolute;top:calc(100% + .5rem);left:0;z-index:20;min-width:21rem;
background:var(--panel);border:1px solid var(--line-hi);border-radius:var(--r-soft);
padding:.4rem;display:flex;flex-direction:column;
box-shadow:0 12px 32px rgba(0,0,0,.75)}
.menu[hidden]{display:none}
.mi{display:flex;align-items:center;gap:.7rem;width:100%;text-align:left;
padding:.62rem .8rem;border:0;border-radius:calc(var(--r-soft) - .35rem);
background:transparent;color:var(--ink);font-family:var(--mono);font-size:.76rem;
letter-spacing:.04em;text-decoration:none;cursor:pointer;line-height:1.3;
white-space:nowrap}
.mi:hover,.mi:focus-visible{background:#181818;outline:none}
.mi .mark{width:1.05rem;height:1.05rem;color:var(--soft)}
.mi:hover .mark{color:var(--ink)}
.mi u{margin-left:auto;color:var(--faint);text-decoration:none;font-size:.68rem;
letter-spacing:.1em;text-transform:uppercase}
/* Works-with bar: one scrolling row, name and mark per agent. */
.strip{border-bottom:1px solid var(--line);background:var(--panel)}
.strip .in{max-width:60rem;margin:0 auto;padding:.9rem 1.75rem;display:flex;
align-items:center;gap:1.25rem;font-family:var(--mono);font-size:.72rem;
letter-spacing:.1em;text-transform:uppercase;color:var(--faint)}
.strip b{color:var(--soft);font-weight:400;flex:0 0 auto;
border-right:1px solid var(--line);padding-right:1.25rem;align-self:stretch;
display:flex;align-items:center}
.bar{display:flex;align-items:center;gap:.5rem;flex:1 1 auto;min-width:0;
overflow-x:auto;scrollbar-width:none;scroll-snap-type:x proximity;
-webkit-overflow-scrolling:touch;
-webkit-mask-image:linear-gradient(90deg,#000 calc(100% - 3rem),transparent);
mask-image:linear-gradient(90deg,#000 calc(100% - 3rem),transparent)}
.bar::-webkit-scrollbar{display:none}
.agent{display:inline-flex;align-items:center;gap:.45rem;flex:0 0 auto;
padding:.4rem .85rem;border:1px solid var(--line);border-radius:var(--r);
color:var(--soft);white-space:nowrap;scroll-snap-align:start;
text-decoration:none;background:var(--ground)}
.agent:hover{color:var(--ink);border-color:var(--line-hi);text-decoration:none}
.agent .mark{width:.95rem;height:.95rem}
h2{font-family:var(--mono);font-size:1.1rem;letter-spacing:-.01em;margin:2.75rem 0 .75rem;font-weight:500}
h3{font-size:1rem;margin:2rem 0 .5rem;font-weight:600}
h4,h5,h6{font-size:.94rem;margin:1.5rem 0 .5rem;font-weight:600}
p{margin:0 0 1rem;max-width:70ch}
ul,ol{margin:0 0 1rem;padding-left:1.35rem;max-width:70ch}
li{margin:.22rem 0}
a{color:var(--ink);text-decoration:underline;text-underline-offset:3px;
text-decoration-color:var(--line-hi)}
a:hover{text-decoration-color:var(--ink)}
a:focus-visible{outline:1px solid var(--ink);outline-offset:3px}
code{font-family:var(--mono);font-size:.85em;background:var(--panel);
border:1px solid var(--line);border-radius:5px;padding:.1em .36em}
pre{background:var(--panel);border:1px solid var(--line);padding:1rem 1.1rem;
border-radius:var(--r-soft);overflow-x:auto;margin:0 0 1.25rem}
pre code{background:none;border:0;padding:0;font-size:.82rem;line-height:1.7}
blockquote{margin:0 0 1rem;padding-left:1rem;border-left:1px solid var(--line-hi);color:var(--soft)}
hr{border:0;border-top:1px solid var(--line);margin:2.5rem 0}
table{border-collapse:collapse;width:100%;margin:0 0 1.25rem;font-size:.92rem;
display:block;overflow-x:auto}
th,td{border:1px solid var(--line);padding:.5rem .7rem;text-align:left}
.curl{display:block;background:var(--panel);border:1px solid var(--line);
border-radius:var(--r-soft);padding:.95rem 1.1rem;font-family:var(--mono);
font-size:.8rem;color:var(--soft);overflow-x:auto;margin:0 0 2.5rem;
white-space:pre;line-height:1.9}
.curl b{color:var(--ink);font-weight:400}
.curl i{font-style:normal;color:var(--faint)}
/* index */
#q{width:100%;padding:.85rem 1.2rem;font-family:var(--mono);font-size:.85rem;color:var(--ink);
background:var(--panel);border:1px solid var(--line);border-radius:var(--r);margin:0 0 .6rem}
#q::placeholder{color:var(--faint);text-transform:uppercase;letter-spacing:.1em;font-size:.72rem}
#q:focus{outline:none;border-color:var(--soft)}
#count{color:var(--faint);font-family:var(--mono);font-size:.72rem;letter-spacing:.1em;
text-transform:uppercase;margin:0 0 2rem}
.cat{margin:2.5rem 0 .5rem;font-family:var(--mono);font-size:.7rem;letter-spacing:.16em;
text-transform:uppercase;color:var(--faint);border-bottom:1px solid var(--line);
padding-bottom:.45rem}
.row{display:block;padding:.6rem .9rem;margin:0 -.9rem;border-bottom:1px solid var(--line);
border-radius:var(--r-soft);text-decoration:none}
.row:hover{background:#0a0a0a}
.nm{font-family:var(--mono);font-size:.84rem;color:var(--ink);display:block}
.ds{color:var(--soft);font-size:.9rem;display:block;margin-top:.12rem}
footer{margin-top:4rem;padding:2rem 0 4rem;border-top:1px solid var(--line);
color:var(--faint);font-size:.85rem}
footer a{color:var(--soft)}
.fmap{display:grid;gap:1.75rem;grid-template-columns:repeat(auto-fit,minmax(11rem,1fr));
margin:0 0 2rem;font-size:.85rem}
.fmap b{display:block;font-family:var(--mono);font-size:.68rem;letter-spacing:.16em;
text-transform:uppercase;color:var(--faint);font-weight:400;margin:0 0 .6rem;
border-bottom:1px solid var(--line);padding-bottom:.4rem}
.fmap a{display:inline-block;color:var(--ink);text-decoration:none;margin:.3rem .35rem 0 0}
.fmap a:hover{text-decoration:underline}
.fmap div{color:var(--faint);line-height:1.5}
.fend{border-top:1px solid var(--line);padding-top:1.25rem;font-size:.82rem}
@media (max-width:640px){
.hero .wrap{padding-top:3.25rem;padding-bottom:2.75rem}
.btns{gap:.5rem}
.btn{padding:.75rem 1.15rem;font-size:.68rem}
.copy,.copy .btn{width:100%;justify-content:center}
.menu{left:0;right:0;min-width:0}
.strip .in{flex-direction:column;align-items:flex-start;gap:.6rem;padding:.9rem 1.75rem}
.strip b{border-right:0;padding-right:0}
.bar{width:100%}
}
</style>
<link rel="alternate" type="text/markdown" href="/debugging-performance-profiling.md">
<link rel="icon" href="/favicon.svg">
<link rel="canonical" href="https://hanzoskills.com/debugging-performance-profiling">
</head>
<!-- Reading this with a machine? Every page here is also plain markdown.
This one: https://hanzoskills.com/debugging-performance-profiling.md
Index: https://hanzoskills.com/llms.txt
Corpus: https://hanzoskills.com/all.md
Append .md to any path for the markdown twin. -->
<body>
<nav><div class="in">
<a class="brand" href="/"><svg class="mark" viewBox="0 0 67 67" fill="currentColor" aria-hidden="true"><path d="M22.21 67V44.6369H0V67H22.21Z"/><path d="M66.7038 22.3184H22.2534L0.0878906 44.6367H44.4634L66.7038 22.3184Z"/><path d="M22.21 0H0V22.3184H22.21V0Z"/><path d="M66.7198 0H44.5098V22.3184H66.7198V0Z"/><path d="M66.7198 67V44.6369H44.5098V67H66.7198Z"/></svg> HANZO<i>SKILLS</i></a>
<a class="up" href="/">All skills</a>
</div></nav>
<div class="wrap">
<h1>debugging-performance-profiling</h1><p class="lede">Performance profiling with CPU profilers (perf, pprof, py-spy), flame graphs, memory profiling (heaptrack, Valgrind), and profile-guided optimization</p><h1>Performance Profiling</h1>
<p><strong>Scope</strong>: CPU profiling (perf, Instruments, pprof, py-spy), flame graphs, sampling vs instrumentation, profile-guided optimization (PGO), memory profiling, I/O profiling, lock contention</p>
<p><strong>Lines</strong>: 450</p>
<p><strong>Last Updated</strong>: 2025-10-26</p>
<hr>
<h2>When to Use This Skill</h2>
<p>Use this skill when:</p>
<ul><li>Identifying performance bottlenecks in production or development</li><li>Optimizing CPU-bound operations (hot paths)</li><li>Debugging memory leaks or excessive allocations</li><li>Analyzing I/O performance (disk, network)</li><li>Investigating lock contention in concurrent programs</li><li>Preparing for profile-guided optimization (PGO)</li><li>Generating flame graphs for visual analysis</li><li>Comparing performance before/after optimizations</li></ul>
<p><strong>Don&#x27;t use</strong> for:</p>
<ul><li>Functional debugging (use debuggers)</li><li>Simple timing measurements (use time.time() or timeit)</li><li>Load testing (use k6, Locust, etc.)</li></ul>
<hr>
<h2>Core Concepts</h2>
<h3>Profiling Approaches</h3>
<p><strong>Sampling Profiling</strong> (recommended for production):</p>
<ul><li>Periodically sample call stack (e.g., 100 Hz)</li><li>Low overhead (~1-5%)</li><li>Statistical approximation</li><li>Tools: perf, py-spy, pprof (sampling mode), Instruments</li></ul>
<p><strong>Instrumentation Profiling</strong> (development only):</p>
<ul><li>Insert timing code at every function call</li><li>High overhead (10-100x slower)</li><li>Exact measurements</li><li>Tools: cProfile, gprof, Valgrind callgrind</li></ul>
<h3>Profiling Types</h3>
<pre><code>CPU Profiling: Which functions consume CPU time?
├─ Sampling: perf, py-spy, pprof (Go), Instruments (macOS)
└─ Instrumentation: cProfile (Python), gprof (C/C++)
Memory Profiling: Where is memory allocated?
├─ Heap profiling: heaptrack, Valgrind massif, memory_profiler
├─ Leak detection: Valgrind memcheck, AddressSanitizer
└─ Allocation tracing: jemalloc, tcmalloc
I/O Profiling: Where is time spent waiting?
├─ Disk I/O: iostat, iotop, strace
├─ Network I/O: tcpdump, Wireshark, netstat
└─ Syscall tracing: strace, dtrace, perf trace
Lock Contention: Where do threads wait for locks?
├─ perf lock
├─ Go pprof (mutex profile)
└─ Instruments (macOS Thread State trace)</code></pre>
<h3>Visualization Formats</h3>
<p>| Format | Description | Use Case | |--------|-------------|----------| | <strong>Flame graph</strong> | Stack samples visualized as horizontal bars | Identify hot paths, CPU bottlenecks | | <strong>Icicle graph</strong> | Inverted flame graph (root at top) | Alternative visualization | | <strong>Call graph</strong> | Directed graph of function calls | Understand call relationships | | <strong>Timeline</strong> | Time-based execution trace | Analyze concurrency, I/O waits |</p>
<hr>
<h2>Patterns</h2>
<h3>Pattern 1: CPU Profiling with perf (Linux)</h3>
<pre><code class="lang-bash"># 1. Record CPU profile (99 Hz sampling)
perf record -F 99 -g -p &lt;pid&gt; -- sleep 30
# -F 99: Sample at 99 Hz (avoid timer bias at 100 Hz)
# -g: Record call stacks
# -p &lt;pid&gt;: Attach to running process
# -- sleep 30: Profile for 30 seconds
# Alternative: Profile command from start
perf record -F 99 -g python app.py
# 2. View report (text)
perf report
# 3. Generate flamegraph
# Install: git clone https://github.com/brendangregg/FlameGraph
perf script | FlameGraph/stackcollapse-perf.pl | FlameGraph/flamegraph.pl &gt; flamegraph.svg
open flamegraph.svg
# 4. Profile all CPUs (system-wide)
sudo perf record -F 99 -g -a -- sleep 30
# 5. Profile specific events
perf list # List available events
perf record -e cache-misses -g -p &lt;pid&gt; -- sleep 30
# 6. Profile with call graph (dwarf)
perf record -F 99 -g --call-graph dwarf -p &lt;pid&gt; -- sleep 30
# Example workflow
# Terminal 1: Run application
python app.py
# Terminal 2: Profile for 30 seconds
PID=$(pgrep -f app.py)
perf record -F 99 -g -p $PID -- sleep 30
perf script | stackcollapse-perf.pl | flamegraph.pl &gt; cpu_profile.svg
# Analyze flamegraph
open cpu_profile.svg
# Look for:
# - Wide bars: CPU-intensive functions
# - Tower shape: Deep call stacks
# - Flat top: Time spent in function itself (not callees)</code></pre>
<h3>Pattern 2: CPU Profiling with py-spy (Python)</h3>
<pre><code class="lang-bash"># Install py-spy
pip install py-spy
# 1. Profile running Python process
py-spy record -o profile.svg --pid &lt;pid&gt;
# 2. Profile for specific duration
py-spy record -o profile.svg --pid &lt;pid&gt; --duration 60
# 3. Profile with higher sample rate
py-spy record -o profile.svg --pid &lt;pid&gt; --rate 200
# 4. Generate flamegraph
py-spy record -o flamegraph.svg --format flamegraph --pid &lt;pid&gt; --duration 30
# 5. Profile all threads
py-spy record -o profile.svg --pid &lt;pid&gt; --threads
# 6. Profile subprocesses
py-spy record -o profile.svg --pid &lt;pid&gt; --subprocesses
# 7. Top-like interface (live view)
py-spy top --pid &lt;pid&gt;
# Example: Profile FastAPI app
# Terminal 1: Start app
uvicorn app:app --host 0.0.0.0 --port 8000
# Terminal 2: Profile
PID=$(pgrep -f uvicorn)
py-spy record -o profile.svg --pid $PID --duration 60 --rate 200
# Terminal 3: Generate load
wrk -t4 -c100 -d60s http://localhost:8000/api/users
# Analyze profile.svg
open profile.svg</code></pre>
<h3>Pattern 3: CPU Profiling with pprof (Go)</h3>
<pre><code class="lang-go">package main
import (
&quot;log&quot;
&quot;net/http&quot;
_ &quot;net/http/pprof&quot;
&quot;os&quot;
&quot;runtime&quot;
&quot;runtime/pprof&quot;
)
func main() {
// Method 1: HTTP server (production)
go func() {
log.Println(http.ListenAndServe(&quot;localhost:6060&quot;, nil))
}()
// Method 2: File-based profiling (development)
f, err := os.Create(&quot;cpu.prof&quot;)
if err != nil {
log.Fatal(err)
}
defer f.Close()
if err := pprof.StartCPUProfile(f); err != nil {
log.Fatal(err)
}
defer pprof.StopCPUProfile()
// Your application code
runApp()
}
func runApp() {
// CPU-intensive work
for i := 0; i &lt; 1000000; i++ {
compute(i)
}
}
func compute(n int) int {
result := 0
for i := 0; i &lt; n; i++ {
result += i
}
return result
}</code></pre>
<p><strong>Analyze Go profiles</strong>:</p>
<pre><code class="lang-bash"># Method 1: HTTP endpoint (live profiling)
go tool pprof http://localhost:6060/debug/pprof/profile?seconds=30
# Interactive commands:
# (pprof) top10 # Top 10 functions
# (pprof) list compute # Show source for function
# (pprof) web # Open in browser (requires graphviz)
# (pprof) svg # Generate SVG
# Method 2: File-based profile
go tool pprof cpu.prof
# Generate flamegraph
go tool pprof -http=:8080 cpu.prof
# Open http://localhost:8080 in browser
# Generate flamegraph SVG
go tool pprof -output=flamegraph.svg -svg cpu.prof
# Compare two profiles (before/after optimization)
go tool pprof -base=cpu_before.prof cpu_after.prof</code></pre>
<h3>Pattern 4: Memory Profiling</h3>
<p><strong>Python (memory_profiler)</strong>:</p>
<pre><code class="lang-python"># Install: pip install memory_profiler
from memory_profiler import profile
@profile
def process_data():
# Large allocation
data = [i for i in range(1000000)]
# Memory leak (never freed)
cache = {}
for i in range(100000):
cache[i] = [0] * 1000
return data
if __name__ == &quot;__main__&quot;:
process_data()</code></pre>
<pre><code class="lang-bash"># Run with memory profiling
python -m memory_profiler app.py
# Output shows memory usage per line:
# Line # Mem usage Increment Line Contents
# ================================================
# 5 38.3 MiB 38.3 MiB @profile
# 6 def process_data():
# 8 45.9 MiB 7.6 MiB data = [i for i in range(1000000)]
# 11 809.4 MiB 763.5 MiB for i in range(100000):
# 12 809.4 MiB 0.0 MiB cache[i] = [0] * 1000</code></pre>
<p><strong>C/C++ (Valgrind Massif)</strong>:</p>
<pre><code class="lang-bash"># Run with heap profiling
valgrind --tool=massif ./myapp
# Generates massif.out.&lt;pid&gt;
# Analyze
ms_print massif.out.12345
# Generate graph
massif-visualizer massif.out.12345 # GUI tool</code></pre>
<p><strong>Go (pprof heap)</strong>:</p>
<pre><code class="lang-go">import (
&quot;net/http&quot;
_ &quot;net/http/pprof&quot;
&quot;runtime&quot;
)
func main() {
// Enable HTTP pprof server
go func() {
http.ListenAndServe(&quot;localhost:6060&quot;, nil)
}()
// Your app code
runApp()
}</code></pre>
<pre><code class="lang-bash"># Capture heap profile
go tool pprof http://localhost:6060/debug/pprof/heap
# Interactive commands
(pprof) top # Top allocations
(pprof) list main # Show allocations in main
# Generate graph
go tool pprof -http=:8080 http://localhost:6060/debug/pprof/heap</code></pre>
<h3>Pattern 5: Flame Graphs</h3>
<pre><code class="lang-bash"># Install FlameGraph tools
git clone https://github.com/brendangregg/FlameGraph
cd FlameGraph
# Generate from perf data (Linux)
perf record -F 99 -g -p &lt;pid&gt; -- sleep 30
perf script | ./stackcollapse-perf.pl | ./flamegraph.pl &gt; flame.svg
# Generate from DTrace (macOS)
sudo dtrace -x ustackframes=100 -n &#x27;profile-997 /pid == &lt;pid&gt;/ { @[ustack()] = count(); }&#x27; -o out.stacks
./stackcollapse.pl out.stacks | ./flamegraph.pl &gt; flame.svg
# Generate from pprof (Go)
go tool pprof -output=flame.svg -svg cpu.prof
# Generate from py-spy (Python)
py-spy record -o flame.svg --format flamegraph --pid &lt;pid&gt; --duration 30
# Reading flame graphs:
# - X-axis: Alphabetical order (not time!)
# - Y-axis: Stack depth
# - Width: Time spent in function
# - Color: Random (some tools use color for libraries)
# - Click to zoom
# - Look for wide bars (hot functions)</code></pre>
<h3>Pattern 6: Profile-Guided Optimization (PGO)</h3>
<p><strong>C/C++ (GCC/Clang)</strong>:</p>
<pre><code class="lang-bash"># Step 1: Compile with instrumentation
gcc -fprofile-generate -O2 myapp.c -o myapp
# Step 2: Run with representative workload
./myapp &lt; typical_input.txt
# Generates *.gcda files
# Step 3: Recompile with profile data
gcc -fprofile-use -O2 myapp.c -o myapp_optimized
# Compare performance
time ./myapp &lt; input.txt
time ./myapp_optimized &lt; input.txt</code></pre>
<p><strong>Go (PGO in Go 1.21+)</strong>:</p>
<pre><code class="lang-bash"># Step 1: Build binary
go build -o myapp
# Step 2: Run with CPU profiling
./myapp --cpuprofile=cpu.prof
# Step 3: Rebuild with profile
go build -pgo=cpu.prof -o myapp_optimized
# Compare
time ./myapp
time ./myapp_optimized</code></pre>
<p><strong>Rust (PGO)</strong>:</p>
<pre><code class="lang-bash"># Step 1: Instrument
RUSTFLAGS=&quot;-Cprofile-generate=/tmp/pgo-data&quot; cargo build --release
# Step 2: Run
./target/release/myapp
# Step 3: Merge profiles
llvm-profdata merge -o /tmp/pgo-data/merged.profdata /tmp/pgo-data
# Step 4: Optimize
RUSTFLAGS=&quot;-Cprofile-use=/tmp/pgo-data/merged.profdata&quot; cargo build --release</code></pre>
<h3>Pattern 7: I/O Profiling</h3>
<pre><code class="lang-bash"># 1. strace (syscall tracing)
strace -c python app.py # Summary
strace -T -e trace=read,write python app.py # Time per syscall
# 2. iostat (disk I/O)
iostat -x 1 # Extended stats, 1 second interval
# 3. iotop (per-process I/O)
sudo iotop -o # Only show processes doing I/O
# 4. lsof (open file descriptors)
lsof -p &lt;pid&gt; # Show open files for process
# 5. perf trace (lightweight strace)
sudo perf trace -p &lt;pid&gt;
# Example: Profile file I/O
# Terminal 1: Run app
python app.py
# Terminal 2: Trace I/O
PID=$(pgrep -f app.py)
strace -e trace=open,read,write,close -c -p $PID
# Ctrl+C after 30 seconds
# Shows count, time, errors per syscall</code></pre>
<h3>Pattern 8: Lock Contention Profiling</h3>
<p><strong>Go (mutex profiling)</strong>:</p>
<pre><code class="lang-go">import (
&quot;net/http&quot;
_ &quot;net/http/pprof&quot;
&quot;runtime&quot;
)
func main() {
// Enable mutex profiling
runtime.SetMutexProfileFraction(1)
// HTTP server for pprof
go func() {
http.ListenAndServe(&quot;localhost:6060&quot;, nil)
}()
runApp()
}</code></pre>
<pre><code class="lang-bash"># Capture mutex profile
go tool pprof http://localhost:6060/debug/pprof/mutex
# Interactive
(pprof) top
(pprof) list main
# Generate graph
go tool pprof -http=:8080 http://localhost:6060/debug/pprof/mutex</code></pre>
<p><strong>Linux (perf lock)</strong>:</p>
<pre><code class="lang-bash"># Record lock contention
sudo perf lock record -p &lt;pid&gt; -- sleep 30
# Report
sudo perf lock report</code></pre>
<hr>
<h2>Quick Reference</h2>
<h3>Profiling Tools by Language</h3>
<p>| Language | CPU Profiling | Memory Profiling | I/O Profiling | |----------|---------------|------------------|---------------| | <strong>Python</strong> | py-spy, cProfile | memory_profiler | strace | | <strong>Go</strong> | pprof (CPU) | pprof (heap) | perf trace | | <strong>Rust</strong> | perf, cargo-flamegraph | heaptrack, Valgrind | strace | | <strong>C/C++</strong> | perf, gprof | Valgrind massif | strace, ltrace | | <strong>Java</strong> | async-profiler | JProfiler, VisualVM | strace | | <strong>Node.js</strong> | 0x, clinic.js | heapdump | strace |</p>
<h3>perf Commands</h3>
<pre><code class="lang-bash"># Record CPU profile
perf record -F 99 -g -p &lt;pid&gt; -- sleep 30
# View report
perf report
# Record specific events
perf record -e cache-misses -g -p &lt;pid&gt;
# Record system-wide
sudo perf record -F 99 -g -a -- sleep 30
# Generate flamegraph
perf script | stackcollapse-perf.pl | flamegraph.pl &gt; flame.svg
# Record with call graph (dwarf)
perf record -F 99 -g --call-graph dwarf -p &lt;pid&gt;</code></pre>
<h3>py-spy Commands</h3>
<pre><code class="lang-bash"># Record flamegraph
py-spy record -o profile.svg --pid &lt;pid&gt; --duration 60
# Live top view
py-spy top --pid &lt;pid&gt;
# Profile all threads
py-spy record -o profile.svg --pid &lt;pid&gt; --threads
# Profile subprocesses
py-spy record -o profile.svg --pid &lt;pid&gt; --subprocesses</code></pre>
<h3>Go pprof Endpoints</h3>
<pre><code class="lang-bash"># CPU profile (30 seconds)
go tool pprof http://localhost:6060/debug/pprof/profile?seconds=30
# Heap profile
go tool pprof http://localhost:6060/debug/pprof/heap
# Goroutine profile
go tool pprof http://localhost:6060/debug/pprof/goroutine
# Mutex profile
go tool pprof http://localhost:6060/debug/pprof/mutex
# Block profile (blocking operations)
go tool pprof http://localhost:6060/debug/pprof/block</code></pre>
<hr>
<h2>Anti-Patterns</h2>
<h3>❌ Using Instrumentation Profiling in Production</h3>
<pre><code class="lang-python"># WRONG: cProfile slows down production by 10-100x
import cProfile
cProfile.run(&#x27;app.run()&#x27;) # NEVER in production!
# CORRECT: Use sampling profiler (py-spy)
# py-spy record -o profile.svg --pid &lt;pid&gt; --duration 30</code></pre>
<h3>❌ Profiling Debug Builds</h3>
<pre><code class="lang-bash"># WRONG: Profile unoptimized build
cargo build # Debug mode!
perf record ./target/debug/myapp
# CORRECT: Profile release build
cargo build --release
perf record ./target/release/myapp</code></pre>
<h3>❌ Profiling Without Representative Load</h3>
<pre><code class="lang-bash"># WRONG: Profile with no traffic
py-spy record -o profile.svg --pid &lt;pid&gt; --duration 60
# App is idle!
# CORRECT: Profile under load
# Terminal 1: Start app
uvicorn app:app
# Terminal 2: Generate load
wrk -t4 -c100 -d60s http://localhost:8000/api/users
# Terminal 3: Profile
py-spy record -o profile.svg --pid &lt;pid&gt; --duration 60</code></pre>
<h3>❌ Ignoring Compiler Optimizations</h3>
<pre><code class="lang-c">// WRONG: Volatile prevents optimization
volatile int result = 0;
for (int i = 0; i &lt; 1000000; i++) {
result += compute(i);
}
// CORRECT: Let compiler optimize
int result = 0;
for (int i = 0; i &lt; 1000000; i++) {
result += compute(i);
}
// Use result to prevent DCE
printf(&quot;%d\n&quot;, result);</code></pre>
<hr>
<h2>Related Skills</h2>
<ul><li><strong>debugging/production-debugging.md</strong> - Non-intrusive production debugging</li><li><strong>testing/performance-testing.md</strong> - Load testing to generate profiling workload</li><li><strong>observability/metrics-instrumentation.md</strong> - Runtime performance metrics</li><li><strong>build-systems/compiler-optimization.md</strong> - Compiler flags for performance</li></ul>
<hr>
<p><strong>Last Updated</strong>: 2025-10-26 <strong>Format Version</strong>: 1.0 (Atomic)</p>
<footer>
<div class="fmap">
<div><b>Build</b>
<a href="https://hanzo.app">hanzo.app</a> app builder
<a href="https://hanzo.codes">hanzo.codes</a> code with an agent
<a href="https://hanzoskills.com">hanzoskills.com</a> what an agent reads</div>
<div><b>Run</b>
<a href="https://console.hanzo.ai">console.hanzo.ai</a> cloud console
<a href="https://api.hanzo.ai">api.hanzo.ai</a> the API</div>
<div><b>Account</b>
<a href="https://hanzo.id">hanzo.id</a> sign in
<a href="https://pay.hanzo.ai">pay.hanzo.ai</a> top up
<a href="https://billing.hanzo.ai">billing.hanzo.ai</a> invoices and usage</div>
<div><b>Company</b>
<a href="https://hanzo.ai">hanzo.ai</a> canonical
<a href="https://hanzo.ventures">hanzo.ventures</a> ventures</div>
</div>
<div class="fend"><a href="/debugging-performance-profiling.md">This page as plain text</a> · <a href="/llms.txt">Index for agents</a></div>
</footer>
</div></body></html>