19 KiB
name, description
| name | description |
|---|---|
| debugging-memory-leak-debugging | Comprehensive memory leak detection and debugging using heap profiling tools across multiple languages |
Memory Leak Debugging
Scope: Heap profiling, memory leak detection patterns, and tools for Python, Go, Rust, C/C++, and .NET Lines: ~440 Last Updated: 2025-10-26 Format Version: 1.0 (Atomic)
When to Use This Skill
Activate this skill when:
- Application memory usage grows unbounded over time
- Out-of-memory errors occur in production or long-running processes
- Need to profile heap allocations and identify leak sources
- Investigating memory bloat in microservices or servers
- Validating memory cleanup in resource-intensive applications
- Debugging reference cycles or unreleased resources
- Optimizing memory footprint for containerized applications
- Analyzing memory patterns in Python, Go, Rust, C/C++, or .NET
Core Concepts
Concept 1: Memory Leak Types
Leak Categories:
- Direct leaks: Allocated memory never freed (no references)
- Indirect leaks: Memory reachable only through leaked blocks
- Still-reachable: Memory with valid references but not freed at exit
- Definitely lost: No pointers to allocated blocks
- Possibly lost: Interior pointers only (offset from allocation start)
Common Causes:
- Forgotten deallocations (missing free/delete)
- Reference cycles in garbage-collected languages
- Event listener leaks (unsubscribed callbacks)
- Cache unbounded growth
- Thread-local storage leaks
- Static container growth
# ❌ Memory leak: cached data never evicted
cache = {}
def get_user(user_id):
if user_id not in cache:
cache[user_id] = fetch_user(user_id) # Cache grows forever
return cache[user_id]
# ✅ Fixed: use LRU cache with size limit
from functools import lru_cache
@lru_cache(maxsize=1000)
def get_user(user_id):
return fetch_user(user_id)
Concept 2: Heap Profiling Workflow
Standard Process:
- Establish baseline: Measure initial memory usage
- Run workload: Execute representative operations
- Capture snapshot: Take heap dump at intervals
- Compare snapshots: Identify growing allocations
- Analyze backtraces: Find allocation call stacks
- Fix and verify: Patch leaks, re-profile to confirm
Sampling vs Tracking:
- Sampling: Periodic snapshots, lower overhead (production-safe)
- Tracking: Record every allocation, high overhead (dev/staging only)
# Production-safe sampling (low overhead)
py-spy record -o profile.svg --duration 60 python app.py
# Development tracking (high overhead)
valgrind --leak-check=full --track-origins=yes ./binary
Concept 3: Leak Detection Tools by Language
Tool Selection Matrix:
| Language | Primary Tool | Secondary Tool | Production-Safe |
|---|---|---|---|
| C/C++ | Valgrind | AddressSanitizer | ASan only |
| Python | tracemalloc | memory_profiler | tracemalloc yes |
| Go | pprof heap | heapster | pprof yes |
| Rust | MIRI | valgrind | No (dev only) |
| .NET | dotnet-dump | PerfView | dotnet-dump yes |
| Java | VisualVM | Eclipse MAT | Both yes |
Key Characteristics:
- Valgrind: Zero code changes, high overhead (10-30x slowdown)
- ASan: Compile-time instrumentation, moderate overhead (2x slowdown)
- tracemalloc: Python stdlib, snapshot comparison, minimal overhead
- pprof: Go runtime profiling, live process inspection
- MIRI: Rust interpreter, catches undefined behavior
Patterns
Pattern 1: Valgrind Leak Detection (C/C++)
When to use:
- C/C++ applications with suspected memory leaks
- Need detailed leak reports without code changes
- Development/staging environments (overhead acceptable)
# Full leak check with origins tracking
valgrind --leak-check=full \
--show-leak-kinds=all \
--track-origins=yes \
--log-file=valgrind.log \
./myapp arg1 arg2
# Analyze log file
grep "definitely lost" valgrind.log
grep "indirectly lost" valgrind.log
# Example output interpretation:
# ==12345== LEAK SUMMARY:
# ==12345== definitely lost: 4,096 bytes in 1 blocks
# ==12345== indirectly lost: 8,192 bytes in 2 blocks
# ==12345== possibly lost: 0 bytes in 0 blocks
# ==12345== still reachable: 512 bytes in 4 blocks
Key Options:
--leak-check=full: Show detailed leak backtraces--track-origins=yes: Track uninitialized value origins--show-leak-kinds=all: Show all leak types (not just definite)--suppressions=file.supp: Suppress known library leaks
Benefits:
- No recompilation required
- Precise leak location reporting
- Handles C libraries transparently
Pattern 2: AddressSanitizer (ASan) for C/C++
When to use:
- Faster leak detection than Valgrind (2x vs 10-30x overhead)
- Integration testing with moderate performance impact
- Compile-time instrumentation acceptable
# Compile with ASan
g++ -fsanitize=address -g -O1 app.cpp -o app
# or
clang++ -fsanitize=address -g -O1 app.cpp -o app
# Run instrumented binary
ASAN_OPTIONS=detect_leaks=1:log_path=asan.log ./app
# Example ASan report:
# =================================================================
# ==12345==ERROR: LeakSanitizer: detected memory leaks
#
# Direct leak of 4096 byte(s) in 1 object(s) allocated from:
# #0 0x7f1234567890 in malloc (/lib/x86_64-linux-gnu/libasan.so.5+0x...)
# #1 0x5678901234ab in allocate_buffer app.cpp:42
# #2 0x5678901234cd in main app.cpp:123
Configuration via ASAN_OPTIONS:
# Common ASan options
export ASAN_OPTIONS="detect_leaks=1:\
fast_unwind_on_malloc=0:\
malloc_context_size=30:\
log_path=asan.log"
Benefits:
- 10-15x faster than Valgrind
- Catches use-after-free, buffer overflows simultaneously
- Integrates with CI/CD pipelines
Pattern 3: Python tracemalloc
When to use:
- Python applications with growing memory usage
- Need allocation snapshots without external tools
- Production environments (low overhead)
import tracemalloc
import linecache
def display_top(snapshot, key_type='lineno', limit=10):
"""Display top memory allocations."""
snapshot = snapshot.filter_traces((
tracemalloc.Filter(False, "<frozen importlib._bootstrap>"),
tracemalloc.Filter(False, "<unknown>"),
))
top_stats = snapshot.statistics(key_type)
print(f"Top {limit} lines:")
for index, stat in enumerate(top_stats[:limit], 1):
frame = stat.traceback[0]
filename = frame.filename
lineno = frame.lineno
print(f"#{index}: {filename}:{lineno}: {stat.size / 1024:.1f} KiB")
line = linecache.getline(filename, lineno).strip()
if line:
print(f" {line}")
# Start tracing
tracemalloc.start()
# Take snapshot before operation
snapshot1 = tracemalloc.take_snapshot()
# Run workload
run_application()
# Take snapshot after operation
snapshot2 = tracemalloc.take_snapshot()
# Compare snapshots
top_stats = snapshot2.compare_to(snapshot1, 'lineno')
print("[ Top 10 differences ]")
for stat in top_stats[:10]:
print(stat)
# Stop tracing
tracemalloc.stop()
Snapshot Comparison:
# Identify memory growth between snapshots
for stat in top_stats[:10]:
print(f"{stat.size_diff / 1024:.1f} KiB | {stat.count_diff:+} allocations")
print(f" {stat.traceback.format()[0]}")
Benefits:
- Built into Python stdlib (no dependencies)
- Low overhead (suitable for production)
- Precise line-level allocation tracking
Pattern 4: Python memory_profiler
When to use:
- Line-by-line memory profiling needed
- Debugging specific function memory usage
- Development environments (higher overhead)
# Install: pip install memory_profiler
from memory_profiler import profile
@profile
def process_data(data):
"""Process large dataset."""
result = []
for item in data:
# This line allocates heavily
result.append(expensive_transform(item))
return result
# Run with: python -m memory_profiler script.py
# Output:
# Line # Mem usage Increment Occurrences Line Contents
# ================================================================
# 5 38.816 MiB 38.816 MiB 1 @profile
# 6 def process_data(data):
# 7 38.816 MiB 0.000 MiB 1 result = []
# 8 156.234 MiB 117.418 MiB 1001 for item in data:
# 9 156.234 MiB 0.000 MiB 1000 result.append(expensive_transform(item))
# 10 156.234 MiB 0.000 MiB 1 return result
mprof Command-Line Tool:
# Record memory usage over time
mprof run python script.py
# Plot memory usage graph
mprof plot
# Show peak memory
mprof peak
Benefits:
- Line-by-line granularity
- Visual plots of memory over time
- Easy decorator-based profiling
Pattern 5: Go pprof Heap Profiling
When to use:
- Go services with memory growth issues
- Live production profiling (minimal overhead)
- Need allocation flame graphs
package main
import (
"net/http"
_ "net/http/pprof" // Registers /debug/pprof/* handlers
"runtime"
)
func main() {
// Enable pprof HTTP endpoint
go func() {
http.ListenAndServe("localhost:6060", nil)
}()
// Your application code
runServer()
}
// Programmatic heap profiling
func captureHeapProfile() {
runtime.GC() // Get up-to-date statistics
f, _ := os.Create("heap.prof")
defer f.Close()
runtime.WriteHeapProfile(f)
}
Analyzing Heap Profiles:
# Capture heap profile from running service
curl http://localhost:6060/debug/pprof/heap > heap.prof
# Interactive analysis
go tool pprof heap.prof
# Commands in interactive mode:
# > top10 # Show top 10 memory consumers
# > list funcName # Show source with allocations
# > web # Open browser with call graph
# Generate flame graph
go tool pprof -http=:8080 heap.prof
# Compare two heap snapshots (find memory growth)
curl http://localhost:6060/debug/pprof/heap > heap1.prof
# ... wait and let app run ...
curl http://localhost:6060/debug/pprof/heap > heap2.prof
go tool pprof -base=heap1.prof heap2.prof
Key pprof Commands:
top: Show functions with highest memory usagelist <func>: Source code with allocation annotationsweb: Visual call graph in browserpdf: Export call graph as PDF
Benefits:
- Production-safe (sampling, low overhead)
- Live process inspection
- Visual call graphs and flame graphs
Pattern 6: Rust MIRI for Memory Debugging
When to use:
- Rust unsafe code memory validation
- Development/testing environments only
- Need to catch undefined behavior
# Install MIRI
rustup +nightly component add miri
# Run tests with MIRI
cargo +nightly miri test
# Run specific binary with MIRI
cargo +nightly miri run
# Example MIRI output for use-after-free:
# error: Undefined Behavior: pointer to alloc123 was dereferenced after this allocation got freed
# --> src/main.rs:10:13
# |
# 10| println!("{}", *ptr);
# | ^^^^^^^^^^^^^^^^^ pointer to alloc123 was dereferenced after this allocation got freed
MIRI Capabilities:
- Detects use-after-free
- Detects invalid pointer arithmetic
- Detects data races (limited)
- Validates unsafe code invariants
Limitations:
- Slow (interpreter, not profiler)
- Dev/test only (cannot run in production)
- Cannot analyze external C libraries
Pattern 7: .NET Memory Diagnostics
When to use:
- .NET Core/5+ applications with memory issues
- Production memory dumps analysis
- Cross-platform .NET diagnostics
# Install dotnet-dump
dotnet tool install --global dotnet-dump
# Capture memory dump from running process
dotnet-dump collect --process-id 12345
# Analyze dump
dotnet-dump analyze core_20251026_123456
# Inside analysis session:
> dumpheap -stat # Heap statistics by type
> dumpheap -mt <MethodTable> # Objects of specific type
> gcroot <address> # Find roots keeping object alive
> eeheap -gc # GC heap statistics
> objsize <address> # Total size including referenced objects
Example Analysis:
# Find objects consuming most memory
> dumpheap -stat
# Output shows types sorted by total size:
# MT Count TotalSize Class Name
# 00007f1234567890 50000 12800000 System.String
# 00007f1234567891 10000 10240000 MyApp.LargeObject
# Investigate specific type
> dumpheap -mt 00007f1234567891
# Shows all instances of MyApp.LargeObject
# Find what's keeping them alive
> gcroot 00007f9876543210
# Shows reference chain from GC root to object
Eclipse MAT (Memory Analyzer Tool):
- Cross-platform Java/.NET heap dump analysis
- Leak suspects report (automatic leak detection)
- Dominator tree (memory ownership visualization)
Benefits:
- Production dump analysis
- No application restart needed
- Cross-platform support
Pattern 8: Heaptrack for C/C++
When to use:
- Linux C/C++ memory profiling
- Need GUI visualization of allocations
- Alternative to Valgrind with better performance
# Install heaptrack
sudo apt install heaptrack heaptrack-gui
# Record heap allocations
heaptrack ./myapp arg1 arg2
# Creates heaptrack.myapp.XXXXX.gz
# Analyze with GUI
heaptrack_gui heaptrack.myapp.12345.gz
# Command-line analysis
heaptrack_print heaptrack.myapp.12345.gz
Heaptrack Features:
- Flame graph visualization
- Timeline of allocations
- Peak memory analysis
- Call stack attribution
- Leak detection
Benefits:
- Lower overhead than Valgrind
- Excellent GUI for visual analysis
- Timeline view shows allocation patterns
Quick Reference
Tool Selection by Use Case
Use Case | Tool | Overhead | Production-Safe
----------------------------|---------------------------|-----------|----------------
C/C++ leak detection | Valgrind | 10-30x | No
C/C++ faster detection | AddressSanitizer | 2x | Maybe (staging)
C/C++ with GUI | heaptrack | Low | Yes
Python stdlib | tracemalloc | <5% | Yes
Python line-by-line | memory_profiler | Medium | No
Go live profiling | pprof | <3% | Yes
Rust unsafe validation | MIRI | 100x+ | No (dev only)
.NET production dumps | dotnet-dump | N/A | Yes
Java heap analysis | Eclipse MAT, VisualVM | N/A | Yes (dumps)
Key Guidelines
✅ DO: Establish memory baseline before profiling
✅ DO: Use sampling profilers in production (pprof, tracemalloc)
✅ DO: Compare heap snapshots to identify growth
✅ DO: Run leak detectors in CI/CD (ASan, Valgrind in tests)
✅ DO: Set memory limits for caches and containers
✅ DO: Profile representative workloads (not synthetic tests)
✅ DO: Suppress known library leaks in Valgrind
❌ DON'T: Run Valgrind in production (too slow)
❌ DON'T: Ignore "still-reachable" leaks (may indicate issues)
❌ DON'T: Profile debug builds (optimizations affect allocations)
❌ DON'T: Assume GC languages never leak (reference cycles do leak)
❌ DON'T: Profile with insufficient workload (leaks may not appear)
❌ DON'T: Mix profiler data from different runs (inconsistent)
Anti-Patterns
Critical Violations
# ❌ NEVER: Unbounded cache growth
class UserService:
def __init__(self):
self.cache = {} # Grows forever
def get_user(self, user_id):
if user_id not in self.cache:
self.cache[user_id] = self.fetch_user(user_id)
return self.cache[user_id]
# ✅ CORRECT: Use LRU cache with size limit
from functools import lru_cache
class UserService:
@lru_cache(maxsize=1000)
def get_user(self, user_id):
return self.fetch_user(user_id)
❌ Unbounded caching: Cache grows indefinitely, causing OOM ✅ Correct approach: Use LRU cache with max size, TTL eviction, or weak references
Common Mistakes
// ❌ Don't: Forget to close resources in goroutines
func processFiles(files []string) {
for _, file := range files {
go func(f string) {
data, _ := os.ReadFile(f) // Leaks if goroutine never exits
process(data)
}(file)
}
}
// ✅ Correct: Use context and cleanup
func processFiles(ctx context.Context, files []string) {
var wg sync.WaitGroup
for _, file := range files {
wg.Add(1)
go func(f string) {
defer wg.Done()
data, err := os.ReadFile(f)
if err != nil {
return
}
process(data)
}(file)
}
wg.Wait()
}
❌ Unclosed resources in goroutines: File handles, connections leak ✅ Better: Use defer, WaitGroup, and context cancellation
// ❌ Don't: Raw pointers without ownership
class Service {
std::vector<Connection*> connections; // Who owns these?
void add_connection(Connection* conn) {
connections.push_back(conn); // Leak risk
}
};
// ✅ Correct: Use smart pointers with clear ownership
class Service {
std::vector<std::unique_ptr<Connection>> connections;
void add_connection(std::unique_ptr<Connection> conn) {
connections.push_back(std::move(conn)); // Clear ownership
}
};
❌ Raw pointer ownership unclear: Memory leaks from ambiguous ownership ✅ Better: Use unique_ptr/shared_ptr for automatic cleanup
# ❌ Don't: Create reference cycles without weak references
class Node:
def __init__(self, value):
self.value = value
self.parent = None # Strong reference
self.children = []
def add_child(self, child):
child.parent = self # Cycle: parent <-> child
self.children.append(child)
# ✅ Correct: Use weak references to break cycles
import weakref
class Node:
def __init__(self, value):
self.value = value
self.parent = None # Will be weakref
self.children = []
def add_child(self, child):
child.parent = weakref.ref(self) # Weak reference
self.children.append(child)
❌ Reference cycles: Python GC can't collect cyclic references efficiently ✅ Better: Use weakref to break cycles in parent-child relationships
# ❌ Don't: Run production with tracking profilers
valgrind ./production-server # 10-30x slowdown!
# ✅ Correct: Use sampling profilers in production
# Go
curl http://localhost:6060/debug/pprof/heap > heap.prof
# Python
python -X tracemalloc=5 app.py # Minimal overhead
❌ Tracking profilers in production: Valgrind, MIRI unusable in production ✅ Better: Use sampling profilers (pprof, tracemalloc, dotnet-dump)
Related Skills
testing/performance-testing.md- Memory profiling integrates with performance testsobservability/metrics-instrumentation.md- Export memory metrics to monitoring systemscontainers/docker-optimization.md- Container memory limits and OOM debuggingplt/rust-memory-safety.md- Rust ownership model prevents many leaksdebugging/crash-debugging.md- OOM crashes require memory analysiscicd/testing-strategy.md- Integrate ASan/Valgrind into CI pipelines
Last Updated: 2025-10-26 Format Version: 1.0 (Atomic)