Files
skills/public/debugging-memory-leak-debugging.md

19 KiB

name, description
name description
debugging-memory-leak-debugging Comprehensive memory leak detection and debugging using heap profiling tools across multiple languages

Memory Leak Debugging

Scope: Heap profiling, memory leak detection patterns, and tools for Python, Go, Rust, C/C++, and .NET Lines: ~440 Last Updated: 2025-10-26 Format Version: 1.0 (Atomic)


When to Use This Skill

Activate this skill when:

  • Application memory usage grows unbounded over time
  • Out-of-memory errors occur in production or long-running processes
  • Need to profile heap allocations and identify leak sources
  • Investigating memory bloat in microservices or servers
  • Validating memory cleanup in resource-intensive applications
  • Debugging reference cycles or unreleased resources
  • Optimizing memory footprint for containerized applications
  • Analyzing memory patterns in Python, Go, Rust, C/C++, or .NET

Core Concepts

Concept 1: Memory Leak Types

Leak Categories:

  • Direct leaks: Allocated memory never freed (no references)
  • Indirect leaks: Memory reachable only through leaked blocks
  • Still-reachable: Memory with valid references but not freed at exit
  • Definitely lost: No pointers to allocated blocks
  • Possibly lost: Interior pointers only (offset from allocation start)

Common Causes:

  • Forgotten deallocations (missing free/delete)
  • Reference cycles in garbage-collected languages
  • Event listener leaks (unsubscribed callbacks)
  • Cache unbounded growth
  • Thread-local storage leaks
  • Static container growth
# ❌ Memory leak: cached data never evicted
cache = {}

def get_user(user_id):
    if user_id not in cache:
        cache[user_id] = fetch_user(user_id)  # Cache grows forever
    return cache[user_id]

# ✅ Fixed: use LRU cache with size limit
from functools import lru_cache

@lru_cache(maxsize=1000)
def get_user(user_id):
    return fetch_user(user_id)

Concept 2: Heap Profiling Workflow

Standard Process:

  1. Establish baseline: Measure initial memory usage
  2. Run workload: Execute representative operations
  3. Capture snapshot: Take heap dump at intervals
  4. Compare snapshots: Identify growing allocations
  5. Analyze backtraces: Find allocation call stacks
  6. Fix and verify: Patch leaks, re-profile to confirm

Sampling vs Tracking:

  • Sampling: Periodic snapshots, lower overhead (production-safe)
  • Tracking: Record every allocation, high overhead (dev/staging only)
# Production-safe sampling (low overhead)
py-spy record -o profile.svg --duration 60 python app.py

# Development tracking (high overhead)
valgrind --leak-check=full --track-origins=yes ./binary

Concept 3: Leak Detection Tools by Language

Tool Selection Matrix:

Language Primary Tool Secondary Tool Production-Safe
C/C++ Valgrind AddressSanitizer ASan only
Python tracemalloc memory_profiler tracemalloc yes
Go pprof heap heapster pprof yes
Rust MIRI valgrind No (dev only)
.NET dotnet-dump PerfView dotnet-dump yes
Java VisualVM Eclipse MAT Both yes

Key Characteristics:

  • Valgrind: Zero code changes, high overhead (10-30x slowdown)
  • ASan: Compile-time instrumentation, moderate overhead (2x slowdown)
  • tracemalloc: Python stdlib, snapshot comparison, minimal overhead
  • pprof: Go runtime profiling, live process inspection
  • MIRI: Rust interpreter, catches undefined behavior

Patterns

Pattern 1: Valgrind Leak Detection (C/C++)

When to use:

  • C/C++ applications with suspected memory leaks
  • Need detailed leak reports without code changes
  • Development/staging environments (overhead acceptable)
# Full leak check with origins tracking
valgrind --leak-check=full \
         --show-leak-kinds=all \
         --track-origins=yes \
         --log-file=valgrind.log \
         ./myapp arg1 arg2

# Analyze log file
grep "definitely lost" valgrind.log
grep "indirectly lost" valgrind.log

# Example output interpretation:
# ==12345== LEAK SUMMARY:
# ==12345==    definitely lost: 4,096 bytes in 1 blocks
# ==12345==    indirectly lost: 8,192 bytes in 2 blocks
# ==12345==      possibly lost: 0 bytes in 0 blocks
# ==12345==    still reachable: 512 bytes in 4 blocks

Key Options:

  • --leak-check=full: Show detailed leak backtraces
  • --track-origins=yes: Track uninitialized value origins
  • --show-leak-kinds=all: Show all leak types (not just definite)
  • --suppressions=file.supp: Suppress known library leaks

Benefits:

  • No recompilation required
  • Precise leak location reporting
  • Handles C libraries transparently

Pattern 2: AddressSanitizer (ASan) for C/C++

When to use:

  • Faster leak detection than Valgrind (2x vs 10-30x overhead)
  • Integration testing with moderate performance impact
  • Compile-time instrumentation acceptable
# Compile with ASan
g++ -fsanitize=address -g -O1 app.cpp -o app
# or
clang++ -fsanitize=address -g -O1 app.cpp -o app

# Run instrumented binary
ASAN_OPTIONS=detect_leaks=1:log_path=asan.log ./app

# Example ASan report:
# =================================================================
# ==12345==ERROR: LeakSanitizer: detected memory leaks
#
# Direct leak of 4096 byte(s) in 1 object(s) allocated from:
#     #0 0x7f1234567890 in malloc (/lib/x86_64-linux-gnu/libasan.so.5+0x...)
#     #1 0x5678901234ab in allocate_buffer app.cpp:42
#     #2 0x5678901234cd in main app.cpp:123

Configuration via ASAN_OPTIONS:

# Common ASan options
export ASAN_OPTIONS="detect_leaks=1:\
                     fast_unwind_on_malloc=0:\
                     malloc_context_size=30:\
                     log_path=asan.log"

Benefits:

  • 10-15x faster than Valgrind
  • Catches use-after-free, buffer overflows simultaneously
  • Integrates with CI/CD pipelines

Pattern 3: Python tracemalloc

When to use:

  • Python applications with growing memory usage
  • Need allocation snapshots without external tools
  • Production environments (low overhead)
import tracemalloc
import linecache

def display_top(snapshot, key_type='lineno', limit=10):
    """Display top memory allocations."""
    snapshot = snapshot.filter_traces((
        tracemalloc.Filter(False, "<frozen importlib._bootstrap>"),
        tracemalloc.Filter(False, "<unknown>"),
    ))
    top_stats = snapshot.statistics(key_type)

    print(f"Top {limit} lines:")
    for index, stat in enumerate(top_stats[:limit], 1):
        frame = stat.traceback[0]
        filename = frame.filename
        lineno = frame.lineno
        print(f"#{index}: {filename}:{lineno}: {stat.size / 1024:.1f} KiB")
        line = linecache.getline(filename, lineno).strip()
        if line:
            print(f"    {line}")

# Start tracing
tracemalloc.start()

# Take snapshot before operation
snapshot1 = tracemalloc.take_snapshot()

# Run workload
run_application()

# Take snapshot after operation
snapshot2 = tracemalloc.take_snapshot()

# Compare snapshots
top_stats = snapshot2.compare_to(snapshot1, 'lineno')

print("[ Top 10 differences ]")
for stat in top_stats[:10]:
    print(stat)

# Stop tracing
tracemalloc.stop()

Snapshot Comparison:

# Identify memory growth between snapshots
for stat in top_stats[:10]:
    print(f"{stat.size_diff / 1024:.1f} KiB | {stat.count_diff:+} allocations")
    print(f"  {stat.traceback.format()[0]}")

Benefits:

  • Built into Python stdlib (no dependencies)
  • Low overhead (suitable for production)
  • Precise line-level allocation tracking

Pattern 4: Python memory_profiler

When to use:

  • Line-by-line memory profiling needed
  • Debugging specific function memory usage
  • Development environments (higher overhead)
# Install: pip install memory_profiler

from memory_profiler import profile

@profile
def process_data(data):
    """Process large dataset."""
    result = []
    for item in data:
        # This line allocates heavily
        result.append(expensive_transform(item))
    return result

# Run with: python -m memory_profiler script.py
# Output:
# Line #    Mem usage    Increment  Occurrences   Line Contents
# ================================================================
#      5   38.816 MiB   38.816 MiB           1   @profile
#      6                                         def process_data(data):
#      7   38.816 MiB    0.000 MiB           1       result = []
#      8  156.234 MiB  117.418 MiB        1001       for item in data:
#      9  156.234 MiB    0.000 MiB        1000           result.append(expensive_transform(item))
#     10  156.234 MiB    0.000 MiB           1       return result

mprof Command-Line Tool:

# Record memory usage over time
mprof run python script.py

# Plot memory usage graph
mprof plot

# Show peak memory
mprof peak

Benefits:

  • Line-by-line granularity
  • Visual plots of memory over time
  • Easy decorator-based profiling

Pattern 5: Go pprof Heap Profiling

When to use:

  • Go services with memory growth issues
  • Live production profiling (minimal overhead)
  • Need allocation flame graphs
package main

import (
    "net/http"
    _ "net/http/pprof"  // Registers /debug/pprof/* handlers
    "runtime"
)

func main() {
    // Enable pprof HTTP endpoint
    go func() {
        http.ListenAndServe("localhost:6060", nil)
    }()

    // Your application code
    runServer()
}

// Programmatic heap profiling
func captureHeapProfile() {
    runtime.GC()  // Get up-to-date statistics
    f, _ := os.Create("heap.prof")
    defer f.Close()
    runtime.WriteHeapProfile(f)
}

Analyzing Heap Profiles:

# Capture heap profile from running service
curl http://localhost:6060/debug/pprof/heap > heap.prof

# Interactive analysis
go tool pprof heap.prof
# Commands in interactive mode:
# > top10        # Show top 10 memory consumers
# > list funcName  # Show source with allocations
# > web          # Open browser with call graph

# Generate flame graph
go tool pprof -http=:8080 heap.prof

# Compare two heap snapshots (find memory growth)
curl http://localhost:6060/debug/pprof/heap > heap1.prof
# ... wait and let app run ...
curl http://localhost:6060/debug/pprof/heap > heap2.prof
go tool pprof -base=heap1.prof heap2.prof

Key pprof Commands:

  • top: Show functions with highest memory usage
  • list <func>: Source code with allocation annotations
  • web: Visual call graph in browser
  • pdf: Export call graph as PDF

Benefits:

  • Production-safe (sampling, low overhead)
  • Live process inspection
  • Visual call graphs and flame graphs

Pattern 6: Rust MIRI for Memory Debugging

When to use:

  • Rust unsafe code memory validation
  • Development/testing environments only
  • Need to catch undefined behavior
# Install MIRI
rustup +nightly component add miri

# Run tests with MIRI
cargo +nightly miri test

# Run specific binary with MIRI
cargo +nightly miri run

# Example MIRI output for use-after-free:
# error: Undefined Behavior: pointer to alloc123 was dereferenced after this allocation got freed
#  --> src/main.rs:10:13
#   |
# 10|     println!("{}", *ptr);
#   |     ^^^^^^^^^^^^^^^^^ pointer to alloc123 was dereferenced after this allocation got freed

MIRI Capabilities:

  • Detects use-after-free
  • Detects invalid pointer arithmetic
  • Detects data races (limited)
  • Validates unsafe code invariants

Limitations:

  • Slow (interpreter, not profiler)
  • Dev/test only (cannot run in production)
  • Cannot analyze external C libraries

Pattern 7: .NET Memory Diagnostics

When to use:

  • .NET Core/5+ applications with memory issues
  • Production memory dumps analysis
  • Cross-platform .NET diagnostics
# Install dotnet-dump
dotnet tool install --global dotnet-dump

# Capture memory dump from running process
dotnet-dump collect --process-id 12345

# Analyze dump
dotnet-dump analyze core_20251026_123456

# Inside analysis session:
> dumpheap -stat                    # Heap statistics by type
> dumpheap -mt <MethodTable>        # Objects of specific type
> gcroot <address>                  # Find roots keeping object alive
> eeheap -gc                        # GC heap statistics
> objsize <address>                 # Total size including referenced objects

Example Analysis:

# Find objects consuming most memory
> dumpheap -stat
# Output shows types sorted by total size:
# MT            Count    TotalSize Class Name
# 00007f1234567890  50000  12800000 System.String
# 00007f1234567891  10000  10240000 MyApp.LargeObject

# Investigate specific type
> dumpheap -mt 00007f1234567891
# Shows all instances of MyApp.LargeObject

# Find what's keeping them alive
> gcroot 00007f9876543210
# Shows reference chain from GC root to object

Eclipse MAT (Memory Analyzer Tool):

  • Cross-platform Java/.NET heap dump analysis
  • Leak suspects report (automatic leak detection)
  • Dominator tree (memory ownership visualization)

Benefits:

  • Production dump analysis
  • No application restart needed
  • Cross-platform support

Pattern 8: Heaptrack for C/C++

When to use:

  • Linux C/C++ memory profiling
  • Need GUI visualization of allocations
  • Alternative to Valgrind with better performance
# Install heaptrack
sudo apt install heaptrack heaptrack-gui

# Record heap allocations
heaptrack ./myapp arg1 arg2
# Creates heaptrack.myapp.XXXXX.gz

# Analyze with GUI
heaptrack_gui heaptrack.myapp.12345.gz

# Command-line analysis
heaptrack_print heaptrack.myapp.12345.gz

Heaptrack Features:

  • Flame graph visualization
  • Timeline of allocations
  • Peak memory analysis
  • Call stack attribution
  • Leak detection

Benefits:

  • Lower overhead than Valgrind
  • Excellent GUI for visual analysis
  • Timeline view shows allocation patterns

Quick Reference

Tool Selection by Use Case

Use Case                    | Tool                      | Overhead  | Production-Safe
----------------------------|---------------------------|-----------|----------------
C/C++ leak detection        | Valgrind                  | 10-30x    | No
C/C++ faster detection      | AddressSanitizer          | 2x        | Maybe (staging)
C/C++ with GUI              | heaptrack                 | Low       | Yes
Python stdlib               | tracemalloc               | <5%       | Yes
Python line-by-line         | memory_profiler           | Medium    | No
Go live profiling           | pprof                     | <3%       | Yes
Rust unsafe validation      | MIRI                      | 100x+     | No (dev only)
.NET production dumps       | dotnet-dump               | N/A       | Yes
Java heap analysis          | Eclipse MAT, VisualVM     | N/A       | Yes (dumps)

Key Guidelines

✅ DO: Establish memory baseline before profiling
✅ DO: Use sampling profilers in production (pprof, tracemalloc)
✅ DO: Compare heap snapshots to identify growth
✅ DO: Run leak detectors in CI/CD (ASan, Valgrind in tests)
✅ DO: Set memory limits for caches and containers
✅ DO: Profile representative workloads (not synthetic tests)
✅ DO: Suppress known library leaks in Valgrind

❌ DON'T: Run Valgrind in production (too slow)
❌ DON'T: Ignore "still-reachable" leaks (may indicate issues)
❌ DON'T: Profile debug builds (optimizations affect allocations)
❌ DON'T: Assume GC languages never leak (reference cycles do leak)
❌ DON'T: Profile with insufficient workload (leaks may not appear)
❌ DON'T: Mix profiler data from different runs (inconsistent)

Anti-Patterns

Critical Violations

# ❌ NEVER: Unbounded cache growth
class UserService:
    def __init__(self):
        self.cache = {}  # Grows forever

    def get_user(self, user_id):
        if user_id not in self.cache:
            self.cache[user_id] = self.fetch_user(user_id)
        return self.cache[user_id]

# ✅ CORRECT: Use LRU cache with size limit
from functools import lru_cache

class UserService:
    @lru_cache(maxsize=1000)
    def get_user(self, user_id):
        return self.fetch_user(user_id)

❌ Unbounded caching: Cache grows indefinitely, causing OOM ✅ Correct approach: Use LRU cache with max size, TTL eviction, or weak references

Common Mistakes

// ❌ Don't: Forget to close resources in goroutines
func processFiles(files []string) {
    for _, file := range files {
        go func(f string) {
            data, _ := os.ReadFile(f)  // Leaks if goroutine never exits
            process(data)
        }(file)
    }
}

// ✅ Correct: Use context and cleanup
func processFiles(ctx context.Context, files []string) {
    var wg sync.WaitGroup
    for _, file := range files {
        wg.Add(1)
        go func(f string) {
            defer wg.Done()
            data, err := os.ReadFile(f)
            if err != nil {
                return
            }
            process(data)
        }(file)
    }
    wg.Wait()
}

❌ Unclosed resources in goroutines: File handles, connections leak ✅ Better: Use defer, WaitGroup, and context cancellation

// ❌ Don't: Raw pointers without ownership
class Service {
    std::vector<Connection*> connections;  // Who owns these?

    void add_connection(Connection* conn) {
        connections.push_back(conn);  // Leak risk
    }
};

// ✅ Correct: Use smart pointers with clear ownership
class Service {
    std::vector<std::unique_ptr<Connection>> connections;

    void add_connection(std::unique_ptr<Connection> conn) {
        connections.push_back(std::move(conn));  // Clear ownership
    }
};

❌ Raw pointer ownership unclear: Memory leaks from ambiguous ownership ✅ Better: Use unique_ptr/shared_ptr for automatic cleanup

# ❌ Don't: Create reference cycles without weak references
class Node:
    def __init__(self, value):
        self.value = value
        self.parent = None  # Strong reference
        self.children = []

    def add_child(self, child):
        child.parent = self  # Cycle: parent <-> child
        self.children.append(child)

# ✅ Correct: Use weak references to break cycles
import weakref

class Node:
    def __init__(self, value):
        self.value = value
        self.parent = None  # Will be weakref
        self.children = []

    def add_child(self, child):
        child.parent = weakref.ref(self)  # Weak reference
        self.children.append(child)

❌ Reference cycles: Python GC can't collect cyclic references efficiently ✅ Better: Use weakref to break cycles in parent-child relationships

# ❌ Don't: Run production with tracking profilers
valgrind ./production-server  # 10-30x slowdown!

# ✅ Correct: Use sampling profilers in production
# Go
curl http://localhost:6060/debug/pprof/heap > heap.prof

# Python
python -X tracemalloc=5 app.py  # Minimal overhead

❌ Tracking profilers in production: Valgrind, MIRI unusable in production ✅ Better: Use sampling profilers (pprof, tracemalloc, dotnet-dump)


  • testing/performance-testing.md - Memory profiling integrates with performance tests
  • observability/metrics-instrumentation.md - Export memory metrics to monitoring systems
  • containers/docker-optimization.md - Container memory limits and OOM debugging
  • plt/rust-memory-safety.md - Rust ownership model prevents many leaks
  • debugging/crash-debugging.md - OOM crashes require memory analysis
  • cicd/testing-strategy.md - Integrate ASan/Valgrind into CI pipelines

Last Updated: 2025-10-26 Format Version: 1.0 (Atomic)