20 KiB
name, description
| name | description |
|---|---|
| debugging-crash-debugging | Signal handling, crash reproduction, fuzzing for crash discovery, telemetry aggregation, and post-mortem workflows |
Crash Debugging
Scope: Signal handling (SIGSEGV, SIGABRT), backtrace generation, crash reproduction, fuzzing (AFL, libFuzzer), crash telemetry Lines: ~410 Last Updated: 2025-10-26 Format Version: 1.0 (Atomic)
When to Use This Skill
Activate this skill when:
- Application crashes unexpectedly (SIGSEGV, SIGABRT, SIGFPE)
- Need to reproduce crashes from bug reports
- Setting up crash telemetry and aggregation
- Implementing signal handlers for graceful crash reporting
- Using fuzzing to discover crash-inducing inputs
- Debugging null pointer dereferences or memory corruption
- Investigating stack overflow or assertion failures
- Building post-mortem debugging workflows
Core Concepts
Concept 1: Common Crash Signals
Signal Types and Causes:
| Signal | Name | Common Causes |
|---|---|---|
| SIGSEGV | Segmentation Fault | Null pointer, invalid memory access, buffer overflow |
| SIGABRT | Abort | assert() failure, std::abort(), memory corruption detected |
| SIGFPE | Floating Point Exception | Division by zero, integer overflow (signed) |
| SIGILL | Illegal Instruction | Corrupted code, wrong architecture binary |
| SIGBUS | Bus Error | Misaligned memory access, mmap failure |
| SIGSTKFLT | Stack Fault | Stack overflow (rare on modern systems) |
Default Signal Behavior:
- Terminate process
- Generate core dump (if enabled)
- No cleanup or logging
#include <signal.h>
#include <stdio.h>
#include <stdlib.h>
void signal_handler(int sig) {
fprintf(stderr, "Caught signal %d (%s)\n", sig, strsignal(sig));
// Log crash, generate backtrace, cleanup resources
exit(1);
}
int main() {
// Install signal handlers
signal(SIGSEGV, signal_handler);
signal(SIGABRT, signal_handler);
signal(SIGFPE, signal_handler);
// Application code
return 0;
}
Concept 2: Stack Unwinding and Backtraces
Backtrace Generation (capturing call stack):
// Linux: backtrace() from execinfo.h
#include <execinfo.h>
#include <stdio.h>
#include <stdlib.h>
#include <signal.h>
void print_backtrace() {
void *array[20];
size_t size = backtrace(array, 20);
char **strings = backtrace_symbols(array, size);
fprintf(stderr, "Backtrace (%zd frames):\n", size);
for (size_t i = 0; i < size; i++) {
fprintf(stderr, " %s\n", strings[i]);
}
free(strings);
}
void crash_handler(int sig) {
fprintf(stderr, "ERROR: Signal %d (%s)\n", sig, strsignal(sig));
print_backtrace();
exit(1);
}
int main() {
signal(SIGSEGV, crash_handler);
signal(SIGABRT, crash_handler);
// Trigger crash for testing
int *ptr = NULL;
*ptr = 42; // SIGSEGV
}
C++ Stack Unwinding (std::stacktrace in C++23):
#include <stacktrace>
#include <iostream>
void crash_handler(int sig) {
std::cerr << "Caught signal " << sig << "\n";
std::cerr << std::stacktrace::current() << "\n";
std::exit(1);
}
Benefits:
- Immediate crash context (call stack)
- Helps identify root cause quickly
- Works without debugger
Concept 3: Crash Reproduction Strategies
Reproduction Categories:
-
Deterministic crashes: Same input always crashes
- Example: Null pointer dereference on specific API call
- Strategy: Capture input, replay in test
-
Non-deterministic crashes: Timing or state-dependent
- Example: Race condition, uninitialized memory
- Strategy: Stress testing, record-replay, sanitizers
-
Environmental crashes: Specific to environment
- Example: Out-of-memory, file descriptor exhaustion
- Strategy: Reproduce constraints (memory limits, resource limits)
Minimal Reproduction:
# ❌ Bug report: "App crashes when processing data"
# Not reproducible (what data? what operation?)
# ✅ Minimal reproduction
import app
# Trigger crash with minimal input
data = {"user_id": None} # Null user_id causes crash
app.process(data) # Crashes on data["user_id"].lower()
# Root cause: No null check before string method
# Fix: Add null check or use data.get("user_id", "")
Patterns
Pattern 1: Comprehensive Signal Handler (Linux)
When to use:
- Production applications requiring crash reporting
- Need backtraces and context before process termination
- Resource cleanup before crash
#include <signal.h>
#include <execinfo.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <time.h>
static void write_crash_log(int sig, void *addr) {
char filename[256];
time_t now = time(NULL);
snprintf(filename, sizeof(filename), "/var/log/crash_%ld.log", (long)now);
FILE *f = fopen(filename, "w");
if (!f) {
f = stderr;
}
fprintf(f, "=== CRASH REPORT ===\n");
fprintf(f, "Signal: %d (%s)\n", sig, strsignal(sig));
fprintf(f, "Fault address: %p\n", addr);
fprintf(f, "PID: %d\n", getpid());
fprintf(f, "Timestamp: %ld\n", now);
// Backtrace
void *array[50];
size_t size = backtrace(array, 50);
fprintf(f, "\nBacktrace (%zd frames):\n", size);
backtrace_symbols_fd(array, size, fileno(f));
if (f != stderr) {
fclose(f);
}
}
void crash_handler(int sig, siginfo_t *info, void *context) {
// Write crash log
write_crash_log(sig, info->si_addr);
// Cleanup resources (be careful: limited operations safe in signal handler)
// - Close log files
// - Flush buffers
// - Release critical locks (if safe)
// Re-raise signal with default handler to generate core dump
signal(sig, SIG_DFL);
raise(sig);
}
void setup_crash_handlers() {
struct sigaction sa;
sa.sa_flags = SA_SIGINFO;
sa.sa_sigaction = crash_handler;
sigemptyset(&sa.sa_mask);
sigaction(SIGSEGV, &sa, NULL);
sigaction(SIGABRT, &sa, NULL);
sigaction(SIGFPE, &sa, NULL);
sigaction(SIGILL, &sa, NULL);
sigaction(SIGBUS, &sa, NULL);
}
int main() {
setup_crash_handlers();
// Application code
}
Signal Handler Safety:
- Only async-signal-safe functions allowed (write, _exit, etc.)
- Avoid: malloc, printf, locks (may deadlock)
- Keep handler minimal (log and exit)
Benefits:
- Captures crash context automatically
- Logs to persistent storage
- Generates core dump after logging
Pattern 2: C++ Exception to Signal Translation
When to use:
- C++ applications with both exceptions and crashes
- Need uniform crash handling for exceptions and signals
- Catching unhandled exceptions
#include <exception>
#include <signal.h>
#include <execinfo.h>
#include <iostream>
void signal_handler(int sig) {
std::cerr << "Signal " << sig << " caught\n";
void *array[20];
size_t size = backtrace(array, 20);
backtrace_symbols_fd(array, size, STDERR_FILENO);
std::abort(); // Generate core dump
}
void terminate_handler() {
std::cerr << "Unhandled exception caught\n";
// Rethrow to get exception details
try {
std::exception_ptr eptr = std::current_exception();
if (eptr) {
std::rethrow_exception(eptr);
}
} catch (const std::exception &e) {
std::cerr << "Exception: " << e.what() << "\n";
} catch (...) {
std::cerr << "Unknown exception\n";
}
void *array[20];
size_t size = backtrace(array, 20);
backtrace_symbols_fd(array, size, STDERR_FILENO);
std::abort();
}
int main() {
// Install handlers
std::set_terminate(terminate_handler);
signal(SIGSEGV, signal_handler);
signal(SIGABRT, signal_handler);
// Application code
}
Benefits:
- Unified crash handling (exceptions + signals)
- Exception details in crash logs
- Backtrace for unhandled exceptions
Pattern 3: Fuzzing with AFL (American Fuzzy Lop)
When to use:
- Discovering crash-inducing inputs
- Security testing (buffer overflows, format string bugs)
- Regression testing (ensure no new crashes)
# Install AFL++
git clone https://github.com/AFLplusplus/AFLplusplus
cd AFLplusplus
make
sudo make install
# Compile target with AFL instrumentation
afl-gcc -g -O0 -fsanitize=address target.c -o target
# Or with afl-clang-fast for better performance:
afl-clang-fast -g -O0 -fsanitize=address target.c -o target
# Prepare input corpus (seed inputs)
mkdir input_corpus
echo "test input" > input_corpus/seed1.txt
# Run fuzzer
afl-fuzz -i input_corpus -o output -- ./target @@
# @@ is replaced with path to generated input file
# AFL output:
# - output/crashes/ → Inputs that caused crashes
# - output/hangs/ → Inputs that caused timeouts
# - output/queue/ → Interesting inputs (code coverage)
# Reproduce crash
./target output/crashes/id:000000,sig:11,src:000000,op:flip1,pos:0
Example Fuzz Target:
// target.c - vulnerable program
#include <stdio.h>
#include <string.h>
int main(int argc, char **argv) {
if (argc != 2) {
return 1;
}
FILE *f = fopen(argv[1], "r");
if (!f) {
return 1;
}
char buffer[16];
// ❌ Vulnerable: no bounds check
fscanf(f, "%s", buffer); // Buffer overflow if input > 15 chars
printf("Read: %s\n", buffer);
fclose(f);
return 0;
}
Benefits:
- Automatic crash discovery
- Code coverage-guided fuzzing (explores code paths)
- Finds edge cases humans miss
Pattern 4: Fuzzing with libFuzzer (LLVM)
When to use:
- In-process fuzzing (faster than AFL)
- C/C++ library fuzzing
- Integration with sanitizers (ASan, UBSan)
// fuzz_target.cpp
#include <stdint.h>
#include <stddef.h>
// Fuzz entry point (called by libFuzzer)
extern "C" int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size) {
if (size < 4) {
return 0; // Ignore small inputs
}
// Call function under test
process_data(data, size);
return 0; // Non-zero = error (stop fuzzing)
}
Compile and Run:
# Compile with libFuzzer and ASan
clang++ -g -O1 -fsanitize=fuzzer,address fuzz_target.cpp -o fuzz_target
# Run fuzzer
./fuzz_target
# LibFuzzer output:
# #1 INITED cov: 42 ft: 56 corp: 1/1b exec/s: 0 rss: 30Mb
# #2 NEW cov: 45 ft: 59 corp: 2/2b exec/s: 0 rss: 30Mb
# ...
# ==12345==ERROR: AddressSanitizer: heap-buffer-overflow
# (Crash found, corpus saved)
# Reproduce crash
./fuzz_target crash-<hash>
Benefits:
- Fast (in-process, no exec overhead)
- Excellent sanitizer integration
- Built into LLVM/Clang
Limitations:
- Requires source code modifications
- In-process only (no stdin fuzzing)
Pattern 5: Crash Telemetry Aggregation
When to use:
- Production applications with many users
- Need crash statistics and trends
- Prioritizing bug fixes by crash frequency
# Using Sentry for crash aggregation
import sentry_sdk
sentry_sdk.init(
dsn="https://[email protected]/0",
environment="production",
release="[email protected]",
)
# Automatic crash reporting
def main():
# Crashes automatically reported with:
# - Stack trace
# - Environment (OS, Python version)
# - Release version
# - User context (if set)
risky_operation() # If this crashes, Sentry captures it
# Sentry dashboard shows:
# - Crash count by version
# - Top crashing functions
# - User impact (unique users affected)
# - Trends (new vs. recurring crashes)
Crash Grouping:
- Sentry groups crashes by stack trace similarity
- Same bug from multiple users → one issue
- New crashes flagged as regressions
Benefits:
- Prioritize fixes by user impact
- Track crash-free user rate
- Identify platform-specific crashes
Pattern 6: Deterministic Crash Reproduction
When to use:
- Bug reports from production crashes
- Need reliable crash reproduction for debugging
- Writing regression tests
# Crash report from production:
# Stack trace shows: process_message() → parse_json() → crash
# Step 1: Extract minimal failing input
def test_crash_reproduction():
# From logs: "Processing message: {id:123,name:null}"
message = '{"id": 123, "name": null}'
# Reproduce crash
with pytest.raises(AttributeError):
process_message(message) # Crashes on name.lower()
# Step 2: Fix bug
def process_message_fixed(msg):
data = json.loads(msg)
# ✅ Add null check
name = data.get("name") or ""
return name.lower()
# Step 3: Regression test
def test_null_name_handling():
message = '{"id": 123, "name": null}'
result = process_message_fixed(message)
assert result == "" # No crash
Crash Reproduction Checklist:
- Capture input that triggers crash (from logs, telemetry)
- Minimize input (remove irrelevant data)
- Write failing test that reproduces crash
- Fix bug
- Verify test passes
- Commit test as regression prevention
Benefits:
- Prevents regressions
- Documents bug and fix
- Enables debugging without production access
Pattern 7: Stack Overflow Detection
When to use:
- Debugging recursive functions
- Investigating SIGSEGV from deep call stacks
- Implementing stack depth limits
#include <sys/resource.h>
#include <stdio.h>
void check_stack_limit() {
struct rlimit rl;
getrlimit(RLIMIT_STACK, &rl);
printf("Stack limit: %ld bytes (soft), %ld bytes (hard)\n",
(long)rl.rlim_cur, (long)rl.rlim_max);
}
void set_stack_limit(size_t bytes) {
struct rlimit rl;
rl.rlim_cur = bytes;
rl.rlim_max = bytes;
if (setrlimit(RLIMIT_STACK, &rl) != 0) {
perror("setrlimit");
}
}
// Detect stack overflow before crash
#include <ucontext.h>
void recursive_function(int depth) {
char buffer[1024]; // Consumes stack
// Estimate stack usage
static void *stack_start = NULL;
if (stack_start == NULL) {
stack_start = &buffer;
}
size_t used = (char*)stack_start - (char*)&buffer;
if (used > 1024 * 1024) { // 1MB threshold
fprintf(stderr, "Stack overflow risk! Depth=%d, Used=%zu\n", depth, used);
return; // Stop recursion
}
recursive_function(depth + 1);
}
Rust Stack Overflow (automatic detection):
// Rust detects stack overflow and panics
fn recursive_function(depth: u32) {
let _buffer = [0u8; 1024]; // Stack allocation
if depth < 1_000_000 {
recursive_function(depth + 1);
}
}
// Stack overflow triggers panic (not SIGSEGV)
// thread 'main' has overflowed its stack
Benefits:
- Early detection (before crash)
- Configurable stack limits
- Graceful degradation
Pattern 8: Post-Mortem Debugging Workflow
When to use:
- Production crash analysis
- Structured approach to crash investigation
- Team crash triage process
Crash Workflow:
1. Capture crash report (telemetry, logs, core dump)
2. Symbolicate stack trace (match symbols to binary version)
3. Identify crash signature (function, file, line)
4. Search for existing issues (check if known bug)
5. Reproduce crash (minimal test case)
6. Analyze root cause (memory, logic, race condition)
7. Fix and test
8. Deploy fix and verify (monitor crash rate)
9. Document (RCA, post-mortem)
Crash Triage Template:
## Crash Report: SIGSEGV in process_request()
**Signature**: `process_request() → parse_header() → strlen(NULL)`
**Stack Trace**:
#0 strlen() at /lib/x86_64-linux-gnu/libc.so.6 #1 parse_header(header=0x0) at server.c:123 #2 process_request(req=0x7f1234567890) at server.c:89
**Root Cause**: NULL pointer passed to parse_header()
**Impact**: 142 crashes in last 24h (0.3% of requests)
**Fix**: Add null check before strlen()
**Test**: Added regression test in test_null_header()
**Deployed**: v1.2.1 (2025-10-26)
**Verification**: Zero crashes in 48h post-deployment
Benefits:
- Structured investigation
- Knowledge sharing (team-wide)
- Prevents duplicate debugging effort
Quick Reference
Signal Quick Reference
Signal | Meaning | Common Cause
----------|--------------------------|---------------------------
SIGSEGV | Segmentation Fault | NULL deref, buffer overflow
SIGABRT | Abort | assert(), abort()
SIGFPE | Floating Point Exception | Division by zero
SIGILL | Illegal Instruction | Corrupted binary
SIGBUS | Bus Error | Misaligned access
SIGSTKFLT | Stack Fault | Stack overflow
Fuzzing Tool Comparison
Tool | Type | Speed | Setup | Best For
----------|-------------|---------|------------|------------------
AFL++ | File-based | Medium | Easy | Programs reading files
libFuzzer | In-process | Fast | Moderate | Libraries, APIs
Honggfuzz | Hybrid | Fast | Easy | Multi-threaded apps
OSS-Fuzz | Continuous | N/A | Complex | Open-source projects
Key Guidelines
✅ DO: Install signal handlers in production (logging, cleanup)
✅ DO: Generate backtraces on crashes (execinfo.h, std::stacktrace)
✅ DO: Integrate crash telemetry (Sentry, Crashlytics)
✅ DO: Write regression tests for all crashes
✅ DO: Fuzz critical parsers and input handlers
✅ DO: Symbolicate stack traces (match binary version)
✅ DO: Set stack limits to detect overflow early
❌ DON'T: Ignore crashes in testing (fix before production)
❌ DON'T: Use unsafe operations in signal handlers (malloc, locks)
❌ DON'T: Deploy without crash reporting infrastructure
❌ DON'T: Assume crashes are user error (investigate all)
❌ DON'T: Skip fuzzing security-critical code
❌ DON'T: Delete crash logs before analysis
Anti-Patterns
Critical Violations
// ❌ NEVER: Empty signal handler (hides crashes)
void signal_handler(int sig) {
// Do nothing - crash ignored!
}
// ✅ CORRECT: Log and abort
void signal_handler(int sig) {
write(STDERR_FILENO, "CRASH\n", 6);
signal(sig, SIG_DFL);
raise(sig); // Re-raise to generate core dump
}
❌ Ignoring crashes: Masks bugs, leads to data corruption ✅ Correct approach: Log, cleanup, generate core dump
Common Mistakes
# ❌ Don't: Generic exception handling (hides crashes)
try:
process_data(user_input)
except Exception:
pass # ❌ Silently ignores all errors
# ✅ Correct: Catch specific exceptions, log unknown
try:
process_data(user_input)
except ValueError as e:
logger.warning(f"Invalid input: {e}")
except Exception as e:
logger.error(f"Unexpected error: {e}", exc_info=True)
sentry_sdk.capture_exception(e)
raise # Re-raise unexpected errors
❌ Broad exception suppression: Hides bugs, makes debugging impossible ✅ Better: Catch specific exceptions, log and report unknown
// ❌ Don't: Allocate memory in signal handler
void crash_handler(int sig) {
std::string msg = "Crash!"; // ❌ Allocates memory (unsafe)
std::cout << msg << std::endl; // ❌ Not async-signal-safe
}
// ✅ Correct: Use only async-signal-safe functions
void crash_handler(int sig) {
const char msg[] = "Crash!\n";
write(STDERR_FILENO, msg, sizeof(msg) - 1); // ✅ Safe
_exit(1); // ✅ Safe (unlike exit())
}
❌ Unsafe signal handlers: May deadlock or corrupt state ✅ Better: Use only async-signal-safe functions (write, _exit)
Related Skills
debugging/core-dump-analysis.md- Analyzing core dumps from crashesdebugging/memory-leak-debugging.md- Memory corruption causes crashesdebugging/concurrency-debugging.md- Race conditions cause crashestesting/integration-testing.md- Crash reproduction in testsobservability/structured-logging.md- Crash context in logscicd/testing-strategy.md- Fuzzing in CI pipelines
Last Updated: 2025-10-26 Format Version: 1.0 (Atomic)