18 KiB
name, description
| name | description |
|---|---|
| debugging-remote-debugging | Comprehensive guide to remote debugging techniques for production and development environments. Covers SSH port forwarding, VSCode Remote Development, JetBrains Gateway, container debugging (Docker), Kubernetes debugging, production-safe debugging, debug symbols, and source maps. |
Remote Debugging
Last Updated: 2025-10-26
Overview
Remote debugging enables debugging applications running on remote servers, containers, or cloud environments. This guide covers safe and effective remote debugging strategies.
Core Concepts
Remote Debugging Architecture
Local Machine (Debugger)
↓ SSH / Network
Remote Machine (Debuggee)
↓
Debug Server (gdbserver, debugpy, etc.)
↓
Application Process
When to Use Remote Debugging
✅ Development on remote server (cloud VM)
✅ Docker container debugging
✅ Kubernetes pod debugging
✅ Production issue investigation (with caution)
✅ IoT / embedded device debugging
✅ Cross-platform debugging (Windows ↔ Linux)
❌ Local development (use local debugger)
❌ Production without monitoring (risky)
❌ High-throughput systems (performance impact)
SSH Port Forwarding
Local Port Forwarding
Forward remote port to local machine:
# Forward remote debugger port 5678 to local 5678
ssh -L 5678:localhost:5678 [email protected]
# Now connect local debugger to localhost:5678
# It tunnels to remote-server.com:5678
Example: Python debugpy:
# On remote server:
python -m debugpy --listen 0.0.0.0:5678 --wait-for-client script.py
# On local machine:
ssh -L 5678:localhost:5678 [email protected]
# In VSCode: Attach to localhost:5678
Remote Port Forwarding
Forward local port to remote machine:
# Make local port 8080 accessible on remote at port 9090
ssh -R 9090:localhost:8080 [email protected]
# Remote can now access localhost:9090 → your local 8080
Dynamic Port Forwarding (SOCKS Proxy)
Create SOCKS proxy for multiple ports:
# Dynamic proxy on local port 1080
ssh -D 1080 [email protected]
# Configure browser/app to use SOCKS proxy localhost:1080
# All connections tunnel through SSH
Persistent SSH Tunnels
Keep tunnel alive:
# Auto-reconnect on disconnect
ssh -L 5678:localhost:5678 -o ServerAliveInterval=60 -o ServerAliveCountMax=3 [email protected]
# Or use autossh
autossh -M 0 -L 5678:localhost:5678 [email protected]
VSCode Remote Development
Remote - SSH Extension
Install extension:
Extensions → Search "Remote - SSH" → Install
Connect to remote:
1. Cmd+Shift+P → "Remote-SSH: Connect to Host"
2. Enter: [email protected]
3. Select platform: Linux / Windows / macOS
4. Opens new VSCode window connected to remote
Now:
- File Explorer shows remote files
- Terminal is remote shell
- Debugger runs on remote
- Extensions installed on remote
SSH Config (~/.ssh/config):
Host my-remote
HostName 192.168.1.100
User myuser
Port 22
IdentityFile ~/.ssh/id_rsa
ForwardAgent yes
# Connect in VSCode: "Remote-SSH: Connect to Host" → my-remote
Remote Debugging Python
Install Python extension on remote:
VSCode → Remote connection → Extensions → Install Python
launch.json (remote debugging):
{
"version": "0.2.0",
"configurations": [
{
"name": "Python: Current File (Remote)",
"type": "python",
"request": "launch",
"program": "${file}",
"console": "integratedTerminal",
"justMyCode": false
}
]
}
Workflow:
1. Open remote folder in VSCode
2. Set breakpoint in Python file
3. F5 to debug
4. Debugger runs on remote, UI in local VSCode
Remote - Containers Extension
Debug inside Docker container:
1. Install "Remote - Containers" extension
2. Cmd+Shift+P → "Remote-Containers: Attach to Running Container"
3. Select container
4. Debug inside container environment
devcontainer.json:
{
"name": "Python Dev",
"image": "python:3.11",
"extensions": [
"ms-python.python"
],
"forwardPorts": [5678],
"postCreateCommand": "pip install debugpy"
}
JetBrains Gateway
Remote Development via Gateway
Setup:
1. Download JetBrains Gateway
2. New Connection → SSH
3. Enter: [email protected]
4. Select IDE (PyCharm, IntelliJ, etc.)
5. Gateway installs IDE backend on remote
6. Opens IDE UI locally, runs on remote
Advantages:
✅ Full IDE features on remote
✅ Large codebase stays on remote (fast indexing)
✅ Automatic port forwarding
✅ Integrated debugger
Remote debugging:
1. Open project on remote via Gateway
2. Set breakpoints
3. Run → Debug (Shift+F9)
4. Debugger runs on remote, UI local
Docker Debugging
Attach to Running Container
VSCode attach:
# Start container with debugger port exposed
docker run -p 5678:5678 my-image
# In VSCode launch.json:
{
"name": "Python: Attach to Docker",
"type": "python",
"request": "attach",
"connect": {
"host": "localhost",
"port": 5678
},
"pathMappings": [
{
"localRoot": "${workspaceFolder}",
"remoteRoot": "/app"
}
]
}
Python container example:
# Dockerfile
FROM python:3.11
WORKDIR /app
COPY requirements.txt .
RUN pip install -r requirements.txt
RUN pip install debugpy
COPY . .
# Start with debugger
CMD ["python", "-m", "debugpy", "--listen", "0.0.0.0:5678", "--wait-for-client", "app.py"]
# Build and run
docker build -t my-app .
docker run -p 5678:5678 my-app
# Attach VSCode debugger to localhost:5678
Docker Exec (Interactive Debugging)
Attach to running container:
# Find container ID
docker ps
# Exec into container
docker exec -it <container-id> /bin/bash
# Install debugger
pip install ipdb
# Add breakpoint in code
import ipdb; ipdb.set_trace()
# Run app, drops into debugger in terminal
Docker Compose Debugging
docker-compose.yml:
services:
app:
build: .
ports:
- "5678:5678"
environment:
- DEBUGPY_ENABLED=true
volumes:
- .:/app # Mount source for live editing
command: python -m debugpy --listen 0.0.0.0:5678 --wait-for-client app.py
# Start with compose
docker-compose up
# Attach debugger to localhost:5678
Kubernetes Debugging
kubectl debug (Ephemeral Containers)
Debug running pod (Kubernetes 1.23+):
# Create ephemeral debug container in pod
kubectl debug my-pod -it --image=busybox --target=my-container
# Or with debug utilities
kubectl debug my-pod -it --image=nicolaka/netshoot --target=my-container
# Inside debug container:
# Can inspect processes, network, filesystem
ps aux
netstat -tuln
ls /proc/1/root/app
Debug with copy:
# Create copy of pod for debugging
kubectl debug my-pod --copy-to=my-pod-debug --container=my-container
# Modify container in copy
kubectl debug my-pod --copy-to=my-pod-debug --container=my-container --image=python:3.11-debug
Port Forwarding
Forward pod port to local:
# Forward pod port 5678 to local 5678
kubectl port-forward pod/my-pod 5678:5678
# Attach debugger to localhost:5678
Forward service:
# Forward service port
kubectl port-forward service/my-service 5678:5678
Remote Debugging in Pod
Deploy with debugger:
# deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-app
spec:
template:
spec:
containers:
- name: app
image: my-app:latest
ports:
- containerPort: 5678 # Debug port
env:
- name: DEBUG_MODE
value: "true"
command: ["python", "-m", "debugpy", "--listen", "0.0.0.0:5678", "--wait-for-client", "app.py"]
# Apply deployment
kubectl apply -f deployment.yaml
# Forward port
kubectl port-forward deployment/my-app 5678:5678
# Attach debugger to localhost:5678
Kubernetes Debugging Best Practices
✅ Use ephemeral containers for live debugging
✅ Create pod copy for intrusive debugging
✅ Use port-forward for debugger access
✅ Remove debug configuration from production
✅ Use separate debug environment/namespace
❌ Don't debug production pods directly (use copy)
❌ Don't leave debug ports exposed
❌ Don't modify production deployments for debugging
Production-Safe Debugging
Principles
Safety rules:
1. NEVER pause execution in production
→ Use logging, tracing, metrics instead
2. MINIMIZE performance impact
→ Use sampling, not continuous debugging
3. ISOLATE debugging sessions
→ Debug copy of pod/container, not live instance
4. TIME-BOX sessions
→ Set max debug duration, auto-timeout
5. AUDIT access
→ Log who debugged what and when
6. REVERT changes
→ Never leave debug code in production
Read-Only Debugging
Observe without modifying:
# ❌ Bad: Modify state in production
def debug_user(user_id):
user = get_user(user_id)
user.role = 'admin' # DANGEROUS IN PRODUCTION
save_user(user)
# ✅ Good: Observe only
def debug_user(user_id):
user = get_user(user_id)
logger.info(f"User state: {user}") # Read-only
# Don't modify, just observe
Logging-Based Debugging
Use structured logging instead of breakpoints:
import logging
import json
logger = logging.getLogger(__name__)
def process_order(order):
# Instead of breakpoint(), log context
logger.info("Processing order", extra={
"order_id": order.id,
"user_id": order.user_id,
"items": [item.id for item in order.items],
"total": order.total
})
result = calculate_total(order)
logger.info("Order processed", extra={
"order_id": order.id,
"result": result
})
return result
Sampling and Rate Limiting
Debug subset of requests:
import random
def should_debug():
# Debug 1% of requests
return random.random() < 0.01
def handle_request(request):
if should_debug():
logger.debug("Full request context", extra={
"headers": request.headers,
"body": request.body,
"params": request.params
})
# Normal processing
return process(request)
Time-Limited Debugging
Auto-disable debug mode:
import time
DEBUG_ENABLED_UNTIL = None # Unix timestamp
def enable_debug(duration_seconds):
global DEBUG_ENABLED_UNTIL
DEBUG_ENABLED_UNTIL = time.time() + duration_seconds
logger.warning(f"Debug mode enabled for {duration_seconds}s")
def is_debug_enabled():
if DEBUG_ENABLED_UNTIL is None:
return False
if time.time() > DEBUG_ENABLED_UNTIL:
logger.info("Debug mode expired")
return False
return True
def process_data(data):
if is_debug_enabled():
logger.debug(f"Processing: {data}")
# Normal processing
Debug Symbols
Debug Symbols for Compiled Languages
C/C++ with GDB:
# Compile with debug symbols
gcc -g program.c -o program
# Strip symbols for production (smaller binary)
strip program -o program-stripped
# Debug with separate symbol file
gcc -g program.c -o program
objcopy --only-keep-debug program program.debug
strip program
objcopy --add-gnu-debuglink=program.debug program
# GDB loads symbols automatically
gdb program
(gdb) file program
(gdb) symbol-file program.debug
Rust:
# Debug build (includes symbols)
cargo build
# Release build (optimized, no symbols)
cargo build --release
# Release with debug symbols
cargo build --release --config profile.release.debug=true
Go:
# Build with symbols (default)
go build -o myapp
# Strip symbols
go build -ldflags="-s -w" -o myapp
# Delve debugger
dlv exec ./myapp
Symbol Servers
Microsoft Symbol Server (Windows debugging):
Use symbol server for Windows binaries:
SRV*c:\symbols*https://msdl.microsoft.com/download/symbols
Linux debug symbols:
# Ubuntu: install debug symbols
sudo apt install <package>-dbg
# Example: Python debug symbols
sudo apt install python3.11-dbg
Source Maps (JavaScript/TypeScript)
Generating Source Maps
TypeScript:
// tsconfig.json
{
"compilerOptions": {
"sourceMap": true,
"inlineSourceMap": false, // Separate .map files
"inlineSources": false,
"sourceRoot": "/"
}
}
Webpack:
// webpack.config.js
module.exports = {
devtool: 'source-map', // Production: separate files
// devtool: 'inline-source-map', // Development: inline
// devtool: 'eval-source-map', // Fastest rebuild
};
Vite:
// vite.config.js
export default {
build: {
sourcemap: true, // Generate .map files
}
}
Using Source Maps
Browser DevTools:
Settings (⚙️) → Sources → Enable JavaScript source maps
Loads .map files automatically:
app.min.js
app.min.js.map // Loaded automatically
Debug shows original TypeScript/source code
Node.js:
# Enable source maps
node --enable-source-maps app.js
# Stack traces show original source, not transpiled
Production Source Maps
Security considerations:
❌ Don't expose source maps publicly
→ Reveals source code to attackers
✅ Host source maps separately, restrict access
✅ Use Sentry/error tracking to symbolicate server-side
✅ Strip source maps from public builds
Sentry source maps:
# Upload source maps to Sentry
sentry-cli releases files <release> upload-sourcemaps ./dist --rewrite
# Sentry symbolicates errors using uploaded maps
# Users never see source maps
Remote Debugging Tools by Language
Python: debugpy
Remote script:
import debugpy
# Listen on all interfaces, port 5678
debugpy.listen(("0.0.0.0", 5678))
print("Waiting for debugger...")
debugpy.wait_for_client() # Block until debugger attaches
# Your code
def main():
for i in range(10):
print(f"Iteration {i}")
if __name__ == "__main__":
main()
VSCode attach:
{
"name": "Python: Remote Attach",
"type": "python",
"request": "attach",
"connect": {
"host": "remote-server.com",
"port": 5678
},
"pathMappings": [
{
"localRoot": "${workspaceFolder}",
"remoteRoot": "/app"
}
]
}
Node.js: Inspector
Start with inspector:
# Remote Node.js
node --inspect=0.0.0.0:9229 app.js
# Or debug-brk (pause at start)
node --inspect-brk=0.0.0.0:9229 app.js
Chrome DevTools:
1. SSH tunnel: ssh -L 9229:localhost:9229 user@remote
2. Chrome: chrome://inspect
3. Configure: Add localhost:9229
4. Click "inspect" under Remote Target
VSCode attach:
{
"name": "Node: Attach to Remote",
"type": "node",
"request": "attach",
"address": "localhost",
"port": 9229,
"localRoot": "${workspaceFolder}",
"remoteRoot": "/app"
}
Go: Delve
Remote debugging:
# On remote server
dlv debug --headless --listen=:2345 --api-version=2 --accept-multiclient
# Local machine
dlv connect remote-server.com:2345
VSCode attach:
{
"name": "Go: Connect to Remote",
"type": "go",
"request": "attach",
"mode": "remote",
"remotePath": "/app",
"port": 2345,
"host": "remote-server.com"
}
Java: JDWP
Start JVM with debug agent:
# Remote Java
java -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005 -jar app.jar
IntelliJ remote debug:
1. Run → Edit Configurations
2. Add → Remote JVM Debug
3. Host: remote-server.com
4. Port: 5005
5. Click Debug
Anti-Patterns
Common Mistakes
❌ NEVER: Debug production without copy/staging
→ Pausing production = downtime
❌ NEVER: Leave debug ports exposed to internet
→ Security vulnerability, remote code execution
❌ NEVER: Commit debug code (breakpoint(), debugger;)
→ Pauses production app
❌ NEVER: Debug without SSH tunnel on public network
→ Unencrypted debug traffic
❌ NEVER: Modify production state while debugging
→ Data corruption, undefined behavior
❌ NEVER: Debug high-traffic endpoints
→ Performance impact affects all users
Best Practices
✅ ALWAYS: Use SSH tunnels for remote debugging
✅ ALWAYS: Debug on copy/staging, not production
✅ ALWAYS: Time-limit debug sessions
✅ ALWAYS: Use read-only debugging in production
✅ ALWAYS: Prefer logging/tracing over breakpoints
✅ ALWAYS: Remove debug configuration after session
✅ ALWAYS: Audit who debugged what and when
Related Skills
- gdb-fundamentals.md: GDB for C/C++/Rust
- lldb-macos-debugging.md: LLDB for macOS/iOS
- python-debugging.md: Python debugging tools
- browser-devtools.md: Browser debugging
- docker-debugging.md: Docker-specific debugging
- kubernetes-debugging.md: Kubernetes debugging strategies
Summary
Remote debugging enables debugging applications in remote environments:
- SSH tunneling: Port forwarding for secure remote access
- VSCode Remote: Remote-SSH, Remote-Containers extensions
- JetBrains Gateway: Full IDE on remote servers
- Docker debugging: Attach to containers, exec debugging
- Kubernetes debugging: Ephemeral containers, port forwarding
- Production safety: Read-only, sampling, time-limited debugging
- Debug symbols: Separate symbols for production binaries
- Source maps: JavaScript/TypeScript source mapping
Quick start (Python):
# Remote server
python -m debugpy --listen 0.0.0.0:5678 --wait-for-client app.py
# Local machine
ssh -L 5678:localhost:5678 [email protected]
# VSCode: Attach to localhost:5678
Production debugging:
# Use logging, not breakpoints
logger.info("Debug context", extra={"user_id": user.id, "state": state})
# Sample 1% of requests
if random.random() < 0.01:
logger.debug("Full request", extra={"request": request})
Master remote debugging for efficient distributed system troubleshooting.