26 KiB
name, description
| name | description |
|---|---|
| engineering-rfc-technical-design | Architecture proposals, technical diagrams, API design, trade-off analysis, and migration strategies for RFCs |
RFC Technical Design
Scope: Techniques for designing architecture, evaluating alternatives, creating technical diagrams, and planning migrations Lines: ~340 Last Updated: 2025-10-25 Format Version: 1.0 (Atomic)
When to Use This Skill
Activate this skill when:
- Designing system architecture for a new feature or service
- Evaluating multiple technical approaches and trade-offs
- Creating technical diagrams (architecture, sequence, data flow)
- Designing APIs (REST, GraphQL, gRPC) with versioning and compatibility
- Planning database schema changes or migrations
- Defining rollout strategies for new systems or migrations
- Analyzing performance, scalability, and reliability trade-offs
- Documenting technical decisions with quantitative justification
Core Concepts
Concept 1: Architecture Design Levels (C4 Model)
Level 1: System Context
- High-level view: System + external dependencies
- Audience: Non-technical stakeholders
- Example: "Web App talks to Stripe, SendGrid, and PostgreSQL"
Level 2: Container Diagram
- Major components (services, databases, message queues)
- Audience: Architects, senior engineers
- Example: "API Server, Worker Queue, Redis Cache, Postgres DB"
Level 3: Component Diagram
- Internal structure of a container (modules, classes)
- Audience: Engineers implementing the system
- Example: "API Server has Auth, User, Project modules"
Level 4: Code Diagram
- Class diagrams, sequence diagrams (optional)
- Audience: Engineers debugging or extending code
Concept 2: Trade-Off Analysis Framework
Evaluation Dimensions:
- Performance: Latency, throughput, resource usage
- Scalability: Horizontal vs vertical, growth limits
- Complexity: Development time, maintenance burden
- Cost: Infrastructure, licensing, operational overhead
- Reliability: Fault tolerance, recovery, uptime
- Maintainability: Code clarity, debugging, extensibility
Quantification: Always include numbers where possible
- "50% faster" vs "faster"
- "$500/month at 10k users" vs "expensive"
- "2 weeks development" vs "takes time"
Concept 3: API Design Principles
Versioning Strategies:
- URI versioning:
/api/v1/users,/api/v2/users - Header versioning:
Accept: application/vnd.api+json; version=1 - Deprecation timeline: Announce → Grace period → Sunset
Backwards Compatibility:
- Add fields, don't remove (deprecate instead)
- Use optional parameters (don't require new fields)
- Version breaking changes (new endpoints)
Error Handling:
- Use HTTP status codes correctly (200, 201, 400, 401, 404, 500)
- Return structured errors (error_code, message, field_errors)
- Include request_id for debugging
Patterns
Pattern 1: Architecture Diagram (C4 Level 2 - Container)
When to use:
- Designing system architecture for RFC
- Illustrating component interactions
## System Architecture: Real-Time Analytics Platform
### Container Diagram (C4 Level 2)
┌─────────────────────────────────────────────────────────────────┐ │ User Clients │ ├─────────────────────────────────────────────────────────────────┤ │ │ │ ┌──────────┐ ┌──────────┐ ┌──────────┐ │ │ │ Web │ │ Mobile │ │ API │ │ │ │ Client │ │ App │ │ Client │ │ │ └────┬─────┘ └────┬─────┘ └────┬─────┘ │ │ │ │ │ │ └─────────┼─────────────────────┼──────────────────────┼───────────┘ │ │ │ │ HTTPS │ HTTPS │ HTTPS │ │ │ ▼ ▼ ▼ ┌─────────────────────────────────────────────────────────────────┐ │ Load Balancer (NGINX) │ └────────────────────────────┬────────────────────────────────────┘ │ ┌──────────────────┼──────────────────┐ │ │ │ ▼ ▼ ▼ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │ API Server │ │ API Server │ │ API Server │ │ (Node.js) │ │ (Node.js) │ │ (Node.js) │ └──────┬───────┘ └──────┬───────┘ └──────┬───────┘ │ │ │ │ │ │ ├──────────────────┴──────────────────┤ │ │ ▼ ▼ ┌─────────────┐ ┌─────────────┐ │ Redis │ │ RabbitMQ │ │ Cache │ │ Queue │ │ │ │ │ └─────────────┘ └──────┬──────┘ │ │ ▼ ┌──────────────┐ │ Worker │ │ Processes │ │ (Python) │ └──────┬───────┘ │ ┌─────────────────────────────────────┤ │ │ ▼ ▼ ┌─────────────┐ ┌─────────────┐ │ PostgreSQL │ │ S3 Bucket │ │ Database │ │ (Logs) │ │ │ │ │ └─────────────┘ └─────────────┘
### Component Responsibilities
**Load Balancer (NGINX)**:
- Distributes traffic across API servers (round-robin)
- Terminates SSL/TLS
- Rate limiting (1000 req/min per IP)
**API Server (Node.js + Express)**:
- Handles HTTP requests
- Authentication/authorization (JWT)
- Business logic
- Publishes jobs to RabbitMQ
**Redis Cache**:
- Session storage (JWT tokens)
- Query result caching (TTL: 5 min)
- Rate limit counters
**RabbitMQ Queue**:
- Async job processing (reports, exports)
- Decouples API from heavy workloads
**Worker Processes (Python)**:
- Consumes jobs from RabbitMQ
- Generates reports, processes data
- Uploads results to S3
**PostgreSQL Database**:
- Primary data store (users, projects, analytics)
- ACID transactions
- Read replicas for analytics queries
**S3 Bucket**:
- Stores generated reports, exports
- CloudFront CDN for fast delivery
Pattern 2: Sequence Diagram for Complex Workflows
Use case: Illustrating multi-step interactions
## Sequence Diagram: User Signup Flow
Client API Server Database SendGrid │ │ │ │ │ POST /signup │ │ │ ├────────────────>│ │ │ │ │ │ │ │ │ Validate │ │ │ │ email/password│ │ │ │ │ │ │ │ Check if │ │ │ │ email exists │ │ │ ├──────────────>│ │ │ │<──────────────┤ │ │ │ (not found) │ │ │ │ │ │ │ │ Create user │ │ │ ├──────────────>│ │ │ │<──────────────┤ │ │ │ (user_id) │ │ │ │ │ │ │ │ Generate │ │ │ │ verify token │ │ │ │ │ │ │ │ Send verification email │ │ ├─────────────────────────────>│ │ │<─────────────────────────────┤ │ │ (200 OK) │ │ │ │ │ │ 201 Created │ │ │ │<────────────────┤ │ │ │ {user_id, ...} │ │ │ │ │ │ │ │ │ │ Email │ │ │ │ delivered │ │ │ │ to user │ │ │ │ │
### Error Scenario: Email Already Exists
Client API Server Database │ │ │ │ POST /signup │ │ ├────────────────>│ │ │ │ Check email │ │ ├──────────────>│ │ │<──────────────┤ │ │ (found!) │ │ │ │ │ 400 Bad Req │ │ │<────────────────┤ │ │ {error_code: │ │ │ EMAIL_IN_USE} │ │
Pattern 3: Trade-Off Matrix for Alternatives
Use case: Comparing technical approaches objectively
## Trade-Off Analysis: Caching Strategy
### Options Evaluated
| Criterion | Redis (In-Memory) | Postgres (DB Cache) | Memcached | CDN (CloudFront) |
|-----------|-------------------|---------------------|-----------|------------------|
| **Latency** | <1ms (local) | ~10ms (network) | <1ms (local) | ~50ms (edge) |
| **Scalability** | Horizontal (sharding) | Vertical (limited) | Horizontal | Infinite |
| **Persistence** | Optional (RDB/AOF) | Yes (durable) | No (memory only) | No (edge cache) |
| **Complexity** | Medium (setup cluster) | Low (already have DB) | Low (simple) | Medium (CDN config) |
| **Cost** | $50/mo (10GB instance) | $0 (included) | $30/mo (5GB) | $0.02/GB (traffic) |
| **Use Case Fit** | Session + query cache | Small datasets | Session only | Static assets |
| **Max TTL** | Custom (hours) | Custom (hours) | 30 days | 1 year |
### Decision: Redis
**Chosen**: Redis (in-memory cache with persistence)
**Rationale**:
- **Latency**: <1ms meets requirement (<10ms)
- **Scalability**: Can shard by cache key if needed (10k→100k users)
- **Persistence**: RDB snapshots prevent cache stampede on restart
- **Cost**: $50/mo acceptable for performance gain (vs $0 DB cache, but 10x slower)
- **Flexibility**: Supports sessions, query cache, rate limiting, pub/sub
**Runner-Up**: Postgres materialized views (for read-heavy analytics only)
**Rejected**:
- **Memcached**: No persistence (unacceptable for rate limiting data)
- **CDN**: Only for static assets (API responses are dynamic)
Pattern 4: API Design with Versioning
Use case: Designing RESTful API with backwards compatibility
## API Design: User Management
### Base URL
- Production: `https://api.example.com`
- Staging: `https://api-staging.example.com`
### Versioning Strategy
- URI versioning: `/v1/users`, `/v2/users`
- Current version: `v1`
- Deprecation policy: 12 months notice, 6 months grace period
### Endpoints
#### `POST /v1/users` - Create User
**Request**:
```json
{
"email": "user@example.com",
"password": "SecurePass1!",
"display_name": "John Doe" // optional
}
Response (201 Created):
{
"user_id": "usr_abc123",
"email": "[email protected]",
"display_name": "John Doe",
"created_at": "2025-10-25T14:30:00Z"
}
Response (400 Bad Request):
{
"error_code": "VALIDATION_ERROR",
"error_message": "Invalid input",
"request_id": "req_xyz789",
"field_errors": {
"email": "Email already in use",
"password": "Password must include a number"
}
}
Rate Limiting:
- 5 signups per IP per hour
- Headers:
X-RateLimit-Limit: 5,X-RateLimit-Remaining: 3
GET /v1/users/:id - Get User Profile
Authentication: Required (JWT in Authorization: Bearer <token>)
Response (200 OK):
{
"user_id": "usr_abc123",
"email": "[email protected]",
"display_name": "John Doe",
"profile_photo_url": "https://cdn.example.com/photos/abc123.jpg",
"created_at": "2025-10-25T14:30:00Z",
"preferences": {
"theme": "dark",
"notifications": true
}
}
Response (404 Not Found):
{
"error_code": "USER_NOT_FOUND",
"error_message": "User with ID usr_abc123 not found",
"request_id": "req_xyz789"
}
Breaking Change Example (v1 → v2)
v1: Returns created_at as Unix timestamp (integer)
{"created_at": 1698249600}
v2: Returns created_at as ISO 8601 string (breaking change)
{"created_at": "2025-10-25T14:30:00Z"}
Migration Plan:
- Announce deprecation: 2025-11-01 (email, docs, changelog)
- Grace period: v1 supported until 2026-05-01 (6 months)
- Sunset: v1 returns 410 Gone after 2026-05-01
### Pattern 5: Database Schema Design with Migrations
**Use case**: Planning schema changes with rollback strategy
```markdown
## Database Schema: User Authentication
### Current Schema (v1)
```sql
CREATE TABLE users (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
email VARCHAR(255) UNIQUE NOT NULL,
password_hash VARCHAR(255) NOT NULL,
created_at TIMESTAMP DEFAULT NOW()
);
Proposed Schema (v2) - Add Email Verification
-- Migration: Add email verification columns
ALTER TABLE users
ADD COLUMN email_verified BOOLEAN DEFAULT FALSE,
ADD COLUMN email_verify_token VARCHAR(255),
ADD COLUMN email_verify_sent_at TIMESTAMP;
-- Create index for fast token lookup
CREATE INDEX idx_users_verify_token ON users(email_verify_token)
WHERE email_verify_token IS NOT NULL;
-- Backfill existing users (mark as verified)
UPDATE users SET email_verified = TRUE WHERE created_at < NOW();
Migration Plan
Step 1: Deploy Schema Change (Zero Downtime)
- Run
ALTER TABLEduring low-traffic window - PostgreSQL locks table briefly (<1s for 100k rows)
- Backfill existing users as verified (no behavior change)
Step 2: Deploy Code Changes
- New signups: Set
email_verified = FALSE, send verification email - Login: Check
email_verified, block if false - Feature flag:
REQUIRE_EMAIL_VERIFICATION(gradual rollout)
Step 3: Monitor
- Track verification rate (target: 80% verify within 24 hours)
- Monitor support tickets (expect questions)
- Alert if verification email delivery rate <95%
Rollback Plan:
- Revert code: Disable
REQUIRE_EMAIL_VERIFICATIONflag - Keep schema changes (no need to drop columns, just stop using)
- If catastrophic: Drop columns (loses verification data)
### Pattern 6: Migration Strategy (Old System → New System)
**Use case**: Transitioning between systems with zero downtime
```markdown
## Migration: Monolith → Microservices (Auth Service)
### Current State
- Monolithic Rails app handles everything (auth, users, projects)
- Auth logic mixed with business logic (tight coupling)
- 100k active users, 10k requests/min peak
### Target State
- Standalone Auth Service (Node.js + JWT)
- Microservices for users, projects (separate from auth)
- Same 100k users, seamless migration
### Migration Strategy: Strangler Fig Pattern
**Phase 1: Build Auth Service (Parallel Run)**
- Build new Auth Service (JWT-based)
- Dual-write: Monolith writes to both old session DB and new Auth Service
- Read from old system only (no behavior change)
- Duration: 2 weeks
**Phase 2: Shadow Traffic**
- Route 1% of login requests to Auth Service (shadow mode)
- Compare results with monolith (validate correctness)
- Fix discrepancies, increase to 10%, then 50%
- Duration: 2 weeks
**Phase 3: Gradual Cutover**
- Route 10% of users to Auth Service (read + write)
- Monitor latency, error rates, user complaints
- Increase to 50%, then 100% over 4 weeks
- Duration: 4 weeks
**Phase 4: Monolith Deprecation**
- All auth traffic goes to Auth Service
- Remove auth code from monolith (dead code)
- Decommission old session database
- Duration: 2 weeks
### Rollback Plan
- Feature flag: `USE_AUTH_SERVICE` (toggle per user)
- If Auth Service fails, revert flag → monolith
- Monitoring: Alert if Auth Service latency >100ms or error rate >1%
### Metrics
- **Latency**: Auth Service <50ms (vs monolith 80ms)
- **Error Rate**: <0.1% (vs monolith 0.2%)
- **User Impact**: Zero failed logins during migration
Pattern 7: Performance Benchmarking
Use case: Quantifying trade-offs with real data
## Performance Benchmark: Caching Strategy
### Test Setup
- Dataset: 1M user records
- Query: `GET /users/:id/profile` (fetch user + projects)
- Load: 1,000 requests/sec (sustained)
- Tool: Apache Bench (`ab -n 10000 -c 100`)
### Results
| Approach | p50 Latency | p95 Latency | p99 Latency | Throughput | CPU Usage |
|----------|-------------|-------------|-------------|------------|-----------|
| **No Cache** (baseline) | 120ms | 350ms | 850ms | 800 req/s | 80% |
| **Redis Cache** | 8ms | 15ms | 45ms | 8,000 req/s | 20% |
| **Postgres Materialized View** | 25ms | 60ms | 150ms | 3,000 req/s | 40% |
| **In-Memory Map** (Node.js) | 2ms | 5ms | 12ms | 10,000 req/s | 30% |
### Analysis
**Winner**: Redis Cache (chosen)
- **Latency**: 15x faster than no cache (p95: 15ms vs 350ms) ✅
- **Throughput**: 10x higher (8k vs 800 req/s) ✅
- **Cost**: $50/mo for Redis instance (acceptable)
- **Risk**: Single point of failure (mitigated with Redis Sentinel)
**Runner-Up**: In-Memory Map
- **Faster**: 2ms latency, 10k req/s
- **Problem**: No persistence (lost on restart), no sharing across servers
- **Verdict**: Only viable for single-server deployment
**Rejected**: Postgres Materialized View
- **Reason**: Still 25ms latency (vs <10ms requirement)
- **Use Case**: Good for analytics, not real-time API
Pattern 8: Failure Mode Analysis
Use case: Anticipating what can go wrong
## Failure Mode Analysis: Payment Processing
### Failure Scenarios
#### 1. Stripe API Down
**Probability**: Low (99.99% uptime SLA)
**Impact**: High (no payments processed)
**Mitigation**:
- Detect: Health check every 30s, alert if 3 failures
- Respond: Show maintenance page, queue payment attempts
- Recover: Retry queued payments when Stripe recovers
#### 2. Database Connection Pool Exhausted
**Probability**: Medium (under high load)
**Impact**: High (API returns 500 errors)
**Mitigation**:
- Prevent: Set connection pool limit (50 connections)
- Detect: Monitor active connections (alert if >80%)
- Respond: Reject new requests with 503 (retry-after header)
- Recover: Scale database read replicas, optimize queries
#### 3. Redis Cache Eviction
**Probability**: Medium (if cache fills up)
**Impact**: Low (fallback to database, slower but works)
**Mitigation**:
- Prevent: Set max memory limit, use LRU eviction policy
- Detect: Monitor cache hit rate (alert if <80%)
- Respond: Cache miss → fetch from DB, repopulate cache
- Recover: Increase Redis memory, optimize cache keys
#### 4. Webhook Delivery Failure
**Probability**: High (network issues, client downtime)
**Impact**: Medium (events not processed by client)
**Mitigation**:
- Detect: HTTP status != 200 or timeout (>30s)
- Respond: Exponential backoff retry (1s, 2s, 4s, 8s, 16s)
- Recover: Dead letter queue after 5 failures, alert for manual review
### Circuit Breaker Pattern
```javascript
// Protect against cascading failures
class CircuitBreaker {
constructor(failureThreshold = 5, timeout = 60000) {
this.failureCount = 0;
this.failureThreshold = failureThreshold;
this.timeout = timeout;
this.state = 'CLOSED'; // CLOSED, OPEN, HALF_OPEN
}
async call(fn) {
if (this.state === 'OPEN') {
throw new Error('Circuit breaker is OPEN');
}
try {
const result = await fn();
this.onSuccess();
return result;
} catch (error) {
this.onFailure();
throw error;
}
}
onSuccess() {
this.failureCount = 0;
this.state = 'CLOSED';
}
onFailure() {
this.failureCount++;
if (this.failureCount >= this.failureThreshold) {
this.state = 'OPEN';
setTimeout(() => { this.state = 'HALF_OPEN'; }, this.timeout);
}
}
}
// Usage: Protect Stripe API calls
const stripeBreaker = new CircuitBreaker(5, 60000);
const charge = await stripeBreaker.call(() => stripe.charges.create({...}));
---
## Quick Reference
### Architecture Diagram Types
| Diagram Type | Use Case | Audience |
|---|---|---|
| System Context (C4-1) | High-level overview | Leadership, PM |
| Container (C4-2) | Major components | Architects, senior eng |
| Component (C4-3) | Internal structure | Engineers |
| Sequence | Multi-step workflows | Engineers debugging |
| Data Flow | Data movement | Data engineers |
### Trade-Off Evaluation Checklist
| Dimension | Question to Ask |
|---|---|
| Performance | What is p95 latency? Throughput? |
| Scalability | How does it scale? (horizontal/vertical) |
| Complexity | How long to implement? Maintain? |
| Cost | $ per month at 10k users? At 100k? |
| Reliability | What's uptime SLA? Failure modes? |
| Maintainability | How easy to debug? Extend? |
### API Versioning Strategies
| Strategy | Example | Pros | Cons |
|---|---|---|---|
| URI | /v1/users | Simple, explicit | URL proliferation |
| Header | Accept: v=1 | Clean URLs | Less visible |
| Query Param | /users?v=1 | Easy to test | Ugly URLs |
### Key Guidelines
✅ DO: Quantify trade-offs (numbers, not adjectives) ✅ DO: Create diagrams for complex architectures ✅ DO: Benchmark alternatives with real data ✅ DO: Plan failure modes and mitigations ✅ DO: Design APIs with backwards compatibility
❌ DON'T: Pick solutions without evaluating alternatives ❌ DON'T: Use vague terms ("fast", "scalable") without metrics ❌ DON'T: Ignore migration complexity ❌ DON'T: Skip performance benchmarking ❌ DON'T: Design APIs without versioning strategy
---
## Anti-Patterns
### Critical Violations
❌ **No Alternatives Analysis**: Presenting one solution without comparison
```markdown
# ❌ NEVER:
We'll use MongoDB for the database.
# ✅ CORRECT:
## Database Options
1. **PostgreSQL**: ACID, relations, complex queries (chosen)
2. **MongoDB**: Flexible schema, horizontal scaling
- Rejected: No ACID for financial transactions (deal-breaker)
3. **DynamoDB**: Managed, infinite scale
- Rejected: Vendor lock-in, limited query flexibility
**Decision**: Postgres for ACID guarantees and query richness.
❌ Premature Optimization: Optimizing without measuring ✅ Correct approach: Benchmark current performance, identify bottleneck, optimize
Common Mistakes
❌ Vague Performance Claims: "Redis is faster than Postgres"
# ❌ Don't:
Redis is faster, so we'll use it for caching.
# ✅ Correct:
## Benchmark Results
- Redis: 8ms p95 latency, 8k req/s throughput
- Postgres: 120ms p95 latency, 800 req/s throughput
- **Decision**: Redis for 15x latency improvement
❌ No Failure Analysis: Assuming happy path only ✅ Better: Document failure modes, circuit breakers, fallbacks
❌ API Without Versioning: No plan for breaking changes ✅ Better: URI versioning, deprecation policy, migration guide
❌ Big Bang Migration: Switching all users at once ✅ Better: Strangler fig pattern (gradual cutover with rollback plan)
Related Skills
rfc-structure-format.md- RFC document templates and formattingrfc-consensus-building.md- Driving approval for technical designsrfc-decision-documentation.md- Documenting decisions and ADRsapi-design-rest.md- Deep dive on REST API best practicesdatabase-postgres-optimization.md- Postgres-specific performance tuning
Last Updated: 2025-10-25 Format Version: 1.0 (Atomic)