14 KiB
name, description
| name | description |
|---|---|
| testing-test-coverage-strategy | Determine what code needs testing vs what to skip |
Test Coverage Strategy
When to Use This Skill
Use this skill when you need to:
- Determine what code needs testing vs what to skip
- Interpret coverage metrics (line, branch, mutation)
- Set realistic coverage goals for your project
- Identify untested critical paths
- Implement mutation testing to verify test quality
- Balance coverage goals with development speed
- Audit existing test suites for gaps
ACTIVATE THIS SKILL: When planning test strategy, auditing coverage, or setting quality gates
Core Concepts
Coverage Metrics
Line Coverage: Percentage of lines executed by tests
def calculate_discount(price, discount_percent):
if discount_percent > 0: # Line executed
return price * (1 - discount_percent / 100) # Line executed
return price # Line NOT executed (100% branch coverage would require this)
# Test only with discount_percent > 0 → 67% line coverage
Branch Coverage: Percentage of decision branches taken
def validate_age(age):
if age < 0: # Branch 1: True path
return False # Covered
elif age < 18: # Branch 2: True path
return False # Covered
else: # Branch 3: Else path
return True # Covered
# Need 3 tests for 100% branch coverage:
# - age < 0
# - 0 <= age < 18
# - age >= 18
Function Coverage: Percentage of functions called
def function_a():
pass
def function_b():
pass
# Test calls function_a only → 50% function coverage
Path Coverage: Percentage of unique execution paths
def process(x, y):
if x > 0:
if y > 0:
return "both positive"
else:
return "x positive, y not"
else:
if y > 0:
return "y positive, x not"
else:
return "both not positive"
# 4 unique paths, need 4 tests for 100% path coverage
Coverage Goals by Project Type
Libraries (Public APIs):
- Target: 90-100% coverage
- Critical: All public APIs tested
- Reason: Used by many consumers, high stability need
Microservices (Business Logic):
- Target: 80-90% coverage
- Critical: Core business logic, error paths
- Reason: Balance between confidence and speed
Web Applications (Full Stack):
- Target: 70-80% coverage
- Critical: Business logic, API endpoints
- Reason: UI changes frequently, focus on backend
Prototypes/Experiments:
- Target: 0-50% coverage
- Critical: None
- Reason: Throwaway code, optimize for learning
What to Test vs Skip
ALWAYS TEST:
- Business logic (pricing, validation, calculations)
- Security-critical code (auth, permissions, encryption)
- Data transformations (parsing, serialization)
- Error handling (edge cases, failure modes)
- Public APIs (library interfaces)
- Bug fixes (regression prevention)
SKIP TESTING:
- Simple getters/setters (no logic)
- Framework/library code (already tested)
- Configuration files (static data)
- Generated code (build artifacts)
- Trivial delegation (one-line wrappers)
- UI layout code (use visual testing instead)
Patterns
Coverage-Guided Test Writing
Step 1: Run coverage report:
# Python
pytest --cov=myapp --cov-report=html tests/
# JavaScript
npm test -- --coverage
# Go
go test -cover ./...
# Rust
cargo tarpaulin --out Html
Step 2: Identify untested code:
# coverage.py report shows:
# myapp/payment.py 45% Lines 23-45, 67-89 not covered
# Look at payment.py:
def process_payment(amount, method):
if method == "credit_card":
return process_credit_card(amount) # Tested
elif method == "paypal":
return process_paypal(amount) # NOT TESTED (line 23-45)
elif method == "bitcoin":
return process_bitcoin(amount) # NOT TESTED (line 67-89)
else:
raise ValueError("Invalid method") # NOT TESTED
Step 3: Write tests for critical gaps:
def test_process_payment_paypal():
result = process_payment(100, "paypal")
assert result.status == "success"
def test_process_payment_bitcoin():
result = process_payment(100, "bitcoin")
assert result.status == "success"
def test_process_payment_invalid_method():
with pytest.raises(ValueError):
process_payment(100, "invalid")
Step 4: Re-run coverage → Improved to 95%
Mutation Testing
Concept: Introduce bugs (mutations) to verify tests catch them
# Original code
def is_adult(age):
return age >= 18
# Test
def test_is_adult():
assert is_adult(18) is True
assert is_adult(17) is False
# Mutation 1: Change >= to >
def is_adult(age):
return age > 18 # Mutant
# Run tests: PASSES (but shouldn't!)
# → Test is weak, doesn't catch boundary error
# Improved test
def test_is_adult_boundary():
assert is_adult(18) is True # Catches >= vs > mutation
assert is_adult(17) is False
assert is_adult(19) is True
Mutation Testing Tools:
# Python (mutmut)
pip install mutmut
mutmut run
mutmut results
# JavaScript (Stryker)
npm install --save-dev @stryker-mutator/core
npx stryker run
# Example output:
# Mutant 1: Changed >= to > → SURVIVED (test didn't catch it)
# Mutant 2: Changed >= to == → KILLED (test caught it)
# Mutation Score: 85% (good)
Critical Path Testing
Identify high-value code paths:
# E-commerce checkout flow
def checkout(cart, user, payment_method):
# CRITICAL: Calculate total (money involved)
total = calculate_total(cart)
# CRITICAL: Apply discounts (business logic)
total = apply_discounts(total, user.discount_codes)
# CRITICAL: Process payment (money exchange)
payment = process_payment(total, payment_method)
if not payment.success:
raise PaymentFailedError()
# CRITICAL: Create order (data integrity)
order = create_order(cart, user, total, payment.id)
# LESS CRITICAL: Send email (nice-to-have)
send_order_confirmation(user.email, order)
# LESS CRITICAL: Analytics (optional)
track_purchase(user.id, total)
return order
# Test priority:
# 1. calculate_total (business logic)
# 2. apply_discounts (business logic)
# 3. process_payment (integration test)
# 4. create_order (integration test)
# 5. send_order_confirmation (optional)
# 6. track_purchase (optional)
Incremental Coverage Improvement
Strategy: Improve coverage over time, focus on new code
# .coveragerc (Python)
[run]
branch = True
source = myapp
[report]
precision = 2
fail_under = 80 # CI fails if coverage drops below 80%
# Track coverage over time
# Week 1: 65%
# Week 2: 70% (added tests for user authentication)
# Week 3: 75% (added tests for payment processing)
# Week 4: 80% (reached goal)
Git hook to prevent coverage regression:
#!/bin/bash
# .git/hooks/pre-push
# Run tests with coverage
coverage run -m pytest
coverage report --fail-under=80
if [ $? -ne 0 ]; then
echo "❌ Coverage below 80%, push rejected"
exit 1
fi
echo "✅ Coverage check passed"
Coverage Exclusions
Explicitly exclude code from coverage:
# Exclude debug code
if DEBUG: # pragma: no cover
print(f"Debug: {variable}")
# Exclude abstract methods
class BaseRepository:
def find_by_id(self, id: int):
raise NotImplementedError() # pragma: no cover
# Exclude platform-specific code
if sys.platform == "win32": # pragma: no cover
import msvcrt
// TypeScript/JavaScript (Istanbul)
/* istanbul ignore next */
function debugLog(message: string) {
console.log(message);
}
/* istanbul ignore else */
if (process.env.NODE_ENV === 'development') {
console.log('Dev mode');
}
Examples by Language
Python (coverage.py)
# Install
pip install pytest-cov
# Run with coverage
pytest --cov=myapp --cov-report=term-missing tests/
# Generate HTML report
pytest --cov=myapp --cov-report=html tests/
# Open htmlcov/index.html
# Configuration (.coveragerc)
[run]
source = myapp
omit =
*/tests/*
*/migrations/*
*/__pycache__/*
[report]
precision = 2
exclude_lines =
pragma: no cover
def __repr__
raise AssertionError
raise NotImplementedError
if __name__ == .__main__.:
# CI integration (GitHub Actions)
- name: Test with coverage
run: |
pytest --cov=myapp --cov-report=xml tests/
- name: Upload coverage to Codecov
uses: codecov/codecov-action@v3
with:
files: ./coverage.xml
TypeScript/JavaScript (Istanbul/c8)
// package.json
{
"scripts": {
"test": "vitest",
"test:coverage": "vitest --coverage"
},
"devDependencies": {
"@vitest/coverage-v8": "^1.0.0"
}
}
// vitest.config.ts
export default defineConfig({
test: {
coverage: {
provider: 'v8',
reporter: ['text', 'html', 'lcov'],
exclude: [
'node_modules/',
'dist/',
'**/*.test.ts',
'**/*.config.ts',
],
lines: 80,
branches: 80,
functions: 80,
statements: 80,
},
},
});
# Run coverage
npm run test:coverage
# Output:
# File | % Stmts | % Branch | % Funcs | % Lines |
# -------------|---------|----------|---------|---------|
# user.ts | 85.71 | 75 | 100 | 85.71 |
# payment.ts | 66.67 | 50 | 75 | 66.67 |
# -------------|---------|----------|---------|---------|
# All files | 76.19 | 62.50 | 87.50 | 76.19 |
Go
// Run with coverage
go test -cover ./...
// Generate coverage profile
go test -coverprofile=coverage.out ./...
// View coverage in browser
go tool cover -html=coverage.out
// Coverage for specific package
go test -cover -coverprofile=coverage.out ./pkg/user
// Coverage threshold (CI)
#!/bin/bash
go test -coverprofile=coverage.out ./...
coverage=$(go tool cover -func=coverage.out | grep total | awk '{print $3}' | sed 's/%//')
threshold=80
if (( $(echo "$coverage < $threshold" | bc -l) )); then
echo "Coverage $coverage% is below threshold $threshold%"
exit 1
fi
Rust (tarpaulin)
# Install
cargo install cargo-tarpaulin
# Run coverage
cargo tarpaulin --out Html
# Output to terminal
cargo tarpaulin --out Stdout
# Configuration (Cargo.toml or tarpaulin.toml)
[tarpaulin]
ignore-tests = true
exclude-files = [
"src/generated/*",
"tests/*"
]
Mutation Testing Examples
Python (mutmut)
# Install
pip install mutmut
# Run mutation testing
mutmut run
# View results
mutmut results
# Show surviving mutants
mutmut show
# Example output:
# Mutant 1 (SURVIVED):
# --- src/calculator.py
# +++ mutant
# @@ -5,7 +5,7 @@
# def add(a, b):
# - return a + b
# + return a - b
# This survived because no test verifies add(2, 3) == 5
# Add test:
def test_add():
assert add(2, 3) == 5
assert add(0, 0) == 0
assert add(-1, 1) == 0
JavaScript (Stryker)
# Install
npm install --save-dev @stryker-mutator/core
# Initialize
npx stryker init
# Run
npx stryker run
# Example stryker.conf.json
{
"mutate": ["src/**/*.ts"],
"testRunner": "vitest",
"coverageAnalysis": "perTest",
"thresholds": {
"high": 80,
"low": 60,
"break": 60
}
}
# Output:
# Mutant killed | 45
# Mutant survived | 5
# Mutation score | 90%
Checklist
Setting Coverage Strategy:
- Define coverage goals by project type (library: 90%+, app: 70-80%)
- Identify critical paths to prioritize
- Decide what to exclude (generated code, debug code)
- Choose coverage metrics (line + branch minimum)
- Set up coverage reporting in CI/CD
Measuring Coverage:
- Run coverage report locally
- Identify untested lines/branches
- Prioritize gaps by business criticality
- Write tests for critical gaps
- Verify coverage improvement
Coverage Quality:
- Use mutation testing to verify test effectiveness
- Check branch coverage (not just line coverage)
- Review surviving mutants (weak tests)
- Test error paths and edge cases
- Avoid testing for coverage sake (test behavior)
CI/CD Integration:
- Fail builds if coverage drops below threshold
- Upload coverage reports (Codecov, Coveralls)
- Track coverage trends over time
- Require coverage for new code (diff coverage)
- Set realistic thresholds (don't aim for 100%)
Anti-Patterns
❌ NEVER: Aim for 100% coverage on everything
→ Waste time testing trivial code
❌ NEVER: Write tests just to increase coverage
→ Tests without assertions, useless tests
❌ NEVER: Only measure line coverage
→ Miss untested branches/paths
❌ NEVER: Ignore coverage in code review
→ Coverage regression over time
❌ NEVER: Test private implementation details
→ Brittle tests, false coverage confidence
❌ NEVER: Exclude all error handling from coverage
→ Untested failure modes
❌ NEVER: Use coverage as only quality metric
→ High coverage ≠ good tests
Related Skills
Foundation:
unit-testing-patterns.md- How to write testable codetest-driven-development.md- TDD workflow for coverage
Advanced:
mutation-testing.md- Advanced mutation testing patternsproperty-based-testing.md- Generate test cases automatically
Integration:
integration-testing.md- Coverage for integration testse2e-testing.md- Coverage for E2E tests
Tools:
- Python: coverage.py, mutmut
- TypeScript: c8, Stryker
- Go: Built-in coverage tool
- Rust: tarpaulin, cargo-mutants
Coverage Services:
- Codecov: Cross-language coverage tracking
- Coveralls: GitHub integration
- SonarQube: Comprehensive code quality