dev/docs/maintenance/upstream-diff.md

Upstream Tracking and Diff Pipeline

This document describes the process for tracking differences between the fork (code-rs) and upstream (codex-rs), identifying critical changes, and planning merges.

Overview

The fork maintains parallel crate trees:

• codex-rs/ - Upstream baseline (synced from https://github.com/openai/codex.git)
• code-rs/ - Fork implementation with enhancements

The tracking pipeline helps identify:

1. Structural differences - Which crates have diverged
2. Critical changes - Changes affecting prompts, API surface, or executor
3. Merge decisions - Historical log of what was adopted/adapted/preserved

Pipeline Scripts

All scripts are located in scripts/upstream-merge/:

1. diff-crates.sh - Structural Diff

Compares codex-rs vs code-rs on a per-crate basis.

Usage:

# Compare a single crate
./scripts/upstream-merge/diff-crates.sh core

# Compare all shared crates
./scripts/upstream-merge/diff-crates.sh --all

# Generate summary report
./scripts/upstream-merge/diff-crates.sh --summary

Output:

• Individual diff files: .github/auto/upstream-diffs/<crate>.diff
• Summary report: .github/auto/upstream-diffs/SUMMARY.md

What it tracks:

• Line-by-line differences between matching crates
• Added/removed lines count
• Files present in one tree but not the other

2. highlight-critical-changes.sh - Critical Change Detection

Analyzes diffs to highlight changes in sensitive subsystems.

Usage:

# Analyze a specific crate
./scripts/upstream-merge/highlight-critical-changes.sh core

# Analyze all crates
./scripts/upstream-merge/highlight-critical-changes.sh --all

Output:

• Per-crate critical changes: .github/auto/upstream-diffs/critical-changes/<crate>-critical.md
• Summary: .github/auto/upstream-diffs/critical-changes/CRITICAL-SUMMARY.md

What it tracks:

• Prompts: Changes to prompts/*.md, openai_tools.rs, agent_tool.rs
• API Surface: Changes to protocol.rs, app-server-protocol, mcp-types
• Executor: Changes to codex.rs, exec, apply-patch, acp.rs
• Config: Changes to config types and behavior toggles
• Function signature changes
• Enum/struct definition changes
• Protocol message changes (EventMsg, Op variants)

3. log-merge.sh - Merge Activity Logger

Tracks merge decisions and resolutions for future reference.

Usage:

# Start a new merge
./scripts/upstream-merge/log-merge.sh init upstream/main

# Add notes during merge
./scripts/upstream-merge/log-merge.sh note conflict "core/src/codex.rs has protocol changes"

# Log decisions
./scripts/upstream-merge/log-merge.sh decision core preserve "Keep fork's QueueUserInput support"
./scripts/upstream-merge/log-merge.sh decision exec adopt "Upstream exec improvements compatible"

# Finalize when done
./scripts/upstream-merge/log-merge.sh finalize

# View historical logs
./scripts/upstream-merge/log-merge.sh summary

Output:

• Timestamped log: docs/maintenance/upstream-merge-logs/merge-YYYYMMDD.md

What it tracks:

• Upstream ref and commit being merged
• Fork state at merge start
• Critical changes identified
• Decisions made (adopt/adapt/preserve) with rationale
• Post-merge checklist status

4. verify.sh - Post-Merge Verification

Validates that the fork still builds and maintains critical functionality after a merge.

Usage:

./scripts/upstream-merge/verify.sh

What it verifies:

• Build passes (build-fast.sh)
• Core API surface compiles
• Static guards (browser/agent tools still registered)
• Version handling intact
• Branding consistency

Monthly Cadence

We follow a two-phase cadence. Use the first Monday for lightweight scans; reserve the second Monday for merge planning if changes warrant action.

First Monday — Quick Diff

# Fetch latest upstream
git fetch upstream

# Check for new commits
git log HEAD..upstream/main --oneline

# Generate structural diff
./scripts/upstream-merge/diff-crates.sh --all

# Review high-level summary
cat .github/auto/upstream-diffs/SUMMARY.md

Goal: understand upstream churn. If nothing critical changed, stop here.

Second Monday — Merge Planning (Conditional)

# Highlight critical changes
./scripts/upstream-merge/highlight-critical-changes.sh --all

# Review critical summary
cat .github/auto/upstream-diffs/critical-changes/CRITICAL-SUMMARY.md

# Initialize merge log when planning an integration
./scripts/upstream-merge/log-merge.sh init upstream/main

Goal: categorize changes (adopt/adapt/preserve) and document decisions.

Typical Workflow

# 1. Initialize merge log
./scripts/upstream-merge/log-merge.sh init upstream/main

# 2. Review diffs
cat .github/auto/upstream-diffs/SUMMARY.md
cat .github/auto/upstream-diffs/critical-changes/CRITICAL-SUMMARY.md

# 3. For each critical crate, make decisions
#    Actions: adopt (take upstream), adapt (merge changes), preserve (keep fork)
./scripts/upstream-merge/log-merge.sh decision core adapt "Merge ACP support while preserving QueueUserInput"
./scripts/upstream-merge/log-merge.sh decision protocol adapt "Add new ACP variants, keep existing Op types"
./scripts/upstream-merge/log-merge.sh decision tui preserve "Fork-only Rust TUI, ignore upstream TS changes"

# 4. Apply changes crate-by-crate
#    (Manual or scripted - depends on complexity)

# 5. Verify after each crate
./scripts/upstream-merge/verify.sh

# 6. Finalize and document
./scripts/upstream-merge/log-merge.sh finalize
git add .
git commit -m "chore(codex-rs): sync with upstream main

See docs/maintenance/upstream-merge-logs/merge-YYYYMMDD.md"

Integration Strategies

Adopt (take upstream as-is):

• Upstream change is a strict improvement
• No fork-specific functionality conflicts
• Example: Performance improvements, bug fixes in non-critical paths

Adapt (merge both):

• Upstream adds new capability, fork has extensions
• Combine both sets of features
• Example: Upstream adds ACP events, fork keeps existing event types

Preserve (keep fork):

• Upstream removes/changes functionality fork depends on
• Fork behavior is intentionally different
• Example: Rust TUI vs TypeScript TUI, fork-specific tools

Critical Subsystems

Pay special attention to changes in:

Prompts and Tools

• prompts/*.md - System prompts
• codex-rs/core/src/openai_tools.rs - Tool definitions
• codex-rs/core/src/agent_tool.rs - Agent-specific tools

Changes here affect LLM behavior and available capabilities.

Protocol and API

• codex-rs/core/src/protocol.rs - Core event types
• codex-rs/app-server-protocol/ - Frontend communication
• codex-rs/mcp-types/ - MCP protocol definitions

Changes here affect client compatibility and event flow.

Executor

• codex-rs/core/src/codex.rs - Main execution loop
• codex-rs/exec/src/ - Command execution
• codex-rs/apply-patch/src/ - File modification
• codex-rs/core/src/acp.rs - Agent Client Protocol bridge

Changes here affect how tools execute and side effects are managed.

Configuration

• codex-rs/core/src/config.rs - Config loading
• codex-rs/core/src/config_types.rs - Config schema

Changes here affect default behavior and user-facing options.

Parity Tracking

After each merge, update docs/code-crate-parity-tracker.md:

• Note which crates moved closer to parity
• Update "Ready to delete code-*?" column
• Add notes about remaining divergence

Goal: Eventually delete code-rs/* crates that match upstream, consolidating to single implementation.

References

• Upstream ACP Integration: docs/upstream-acp-merge.md - Detailed notes on integrating Agent Client Protocol
• Parity Tracker: docs/code-crate-parity-tracker.md - Per-crate divergence status
• Merge Prompts: prompts/MERGE.md - Guidance for resolving conflicts
• Triage Prompts: prompts/TRIAGE.md - Categorizing upstream changes

Automation Opportunities

Future enhancements to pipeline:

• Auto-detect upstream releases via GitHub API
• Generate merge preview PRs automatically
• Classify changes by semantic impact (breaking/compatible/internal)
• Extract commit messages for changelog integration
• Cross-reference with fork's issue tracker

Troubleshooting

Diff shows everything changed:

• Check line endings (CRLF vs LF)
• Verify both trees are clean (git status)
• Ensure upstream remote is correctly configured

Critical changes not detected:

• Update patterns in highlight-critical-changes.sh
• Check that diff files exist in .github/auto/upstream-diffs/

Verify script fails:

• Review .github/auto/VERIFY.json for specific failures
• Check logs in .github/auto/VERIFY_*.log
• Ensure all fork-specific guards are still valid

Merge log not found:

• Logs are in docs/maintenance/upstream-merge-logs/
• Initialize with log-merge.sh init before logging notes/decisions

Maintenance

Update these scripts when:

• New critical files/patterns emerge (update CRITICAL_PATTERNS in highlight-critical-changes.sh)
• New crates are added to either tree (update SHARED_CRATES in diff-crates.sh)
• New verification guards needed (update verify.sh)
• Merge workflow evolves (update this doc and log-merge.sh)

---

Document created: 2025-10-05 Last updated: 2025-10-05