hanzoai/dev
2
0
Fork 0
mirror of https://github.com/hanzoai/dev.git synced 2026-10-17 15:34:54 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs: reset roadmap and add maintenance guides

Browse Source
This commit is contained in:
James Peter
2025-10-05 20:48:06 +10:00
parent f95ab56e7c
commit f2f0a07b38
15 changed files with 2997 additions and 153 deletions
Download Patch File Download Diff File Expand all files Collapse all files
DEAD_CODE_INVENTORY.md
+677
View File
@@ -0,0 +1,677 @@
# Dead Code Inventory and Deletion Plan for code-rs
**Generated:** 2025-10-05
**Scope:** code-rs/tui, code-rs/core, and related crates
**Purpose:** Identify and plan removal of dead compatibility scaffolding, orphaned modules, and unused code
---
## Executive Summary
This document provides a comprehensive inventory of dead or unused code in the code-rs codebase, organized by category with specific deletion plans and verification steps.
### Key Findings
- **~11,200 lines** of dead test code behind `legacy_tests` feature flag
- **6 orphaned modules** totaling ~72,700 bytes
- **10+ entire modules** marked with `#![allow(dead_code)]`
- **1 unused feature flag** (`vt100-tests` in code-rs/tui)
- **0 orphaned prompt files** (all are actively used)
- **100+ stale comments** about removed code
---
## Category 1: Feature Flags
### 1.1 legacy_tests Feature Flag (HIGH PRIORITY)
**Location:** `code-rs/tui/Cargo.toml:20`
**Status:** BROKEN - Does not compile (106+ errors)
**Scope:**
- 2 dedicated test files: 4,985 lines
- 23 modules with gated test code: ~6,247 lines
- Total: **~11,232 lines of dead code**
**Files Affected:**
```
code-rs/tui/src/chatwidget/tests.rs (4,634 lines)
code-rs/tui/src/chatwidget/tests_retry.rs (351 lines)
code-rs/tui/src/text_formatting.rs (~164 lines of tests)
code-rs/tui/src/text_processing.rs (~24 lines of tests)
code-rs/tui/src/updates.rs (~23 lines of tests)
code-rs/tui/src/status_indicator_widget.rs (~53 lines of tests)
code-rs/tui/src/pager_overlay.rs (~209 lines of tests)
code-rs/tui/src/markdown_renderer.rs (~75 lines of tests)
code-rs/tui/src/markdown_stream.rs (~499 lines of tests)
code-rs/tui/src/markdown.rs (~251 lines of tests)
code-rs/tui/src/live_wrap.rs (~81 lines of tests)
code-rs/tui/src/insert_history.rs (~80 lines of tests)
code-rs/tui/src/exec_command.rs (~17 lines of tests)
code-rs/tui/src/diff_render.rs (~155 lines of tests)
code-rs/tui/src/clipboard_paste.rs (~115 lines of tests)
code-rs/tui/src/user_approval_widget.rs (~138 lines of tests)
code-rs/tui/src/backtrack_helpers.rs (~41 lines of tests)
code-rs/tui/src/streaming/controller.rs (~180 lines of tests)
code-rs/tui/src/bottom_pane/textarea.rs (~914 lines of tests)
code-rs/tui/src/bottom_pane/mod.rs (~256 lines of tests)
code-rs/tui/src/bottom_pane/chat_composer.rs (~751 lines of tests)
code-rs/tui/src/bottom_pane/chat_composer_history.rs (~61 lines of tests)
code-rs/tui/src/bottom_pane/command_popup.rs (~319 lines of tests)
code-rs/tui/src/bottom_pane/scroll_state.rs (~28 lines of tests)
code-rs/tui/src/bottom_pane/approval_modal_view.rs (~37 lines of tests)
```
**Deletion Plan:**
1. **Delete entire test files:**
```bash
rm code-rs/tui/src/chatwidget/tests.rs
rm code-rs/tui/src/chatwidget/tests_retry.rs
```
2. **Remove gated test modules from each file:**
Search for `#[cfg(all(test, feature = "legacy_tests"))]` and delete the entire module block in each of the 23 files listed above.
3. **Remove feature flag from Cargo.toml:**
```bash
# Edit code-rs/tui/Cargo.toml, delete lines 19-20:
# # Disable legacy tests by default; enable with `--features legacy_tests` if needed.
# legacy_tests = []
```
4. **Remove test-only helper functions:**
- `simulate_stream_markdown_for_tests()` in `markdown_stream.rs`
- `CommandPopup::new()` test constructor
- `ChatWidget::test_for_request()` constructor
**Verification Steps:**
```bash
# 1. Verify feature is not in CI/CD
grep -r "legacy_tests" .github/ scripts/ justfile
# 2. Attempt build before deletion (should succeed)
cd code-rs/tui && cargo build
# 3. Perform deletion
# 4. Verify build still succeeds
cargo build
# 5. Verify tests run (non-legacy tests)
cargo test
# 6. Check for any remaining references
rg "legacy_tests" code-rs/
```
**Risk:** LOW - Code is already broken and unused
---
### 1.2 vt100-tests Feature Flag (MEDIUM PRIORITY)
**Location:** `code-rs/tui/Cargo.toml:16`
**Status:** UNUSED - No tests exist in code-rs/tui (tests exist in codex-rs/tui)
**Deletion Plan:**
```bash
# Remove feature flag definition
# Edit code-rs/tui/Cargo.toml, delete line 16:
# vt100-tests = []
```
**Note:** Keep `vt100` dependency in main dependencies (line 104) as it's used in production code (`chatwidget/terminal.rs`).
**Verification Steps:**
```bash
# 1. Verify no tests use this feature
find code-rs/tui/tests -name "*.rs" 2>/dev/null
# 2. Check if vt100 is used in production
rg "use vt100" code-rs/tui/src/
# 3. Remove feature flag
# 4. Verify build
cargo build -p code-tui
```
**Risk:** LOW - Feature has no tests
---
## Category 2: Orphaned Modules and Files
### 2.1 Confirmed Orphaned Files (code-rs/tui)
#### 2.1.1 text_block.rs
**Location:** `code-rs/tui/src/text_block.rs`
**Size:** 278 bytes
**Status:** Commented out in lib.rs:73 with note "Orphaned after trait-based HistoryCell migration"
**Deletion Plan:**
```bash
rm code-rs/tui/src/text_block.rs
# Also remove commented line from code-rs/tui/src/lib.rs:73
```
**Verification:**
```bash
rg "text_block" code-rs/tui/
cargo build -p code-tui
```
**Risk:** NONE - Already acknowledged as orphaned
---
#### 2.1.2 scroll_view.rs
**Location:** `code-rs/tui/src/scroll_view.rs`
**Size:** 4,529 bytes
**Status:** Commented out in lib.rs:63 with note "Orphaned after trait-based HistoryCell migration"
**Deletion Plan:**
```bash
rm code-rs/tui/src/scroll_view.rs
# Also remove commented line from code-rs/tui/src/lib.rs:63
```
**Verification:**
```bash
rg "scroll_view" code-rs/tui/
cargo build -p code-tui
```
**Risk:** NONE - Already acknowledged as orphaned
---
#### 2.1.3 app_backtrack.rs
**Location:** `code-rs/tui/src/app_backtrack.rs`
**Size:** 14,257 bytes
**Status:** Not declared in any mod.rs/lib.rs
**Deletion Plan:**
```bash
rm code-rs/tui/src/app_backtrack.rs
```
**Verification:**
```bash
# Check for any references to BacktrackState
rg "BacktrackState" code-rs/tui/
rg "app_backtrack" code-rs/tui/
cargo build -p code-tui
```
**Risk:** LOW - No module declaration found
---
#### 2.1.4 resume_picker.rs
**Location:** `code-rs/tui/src/resume_picker.rs`
**Size:** 37,218 bytes
**Status:** Not declared in module tree (only `mod resume` exists)
**Note:** Depends on `custom_terminal.rs` which is also orphaned
**Deletion Plan:**
```bash
rm code-rs/tui/src/resume_picker.rs
```
**Verification:**
```bash
rg "resume_picker" code-rs/tui/
rg "PickerState" code-rs/tui/
cargo build -p code-tui
```
**Risk:** LOW - Not integrated into module tree
---
#### 2.1.5 custom_terminal.rs
**Location:** `code-rs/tui/src/custom_terminal.rs`
**Size:** 22,672 bytes
**Status:** USED by tests but not declared as module
**Note:** This is referenced by `resume_picker.rs` and `chatwidget/tests.rs` (which is behind legacy_tests)
**Deletion Plan:**
**Option A:** Delete if legacy_tests are removed:
```bash
rm code-rs/tui/src/custom_terminal.rs
```
**Option B:** Add module declaration if needed:
```rust
// In code-rs/tui/src/lib.rs, add:
#[cfg(test)]
mod custom_terminal;
```
**Verification:**
```bash
rg "custom_terminal" code-rs/tui/
cargo build -p code-tui
cargo test -p code-tui
```
**Risk:** MEDIUM - Used by test code, but those tests are broken
**Recommendation:** Delete along with legacy_tests removal
---
#### 2.1.6 exec_cell/ directory
**Location:** `code-rs/tui/src/exec_cell/`
**Contents:** `render.rs` (16,444 bytes)
**Status:** Orphaned - references non-existent `super::model::ExecCell`
**Note:** The actual ExecCell is in `history_cell/exec.rs`, this appears to be old/duplicate code
**Deletion Plan:**
```bash
rm -rf code-rs/tui/src/exec_cell/
```
**Verification:**
```bash
rg "exec_cell" code-rs/tui/
rg "crate::exec_cell" code-rs/tui/
cargo build -p code-tui
```
**Risk:** LOW - No references found
---
### 2.2 Module Declaration Issues (code-rs/core)
#### 2.2.1 tasks/ directory
**Location:** `code-rs/core/src/tasks/`
**Contents:** `mod.rs`, `compact.rs`, `regular.rs`, `review.rs`
**Status:** USED but not declared as module
**Issue:** Code in `state/turn.rs` references `use crate::tasks::SessionTask;` but `mod tasks;` is missing from lib.rs
**Action Required:** ADD MODULE DECLARATION (not removal)
```rust
// In code-rs/core/src/lib.rs, add:
pub mod tasks;
```
**Verification:**
```bash
cargo build -p code-core
cargo test -p code-core
```
**Risk:** NONE - This fixes a bug
---
#### 2.2.2 tests/suite/otel.rs
**Location:** `code-rs/core/tests/suite/otel.rs`
**Status:** Test file not declared in `tests/suite/mod.rs`
**Action Required:** Either add to mod.rs or delete
**Option A:** Add to suite:
```rust
// In code-rs/core/tests/suite/mod.rs, add:
mod otel;
```
**Option B:** Delete if unused:
```bash
rm code-rs/core/tests/suite/otel.rs
```
**Verification:**
```bash
cargo test -p code-core --test suite
```
---
## Category 3: Modules Marked #[allow(dead_code)]
These entire modules are marked with `#![allow(dead_code)]` at the file level, indicating they may be completely unused:
### 3.1 High Priority Candidates (code-rs/tui)
| File | Size | Status |
|------|------|--------|
| `src/pager_overlay.rs` | 25,058 B | `#![allow(dead_code, unused_imports, unused_variables)]` |
| `src/streaming/controller.rs` | 18,450 B | `#![allow(dead_code)]` |
| `src/streaming/mod.rs` | 3,952 B | `#![allow(dead_code)]` |
| `src/markdown_stream.rs` | 41,063 B | `#![allow(dead_code)]` |
| `src/markdown.rs` | 28,666 B | `#![allow(dead_code)]` |
| `src/transcript_app.rs` | 9,904 B | `#![allow(dead_code)]` |
| `src/backtrack_helpers.rs` | 4,919 B | `#![allow(dead_code)]` |
| `src/bottom_pane/list_selection_view.rs` | ? | `#![allow(dead_code)]` |
| `src/bottom_pane/paste_burst.rs` | ? | `#![allow(dead_code, unused_imports, unused_variables)]` |
**Total:** ~131,000+ bytes
### 3.2 Core Modules (code-rs/core)
| File | Status |
|------|--------|
| `src/acp.rs` | `#![allow(dead_code)]` |
| `src/function_tool.rs` | `#[allow(dead_code)]` on module |
**Investigation Plan:**
For each file:
1. **Check for external references:**
```bash
# Example for pager_overlay.rs
rg "pager_overlay" code-rs/ --type rust | grep -v "src/pager_overlay.rs"
rg "PagerOverlay" code-rs/ --type rust | grep -v "src/pager_overlay.rs"
```
2. **Try compilation without file:**
```bash
# Temporarily rename file
mv src/pager_overlay.rs src/pager_overlay.rs.bak
cargo build -p code-tui
# If build succeeds, the file is safe to delete
```
3. **Check git history:**
```bash
git log --oneline --follow src/pager_overlay.rs | head -20
```
**Verification Process:**
```bash
# For each potentially dead module:
# 1. Search for usage
rg "ModuleName" code-rs/
# 2. Comment out module declaration in lib.rs
# (or rename file to .bak)
# 3. Build
cargo build
# 4. Test
cargo test
# 5. If successful, proceed with deletion
```
**Risk:** MEDIUM - Requires individual verification of each module
---
## Category 4: Stale Comments
### 4.1 Comments About Removed Code (Safe to Delete)
These are comments documenting code that has already been removed:
**code-rs/core/src/codex.rs:**
- Line 9010: `// removed upstream exit_review_mode helper: not used in fork`
- Lines 2539, 2542, 2554, 2556, 2564: `// (debug removed)`
- Line 2884: `// (submission diagnostics removed)`
**code-rs/core/src/client.rs:**
- Line 645: `// duplicate of earlier helpers removed during merge cleanup`
**code-rs/tui/src/chatwidget.rs:**
- Line 1235: `// Removed legacy turn-window logic; ordering is strictly global.`
- Line 2674: `// Legacy helper removed: streaming now requires explicit sequence numbers.`
- Line 10905: `// Legacy show_agents_settings_ui removed — overview/Direct editors replace it`
- Line 15695: `// removed legacy ensure_stream_order_key; strict variant is used instead`
**code-rs/tui/src/chatwidget/exec_tools.rs:**
- Line 1023: `// Stable ordering now inserts at the correct position; these helpers are removed.`
**Deletion Plan:**
```bash
# Manually remove these comments from the files listed above
# This is a code cleanup task with no functional impact
```
**Verification:** Visual inspection + `cargo build`
**Risk:** NONE
---
### 4.2 Comments to Keep (Important Documentation)
These comments document architectural decisions and should be retained:
- `code-rs/core/src/codex.rs:3843` - Documents intentional review flow divergence
- `code-rs/core/src/codex.rs:3545` - Documents upstream protocol changes
- `code-rs/core/src/client.rs:1212-1214` - Documents where to find removed helpers if needed
- `code-rs/core/src/exec.rs:35` - Documents architectural change
- All "legacy" config compatibility comments in `config_types.rs`
---
## Category 5: Documentation Files
### 5.1 Planning/Design Documents (Keep or Archive)
**Location:** `code-rs/tui/agent_tasks/`
**Files:**
- `plain_loading_wait_migration.md`
- `renderer_cache.md`
- `stream_exec_assistant_migration.md`
**Status:** Not referenced in code, appear to be design documents
**Recommendation:**
**Option A:** Archive to docs/archive/ directory:
```bash
mkdir -p docs/archive/tui-migrations
mv code-rs/tui/agent_tasks/*.md docs/archive/tui-migrations/
rmdir code-rs/tui/agent_tasks
```
**Option B:** Keep as-is if they contain useful migration context
**Option C:** Delete if outdated:
```bash
rm -rf code-rs/tui/agent_tasks/
```
**Verification:** Review content to determine historical value
**Risk:** LOW - Documentation only
---
### 5.2 Active Documentation (Keep)
**code-rs/tui/HISTORY_CELLS_PLAN.md** - Active planning doc, keep
**code-rs/tui/styles.md** - Developer reference, keep (or move to docs/)
---
## Category 6: Prompt Files
### Status: ALL ACTIVE
No orphaned prompt files found. All prompt files have confirmed `include_str!()` usage:
- ✅ `code-rs/core/gpt_5_code_prompt.md` - Used by gpt-5-codex models
- ✅ `code-rs/core/prompt.md` - Base instructions for all models
- ✅ `code-rs/core/prompt_coder.md` - Additional developer instructions
- ✅ `code-rs/core/review_prompt.md` - Review mode functionality
- ✅ `code-rs/tui/prompt_for_init_command.md` - /init slash command
- ✅ `code-rs/core/templates/compact/prompt.md` - Context compaction
- ✅ `code-rs/core/templates/compact/history_bridge.md` - Context compaction
**Action:** None required
---
## Deletion Priority Matrix
### P0 - Immediate (Zero Risk)
1. Delete `text_block.rs` + comment in lib.rs
2. Delete `scroll_view.rs` + comment in lib.rs
3. Delete stale comments in codex.rs, client.rs, chatwidget.rs
4. Fix module declaration for `tasks/` directory
**Estimated cleanup:** ~5,000 bytes + improved code clarity
---
### P1 - High Priority (Low Risk)
1. Remove `legacy_tests` feature flag and all gated code (~11,232 lines)
2. Delete `app_backtrack.rs`
3. Delete `resume_picker.rs`
4. Delete `custom_terminal.rs`
5. Delete `exec_cell/` directory
6. Remove `vt100-tests` feature flag from code-rs/tui
**Estimated cleanup:** ~11,300 lines + ~90KB
---
### P2 - Medium Priority (Requires Investigation)
1. Investigate and remove modules marked `#![allow(dead_code)]`
- Start with `pager_overlay.rs`, `transcript_app.rs`, `backtrack_helpers.rs`
- Then `streaming/` directory modules
- Then `markdown.rs` and `markdown_stream.rs`
2. Fix or delete `tests/suite/otel.rs`
3. Archive or delete `agent_tasks/` directory
**Estimated cleanup:** ~131KB
---
### P3 - Low Priority (Documentation)
1. Review and archive old planning docs in `agent_tasks/`
2. Consider moving `styles.md` to docs/ directory
**Estimated cleanup:** Organizational only
---
## Verification Checklist
After each deletion phase:
```bash
# 1. Build succeeds
cargo build --workspace
# 2. Tests pass (non-legacy)
cargo test --workspace
# 3. Clippy is happy
cargo clippy --workspace -- -D warnings
# 4. Format is correct
cargo fmt --check
# 5. No new warnings
cargo build --workspace 2>&1 | grep -i warning
# 6. Git status is clean (except intended changes)
git status
```
---
## Rollback Plan
Before any deletions:
```bash
# Create a backup branch
git checkout -b backup-before-dead-code-cleanup
git push origin backup-before-dead-code-cleanup
# Create working branch
git checkout -b cleanup-dead-code-phase-1
# Perform deletions with individual commits
git commit -m "Remove legacy_tests feature and gated code"
git commit -m "Remove orphaned modules: text_block, scroll_view"
# etc.
# If issues arise, revert specific commits or reset to backup branch
```
---
## Success Metrics
- **Lines of code removed:** Target ~11,500+ lines
- **Files removed:** Target ~8-10 files
- **Build time improvement:** Measure before/after compilation time
- **Code complexity reduction:** Fewer modules to maintain
- **Clarity improvement:** No more `#![allow(dead_code)]` suppressions
---
## Related Issues
During this investigation, the following potential issues were identified:
1. **Missing module declarations:**
- `code-rs/core/src/tasks/` is used but not declared
- `code-rs/tui/src/custom_terminal.rs` is used in tests but not declared
2. **Broken test infrastructure:**
- Unit tests in code-rs/tui have 11 compilation errors even without legacy_tests
- May need separate cleanup effort for test infrastructure
3. **Upstream sync opportunity:**
- codex-rs/tui has working vt100-tests
- Consider whether to port these tests or maintain divergence
---
## Notes for Future Maintenance
1. **Avoid `#![allow(dead_code)]` at file level** - This suppresses important warnings and makes it hard to identify truly dead code. Use at item level only when justified.
2. **Remove code instead of commenting it out** - Git preserves history; commented-out module declarations (like `text_block.rs` in lib.rs) create confusion.
3. **Clean up stale comments** - Comments about "removed" code should be deleted along with the code.
4. **Keep feature flags clean** - Remove unused feature flags promptly to avoid confusion.
5. **Test infrastructure hygiene** - If tests don't compile, either fix them or delete them. Don't hide behind feature flags.
---
**End of Document**
INVESTIGATION_SUMMARY.md
+116
View File
@@ -0,0 +1,116 @@
# MCP Client Upstream Reuse Investigation Summary
**Branch**: `code-claude-investigate-reusing-codex-rs-mcp-client`
**Date**: 2025-10-05
**Investigator**: Claude Code Agent
## Key Findings
### **RECOMMENDATION: Reuse upstream codex-rs MCP components directly** ✅
The investigation reveals **minimal divergence** between code-rs and codex-rs implementations:
#### 1. MCP Client (`mcp-client`)
- **Only difference**: 1-line buffer size change (1MB vs default)
- **File similarity**: 476 vs 475 lines (99.8% identical)
- **Action**: Test if upstream buffer is sufficient; if not, contribute config option upstream
#### 2. Responses API Proxy (`responses-api-proxy`)
- **Difference**: ZERO - byte-for-byte identical logic
- **Only variation**: Package/binary naming (`code-*` vs `codex-*`)
- **Action**: Use upstream with build-time binary rename OR accept upstream naming
#### 3. Process Hardening (`process-hardening`)
- **Difference**: Cosmetic code style only (functionally identical)
- **Action**: Use upstream directly via re-export
## Implementation Strategy
**Phase 1 (Immediate)**:
- Switch to `codex-mcp-client` dependency
- Test with default buffer size
- Use `codex-process-hardening` via re-export
**Phase 2 (1-2 weeks)**:
- Propose upstream PR for buffer configuration (if needed)
- Migrate responses-api-proxy with binary rename
- Establish automated upstream monitoring
**Phase 3 (Ongoing)**:
- Weekly upstream change checks
- Participate in upstream development
- Maintain zero-patch strategy
## Detailed Documentation
1. **[upstream-mcp-reuse-strategy.md](docs/upstream-mcp-reuse-strategy.md)**
- Comprehensive analysis of all differences
- Three proposed strategies (A: Direct Dependency, B: Feature-Flag, C: Automated Sync)
- Risk assessment and mitigation
- Maintenance checklist
2. **[upstream-mcp-implementation-guide.md](docs/upstream-mcp-implementation-guide.md)**
- Step-by-step migration instructions
- Code examples for wrapper/config approach
- Testing and validation procedures
- Upstream contribution workflow
- Rollback plan
## Fork-Specific Tweaks Required
| Component | Tweak | Complexity | Mitigation |
|-----------|-------|------------|------------|
| Buffer size | 1MB vs default | Trivial | Config option (upstream PR) or wrapper |
| Binary names | `code-*` vs `codex-*` | Low | Build script rename OR accept upstream |
| Process hardening | Re-export naming | Trivial | `pub use codex_process_hardening as code_process_hardening` |
## Upstream Parity Maintenance
**Weekly**:
- Check codex-rs commits to MCP components
- Run integration tests with latest upstream
- Review upstream issues/PRs
**Per-Release**:
- Review changelog for breaking changes
- Full test suite validation
- Update documentation if needed
**Monthly**:
- Performance benchmarks
- Security audit review
- Evaluate sunset of workarounds
## Risk Assessment: LOW ✅
| Risk | Likelihood | Impact | Status |
|------|------------|--------|--------|
| API breakage | Low | High | Mitigated: Pin versions, CI tests |
| Buffer regression | Medium | Medium | Mitigated: Integration tests |
| Upstream abandonment | Very Low | High | Fallback: Current fork exists |
## Conclusion
**The fork is unnecessary.** Code-rs can safely depend on upstream codex-rs MCP components with:
- Zero maintenance burden for core functionality
- Automatic security updates
- Trivial customization via wrapper or upstream contribution
**Recommended Action**: Implement Strategy A (Direct Upstream Dependency) immediately.
---
## Files Modified/Created
- `code-rs/docs/upstream-mcp-reuse-strategy.md` - Comprehensive analysis
- `code-rs/docs/upstream-mcp-implementation-guide.md` - Implementation guide
- `INVESTIGATION_SUMMARY.md` - This summary
## Next Steps
1. Review documents with team
2. Decide on binary naming convention
3. Create test branch with upstream dependencies
4. Run full test suite
5. Benchmark performance with actual workloads
6. If successful, merge and close fork divergence
ROADMAP.md
+76
View File
@@ -0,0 +1,76 @@
# Roadmap
## Objectives
- Protect our ability to absorb upstream changes quickly while letting `code-rs` own the fork-only UX, policy, and tooling layers.
- Remove legacy compatibility scaffolding (dead modules, feature flags, tests, assets) so the tree matches the product we actually ship.
- Rebuild the test suite around lightweight smoke/targeted coverage that reflects `code-rs` behavior today instead of the old codex execution model.
- Institutionalize upstream tracking, reuse decisions, and maintenance checklists so each merge is predictable.
- Standardize parallel-agent workflows (one worktree per track) to keep large refactors moving without blocking on long-running investigations.
## Upstream Reuse Strategy
| Area | Current Delta | Strategy | Immediate Actions |
| --- | --- | --- | --- |
| `mcp-client` | Rename + larger stdout buffer. | Adopt upstream crate directly, keep fork-only buffer change via wrapper if still required. | See `INVESTIGATION_SUMMARY.md` for options; test whether the 1 MB buffer is needed, then re-export upstream `codex-mcp-client` with a thin CLI shim if so. |
| `responses-api-proxy` | Rename + larger API key buffer + extra test. | Consume upstream crate, keep buffer tweak behind feature or downstream patch. | Follow `docs/upstream-mcp-reuse-strategy.md` to rebase local buffer changes into a minimal patch and depend on upstream. |
| `exec` | Substantial policy/output changes. | Build adapters for upstream executor where possible; retain fork-specific approval pipeline. | Catalogue fork-only code paths (policy prompts, streaming) and design adapter boundaries in `code-core`. |
| `login` | Auth flow tweaks, HTML assets, approval prompts. | Keep forked implementation, but cross-check with upstream helpers for OAuth updates. | Diff upstream releases regularly; cherry-pick auth fixes while preserving fork UX. |
| `protocol` | Added MCP protocol types. | Maintain fork version; push reusable schema changes upstream if possible. | Document any upstream schema additions needed for fork behavior. |
| `core`, `mcp-server`, `tui`, `app-server` | Massive divergence (fork-only features). | Treat as fork-owned; only pull upstream when changes unblock us. | Record upstream changes that impact prompts/executor and implement targeted integrations. |
## Codebase Cleanup Plan
Supporting detail: `DEAD_CODE_INVENTORY.md`
1. **Remove stale compat shims**
- Delete `code-rs/tui/src/compat.rs`, `foundation.rs`, and other adapter modules introduced for abandoned compatibility work.
- Strip feature flags such as `code-fork` and `legacy_tests` where they no longer gate real behavior.
2. **Prune unused modules and assets**
- Audit `code-rs/tui` for modules ported from upstream but unused (e.g., status widgets, legacy assets).
- Review `code-rs/core` for upstream executor skeletons left over from earlier merges (`tools/`, `executor/`, etc.) and remove or quarantine them.
3. **Consolidate prompts and config**
- Keep only fork-specific prompt files (e.g., `prompt_coder.md`); archive or delete upstream variants that are no longer consumed.
4. **Simplify feature flags and cfgs**
- Replace `cfg(feature = "legacy_tests")` blocks with explicit test modules or drop them entirely.
- Ensure crate names and binaries use the `code-*` prefix consistently.
## Test Suite Reset
Supporting detail: `TEST_SUITE_RESET.md`
1. **Retire legacy suites**
- Remove `legacy_tests` feature and delete unit tests that target the old codex execution flow.
- Drop vt100 fixtures and binary-size snapshots that no longer reflect current UI behavior.
2. **Seed minimal, reliable coverage**
- Keep smoke-style tests (e.g., new `tests/ui_smoke.rs`) that exercise critical flows without upstream dependencies.
- Add targeted executor and MCP tests that validate current fork behavior.
3. **Create new test scaffolding**
- Introduce helper modules for building `ChatWidget`, `App` instances, and executor contexts tailored to `code-rs`.
- Document how to add new tests; enforce `./build-fast.sh --workspace code` as the only required gate.
## Upstream Tracking Process
- Use the scripts under `scripts/upstream-merge/` (`diff-crates.sh`, `highlight-critical-changes.sh`, `log-merge.sh`) as described in `docs/maintenance/upstream-diff.md`.
- Monitor `codex-rs/core/prompt.md`, executor/plan tool updates, and API schema changes during each upstream merge.
- Maintain a CHANGELOG section summarizing upstream features adopted vs intentionally skipped.
- For reusable crates we adopt, pin the upstream commit hash and bump deliberately (documenting any fork-specific patches).
## Execution Phases (No Dates)
1. **Baseline & Tooling**
- Revert or delete unused compatibility modules and features (see `DEAD_CODE_INVENTORY.md`).
- Install upstream diff scripts; verify they run and produce baseline reports (`scripts/upstream-merge/`).
- Establish parallel worktree workflow (one agent per track, summarized back into ROADMAP).
2. **Crate Adoption Pass**
- Trial direct dependencies on `codex-mcp-client` / `codex-responses-api-proxy` (guided by `INVESTIGATION_SUMMARY.md`).
- Document wrapper decisions and pin versions in `UPSTREAM_PARITY_CHECKLIST.md`.
3. **Fork Cleanup Pass**
- Execute P0/P1 deletions from `DEAD_CODE_INVENTORY.md` (feature flags, orphan modules, stale comments).
- Archive or delete outdated planning docs that referenced abandoned compat efforts.
4. **Test Suite Rebuild**
- Remove legacy suites per `TEST_SUITE_RESET.md`; preserve only smoke/critical tests.
- Add new helper modules so future tests can be written without reintroducing upstream assumptions.
5. **Upstream Tracking & Maintenance**
- For each upstream merge, run diff scripts, log decisions with `log-merge.sh`, and update ROADMAP sections.
- Capture learnings from each parallel-agent run (what worked, what stalled) so future phases stay efficient.
## Deliverables
- Clean `code-rs` tree with only fork-relevant modules.
- Minimal but working test suite aligned with current behavior.
- Clearly documented integration points for upstream crates and prompts.
- ROADMAP updates as tasks complete; old planning documents removed.
TEST_SUITE_RESET.md
+314
View File
@@ -0,0 +1,314 @@
# Test Suite Reset Plan for code-rs
## Executive Summary
This document outlines the plan to reset the code-rs test infrastructure by removing the legacy integration test suite and establishing a minimal, maintainable testing baseline. The primary goal is to reduce CI time and maintenance burden while ensuring `./build-fast.sh --workspace code` remains the sole required gate.
## Current State Analysis
### Test Infrastructure Overview
- **Total test files**: ~99 Rust test files across 13 test directories
- **Major test suites**:
- `core/tests/`: 32 files (largest suite with ~18k LOC total)
- Integration tests for CLI streaming, compact/resume/fork, prompt caching, OpenTelemetry, MCP client
- Largest individual files: `client.rs` (1489 LOC), `otel.rs` (1196 LOC), `compact.rs` (1035 LOC)
- `app-server/tests/`: 19 files
- Message processor flows, auth, conversation management, fuzzy search
- `exec/tests/`: 10 files
- Event processor, sandbox, resume, output schema
- `execpolicy/tests/`: 11 files
- Command parsing tests for various shell commands
- Smaller suites: `mcp-server` (7 files), `login` (4 files), `apply-patch` (3 files), `chatgpt` (3 files), etc.
### TUI Legacy Tests (Feature-Gated)
The `tui` crate contains **27 inline test modules** gated behind `#[cfg(all(test, feature = "legacy_tests"))]`:
- Text formatting, markdown rendering, live wrapping
- Chat composer, textarea, command popup
- Status indicators, pager overlay, clipboard handling
- Backtrack helpers, user approval widgets
- VT100-based replay tests (`chatwidget_stream_tests`)
These tests are **disabled by default** (feature `legacy_tests = []` in `tui/Cargo.toml` line 20).
### Test Helper Infrastructure
Three shared test support crates exist:
1. **`core_test_support`** (`core/tests/common/`): SSE fixture loading, default config setup, event waiting utilities
2. **`app_test_support`** (`app-server/tests/common/`): MCP process management, mock model server, SSE response helpers
3. **`mcp_test_support`** (`mcp-server/tests/common/`): Similar to app_test_support for MCP server tests
All three are workspace dependencies (defined in root `Cargo.toml`).
### CI Gating Status
- **Current gate**: `./build-fast.sh --workspace code` (confirmed in `.github/workflows/`)
- Used in `issue-triage.yml` line 188
- Used in `upstream-merge.yml` lines 351, 676-712
- Used in `issue-code.yml` lines 170, 185, 251
- **No `cargo test` invocations** found in any workflow file
- Build-only verification is **already enforced** as the sole requirement
---
## Removal Plan
### Phase 1: Delete Legacy Integration Tests
**Target for removal** (complete deletion recommended):
#### Core Tests (`code-rs/core/tests/`)
- **Delete entire directory**: `code-rs/core/tests/`
- Files to remove (32 total):
- `all.rs`, `api_surface.rs`, `chat_completions_payload.rs`, `chat_completions_sse.rs`, `mcp_manager.rs`
- `suite/`: All 20 test modules including `client.rs`, `compact.rs`, `compact_resume_fork.rs`, `cli_stream.rs`, `exec.rs`, `exec_stream_events.rs`, `fork_conversation.rs`, `json_result.rs`, `live_cli.rs`, `model_overrides.rs`, `otel.rs`, `prompt_caching.rs`, `review.rs`, `rmcp_client.rs`, `rollout_list_find.rs`, `rollout_resume.rs`, `seatbelt.rs`, `stream_error_allows_next_turn.rs`, `stream_no_completed.rs`, `stream_order.rs`, `user_notification.rs`, `abort_tasks.rs`
- `common/`: Test fixtures and helpers (`lib.rs`, `responses.rs`, `test_codex.rs`, `test_codex_exec.rs`)
#### App Server Tests (`code-rs/app-server/tests/`)
- **Delete entire directory**: `code-rs/app-server/tests/`
- Files to remove (19 total):
- `all.rs`
- `suite/`: `archive_conversation.rs`, `auth.rs`, `code_message_processor_flow.rs`, `config.rs`, `create_conversation.rs`, `fuzzy_file_search.rs`, `interrupt.rs`, `list_resume.rs`, `login.rs`, `send_message.rs`, `set_default_model.rs`, `user_agent.rs`, `user_info.rs`
- `common/`: `lib.rs`, `mcp_process.rs`, `mock_model_server.rs`, `responses.rs`
#### Exec Tests (`code-rs/exec/tests/`)
- **Delete entire directory**: `code-rs/exec/tests/`
- Files to remove (10 total):
- `all.rs`, `event_processor_with_json_output.rs`
- `suite/`: `apply_patch.rs`, `auth_env.rs`, `common.rs`, `mod.rs`, `output_schema.rs`, `resume.rs`, `sandbox.rs`, `server_error_exit.rs`
#### MCP Server Tests (`code-rs/mcp-server/tests/`)
- **Delete entire directory**: `code-rs/mcp-server/tests/`
- Files to remove (7 total):
- `all.rs`
- `suite/`: `codex_tool.rs`, `mod.rs`
- `common/`: `lib.rs`, `mcp_process.rs`, `mock_model_server.rs`, `responses.rs`
#### Login Tests (`code-rs/login/tests/`)
- **Delete entire directory**: `code-rs/login/tests/`
- Files to remove (4 total): `all.rs`, `suite/device_code_login.rs`, `suite/login_server_e2e.rs`, `suite/mod.rs`
#### ChatGPT Tests (`code-rs/chatgpt/tests/`)
- **Delete entire directory**: `code-rs/chatgpt/tests/`
- Files to remove (3 total): `all.rs`, `suite/apply_command_e2e.rs`, `suite/mod.rs`
#### Apply Patch Tests (`code-rs/apply-patch/tests/`)
- **Delete entire directory**: `code-rs/apply-patch/tests/`
- Files to remove (3 total): `all.rs`, `suite/cli.rs`, `suite/mod.rs`
#### ExecPolicy Tests (`code-rs/execpolicy/tests/`)
- **Keep minimal**: These are lightweight unit tests for command parsing logic
- **Action**: Review and potentially keep 2-3 critical tests (e.g., `literal.rs`, `sed.rs`) as examples
- Consider consolidating into inline `#[cfg(test)]` modules instead
#### Other Test Directories (Evaluate Case-by-Case)
- **`cli/tests/`** (2 files: `mcp_add_remove.rs`, `mcp_list.rs`): **DELETE** - MCP config tests
- **`mcp-types/tests/`** (4 files): **KEEP** - Protocol serialization tests are valuable for stability
- **`linux-sandbox/tests/`** (3 files): **KEEP** - Security-critical landlock tests
- **`cloud-tasks/tests/`** (1 file: `env_filter.rs`): **KEEP** - Minimal, focused test
### Phase 2: Remove TUI Legacy Tests
- **Action**: Remove `legacy_tests` feature entirely from `code-rs/tui/Cargo.toml`
- Delete all `#[cfg(all(test, feature = "legacy_tests"))]` test modules from `tui/src/`:
- 27 test module blocks across files like `text_formatting.rs`, `markdown_renderer.rs`, `live_wrap.rs`, `chatwidget.rs`, etc.
- Special file: `chatwidget_stream_tests.rs` (VT100 replay tests in `lib.rs` line 98)
- Test-only helper methods (e.g., `chatwidget.rs` line 911: `pub(crate) fn test_for_request`)
- Also remove `vt100-tests` feature if no longer needed (line 16 in `tui/Cargo.toml`)
### Phase 3: Clean Up Test Dependencies
Remove from `code-rs/Cargo.toml` workspace dependencies:
- `app_test_support = { path = "app-server/tests/common" }` (line 48)
- `core_test_support = { path = "core/tests/common" }` (line 76)
- `mcp_test_support = { path = "mcp-server/tests/common" }` (line 78)
Remove or minimize `[dev-dependencies]` across affected crates:
- `core/Cargo.toml`: Remove `assert_cmd`, `predicates`, `tokio-test`, `wiremock` (keep `tempfile`, `pretty_assertions` if needed for doc tests)
- `app-server/Cargo.toml`: Remove `app_test_support`, `core_test_support`, `assert_cmd`, `base64`, `os_info`, `pretty_assertions`, `tempfile`, `toml`, `wiremock`
- `exec/Cargo.toml`: Review and remove test-specific dependencies
- `tui/Cargo.toml`: Remove `insta`, `strip-ansi-escapes`, `rand`, `pretty_assertions` if only used in legacy tests
---
## Minimal Tests to Keep
### Recommended Smoke/Unit Tests
1. **Protocol/Types Tests** (`mcp-types/tests/`):
- JSON-RPC serialization/deserialization correctness
- Progress notification format
- Initialize handshake format
- **Rationale**: Catches breaking API changes early
2. **Security Tests** (`linux-sandbox/tests/`):
- Landlock sandboxing validation
- **Rationale**: Security-critical, platform-specific
3. **Utility Tests** (`cloud-tasks/tests/env_filter.rs`):
- Environment variable filtering logic
- **Rationale**: Low-cost, high-value edge case coverage
4. **ExecPolicy (Minimal)** (`execpolicy/tests/`):
- Keep 2-3 critical command parsing tests (e.g., `literal.rs`, `sed.rs`)
- Move to inline `#[cfg(test)]` modules if possible
- **Rationale**: Critical correctness for shell command allowlisting
5. **Doc Tests** (Existing `#[doc]` examples):
- Already run by `cargo test` with minimal cost
- Keep inline examples in library code where helpful
- **Rationale**: Ensures public API examples compile
### Total Retained Test LOC
- Estimated: **<500 lines** (down from ~18k+ integration tests)
---
## Required Helper Refactors
### No Major Refactors Needed
Since we're **deleting** rather than migrating tests, minimal code changes are required:
1. **Remove workspace test support dependencies** (see Phase 3 above)
2. **Remove test-only pub methods** in production code:
- Search for `#[cfg(any(test, feature = "legacy_tests"))]` and remove those methods
- Example: `tui/src/chatwidget.rs` line 911 `pub(crate) fn test_for_request`
3. **Clean up unused imports** flagged by clippy after deletion
4. **Verify no CI breakage** from removed test crates (unlikely since tests aren't run in CI)
### Optional: Consolidate Remaining Tests
For `execpolicy` and other small suites, consider:
- Moving tests from `tests/` directory to inline `#[cfg(test)] mod tests` in `src/`
- Reduces filesystem clutter
- Allows direct access to private functions
- Example pattern:
```rust
// src/lib.rs or src/module.rs
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_parsing_logic() { /* ... */ }
}
```
---
## CI Gating Strategy
### Current Behavior (No Change Needed)
- **Primary gate**: `./build-fast.sh --workspace code` (build-only)
- **No test execution** in any workflow
- Workflows already enforce:
- Build success (compile check)
- Clippy lints pass
- Formatting via `rustfmt` (if configured)
### Post-Reset Behavior
- **Unchanged**: `./build-fast.sh --workspace code` remains sole gate
- Minimal test suites (mcp-types, linux-sandbox, etc.) can be run **locally** on-demand:
```bash
cargo test -p mcp-types
cargo test -p code-linux-sandbox
cargo test -p code-cloud-tasks
```
- No automatic test execution in CI prevents:
- Slow PR feedback
- Flaky test maintenance burden
- Need for test infrastructure (mock servers, fixtures, etc.)
### Future Considerations
If integration tests are re-introduced later:
- Gate behind explicit workflow dispatch or nightly schedule
- Use separate workflow file (e.g., `.github/workflows/integration-tests.yml`)
- Mark as non-blocking advisory check
---
## Migration Checklist
### Pre-Deletion Validation
- [ ] Archive current test suite state to a branch (e.g., `archive/legacy-tests-2025-10`)
- [ ] Document any critical test scenarios that should be preserved as manual validation steps
- [ ] Verify no production code has hard dependencies on test crates
### Deletion Steps
1. [ ] Delete test directories (Phase 1):
- [ ] `code-rs/core/tests/`
- [ ] `code-rs/app-server/tests/`
- [ ] `code-rs/exec/tests/`
- [ ] `code-rs/mcp-server/tests/`
- [ ] `code-rs/login/tests/`
- [ ] `code-rs/chatgpt/tests/`
- [ ] `code-rs/apply-patch/tests/`
- [ ] `code-rs/cli/tests/`
2. [ ] Remove TUI legacy tests (Phase 2):
- [ ] Delete `legacy_tests` feature from `tui/Cargo.toml`
- [ ] Remove all 27 gated test modules in `tui/src/`
- [ ] Remove `chatwidget_stream_tests.rs`
- [ ] Remove `vt100-tests` feature if unused
3. [ ] Clean dependencies (Phase 3):
- [ ] Remove `app_test_support`, `core_test_support`, `mcp_test_support` from workspace `Cargo.toml`
- [ ] Prune `[dev-dependencies]` in affected crates
4. [ ] Remove test-only code:
- [ ] Search and remove `#[cfg(any(test, feature = "legacy_tests"))]` methods
- [ ] Clean up unused imports (run `cargo clippy --fix`)
### Post-Deletion Validation
- [ ] Run `./build-fast.sh --workspace code` successfully
- [ ] Run `cargo clippy --workspace` with no errors
- [ ] Run remaining tests locally: `cargo test -p mcp-types -p code-linux-sandbox -p code-cloud-tasks`
- [ ] Verify no warnings about missing workspace members
- [ ] Confirm CI workflows still pass
---
## Rollback Plan
If issues arise post-deletion:
1. Revert to archived branch: `git checkout archive/legacy-tests-2025-10`
2. Cherry-pick critical fixes onto legacy test branch
3. Evaluate minimal subset of tests to restore
---
## Benefits Summary
1. **Reduced Maintenance**: No need to update/fix tests when refactoring internal APIs
2. **Faster Local Builds**: No test compilation overhead for developers
3. **CI Simplicity**: Single build gate reduces complexity and failure modes
4. **Codebase Clarity**: Less test scaffolding obscuring production code
5. **Lower Barrier to Entry**: New contributors only need to ensure build passes
---
## Risks & Mitigations
| Risk | Likelihood | Mitigation |
|------|------------|------------|
| Loss of regression coverage | Medium | Archive tests; rely on manual validation for critical paths; re-introduce targeted tests if regressions occur |
| Breaking changes undetected | Medium | Build gate + clippy catches most issues; user reports catch integration issues |
| Reduced confidence in refactoring | Low | Strong type system + compiler checks provide safety net |
| Difficulty debugging production issues | Low | Logging/telemetry more valuable than stale tests; can add focused repro tests on-demand |
---
## References
- Current build script: `./build-fast.sh` (lines 465, 633-636 confirm no test execution)
- Workflow files:
- `.github/workflows/issue-triage.yml` line 188
- `.github/workflows/upstream-merge.yml` lines 351, 676-712, 733
- `.github/workflows/issue-code.yml` lines 170, 185, 251
- Feature flags: `code-rs/tui/Cargo.toml` lines 16, 20
- Test helpers: `code-rs/Cargo.toml` lines 48, 76, 78
---
**Document Version**: 1.0
**Last Updated**: 2025-10-05
**Status**: Ready for implementation
UPSTREAM_PARITY_CHECKLIST.md
+186
View File
@@ -0,0 +1,186 @@
# Upstream Parity Maintenance Checklist
Quick reference for maintaining parity with codex-rs MCP components.
## Weekly Monitoring (Every Monday)
### Code Changes
- [ ] Check [codex-rs commits](https://github.com/openai/codex-rs/commits/main) for changes to:
- [ ] `mcp-client/`
- [ ] `responses-api-proxy/`
- [ ] `process-hardening/`
- [ ] Review commit messages for breaking changes or new features
- [ ] Check if changes affect code-rs usage patterns
### Testing
- [ ] Run `cargo test -p core -- mcp::` with latest upstream
- [ ] Run `cargo test -p cli -- proxy::` if proxy changes detected
- [ ] Verify no new test failures introduced by upstream changes
### Issues/PRs
- [ ] Review open issues in codex-rs relevant to MCP
- [ ] Check open PRs that may affect our components
- [ ] Comment on issues affecting code-rs use cases
## Monthly Deep Dive
### Performance
- [ ] Benchmark MCP client with large responses (1MB, 2MB, 5MB)
- [ ] Measure proxy latency and throughput
- [ ] Compare performance vs previous month baseline
- [ ] Document any regressions or improvements
### Security
- [ ] Run `cargo audit` on full dependency tree
- [ ] Verify process hardening on all platforms:
- [ ] Linux: `cat /proc/$(pgrep code-responses)/status | grep Dumpable`
- [ ] macOS: Attempt lldb attach (should fail)
- [ ] Windows: (TODO: Define verification)
- [ ] Review upstream security advisories
- [ ] Check for CVEs in MCP component dependencies
### Code Review
- [ ] Review all upstream changes since last month
- [ ] Identify opportunities for code-rs improvements
- [ ] Identify code-rs features worth contributing upstream
- [ ] Update fork divergence documentation if needed
### Documentation
- [ ] Update upstream-mcp-reuse-strategy.md with new findings
- [ ] Document any new workarounds or patches needed
- [ ] Review and update implementation guide if API changed
- [ ] Update this checklist with new monitoring needs
## Per Upstream Release
### Pre-Upgrade
- [ ] Read full changelog for breaking changes
- [ ] Identify deprecated APIs used by code-rs
- [ ] Plan migration for breaking changes
- [ ] Create test branch for upgrade
### Testing
- [ ] Run full test suite: `cargo test --workspace`
- [ ] Integration tests with real MCP servers
- [ ] Performance regression tests
- [ ] Security hardening validation
- [ ] Cross-platform testing (Linux, macOS, Windows)
### Documentation
- [ ] Update dependency versions in docs
- [ ] Document migration steps for breaking changes
- [ ] Update examples if API changed
- [ ] Add release notes to code-rs changelog
### Deployment
- [ ] Update Cargo.toml with new version pins
- [ ] Create PR with upgrade changes
- [ ] Run CI/CD pipeline
- [ ] Deploy to staging environment
- [ ] Monitor for issues post-deployment
## Critical Alerts (Immediate Action)
### Security Advisory
- [ ] Assess impact on code-rs
- [ ] Determine if affected components are used
- [ ] Create hotfix branch if needed
- [ ] Apply patches immediately
- [ ] Notify team of security update
- [ ] Deploy emergency update
### Upstream API Breakage
- [ ] Identify affected code in code-rs
- [ ] Assess migration effort
- [ ] Choose mitigation strategy:
- [ ] Pin to last working version (temporary)
- [ ] Implement adapter/wrapper
- [ ] Migrate to new API
- [ ] Fork and maintain (last resort)
- [ ] Create migration plan
- [ ] Execute migration or pin version
### Upstream Abandonment
- [ ] Assess recent activity level
- [ ] Contact upstream maintainers
- [ ] Evaluate fork maintenance cost
- [ ] Decision matrix:
- [ ] Wait and monitor (if temporary)
- [ ] Take over maintenance
- [ ] Fork permanently
- [ ] Replace with alternative
- [ ] Document decision and rationale
## Quick Commands Reference
```bash
# Check upstream changes
cd codex-rs
git fetch origin
git log origin/main --since="1 week ago" -- mcp-client/ responses-api-proxy/ process-hardening/
# Run MCP-specific tests
cd code-rs
cargo test -p core -- mcp::
cargo test -p cli -- proxy::
# Security audit
cargo audit
# Performance benchmark
cargo bench --bench mcp_client_bench
# Check process hardening (Linux)
ps aux | grep code-responses
cat /proc/$(pgrep code-responses)/status | grep Dumpable
# Check process hardening (macOS)
ps aux | grep code-responses
lldb -p $(pgrep code-responses) # Should fail
# Diff implementations (if maintaining fork)
diff -u codex-rs/mcp-client/src/mcp_client.rs code-rs/mcp-client/src/mcp_client.rs
diff -u codex-rs/responses-api-proxy/src/lib.rs code-rs/responses-api-proxy/src/lib.rs
```
## GitHub Notifications Setup
1. Go to: https://github.com/openai/codex-rs
2. Click "Watch" → "Custom"
3. Enable notifications for:
- [x] Issues
- [x] Pull requests
- [x] Releases
- [x] Security alerts
4. Set up custom rules:
- Path filter: `mcp-client/**`
- Path filter: `responses-api-proxy/**`
- Path filter: `process-hardening/**`
## Metrics to Track
| Metric | Target | Current | Trend |
|--------|--------|---------|-------|
| Upstream divergence (lines) | <10 | 1 | ↓ |
| Test coverage | >90% | - | - |
| Large response handling (2MB) | <100ms | - | - |
| Security audit issues | 0 | - | - |
| Days since upstream sync | <7 | - | - |
## Contact
- **Upstream Issues**: https://github.com/openai/codex-rs/issues
- **Upstream Discussions**: https://github.com/openai/codex-rs/discussions
- **Security**: security@openai.com (for vulnerabilities)
## Archive
When upstream reuse is complete and stable, archive this checklist with:
- Final divergence report
- Lessons learned
- Recommendations for future component reuse
---
**Last Updated**: 2025-10-05
**Next Review**: 2025-10-12
docs/code-crate-parity-tracker.md
-18
View File
@@ -1,18 +0,0 @@
# `code-*` vs `codex-*` Crate Parity Tracker
Goal: delete `code-*` crates once the forked implementation matches the new
upstream baseline (or the upstream crate already covers our needs).
| Crate | Diff summary (code vs codex) | Ready to delete `code-*`? | Notes |
| --- | --- | --- | --- |
| ansi-escape | Renamed crate IDs (`code-ansi-escape` vs `codex-ansi-escape`), otherwise same API but we ship fork-specific spinner assets. | No | Wait until spinner grouping lands upstream or we can fully adopt theirs. |
| arg0 | Fork keeps `--code-run-as-apply-patch` flag; upstream uses `codex`. | No | Needs flag rename adapters before deleting. |
| browser | Fork-only feature. | No | Upstream lacks browser integration. |
| core | Diverges heavily (executor/tool router, approval policy). | No | Bridge in progress (see `docs/subsystem-migration-status.md`). |
| exec | Fork adds policy prompts + streaming bridges. | No | Depends on core migration. |
| login | Fork adds approval + device flow tweaks; upstream gained OAuth helpers. | No | Reconcile after validating new helpers in fork. |
| protocol | Upstream introducing new tool schema (ACP). | No | Needs schema audit before removal. |
| tui | Fork retains browser UI, streaming invariants. | No | Must port status UI first. |
Update this table as diffs shrink. When “Yes” appears, schedule deletion in the
next release window and update documentation accordingly.
docs/code-dead-code-sweeps.md
-26
View File
@@ -1,26 +0,0 @@
# `code-rs` Dead-Code Sweep Plan
We now carry the upstream tree side-by-side. To keep `code-rs/` lean while
we bridge to the upstream crates, run focused dead-code sweeps on a schedule.
## Cadence
| Window | Focus areas | Owner |
| --- | --- | --- |
| 2025-10-20 → 2025-10-24 | `code-rs/tui/*` legacy render helpers, unused history cell modules. | TUI pod |
| 2025-11-17 → 2025-11-21 | `code-rs/core/*` legacy executor adapters after the upstream router lands. | Core runtime pod |
| 2025-12-15 → 2025-12-19 | Remaining crates (CLI, browser, app-server) post-executor migration. | Repo maintainers |
Add follow-up rows as additional migrations finish.
## Sweep checklist
1. Run `rg --files-without-match` patterns to detect unused modules (e.g. `mod
` declarations).
2. Confirm no crate-level `pub use` exposes the target symbol externally.
3. Delete the dead module/file and rerun `./build-fast.sh --workspace code`.
4. If the removal mirrors an upstream deletion, record it in
`docs/subsystem-migration-status.md` so future merges know the fork matches
upstream.
Track outcomes in PR descriptions so we know which areas are already cleaned.
docs/code-refactor-plan.md
-91
View File
@@ -1,91 +0,0 @@
# Code/Core Refactor – Wrapper-first Reset Plan
## Goals
- Keep upstream `codex-*` crates pristine so we can fast-forward to
`openai/codex:main` without hand-merging fork logic every time.
- Host all fork-only functionality (binaries, features, modules, assets)
under parallel `code-*` crates that depend on the upstream crates.
- Preserve a working build throughout the transition so we can ship fixes
while the migration is underway.
## High-Level Strategy
We briefly diverged with piecemeal module copies. To simplify, we will:
### What to Avoid
We already tried copying modules into `code-*` while leaving the originals in `codex-*`; nothing pointed at the new copies, merges stayed painful, and the workspace exploded in size. This plan keeps the forked tree separate from the start so every step removes upstream churn instead of doubling it.
1. **Duplicate the tree once, then freeze upstream**
- Move (via `git mv`) the existing `codex-rs/` working tree into
`code-rs/` so history follows the forked code.
- Run a scripted rename that turns every crate/package prefix
`codex-` → `code-` inside the moved tree (Cargo manifests, feature
names, `use` statements, module paths, tool configs).
- Update workspace manifests, scripts, and build tooling so the renamed
crates compile from their new location. Verify with `./build-fast.sh`.
2. **Restore upstream into `codex-rs/`**
- Check out the latest `openai/codex:main` tree into `codex-rs/`.
- Confirm `codex-rs/` contains zero fork-only files (just upstream).
- Keep both trees side by side: `code-rs/` (fork) and `codex-rs/`
(baseline).
3. **Bridge and prune incrementally**
- Point binaries and downstream crates at the `code-*` versions first so
the product keeps working.
- For each fork feature area (TUI, app-server, CLI, core, browser, etc.)
decide whether we can extend upstream `codex-*` code or must keep the
forked implementation. Favor reusing large upstream sections where the
integration cost is manageable; otherwise, maintain the fork logic and
document why it diverges.
- When we do integrate, prefer targeted extension points or light
wrappers instead of one-to-one rewrites, and retire duplicate modules
only after the shared path is verified.
- After each feature area decision, re-run `./build-fast.sh` and capture
any remaining fork patches in the matching `codex-*` crate for future
cleanup.
4. **Ongoing maintenance**
- Pull upstream changes directly into `codex-rs/`, resolve conflicts only
in `code-rs/` when APIs shift, and document deltas in
`docs/fork-enhancements.md`.
- Track remaining wrapper-only surface in a status doc (e.g.
`docs/tui-module-migration-status.md`).
## Execution Checklist
1. **DONE (2025-10-04):** Snapshot current state (branch + optional tag) so
we can recover if the reset uncovers regressions. Created
`snapshot/pre-upstream-reset-20251004` (branch) and
`snapshot-pre-upstream-reset-20251004` (tag) before upstream restore
work begins.
2. **DONE (2025-10-04):** Duplicate `codex-rs/` into `code-rs/` so history
follows the fork. Bulk-renamed all crates/binaries from `codex-*` to
`code-*`, fixed module/file names, updated manifests, and confirmed
`./build-fast.sh --workspace code` passes.
3. **DONE (2025-10-04):** Update shared tooling (`build-fast.sh`) to target
both workspaces and keep existing `codex` builds green (`./build-fast.sh`).
4. **DONE (2025-10-04):** Audited all scripts, docs, and SDK samples to swap
`codex-rs/…` references over to `code-rs/…` (see commits touching
`build-fast.sh`, `scripts/ci-tests.sh`, TypeScript SDK, installation docs).
5. **DONE (2025-10-04):** Checked out upstream `codex-rs/` at
`openai/codex@d7acd146` and re-ran `./build-fast.sh --workspace codex`
plus `./build-fast.sh --workspace code` to confirm both workspaces build.
6. **DONE (2025-10-04):** Captured subsystem decisions in
`docs/subsystem-migration-status.md`, highlighting where we will reuse
upstream code versus keep fork-specialised wrappers.
7. **DONE (2025-10-04):** Scheduled recurring `code-rs/` dead-code sweeps and
documented the cadence in `docs/code-dead-code-sweeps.md`.
8. **DONE (2025-10-04):** Opened `docs/code-crate-parity-tracker.md` to record
when forked crates reach parity with upstream so we can delete duplicates
ahead of releases.
## Tracking
- Maintain per-area migration status docs under `docs/`.
- Record any upstream patches we must keep in `codex-*` (policy/legal) so
future merges understand why they exist.
- Treat `./build-fast.sh` as the sole required validation step after each
migration chunk; fix all warnings before landing.
docs/maintenance/upstream-diff.md
+271
View File
@@ -0,0 +1,271 @@
# Upstream Tracking and Diff Pipeline
This document describes the process for tracking differences between the fork (`code-rs`) and upstream (`codex-rs`), identifying critical changes, and planning merges.
## Overview
The fork maintains parallel crate trees:
- **`codex-rs/`** - Upstream baseline (synced from `https://github.com/openai/codex.git`)
- **`code-rs/`** - Fork implementation with enhancements
The tracking pipeline helps identify:
1. **Structural differences** - Which crates have diverged
2. **Critical changes** - Changes affecting prompts, API surface, or executor
3. **Merge decisions** - Historical log of what was adopted/adapted/preserved
## Pipeline Scripts
All scripts are located in `scripts/upstream-merge/`:
### 1. `diff-crates.sh` - Structural Diff
Compares codex-rs vs code-rs on a per-crate basis.
**Usage:**
```bash
# Compare a single crate
./scripts/upstream-merge/diff-crates.sh core
# Compare all shared crates
./scripts/upstream-merge/diff-crates.sh --all
# Generate summary report
./scripts/upstream-merge/diff-crates.sh --summary
```
**Output:**
- Individual diff files: `.github/auto/upstream-diffs/<crate>.diff`
- Summary report: `.github/auto/upstream-diffs/SUMMARY.md`
**What it tracks:**
- Line-by-line differences between matching crates
- Added/removed lines count
- Files present in one tree but not the other
### 2. `highlight-critical-changes.sh` - Critical Change Detection
Analyzes diffs to highlight changes in sensitive subsystems.
**Usage:**
```bash
# Analyze a specific crate
./scripts/upstream-merge/highlight-critical-changes.sh core
# Analyze all crates
./scripts/upstream-merge/highlight-critical-changes.sh --all
```
**Output:**
- Per-crate critical changes: `.github/auto/upstream-diffs/critical-changes/<crate>-critical.md`
- Summary: `.github/auto/upstream-diffs/critical-changes/CRITICAL-SUMMARY.md`
**What it tracks:**
- **Prompts**: Changes to `prompts/*.md`, `openai_tools.rs`, `agent_tool.rs`
- **API Surface**: Changes to `protocol.rs`, `app-server-protocol`, `mcp-types`
- **Executor**: Changes to `codex.rs`, `exec`, `apply-patch`, `acp.rs`
- **Config**: Changes to config types and behavior toggles
- Function signature changes
- Enum/struct definition changes
- Protocol message changes (EventMsg, Op variants)
### 3. `log-merge.sh` - Merge Activity Logger
Tracks merge decisions and resolutions for future reference.
**Usage:**
```bash
# Start a new merge
./scripts/upstream-merge/log-merge.sh init upstream/main
# Add notes during merge
./scripts/upstream-merge/log-merge.sh note conflict "core/src/codex.rs has protocol changes"
# Log decisions
./scripts/upstream-merge/log-merge.sh decision core preserve "Keep fork's QueueUserInput support"
./scripts/upstream-merge/log-merge.sh decision exec adopt "Upstream exec improvements compatible"
# Finalize when done
./scripts/upstream-merge/log-merge.sh finalize
# View historical logs
./scripts/upstream-merge/log-merge.sh summary
```
**Output:**
- Timestamped log: `docs/maintenance/upstream-merge-logs/merge-YYYYMMDD.md`
**What it tracks:**
- Upstream ref and commit being merged
- Fork state at merge start
- Critical changes identified
- Decisions made (adopt/adapt/preserve) with rationale
- Post-merge checklist status
### 4. `verify.sh` - Post-Merge Verification
Validates that the fork still builds and maintains critical functionality after a merge.
**Usage:**
```bash
./scripts/upstream-merge/verify.sh
```
**What it verifies:**
- Build passes (`build-fast.sh`)
- Core API surface compiles
- Static guards (browser/agent tools still registered)
- Version handling intact
- Branding consistency
## Typical Workflow
### Weekly Upstream Check
```bash
# Fetch latest upstream
git fetch upstream
# Check for new commits
git log HEAD..upstream/main --oneline
# If changes exist, start tracking
./scripts/upstream-merge/diff-crates.sh --all
./scripts/upstream-merge/highlight-critical-changes.sh --all
```
### Planning a Merge
```bash
# 1. Initialize merge log
./scripts/upstream-merge/log-merge.sh init upstream/main
# 2. Review diffs
cat .github/auto/upstream-diffs/SUMMARY.md
cat .github/auto/upstream-diffs/critical-changes/CRITICAL-SUMMARY.md
# 3. For each critical crate, make decisions
# Actions: adopt (take upstream), adapt (merge changes), preserve (keep fork)
./scripts/upstream-merge/log-merge.sh decision core adapt "Merge ACP support while preserving QueueUserInput"
./scripts/upstream-merge/log-merge.sh decision protocol adapt "Add new ACP variants, keep existing Op types"
./scripts/upstream-merge/log-merge.sh decision tui preserve "Fork-only Rust TUI, ignore upstream TS changes"
# 4. Apply changes crate-by-crate
# (Manual or scripted - depends on complexity)
# 5. Verify after each crate
./scripts/upstream-merge/verify.sh
# 6. Finalize and document
./scripts/upstream-merge/log-merge.sh finalize
git add .
git commit -m "chore(codex-rs): sync with upstream main
See docs/maintenance/upstream-merge-logs/merge-YYYYMMDD.md"
```
### Integration Strategies
**Adopt (take upstream as-is):**
- Upstream change is a strict improvement
- No fork-specific functionality conflicts
- Example: Performance improvements, bug fixes in non-critical paths
**Adapt (merge both):**
- Upstream adds new capability, fork has extensions
- Combine both sets of features
- Example: Upstream adds ACP events, fork keeps existing event types
**Preserve (keep fork):**
- Upstream removes/changes functionality fork depends on
- Fork behavior is intentionally different
- Example: Rust TUI vs TypeScript TUI, fork-specific tools
## Critical Subsystems
Pay special attention to changes in:
### Prompts and Tools
- `prompts/*.md` - System prompts
- `codex-rs/core/src/openai_tools.rs` - Tool definitions
- `codex-rs/core/src/agent_tool.rs` - Agent-specific tools
Changes here affect LLM behavior and available capabilities.
### Protocol and API
- `codex-rs/core/src/protocol.rs` - Core event types
- `codex-rs/app-server-protocol/` - Frontend communication
- `codex-rs/mcp-types/` - MCP protocol definitions
Changes here affect client compatibility and event flow.
### Executor
- `codex-rs/core/src/codex.rs` - Main execution loop
- `codex-rs/exec/src/` - Command execution
- `codex-rs/apply-patch/src/` - File modification
- `codex-rs/core/src/acp.rs` - Agent Client Protocol bridge
Changes here affect how tools execute and side effects are managed.
### Configuration
- `codex-rs/core/src/config.rs` - Config loading
- `codex-rs/core/src/config_types.rs` - Config schema
Changes here affect default behavior and user-facing options.
## Parity Tracking
After each merge, update `docs/code-crate-parity-tracker.md`:
- Note which crates moved closer to parity
- Update "Ready to delete code-*?" column
- Add notes about remaining divergence
Goal: Eventually delete `code-rs/*` crates that match upstream, consolidating to single implementation.
## References
- **Upstream ACP Integration**: `docs/upstream-acp-merge.md` - Detailed notes on integrating Agent Client Protocol
- **Parity Tracker**: `docs/code-crate-parity-tracker.md` - Per-crate divergence status
- **Merge Prompts**: `prompts/MERGE.md` - Guidance for resolving conflicts
- **Triage Prompts**: `prompts/TRIAGE.md` - Categorizing upstream changes
## Automation Opportunities
Future enhancements to pipeline:
- Auto-detect upstream releases via GitHub API
- Generate merge preview PRs automatically
- Classify changes by semantic impact (breaking/compatible/internal)
- Extract commit messages for changelog integration
- Cross-reference with fork's issue tracker
## Troubleshooting
**Diff shows everything changed:**
- Check line endings (CRLF vs LF)
- Verify both trees are clean (`git status`)
- Ensure upstream remote is correctly configured
**Critical changes not detected:**
- Update patterns in `highlight-critical-changes.sh`
- Check that diff files exist in `.github/auto/upstream-diffs/`
**Verify script fails:**
- Review `.github/auto/VERIFY.json` for specific failures
- Check logs in `.github/auto/VERIFY_*.log`
- Ensure all fork-specific guards are still valid
**Merge log not found:**
- Logs are in `docs/maintenance/upstream-merge-logs/`
- Initialize with `log-merge.sh init` before logging notes/decisions
## Maintenance
Update these scripts when:
- New critical files/patterns emerge (update `CRITICAL_PATTERNS` in `highlight-critical-changes.sh`)
- New crates are added to either tree (update `SHARED_CRATES` in `diff-crates.sh`)
- New verification guards needed (update `verify.sh`)
- Merge workflow evolves (update this doc and `log-merge.sh`)
---
Document created: 2025-10-05
Last updated: 2025-10-05
docs/subsystem-migration-status.md
-18
View File
@@ -1,18 +0,0 @@
# Subsystem Migration Status (2025-10-05)
This tracker documents how each major area of the fork compares against the
fresh upstream snapshot in `codex-rs/` and whether we plan to reuse the upstream
implementation or keep a forked variant living under `code-rs/`.
| Subsystem | Upstream delta summary | Decision | Immediate next steps |
| --- | --- | --- | --- |
| Core runtime (`core/`, `exec/`, `protocol/`, `apply-patch/`) | Upstream added a unified `executor` module, tool router, and richer MCP hooks. Fork keeps pre-refactor pipeline plus fork-only policy/approval features. | **Hybrid** — adopt upstream executor/tool router incrementally while preserving fork-only approval flow. | Diff `code-rs/core` vs `codex-rs/core` focusing on `executor/*` and `tools/*`; design an adapter so forked approval + sandbox policy can plug into the upstream router without copy-pasting. |
| TUI (`tui/`) | Upstream reorganised history cells into typed renderers, added status dashboards/tests, and removed the older per-cell module tree. Fork still uses the legacy layout with Code-specific UX. | **Fork-primary** — keep `code-rs/tui` as the shipping implementation; cherry-pick upstream status widgets/tests once we add equivalent APIs in `code-rs`. | ✅ History/approval/event facade adapters landed behind `code-fork`. Next: finish routing remaining direct imports through compat modules, run manual smoke checklist, and continue pruning legacy test suites once replacements exist. |
| CLI (`cli/`, `login/`, `mcp-*`, `responses-api-proxy/`) | Upstream refreshed auth flows (OAuth helpers in `rmcp-client`) and expanded MCP tooling; fork adds policy prompts, approval gating. | **Hybrid** — reuse upstream auth/mcp client improvements by wiring them behind feature flags in `code-rs` while keeping fork-only prompts. | Prototype re-export of `codex-rs/rmcp-client` login helpers inside `code-rs/login`; confirm CLI flag parity before deleting fork copies. |
| Browser tooling (`browser/`, TUI browser events) | Upstream still lacks the browser agent; fork-only feature. | **Fork-only** — retain `code-rs/browser` and related TUI integrations. | Document browser integration in `docs/fork-enhancements.md` (already tracked); no upstream merge needed. |
| App server (`app-server`, `app-server-protocol`) | Upstream rewrote message processor to use new executor/tool traits. Fork still references old pipeline. | **Monitor** — do not adopt until core executor bridge (above) lands; otherwise duplicate effort. | Once the core adapter is ready, revisit to see if we can reuse upstream app-server entirely. |
| SDK / tooling (`sdk/typescript`, `scripts/*`) | Upstream scripts now assume `codex-*` bins. Fork updated to `code-*`. | **Fork-primary** with compatibility fallbacks. | Ensure scripts that read templates (e.g., `test-responses.js`) load both paths (already updated). Review future upstream changes quarterly. |
| Build & release tooling (`build-fast.sh`, nix flakes) | Upstream still targets a single workspace. Fork must handle both. | **Fork-only** — maintain dual-workspace script. | Keep script in sync with upstream bugfixes manually; add regression tests later. |
Last updated **2025-10-04**. Update this table whenever a subsystem decision
changes or a milestone completes.
docs/upstream-mcp-implementation-guide.md
+369
View File
@@ -0,0 +1,369 @@
# Upstream MCP Client Implementation Guide
Quick reference for implementing the upstream reuse strategy.
## Quick Start: Migrate to Upstream Dependencies
### 1. Update Workspace Cargo.toml
```toml
# code-rs/Cargo.toml
[workspace.dependencies]
# Replace fork dependencies with upstream
codex-mcp-client = { path = "../codex-rs/mcp-client" }
codex-responses-api-proxy = { path = "../codex-rs/responses-api-proxy" }
codex-process-hardening = { path = "../codex-rs/process-hardening" }
# Optional: Keep re-export aliases for gradual migration
code-mcp-client = { path = "../codex-rs/mcp-client", package = "codex-mcp-client" }
code-responses-api-proxy = { path = "../codex-rs/responses-api-proxy", package = "codex-responses-api-proxy" }
code-process-hardening = { path = "../codex-rs/process-hardening", package = "codex-process-hardening" }
```
### 2. Update Core Dependencies
```toml
# code-rs/core/Cargo.toml
[dependencies]
# Option A: Direct migration
codex-mcp-client = { workspace = true }
# Option B: Gradual migration with alias
code-mcp-client = { workspace = true }
```
### 3. Handle Binary Naming (If Required)
#### Option A: Build Script Rename
```rust
// code-rs/cli/build.rs
use std::env;
use std::fs;
use std::path::PathBuf;
fn main() {
// If embedding binary
let out_dir = PathBuf::from(env::var("OUT_DIR").unwrap());
// Copy upstream binary with code- prefix
let upstream_bin = "../target/release/codex-responses-api-proxy";
let renamed_bin = out_dir.join("code-responses-api-proxy");
if PathBuf::from(upstream_bin).exists() {
fs::copy(upstream_bin, renamed_bin)
.expect("Failed to copy responses-api-proxy binary");
}
}
```
#### Option B: Accept Upstream Naming
```rust
// code-rs/cli/src/whatever_uses_the_binary.rs
// Simply use codex- prefix everywhere
const PROXY_BINARY: &str = "codex-responses-api-proxy";
```
### 4. Create Buffer Size Wrapper (If Needed)
Test first if default buffer is sufficient. If not:
```rust
// code-rs/core/src/mcp/client_wrapper.rs
use codex_mcp_client::McpClient;
use std::collections::HashMap;
use std::ffi::OsString;
/// Creates MCP client optimized for large responses (code-rs specific)
pub async fn create_large_response_client(
program: OsString,
args: Vec<OsString>,
env: Option<HashMap<String, String>>,
) -> std::io::Result<McpClient> {
// For now, upstream implementation may already handle this well
// TODO: Benchmark with actual large responses before customizing
McpClient::new_stdio_client(program, args, env).await
}
// If buffer customization proves necessary, contribute upstream PR
// See: upstream-mcp-reuse-strategy.md Strategy B
```
### 5. Update Imports
```rust
// Before (code-rs fork)
use code_mcp_client::McpClient;
use code_process_hardening;
// After (upstream direct)
use codex_mcp_client::McpClient;
use codex_process_hardening;
// Or with re-export alias (no code changes needed)
use code_mcp_client::McpClient; // Still works via Cargo.toml alias
use code_process_hardening; // Still works via Cargo.toml alias
```
## Testing the Migration
### 1. Unit Tests
```bash
cd code-rs
cargo test -p core # Test MCP client integration
cargo test -p cli # Test proxy binary handling
```
### 2. Integration Tests with Large Payloads
```rust
// code-rs/core/tests/mcp_large_response_test.rs
#[tokio::test]
async fn test_large_tool_response() {
// Generate 2MB JSON response
let large_payload = generate_large_json_payload(2 * 1024 * 1024);
// Test MCP client handles it without truncation
let client = create_test_mcp_client().await.unwrap();
let result = client.call_tool(
"large_data_tool".to_string(),
Some(large_payload),
Some(Duration::from_secs(30))
).await;
assert!(result.is_ok());
// Validate full payload received
}
```
### 3. Security Validation
```bash
# Verify process hardening still active
cargo build --release
./target/release/code-responses-api-proxy --help
# On Linux: Verify non-dumpable
cat /proc/$(pgrep code-responses)/status | grep Dumpable
# Should show: Dumpable: 0
# On macOS: Verify ptrace protection
# Attempt: lldb -p $(pgrep code-responses)
# Should fail: "Operation not permitted"
```
## Upstream Contribution Workflow
### Proposing Buffer Configuration Feature
```rust
// Proposed upstream change for codex-rs/mcp-client
// File: codex-rs/mcp-client/src/mcp_client.rs
/// Configuration options for MCP client
#[derive(Debug, Clone)]
pub struct McpClientConfig {
/// Buffer capacity for reading server responses.
/// Default: 8KB (Tokio default)
/// Use larger values (e.g., 1MB) for servers with large tool responses
pub buffer_capacity: Option<usize>,
}
impl Default for McpClientConfig {
fn default() -> Self {
Self {
buffer_capacity: None,
}
}
}
impl McpClient {
// Existing method unchanged for compatibility
pub async fn new_stdio_client(
program: OsString,
args: Vec<OsString>,
env: Option<HashMap<String, String>>,
) -> std::io::Result<Self> {
Self::new_stdio_client_with_config(
program,
args,
env,
McpClientConfig::default()
).await
}
// New method with configuration
pub async fn new_stdio_client_with_config(
program: OsString,
args: Vec<OsString>,
env: Option<HashMap<String, String>>,
config: McpClientConfig,
) -> std::io::Result<Self> {
// ... existing setup code ...
let reader_handle = {
let pending = pending.clone();
let mut lines = match config.buffer_capacity {
Some(capacity) => BufReader::with_capacity(capacity, stdout).lines(),
None => BufReader::new(stdout).lines(),
};
// ... rest of implementation ...
};
// ... rest of implementation ...
}
}
```
### PR Description Template
```markdown
## Add buffer configuration to MCP client
### Motivation
When working with MCP servers that return large tool responses (>100KB),
the default buffer size can impact performance. This PR adds optional
buffer configuration while maintaining backward compatibility.
### Changes
- Add `McpClientConfig` struct with optional buffer_capacity
- Add `new_stdio_client_with_config` method
- Keep existing `new_stdio_client` method unchanged (uses default config)
### Testing
- Existing tests pass (backward compatibility verified)
- New test: large response handling with 1MB buffer
- Benchmark: 2MB response ~30% faster with larger buffer
### Breaking Changes
None - existing API unchanged, new method is additive
### Use Case (code-rs)
We use this in code-rs for MCP servers that return large file contents
or extensive analysis results. Setting buffer to 1MB eliminates multiple
read syscalls for these responses.
```
## Rollback Plan
If migration causes issues:
```toml
# Revert to fork in code-rs/Cargo.toml
[workspace.dependencies]
code-mcp-client = { path = "mcp-client" }
code-responses-api-proxy = { path = "responses-api-proxy" }
code-process-hardening = { path = "process-hardening" }
```
```bash
# Restore fork code
git checkout main -- code-rs/mcp-client
git checkout main -- code-rs/responses-api-proxy
git checkout main -- code-rs/process-hardening
```
## Performance Benchmarks
Before migration, establish baseline:
```rust
// code-rs/benches/mcp_client_bench.rs
use criterion::{black_box, criterion_group, criterion_main, Criterion};
fn benchmark_large_response(c: &mut Criterion) {
c.bench_function("mcp_client_1mb_response", |b| {
b.iter(|| {
// Test current fork implementation
let client = create_fork_client();
let result = client.call_tool_with_large_response();
black_box(result)
})
});
}
criterion_group!(benches, benchmark_large_response);
criterion_main!(benches);
```
After migration, compare:
```bash
cargo bench --bench mcp_client_bench > before.txt
# Apply migration
cargo bench --bench mcp_client_bench > after.txt
# Compare results
diff before.txt after.txt
```
## Monitoring Post-Migration
### CI/CD Checks
```yaml
# .github/workflows/mcp-upstream-health.yml
name: MCP Upstream Health Check
on:
schedule:
- cron: '0 8 * * 1' # Weekly Monday 8am
workflow_dispatch:
jobs:
check-upstream:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
with:
submodules: true
- name: Check for upstream changes
run: |
cd codex-rs
git fetch origin
git diff origin/main -- mcp-client/ responses-api-proxy/ process-hardening/
- name: Run MCP integration tests
run: |
cargo test -p core -- mcp::
cargo test -p cli -- proxy::
- name: Security audit
run: cargo audit
```
### Alerting
Set up GitHub notifications:
1. Watch codex-rs repository
2. Custom notification rules:
- All activity in `mcp-client/`
- All activity in `responses-api-proxy/`
- All activity in `process-hardening/`
- Security advisories
## Summary Checklist
- [ ] Update workspace dependencies to point to codex-rs
- [ ] Test with default buffer size first
- [ ] Implement wrapper only if needed (benchmark first)
- [ ] Update imports or use Cargo.toml aliases
- [ ] Run full test suite
- [ ] Validate process hardening on all platforms
- [ ] Benchmark performance before/after
- [ ] Set up upstream monitoring
- [ ] Prepare upstream PR for buffer config (if needed)
- [ ] Document migration in CHANGELOG
- [ ] Update internal documentation references
## Next Steps
1. **Immediate**: Test upstream `codex-mcp-client` with code-rs workloads
2. **Week 1**: Migrate mcp-client dependency if tests pass
3. **Week 2**: Migrate responses-api-proxy with binary rename
4. **Week 3**: Propose upstream buffer config PR
5. **Ongoing**: Monitor upstream changes weekly
See [upstream-mcp-reuse-strategy.md](./upstream-mcp-reuse-strategy.md) for detailed analysis and rationale.
docs/upstream-mcp-reuse-strategy.md
+325
View File
@@ -0,0 +1,325 @@
# Upstream MCP Client Reuse Strategy
## Executive Summary
This document analyzes the feasibility of reusing `codex-rs/mcp-client` and `codex-rs/responses-api-proxy` directly from code-rs without maintaining separate forks. The investigation reveals **minimal divergence** between implementations, making upstream reuse highly feasible with a thin wrapper/patch strategy.
## Analysis of Current Divergence
### 1. MCP Client (`mcp-client`)
**Differences Found:**
- **Buffer Size**: code-rs uses 1MB buffer (`BufReader::with_capacity(1024 * 1024, stdout)`) vs codex-rs default buffer
- Location: `mcp-client/src/mcp_client.rs:145`
- Reason: Handle large tool responses without truncation
- Impact: Performance optimization for specific use cases
**Similarities:**
- Identical package structure and dependencies
- Same API surface and MCP protocol implementation
- Same JSON-RPC message handling
- Same environment variable filtering logic
- File sizes: 476 vs 475 lines (virtually identical)
### 2. Responses API Proxy (`responses-api-proxy`)
**Differences Found:**
- **None** - The lib.rs files are byte-for-byte identical
**Naming Differences:**
- Package name: `code-responses-api-proxy` vs `codex-responses-api-proxy`
- Binary name: `code-responses-api-proxy` vs `codex-responses-api-proxy`
- Library name: `code_responses_api_proxy` vs `codex_responses_api_proxy`
- Process hardening dependency: `code-process-hardening` vs `codex-process-hardening`
### 3. Process Hardening (`process-hardening`)
**Differences Found:**
- **Minor style difference** in filter_map closure (code-rs uses compact `.then_some()`, codex-rs uses verbose if/else)
- Location: `process-hardening/src/lib.rs:44-50` (Linux), `77-84` (macOS)
- Functional equivalence: Identical behavior
- Comment difference: Line 113 has `TODO(mbolin):` in codex-rs vs `TODO:` in code-rs
**Similarities:**
- Identical hardening strategy (core dumps, ptrace, env vars)
- Same platform-specific implementations
- Same error handling and exit codes
## Proposed Wrapper/Patch Strategy
### Strategy A: Direct Upstream Dependency (Recommended)
Use codex-rs crates directly with minimal wrapper to handle fork-specific needs.
**Implementation:**
1. **MCP Client**: Add configuration parameter for buffer size
```rust
// In code-rs workspace Cargo.toml
[dependencies]
codex-mcp-client = { path = "../codex-rs/mcp-client" }
// Create thin wrapper in code-rs if buffer config needed
pub fn create_mcp_client_large_responses(
program: OsString,
args: Vec<OsString>,
env: Option<HashMap<String, String>>,
) -> std::io::Result<codex_mcp_client::McpClient> {
// Option 1: Use as-is (upstream buffer may be sufficient)
codex_mcp_client::McpClient::new_stdio_client(program, args, env)
// Option 2: Propose buffer size config upstream
// codex_mcp_client::McpClient::with_buffer_capacity(1024*1024)
// .new_stdio_client(program, args, env)
}
```
2. **Responses API Proxy**: Use upstream with renamed binary
```rust
// In code-rs/cli binary embedding
#[cfg(feature = "embed-binaries")]
const RESPONSES_PROXY_BINARY: &[u8] =
include_bytes!(concat!(env!("OUT_DIR"), "/codex-responses-api-proxy"));
// Or: Copy/symlink binary during build with code- prefix
```
3. **Process Hardening**: Direct dependency, re-export if needed
```rust
// In code-rs workspace
pub use codex_process_hardening as code_process_hardening;
```
**Advantages:**
- Zero maintenance burden for core logic
- Automatic upstream security fixes and improvements
- Minimal code duplication
**Disadvantages:**
- Dependency on external codebase structure
- Binary naming requires build script handling
### Strategy B: Feature-Flag in Upstream
Contribute buffer configuration as optional feature to codex-rs.
**Upstream PR Proposal:**
```rust
// In codex-rs/mcp-client/src/mcp_client.rs
pub struct McpClientConfig {
pub buffer_capacity: Option<usize>,
// Future: timeout configs, retry logic, etc.
}
impl Default for McpClientConfig {
fn default() -> Self {
Self { buffer_capacity: None }
}
}
impl McpClient {
pub async fn new_stdio_client_with_config(
program: OsString,
args: Vec<OsString>,
env: Option<HashMap<String, String>>,
config: McpClientConfig,
) -> std::io::Result<Self> {
// ... existing code ...
let mut lines = if let Some(capacity) = config.buffer_capacity {
BufReader::with_capacity(capacity, stdout).lines()
} else {
BufReader::new(stdout).lines()
};
// ... rest of implementation ...
}
}
```
**Advantages:**
- Cleanest long-term solution
- Benefits both codebases
- No wrapper needed
**Disadvantages:**
- Requires upstream acceptance
- Timeline depends on upstream review
### Strategy C: Minimal Fork with Automated Sync
Maintain current fork structure but automate sync from upstream.
**Implementation:**
```bash
# .github/workflows/sync-upstream-mcp.yml
name: Sync Upstream MCP Components
on:
schedule:
- cron: '0 0 * * 1' # Weekly
workflow_dispatch:
jobs:
sync:
runs-on: ubuntu-latest
steps:
- name: Sync mcp-client
run: |
rsync -av --exclude=Cargo.toml \
../codex-rs/mcp-client/src/ \
./code-rs/mcp-client/src/
# Apply code-rs specific patches
patch -p1 < patches/mcp-client-buffer-size.patch
```
**Patch file** (`patches/mcp-client-buffer-size.patch`):
```diff
--- a/code-rs/mcp-client/src/mcp_client.rs
+++ b/code-rs/mcp-client/src/mcp_client.rs
@@ -141,7 +141,8 @@
let reader_handle = {
let pending = pending.clone();
- let mut lines = BufReader::new(stdout).lines();
+ // Use a larger buffer size (1MB) to handle large tool responses
+ let mut lines = BufReader::with_capacity(1024 * 1024, stdout).lines();
```
**Advantages:**
- Automated tracking of upstream changes
- Clear patch management
- Fork-specific customization preserved
**Disadvantages:**
- Still maintains separate crate
- Patch conflicts require manual resolution
## Fork-Specific Modifications Required
### 1. Binary/Package Naming
**Current Naming:**
| Component | codex-rs | code-rs |
|-----------|----------|---------|
| MCP client package | `codex-mcp-client` | `code-mcp-client` |
| Proxy package | `codex-responses-api-proxy` | `code-responses-api-proxy` |
| Proxy binary | `codex-responses-api-proxy` | `code-responses-api-proxy` |
| Process hardening | `codex-process-hardening` | `code-process-hardening` |
**Resolution Strategies:**
1. **Build-time rename**: Copy and rename binaries during build
2. **Cargo alias**: Use `[[bin]]` section with custom name
3. **Accept upstream names**: Use `codex-*` binaries in code-rs (simplest)
### 2. Buffer Size Configuration
**Options:**
1. **Environment variable**: `MCP_BUFFER_SIZE=1048576`
2. **Config file parameter**: Add to MCP server config
3. **Upstream contribution**: Add to McpClientConfig (Strategy B)
4. **Accept default**: Test if upstream buffer is sufficient
### 3. Process Hardening Integration
**Current approach:**
```rust
// code-rs binaries
use code_process_hardening;
#[ctor::ctor]
fn pre_main() {
code_process_hardening::pre_main_hardening();
}
```
**Upstream reuse:**
```rust
// Option 1: Re-export
pub use codex_process_hardening as code_process_hardening;
// Option 2: Direct use
use codex_process_hardening;
#[ctor::ctor]
fn pre_main() {
codex_process_hardening::pre_main_hardening();
}
```
## Recommended Approach
**Phase 1: Immediate (Low-Risk Migration)**
1. Depend on `codex-mcp-client` directly from code-rs
2. Test with default buffer size - may already be sufficient
3. If buffer issues arise, implement thin wrapper (Strategy A, Option 1)
4. Use `codex-process-hardening` via re-export
**Phase 2: Short-Term (1-2 weeks)**
1. Propose upstream PR for buffer configuration (Strategy B)
2. Migrate responses-api-proxy to use upstream with build-time binary rename
3. Document any remaining customization needs
**Phase 3: Long-Term Maintenance**
1. Establish weekly automated checks for upstream changes
2. Participate in upstream development for shared needs
3. Maintain minimal patch set (ideally zero) for fork-specific requirements
## Upstream Parity Maintenance Checklist
### Weekly Tasks
- [ ] Check codex-rs commits to mcp-client, responses-api-proxy, process-hardening
- [ ] Review upstream issues/PRs that may affect code-rs usage
- [ ] Run integration tests with latest upstream commits
- [ ] Update dependency pins if changes detected
### Per-Upstream-Release Tasks
- [ ] Review changelog for breaking changes
- [ ] Test buffer size behavior with new release
- [ ] Validate process hardening still meets security requirements
- [ ] Update code-rs documentation if API changes
- [ ] Run full test suite with new upstream version
### Monthly Tasks
- [ ] Measure and document any performance differences
- [ ] Review need for fork-specific customizations
- [ ] Propose upstream contributions for shared needs
- [ ] Audit dependency tree for security updates
### Monitoring Triggers
- [ ] Set up GitHub notifications for codex-rs MCP-related commits
- [ ] Create alerts for upstream security advisories
- [ ] Track upstream issue tracker for relevant bug reports
- [ ] Monitor codex-rs release notes for MCP changes
### Testing Requirements
- [ ] Verify large tool response handling (>1MB payloads)
- [ ] Test MCP client with various server implementations
- [ ] Validate process hardening on all target platforms (Linux, macOS, Windows)
- [ ] Benchmark buffer size impact on performance
- [ ] Security audit of responses-api-proxy authentication
### Documentation Maintenance
- [ ] Keep this document updated with any new divergences
- [ ] Document reasons for any fork-specific patches
- [ ] Maintain migration guide for upstream API changes
- [ ] Track technical debt and sunset timeline for workarounds
## Risk Assessment
| Risk | Likelihood | Impact | Mitigation |
|------|------------|--------|------------|
| Upstream breaks API | Low | High | Pin versions, automated testing |
| Buffer size regression | Medium | Medium | Integration tests with large payloads |
| Binary rename conflicts | Low | Low | Build script validation |
| Security patch delay | Low | High | Automated weekly sync, subscribe to advisories |
| Upstream abandonment | Very Low | High | Fork maintained as fallback (current state) |
## Conclusion
**The investigation reveals that code-rs can effectively reuse codex-rs MCP components with minimal overhead.** The primary difference (1MB buffer) is a trivial configuration change that can be:
1. Tested to determine if even necessary
2. Implemented as a thin wrapper
3. Proposed upstream for mutual benefit
**Recommended Action**: Proceed with **Strategy A** (Direct Upstream Dependency) for immediate benefits, while pursuing **Strategy B** (Feature-Flag in Upstream) for long-term sustainability.
**Expected Outcome**: Reduced maintenance burden, automatic security updates, and stronger collaboration with upstream codex-rs development.
scripts/upstream-merge/diff-crates.sh
Executable
+173
View File
@@ -0,0 +1,173 @@
#!/usr/bin/env bash
set -euo pipefail
# diff-crates.sh: Compare codex-rs vs code-rs per crate
#
# Usage:
# ./scripts/upstream-merge/diff-crates.sh [crate-name]
# ./scripts/upstream-merge/diff-crates.sh --all
# ./scripts/upstream-merge/diff-crates.sh --summary
#
# Examples:
# ./scripts/upstream-merge/diff-crates.sh core
# ./scripts/upstream-merge/diff-crates.sh --all
# ./scripts/upstream-merge/diff-crates.sh --summary
ROOT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/../.." >/dev/null 2>&1 && pwd)"
cd "$ROOT_DIR"
OUTPUT_DIR=".github/auto/upstream-diffs"
mkdir -p "$OUTPUT_DIR"
# List of crates that exist in both codex-rs and code-rs
SHARED_CRATES=(
"ansi-escape"
"app-server"
"app-server-protocol"
"apply-patch"
"arg0"
"backend-client"
"chatgpt"
"cli"
"cloud-tasks"
"cloud-tasks-client"
"common"
"core"
"exec"
"execpolicy"
"file-search"
"git-apply"
"git-tooling"
"linux-sandbox"
"login"
"mcp-client"
"mcp-server"
"mcp-types"
"ollama"
"otel"
"protocol"
"protocol-ts"
"process-hardening"
"rmcp-client"
"responses-api-proxy"
"tui"
)
# Function to diff a single crate
diff_crate() {
local crate_name="$1"
local codex_path="codex-rs/${crate_name}"
local code_path="code-rs/${crate_name}"
# Check if both directories exist
if [[ ! -d "$codex_path" ]]; then
echo "⚠️ Warning: $codex_path does not exist"
return 1
fi
if [[ ! -d "$code_path" ]]; then
echo "⚠️ Warning: $code_path does not exist (fork-only crate)"
return 1
fi
local output_file="${OUTPUT_DIR}/${crate_name}.diff"
echo "📊 Comparing ${crate_name}..."
# Generate diff with context
if diff -Naur --exclude="target" --exclude="*.lock" --exclude="node_modules" \
"$codex_path" "$code_path" > "$output_file" 2>&1; then
echo " ✅ No differences found"
rm "$output_file"
return 0
else
local line_count=$(wc -l < "$output_file")
echo " 📝 Differences found: ${line_count} lines written to ${output_file}"
return 0
fi
}
# Function to generate summary
generate_summary() {
echo "📋 Generating diff summary..."
local summary_file="${OUTPUT_DIR}/SUMMARY.md"
cat > "$summary_file" <<'HEADER'
# Upstream Diff Summary
Generated: $(date -u +%Y-%m-%d\ %H:%M:%S\ UTC)
This report compares `codex-rs` (upstream baseline) vs `code-rs` (fork) for each shared crate.
## Overview
HEADER
echo "| Crate | Status | Diff Size | Notes |" >> "$summary_file"
echo "|-------|--------|-----------|-------|" >> "$summary_file"
for crate in "${SHARED_CRATES[@]}"; do
local diff_file="${OUTPUT_DIR}/${crate}.diff"
if [[ ! -f "$diff_file" ]]; then
echo "| ${crate} | ✅ Identical | 0 lines | - |" >> "$summary_file"
else
local line_count=$(wc -l < "$diff_file")
echo "| ${crate} | 📝 Differs | ${line_count} lines | See \`${crate}.diff\` |" >> "$summary_file"
fi
done
echo "" >> "$summary_file"
echo "## Crates with Differences" >> "$summary_file"
echo "" >> "$summary_file"
for crate in "${SHARED_CRATES[@]}"; do
local diff_file="${OUTPUT_DIR}/${crate}.diff"
if [[ -f "$diff_file" ]]; then
echo "### ${crate}" >> "$summary_file"
echo "" >> "$summary_file"
echo "Diff file: \`${crate}.diff\`" >> "$summary_file"
echo "" >> "$summary_file"
# Extract added/removed line counts
local added=$(grep -c "^+" "$diff_file" || echo "0")
local removed=$(grep -c "^-" "$diff_file" || echo "0")
echo "- Lines added: ${added}" >> "$summary_file"
echo "- Lines removed: ${removed}" >> "$summary_file"
echo "" >> "$summary_file"
fi
done
echo "✅ Summary written to ${summary_file}"
cat "$summary_file"
}
# Main logic
main() {
if [[ $# -eq 0 ]]; then
echo "Usage: $0 [crate-name | --all | --summary]"
echo ""
echo "Available crates:"
printf " %s\n" "${SHARED_CRATES[@]}"
exit 1
fi
case "$1" in
--all)
echo "🔍 Comparing all shared crates..."
for crate in "${SHARED_CRATES[@]}"; do
diff_crate "$crate"
done
generate_summary
;;
--summary)
generate_summary
;;
*)
diff_crate "$1"
;;
esac
}
main "$@"
scripts/upstream-merge/highlight-critical-changes.sh
Executable
+224
View File
@@ -0,0 +1,224 @@
#!/usr/bin/env bash
set -euo pipefail
# highlight-critical-changes.sh: Identify changes affecting prompts/API/executor
#
# This script searches for changes in critical subsystems that affect:
# - System prompts and tool definitions
# - API surface (protocol, event types)
# - Execution flow (codex.rs, exec, apply-patch)
#
# Usage:
# ./scripts/upstream-merge/highlight-critical-changes.sh [crate-name]
# ./scripts/upstream-merge/highlight-critical-changes.sh --all
#
# Examples:
# ./scripts/upstream-merge/highlight-critical-changes.sh core
# ./scripts/upstream-merge/highlight-critical-changes.sh --all
ROOT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/../.." >/dev/null 2>&1 && pwd)"
cd "$ROOT_DIR"
OUTPUT_DIR=".github/auto/upstream-diffs"
CRITICAL_DIR="${OUTPUT_DIR}/critical-changes"
mkdir -p "$CRITICAL_DIR"
# Critical files and patterns to watch
declare -A CRITICAL_PATTERNS=(
# Prompts and tool definitions
["prompts"]="prompts/.*\.md"
["openai_tools"]="codex-rs/core/src/openai_tools\.rs|code-rs/core/src/openai_tools\.rs"
["agent_tool"]="codex-rs/core/src/agent_tool\.rs|code-rs/core/src/agent_tool\.rs"
# Protocol and API surface
["protocol"]="codex-rs/core/src/protocol\.rs|code-rs/core/src/protocol\.rs"
["app_server_protocol"]="codex-rs/app-server-protocol/src/.*|code-rs/app-server-protocol/src/.*"
["mcp_types"]="codex-rs/mcp-types/src/.*|code-rs/mcp-types/src/.*"
# Executor and core logic
["codex_main"]="codex-rs/core/src/codex\.rs|code-rs/core/src/codex\.rs"
["exec"]="codex-rs/exec/src/.*|code-rs/exec/src/.*"
["apply_patch"]="codex-rs/apply-patch/src/.*|code-rs/apply-patch/src/.*"
["acp"]="codex-rs/core/src/acp\.rs|code-rs/core/src/acp\.rs"
# Config and behavior toggles
["config"]="codex-rs/core/src/config.*\.rs|code-rs/core/src/config.*\.rs"
)
# Color codes for output
RED='\033[0;31m'
YELLOW='\033[1;33m'
GREEN='\033[0;32m'
BLUE='\033[0;34m'
NC='\033[0m' # No Color
# Function to analyze a diff file for critical changes
analyze_diff() {
local crate_name="$1"
local diff_file="${OUTPUT_DIR}/${crate_name}.diff"
if [[ ! -f "$diff_file" ]]; then
echo -e "${GREEN}✅ ${crate_name}: No differences${NC}"
return 0
fi
echo -e "${BLUE}🔍 Analyzing ${crate_name}...${NC}"
local critical_file="${CRITICAL_DIR}/${crate_name}-critical.md"
local has_critical=0
# Initialize critical changes file
cat > "$critical_file" <<HEADER
# Critical Changes in ${crate_name}
Generated: $(date -u +%Y-%m-%d\ %H:%M:%S\ UTC)
HEADER
# Check for each critical pattern
for category in "${!CRITICAL_PATTERNS[@]}"; do
local pattern="${CRITICAL_PATTERNS[$category]}"
# Extract relevant sections from diff
if grep -E "^\+\+\+ |^--- " "$diff_file" | grep -E "$pattern" > /dev/null 2>&1; then
echo "## ${category}" >> "$critical_file"
echo "" >> "$critical_file"
# Extract the diff sections for files matching this pattern
awk -v pattern="$pattern" '
/^\+\+\+ |^--- / {
if ($0 ~ pattern) {
in_section=1
print $0
} else {
in_section=0
}
next
}
in_section {
print
if (/^diff --git/) in_section=0
}
' "$diff_file" >> "$critical_file"
echo "" >> "$critical_file"
has_critical=1
fi
done
# Check for specific high-impact changes
echo "## High-Impact Changes" >> "$critical_file"
echo "" >> "$critical_file"
# Function signature changes
if grep -E "^\+.*fn |^-.*fn " "$diff_file" | head -20 >> "$critical_file" 2>/dev/null; then
echo -e "${YELLOW}⚠️ Function signature changes detected${NC}"
has_critical=1
fi
# Enum/struct changes
if grep -E "^\+.*enum |^-.*enum |^\+.*struct |^-.*struct " "$diff_file" | head -20 >> "$critical_file" 2>/dev/null; then
echo -e "${YELLOW}⚠️ Type definition changes detected${NC}"
has_critical=1
fi
# Protocol message changes
if grep -E "^\+.*EventMsg|^-.*EventMsg|^\+.*Op::|^-.*Op::" "$diff_file" | head -20 >> "$critical_file" 2>/dev/null; then
echo -e "${YELLOW}⚠️ Protocol message changes detected${NC}"
has_critical=1
fi
if [[ $has_critical -eq 1 ]]; then
echo -e "${RED}🚨 Critical changes found - see ${critical_file}${NC}"
return 1
else
rm "$critical_file"
echo -e "${GREEN}✅ No critical changes detected${NC}"
return 0
fi
}
# Function to generate critical changes summary
generate_critical_summary() {
echo "📋 Generating critical changes summary..."
local summary_file="${CRITICAL_DIR}/CRITICAL-SUMMARY.md"
cat > "$summary_file" <<'HEADER'
# Critical Changes Summary
Generated: $(date -u +%Y-%m-%d\ %H:%M:%S\ UTC)
This report highlights changes in critical subsystems that affect:
- **Prompts**: System prompts and tool definitions
- **API**: Protocol definitions and event types
- **Executor**: Core execution logic and tool handlers
## Critical Crates
HEADER
local critical_count=0
for crate_file in "$CRITICAL_DIR"/*-critical.md; do
if [[ -f "$crate_file" ]]; then
local crate_name=$(basename "$crate_file" -critical.md)
echo "### ${crate_name}" >> "$summary_file"
echo "" >> "$summary_file"
echo "See [\`${crate_name}-critical.md\`](./${crate_name}-critical.md) for details." >> "$summary_file"
echo "" >> "$summary_file"
((critical_count++))
fi
done
if [[ $critical_count -eq 0 ]]; then
echo "✅ No critical changes detected across all crates." >> "$summary_file"
else
echo "" >> "$summary_file"
echo "## Action Items" >> "$summary_file"
echo "" >> "$summary_file"
echo "1. Review each critical change file for merge impact" >> "$summary_file"
echo "2. Identify which changes are upstream improvements vs. fork regressions" >> "$summary_file"
echo "3. Plan integration strategy (adopt, adapt, or preserve fork behavior)" >> "$summary_file"
echo "4. Update \`docs/code-crate-parity-tracker.md\` with findings" >> "$summary_file"
fi
echo "✅ Critical summary written to ${summary_file}"
cat "$summary_file"
}
# Main logic
main() {
# Check if diff files exist
if [[ ! -d "$OUTPUT_DIR" ]] || [[ -z "$(ls -A $OUTPUT_DIR/*.diff 2>/dev/null)" ]]; then
echo "⚠️ No diff files found. Run diff-crates.sh first:"
echo " ./scripts/upstream-merge/diff-crates.sh --all"
exit 1
fi
if [[ $# -eq 0 ]]; then
echo "Usage: $0 [crate-name | --all]"
echo ""
echo "Available options:"
echo " --all Analyze all crate diffs"
echo " <crate-name> Analyze specific crate"
exit 1
fi
case "$1" in
--all)
echo "🔍 Analyzing all crate diffs for critical changes..."
for diff_file in "$OUTPUT_DIR"/*.diff; do
if [[ -f "$diff_file" ]]; then
local crate_name=$(basename "$diff_file" .diff)
analyze_diff "$crate_name" || true
fi
done
generate_critical_summary
;;
*)
analyze_diff "$1"
;;
esac
}
main "$@"
scripts/upstream-merge/log-merge.sh
Executable
+266
View File
@@ -0,0 +1,266 @@
#!/usr/bin/env bash
set -euo pipefail
# log-merge.sh: Log upstream merge activity and decisions
#
# This script helps track merge decisions, conflicts, and resolutions
# for future reference. It creates structured logs that can be referenced
# when doing subsequent merges.
#
# Usage:
# ./scripts/upstream-merge/log-merge.sh init <upstream-ref>
# ./scripts/upstream-merge/log-merge.sh note <category> <message>
# ./scripts/upstream-merge/log-merge.sh decision <crate> <action> <reason>
# ./scripts/upstream-merge/log-merge.sh finalize
#
# Examples:
# ./scripts/upstream-merge/log-merge.sh init upstream/main
# ./scripts/upstream-merge/log-merge.sh note conflict "core/src/codex.rs has protocol changes"
# ./scripts/upstream-merge/log-merge.sh decision core preserve "Keep fork's QueueUserInput support"
# ./scripts/upstream-merge/log-merge.sh finalize
ROOT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/../.." >/dev/null 2>&1 && pwd)"
cd "$ROOT_DIR"
LOGS_DIR="docs/maintenance/upstream-merge-logs"
mkdir -p "$LOGS_DIR"
# Get current timestamp
timestamp() {
date -u +%Y-%m-%d\ %H:%M:%S\ UTC
}
# Get current date for log filename
log_date() {
date -u +%Y%m%d
}
# Initialize a new merge log
init_merge_log() {
local upstream_ref="$1"
local log_file="${LOGS_DIR}/merge-$(log_date).md"
if [[ -f "$log_file" ]]; then
echo "⚠️ Log file already exists: $log_file"
read -p "Overwrite? (y/N) " -n 1 -r
echo
if [[ ! $REPLY =~ ^[Yy]$ ]]; then
echo "Aborted."
exit 1
fi
fi
# Get upstream commit info
local upstream_commit=$(git rev-parse "$upstream_ref" 2>/dev/null || echo "unknown")
local upstream_date=$(git show -s --format=%ci "$upstream_ref" 2>/dev/null || echo "unknown")
local fork_commit=$(git rev-parse HEAD)
local fork_branch=$(git branch --show-current)
cat > "$log_file" <<HEADER
# Upstream Merge Log - $(log_date)
**Started:** $(timestamp)
**Upstream Ref:** ${upstream_ref} (${upstream_commit})
**Upstream Date:** ${upstream_date}
**Fork Branch:** ${fork_branch} (${fork_commit})
## Overview
This log tracks the merge of upstream changes into the fork.
## Merge Strategy
- [ ] Review diff summary (\`diff-crates.sh --all\`)
- [ ] Identify critical changes (\`highlight-critical-changes.sh --all\`)
- [ ] Plan integration approach
- [ ] Apply changes crate-by-crate
- [ ] Verify build and tests
- [ ] Update parity tracker
## Critical Changes
HEADER
echo "✅ Initialized merge log: $log_file"
echo "$log_file"
}
# Add a note to the current merge log
add_note() {
local category="$1"
shift
local message="$*"
local log_file=$(ls -t "${LOGS_DIR}"/merge-*.md 2>/dev/null | head -1)
if [[ -z "$log_file" ]]; then
echo "❌ No active merge log found. Run 'init' first."
exit 1
fi
echo "### ${category} - $(timestamp)" >> "$log_file"
echo "" >> "$log_file"
echo "${message}" >> "$log_file"
echo "" >> "$log_file"
echo "📝 Added note to $log_file"
}
# Log a merge decision for a specific crate
log_decision() {
local crate="$1"
local action="$2"
local reason="$3"
local log_file=$(ls -t "${LOGS_DIR}"/merge-*.md 2>/dev/null | head -1)
if [[ -z "$log_file" ]]; then
echo "❌ No active merge log found. Run 'init' first."
exit 1
fi
# Check if Decisions section exists, create if not
if ! grep -q "^## Decisions$" "$log_file"; then
echo "" >> "$log_file"
echo "## Decisions" >> "$log_file"
echo "" >> "$log_file"
echo "| Crate | Action | Reason | Timestamp |" >> "$log_file"
echo "|-------|--------|--------|-----------|" >> "$log_file"
fi
# Add decision row
echo "| ${crate} | ${action} | ${reason} | $(timestamp) |" >> "$log_file"
echo "📝 Logged decision for ${crate}: ${action}"
}
# Finalize the merge log
finalize_log() {
local log_file=$(ls -t "${LOGS_DIR}"/merge-*.md 2>/dev/null | head -1)
if [[ -z "$log_file" ]]; then
echo "❌ No active merge log found. Run 'init' first."
exit 1
fi
# Add completion timestamp
cat >> "$log_file" <<FOOTER
## Completion
**Finished:** $(timestamp)
### Post-Merge Checklist
- [ ] All critical changes reviewed and integrated
- [ ] Build passes (\`./build-fast.sh\`)
- [ ] Tests pass (\`./scripts/ci-tests.sh\`)
- [ ] Verification guards pass (\`./scripts/upstream-merge/verify.sh\`)
- [ ] Parity tracker updated (\`docs/code-crate-parity-tracker.md\`)
- [ ] Changelog updated if user-facing changes
- [ ] Commit created with merge details
### Files Modified
\`\`\`
$(git status --short)
\`\`\`
### Next Upstream Sync
Recommended: Check upstream weekly for new changes
Command: \`git fetch upstream && git log HEAD..upstream/main --oneline\`
FOOTER
echo "✅ Finalized merge log: $log_file"
echo ""
echo "Next steps:"
echo " 1. Complete post-merge checklist in the log"
echo " 2. Commit changes with reference to log file"
echo " 3. Update docs/code-crate-parity-tracker.md"
}
# Generate a summary of all merge logs
summarize_logs() {
echo "# Historical Merge Summary"
echo ""
echo "All upstream merge logs:"
echo ""
for log_file in "${LOGS_DIR}"/merge-*.md; do
if [[ -f "$log_file" ]]; then
local filename=$(basename "$log_file")
local date=$(echo "$filename" | sed 's/merge-\(.*\)\.md/\1/')
local started=$(grep "^\*\*Started:\*\*" "$log_file" | sed 's/\*\*Started:\*\* //')
local upstream=$(grep "^\*\*Upstream Ref:\*\*" "$log_file" | sed 's/\*\*Upstream Ref:\*\* //')
echo "## ${date}"
echo ""
echo "- File: [\`${filename}\`](./upstream-merge-logs/${filename})"
echo "- Started: ${started}"
echo "- Upstream: ${upstream}"
echo ""
fi
done
}
# Main logic
main() {
if [[ $# -eq 0 ]]; then
echo "Usage: $0 <command> [args...]"
echo ""
echo "Commands:"
echo " init <upstream-ref> Initialize new merge log"
echo " note <category> <message> Add a note to current log"
echo " decision <crate> <action> <reason> Log a merge decision"
echo " finalize Finalize current merge log"
echo " summary Show all merge logs"
echo ""
echo "Examples:"
echo " $0 init upstream/main"
echo " $0 note conflict 'core/src/codex.rs has protocol changes'"
echo " $0 decision core preserve 'Keep fork QueueUserInput support'"
echo " $0 finalize"
exit 1
fi
local command="$1"
shift
case "$command" in
init)
if [[ $# -lt 1 ]]; then
echo "❌ Usage: $0 init <upstream-ref>"
exit 1
fi
init_merge_log "$@"
;;
note)
if [[ $# -lt 2 ]]; then
echo "❌ Usage: $0 note <category> <message>"
exit 1
fi
add_note "$@"
;;
decision)
if [[ $# -lt 3 ]]; then
echo "❌ Usage: $0 decision <crate> <action> <reason>"
exit 1
fi
log_decision "$@"
;;
finalize)
finalize_log
;;
summary)
summarize_logs
;;
*)
echo "❌ Unknown command: $command"
exit 1
;;
esac
}
main "$@"