Files
dev/scripts/mcp_conformance
zeekayandHanzo Dev 6cbbea2d03 merge: just-every/code v0.6.169 (e4fe007150) — keep Hanzo identity
Our history is just-every's with author emails rewritten, so the graphs are
disjoint and git finds no merge base. Supplied one by hand: d1f1f51a41, the
just-every commit matching our Oct-2025 fork point (22 files apart). 9618
upstream commits, 7004 files, +1.81M/-159k, 37 conflicts.

Resolution:
- Upstream code where our side was the Oct-2025 prototype of a feature they
  have since reimplemented 3-5x larger (agent_runs, browser_sessions,
  history_cell/{agent,browser}, the vt100 harness and its snapshots).
- Upstream's config.rs module split (builder/defaults/sources/validation)
  kept; zen5-coder and the provider-follows-model rule re-applied on top.
- Ours where the file IS the identity: README, package.json, postinstall,
  release notes pipeline, integration-zed.
- Hybrid where upstream improved a file we had branded: updates.rs (their
  release-repo migration, our hanzoai/dev), release.yml (their fail-fast npm
  auth check, our package name), chatwidget.rs (their turn handling, our
  HANZO_USER_KEY hint), exec human output (their 237 lines, our banner).

Superseded and dropped deliberately: our resume_command_name() coder-alias
hack — upstream's derive_resume_command_name() reads the invocation basename
generically, which yields "dev" for us.

LLM.md now records the real topology and the sync point; AGENTS.md and
CLAUDE.md are symlinks to it again. Single parent on purpose: adopting
jecode/main as a second parent would re-import 13,742 commits carrying the
upstream author emails a prior rewrite deliberately removed.

Version 0.2.188 -> 0.2.189.

Co-authored-by: Hanzo Dev <[email protected]>
2026-08-05 23:09:12 -07:00
..

MCP client conformance

This directory tests the actual Codex executable against the official Model Context Protocol client conformance suite. It exercises the shipping legacy, intermediate 2025-11-25, and modern 2026-07-28 protocols, localhost HTTP, stdio, OAuth, and additional transport and security regression fixtures.

The official upstream suite is pinned to modelcontextprotocol/conformance@49103de6ed70804e940637bf3e9e29e4a3f54e64. Use Node.js 22 and Python 3.10 or later.

Run the conformance gate

First install the frozen workspace dependencies and build Codex:

pnpm install --frozen-lockfile
cargo build --locked --manifest-path codex-rs/Cargo.toml -p codex-cli --bin codex

From a published Codex checkout, run:

python3 scripts/mcp_conformance/run_codex_compliance.py \
  codex-rs/target/debug/codex \
  --conformance-cli node_modules/@modelcontextprotocol/conformance/dist/index.js \
  --baseline-report scripts/mcp_conformance/regression-baseline-v1.json \
  --report /tmp/codex-mcp-conformance.json

The positional executable can also point to an already built Codex binary. --conformance-cli selects the exact, lockfile-installed upstream JavaScript runner instead of downloading a moving version during a test.

What the baseline means

regression-baseline-v1.json is a compact, reviewed snapshot of the upstream revision, required protocol versions, HTTP and stdio transports, enabled modern feature, OAuth coverage, and individual passing and failing check identities.

The gate exits successfully only when:

  • The upstream suite and modern feature match the committed baseline.
  • The shipping legacy, intermediate, and modern protocols are actually tested.
  • The required HTTP, stdio, and authentication scenarios are actually run.
  • Every previously passing check still passes.
  • No additional check fails.

Existing known failures remain visible in the complete JSON report. In particular, success describes complete upstream conformance and regressionGate.success describes the no-new-regressions merge gate; the gate does not relabel an existing failure as a pass.

Create a compact baseline from a reviewed complete report without contacting the upstream suite again:

python3 scripts/mcp_conformance/run_codex_compliance.py \
  /absolute/path/to/codex \
  --baseline-report /absolute/path/to/full-conformance-report.json \
  --extract-baseline /tmp/mcp-conformance-regression-baseline-v1.json

Alternatively, add --write-baseline /tmp/mcp-conformance-regression-baseline-v1.json to a complete conformance run. Review every baseline change; do not regenerate it to conceal a regression.

Run the production reviewer regression gate

The separate reviewer gate tests the real Codex app-server across all three shipping, legacy, and modern protocol modes. It covers stdio and localhost HTTP, exact-integer tool and elicitation schemas, bounded multi-round requests, malformed discovery response IDs, repeated pagination cursors, SSE framing and keepalives, and catalog boundaries. In a published Codex checkout, run:

python3 scripts/mcp_conformance/review_regressions.py \
  /absolute/path/to/codex \
  --mode all \
  --baseline-report scripts/mcp_conformance/review-regression-baseline-v1.json \
  --report /tmp/codex-mcp-review-regressions.json

A complete, main-derived baseline records all 186 real check identities and all 21 required cases. Existing failures remain explicitly visible in the complete report; regressionGate.success: true means there are no newly failing or missing checks. Improvements are recorded under fixedChecks. The gate never classifies an existing failure as a passing check.

Extract a compact deterministic reviewer baseline from a reviewed complete production report without rerunning the client:

python3 scripts/mcp_conformance/review_regressions.py \
  /absolute/path/to/codex \
  --baseline-report /absolute/path/to/full-review-regressions.json \
  --extract-baseline /tmp/review-regression-baseline-v1.json

Review every baseline update. Do not regenerate a baseline to hide a regression.

Run the fixture self-tests

env PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 \
  python3 -m pytest -q scripts/mcp_conformance

Run the required SDK integration

The existing required SDK workflow runs the complete Python fixture self-tests. Its TypeScript job builds the actual Codex executable, sets CODEX_EXEC_PATH, installs the pinned upstream conformance runner, and runs both the official authenticated suite and the separate production reviewer regression matrix. Neither gate can be skipped. To reproduce the focused integration locally:

CODEX_EXEC_PATH=/absolute/path/to/codex \
  pnpm --filter @openai/codex-sdk test -- \
  --runInBand tests/mcpConformance.test.ts