mirror of
https://github.com/hanzoai/dev.git
synced 2026-10-25 03:32:32 +00:00
Michael BolinandGitHub
af37785bca
We try to ensure ripgrep (`rg`) is provided with Codex. - For `brew`, we declare it as a dependency of our formula: https://github.com/Homebrew/homebrew-core/blob/08d82d8b006a19efbe234477bc8b18d35b5fef50/Formula/c/codex.rb#L24 - For `npm`, we declare `@vscode/ripgrep` as a dependency, which installs the platform-specific binary as part of a `postinstall` script: https://github.com/openai/codex/blob/fdb8dadcae9f8eec91bc3eb5a17b3f9b19e28505/codex-cli/package.json#L22 - Users who download the CLI directly from GitHub Releases are on their own. In practice, I have seen `@vscode/ripgrep` fail on occasion. Here is a trace from a GitHub workflow: ``` npm error code 1 npm error path /Users/runner/hostedtoolcache/node/20.19.5/arm64/lib/node_modules/@openai/codex/node_modules/@vscode/ripgrep npm error command failed npm error command sh -c node ./lib/postinstall.js npm error Finding release for v13.0.0-13 npm error GET https://api.github.com/repos/microsoft/ripgrep-prebuilt/releases/tags/v13.0.0-13 npm error Deleting invalid download cache npm error Download attempt 1 failed, retrying in 2 seconds... npm error Finding release for v13.0.0-13 npm error GET https://api.github.com/repos/microsoft/ripgrep-prebuilt/releases/tags/v13.0.0-13 npm error Deleting invalid download cache npm error Download attempt 2 failed, retrying in 4 seconds... npm error Finding release for v13.0.0-13 npm error GET https://api.github.com/repos/microsoft/ripgrep-prebuilt/releases/tags/v13.0.0-13 npm error Deleting invalid download cache npm error Download attempt 3 failed, retrying in 8 seconds... npm error Finding release for v13.0.0-13 npm error GET https://api.github.com/repos/microsoft/ripgrep-prebuilt/releases/tags/v13.0.0-13 npm error Deleting invalid download cache npm error Download attempt 4 failed, retrying in 16 seconds... npm error Finding release for v13.0.0-13 npm error GET https://api.github.com/repos/microsoft/ripgrep-prebuilt/releases/tags/v13.0.0-13 npm error Deleting invalid download cache npm error Error: Request failed: 403 ``` To eliminate this error, this PR changes things so that we vendor the `rg` binary into https://www.npmjs.com/package/@openai/codex so it is guaranteed to be included when a user runs `npm i -g @openai/codex`. The downside of this approach is the increase in package size: we include the `rg` executable for six architectures (in addition to the six copies of `codex` we already include). In a follow-up, I plan to add support for "slices" of our npm module, so that soon users will be able to do: ``` npm install -g @openai/codex@aarch64-apple-darwin ``` Admittedly, this is a sizable change and I tried to clean some things up in the process: - `install_native_deps.sh` has been replaced by `install_native_deps.py` - `stage_release.sh` and `stage_rust_release.py` has been replaced by `build_npm_package.py` We now vendor in a DotSlash file for ripgrep (as a modest attempt to facilitate local testing) and then build up the extension by: - creating a temp directory and copying `package.json` over to it with the target value for `"version"` - finding the GitHub workflow that corresponds to the `--release-version` and copying the various `codex` artifacts to respective `vendor/TARGET_TRIPLE/codex` folder - downloading the `rg` artifacts specified in the DotSlash file and copying them over to the respective `vendor/TARGET_TRIPLE/path` folder - if `--pack-output` is specified, runs `npm pack` on the temp directory To test, I downloaded the artifact produced by this CI job: https://github.com/openai/codex/actions/runs/17961595388/job/51085840022?pr=3660 and verified that `node ./bin/codex.js 'which -a rg'` worked as intended.
92 lines
2.9 KiB
Bash
Executable File
92 lines
2.9 KiB
Bash
Executable File
#!/usr/bin/env bash
|
||
|
||
# Install native runtime dependencies for codex-cli.
|
||
#
|
||
# Usage
|
||
# install_native_deps.sh [--workflow-url URL] [CODEX_CLI_ROOT]
|
||
#
|
||
# The optional RELEASE_ROOT is the path that contains package.json. Omitting
|
||
# it installs the binaries into the repository's own bin/ folder to support
|
||
# local development.
|
||
|
||
set -euo pipefail
|
||
|
||
# ------------------
|
||
# Parse arguments
|
||
# ------------------
|
||
|
||
CODEX_CLI_ROOT=""
|
||
|
||
# Until we start publishing stable GitHub releases, we have to grab the binaries
|
||
# from the GitHub Action that created them. Update the URL below to point to the
|
||
# appropriate workflow run:
|
||
WORKFLOW_URL="" # Optional override; if empty, caller must pass --workflow-url
|
||
|
||
while [[ $# -gt 0 ]]; do
|
||
case "$1" in
|
||
--workflow-url)
|
||
shift || { echo "--workflow-url requires an argument"; exit 1; }
|
||
if [ -n "$1" ]; then
|
||
WORKFLOW_URL="$1"
|
||
fi
|
||
;;
|
||
*)
|
||
if [[ -z "$CODEX_CLI_ROOT" ]]; then
|
||
CODEX_CLI_ROOT="$1"
|
||
else
|
||
echo "Unexpected argument: $1" >&2
|
||
exit 1
|
||
fi
|
||
;;
|
||
esac
|
||
shift
|
||
done
|
||
|
||
# ----------------------------------------------------------------------------
|
||
# Determine where the binaries should be installed.
|
||
# ----------------------------------------------------------------------------
|
||
|
||
if [ -n "$CODEX_CLI_ROOT" ]; then
|
||
# The caller supplied a release root directory.
|
||
BIN_DIR="$CODEX_CLI_ROOT/bin"
|
||
else
|
||
# No argument; fall back to the repo’s own bin directory.
|
||
# Resolve the path of this script, then walk up to the repo root.
|
||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||
CODEX_CLI_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
|
||
BIN_DIR="$CODEX_CLI_ROOT/bin"
|
||
fi
|
||
|
||
# Make sure the destination directory exists.
|
||
mkdir -p "$BIN_DIR"
|
||
|
||
# ----------------------------------------------------------------------------
|
||
# Download and decompress the artifacts from the GitHub Actions workflow.
|
||
# ----------------------------------------------------------------------------
|
||
|
||
WORKFLOW_ID="${WORKFLOW_URL##*/}"
|
||
|
||
ARTIFACTS_DIR="$(mktemp -d)"
|
||
trap 'rm -rf "$ARTIFACTS_DIR"' EXIT
|
||
|
||
# NB: The GitHub CLI `gh` must be installed and authenticated.
|
||
gh run download --dir "$ARTIFACTS_DIR" --repo just-every/code "$WORKFLOW_ID"
|
||
|
||
# x64 Linux
|
||
zstd -d "$ARTIFACTS_DIR/x86_64-unknown-linux-musl/code-x86_64-unknown-linux-musl.zst" \
|
||
-o "$BIN_DIR/code-x86_64-unknown-linux-musl"
|
||
# ARM64 Linux
|
||
zstd -d "$ARTIFACTS_DIR/aarch64-unknown-linux-musl/code-aarch64-unknown-linux-musl.zst" \
|
||
-o "$BIN_DIR/code-aarch64-unknown-linux-musl"
|
||
# x64 macOS
|
||
zstd -d "$ARTIFACTS_DIR/x86_64-apple-darwin/code-x86_64-apple-darwin.zst" \
|
||
-o "$BIN_DIR/code-x86_64-apple-darwin"
|
||
# ARM64 macOS
|
||
zstd -d "$ARTIFACTS_DIR/aarch64-apple-darwin/code-aarch64-apple-darwin.zst" \
|
||
-o "$BIN_DIR/code-aarch64-apple-darwin"
|
||
# x64 Windows
|
||
zstd -d "$ARTIFACTS_DIR/x86_64-pc-windows-msvc/code-x86_64-pc-windows-msvc.exe.zst" \
|
||
-o "$BIN_DIR/code-x86_64-pc-windows-msvc.exe"
|
||
|
||
echo "Installed native dependencies into $BIN_DIR"
|