mirror of
https://github.com/hanzoai/cpy3.git
synced 2026-10-30 18:42:30 +00:00
zeekayandGitHub
aa54594e88
* feat: Python 3.12+/3.13 compat + sub-interpreter wrapper
Hanzo fork of go-python/cpy3. See HANZO_CHANGES.md for full diff.
- Polyfill APIs removed in 3.12 (per-type ClearFreeList helpers) and
3.13 (PyEval_InitThreads, Py_SetPath/PythonHome/ProgramName,
PySys_SetArgv/AddWarnOption/AddXOption/SetPath,
Py_SetStandardStreamEncoding, PyEval_ThreadsInitialized) so existing
call sites compile against modern CPython. Polyfills route to the
PyConfig API where there's a meaningful replacement, return 0/no-op
otherwise.
- New PEP 684 SubInterpreter wrapper (subinterp.go) — recommended
primitive for tenant/per-worker isolation on 3.12+. Defaults follow
PEP 684 (own GIL, threads on, no daemon threads, refuse non-isolated
C extensions). Falls back to Py_NewInterpreter on 3.11 and older.
- Free-threading awareness (PEP 703) — IsGILDisabled() reports at
runtime whether the linked CPython was built with --disable-gil.
- pkg-config switched from "python3" to "python3-embed" so libpython
links cleanly on Linux distros that split the embed flags.
- Module path: github.com/go-python/cpy3 -> github.com/hanzoai/cpy3.
Go 1.21+. Package name (python3) unchanged.
- CI matrix: Python {3.11, 3.12, 3.13} x Go {1.21, 1.22} on Ubuntu.
License remains MIT. Original go-python team credited in HANZO_CHANGES.md
and every modified file's header.
* feat: JSON hot-path helper + 3.13t build script
- LoadSource + CallJSONFunctionByName: single-cgo-crossing helpers
for the common "JSON-in, JSON-out" extension embedding contract.
Roughly 2x faster than the long form via individual wrappers and
stack-safer under sustained load (macOS SIGBUS at ~400k iters of
the 9-call long form is mitigated by collapsing to 1 cgo call).
- ErrUnknownFunction: sentinel error for missing/non-callable
lookups so callers can errors.Is.
- scripts/build-python313t.sh: idempotent local build of CPython
3.13.x with --disable-gil. Writes a python3-embed.pc so callers
steer via PKG_CONFIG_PATH.
- HANZO_CHANGES.md + README updated.
- .gitignore: exclude python313t-build/ python313t-src/.
* ci: split test runs by Python global state
TestSubInterpreter_* and TestCallJSONFunctionByName_* / TestLoadSource_*
share global Py state with the data-type/error tests. Running them
in the same `go test` invocation can hang on darwin after the
data-type tests have already touched the main interpreter. Split
into two test steps so each starts from a fresh process.
* test: move json_hotpath tests in-package + share Py init
Move from package python3_test to package python3 so they share
Py_Initialize() state with subinterp_test.go. Each test pins its
own OS thread, calls Py_Initialize (idempotent), creates a fresh
sub-interp, and unpins on return. No PyEval_SaveThread in teardown
— the sub-interp's Close already released the calling thread's GIL.
This eliminates a SIGSEGV-on-second-test pattern caused by trying
to PyEval_SaveThread a state that's already been saved.
* docs: README example for the JSON hot path
Show LoadSource + CallJSONFunctionByName end-to-end with luxfi/log
for context. The helpers themselves take no logger dependency.
137 lines
4.1 KiB
Bash
Executable File
137 lines
4.1 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# Build CPython 3.13.x with --disable-gil (PEP 703 free-threading).
|
|
#
|
|
# Output: ./python313t-build/bin/python3.13t plus a usable
|
|
# pkg-config/embed setup at:
|
|
# ./python313t-build/lib/pkgconfig/python-3.13t-embed.pc
|
|
#
|
|
# Idempotent: re-running is a no-op if the binary is present and
|
|
# correctly reports "GIL disabled: True".
|
|
#
|
|
# Hosts: macOS 14+ (Apple Silicon or Intel) and Linux. Requires a
|
|
# working C toolchain (xcode-select on macOS; build-essential on
|
|
# Debian/Ubuntu) and the usual libssl/libffi headers for stdlib
|
|
# extensions.
|
|
#
|
|
# Time budget: 10-20 minutes for a clean build on a modern laptop.
|
|
|
|
set -euo pipefail
|
|
|
|
PYTHON_VERSION="${PYTHON_VERSION:-3.13.5}"
|
|
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
|
|
ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
|
|
PREFIX="$ROOT/python313t-build"
|
|
SRC_DIR="$ROOT/python313t-src"
|
|
TARBALL="Python-${PYTHON_VERSION}.tgz"
|
|
URL="https://www.python.org/ftp/python/${PYTHON_VERSION}/${TARBALL}"
|
|
|
|
# --- Idempotency check ---
|
|
if [[ -x "$PREFIX/bin/python3.13t" ]]; then
|
|
if "$PREFIX/bin/python3.13t" -c 'import sys; raise SystemExit(0 if not sys._is_gil_enabled() else 1)'; then
|
|
echo "[build-python313t] already built at $PREFIX (GIL disabled). Nothing to do."
|
|
exit 0
|
|
fi
|
|
echo "[build-python313t] existing binary does not report GIL disabled; rebuilding."
|
|
fi
|
|
|
|
# --- Toolchain probe ---
|
|
if ! command -v curl >/dev/null 2>&1 && ! command -v wget >/dev/null 2>&1; then
|
|
echo "build-python313t: need curl or wget" >&2; exit 1
|
|
fi
|
|
if ! command -v make >/dev/null 2>&1; then
|
|
echo "build-python313t: need make" >&2; exit 1
|
|
fi
|
|
if ! command -v cc >/dev/null 2>&1; then
|
|
echo "build-python313t: need a C compiler (cc)" >&2; exit 1
|
|
fi
|
|
|
|
# --- Parallelism ---
|
|
if command -v sysctl >/dev/null 2>&1 && sysctl -n hw.ncpu >/dev/null 2>&1; then
|
|
JOBS="$(sysctl -n hw.ncpu)"
|
|
elif command -v nproc >/dev/null 2>&1; then
|
|
JOBS="$(nproc)"
|
|
else
|
|
JOBS=4
|
|
fi
|
|
|
|
mkdir -p "$SRC_DIR"
|
|
cd "$SRC_DIR"
|
|
|
|
if [[ ! -f "$TARBALL" ]]; then
|
|
echo "[build-python313t] fetching $URL"
|
|
if command -v curl >/dev/null 2>&1; then
|
|
curl -fSLo "$TARBALL" "$URL"
|
|
else
|
|
wget -O "$TARBALL" "$URL"
|
|
fi
|
|
fi
|
|
|
|
if [[ ! -d "Python-${PYTHON_VERSION}" ]]; then
|
|
tar -xzf "$TARBALL"
|
|
fi
|
|
|
|
cd "Python-${PYTHON_VERSION}"
|
|
|
|
# Honor SDKROOT on macOS so headers resolve (xcrun-style).
|
|
if [[ "$(uname -s)" == "Darwin" ]]; then
|
|
if [[ -z "${SDKROOT:-}" ]] && command -v xcrun >/dev/null 2>&1; then
|
|
export SDKROOT="$(xcrun --show-sdk-path 2>/dev/null || true)"
|
|
fi
|
|
fi
|
|
|
|
echo "[build-python313t] configuring (prefix=$PREFIX, jobs=$JOBS)"
|
|
./configure \
|
|
--disable-gil \
|
|
--enable-shared \
|
|
--with-ensurepip=no \
|
|
--prefix="$PREFIX"
|
|
|
|
echo "[build-python313t] make -j$JOBS"
|
|
make -j"$JOBS"
|
|
|
|
echo "[build-python313t] installing"
|
|
make install
|
|
|
|
# --- Verification ---
|
|
PYTHON_BIN="$PREFIX/bin/python3.13t"
|
|
if [[ ! -x "$PYTHON_BIN" ]]; then
|
|
# Some installs symlink python3.13t to python3.13 when GIL is disabled
|
|
if [[ -x "$PREFIX/bin/python3.13" ]]; then
|
|
ln -sf python3.13 "$PYTHON_BIN" || cp "$PREFIX/bin/python3.13" "$PYTHON_BIN"
|
|
fi
|
|
fi
|
|
|
|
echo "[build-python313t] verification:"
|
|
"$PYTHON_BIN" --version
|
|
"$PYTHON_BIN" -c 'import sys; print("GIL disabled:", not sys._is_gil_enabled())'
|
|
|
|
# --- Write a tagged pkg-config so cpy3 (which uses pkg-config:
|
|
# python3-embed) can be steered with PKG_CONFIG_PATH ---
|
|
PC_DIR="$PREFIX/lib/pkgconfig"
|
|
mkdir -p "$PC_DIR"
|
|
cat > "$PC_DIR/python3-embed.pc" <<EOF
|
|
# Generated by scripts/build-python313t.sh — points to the
|
|
# free-threaded CPython 3.13t build.
|
|
prefix=$PREFIX
|
|
exec_prefix=\${prefix}
|
|
libdir=\${exec_prefix}/lib
|
|
includedir=\${prefix}/include
|
|
|
|
Name: Python
|
|
Description: Embed free-threaded Python into an application
|
|
Requires:
|
|
Version: 3.13
|
|
Libs: -L\${libdir} -lpython3.13t
|
|
Cflags: -I\${includedir}/python3.13t
|
|
EOF
|
|
|
|
echo "[build-python313t] done."
|
|
echo " binary: $PYTHON_BIN"
|
|
echo " prefix: $PREFIX"
|
|
echo " pkgconfig: $PC_DIR/python3-embed.pc"
|
|
echo
|
|
echo "To benchmark pyvm against this Python, set:"
|
|
echo " export PKG_CONFIG_PATH=$PC_DIR:\${PKG_CONFIG_PATH:-}"
|
|
echo " export DYLD_LIBRARY_PATH=$PREFIX/lib:\${DYLD_LIBRARY_PATH:-} # macOS"
|
|
echo " export LD_LIBRARY_PATH=$PREFIX/lib:\${LD_LIBRARY_PATH:-} # Linux"
|