Files
ml/scripts/ultimate_setup.sh
zeekayandHanzo Dev 6a363f7ee3 merge: upstream main ddf1b879 and the aarch64 repack branch 35d7ae7c
Three-way per file through the rename: base 39355c6c91, theirs the upstream
merge of main with 35d7ae7c (via 7ef6a694d), ours origin/main.

Lands the barrier thread pool, runtime CPU feature dispatch, aarch64 and x86
repacked quantized kernels (indexed_gemv, gemv_fused_shared_lhs), quantized
embedding on CPU/Metal/CUDA, paged flash-attn kernels, cutile fused MoE, the
in-place op aliasing API, parking_lot storage locks, Metal buffer builders and
debug labels, safetensors 0.8, hf-hub 1.0, tokenizers 0.23.

Ours keeps ROCm, Vulkan, wgpu, the IQ/MXFP4/NVFP4/TQ dtypes, and the i-quant
weight/activation block split in matmul. Our backends ride InplaceOpN and the
quantized embedding; Metal get_rows gains the IQ kernels. ug leaves, as upstream.

Co-authored-by: Hanzo Dev <[email protected]>
2026-09-15 15:52:28 -07:00

79 lines
2.5 KiB
Bash
Executable File

#!/bin/bash
# Hanzo ML Setup Script
# Updates from upstream Hanzo, checks compilation, sets up training
set -e
HANZO_ML_DIR="/Users/z/work/hanzo/ml"
UPSTREAM_HANZO_ML="/Users/z/work/hf/hanzo-ml"
ZEN_DATASET="/Users/z/work/zen/zen-agentic-dataset"
HANZO_ENGINE="/Users/z/work/hanzo/engine"
cd "$HANZO_ML_DIR"
# Pull latest upstream changes
echo "Pulling latest Hanzo upstream changes..."
if [ -d "$UPSTREAM_HANZO_ML" ]; then
cd "$UPSTREAM_HANZO_ML"
git pull origin main || true
LATEST_COMMIT=$(git rev-parse HEAD)
echo "Latest upstream: $LATEST_COMMIT"
cd "$HANZO_ML_DIR"
fi
# Test compilation
echo "Testing hanzo-training compilation..."
if cargo check --package hanzo-training --quiet 2>/dev/null; then
echo "hanzo-training compiles successfully"
else
echo "hanzo-training has compilation errors"
cargo check --package hanzo-training 2>&1 | head -20
fi
# Check published crates
echo "Checking published crates..."
CRATES=("hanzo-ml" "hanzo-nn" "hanzo-transformers" "hanzo-datasets")
for crate in "${CRATES[@]}"; do
if cargo search "$crate" --limit 1 2>/dev/null | grep -q "0.9.2-alpha.2"; then
echo " $crate v0.9.2-alpha.2 published"
else
echo " $crate v0.9.2-alpha.2 not found"
fi
done
# Verify licenses
BSD_COUNT=$(find . -name "Cargo.toml" -not -path "./target/*" -exec grep -l "BSD-3-Clause" {} \; 2>/dev/null | wc -l)
echo "Found $BSD_COUNT crates with BSD-3-Clause license"
# Check dataset
if [ -d "$ZEN_DATASET" ]; then
echo "Zen Agentic Dataset found"
DATASET_SIZE=$(find "$ZEN_DATASET" -name "*.json*" 2>/dev/null | wc -l)
echo " Dataset files: $DATASET_SIZE"
else
echo "Zen Agentic Dataset not found at $ZEN_DATASET"
fi
# Check engine
if [ -d "$HANZO_ENGINE" ]; then
echo "Hanzo Engine found"
if [ -f "$HANZO_ENGINE/Cargo.toml" ]; then
echo " Engine version: $(grep '^version' "$HANZO_ENGINE/Cargo.toml" | head -1)"
fi
else
echo "Hanzo Engine not found at $HANZO_ENGINE"
fi
# Build optimized training binaries
echo "Building hanzo-training with release optimizations..."
cargo build --release --package hanzo-training --features "metal" 2>/dev/null || echo "Build failed (expected if metal feature unavailable)"
# The package built above is hanzo-training, so its binary is target/release/hanzo-training.
# target/release/hanzo-train belongs to the hanzo-train package (the DSpark draft trainer).
if [ -f "target/release/hanzo-training" ]; then
echo "hanzo-training binary ready"
ls -lh target/release/hanzo-training
fi
echo "Setup complete"