Files
engine/tools/upstream-map.sh
zeekay c7afc3bf94 tools: route upstream mistral.rs changes onto our tree
We forked at 7483b396f (2024-02-27) and renamed every crate, so git cannot follow a
change across the boundary: rename detection finds ONE rename between the merge-base
and HEAD, because the files diverged past any similarity threshold. Upstream is
therefore unreadable by tooling -- "what did they do here, and do we have it?" has no
mechanical answer. This is the missing dictionary. mistralrs-core maps to hanzo-engine;
the rest are mistralrs-X to hanzo-X.

  upstream-map.sh path <p>    upstream path -> ours
  upstream-map.sh show <sha>  what a commit touches, routed, marking what we deleted
  upstream-map.sh diff <sub>  their file against ours
  upstream-map.sh harvest     correctness commits since the fork, MoE and perf filtered out

For studying upstream, not merging it. Merging is a measured downgrade: same M4 Max,
same Qwen3-30B-A3B Q4_K_M GGUF, same bench flags, upstream decodes 0.20 t/s against our
94.4 -- Metal-mapped but with one CPU core pinned, a host-side per-expert MoE loop. Their
tree also carries no Vulkan and no ROCm. And merge-tree puts the whole merge at 921
conflict hunks over 449k lines, with 64% of the upstream files they touch absent from our
tree. Take their correctness fixes and new architectures; leave the MoE dispatch and the
perf work, which is what we are ahead of.

`git worktree add --detach ../engine-upstream upstream/master` gives a side-by-side tree
to read. It is gitignored.
2026-07-16 21:40:11 -07:00

83 lines
3.5 KiB
Bash

#!/usr/bin/env bash
# Route an upstream (EricLBuehler/mistral.rs) path or commit onto our tree.
#
# We forked at 7483b396f (2024-02-27) and renamed every crate, so `git log` cannot follow a change
# across the boundary: rename detection finds ONE rename between the merge-base and HEAD because the
# files diverged far past any similarity threshold. That makes upstream unreadable-by-tooling and is
# why "what did they do here?" is hard to answer. This is the missing dictionary.
#
# This exists to STUDY upstream, not to merge it. A wholesale merge is a measured downgrade: on an
# M4 Max, identical Qwen3-30B-A3B Q4_K_M GGUF and identical bench flags, upstream decodes at 0.20 t/s
# (Metal-mapped, one CPU core pinned -- a host-side per-expert MoE loop) against our 94.4. Their tree
# also has zero Vulkan and zero ROCm. Harvest correctness fixes and new architectures; never their
# MoE/expert dispatch or perf work.
#
# ./tools/upstream-map.sh path mistralrs-core/src/foo.rs # -> our path
# ./tools/upstream-map.sh show <upstream-sha> # what a commit touches, routed to ours
# ./tools/upstream-map.sh diff <subsystem> # their file vs ours, side by side
# ./tools/upstream-map.sh harvest # correctness commits we may not have
set -euo pipefail
cd "$(git rev-parse --show-toplevel)"
UP=upstream/master
# The dictionary. Mostly mistralrs-X -> hanzo-X; `core` is the exception that matters most.
map_path() {
sed -E \
-e 's|^mistralrs-core/|hanzo-engine/|' \
-e 's|^mistralrs-cli/|hanzo-cli/|' \
-e 's|^mistralrs-server-core/|hanzo-server-core/|' \
-e 's|^mistralrs-quant/|hanzo-quant/|' \
-e 's|^mistralrs-vision/|hanzo-vision/|' \
-e 's|^mistralrs-audio/|hanzo-audio/|' \
-e 's|^mistralrs-paged-attn/|hanzo-paged-attn/|' \
-e 's|^mistralrs-flash-attn/|hanzo-flash-attn/|' \
-e 's|^mistralrs-code-exec/|hanzo-code-exec/|' \
-e 's|^mistralrs-mcp/|hanzo-llm-mcp/|' \
-e 's|^mistralrs-macros/|hanzo-macros/|' \
-e 's|^mistralrs-bench/|hanzo-bench/|' \
-e 's|^mistralrs/|hanzo/|'
}
case "${1:-}" in
path) echo "$2" | map_path ;;
show)
sha="$2"
git log -1 --format='%h %ad %s' --date=short "$sha"
echo
git show --stat --format= "$sha" | sed -E 's/\|.*//' | awk 'NF' | while read -r f; do
ours=$(echo "$f" | map_path)
if [ -e "$ours" ]; then st="EXISTS"; else st="ABSENT (we deleted/rewrote it -> likely a NO-OP)"; fi
printf ' %-52s -> %-52s %s\n' "$f" "$ours" "$st"
done
;;
diff)
sub="$2"
their=$(git ls-tree -r --name-only "$UP" | grep -i "$sub" | head -1)
[ -n "$their" ] || { echo "no upstream file matching '$sub'"; exit 1; }
ours=$(echo "$their" | map_path)
echo "theirs: $their"
echo "ours: $ours $([ -e "$ours" ] || echo '(ABSENT)')"
[ -e "$ours" ] || exit 0
diff -u <(git show "$UP:$their") "$ours" || true
;;
harvest)
# Correctness-signal commits since the fork. Perf/MoE deliberately excluded: we are ~470x ahead
# of their MoE path, so importing it is the regression, not the fix.
git fetch upstream --quiet
git log --oneline HEAD.."$UP" \
--grep='fix' --grep='correctness' --grep='regression' --grep='panic' --grep='unsound' \
--grep='overflow' --grep='incorrect' --grep='wrong' --grep='leak' --grep='race' \
--regexp-ignore-case \
| grep -viE 'moe|expert|perf|speed|faster|optimi' \
| head -60
echo
echo "Run monthly. Route each with: $0 show <sha>. ABSENT => we rewrote that code; skip."
;;
*) sed -n '2,26p' "$0"; exit 1 ;;
esac