mirror of
https://github.com/hanzoai/engine.git
synced 2026-11-20 14:17:52 +00:00
We forked at 7483b396f (2024-02-27) and renamed every crate, so git cannot follow a
change across the boundary: rename detection finds ONE rename between the merge-base
and HEAD, because the files diverged past any similarity threshold. Upstream is
therefore unreadable by tooling -- "what did they do here, and do we have it?" has no
mechanical answer. This is the missing dictionary. mistralrs-core maps to hanzo-engine;
the rest are mistralrs-X to hanzo-X.
upstream-map.sh path <p> upstream path -> ours
upstream-map.sh show <sha> what a commit touches, routed, marking what we deleted
upstream-map.sh diff <sub> their file against ours
upstream-map.sh harvest correctness commits since the fork, MoE and perf filtered out
For studying upstream, not merging it. Merging is a measured downgrade: same M4 Max,
same Qwen3-30B-A3B Q4_K_M GGUF, same bench flags, upstream decodes 0.20 t/s against our
94.4 -- Metal-mapped but with one CPU core pinned, a host-side per-expert MoE loop. Their
tree also carries no Vulkan and no ROCm. And merge-tree puts the whole merge at 921
conflict hunks over 449k lines, with 64% of the upstream files they touch absent from our
tree. Take their correctness fixes and new architectures; leave the MoE dispatch and the
perf work, which is what we are ahead of.
`git worktree add --detach ../engine-upstream upstream/master` gives a side-by-side tree
to read. It is gitignored.
83 lines
3.5 KiB
Bash
83 lines
3.5 KiB
Bash
#!/usr/bin/env bash
|
|
# Route an upstream (EricLBuehler/mistral.rs) path or commit onto our tree.
|
|
#
|
|
# We forked at 7483b396f (2024-02-27) and renamed every crate, so `git log` cannot follow a change
|
|
# across the boundary: rename detection finds ONE rename between the merge-base and HEAD because the
|
|
# files diverged far past any similarity threshold. That makes upstream unreadable-by-tooling and is
|
|
# why "what did they do here?" is hard to answer. This is the missing dictionary.
|
|
#
|
|
# This exists to STUDY upstream, not to merge it. A wholesale merge is a measured downgrade: on an
|
|
# M4 Max, identical Qwen3-30B-A3B Q4_K_M GGUF and identical bench flags, upstream decodes at 0.20 t/s
|
|
# (Metal-mapped, one CPU core pinned -- a host-side per-expert MoE loop) against our 94.4. Their tree
|
|
# also has zero Vulkan and zero ROCm. Harvest correctness fixes and new architectures; never their
|
|
# MoE/expert dispatch or perf work.
|
|
#
|
|
# ./tools/upstream-map.sh path mistralrs-core/src/foo.rs # -> our path
|
|
# ./tools/upstream-map.sh show <upstream-sha> # what a commit touches, routed to ours
|
|
# ./tools/upstream-map.sh diff <subsystem> # their file vs ours, side by side
|
|
# ./tools/upstream-map.sh harvest # correctness commits we may not have
|
|
set -euo pipefail
|
|
cd "$(git rev-parse --show-toplevel)"
|
|
|
|
UP=upstream/master
|
|
|
|
# The dictionary. Mostly mistralrs-X -> hanzo-X; `core` is the exception that matters most.
|
|
map_path() {
|
|
sed -E \
|
|
-e 's|^mistralrs-core/|hanzo-engine/|' \
|
|
-e 's|^mistralrs-cli/|hanzo-cli/|' \
|
|
-e 's|^mistralrs-server-core/|hanzo-server-core/|' \
|
|
-e 's|^mistralrs-quant/|hanzo-quant/|' \
|
|
-e 's|^mistralrs-vision/|hanzo-vision/|' \
|
|
-e 's|^mistralrs-audio/|hanzo-audio/|' \
|
|
-e 's|^mistralrs-paged-attn/|hanzo-paged-attn/|' \
|
|
-e 's|^mistralrs-flash-attn/|hanzo-flash-attn/|' \
|
|
-e 's|^mistralrs-code-exec/|hanzo-code-exec/|' \
|
|
-e 's|^mistralrs-mcp/|hanzo-llm-mcp/|' \
|
|
-e 's|^mistralrs-macros/|hanzo-macros/|' \
|
|
-e 's|^mistralrs-bench/|hanzo-bench/|' \
|
|
-e 's|^mistralrs/|hanzo/|'
|
|
}
|
|
|
|
case "${1:-}" in
|
|
path) echo "$2" | map_path ;;
|
|
|
|
show)
|
|
sha="$2"
|
|
git log -1 --format='%h %ad %s' --date=short "$sha"
|
|
echo
|
|
git show --stat --format= "$sha" | sed -E 's/\|.*//' | awk 'NF' | while read -r f; do
|
|
ours=$(echo "$f" | map_path)
|
|
if [ -e "$ours" ]; then st="EXISTS"; else st="ABSENT (we deleted/rewrote it -> likely a NO-OP)"; fi
|
|
printf ' %-52s -> %-52s %s\n' "$f" "$ours" "$st"
|
|
done
|
|
;;
|
|
|
|
diff)
|
|
sub="$2"
|
|
their=$(git ls-tree -r --name-only "$UP" | grep -i "$sub" | head -1)
|
|
[ -n "$their" ] || { echo "no upstream file matching '$sub'"; exit 1; }
|
|
ours=$(echo "$their" | map_path)
|
|
echo "theirs: $their"
|
|
echo "ours: $ours $([ -e "$ours" ] || echo '(ABSENT)')"
|
|
[ -e "$ours" ] || exit 0
|
|
diff -u <(git show "$UP:$their") "$ours" || true
|
|
;;
|
|
|
|
harvest)
|
|
# Correctness-signal commits since the fork. Perf/MoE deliberately excluded: we are ~470x ahead
|
|
# of their MoE path, so importing it is the regression, not the fix.
|
|
git fetch upstream --quiet
|
|
git log --oneline HEAD.."$UP" \
|
|
--grep='fix' --grep='correctness' --grep='regression' --grep='panic' --grep='unsound' \
|
|
--grep='overflow' --grep='incorrect' --grep='wrong' --grep='leak' --grep='race' \
|
|
--regexp-ignore-case \
|
|
| grep -viE 'moe|expert|perf|speed|faster|optimi' \
|
|
| head -60
|
|
echo
|
|
echo "Run monthly. Route each with: $0 show <sha>. ABSENT => we rewrote that code; skip."
|
|
;;
|
|
|
|
*) sed -n '2,26p' "$0"; exit 1 ;;
|
|
esac
|