mirror of
https://github.com/hanzoai/benchmarks.git
synced 2026-10-30 14:59:11 +00:00
MEASURED.md said memory is deterministic and has no range. On a host at load average 3 it is: 573.0 MB and 601 bytes per agent, twelve for twelve, byte-identical across three separately built binaries. On a second host at load 17 to 73 the same lane read 594, 595, 596 and 599. Two things differ between those and this pass cannot separate them — the machines are not the same machine, and only one was busy. The cross-machine part is already known, an M1 Max and an M4 Max differ by about five bytes here. So the line now says what was observed and not a cause: determinism under quiet conditions, about 1% of variation under load, unresolved. AND THE LANES NOW SAY WHICH THEY WERE. host.sh, say.mjs and goroutine/main.go all print the one-minute load beside the CPU and core count, because a reader cannot tell a busy machine from a quiet one after the fact and this is exactly the distinction the paragraph above turns on. While wiring it: MEASURED.md claimed "every lane that measures time prints its host", and fleet, sandbox and pricing printed none at all — platform.mjs harvests lane output by looking for that very line, so it recorded no host for any of the three. say.mjs has the function now and all three call it, so the sentence is true of both halves. Co-authored-by: Hanzo Dev <[email protected]>
89 lines
3.9 KiB
Bash
89 lines
3.9 KiB
Bash
# The machine a number came from, printed by every lane that measures time.
|
|
#
|
|
# "An M-series laptop" covered an M1 Max with 10 cores and an M4 Max with 16,
|
|
# and the goroutine lane reads about twice as fast on the second. A timing
|
|
# without its host is not reproducible and not comparable, including against
|
|
# itself a month later.
|
|
#
|
|
# Sourced, not run: `. "$(dirname "$0")/../host.sh"; host`
|
|
host() {
|
|
local cpu cores os
|
|
if [ "$(uname -s)" = Darwin ]; then
|
|
cpu=$(sysctl -n machdep.cpu.brand_string 2>/dev/null)
|
|
cores=$(sysctl -n hw.ncpu 2>/dev/null)
|
|
os="macOS $(sw_vers -productVersion 2>/dev/null)"
|
|
else
|
|
cpu=$(awk -F': ' '/model name/ {print $2; exit}' /proc/cpuinfo 2>/dev/null)
|
|
cores=$(nproc 2>/dev/null)
|
|
os=$(. /etc/os-release 2>/dev/null && echo "$PRETTY_NAME")
|
|
fi
|
|
# THE LOAD IS PART OF THE HOST. The same lane on the same laptop read 601
|
|
# bytes per agent at load 3 and 594 at load 73 — a machine that is busy is not
|
|
# the machine the number claims to be from, and a reader cannot tell after the
|
|
# fact. One field, printed with the rest.
|
|
local load
|
|
load=$(uptime 2>/dev/null | sed 's/.*averages*: *//' | awk '{print $1}')
|
|
printf 'host: %s · %s cores · %s · %s · load %s\n\n' \
|
|
"${cpu:-unknown}" "${cores:-?}" "$(uname -m)" "${os:-$(uname -s)}" "${load:-?}"
|
|
}
|
|
|
|
# One row of a lane's table, and — when `BENCH_JSON` names a file — one entry in
|
|
# it. Every lane already funnelled its output through a local `say`; this is that
|
|
# function, once, so a row reaches the terminal and the file by the same call and
|
|
# the two cannot disagree.
|
|
#
|
|
# The value is written as a string. A lane's rows are as often "0 of 7" or
|
|
# "refuses — …" as they are numbers, and a reader that wants a number knows which
|
|
# row it asked for.
|
|
say() {
|
|
printf '%-34s %s\n' "$1" "$2"
|
|
[ -n "${BENCH_JSON:-}" ] || return 0
|
|
python3 "$(dirname "${BASH_SOURCE[0]}")/row.py" "$BENCH_JSON" "$1" "$2"
|
|
}
|
|
|
|
# Where the cloud source is, for the lanes that build it.
|
|
#
|
|
# Four lanes — self, egress, cipher, transport — measure a BINARY, so they need
|
|
# the source to build one. In hanzoai/cloud they got it from `cd ../..`, which
|
|
# was the repo they lived in. Here it is a different repository, and a relative
|
|
# climb would reach this one and fail on a missing ./cmd/cloud with no hint why.
|
|
#
|
|
# Resolution order, and it stops at the first that exists:
|
|
# $CLOUD_SRC named outright
|
|
# ../cloud a sibling checkout, which is how the estate lays out
|
|
# $HOME/work/hanzo/cloud
|
|
#
|
|
# It does not clone. A benchmark that silently fetches a tree decides which
|
|
# revision you measured, and the whole point of the run is that you chose.
|
|
cloud_src() {
|
|
local c
|
|
for c in "${CLOUD_SRC:-}" "$(dirname "${BASH_SOURCE[0]}")/../cloud" "$HOME/work/hanzo/cloud"; do
|
|
[ -n "$c" ] || continue
|
|
[ -f "$c/go.mod" ] && [ -d "$c/cmd/cloud" ] || continue
|
|
# SAY WHICH SOURCE, for the same reason every timing says which host.
|
|
# `hanzoai/cloud` and `hanzo-inc/cloud` are different repositories that both
|
|
# check out as `cloud`, and a lane pointed at the wrong one answers a
|
|
# different question without saying so: self/modules.sh reads 127 modules on
|
|
# one and 55 on the other, and neither number is wrong about the tree it was
|
|
# given. Printed to stderr so `$(cloud_src)` still yields only the path.
|
|
(cd "$c" && printf 'cloud: %s · %s · %s\n\n' \
|
|
"$(pwd)" \
|
|
"$(git remote get-url origin 2>/dev/null | sed 's#.*[:/]\([^/]*/[^/]*\)$#\1#; s#\.git$##' || echo 'not a checkout')" \
|
|
"$(git rev-parse --short=12 HEAD 2>/dev/null || echo 'no commit')" >&2)
|
|
(cd "$c" && pwd)
|
|
return 0
|
|
done
|
|
cat >&2 <<'WHY'
|
|
This lane builds the cloud binary and cannot find its source.
|
|
|
|
git clone https://github.com/hanzoai/cloud ../cloud
|
|
|
|
or point at a checkout you already have:
|
|
|
|
CLOUD_SRC=/path/to/cloud <this script>
|
|
|
|
Looked for a directory holding both go.mod and cmd/cloud.
|
|
WHY
|
|
return 1
|
|
}
|