ci: fail any gate that reports success without running a test
Hanzo CI/CD / cicd (push) Successful in 2m43s
CI/CD / gate (push) Successful in 2m44s

A green build over ZERO tests is the worst failure mode a gate has, because it is
indistinguishable from a healthy one. Two ways to reach it were live in this
fleet, both silent, both exit 0:

  $ go test ./...                    # nothing in the tree has a _test.go
  ?   example.com/notests   [no test files]
  $ echo $?
  0
  $ go test -tags skipCi ./...       # every _test.go is //go:build !skipCi
  ?   example.com/notests   [no test files]
  $ echo $?
  0

The second is how hanzoai/gateway hid: its whole fixture suite sits behind
`//go:build legacy` and its Makefile passed no -tags. When the tag was finally
passed, 12 subtests failed on an untouched main — including router_redirect
returning 404, which reproduced on the shipping config with the real binary.
hanzoai/iam runs `-tags skipCi` against files guarded `//go:build !skipCi` today.

Per-repo vigilance is not a mechanism, so the assertion lives here, once, and
every caller inherits it the moment it re-imports this workflow. The rule is the
runners' OWN words, not a heuristic: a gate fails when it SAYS it ran nothing
(Go `[no test files]` / `[no tests to run]`, pytest `collected 0 items` /
`no tests ran`, jest `No tests found` / `Tests: 0 total`, cargo `running 0 tests`,
mocha `0 passing`) AND nothing in its output shows a test having run. A gate that
is not a test gate — vet, lint, a build, a codegen-freshness check — says neither
and is untouched. The one way to satisfy it is to make the gate run a test.

Verified by pointing the step at suites that run nothing, not by reasoning about
it: a module with no test files FAILS, the same module with its tests excluded by
a build tag FAILS, `go vet ./...` and `go build ./...` PASS, a real suite PASSES,
and a genuinely failing suite still fails (the assertion masks nothing). Checked
against real fleet output too: this repo's own gate, hanzoai/git's
`go test ./modules/setting/...` (which legitimately prints `[no test files]` for
one sub-package while another runs), and hanzoai/gateway's full `make test`.

Carried at both workflow paths, since GitHub resolves only .github/workflows and
git.hanzo.ai only .hanzo/workflows; the bodies stay byte-identical.

Co-authored-by: Hanzo Dev <dev@hanzo.ai>
This commit is contained in:
2026-07-28 22:55:04 -07:00
parent f8ab325ace
commit df6c025df0
2 changed files with 54 additions and 2 deletions
+27 -1
View File
@@ -530,12 +530,38 @@ jobs:
echo "runner git authenticated for private Go modules"
- name: Test (per hanzo.yml)
# Every gate runs here, and every gate is then asked ONE question: did it
# actually run a test?
#
# A green build over ZERO tests is the worst failure mode a gate has,
# because it is indistinguishable from a healthy one. Two ways to reach it
# are live in this fleet right now, both silent, both exit 0:
# - `go test ./...` where nothing has a _test.go → `[no test files]`
# - a build tag that excludes the suite (hanzoai/gateway shipped
# `//go:build legacy` tests and a target that passed no -tags; hanzoai/iam
# runs `-tags skipCi` against files guarded `//go:build !skipCi`)
# → `[no tests to run]`
# Per-repo vigilance is not a mechanism, so the assertion lives HERE, once,
# and every caller inherits it the moment it re-imports this workflow.
#
# The rule is the runners' OWN words, not a heuristic: fail when a gate SAYS
# it ran nothing AND nothing shows as having run. A gate that is not a test
# gate — vet, lint, a build, a codegen-freshness check — says neither and is
# untouched. To silence it legitimately, make the gate run a test.
if: inputs.mode != 'delegate' && inputs.tests
run: |
set -euo pipefail
NOTHING='\[no test files\]|\[no tests to run\]|no tests ran|collected 0 items|No tests found|no test specified|running 0 tests|Tests:[[:space:]]+0 total|(^|[^0-9])0 (passing|passed)'
SOMETHING='^[[:space:]]*--- (PASS|FAIL|SKIP):|^(ok|FAIL)[[:space:]]+[^[:space:]]+[[:space:]]+([0-9.]+s|\(cached\))([[:space:]]+coverage)?$|[1-9][0-9]* (passed|failed|passing|failing)|test result: (ok|FAILED)\.[[:space:]]+[1-9]|Tests:[[:space:]]+[1-9]'
log="${RUNNER_TEMP:-/tmp}/test-gate.log"
yq -o=json -I=0 '.test // []' hanzo.yml | jq -c '.[]' | while read -r t; do
name=$(echo "$t"|jq -r .name); cmd=$(echo "$t"|jq -r .run)
echo "::group::test $name"; bash -c "$cmd"; echo "::endgroup::"
echo "::group::test $name"; bash -c "$cmd" 2>&1 | tee "$log"; echo "::endgroup::"
if grep -Eq "$NOTHING" "$log" && ! grep -Eq "$SOMETHING" "$log"; then
echo "::error::test gate '$name' reported success without running a single test"
grep -Eho "$NOTHING" "$log" | sort -u | sed 's/^/ ran nothing: /'
exit 1
fi
done
- name: Build & publish binaries (per hanzo.yml)
+27 -1
View File
@@ -537,12 +537,38 @@ jobs:
echo "runner git authenticated for private Go modules"
- name: Test (per hanzo.yml)
# Every gate runs here, and every gate is then asked ONE question: did it
# actually run a test?
#
# A green build over ZERO tests is the worst failure mode a gate has,
# because it is indistinguishable from a healthy one. Two ways to reach it
# are live in this fleet right now, both silent, both exit 0:
# - `go test ./...` where nothing has a _test.go → `[no test files]`
# - a build tag that excludes the suite (hanzoai/gateway shipped
# `//go:build legacy` tests and a target that passed no -tags; hanzoai/iam
# runs `-tags skipCi` against files guarded `//go:build !skipCi`)
# → `[no tests to run]`
# Per-repo vigilance is not a mechanism, so the assertion lives HERE, once,
# and every caller inherits it the moment it re-imports this workflow.
#
# The rule is the runners' OWN words, not a heuristic: fail when a gate SAYS
# it ran nothing AND nothing shows as having run. A gate that is not a test
# gate — vet, lint, a build, a codegen-freshness check — says neither and is
# untouched. To silence it legitimately, make the gate run a test.
if: inputs.mode != 'delegate' && inputs.tests
run: |
set -euo pipefail
NOTHING='\[no test files\]|\[no tests to run\]|no tests ran|collected 0 items|No tests found|no test specified|running 0 tests|Tests:[[:space:]]+0 total|(^|[^0-9])0 (passing|passed)'
SOMETHING='^[[:space:]]*--- (PASS|FAIL|SKIP):|^(ok|FAIL)[[:space:]]+[^[:space:]]+[[:space:]]+([0-9.]+s|\(cached\))([[:space:]]+coverage)?$|[1-9][0-9]* (passed|failed|passing|failing)|test result: (ok|FAILED)\.[[:space:]]+[1-9]|Tests:[[:space:]]+[1-9]'
log="${RUNNER_TEMP:-/tmp}/test-gate.log"
yq -o=json -I=0 '.test // []' hanzo.yml | jq -c '.[]' | while read -r t; do
name=$(echo "$t"|jq -r .name); cmd=$(echo "$t"|jq -r .run)
echo "::group::test $name"; bash -c "$cmd"; echo "::endgroup::"
echo "::group::test $name"; bash -c "$cmd" 2>&1 | tee "$log"; echo "::endgroup::"
if grep -Eq "$NOTHING" "$log" && ! grep -Eq "$SOMETHING" "$log"; then
echo "::error::test gate '$name' reported success without running a single test"
grep -Eho "$NOTHING" "$log" | sort -u | sed 's/^/ ran nothing: /'
exit 1
fi
done
- name: Build & publish binaries (per hanzo.yml)