ci: fail any gate that reports success without running a test
A green build over ZERO tests is the worst failure mode a gate has, because it is indistinguishable from a healthy one. Two ways to reach it were live in this fleet, both silent, both exit 0: $ go test ./... # nothing in the tree has a _test.go ? example.com/notests [no test files] $ echo $? 0 $ go test -tags skipCi ./... # every _test.go is //go:build !skipCi ? example.com/notests [no test files] $ echo $? 0 The second is how hanzoai/gateway hid: its whole fixture suite sits behind `//go:build legacy` and its Makefile passed no -tags. When the tag was finally passed, 12 subtests failed on an untouched main — including router_redirect returning 404, which reproduced on the shipping config with the real binary. hanzoai/iam runs `-tags skipCi` against files guarded `//go:build !skipCi` today. Per-repo vigilance is not a mechanism, so the assertion lives here, once, and every caller inherits it the moment it re-imports this workflow. The rule is the runners' OWN words, not a heuristic: a gate fails when it SAYS it ran nothing (Go `[no test files]` / `[no tests to run]`, pytest `collected 0 items` / `no tests ran`, jest `No tests found` / `Tests: 0 total`, cargo `running 0 tests`, mocha `0 passing`) AND nothing in its output shows a test having run. A gate that is not a test gate — vet, lint, a build, a codegen-freshness check — says neither and is untouched. The one way to satisfy it is to make the gate run a test. Verified by pointing the step at suites that run nothing, not by reasoning about it: a module with no test files FAILS, the same module with its tests excluded by a build tag FAILS, `go vet ./...` and `go build ./...` PASS, a real suite PASSES, and a genuinely failing suite still fails (the assertion masks nothing). Checked against real fleet output too: this repo's own gate, hanzoai/git's `go test ./modules/setting/...` (which legitimately prints `[no test files]` for one sub-package while another runs), and hanzoai/gateway's full `make test`. Carried at both workflow paths, since GitHub resolves only .github/workflows and git.hanzo.ai only .hanzo/workflows; the bodies stay byte-identical. Co-authored-by: Hanzo Dev <dev@hanzo.ai>
This commit is contained in:
@@ -530,12 +530,38 @@ jobs:
|
||||
echo "runner git authenticated for private Go modules"
|
||||
|
||||
- name: Test (per hanzo.yml)
|
||||
# Every gate runs here, and every gate is then asked ONE question: did it
|
||||
# actually run a test?
|
||||
#
|
||||
# A green build over ZERO tests is the worst failure mode a gate has,
|
||||
# because it is indistinguishable from a healthy one. Two ways to reach it
|
||||
# are live in this fleet right now, both silent, both exit 0:
|
||||
# - `go test ./...` where nothing has a _test.go → `[no test files]`
|
||||
# - a build tag that excludes the suite (hanzoai/gateway shipped
|
||||
# `//go:build legacy` tests and a target that passed no -tags; hanzoai/iam
|
||||
# runs `-tags skipCi` against files guarded `//go:build !skipCi`)
|
||||
# → `[no tests to run]`
|
||||
# Per-repo vigilance is not a mechanism, so the assertion lives HERE, once,
|
||||
# and every caller inherits it the moment it re-imports this workflow.
|
||||
#
|
||||
# The rule is the runners' OWN words, not a heuristic: fail when a gate SAYS
|
||||
# it ran nothing AND nothing shows as having run. A gate that is not a test
|
||||
# gate — vet, lint, a build, a codegen-freshness check — says neither and is
|
||||
# untouched. To silence it legitimately, make the gate run a test.
|
||||
if: inputs.mode != 'delegate' && inputs.tests
|
||||
run: |
|
||||
set -euo pipefail
|
||||
NOTHING='\[no test files\]|\[no tests to run\]|no tests ran|collected 0 items|No tests found|no test specified|running 0 tests|Tests:[[:space:]]+0 total|(^|[^0-9])0 (passing|passed)'
|
||||
SOMETHING='^[[:space:]]*--- (PASS|FAIL|SKIP):|^(ok|FAIL)[[:space:]]+[^[:space:]]+[[:space:]]+([0-9.]+s|\(cached\))([[:space:]]+coverage)?$|[1-9][0-9]* (passed|failed|passing|failing)|test result: (ok|FAILED)\.[[:space:]]+[1-9]|Tests:[[:space:]]+[1-9]'
|
||||
log="${RUNNER_TEMP:-/tmp}/test-gate.log"
|
||||
yq -o=json -I=0 '.test // []' hanzo.yml | jq -c '.[]' | while read -r t; do
|
||||
name=$(echo "$t"|jq -r .name); cmd=$(echo "$t"|jq -r .run)
|
||||
echo "::group::test $name"; bash -c "$cmd"; echo "::endgroup::"
|
||||
echo "::group::test $name"; bash -c "$cmd" 2>&1 | tee "$log"; echo "::endgroup::"
|
||||
if grep -Eq "$NOTHING" "$log" && ! grep -Eq "$SOMETHING" "$log"; then
|
||||
echo "::error::test gate '$name' reported success without running a single test"
|
||||
grep -Eho "$NOTHING" "$log" | sort -u | sed 's/^/ ran nothing: /'
|
||||
exit 1
|
||||
fi
|
||||
done
|
||||
|
||||
- name: Build & publish binaries (per hanzo.yml)
|
||||
|
||||
@@ -537,12 +537,38 @@ jobs:
|
||||
echo "runner git authenticated for private Go modules"
|
||||
|
||||
- name: Test (per hanzo.yml)
|
||||
# Every gate runs here, and every gate is then asked ONE question: did it
|
||||
# actually run a test?
|
||||
#
|
||||
# A green build over ZERO tests is the worst failure mode a gate has,
|
||||
# because it is indistinguishable from a healthy one. Two ways to reach it
|
||||
# are live in this fleet right now, both silent, both exit 0:
|
||||
# - `go test ./...` where nothing has a _test.go → `[no test files]`
|
||||
# - a build tag that excludes the suite (hanzoai/gateway shipped
|
||||
# `//go:build legacy` tests and a target that passed no -tags; hanzoai/iam
|
||||
# runs `-tags skipCi` against files guarded `//go:build !skipCi`)
|
||||
# → `[no tests to run]`
|
||||
# Per-repo vigilance is not a mechanism, so the assertion lives HERE, once,
|
||||
# and every caller inherits it the moment it re-imports this workflow.
|
||||
#
|
||||
# The rule is the runners' OWN words, not a heuristic: fail when a gate SAYS
|
||||
# it ran nothing AND nothing shows as having run. A gate that is not a test
|
||||
# gate — vet, lint, a build, a codegen-freshness check — says neither and is
|
||||
# untouched. To silence it legitimately, make the gate run a test.
|
||||
if: inputs.mode != 'delegate' && inputs.tests
|
||||
run: |
|
||||
set -euo pipefail
|
||||
NOTHING='\[no test files\]|\[no tests to run\]|no tests ran|collected 0 items|No tests found|no test specified|running 0 tests|Tests:[[:space:]]+0 total|(^|[^0-9])0 (passing|passed)'
|
||||
SOMETHING='^[[:space:]]*--- (PASS|FAIL|SKIP):|^(ok|FAIL)[[:space:]]+[^[:space:]]+[[:space:]]+([0-9.]+s|\(cached\))([[:space:]]+coverage)?$|[1-9][0-9]* (passed|failed|passing|failing)|test result: (ok|FAILED)\.[[:space:]]+[1-9]|Tests:[[:space:]]+[1-9]'
|
||||
log="${RUNNER_TEMP:-/tmp}/test-gate.log"
|
||||
yq -o=json -I=0 '.test // []' hanzo.yml | jq -c '.[]' | while read -r t; do
|
||||
name=$(echo "$t"|jq -r .name); cmd=$(echo "$t"|jq -r .run)
|
||||
echo "::group::test $name"; bash -c "$cmd"; echo "::endgroup::"
|
||||
echo "::group::test $name"; bash -c "$cmd" 2>&1 | tee "$log"; echo "::endgroup::"
|
||||
if grep -Eq "$NOTHING" "$log" && ! grep -Eq "$SOMETHING" "$log"; then
|
||||
echo "::error::test gate '$name' reported success without running a single test"
|
||||
grep -Eho "$NOTHING" "$log" | sort -u | sed 's/^/ ran nothing: /'
|
||||
exit 1
|
||||
fi
|
||||
done
|
||||
|
||||
- name: Build & publish binaries (per hanzo.yml)
|
||||
|
||||
Reference in New Issue
Block a user