mirror of
https://github.com/hanzoai/insights.git
synced 2026-10-30 16:15:31 +00:00
zeekay
381f27d7b1
Arming the boot gate means a failing check crash-loops every pod, so what each check couples boot to matters. The three local checks are boot-critical: a schema this process cannot serve produces 500s, so failing closed is right. schema_drift joins them because the ledger checks passed for a long time against a database missing eleven tables. The datastore is a separate process on a separate node, and `migrate_datastore --check` fails the same way whether a migration is unapplied or the store is simply unreachable. One of those should stop a boot; the other should not. On 2026-08-09 the store wedged above its own memory limit and refused every query for hours -- with the gate armed and no distinction, that analytics incident would have become a total outage, with no running pod left to exec into. Proven in the pod: healthy exit 0; unreachable exit 0 with a warning; and the classifier fails an unapplied migration and an inconsistent ledger while continuing on refused/timeout/gaierror/memory limit.
31 lines
1.4 KiB
Bash
Executable File
31 lines
1.4 KiB
Bash
Executable File
#!/bin/bash
|
|
set -e
|
|
|
|
if [ -n "$INSIGHTS_SKIP_MIGRATION_CHECKS" ]; then
|
|
exit 0
|
|
fi
|
|
|
|
# Local schema first: cheap, and a mismatch here means this process would serve 500s.
|
|
python manage.py migrate --check
|
|
python manage.py run_async_migrations --check
|
|
|
|
# The three ledger checks answer "is every migration recorded applied", which is a different
|
|
# question from "does the database have what the models declare". Production served for a long
|
|
# time with eleven absent tables while that passed, because those rows had been faked past a
|
|
# failure and a migration recorded as applied can never run again. This reads the schema.
|
|
python manage.py schema_drift
|
|
|
|
# The datastore is a separate process on a separate node. An unapplied migration there should
|
|
# stop a boot; the store being briefly unreachable should NOT -- otherwise a datastore incident
|
|
# (one lasted hours on 2026-08-09) takes the entire web tier down with it, turning "analytics is
|
|
# unavailable" into "the product is unavailable", and leaving no running pod to exec into. The
|
|
# two failures are worth distinguishing.
|
|
if ! datastore_out=$(python manage.py migrate_datastore --check 2>&1); then
|
|
if printf '%s' "$datastore_out" | grep -qiE 'connection|refused|timed out|timeout|unreachable|resolve|gaierror|memory limit'; then
|
|
printf 'migrate-check: datastore unreachable, continuing: %s\n' "$(printf '%s' "$datastore_out" | tail -1)" >&2
|
|
else
|
|
printf '%s\n' "$datastore_out" >&2
|
|
exit 1
|
|
fi
|
|
fi
|