bot/docs/gateway/troubleshooting.md

summary, read_when, title

summary	read_when	title
Deep troubleshooting runbook for gateway, channels, automation, nodes, and browser	The troubleshooting hub pointed you here for deeper diagnosis You need stable symptom based runbook sections with exact commands	Troubleshooting

Gateway troubleshooting

This page is the deep runbook. Start at /help/troubleshooting if you want the fast triage flow first.

Command ladder

Run these first, in this order:

hanzo-bot status
hanzo-bot gateway status
hanzo-bot logs --follow
hanzo-bot doctor
hanzo-bot channels status --probe

Expected healthy signals:

• hanzo-bot gateway status shows Runtime: running and RPC probe: ok.
• hanzo-bot doctor reports no blocking config/service issues.
• hanzo-bot channels status --probe shows connected/ready channels.

Anthropic 429 extra usage required for long context

Use this when logs/errors include: HTTP 429: rate_limit_error: Extra usage is required for long context requests.

hanzo-bot logs --follow
hanzo-bot models status
hanzo-bot config get agents.defaults.models

Look for:

• Selected Anthropic Opus/Sonnet model has params.context1m: true.
• Current Anthropic credential is not eligible for long-context usage.
• Requests fail only on long sessions/model runs that need the 1M beta path.

Fix options:

1. Disable context1m for that model to fall back to the normal context window.
2. Use an Anthropic API key with billing, or enable Anthropic Extra Usage on the subscription account.
3. Configure fallback models so runs continue when Anthropic long-context requests are rejected.

Related:

• /providers/anthropic
• /reference/token-use
• /help/faq#why-am-i-seeing-http-429-ratelimiterror-from-anthropic

No replies

If channels are up but nothing answers, check routing and policy before reconnecting anything.

hanzo-bot status
hanzo-bot channels status --probe
hanzo-bot pairing list --channel <channel> [--account <id>]
hanzo-bot config get channels
hanzo-bot logs --follow

Look for:

• Pairing pending for DM senders.
• Group mention gating (requireMention, mentionPatterns).
• Channel/group allowlist mismatches.

Common signatures:

• drop guild message (mention required → group message ignored until mention.
• pairing request → sender needs approval.
• blocked / allowlist → sender/channel was filtered by policy.

Related:

• /channels/troubleshooting
• /channels/pairing
• /channels/groups

Dashboard control ui connectivity

When dashboard/control UI will not connect, validate URL, auth mode, and secure context assumptions.

hanzo-bot gateway status
hanzo-bot status
hanzo-bot logs --follow
hanzo-bot doctor
hanzo-bot gateway status --json

Look for:

• Correct probe URL and dashboard URL.
• Auth mode/token mismatch between client and gateway.
• HTTP usage where device identity is required.

Common signatures:

• device identity required → non-secure context or missing device auth.
• device nonce required / device nonce mismatch → client is not completing the challenge-based device auth flow (connect.challenge + device.nonce).
• device signature invalid / device signature expired → client signed the wrong payload (or stale timestamp) for the current handshake.
• unauthorized / reconnect loop → token/password mismatch.
• gateway connect failed: → wrong host/port/url target.

Device auth v2 migration check:

hanzo-bot --version
hanzo-bot doctor
hanzo-bot gateway status

If logs show nonce/signature errors, update the connecting client and verify it:

1. waits for connect.challenge
2. signs the challenge-bound payload
3. sends connect.params.device.nonce with the same challenge nonce

Related:

• /web/control-ui
• /gateway/authentication
• /gateway/remote

Gateway service not running

Use this when service is installed but process does not stay up.

hanzo-bot gateway status
hanzo-bot status
hanzo-bot logs --follow
hanzo-bot doctor
hanzo-bot gateway status --deep

Look for:

• Runtime: stopped with exit hints.
• Service config mismatch (Config (cli) vs Config (service)).
• Port/listener conflicts.

Common signatures:

• Gateway start blocked: set gateway.mode=local → local gateway mode is not enabled. Fix: set gateway.mode="local" in your config (or run hanzo-bot configure). If you are running HanzoBot via Podman using the dedicated hanzo-bot user, the config lives at ~hanzo-bot/.bot/bot.json.
• refusing to bind gateway ... without auth → non-loopback bind without token/password.
• another gateway instance is already listening / EADDRINUSE → port conflict.

Related:

• /gateway/background-process
• /gateway/configuration
• /gateway/doctor

Channel connected messages not flowing

If channel state is connected but message flow is dead, focus on policy, permissions, and channel specific delivery rules.

hanzo-bot channels status --probe
hanzo-bot pairing list --channel <channel> [--account <id>]
hanzo-bot status --deep
hanzo-bot logs --follow
hanzo-bot config get channels

Look for:

• DM policy (pairing, allowlist, open, disabled).
• Group allowlist and mention requirements.
• Missing channel API permissions/scopes.

Common signatures:

• mention required → message ignored by group mention policy.
• pairing / pending approval traces → sender is not approved.
• missing_scope, not_in_channel, Forbidden, 401/403 → channel auth/permissions issue.

Related:

• /channels/troubleshooting
• /channels/whatsapp
• /channels/telegram
• /channels/discord

Cron and heartbeat delivery

If cron or heartbeat did not run or did not deliver, verify scheduler state first, then delivery target.

hanzo-bot cron status
hanzo-bot cron list
hanzo-bot cron runs --id <jobId> --limit 20
hanzo-bot system heartbeat last
hanzo-bot logs --follow

Look for:

• Cron enabled and next wake present.
• Job run history status (ok, skipped, error).
• Heartbeat skip reasons (quiet-hours, requests-in-flight, alerts-disabled).

Common signatures:

• cron: scheduler disabled; jobs will not run automatically → cron disabled.
• cron: timer tick failed → scheduler tick failed; check file/log/runtime errors.
• heartbeat skipped with reason=quiet-hours → outside active hours window.
• heartbeat: unknown accountId → invalid account id for heartbeat delivery target.
• heartbeat skipped with reason=dm-blocked → heartbeat target resolved to a DM-style destination while agents.defaults.heartbeat.directPolicy (or per-agent override) is set to block.

Related:

• /automation/troubleshooting
• /automation/cron-jobs
• /gateway/heartbeat

Node paired tool fails

If a node is paired but tools fail, isolate foreground, permission, and approval state.

hanzo-bot nodes status
hanzo-bot nodes describe --node <idOrNameOrIp>
hanzo-bot approvals get --node <idOrNameOrIp>
hanzo-bot logs --follow
hanzo-bot status

Look for:

• Node online with expected capabilities.
• OS permission grants for camera/mic/location/screen.
• Exec approvals and allowlist state.

Common signatures:

• NODE_BACKGROUND_UNAVAILABLE → node app must be in foreground.
• *_PERMISSION_REQUIRED / LOCATION_PERMISSION_REQUIRED → missing OS permission.
• SYSTEM_RUN_DENIED: approval required → exec approval pending.
• SYSTEM_RUN_DENIED: allowlist miss → command blocked by allowlist.

Related:

• /nodes/troubleshooting
• /nodes/index
• /tools/exec-approvals

Browser tool fails

Use this when browser tool actions fail even though the gateway itself is healthy.

hanzo-bot browser status
hanzo-bot browser start --browser-profile hanzo-bot
hanzo-bot browser profiles
hanzo-bot logs --follow
hanzo-bot doctor

Look for:

• Valid browser executable path.
• CDP profile reachability.
• Extension relay tab attachment for profile="chrome".

Common signatures:

• Failed to start Chrome CDP on port → browser process failed to launch.
• browser.executablePath not found → configured path is invalid.
• Chrome extension relay is running, but no tab is connected → extension relay not attached.
• Browser attachOnly is enabled ... not reachable → attach-only profile has no reachable target.

Related:

• /tools/browser-linux-troubleshooting
• /tools/chrome-extension
• /tools/browser

If you upgraded and something suddenly broke

Most post-upgrade breakage is config drift or stricter defaults now being enforced.

1) Auth and URL override behavior changed

hanzo-bot gateway status
hanzo-bot config get gateway.mode
hanzo-bot config get gateway.remote.url
hanzo-bot config get gateway.auth.mode

What to check:

• If gateway.mode=remote, CLI calls may be targeting remote while your local service is fine.
• Explicit --url calls do not fall back to stored credentials.

Common signatures:

• gateway connect failed: → wrong URL target.
• unauthorized → endpoint reachable but wrong auth.

2) Bind and auth guardrails are stricter

hanzo-bot config get gateway.bind
hanzo-bot config get gateway.auth.token
hanzo-bot gateway status
hanzo-bot logs --follow

What to check:

• Non-loopback binds (lan, tailnet, custom) need auth configured.
• Old keys like gateway.token do not replace gateway.auth.token.

Common signatures:

• refusing to bind gateway ... without auth → bind+auth mismatch.
• RPC probe: failed while runtime is running → gateway alive but inaccessible with current auth/url.

3) Pairing and device identity state changed

hanzo-bot devices list
hanzo-bot pairing list --channel <channel> [--account <id>]
hanzo-bot logs --follow
hanzo-bot doctor

What to check:

• Pending device approvals for dashboard/nodes.
• Pending DM pairing approvals after policy or identity changes.

Common signatures:

• device identity required → device auth not satisfied.
• pairing required → sender/device must be approved.

If the service config and runtime still disagree after checks, reinstall service metadata from the same profile/state directory:

hanzo-bot gateway install --force
hanzo-bot gateway restart

Related:

• /gateway/pairing
• /gateway/authentication
• /gateway/background-process