Files
skills/public/hanzo-sentry.md

10 KiB

Hanzo Sentry - Error Tracking & Performance Monitoring

Category: Hanzo Ecosystem Related Skills: hanzo/hanzo-o11y.md, hanzo/hanzo-kms.md, hanzo/hanzo-id.md, hanzo/hanzo-universe.md

Overview

Hanzo Sentry is a self-hosted Sentry 24.2.0 deployment for error tracking and performance monitoring across all Hanzo, Lux, Zoo, and Pars services. It is a fork of getsentry/sentry with a custom Dockerfile branded for Hanzo, configured for OIDC SSO via hanzo.id, and deployed to the hanzo-k8s cluster.

Why Hanzo Sentry?

  • Self-hosted: Full control over data, no SaaS dependency
  • Multi-tenant: Single instance serving four domains (hanzo, lux, pars, zoo)
  • IAM integrated: OIDC SSO login via hanzo.id (app-sentry client)
  • Secrets via KMS: All credentials synced from kms.hanzo.ai
  • OTEL instrumented: Exports traces to the shared OTEL collector
  • Shared infrastructure: Uses hanzo-sql (PostgreSQL) and shared Redis

Tech Stack

  • Backend: Python 3.11 (Django 5.x, Celery, DRF)
  • Frontend: React + TypeScript (Webpack, Emotion CSS)
  • Queue: Celery via Redis (or RabbitMQ if configured)
  • Database: PostgreSQL via hanzo-sql.hanzo.svc
  • Cache: Redis via redis.hanzo.svc
  • Image: ghcr.io/hanzoai/sentry:latest
  • Base version: Sentry 24.2.0 (FSL-1.0-Apache-2.0 license)

OSS Base

Fork of getsentry/sentry. Local clone at ~/work/hanzo/sentry/. The GitHub repo hanzoai/sentry does not yet exist as a public fork -- currently origin points to getsentry/sentry and Hanzo customizations live in the self-hosted/ directory and K8s manifests in universe/infra/k8s/sentry/.

When to use

  • Instrumenting Hanzo services with error tracking
  • Monitoring performance of API endpoints, workers, and frontends
  • Viewing crash reports, stack traces, and breadcrumbs
  • Setting up alerting rules for error spikes
  • Tracking release health across deployments

Hard requirements

  1. PostgreSQL via hanzo-sql.hanzo.svc (database sentry)
  2. Redis via redis.hanzo.svc (cache, queues, rate limits, buffers)
  3. KMS at kms.hanzo.ai for secret sync (secret-key, db-password, oidc-client-secret)
  4. IAM at hanzo.id with app-sentry OIDC client for SSO

Quick reference

Item Value
UI https://sentry.hanzo.ai
Alt domains sentry.lux.network, sentry.pars.network, sentry.zoo.network
Port 9000 (web), internal K8s service
Image ghcr.io/hanzoai/sentry:latest
Base version Sentry 24.2.0
Language Python 3.11 + TypeScript
K8s namespace hanzo
Deployments sentry-web (2 replicas), sentry-worker (2 replicas), sentry-cron (1 replica)
Database hanzo-sql.hanzo.svc:5432/sentry
Redis redis.hanzo.svc:6379
Auth OIDC SSO via hanzo.id (app-sentry)
KMS project hanzo-k8s-epiq, path /sentry
Local clone ~/work/hanzo/sentry/
K8s manifests ~/work/hanzo/universe/infra/k8s/sentry/
Health check GET /_health/

One-file quickstart

Instrument a Python service

import sentry_sdk

sentry_sdk.init(
 dsn="https://<key>@sentry.hanzo.ai/<project-id>",
 traces_sample_rate=0.1,
 environment="production",
 release="[email protected]",
)

Instrument a Node.js service

const Sentry = require("@sentry/node");

Sentry.init({
 dsn: "https://<key>@sentry.hanzo.ai/<project-id>",
 tracesSampleRate: 0.1,
 environment: "production",
 release: "[email protected]",
});

Instrument a Go service

import "github.com/getsentry/sentry-go"

func main() {
 sentry.Init(sentry.ClientOptions{
 Dsn: "https://<key>@sentry.hanzo.ai/<project-id>",
 TracesSampleRate: 0.1,
 Environment: "production",
 Release: "[email protected]",
 })
 defer sentry.Flush(2 * time.Second)
}

Docker Compose (local development)

# compose.yml
services:
 sentry-web:
 image: ghcr.io/hanzoai/sentry:latest
 command: ["sentry", "run", "web"]
 ports:
 - "9000:9000"
 environment:
 SENTRY_SECRET_KEY: "${SENTRY_SECRET_KEY}"
 SENTRY_POSTGRES_HOST: postgres
 SENTRY_DB_NAME: sentry
 SENTRY_DB_USER: sentry
 SENTRY_DB_PASSWORD: "${SENTRY_DB_PASSWORD}"
 SENTRY_REDIS_HOST: redis

 sentry-worker:
 image: ghcr.io/hanzoai/sentry:latest
 command: ["sentry", "run", "worker"]
 environment:
 SENTRY_SECRET_KEY: "${SENTRY_SECRET_KEY}"
 SENTRY_POSTGRES_HOST: postgres
 SENTRY_DB_NAME: sentry
 SENTRY_DB_USER: sentry
 SENTRY_DB_PASSWORD: "${SENTRY_DB_PASSWORD}"
 SENTRY_REDIS_HOST: redis

 sentry-cron:
 image: ghcr.io/hanzoai/sentry:latest
 command: ["sentry", "run", "cron"]
 environment:
 SENTRY_SECRET_KEY: "${SENTRY_SECRET_KEY}"
 SENTRY_POSTGRES_HOST: postgres
 SENTRY_DB_NAME: sentry
 SENTRY_DB_USER: sentry
 SENTRY_DB_PASSWORD: "${SENTRY_DB_PASSWORD}"
 SENTRY_REDIS_HOST: redis

 postgres:
 image: postgres:16
 environment:
 POSTGRES_DB: sentry
 POSTGRES_USER: sentry
 POSTGRES_PASSWORD: "${SENTRY_DB_PASSWORD}"

 redis:
 image: redis:7-alpine

Core Concepts

Architecture

 ┌──────────────────────────────────┐
 │ K8s Ingress │
 │ sentry.hanzo.ai │
 │ sentry.lux.network │
 │ sentry.pars.network │
 │ sentry.zoo.network │
 └───────────────┬──────────────────┘
 │
 ┌───────────────▼──────────────────┐
 │ sentry-web (2 replicas) │
 │ Django + uWSGI on port 9000 │
 └──┬────────────┬─────────────┬────┘
 │ │ │
 ┌──────▼──┐ ┌─────▼─────┐ ┌───▼────────┐
 │ Postgres │ │ Redis │ │ OTEL │
 │ hanzo-sql│ │ (shared) │ │ Collector │
 └─────────┘ └───────────┘ └────────────┘
 │ │
 ┌──────▼──┐ ┌─────▼──────────────┐
 │ sentry- │ │ sentry-worker │
 │ cron │ │ (2 replicas, Celery) │
 └─────────┘ └─────────────────────┘

Three Deployments

Component Command Replicas Purpose
sentry-web sentry run web 2 Django web server (uWSGI)
sentry-worker sentry run worker 2 Celery async task processing
sentry-cron sentry run cron 1 Scheduled tasks (digests, cleanup)

Multi-Tenant Domains

A single Sentry instance serves four domains via K8s Ingress. Each domain maps to a separate Sentry organization, allowing per-org project isolation:

  • sentry.hanzo.ai -- Hanzo AI services
  • sentry.lux.network -- Lux blockchain nodes
  • sentry.pars.network -- Pars network services
  • sentry.zoo.network -- Zoo Labs experiments

Auth Flow

SSO is via OIDC against hanzo.id:

  1. User visits sentry.hanzo.ai and clicks "SSO Login"
  2. Redirected to hanzo.id for authentication (client app-sentry)
  3. OIDC callback returns to Sentry with identity token
  4. Sentry creates/links user account and grants org membership

Dockerfile Customizations

The self-hosted/Dockerfile in the local clone contains Hanzo-specific changes:

  • OCI labels with [email protected] maintainer and Hanzo branding
  • References ghcr.io/hanzoai/sentry as image source
  • Copies self-hosted/sentry.conf.py and self-hosted/config.yml into /etc/sentry/
  • Entrypoint via tini + gosu for proper signal handling

KMS Secret Sync

Secrets are managed via the KMSSecret CRD that syncs from kms.hanzo.ai:

# kms-secrets.yaml
spec:
 hostAPI: https://kms.hanzo.ai/api
 managedSecretReference:
 secretName: sentry-secrets
 secretNamespace: hanzo
 authentication:
 universalAuth:
 secretsScope:
 projectSlug: hanzo-k8s-epiq
 envSlug: prod
 secretsPath: /sentry

Required secrets: secret-key, db-password, oidc-client-secret, commerce-api-key (optional).

Environment Variables

# Database (PostgreSQL)
SENTRY_POSTGRES_HOST=hanzo-sql.hanzo.svc
SENTRY_DB_NAME=sentry
SENTRY_DB_USER=sentry
SENTRY_DB_PASSWORD=<from KMS>
SENTRY_DB_PORT=5432

# Redis
SENTRY_REDIS_HOST=redis.hanzo.svc
SENTRY_REDIS_PORT=6379

# Core
SENTRY_SECRET_KEY=<from KMS>

# OIDC SSO
SENTRY_OIDC_ENABLED=true
SENTRY_OIDC_ISSUER=https://hanzo.id
SENTRY_OIDC_CLIENT_ID=app-sentry
SENTRY_OIDC_CLIENT_SECRET=<from KMS>

# Observability
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector.hanzo.svc:4318
OTEL_SERVICE_NAME=sentry

# Optional: Commerce billing
HANZO_COMMERCE_URL=https://commerce.hanzo.ai
HANZO_COMMERCE_API_KEY=<from KMS>

Local Development

cd ~/work/hanzo/sentry

# Python setup (uses direnv + pyenv, not uv — upstream pattern)
make develop

# Run dependent services (Postgres, Redis, Kafka, etc.)
make run-dependent-services

# Apply database migrations
make apply-migrations

# Start web server
sentry run web

# In another terminal: start worker
sentry run worker

# Run Python tests
make test-python-ci

# Run JavaScript tests
make test-js

Troubleshooting

Issue Cause Solution
OIDC login fails app-sentry not configured in IAM Create OIDC client at hanzo.id with redirect URI https://sentry.hanzo.ai/auth/sso/
Redis connection error Missing SENTRY_REDIS_HOST Ensure redis.hanzo.svc is reachable in cluster
SENTRY_SECRET_KEY is undefined KMS sync not running Verify sentry-secrets K8s secret exists in hanzo namespace
Migration errors DB not created Run sentry upgrade to create tables and apply migrations
Worker not processing Celery not connected to broker Check Redis connectivity or RabbitMQ config
Health check failing Sentry not fully started Wait for initialDelaySeconds (60s); check logs for startup errors
npm ci fails Peer dep conflicts Use npm install with .npmrc containing legacy-peer-deps=true
  • hanzo/hanzo-o11y.md - Observability stack (OTEL, Prometheus, Grafana)
  • hanzo/hanzo-id.md - IAM and OIDC SSO configuration
  • hanzo/hanzo-kms.md - Secret management via KMSSecret CRDs
  • hanzo/hanzo-universe.md - Production K8s infrastructure and manifests
  • hanzo/hanzo-database.md - Shared PostgreSQL (hanzo-sql)

Last Updated: 2026-03-13 Category: Hanzo Ecosystem Related: sentry, error-tracking, monitoring, observability, self-hosted Prerequisites: Kubernetes, PostgreSQL, Redis, OIDC basics