chore: release v3.181.0

* fix(sessions): stabilize virtualized session rows

* refactor(sessions): extract session JSON download action

* fix(annotation-queues): restore session item type contract

* fix(sessions): clear stale virtual row height

* refactor(sessions): isolate virtual row measurement state

* fix(sessions): release stale virtual row height freeze

* fix(session): lazy initialize virtual row measurement state

* fix(session): small polishments to sessions virtualization

* fix(session): keep virtual row freeze active during oscillation

* fix(session): keep virtual row freeze active after scroll

---------

Co-authored-by: Nimar <l.nimar.b@gmail.com>

.agents/AGENTS.md

@@ -0,0 +1,101 @@
+# Agent Guidelines for Langfuse
+
+Langfuse is an open source LLM engineering platform for developing, monitoring,
+evaluating, and debugging AI applications.
+
+## How To Work
+
+- Read the minimal local context required for the task.
+- Keep changes scoped and avoid unrelated refactors.
+- For bug fixes, write the failing test first, confirm it fails, then fix the
+  bug.
+- For user-visible frontend changes in `web/**`, review the affected flow in a
+  real browser before signoff.
+- For documentation screenshots in Markdown, avoid fixed `height` on `<img>`
+  tags; prefer Markdown images or width-only HTML so previews preserve aspect
+  ratio.
+- Never commit secrets or credentials. Keep `.env*.example` files in
+  sync with required env vars.
+
+## Project Structure
+
+```text
+langfuse/
+|- web/                     # Next.js app (UI + tRPC + public REST)
+|- worker/                  # Queue consumers and background processing
+|- packages/shared/         # Shared domain, DB, queue contracts, repositories
+|- ee/                      # Enterprise package consumed by web
+|- generated/               # Generated API clients (do not hand-edit)
+|- fern/                    # API definition sources
+`- scripts/                 # Repo scripts
+```
+
+- Dependency direction:
+  - `web` -> `@langfuse/shared`, `@langfuse/ee`
+  - `worker` -> `@langfuse/shared`
+  - `@langfuse/ee` -> `@langfuse/shared`
+  - `@langfuse/shared` -> no imports from `web`, `worker`, or `ee`
+- Queue payload schemas and queue-name contracts are owned by
+  `packages/shared/src/server/queues.ts`.
+- High-signal shared entry points:
+  - Domain models: `packages/shared/src/domain/{observations,traces,scores}.ts`
+  - Postgres schema: `packages/shared/prisma/schema.prisma`
+  - ClickHouse migrations:
+    `packages/shared/clickhouse/migrations/{clustered,unclustered}/*.sql`
+- Architecture principles live in `.agents/ARCHITECTURE_PRINCIPLES.md`.
+
+## Core Commands
+
+- Install deps: `pnpm install`
+- Dev all packages: `pnpm run dev`
+- Dev web only: `pnpm run dev:web`
+- Dev worker only: `pnpm run dev:worker`
+- Lint all: `pnpm run lint`
+- Typecheck all: `pnpm run typecheck` / `pnpm tc`
+- Build check: `pnpm run build:check`
+- Full build: `pnpm run build`
+- Worktree bootstrap: `bash scripts/codex/setup.sh`
+- Worktree maintenance: `bash scripts/codex/maintenance.sh`
+- Install Playwright Chromium: `pnpm run playwright:install`
+
+## Verification
+
+- `web/**`: `pnpm run lint` plus targeted web tests.
+- `worker/**`: `pnpm run lint` plus targeted worker tests.
+- `packages/shared/**` non-schema changes:
+  `pnpm run lint` plus one targeted web check and one targeted worker check.
+- `packages/shared/prisma/**` or `packages/shared/clickhouse/**`:
+  `pnpm run lint`, `pnpm run db:generate`, and targeted web/worker
+  regressions.
+- Public API contracts in `web/src/pages/api/public/**`,
+  `web/src/features/public-api/types/**`, or `fern/apis/**`: `pnpm run lint`,
+  targeted server API tests, and Fern update/regeneration.
+- Cross-package refactors: `pnpm run lint`, `pnpm run typecheck`, and targeted
+  tests for impacted packages.
+
+## Generated Files
+
+Do not hand-edit generated or build artifacts:
+
+- `generated/*`
+- `web/.next/*`
+- `web/.next-check/*`
+- `*/dist/*`
+- `packages/shared/prisma/generated/*`
+
+Public API contract changes must update Fern sources in `fern/apis/**` and
+regenerated outputs. Never hand-edit `generated/**`.
+
+## Shared Agent Setup
+
+- `.agents/AGENTS.md` is the canonical root guide.
+- Root `AGENTS.md` is a symlink to `.agents/AGENTS.md`.
+- Root `CLAUDE.md` is a compatibility symlink to `AGENTS.md`.
+- When creating or editing `.agents/skills/**`, use
+  `.agents/skills/skill-creator/SKILL.md`; keep skills concise with
+  progressive disclosure.
+- After changing shared agent setup, run `pnpm run agents:sync` and
+  `pnpm run agents:check`.
+- Generated provider config and shim outputs under `.claude/`, `.cursor/`,
+  `.codex/`, `.vscode/`, or `.mcp.json` are local artifacts, not source of
+  truth files.

.agents/ARCHITECTURE_PRINCIPLES.md

@@ -0,0 +1,49 @@
+# Underlying Architecture Principles
+
+Langfuse architecture should optimize for high-scale, exploratory observability
+on wide, structured event data. These principles are grounded in current
+production scale and the reference material below.
+
+## Reference Posts
+
+- [Simplifying Langfuse for Scale](https://langfuse.com/blog/2026-03-10-simplify-langfuse-for-scale)
+- [Charity Majors on Observability 2.0](https://charity.wtf/tag/observability-2-0/)
+- [All you need is Wide Events, not "Metrics, Logs and Traces"](https://isburmistrov.substack.com/p/all-you-need-is-wide-events-not-metrics)
+
+## Principles
+
+- Model observations as the primary analytical unit. A trace is a correlation
+  handle that links related observations, not the only useful entry point.
+- Prefer wide, richly attributed events over fragmented metrics, logs, and trace
+  records that require later reconstruction.
+- Preserve high-cardinality context so users can slice, group, filter, and debug
+  unknown unknowns without predefining every future question.
+- Favor immutable or append-oriented event records for high-volume telemetry.
+  Updates that force read-time deduplication create hidden query costs at scale.
+- Denormalize carefully when it removes hot-path joins and makes common filters
+  into direct column predicates.
+- Design storage and query paths around columnar access patterns: narrow field
+  selection, time-bounded scans, useful ordering keys, and data pruning.
+- Keep list, dashboard, and aggregate views on compact query-optimized
+  representations. Fetch large raw payloads only for focused detail views.
+- Make API contracts scale-aware: require time windows where needed, expose field
+  selection, use token pagination, and avoid defaults that can scan all history.
+- Treat cost and operational simplicity as architectural constraints. Extra
+  databases, queues, materialized views, and migrations must earn their long-term
+  operational burden.
+- Preserve real-time or near-real-time debugging workflows. Batch processing can
+  help, but it should not make fresh production behavior invisible.
+
+## Practical Defaults For Agents
+
+- Before adding a metric, ask whether the same question is better answered from
+  wide event data.
+- Before adding a join, ask whether the attribute should be propagated or
+  denormalized onto the observation path.
+- Before reading large fields, ask whether the view needs them or can defer them
+  until a single-record fetch.
+- Before adding an update-heavy design, ask whether immutable events plus
+  derived representations would be simpler at production scale.
+- Before documenting public behavior, separate stable public contracts from
+  private production topology, account details, secret names, and incident
+  runbooks.

.agents/README.md

@@ -0,0 +1,209 @@
+# Shared Agent Setup
+
+This directory is the neutral, repo-owned source of truth for agent behavior in
+Langfuse.
+
+Use `.agents/` for configuration and guidance that should apply across tools.
+Do not put durable shared guidance only in `.claude/`, `.codex/`, `.cursor/`,
+or `.vscode/`.
+
+## Layout
+
+- `AGENTS.md`: canonical shared root instructions
+- `ARCHITECTURE_PRINCIPLES.md`: architecture principles for high-scale
+  observability
+- `config.json`: shared bootstrap and MCP configuration used to generate
+  tool-specific shims
+- `skills/`: shared, tool-neutral implementation guidance for recurring
+  workflows
+
+## `config.json`
+
+`.agents/config.json` contains four kinds of data:
+
+- `shared`: defaults used across tools
+- `mcpServers`: project MCP servers and how to connect to them
+- `claude`: Claude-specific generated settings inputs
+- `codex`: Codex-specific generated settings inputs
+- `cursor`: Cursor-specific generated settings inputs
+
+Current shape:
+
+```json
+{
+  "shared": {
+    "setupScript": "bash scripts/codex/setup.sh",
+    "devCommand": "pnpm run dev",
+    "devTerminalDescription": "Main development terminal running the development server"
+  },
+  "mcpServers": {
+    "playwright": {
+      "transport": "stdio",
+      "command": "npx",
+      "args": [
+        "-y",
+        "@playwright/mcp@latest",
+        "--isolated",
+        "--save-session",
+        "--output-dir",
+        "/tmp/playwright-mcp",
+        "--test-id-attribute",
+        "data-testid"
+      ]
+    },
+    "langfuse-docs": {
+      "transport": "http",
+      "url": "https://langfuse.com/api/mcp"
+    },
+    "linear": {
+      "transport": "http",
+      "url": "https://mcp.linear.app/mcp"
+    }
+  },
+  "claude": {
+    "settings": {
+      "permissions": {
+        "allow": [
+          "Bash(find:*)",
+          "Bash(rg:*)",
+          "Bash(grep:*)",
+          "Bash(ls:*)",
+          "Bash(cat:*)",
+          "Bash(head:*)",
+          "Bash(tail:*)"
+        ],
+        "deny": []
+      },
+      "enableAllProjectMcpServers": true
+    }
+  },
+  "codex": {
+    "environment": {
+      "version": 1,
+      "name": "langfuse"
+    }
+  },
+  "cursor": {
+    "environment": {
+      "agentCanUpdateSnapshot": false
+    }
+  }
+}
+```
+
+## How Shims Are Generated
+
+`scripts/agents/sync-agent-shims.mjs` reads `.agents/config.json` and writes the
+tool discovery files that those products require.
+
+Generated local artifacts:
+
+- `.claude/settings.json`
+- `.claude/skills/*`
+- `.cursor/environment.json`
+- `.cursor/mcp.json`
+- `.vscode/mcp.json`
+- `.mcp.json`
+- `.codex/config.toml`
+- `.codex/environments/environment.toml`
+
+The repo root discovery files remain committed as symlinks:
+
+- `AGENTS.md` -> `.agents/AGENTS.md`
+- `CLAUDE.md` -> `AGENTS.md`
+
+This keeps provider discovery stable while `.agents/` remains the source of
+truth.
+
+## When To Edit `config.json`
+
+Edit `.agents/config.json` when you need to:
+
+- add, remove, or update a shared MCP server
+- change the shared setup/bootstrap command
+- change the default dev command or terminal label used by generated shims
+- adjust generated Claude, Cursor, or Codex settings that are intentionally
+  modeled in the shared config
+
+Do not edit generated shim files by hand. Edit the canonical files in
+`.agents/` instead.
+
+## How To Extend `config.json`
+
+### Add an MCP server
+
+Add a new entry under `mcpServers`.
+
+For `stdio` servers:
+
+```json
+{
+  "mcpServers": {
+    "example": {
+      "transport": "stdio",
+      "command": "npx",
+      "args": ["-y", "some-package"]
+    }
+  }
+}
+```
+
+For HTTP servers:
+
+```json
+{
+  "mcpServers": {
+    "example": {
+      "transport": "http",
+      "url": "https://example.com/mcp"
+    }
+  }
+}
+```
+
+Optional fields:
+
+- `env` for `stdio` servers
+- `headers` for HTTP servers
+
+### Change bootstrap or default dev command
+
+Update values in `shared`:
+
+- `setupScript`
+- `devCommand`
+- `devTerminalDescription`
+
+### Add tool-specific generated inputs
+
+Only add tool-specific fields when they are required to generate a discovery
+file for a supported tool. Keep the shared config minimal and neutral.
+
+## Workflow
+
+After editing `.agents/config.json`:
+
+1. Run `pnpm run agents:sync`
+2. Run `pnpm run agents:check`
+3. Verify you did not stage any generated files under `.claude/skills/` or the
+   generated MCP/runtime config paths
+4. Update `AGENTS.md` or `CONTRIBUTING.md` if the shared workflow materially
+   changed
+
+`pnpm install` also runs the sync/check flow via `postinstall`.
+
+## Adding Shared Skills
+
+Shared skills live under `.agents/skills/`.
+
+Use them for durable, reusable guidance such as:
+
+- backend implementation patterns
+- provider-specific maintenance workflows
+- repeated repo-specific review checklists
+
+Do not use skills for one-off task notes or tool runtime configuration.
+
+Use `skills/skill-creator/SKILL.md` when creating or editing shared skills.
+`pnpm run agents:sync` projects the shared skills into `.claude/skills/` so
+Claude can discover the same repo-owned skills.

.agents/config.json

@@ -0,0 +1,59 @@
+{
+  "shared": {
+    "setupScript": "bash scripts/codex/setup.sh",
+    "devCommand": "pnpm run dev",
+    "devTerminalDescription": "Main development terminal running the development server"
+  },
+  "mcpServers": {
+    "playwright": {
+      "transport": "stdio",
+      "command": "npx",
+      "args": [
+        "-y",
+        "@playwright/mcp@latest",
+        "--isolated",
+        "--save-session",
+        "--output-dir",
+        "/tmp/playwright-mcp",
+        "--test-id-attribute",
+        "data-testid"
+      ]
+    },
+    "langfuse-docs": {
+      "transport": "http",
+      "url": "https://langfuse.com/api/mcp"
+    },
+    "linear": {
+      "transport": "http",
+      "url": "https://mcp.linear.app/mcp"
+    }
+  },
+  "claude": {
+    "settings": {
+      "permissions": {
+        "allow": [
+          "Bash(find:*)",
+          "Bash(rg:*)",
+          "Bash(grep:*)",
+          "Bash(ls:*)",
+          "Bash(cat:*)",
+          "Bash(head:*)",
+          "Bash(tail:*)"
+        ],
+        "deny": []
+      },
+      "enableAllProjectMcpServers": true
+    }
+  },
+  "codex": {
+    "environment": {
+      "version": 1,
+      "name": "langfuse"
+    }
+  },
+  "cursor": {
+    "environment": {
+      "agentCanUpdateSnapshot": false
+    }
+  }
+}

.agents/skills/add-model-price/SKILL.md

@@ -0,0 +1,60 @@
+---
+name: add-model-price
+description: Use when editing worker/src/constants/default-model-prices.json, packages/shared/src/server/llm/types.ts, pricing tiers, tokenizer IDs, or matchPattern regexes for OpenAI, Anthropic, Bedrock, Vertex, Azure, or Gemini model pricing.
+---
+
+# Add Model Price
+
+Use this skill for model pricing changes in `worker/` and shared LLM type
+updates in `packages/shared/`.
+
+## When to Apply
+
+- Editing `worker/src/constants/default-model-prices.json`
+- Editing `packages/shared/src/server/llm/types.ts`
+- Adding a new priced model
+- Updating provider prices, cache pricing, or tier conditions
+- Expanding regex coverage for Bedrock, Vertex, Azure, or provider-prefixed
+  model names
+
+## How to Read This Skill
+
+- Use this `SKILL.md` as the high-level workflow and helper index.
+- Open only the specific reference file that matches the task.
+
+## Quick Start Checklist
+
+### Adding a New Model
+
+- Gather official pricing from the provider documentation.
+- Generate a lowercase UUID for the model entry.
+- Create a `matchPattern` that covers supported provider formats.
+- Add at least one default pricing tier.
+- Insert the pricing entry into `worker/src/constants/default-model-prices.json`.
+- Update `packages/shared/src/server/llm/types.ts` if the model should be
+  selectable in playground or evaluation flows.
+- Validate the JSON after editing.
+
+### Updating an Existing Model
+
+- Update the relevant prices, keys, tiers, or regexes.
+- Refresh `updatedAt` to today's ISO-8601 timestamp.
+- Validate the JSON after editing.
+
+## Reference Map
+
+| Topic                           | Read this when                                                                        | File                                                                                           |
+| ------------------------------- | ------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------- |
+| Schema and tier rules           | You need the entry shape or pricing-tier invariants                                   | [references/schema-and-tiers.md](references/schema-and-tiers.md)                               |
+| Provider sources and price keys | You need official pricing URLs, per-token conversion, or provider-specific usage keys | [references/provider-sources-and-price-keys.md](references/provider-sources-and-price-keys.md) |
+| Match patterns                  | You are editing `matchPattern` regexes or provider coverage                           | [references/match-patterns.md](references/match-patterns.md)                                   |
+| Workflow and validation         | You are applying the end-to-end edit process or checking common mistakes              | [references/workflow-and-validation.md](references/workflow-and-validation.md)                 |
+
+## Deterministic Helpers
+
+- Pricing file validator:
+  `node .agents/skills/add-model-price/scripts/validate-pricing-file.mjs`
+- Match-pattern tester:
+  `node .agents/skills/add-model-price/scripts/test-match-pattern.mjs --model <modelName> --accept <sample...> --reject <sample...>`
+- Direct regex tester:
+  `node .agents/skills/add-model-price/scripts/test-match-pattern.mjs --pattern '(?i)^(openai/)?(gpt-4o)$' --accept gpt-4o openai/gpt-4o --reject gpt-4o-mini`

.agents/skills/add-model-price/references/match-patterns.md

@@ -0,0 +1,51 @@
+# Match Patterns
+
+## Anthropic Claude: API + Bedrock + Vertex
+
+```regex
+(?i)^(anthropic\/)?(claude-opus-4-6|(eu\\.|us\\.|apac\\.)?anthropic\\.claude-opus-4-6-v1(:0)?|claude-opus-4-6)$
+```
+
+Matches:
+
+- `claude-opus-4-6`
+- `anthropic/claude-opus-4-6`
+- `anthropic.claude-opus-4-6-v1:0`
+- `us.anthropic.claude-opus-4-6-v1:0`
+
+## With Version Date
+
+```regex
+(?i)^(anthropic\/)?(claude-opus-4-5-20251101|(eu\\.|us\\.|apac\\.)?anthropic\\.claude-opus-4-5-20251101-v1:0|claude-opus-4-5@20251101)$
+```
+
+## OpenAI
+
+```regex
+(?i)^(openai\/)?(gpt-4o)$
+```
+
+## Google Gemini
+
+```regex
+(?i)^(google\/)?(gemini-2.5-pro)$
+```
+
+## Pattern Components
+
+| Component | Purpose | Example |
+| --- | --- | --- |
+| `(?i)` | Case-insensitive match | `gpt-4o` and `GPT-4O` |
+| `^...$` | Full-string match | Avoids partial matches |
+| `(provider\/)?` | Optional provider prefix | `openai/gpt-4o` |
+| `(eu\\.|us\\.|apac\\.)?` | Optional AWS region prefix | `us.anthropic.model` |
+| `(:0)?` | Optional version suffix | Bedrock model versions |
+| `@date` | Vertex AI version format | `claude-3-5-sonnet@20240620` |
+
+## Testing Patterns
+
+Use the bundled helper script:
+
+```bash
+node .agents/skills/add-model-price/scripts/test-match-pattern.mjs --model gpt-4o --accept gpt-4o openai/gpt-4o --reject gpt-4o-mini
+```

.agents/skills/add-model-price/references/provider-sources-and-price-keys.md

@@ -0,0 +1,84 @@
+# Provider Sources and Price Keys
+
+## Official Pricing Sources
+
+Always fetch pricing from the provider's official docs before editing.
+
+| Provider | Source |
+| --- | --- |
+| Anthropic Claude | `https://platform.claude.com/docs/en/about-claude/pricing` |
+| OpenAI | `https://openai.com/api/pricing/` |
+| Google Gemini | `https://ai.google.dev/pricing` |
+| AWS Bedrock | `https://aws.amazon.com/bedrock/pricing/` |
+| Azure OpenAI | `https://azure.microsoft.com/pricing/details/cognitive-services/openai-service/` |
+
+Capture:
+
+1. Base input token price per million tokens
+2. Output token price per million tokens
+3. Cache write price when supported
+4. Cache read price when supported
+5. Any long-context or conditional pricing
+6. All model ID variants that Langfuse should match
+
+## Price Conversion
+
+Values in `default-model-prices.json` are per token, not per million tokens.
+
+| Provider Price | JSON Value |
+| --- | --- |
+| `$5 / MTok` | `5e-6` |
+| `$25 / MTok` | `25e-6` |
+| `$0.50 / MTok` | `0.5e-6` |
+| `$6.25 / MTok` | `6.25e-6` |
+
+Formula:
+
+```text
+price_per_token = price_per_mtok / 1_000_000
+```
+
+## Common Price Keys by Provider
+
+### Anthropic Claude
+
+```json
+{
+  "input": "<base_input_price>",
+  "input_tokens": "<base_input_price>",
+  "output": "<output_price>",
+  "output_tokens": "<output_price>",
+  "cache_creation_input_tokens": "<cache_write_price>",
+  "input_cache_creation": "<cache_write_price>",
+  "cache_read_input_tokens": "<cache_read_price>",
+  "input_cache_read": "<cache_read_price>"
+}
+```
+
+### OpenAI
+
+```json
+{
+  "input": "<input_price>",
+  "input_cached_tokens": "<cached_input_price>",
+  "input_cache_read": "<cached_input_price>",
+  "output": "<output_price>"
+}
+```
+
+### Google Gemini
+
+```json
+{
+  "input": "<input_price>",
+  "input_modality_1": "<input_price>",
+  "prompt_token_count": "<input_price>",
+  "promptTokenCount": "<input_price>",
+  "input_cached_tokens": "<cached_price>",
+  "cached_content_token_count": "<cached_price>",
+  "output": "<output_price>",
+  "output_modality_1": "<output_price>",
+  "candidates_token_count": "<output_price>",
+  "candidatesTokenCount": "<output_price>"
+}
+```

.agents/skills/add-model-price/references/schema-and-tiers.md

@@ -0,0 +1,138 @@
+# Schema and Tiers
+
+## Target Files
+
+- Pricing data: `worker/src/constants/default-model-prices.json`
+- Shared model types: `packages/shared/src/server/llm/types.ts`
+
+## Complete Model Entry Schema
+
+```json
+{
+  "id": "uuid-generated-with-uuidgen",
+  "modelName": "model-name-identifier",
+  "matchPattern": "(?i)^regex-pattern$",
+  "createdAt": "ISO-8601-timestamp",
+  "updatedAt": "ISO-8601-timestamp",
+  "tokenizerConfig": null,
+  "tokenizerId": "claude|openai|null",
+  "pricingTiers": [
+    {
+      "id": "model-uuid_tier_default",
+      "name": "Standard",
+      "isDefault": true,
+      "priority": 0,
+      "conditions": [],
+      "prices": {
+        "input": 0.000005,
+        "output": 0.000025
+      }
+    }
+  ]
+}
+```
+
+## Required Fields
+
+| Field | Type | Description |
+| --- | --- | --- |
+| `id` | string | Unique lowercase ID used by the pricing file |
+| `modelName` | string | Primary model identifier |
+| `matchPattern` | string | Regex used to match provider model names |
+| `createdAt` | string | ISO-8601 timestamp set on creation |
+| `updatedAt` | string | ISO-8601 timestamp refreshed whenever the entry changes |
+| `pricingTiers` | array | At least one pricing tier |
+
+## Optional Fields
+
+| Field | Type | Default | Description |
+| --- | --- | --- | --- |
+| `tokenizerId` | string | `null` | Usually `"claude"`, `"openai"`, or `null` |
+| `tokenizerConfig` | object | `null` | Custom tokenizer settings |
+
+## Default Tier
+
+Every model must have exactly one default tier:
+
+```json
+{
+  "id": "{model-id}_tier_default",
+  "name": "Standard",
+  "isDefault": true,
+  "priority": 0,
+  "conditions": [],
+  "prices": {}
+}
+```
+
+Rules:
+
+- `isDefault` must be `true`
+- `priority` must be `0`
+- `conditions` must be `[]`
+
+## Additional Tiers
+
+Use extra tiers for context-window or usage-based pricing:
+
+```json
+{
+  "id": "uuid-for-tier",
+  "name": "Large Context (>200K)",
+  "isDefault": false,
+  "priority": 1,
+  "conditions": [
+    {
+      "usageDetailPattern": "(input|prompt|cached)",
+      "operator": "gt",
+      "value": 200000,
+      "caseSensitive": false
+    }
+  ],
+  "prices": {}
+}
+```
+
+Supported operators: `gt`, `gte`, `lt`, `lte`, `eq`, `neq`
+
+## Multi-Tier Example
+
+Use multiple tiers when a provider changes pricing by context window or usage
+class. Keep exactly one default tier and assign higher priorities to conditional
+tiers:
+
+```json
+{
+  "pricingTiers": [
+    {
+      "id": "{model-id}_tier_default",
+      "name": "Standard",
+      "isDefault": true,
+      "priority": 0,
+      "conditions": [],
+      "prices": {
+        "input": 0.000001,
+        "output": 0.000002
+      }
+    },
+    {
+      "id": "uuid-for-large-context-tier",
+      "name": "Large Context (>200K)",
+      "isDefault": false,
+      "priority": 1,
+      "conditions": [
+        {
+          "usageDetailPattern": "(input|prompt|cached)",
+          "operator": "gt",
+          "value": 200000,
+          "caseSensitive": false
+        }
+      ],
+      "prices": {
+        "input": 0.000002,
+        "output": 0.000004
+      }
+    }
+  ]
+}
+```

.agents/skills/add-model-price/references/workflow-and-validation.md

@@ -0,0 +1,128 @@
+# Workflow and Validation
+
+## Step-by-Step Workflow
+
+### 1. Fetch Official Pricing
+
+Open the provider's official pricing page and collect input, output, cache
+write, and cache read prices.
+
+### 2. Generate a Lowercase ID
+
+```bash
+uuidgen
+```
+
+Convert the output to lowercase before using it.
+
+### 3. Create or Update the Entry
+
+Use nearby models in `worker/src/constants/default-model-prices.json` as the
+template, then:
+
+- add the new entry near related models
+- refresh `updatedAt` when editing an existing entry
+
+Example for a model with `$5` input, `$25` output, `$6.25` cache write, and
+`$0.50` cache read per million tokens:
+
+```json
+{
+  "id": "13458bc0-1c20-44c2-8753-172f54b67647",
+  "modelName": "claude-opus-4-6",
+  "matchPattern": "(?i)^(anthropic\/)?(claude-opus-4-6|(eu\\.|us\\.|apac\\.)?anthropic\\.claude-opus-4-6-v1(:0)?)$",
+  "createdAt": "2026-03-09T00:00:00.000Z",
+  "updatedAt": "2026-03-09T00:00:00.000Z",
+  "tokenizerConfig": null,
+  "tokenizerId": "claude",
+  "pricingTiers": [
+    {
+      "id": "13458bc0-1c20-44c2-8753-172f54b67647_tier_default",
+      "name": "Standard",
+      "isDefault": true,
+      "priority": 0,
+      "conditions": [],
+      "prices": {
+        "input": 5e-6,
+        "input_tokens": 5e-6,
+        "output": 25e-6,
+        "output_tokens": 25e-6,
+        "cache_creation_input_tokens": 6.25e-6,
+        "input_cache_creation": 6.25e-6,
+        "cache_read_input_tokens": 0.5e-6,
+        "input_cache_read": 0.5e-6
+      }
+    }
+  ]
+}
+```
+
+### 4. Update Shared Model Types When Needed
+
+If the model should be available in playground or LLM-as-judge flows, add it to
+the correct array in `packages/shared/src/server/llm/types.ts`.
+
+Common arrays include:
+
+- `anthropicModels`
+- `openAIModels`
+- `vertexAIModels`
+- `googleAIStudioModels`
+
+Do not add a new model as the first entry in one of these arrays. The first
+entry is used as a default model in some test or evaluation paths, and newer
+models may not be available to all users yet.
+
+### 5. Validate the Result
+
+Run the bundled validator:
+
+```bash
+node .agents/skills/add-model-price/scripts/validate-pricing-file.mjs
+```
+
+For quick manual inspection, use `jq`:
+
+```bash
+jq '.[] | select(.modelName == "claude-opus-4-6")' worker/src/constants/default-model-prices.json
+```
+
+## Validation Rules
+
+1. Exactly one default tier must have `isDefault: true`
+2. The default tier must have `priority: 0`
+3. The default tier must have `conditions: []`
+4. Non-default tiers must have `priority > 0`
+5. Non-default tiers must have at least one condition
+6. Priorities must be unique within a model
+7. Tier names must be unique within a model
+8. Each tier must contain at least one price
+9. All tiers must expose the same usage-type keys
+10. Regex patterns must be valid
+
+## Testing Model Matching
+
+Use the bundled tester before finishing any `matchPattern` change:
+
+```bash
+node .agents/skills/add-model-price/scripts/test-match-pattern.mjs --model <modelName> --accept <sample...> --reject <sample...>
+```
+
+Use representative accepted and rejected model IDs for every provider format the
+regex is intended to cover.
+
+## Common Mistakes
+
+- Guessing prices instead of using official provider docs
+- Using MTok values directly instead of per-token values
+- Forgetting the `_tier_default` suffix on the default tier ID
+- Forgetting to escape regex metacharacters such as `.`
+- Forgetting to refresh `updatedAt`
+
+## Existing Model Templates
+
+Use nearby entries as templates:
+
+- `claude-opus-4-5-20251101` for Anthropic multi-provider patterns
+- `gpt-4o` for a simple OpenAI pattern
+- `gemini-2.5-pro` for a multi-tier Gemini entry

.agents/skills/add-model-price/scripts/test-match-pattern.mjs

@@ -0,0 +1,115 @@
+#!/usr/bin/env node
+
+import fs from "node:fs/promises";
+import path from "node:path";
+
+const repoRoot = process.cwd();
+const defaultFile = path.resolve(
+  repoRoot,
+  "worker/src/constants/default-model-prices.json",
+);
+const args = process.argv.slice(2);
+
+function readOption(name) {
+  const index = args.indexOf(name);
+  if (index === -1) {
+    return null;
+  }
+
+  return args[index + 1] ?? null;
+}
+
+function readListOption(name) {
+  const index = args.indexOf(name);
+  if (index === -1) {
+    return [];
+  }
+
+  const values = [];
+  for (let i = index + 1; i < args.length; i += 1) {
+    if (args[i].startsWith("--")) {
+      break;
+    }
+    values.push(args[i]);
+  }
+  return values;
+}
+
+function compilePattern(rawPattern) {
+  let source = rawPattern;
+  let flags = "";
+  const inlineFlags = rawPattern.match(/^\(\?([dgimsuvy]*)\)/);
+
+  if (inlineFlags) {
+    flags = inlineFlags[1];
+    source = rawPattern.slice(inlineFlags[0].length);
+  }
+
+  return new RegExp(source, flags);
+}
+
+let pattern = readOption("--pattern");
+const modelName = readOption("--model");
+const accepted = readListOption("--accept");
+const rejected = readListOption("--reject");
+
+if (!pattern && !modelName) {
+  console.error("Pass either --pattern <regex> or --model <modelName>.");
+  process.exit(1);
+}
+
+if (accepted.length === 0 && rejected.length === 0) {
+  console.error("Provide samples with --accept and/or --reject.");
+  process.exit(1);
+}
+
+if (!pattern && modelName) {
+  const models = JSON.parse(await fs.readFile(defaultFile, "utf8"));
+  const model = models.find((entry) => entry.modelName === modelName);
+
+  if (!model) {
+    console.error(`Model not found in pricing file: ${modelName}`);
+    process.exit(1);
+  }
+
+  pattern = model.matchPattern;
+}
+
+let regex;
+try {
+  regex = compilePattern(pattern);
+} catch (error) {
+  console.error(`Invalid pattern: ${error.message}`);
+  process.exit(1);
+}
+
+const failures = [];
+
+for (const sample of accepted) {
+  const matched = regex.test(sample);
+  console.log(`${matched ? "PASS" : "FAIL"} accept ${sample}`);
+  if (!matched) {
+    failures.push(`Expected pattern to match: ${sample}`);
+  }
+}
+
+for (const sample of rejected) {
+  const matched = regex.test(sample);
+  console.log(`${!matched ? "PASS" : "FAIL"} reject ${sample}`);
+  if (matched) {
+    failures.push(`Expected pattern to reject: ${sample}`);
+  }
+}
+
+if (failures.length > 0) {
+  console.error("");
+  for (const failure of failures) {
+    console.error(`- ${failure}`);
+  }
+  process.exit(1);
+}
+
+console.log("");
+console.log(
+  `Pattern is valid for ${accepted.length + rejected.length} sample(s).`,
+);

.agents/skills/add-model-price/scripts/validate-pricing-file.mjs

@@ -0,0 +1,150 @@
+#!/usr/bin/env node
+
+import fs from "node:fs/promises";
+import path from "node:path";
+
+const defaultFile = "worker/src/constants/default-model-prices.json";
+const repoRoot = process.cwd();
+const filePath = path.resolve(repoRoot, process.argv[2] ?? defaultFile);
+const failures = [];
+
+function compileMatchPattern(rawPattern, label) {
+  let source = rawPattern;
+  let flags = "";
+  const inlineFlags = rawPattern.match(/^\(\?([dgimsuvy]*)\)/);
+
+  if (inlineFlags) {
+    flags = inlineFlags[1];
+    source = rawPattern.slice(inlineFlags[0].length);
+  }
+
+  try {
+    return new RegExp(source, flags);
+  } catch (error) {
+    failures.push(`${label}: invalid matchPattern (${error.message})`);
+    return null;
+  }
+}
+
+function keysOfPrices(prices) {
+  return Object.keys(prices).sort();
+}
+
+const raw = await fs.readFile(filePath, "utf8");
+const models = JSON.parse(raw);
+
+if (!Array.isArray(models)) {
+  throw new Error("Expected the pricing file to be a JSON array.");
+}
+
+for (const model of models) {
+  const label = model.modelName ?? model.id ?? "<unknown-model>";
+
+  if (!model.id || typeof model.id !== "string") {
+    failures.push(`${label}: missing string id`);
+  }
+
+  if (!model.modelName || typeof model.modelName !== "string") {
+    failures.push(`${label}: missing string modelName`);
+  }
+
+  if (!model.matchPattern || typeof model.matchPattern !== "string") {
+    failures.push(`${label}: missing string matchPattern`);
+  } else {
+    compileMatchPattern(model.matchPattern, label);
+  }
+
+  if (Number.isNaN(Date.parse(model.createdAt ?? ""))) {
+    failures.push(`${label}: invalid createdAt timestamp`);
+  }
+
+  if (Number.isNaN(Date.parse(model.updatedAt ?? ""))) {
+    failures.push(`${label}: invalid updatedAt timestamp`);
+  }
+
+  if (!Array.isArray(model.pricingTiers) || model.pricingTiers.length === 0) {
+    failures.push(`${label}: pricingTiers must be a non-empty array`);
+    continue;
+  }
+
+  const defaultTiers = model.pricingTiers.filter((tier) => tier.isDefault);
+  if (defaultTiers.length !== 1) {
+    failures.push(`${label}: must have exactly one default tier`);
+  }
+
+  const seenPriorities = new Set();
+  const seenNames = new Set();
+  let expectedPriceKeys = null;
+
+  for (const tier of model.pricingTiers) {
+    const tierLabel = `${label}/${tier.name ?? tier.id ?? "<unknown-tier>"}`;
+
+    if (seenPriorities.has(tier.priority)) {
+      failures.push(`${tierLabel}: duplicate tier priority ${tier.priority}`);
+    } else {
+      seenPriorities.add(tier.priority);
+    }
+
+    if (seenNames.has(tier.name)) {
+      failures.push(`${tierLabel}: duplicate tier name ${tier.name}`);
+    } else {
+      seenNames.add(tier.name);
+    }
+
+    if (!tier.prices || typeof tier.prices !== "object") {
+      failures.push(`${tierLabel}: missing prices object`);
+      continue;
+    }
+
+    const priceKeys = keysOfPrices(tier.prices);
+    if (priceKeys.length === 0) {
+      failures.push(`${tierLabel}: prices object must not be empty`);
+    }
+
+    for (const [usageType, price] of Object.entries(tier.prices)) {
+      if (typeof price !== "number" || Number.isNaN(price) || price < 0) {
+        failures.push(`${tierLabel}: invalid price for ${usageType}`);
+      }
+    }
+
+    if (tier.isDefault) {
+      if (tier.priority !== 0) {
+        failures.push(`${tierLabel}: default tier priority must be 0`);
+      }
+
+      if (!Array.isArray(tier.conditions) || tier.conditions.length !== 0) {
+        failures.push(`${tierLabel}: default tier conditions must be []`);
+      }
+    } else {
+      if (!(tier.priority > 0)) {
+        failures.push(`${tierLabel}: non-default tier priority must be > 0`);
+      }
+
+      if (!Array.isArray(tier.conditions) || tier.conditions.length === 0) {
+        failures.push(
+          `${tierLabel}: non-default tiers must define at least one condition`,
+        );
+      }
+    }
+
+    if (!expectedPriceKeys) {
+      expectedPriceKeys = priceKeys.join(",");
+    } else if (expectedPriceKeys !== priceKeys.join(",")) {
+      failures.push(
+        `${tierLabel}: price keys must match the other tiers for ${label}`,
+      );
+    }
+  }
+}
+
+if (failures.length > 0) {
+  console.error("Pricing validation failed:\n");
+  for (const failure of failures) {
+    console.error(`- ${failure}`);
+  }
+  process.exit(1);
+}
+
+console.log(
+  `Validated ${models.length} pricing entries in ${path.relative(repoRoot, filePath)}.`,
+);

.agents/skills/agent-setup-maintenance/SKILL.md

@@ -0,0 +1,71 @@
+---
+name: agent-setup-maintenance
+description: |
+  Shared workflow for editing Langfuse's repo-owned agent setup under `.agents/`.
+  Use when changing AGENTS files, shared skills, `.agents/config.json`,
+  generated shim behavior, provider discovery paths, or install-time agent sync.
+---
+
+# Agent Setup Maintenance
+
+Use this skill when changing the shared agent setup for the repository.
+
+## Start Here
+
+- Read [`../../README.md`](../../README.md) for the shared config and shim model.
+- Read root [`../../AGENTS.md`](../../AGENTS.md) for repo-level expectations.
+- When adding or editing shared skills, use
+  [`../skill-creator/SKILL.md`](../skill-creator/SKILL.md), then apply the
+  repo-specific checks in this skill.
+- Inspect [`../../../scripts/agents/sync-agent-shims.mjs`](../../../scripts/agents/sync-agent-shims.mjs)
+  before changing generated outputs or provider discovery behavior.
+- Inspect [`../../../scripts/postinstall.sh`](../../../scripts/postinstall.sh)
+  and [`../../../package.json`](../../../package.json) when changing install-time
+  sync behavior.
+
+## Workflow
+
+1. Edit the canonical files under `.agents/`, not generated provider outputs.
+2. Keep root `AGENTS.md` and `CLAUDE.md` as discovery symlinks; do not turn
+   them back into manually maintained copies.
+3. Treat tool-specific directories such as `.claude/`, `.cursor/`, `.codex/`,
+   `.vscode/`, and `.mcp.json` as generated discovery surfaces unless the tool
+   requires a truly tool-specific feature.
+4. Keep root `AGENTS.md` concise. Move detailed or conditional workflows into
+   shared skills or package `AGENTS.md` files.
+5. Treat developer feedback as a learning loop: when a task reveals a durable
+   repo convention, recurring pitfall, reusable workflow, or verification
+   pattern, update the smallest relevant `AGENTS.md` or shared skill.
+6. When adding or changing a shared skill, keep `SKILL.md` as the entrypoint; do
+   not add skill-by-skill links to root `AGENTS.md`.
+7. When shared setup behavior changes materially, update `README.md` and
+   contributor-facing docs in the same PR.
+
+## Docker / Install-Time Constraint
+
+- `pnpm install` runs in environments that may not contain the full repo source
+  tree.
+- In Docker builds, Turbo's pruned install stage can run root `postinstall`
+  before `scripts/` and `.agents/` are available in the image.
+- Keep install-time agent setup logic robust in those pruned contexts: skip
+  cleanly when the required repo-owned files are not present.
+
+## Required Verification
+
+Run after changing shared agent setup:
+
+- `pnpm run agents:sync`
+- `pnpm run agents:check`
+
+Run additional verification when relevant:
+
+- `pnpm run postinstall` when install-time behavior changes
+- targeted tests for any scripts you changed
+
+## Design Rules
+
+- Prefer one repo-owned source of truth over duplicated provider-specific files.
+- Keep shared setup tool-neutral where possible.
+- Only keep provider-specific files in source control when the provider requires
+  a fixed discovery path or feature that cannot be expressed through the shared
+  setup model.

.agents/skills/analyze-cloud-costs/SKILL.md

@@ -0,0 +1,62 @@
+---
+name: analyze-cloud-costs
+description: |
+  Analyze Langfuse Cloud infrastructure cost structure using Metabase cost
+  marts. Use when asked about cloud spend, AWS versus ClickHouse cost splits,
+  cost drivers by provider/service/usage type/account, daily cost per tracing
+  event, infra cost dashboards, or cost regressions visible in Metabase.
+---
+
+# Analyze Cloud Costs
+
+## Overview
+
+Use this skill for evidence-backed Langfuse Cloud cost analysis. The primary
+source is the Metabase infra cost dashboard and its production cost marts; the
+deliverable should name the time window, query grain, top drivers, and caveats.
+
+## Workflow
+
+1. Clarify the question and choose the grain:
+   - Headline daily totals: total, AWS, ClickHouse, tracing events, and cost per
+     100k events.
+   - Cost structure: provider, service, usage type, operation, account, and day.
+   - Driver or regression analysis: compare a recent complete-day window against
+     a prior baseline.
+2. Load [`references/cost-marts.md`](references/cost-marts.md) for table IDs,
+   field IDs, query examples, and caveats.
+3. Use the Metabase MCP. If the Metabase tools are not visible, discover them
+   with tool search before falling back to manual interpretation.
+4. Prefer complete UTC days. Avoid treating current-day AWS cost as final
+   because AWS CUR rows can arrive late.
+5. Start broad, then drill down:
+   - Provider split.
+   - Service split within the dominant provider.
+   - Usage type, operation, and account split for the top services.
+   - Daily trend when explaining change over time.
+6. Report only what the queried data supports. If a requested slice is absent,
+   say that no rows were found for that slice instead of inventing a driver.
+
+## Query Rules
+
+- Use `mcp__metabase__.query` for quick reads. Use
+  `construct_query` plus `execute_query` when you need to inspect or reuse the
+  opaque query.
+- Pass `filters`, `aggregations`, `group_by`, and `fields` as JSON arrays. Some
+  tool schemas may display these as strings; if that happens, serialize the same
+  arrays without changing their shape.
+- Keep limits explicit and small enough for analysis. Use pagination only when
+  the continuation token is needed.
+- Include the Metabase dashboard link or query result context in the final
+  answer when useful.
+
+## Output Expectations
+
+Summarize:
+
+- Time window and whether it uses complete UTC days.
+- Total cost and provider split when relevant.
+- Top cost drivers by service, usage type, operation, or account.
+- Trend or baseline comparison when the user asks "why did this change?"
+- Caveats, especially incomplete current-day AWS data and ClickHouse credit
+  labeling in the unified mart.

.agents/skills/analyze-cloud-costs/agents/openai.yaml

@@ -0,0 +1,4 @@
+interface:
+  display_name: "Analyze Cloud Costs"
+  short_description: "Analyze Langfuse Cloud cost structure"
+  default_prompt: "Use $analyze-cloud-costs to explain recent Langfuse Cloud cost drivers from Metabase."

.agents/skills/analyze-cloud-costs/references/cost-marts.md

@@ -0,0 +1,137 @@
+# Langfuse Cloud Cost Marts
+
+Use this reference when querying or explaining Langfuse Cloud cost structure.
+
+Dashboard:
+
+- https://langfuse.metabaseapp.com/dashboard/22-infra-cost?account=&date=past90days&tab=20-tab-1
+
+## Primary Tables
+
+| Purpose | Table | ID |
+| --- | --- | --- |
+| Unified AWS and ClickHouse cost rows by provider, service, usage type, account, and day | `langfuse_prod.mart_daily_cost_chart` | `739` |
+| Daily headline totals plus tracing event counts and cost per 100k events | `langfuse_prod.mart_daily_cost_with_events` | `784` |
+| Detailed AWS CUR summary by product, operation, account, and usage type | `langfuse_prod.mart_aws_cost_daily_by_service` | `610` |
+| Detailed ClickHouse costs by entity and metric | `langfuse_prod.mart_clickhouse_daily_cost` | `689` |
+
+Prefer table `739` for structural breakdowns. Prefer table `784` for daily
+headline totals.
+
+## Field IDs
+
+### `mart_daily_cost_chart` (`739`)
+
+| Field ID | Field |
+| --- | --- |
+| `t739-0` | `usage_date` |
+| `t739-1` | `service_provider` |
+| `t739-2` | `service_name` |
+| `t739-3` | `operation` |
+| `t739-4` | `usage_type` |
+| `t739-5` | `account_name` |
+| `t739-6` | `cost_usd` |
+
+### `mart_daily_cost_with_events` (`784`)
+
+| Field ID | Field |
+| --- | --- |
+| `t784-0` | `usage_date` |
+| `t784-1` | `total_cost_usd` |
+| `t784-2` | `clickhouse_cost_usd` |
+| `t784-3` | `aws_cost_usd` |
+| `t784-5` | `s3_api_operations_cost_usd` |
+| `t784-6` | `total_tracing_events` |
+| `t784-7` | `total_cost_per_100k_events` |
+
+## Metabase MCP Patterns
+
+The Metabase MCP supports `query` for direct reads and
+`construct_query` plus `execute_query` for reusable opaque queries. In practice,
+pass `filters`, `aggregations`, `group_by`, and `fields` as JSON arrays:
+
+```json
+{
+  "table_id": 739,
+  "filters": [
+    {
+      "field_id": "t739-0",
+      "operation": "greater-than-or-equal",
+      "value": "2026-05-09"
+    }
+  ],
+  "aggregations": [
+    {
+      "function": "sum",
+      "field_id": "t739-6"
+    }
+  ],
+  "group_by": [
+    { "field_id": "t739-1" },
+    { "field_id": "t739-2" }
+  ],
+  "limit": "200"
+}
+```
+
+If a tool surface insists on strings for those parameters, serialize the same
+arrays as JSON strings.
+
+## Common Breakdowns
+
+Provider split:
+
+- Table `739`
+- Aggregate `sum(t739-6)`
+- Group by `t739-1`
+
+Service split:
+
+- Table `739`
+- Aggregate `sum(t739-6)`
+- Group by `t739-1`, `t739-2`
+
+Usage type split:
+
+- Table `739`
+- Aggregate `sum(t739-6)`
+- Group by `t739-1`, `t739-4`
+
+Environment/account split:
+
+- Table `739`
+- Aggregate `sum(t739-6)`
+- Group by `t739-1`, `t739-5`
+
+Daily headline totals:
+
+- Table `784`
+- Filter `t784-0` by date.
+- Read `total_cost_usd`, `clickhouse_cost_usd`, `aws_cost_usd`,
+  `total_tracing_events`, and `total_cost_per_100k_events`.
+
+Daily trend by provider:
+
+- Table `739`
+- Aggregate `sum(t739-6)`
+- Group by `t739-0`, `t739-1`
+
+Drilldown sequence for a cost spike:
+
+1. Compare total daily cost in table `784`.
+2. Split the same days by provider in table `739`.
+3. Split the dominant provider by service.
+4. Split the dominant service by usage type, operation, and account.
+
+## Caveats
+
+- Current-day AWS cost can be incomplete because AWS CUR data may not have
+  landed yet.
+- For stable recent analysis, prefer the last complete UTC days rather than
+  including today.
+- ClickHouse cost rows are labeled `cost_usd` in the unified mart, but the
+  source metric is ClickHouse credits. Mention this when precision or billing
+  interpretation matters.
+- Field IDs can change if Metabase models are rebuilt. If a query fails, search
+  Metabase for the table name and inspect the returned metadata before
+  changing the analysis.

.agents/skills/backend-dev-guidelines/SKILL.md

@@ -0,0 +1,116 @@
+---
+name: backend-dev-guidelines
+description: Shared backend guide for Langfuse's Next.js, tRPC, BullMQ, and TypeScript monorepo. Use when creating or reviewing tRPC routers, public REST endpoints, BullMQ queue processors, backend services, middleware, Prisma or ClickHouse data access, OpenTelemetry instrumentation, Zod validation, env configuration, or backend tests across web, worker, or packages/shared.
+---
+
+# Backend Development Guidelines
+
+Use this skill for backend and API work across `web/`, `worker/`, and
+`packages/shared/`.
+
+## When to Apply
+
+- Creating or modifying tRPC routers and procedures
+- Creating or modifying public API endpoints
+- Creating or modifying queue processors, producers, or queue-backed workflows
+- Building or refactoring backend services and repositories
+- Working on backend auth, middleware, validation, or observability
+- Updating Prisma or ClickHouse access patterns
+- Adding or fixing backend tests
+
+## How to Read This Skill
+
+- Use this `SKILL.md` when the task spans multiple backend areas or you need the
+  end-to-end reference map.
+- Read only the specific reference file that matches the work when the scope is
+  narrower.
+- If the task introduces a user-supplied URL, an outbound HTTP request, a new
+  integration, or touches secrets, RBAC, or redirect handling, also load the
+  shared [`security-review`](../security-review/SKILL.md) skill before
+  designing or implementing the change.
+
+## Quick Start Checklists
+
+### UI: New tRPC Feature
+
+- Define the router in `features/[feature]/server/*Router.ts`.
+- Use the appropriate protected or public procedure.
+- Authenticate with JWT-aware middleware.
+- Check project/resource access and entitlements.
+- Validate input with Zod v4.
+- Put business logic in a service file.
+- Use `traceException` for error handling where relevant.
+- Add unit or integration tests in `__tests__/`.
+- Access config via `env.mjs`.
+
+### SDKs: New Public API Endpoint
+
+- Create the route in `pages/api/public/`.
+- Wrap it with `withMiddlewares` and `createAuthedProjectAPIRoute`.
+- Define types in `features/public-api/types/`.
+- Authenticate with basic auth.
+- Validate query, body, and response with Zod schemas.
+- Include API versioning in paths and schemas.
+- Update Fern API definitions to match TypeScript types.
+- Add end-to-end tests in `__tests__/async/`.
+
+### Worker: New Queue Processor
+
+- Create the processor in `worker/src/queues/`.
+- Define queue types in `packages/shared/src/server/queues`.
+- Place business logic in `features/` or `worker/src/features/`.
+- Distinguish failed jobs from jobs that should succeed with a recorded error.
+- Register the queue in `WorkerManager` in `app.ts`.
+- Add worker vitest coverage.
+
+## Core Principles
+
+- tRPC procedures, public API routes, and queue processors delegate business
+  logic to services.
+- Access configuration through `env.mjs`; do not read `process.env` directly
+  outside env setup.
+- Validate all external input with Zod v4.
+- Use Prisma directly for simple CRUD and repositories for complex query access.
+- Use OpenTelemetry and DataDog for backend observability.
+- Always filter project-scoped database queries by `projectId`.
+- Keep Fern API definitions in sync with public TypeScript API contracts.
+- Keep backend tests independent and parallel-safe.
+
+## Live Examples
+
+- tRPC router with project auth and Zod input:
+  `web/src/features/events/server/eventsRouter.ts`.
+- Public API route with middleware and typed request/response schemas:
+  `web/src/pages/api/public/datasets/index.ts`.
+- Worker queue processor with typed jobs, logging, and retry behavior:
+  `worker/src/queues/evalQueue.ts`.
+- Tenant filters for Prisma and ClickHouse:
+  `references/database-patterns.md`.
+
+## Naming Conventions
+
+- tRPC routers: `camelCaseRouter.ts`, for example `datasetRouter.ts`.
+- Services: `service.ts` in the feature server directory.
+- Queue processors: `camelCaseQueue.ts`, for example `evalQueue.ts`.
+- Public API routes: kebab-case filenames, for example `dataset-items.ts`.
+
+## Anti-Patterns to Avoid
+
+- Business logic in routes or procedures.
+- Direct `process.env` usage instead of `env.mjs` / `env.ts`.
+- Missing error handling.
+- Missing input validation.
+- Missing `projectId` filters on tenant-scoped queries.
+- `console.log` instead of `logger` / `traceException`.
+
+## Reference Map
+
+| Topic                               | Read this when                                                           | File                                                                               |
+| ----------------------------------- | ------------------------------------------------------------------------ | ---------------------------------------------------------------------------------- |
+| Architecture and package boundaries | You need the web/worker/shared split, request flow, or queue lifecycle   | [references/architecture-overview.md](references/architecture-overview.md)         |
+| Routing and controllers             | You are writing tRPC procedures, public API routes, or queue entrypoints | [references/routing-and-controllers.md](references/routing-and-controllers.md)     |
+| Middleware and auth                 | You are changing request auth, permissions, or middleware composition    | [references/middleware-guide.md](references/middleware-guide.md)                   |
+| Services and repositories           | You are placing business logic, repository code, or DI patterns          | [references/services-and-repositories.md](references/services-and-repositories.md) |
+| Database access                     | You are touching Prisma, ClickHouse, tenant filters, or query patterns   | [references/database-patterns.md](references/database-patterns.md)                 |
+| Configuration                       | You are adding env vars, startup config, or runtime toggles              | [references/configuration.md](references/configuration.md)                         |
+| Testing                             | You are adding or updating backend tests                                 | [references/testing-guide.md](references/testing-guide.md)                         |

.agents/skills/backend-dev-guidelines/references/architecture-overview.md

@@ -0,0 +1,870 @@
+# Architecture Overview - Langfuse Backend
+
+Complete guide to the layered architecture pattern used in Langfuse's Next.js/tRPC/Express monorepo. Check package manifests such as `web/package.json` for current framework versions before version-sensitive work.
+
+## Table of Contents
+
+- [Layered Architecture Pattern](#layered-architecture-pattern)
+- [Request Lifecycle](#request-lifecycle)
+- [Directory Structure](#directory-structure)
+- [Module Organization](#module-organization)
+- [Separation of Concerns](#separation-of-concerns)
+- [Database Architecture](#database-architecture)
+
+---
+
+## Layered Architecture Pattern
+
+Langfuse uses a **three-layer architecture** with two primary entry points (tRPC and Public API) plus async processing via Worker.
+
+### The Three Layers
+
+```
+# Web Package (Next.js)
+
+┌─ tRPC API ──────────────────┐   ┌── Public REST API ──────────┐
+│                             │   │                             │
+│  HTTP Request               │   │  HTTP Request               │
+│      ↓                      │   │      ↓                      │
+│  tRPC Procedure             │   │  withMiddlewares +          │
+│  (protectedProjectProcedure)│   │  createAuthedProjectAPIRoute│
+│      ↓                      │   │      ↓                      │
+│  Service (business logic)   │   │  Service (business logic)   │
+│      ↓                      │   │      ↓                      │
+│  Prisma / ClickHouse        │   │  Prisma / ClickHouse        │
+│                             │   │                             │
+└─────────────────────────────┘   └─────────────────────────────┘
+                 ↓
+            [optional]: Publish to Redis BullMQ queue
+                 ↓
+┌─ Worker Package (Express) ──────────────────────────────────┐
+│                                                             │
+│  BullMQ Queue Job                                           │
+│      ↓                                                      │
+│  Queue Processor (handles job)                              │
+│      ↓                                                      │
+│  Service (business logic)                                   │
+│      ↓                                                      │
+│  Prisma / ClickHouse                                        │
+│                                                             │
+└─────────────────────────────────────────────────────────────┘
+```
+
+### Layer Breakdown
+
+**Layer 1: API Entry Points**
+
+Two types of entry points:
+
+- **tRPC Procedures** - Type-safe RPC for UI
+  - Located in `features/[feature]/server/*Router.ts`
+  - Uses middleware for auth/validation
+  - Types shared between client/server
+
+- **Public REST APIs** - REST endpoints for SDKs
+  - Located in `pages/api/public/`
+  - Uses `withMiddlewares` + `createAuthedProjectAPIRoute`
+  - Versioned with Zod schemas
+
+**Layer 2: Services**
+
+- Business logic and orchestration
+- Shared between tRPC, Public API, and Worker
+- Located in `features/[feature]/server/service.ts`
+- No HTTP/Request/Response knowledge
+- Use repositories for complex queries or Prisma directly for simple CRUD
+
+**Layer 3: Data Access**
+
+- **Repositories** for complex data access patterns (traces, observations, scores, events)
+- **Direct Prisma** for simple CRUD operations in services
+- PostgreSQL for transactional data
+- ClickHouse for analytics/traces (accessed via repositories)
+- Redis for caching/queues
+
+**Async Processing Layer: Worker**
+
+- BullMQ queue processors
+- Same service layer as Web
+- Handles long-running operations
+
+### Why This Architecture?
+
+**Testability:**
+
+- tRPC procedures easily testable with type-safe callers
+- Services tested independently with mocked DB
+- Queue processors tested with vitest
+- Clear test boundaries
+
+**Maintainability:**
+
+- Business logic isolated in services
+- tRPC provides type safety end-to-end
+- Changes to API don't affect service layer
+- Easy to locate and fix bugs
+
+**Reusability:**
+
+- Services used by tRPC, Public API, Worker, and scripts
+- Business logic not tied to HTTP or tRPC
+- Consistent patterns across packages
+
+**Scalability:**
+
+- Worker handles async operations separately
+- Easy to add new tRPC procedures
+- Clear patterns to follow
+- Shared code in packages/shared
+
+---
+
+## Request Lifecycle
+
+### tRPC Request Flow (UI)
+
+```typescript
+1. HTTP POST /api/trpc/datasets.create
+   ↓
+2. Next.js API route catches request (pages/api/trpc/[trpc].ts)
+   ↓
+3. tRPC router resolves procedure:
+   - Match route to procedure in datasetRouter.ts
+   ↓
+4. tRPC middleware chain executes:
+   - protectedProjectProcedure (authentication)
+   - hasEntitlement checks
+   - Input validation with Zod v4
+   ↓
+5. Procedure handler calls service:
+   export const datasetRouter = createTRPCRouter({
+     create: protectedProjectProcedure
+       .input(createDatasetSchema)
+       .mutation(async ({ input, ctx }) => {
+         return await createDataset(input, ctx.session);
+       }),
+   })
+   ↓
+6. Service executes business logic:
+   - Validate business rules
+   - Use repositories for complex queries or Prisma directly
+   - ClickHouse queries via repositories if needed
+   ↓
+7. Database operations:
+   - prisma.dataset.create({ data })
+   - clickhouse queries via getTracesTable()
+   ↓
+8. Response flows back:
+   Database → Service → Procedure → tRPC → Client
+```
+
+### Public API Request Flow (SDKs)
+
+```typescript
+1. HTTP POST /api/public/datasets
+   ↓
+2. Next.js API route handler (pages/api/public/datasets/index.ts)
+   ↓
+3. withMiddlewares wrapper executes:
+   - Basic auth verification
+   - Rate limiting
+   - CORS handling
+   ↓
+4. createAuthedProjectAPIRoute handler:
+   - Parse and validate request with Zod v4
+   - Extract auth context (project, user)
+   ↓
+5. Handler calls service function:
+   const dataset = await createDataset({
+     name: req.body.name,
+     projectId: req.auth.projectId,
+   });
+   ↓
+6. Service executes (same as tRPC path)
+   ↓
+7. Response formatted and returned:
+   res.status(201).json(dataset);
+```
+
+### Worker/Queue Processing Flow
+
+```typescript
+1. Job added to Redis BullMQ queue:
+   await evalQueue.add("eval-job", {
+     evalId, projectId
+   });
+   ↓
+2. Worker picks up job from Redis
+   ↓
+3. Queue processor handles job:
+   // worker/src/queues/evalQueue.ts
+   async process(job: Job<EvalJobType>) {
+     await processEvaluation(job.data);
+   }
+   ↓
+4. Processor calls service:
+   - Same service layer as Web
+   - Business logic execution
+   ↓
+5. Service performs operations:
+   - Prisma transactions
+   - ClickHouse queries
+   - External API calls (LLMs)
+   ↓
+6. Job completes or fails:
+   - Success: job.updateProgress(100)
+   - Failure: throw error for retry
+```
+
+---
+
+## Directory Structure
+
+### Web Package (`/web/src/`)
+
+```
+web/src/
+├── features/                    # Feature-organized code
+│   ├── datasets/
+│   │   ├── server/             # Backend logic
+│   │   │   ├── datasetRouter.ts    # tRPC router
+│   │   │   └── datasetService.ts   # Business logic
+│   │   ├── components/         # React components
+│   │   └── types/              # Feature types
+│   │
+│   ├── public-api/
+│   │   ├── server/
+│   │   │   ├── withMiddlewares.ts
+│   │   │   └── createAuthedProjectAPIRoute.ts
+│   │   └── types/              # API schemas
+│   │
+│   └── [feature-name]/
+│       ├── server/
+│       │   ├── *Router.ts      # tRPC router
+│       │   └── service.ts      # Business logic
+│       ├── components/
+│       └── types/
+│
+├── server/
+│   ├── api/
+│   │   ├── routers/            # tRPC routers
+│   │   ├── trpc.ts             # tRPC setup & middleware
+│   │   └── root.ts             # Main router combining all
+│   ├── auth.ts                 # NextAuth.js config
+│   └── db.ts                   # Database utilities
+│
+├── pages/
+│   ├── api/
+│   │   ├── public/             # Public REST APIs
+│   │   │   ├── datasets.ts
+│   │   │   └── traces.ts
+│   │   └── trpc/
+│   │       └── [trpc].ts       # tRPC endpoint
+│   └── [routes].tsx            # Next.js pages
+│
+├── __tests__/                  # Jest tests
+│   ├── async/                  # Integration tests
+│   └── sync/                   # Unit tests
+│
+├── instrumentation.ts          # OpenTelemetry (FIRST IMPORT)
+└── env.mjs                     # Environment config
+```
+
+### Worker Package (`/worker/src/`)
+
+```
+worker/src/
+├── queues/                     # BullMQ processors
+│   ├── evalQueue.ts           # Evaluation jobs
+│   ├── ingestionQueue.ts      # Data ingestion
+│   ├── batchExportQueue.ts    # Batch exports
+│   └── workerManager.ts       # Queue registration
+│
+├── features/                   # Business logic
+│   └── [feature]/
+│       └── service.ts
+│
+├── __tests__/                  # Vitest tests
+│
+├── instrumentation.ts          # OpenTelemetry (FIRST IMPORT)
+├── app.ts                      # Express setup + queue registration
+├── env.ts                      # Environment config
+└── index.ts                    # Server start
+```
+
+### Shared Package (`/packages/shared/`)
+
+The shared package provides types, utilities, and server code used by both web and worker packages. It has **5 export paths** that control frontend vs backend access:
+
+| Import Path                                | Usage                 | What's Included                                                                    |
+| ------------------------------------------ | --------------------- | ---------------------------------------------------------------------------------- |
+| `@langfuse/shared`                         | ✅ Frontend + Backend | Prisma types, Zod schemas, constants, table definitions, domain models, utilities  |
+| `@langfuse/shared/src/db`                  | 🔒 Backend only       | Prisma client instance                                                             |
+| `@langfuse/shared/src/server`              | 🔒 Backend only       | Services, repositories, queues, auth, ClickHouse, LLM integration, instrumentation |
+| `@langfuse/shared/src/server/auth/apiKeys` | 🔒 Backend only       | API key management (separated to avoid circular deps)                              |
+| `@langfuse/shared/encryption`              | 🔒 Backend only       | Database field encryption/decryption                                               |
+
+**Key Structure:**
+
+```
+packages/shared/src/
+├── server/                  # 🔒 All server-only code
+│   ├── auth/                # Authentication & authorization
+│   ├── clickhouse/          # ClickHouse client & queries
+│   ├── redis/               # Redis client & 30+ queue types
+│   ├── repositories/        # Data access (traces, observations, scores, events)
+│   ├── services/            # Business services (Storage, Email, Slack, etc.)
+│   ├── llm/                 # LLM integration
+│   ├── instrumentation/     # OpenTelemetry
+│   └── queues.ts, logger.ts, filterToPrisma.ts, etc.
+│
+├── features/                # ✅ Feature types (evals, scores, prompts, datasets)
+├── domain/                  # ✅ Domain models (automations, webhooks, etc.)
+├── tableDefinitions/        # ✅ Table schemas
+├── interfaces/              # ✅ Shared interfaces (filters, orderBy)
+├── utils/                   # ✅ Utilities (JSON, Zod, string checks)
+├── encryption/              # 🔒 Encryption utilities
+└── db.ts, constants.ts, types.ts, etc.
+```
+
+**Common Import Patterns:**
+
+```typescript
+// ✅ Main export - Safe for frontend + backend
+import {
+  Prisma,
+  Role,
+  type Dataset,
+  CloudConfigSchema,
+} from "@langfuse/shared";
+
+// 🔒 Database - Backend only
+import { prisma } from "@langfuse/shared/src/db";
+
+// 🔒 Server utilities - Backend only
+import {
+  logger,
+  instrumentAsync,
+  traceException,
+  redis,
+  clickhouseClient,
+  StorageService,
+  fetchLLMCompletion,
+  filterToPrisma,
+} from "@langfuse/shared/src/server";
+
+// 🔒 API keys - Backend only
+import { createAndAddApiKeysToDb } from "@langfuse/shared/src/server/auth/apiKeys";
+
+// 🔒 Encryption - Backend only
+import { encrypt, decrypt } from "@langfuse/shared/encryption";
+```
+
+---
+
+## Module Organization
+
+### Feature-Based Organization (Recommended)
+
+For most features, organize by domain within `features/`:
+
+```
+src/features/datasets/
+├── server/                 # Backend code
+│   ├── datasetRouter.ts   # tRPC procedures
+│   └── service.ts         # Business logic
+├── components/            # React components
+│   ├── DatasetTable.tsx
+│   └── DatasetForm.tsx
+├── types/                 # Feature types
+│   └── index.ts
+└── utils/                 # Feature utilities
+```
+
+**When to use:**
+
+- Any feature with UI + API
+- Clear domain boundary
+- Multiple related procedures
+
+### Subdomain Organization
+
+For complex features with multiple subdomains:
+
+```
+src/features/evaluations/
+├── server/
+│   ├── evalRouter.ts          # Main router
+│   ├── evalService.ts         # Core service
+│   ├── templates/             # Template subdomain
+│   │   ├── templateRouter.ts
+│   │   └── templateService.ts
+│   └── configs/               # Config subdomain
+│       ├── configRouter.ts
+│       └── configService.ts
+├── components/
+│   ├── templates/
+│   └── configs/
+└── types/
+```
+
+**When to use:**
+
+- Feature has 10+ files
+- Clear subdomains exist
+- Logical grouping improves clarity
+
+### Flat Organization (Rare)
+
+For small, standalone features:
+
+```
+src/server/api/routers/
+├── healthRouter.ts            # Simple health check
+└── versionRouter.ts           # Version info
+```
+
+**When to use:**
+
+- Simple features (1-2 procedures)
+- No UI components
+- Standalone utilities
+
+---
+
+## Separation of Concerns
+
+### What Goes Where
+
+**tRPC Procedures (Entry Layer):**
+
+- ✅ Procedure definitions (query/mutation)
+- ✅ Middleware application (auth, validation)
+- ✅ Input schemas (Zod v4)
+- ✅ Service delegation
+- ✅ Error transformation (TRPCError)
+- ❌ Business logic (belongs in services)
+- ❌ Database operations (belongs in services)
+- ❌ Complex validation (belongs in services)
+
+**Public API Routes (Entry Layer):**
+
+- ✅ Route registration
+- ✅ Middleware wrapper application
+- ✅ Input validation (Zod v4)
+- ✅ Service delegation
+- ✅ Response formatting
+- ✅ HTTP status codes
+- ❌ Business logic (belongs in services)
+- ❌ Database operations (belongs in services)
+
+**Services Layer:**
+
+- ✅ Business logic
+- ✅ Business rules enforcement
+- ✅ Transaction orchestration
+- ✅ Repository calls for complex queries
+- ✅ Direct Prisma operations for simple CRUD
+- ✅ ClickHouse queries (via repositories)
+- ✅ Redis cache access
+- ✅ External API calls (LLMs, etc.)
+- ❌ HTTP concerns (Request/Response)
+- ❌ tRPC-specific types (TRPCError in entry layer)
+- ❌ NextAuth session handling (passed as parameter)
+
+**Queue Processors (Worker):**
+
+- ✅ Job registration and configuration
+- ✅ Job data extraction
+- ✅ Service delegation
+- ✅ Progress updates
+- ✅ Error handling (retry logic)
+- ❌ Business logic (belongs in services)
+- ❌ Database operations (belongs in services)
+
+### Example: Dataset Creation
+
+**tRPC Procedure (Entry Point):**
+
+```typescript
+// web/src/features/datasets/server/datasetRouter.ts
+import { z } from "zod/v4";
+import {
+  createTRPCRouter,
+  protectedProjectProcedure,
+} from "@/src/server/api/trpc";
+import { TRPCError } from "@trpc/server";
+import { createDataset } from "./service";
+
+export const datasetRouter = createTRPCRouter({
+  create: protectedProjectProcedure
+    .input(
+      z.object({
+        name: z.string(),
+        description: z.string().optional(),
+        projectId: z.string(),
+      }),
+    )
+    .mutation(async ({ input, ctx }) => {
+      try {
+        return await createDataset({
+          ...input,
+          userId: ctx.session.user.id,
+        });
+      } catch (error) {
+        throw new TRPCError({
+          code: "INTERNAL_SERVER_ERROR",
+          message: "Failed to create dataset",
+          cause: error,
+        });
+      }
+    }),
+});
+```
+
+**Service (Business Logic):**
+
+```typescript
+// web/src/features/datasets/server/service.ts
+import { prisma } from "@langfuse/shared/src/db";
+import { instrumentAsync, traceException } from "@langfuse/shared/src/server";
+
+export async function createDataset(data: {
+  name: string;
+  description?: string;
+  projectId: string;
+  userId: string;
+}) {
+  return await instrumentAsync({ name: "dataset.create" }, async (span) => {
+    // Business rule: Check for duplicate names in project
+    const existing = await prisma.dataset.findFirst({
+      where: {
+        name: data.name,
+        projectId: data.projectId,
+      },
+    });
+
+    if (existing) {
+      throw new Error(`Dataset with name "${data.name}" already exists`);
+    }
+
+    // Create dataset
+    const dataset = await prisma.dataset.create({
+      data: {
+        name: data.name,
+        description: data.description,
+        projectId: data.projectId,
+        createdById: data.userId,
+      },
+    });
+
+    span.setAttributes({
+      datasetId: dataset.id,
+      projectId: dataset.projectId,
+    });
+
+    return dataset;
+  });
+}
+```
+
+**Public API (Alternative Entry Point):**
+
+```typescript
+// web/src/pages/api/public/datasets/index.ts
+import { withMiddlewares } from "@/src/features/public-api/server/withMiddlewares";
+import { createAuthedProjectAPIRoute } from "@/src/features/public-api/server/createAuthedProjectAPIRoute";
+import { createDataset } from "@/src/features/datasets/server/service";
+import { z } from "zod/v4";
+
+const createDatasetSchema = z.object({
+  name: z.string(),
+  description: z.string().optional(),
+});
+
+export default withMiddlewares({
+  POST: createAuthedProjectAPIRoute({
+    name: "Create Dataset",
+    bodySchema: createDatasetSchema,
+    fn: async ({ body, auth, res }) => {
+      const dataset = await createDataset({
+        name: body.name,
+        description: body.description,
+        projectId: auth.scope.projectId,
+        userId: auth.scope.userId,
+      });
+
+      return res.status(201).json(dataset);
+    },
+  }),
+});
+```
+
+**Queue Processor (Async Processing):**
+
+```typescript
+// worker/src/queues/datasetExportQueue.ts
+import { Job } from "bullmq";
+import { exportDataset } from "../features/datasets/exportService";
+
+export async function processDatasetExport(
+  job: Job<{ datasetId: string; projectId: string; format: string }>,
+) {
+  const { datasetId, projectId, format } = job.data;
+
+  await job.updateProgress(10);
+
+  // Delegate to service
+  const exportUrl = await exportDataset({
+    datasetId,
+    projectId,
+    format,
+    onProgress: (percent) => job.updateProgress(percent),
+  });
+
+  await job.updateProgress(100);
+
+  return { exportUrl };
+}
+```
+
+**Notice:** Each layer has clear, distinct responsibilities!
+
+- **Entry layers** (tRPC/Public API/Queue) handle protocol concerns
+- **Service layer** contains all business logic
+- **Data layer** accessed via repositories (complex queries) or Prisma directly (simple CRUD)
+
+---
+
+## Database Architecture
+
+### Dual Database System
+
+Langfuse uses two databases with different purposes:
+
+```
+┌─────────────────────────────────────────────────────────────┐
+│                       Application                           │
+│                                                             │
+│  ┌──────────────┐              ┌──────────────┐           │
+│  │  PostgreSQL  │              │  ClickHouse  │           │
+│  │              │              │              │           │
+│  │ Transactional│              │  Analytics   │           │
+│  │    Data      │              │    Data      │           │
+│  └──────────────┘              └──────────────┘           │
+│         ↑                              ↑                   │
+│         │                              │                   │
+│    Prisma ORM                    Direct SQL               │
+│  (schema migrations)              (via client)            │
+└─────────────────────────────────────────────────────────────┘
+```
+
+**PostgreSQL (Primary Database):**
+
+- Accessed via Prisma ORM
+- Transactional data (users, projects, datasets, etc.)
+- ACID guarantees
+- Schema managed via `prisma migrate`
+- Located in `packages/shared/prisma/`
+
+**ClickHouse (Analytics Database):**
+
+- Accessed via direct SQL queries
+- High-volume trace/observation data
+- Columnar storage for analytics
+- Optimized for aggregations
+- Schema in `packages/shared/src/server/clickhouse/`
+- Schema managed via `golang-migrate`
+
+**Redis (Cache & Queues):**
+
+- BullMQ job queues
+- Caching layer
+- Session storage
+- Rate limiting
+
+### Data Access Pattern
+
+**Services access databases directly:**
+
+```typescript
+// PostgreSQL via Prisma
+import { prisma } from "@langfuse/shared/src/db";
+
+const dataset = await prisma.dataset.create({ data });
+
+// ClickHouse via helper functions
+import { getTracesTable } from "@langfuse/shared/src/server";
+
+const traces = await getTracesTable({
+  projectId,
+  filter: [...],
+  limit: 1000,
+});
+
+// Redis via queue/cache utilities
+import { redis } from "@langfuse/shared/src/server";
+
+await redis.set(`cache:${key}`, value, "EX", 3600);
+```
+
+**Repository Pattern:**
+
+Langfuse uses repositories in `packages/shared/src/server/repositories/` for complex data access patterns. Repositories provide:
+
+- Abstraction over complex queries (traces, observations, scores, events)
+- Data converters for transforming database models to application models
+- ClickHouse query builders and stream processing
+- Reusable query logic across services
+
+Services can use repositories for complex operations OR Prisma directly for simple CRUD operations.
+
+---
+
+## Best Practices
+
+### 1. Keep Procedures Thin
+
+tRPC procedures should only handle protocol concerns:
+
+```typescript
+// ❌ BAD: Business logic in procedure
+export const datasetRouter = createTRPCRouter({
+  create: protectedProjectProcedure
+    .input(createSchema)
+    .mutation(async ({ input, ctx }) => {
+      // 200 lines of business logic here
+      const existing = await prisma.dataset.findFirst(...);
+      if (existing) throw new Error(...);
+      const dataset = await prisma.dataset.create(...);
+      await sendNotification(...);
+      return dataset;
+    }),
+});
+
+// ✅ GOOD: Delegate to service
+export const datasetRouter = createTRPCRouter({
+  create: protectedProjectProcedure
+    .input(createSchema)
+    .mutation(async ({ input, ctx }) => {
+      return await createDataset(input, ctx.session);
+    }),
+});
+```
+
+### 2. Services Should Be Protocol-Agnostic
+
+Services should work regardless of entry point:
+
+```typescript
+// ✅ GOOD: No HTTP/tRPC knowledge
+export async function createDataset(data: CreateDatasetInput) {
+  // Pure business logic
+  return await prisma.dataset.create({ data });
+}
+
+// ❌ BAD: tRPC-specific
+export async function createDataset(ctx: TRPCContext) {
+  // Coupled to tRPC
+}
+```
+
+### 3. Observability with OpenTelemetry + DataDog
+
+**Langfuse uses OpenTelemetry for backend observability, with traces and logs sent to DataDog.**
+
+Use structured logging and instrumentation:
+
+```typescript
+import {
+  logger,
+  traceException,
+  instrumentAsync,
+} from "@langfuse/shared/src/server";
+
+export async function processEvaluation(evalId: string) {
+  return await instrumentAsync(
+    { name: "evaluation.process", attributes: { evalId } },
+    async (span) => {
+      // Structured logging (includes trace_id, span_id, dd.trace_id)
+      logger.info("Starting evaluation", { evalId });
+
+      try {
+        // Operation here
+        const result = await runEvaluation(evalId);
+
+        span.setAttributes({
+          score: result.score,
+          status: "success",
+        });
+
+        return result;
+      } catch (error) {
+        // Record exception to OpenTelemetry span (sent to DataDog)
+        traceException(error, span);
+        logger.error("Evaluation failed", { evalId, error: error.message });
+        throw error;
+      }
+    },
+  );
+}
+```
+
+**Note**: Frontend uses Sentry for error tracking, but backend (tRPC, API routes, services, worker) uses OpenTelemetry + DataDog.
+
+### 4. Use Proper Error Handling
+
+Transform errors at entry points:
+
+```typescript
+// tRPC procedure
+try {
+  return await service();
+} catch (error) {
+  traceException(error); // Record to OpenTelemetry span (sent to DataDog)
+  throw new TRPCError({
+    code: "INTERNAL_SERVER_ERROR",
+    message: "User-friendly message",
+    cause: error,
+  });
+}
+
+// Public API
+try {
+  return await service();
+} catch (error) {
+  traceException(error); // Record to OpenTelemetry span (sent to DataDog)
+  return res.status(500).json({
+    error: "User-friendly message",
+  });
+}
+```
+
+### 5. Validate at Entry Points
+
+Use Zod v4 for all input validation:
+
+```typescript
+import { z } from "zod/v4";
+
+// tRPC
+.input(z.object({
+  name: z.string().min(1).max(255),
+  projectId: z.string(),
+}))
+
+// Public API
+const bodySchema = z.object({
+  name: z.string().min(1).max(255),
+});
+const validated = bodySchema.parse(req.body);
+```
+
+---
+
+**Related Files:**
+
+- [../SKILL.md](../SKILL.md) - Main guide
+- [routing-and-controllers.md](routing-and-controllers.md) - tRPC and Public API details
+- [services-and-repositories.md](services-and-repositories.md) - Service patterns
+- [testing-guide.md](testing-guide.md) - Testing strategies

.agents/skills/backend-dev-guidelines/references/configuration.md

@@ -0,0 +1,563 @@
+# Configuration Management - Environment Variables
+
+Complete guide to managing configuration across Langfuse's monorepo packages.
+
+## Table of Contents
+
+- [Environment Variable Pattern](#environment-variable-pattern)
+- [Package-Specific Configuration](#package-specific-configuration)
+- [Special Environment Variables](#special-environment-variables)
+- [Best Practices](#best-practices)
+
+---
+
+## Environment Variable Pattern
+
+### Why Zod-Validated Environment Variables?
+
+**Problems with raw process.env:**
+
+- ❌ No type safety
+- ❌ No validation
+- ❌ Hard to test
+- ❌ Runtime errors for typos
+- ❌ No default values
+
+**Benefits of Zod validation:**
+
+- ✅ Type-safe configuration
+- ✅ Validated at startup
+- ✅ Clear error messages
+- ✅ Default values
+- ✅ Environment-specific transformation
+
+---
+
+## Package-Specific Configuration
+
+Each package has its own `env.ts` or `env.mjs` file that validates and exports environment variables:
+
+```
+langfuse/
+├── web/src/env.mjs              # Next.js app (t3-env pattern)
+├── worker/src/env.ts            # Worker service (Zod schema)
+├── packages/shared/src/env.ts   # Shared config (Zod schema)
+└── ee/src/env.ts                # Enterprise Edition (Zod schema)
+```
+
+### Web Package (`web/src/env.mjs`)
+
+Uses **t3-oss/env-nextjs** for Next.js-specific validation with server/client separation.
+
+**Key Features:**
+
+- Separates server-side and client-side environment variables
+- Client variables must be prefixed with `NEXT_PUBLIC_`
+- Validates at build time (unless `DOCKER_BUILD=1`)
+- `runtimeEnv` section manually maps all variables
+
+**Structure:**
+
+```typescript
+import { createEnv } from "@t3-oss/env-nextjs";
+import { z } from "zod";
+
+export const env = createEnv({
+  // Server-side only variables (never exposed to client)
+  server: {
+    DATABASE_URL: z.string().url(),
+    NEXTAUTH_SECRET: z.string().min(1),
+    SALT: z.string(),
+    CLICKHOUSE_URL: z.string().url(),
+    // ... 100+ server variables
+  },
+
+  // Client-side variables (exposed to browser)
+  client: {
+    NEXT_PUBLIC_LANGFUSE_CLOUD_REGION: z
+      .enum(["US", "EU", "STAGING", "DEV", "HIPAA", "JP"])
+      .optional(),
+    NEXT_PUBLIC_SIGN_UP_DISABLED: z.enum(["true", "false"]).default("false"),
+    // ... client variables
+  },
+
+  // Runtime mapping (required for Next.js edge runtime)
+  runtimeEnv: {
+    DATABASE_URL: process.env.DATABASE_URL,
+    NEXTAUTH_SECRET: process.env.NEXTAUTH_SECRET,
+    NEXT_PUBLIC_LANGFUSE_CLOUD_REGION:
+      process.env.NEXT_PUBLIC_LANGFUSE_CLOUD_REGION,
+    // ... must map ALL variables
+  },
+
+  // Skip validation in Docker builds
+  skipValidation: process.env.DOCKER_BUILD === "1",
+  emptyStringAsUndefined: true,
+});
+```
+
+**Usage:**
+
+```typescript
+// In server-side code (tRPC, API routes)
+import { env } from "@/src/env.mjs";
+
+const dbUrl = env.DATABASE_URL;
+const salt = env.SALT;
+
+// In client-side code (React components)
+import { env } from "@/src/env.mjs";
+
+const region = env.NEXT_PUBLIC_LANGFUSE_CLOUD_REGION;
+```
+
+### Worker Package (`worker/src/env.ts`)
+
+Uses **plain Zod schema** for Express.js worker service.
+
+**Structure:**
+
+```typescript
+import { z } from "zod/v4";
+import { removeEmptyEnvVariables } from "@langfuse/shared";
+
+const EnvSchema = z.object({
+  BUILD_ID: z.string().optional(),
+  NODE_ENV: z
+    .enum(["development", "test", "production"])
+    .default("development"),
+  DATABASE_URL: z.string(),
+  PORT: z.coerce.number().positive().max(65536).default(3030),
+
+  // ClickHouse
+  CLICKHOUSE_URL: z.string().url(),
+  CLICKHOUSE_USER: z.string(),
+  CLICKHOUSE_PASSWORD: z.string(),
+
+  // S3 Event Upload (required)
+  LANGFUSE_S3_EVENT_UPLOAD_BUCKET: z.string({
+    error: "Langfuse requires a bucket name for S3 Event Uploads.",
+  }),
+
+  // Queue concurrency settings
+  LANGFUSE_INGESTION_QUEUE_PROCESSING_CONCURRENCY: z.coerce
+    .number()
+    .positive()
+    .default(20),
+  LANGFUSE_EVAL_EXECUTION_WORKER_CONCURRENCY: z.coerce
+    .number()
+    .positive()
+    .default(5),
+
+  // Queue consumer toggles
+  QUEUE_CONSUMER_INGESTION_QUEUE_IS_ENABLED: z
+    .enum(["true", "false"])
+    .default("true"),
+  QUEUE_CONSUMER_BATCH_EXPORT_QUEUE_IS_ENABLED: z
+    .enum(["true", "false"])
+    .default("true"),
+
+  // ... 150+ worker-specific variables
+});
+
+export const env: z.infer<typeof EnvSchema> =
+  process.env.DOCKER_BUILD === "1"
+    ? (process.env as any)
+    : EnvSchema.parse(removeEmptyEnvVariables(process.env));
+```
+
+**Usage:**
+
+```typescript
+import { env } from "./env";
+
+const concurrency = env.LANGFUSE_INGESTION_QUEUE_PROCESSING_CONCURRENCY;
+const s3Bucket = env.LANGFUSE_S3_EVENT_UPLOAD_BUCKET;
+```
+
+### Shared Package (`packages/shared/src/env.ts`)
+
+Uses **plain Zod schema** for configuration shared between web and worker.
+
+**Structure:**
+
+```typescript
+import { z } from "zod/v4";
+import { removeEmptyEnvVariables } from "./utils/environment";
+
+const EnvSchema = z.object({
+  NODE_ENV: z
+    .enum(["development", "test", "production"])
+    .default("development"),
+
+  // Redis configuration
+  REDIS_HOST: z.string().nullish(),
+  REDIS_PORT: z.coerce.number().positive().max(65536).default(6379).nullable(),
+  REDIS_AUTH: z.string().nullish(),
+  REDIS_CONNECTION_STRING: z.string().nullish(),
+  REDIS_CLUSTER_ENABLED: z.enum(["true", "false"]).default("false"),
+
+  // ClickHouse
+  CLICKHOUSE_URL: z.string().url(),
+  CLICKHOUSE_USER: z.string(),
+  CLICKHOUSE_PASSWORD: z.string(),
+  CLICKHOUSE_MAX_OPEN_CONNECTIONS: z.coerce.number().int().default(25),
+
+  // S3 Event Upload
+  LANGFUSE_S3_EVENT_UPLOAD_BUCKET: z.string(),
+  LANGFUSE_S3_EVENT_UPLOAD_REGION: z.string().optional(),
+
+  // Logging
+  LANGFUSE_LOG_LEVEL: z
+    .enum(["trace", "debug", "info", "warn", "error", "fatal"])
+    .optional(),
+  LANGFUSE_LOG_FORMAT: z.enum(["text", "json"]).default("text"),
+
+  // Encryption
+  ENCRYPTION_KEY: z
+    .string()
+    .length(
+      64,
+      "ENCRYPTION_KEY must be 256 bits, 64 string characters in hex format, generate via: openssl rand -hex 32",
+    )
+    .optional(),
+
+  // ... 80+ shared variables
+});
+
+export const env: z.infer<typeof EnvSchema> =
+  process.env.DOCKER_BUILD === "1"
+    ? (process.env as any)
+    : EnvSchema.parse(removeEmptyEnvVariables(process.env));
+```
+
+**Usage:**
+
+```typescript
+import { env } from "@langfuse/shared/src/env";
+
+const redisHost = env.REDIS_HOST;
+const clickhouseUrl = env.CLICKHOUSE_URL;
+```
+
+### Enterprise Edition Package (`ee/src/env.ts`)
+
+Minimal Zod schema for EE-specific variables.
+
+**Structure:**
+
+```typescript
+import { z } from "zod/v4";
+import { removeEmptyEnvVariables } from "@langfuse/shared";
+
+const EnvSchema = z.object({
+  NEXT_PUBLIC_LANGFUSE_CLOUD_REGION: z.string().optional(),
+  LANGFUSE_EE_LICENSE_KEY: z.string().optional(),
+});
+
+export const env = EnvSchema.parse(removeEmptyEnvVariables(process.env));
+```
+
+**Usage:**
+
+```typescript
+import { env } from "@langfuse/ee/src/env";
+
+const licenseKey = env.LANGFUSE_EE_LICENSE_KEY;
+```
+
+---
+
+## Special Environment Variables
+
+### NEXT_PUBLIC_LANGFUSE_CLOUD_REGION
+
+**Purpose:** Identifies the cloud deployment region for Langfuse Cloud.
+
+**Type:** `"US" | "EU" | "STAGING" | "DEV" | "HIPAA" | "JP" | undefined`
+
+**Where Used:**
+
+- **web/src/env.mjs** - Client-side accessible (prefixed with `NEXT_PUBLIC_`)
+- **ee/src/env.ts** - Enterprise features
+- **packages/shared/src/env.ts** - Shared logic
+- **worker/src/env.ts** - Worker processing
+
+**When Set:**
+
+| Environment              | Value                  | Purpose                                        |
+| ------------------------ | ---------------------- | ---------------------------------------------- |
+| **Developer Laptop**     | `"DEV"` or `"STAGING"` | Local development against cloud infrastructure |
+| **Langfuse Cloud US**    | `"US"`                 | Production US region                           |
+| **Langfuse Cloud EU**    | `"EU"`                 | Production EU region                           |
+| **Langfuse Cloud HIPAA** | `"HIPAA"`              | HIPAA-compliant region                         |
+| **Langfuse Cloud JP**    | `"JP"`                 | Production JP region                           |
+| **OSS Self-Hosted**      | `undefined` (not set)  | Self-hosted deployments don't have region      |
+
+**Use Cases:**
+
+```typescript
+// Check if running in cloud
+if (env.NEXT_PUBLIC_LANGFUSE_CLOUD_REGION) {
+  // Enable cloud-specific features
+  - Usage metering and billing
+  - Cloud spend alerts
+  - Free tier enforcement
+  - Stripe integration
+  - PostHog analytics
+}
+
+// Region-specific behavior
+if (env.NEXT_PUBLIC_LANGFUSE_CLOUD_REGION === "HIPAA") {
+  // HIPAA compliance features
+}
+
+// Development/staging checks
+if (env.NEXT_PUBLIC_LANGFUSE_CLOUD_REGION === "DEV") {
+  // Enable debug features
+}
+```
+
+**Example Configuration:**
+
+```bash
+# .env file on developer laptop
+NEXT_PUBLIC_LANGFUSE_CLOUD_REGION=DEV
+
+# Cloud US deployment
+NEXT_PUBLIC_LANGFUSE_CLOUD_REGION=US
+
+# Self-hosted OSS deployment
+# (variable not set)
+```
+
+### LANGFUSE_EE_LICENSE_KEY
+
+**Purpose:** Enables Enterprise Edition features in self-hosted deployments.
+
+**Type:** `string | undefined`
+
+**Where Used:**
+
+- **web/src/env.mjs** - Web app EE features
+- **ee/src/env.ts** - EE package
+
+**When Set:**
+
+| Deployment          | Value              | Features Enabled                                                 |
+| ------------------- | ------------------ | ---------------------------------------------------------------- |
+| **Langfuse Cloud**  | Not set            | Cloud features controlled by `NEXT_PUBLIC_LANGFUSE_CLOUD_REGION` |
+| **OSS Self-Hosted** | Not set            | Core open-source features only                                   |
+| **EE Self-Hosted**  | License key string | Enterprise features enabled                                      |
+
+**Enterprise Features Controlled:**
+
+When `LANGFUSE_EE_LICENSE_KEY` is set and valid:
+
+- SSO integrations (custom OIDC, SAML)
+- Advanced RBAC
+- Audit logging
+- Custom branding
+- SLA support
+- Advanced security features
+
+**Usage Pattern:**
+
+```typescript
+import { env } from "@/src/env.mjs";
+
+// Check if EE license is present
+if (env.LANGFUSE_EE_LICENSE_KEY) {
+  // Validate license
+  const isValidLicense = await validateEELicense(env.LANGFUSE_EE_LICENSE_KEY);
+
+  if (isValidLicense) {
+    // Enable EE features
+    enableCustomSSO();
+    enableAdvancedRBAC();
+  }
+}
+```
+
+**Example Configuration:**
+
+```bash
+# OSS self-hosted (no license)
+# LANGFUSE_EE_LICENSE_KEY not set
+
+# EE self-hosted
+LANGFUSE_EE_LICENSE_KEY=ee_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
+
+# Langfuse Cloud (uses region instead)
+NEXT_PUBLIC_LANGFUSE_CLOUD_REGION=US
+# LANGFUSE_EE_LICENSE_KEY not used
+```
+
+### Other Important Variables
+
+**DOCKER_BUILD**
+
+```typescript
+// Skip validation during Docker builds
+skipValidation: process.env.DOCKER_BUILD === "1";
+```
+
+**Purpose:** Docker builds happen before runtime env vars are available, so validation must be skipped.
+
+**SALT**
+
+```typescript
+SALT: z.string({
+  required_error: "A strong Salt is required to encrypt API keys securely.",
+});
+```
+
+**Purpose:** Required for encrypting API keys in database. Must be set in production.
+
+**ENCRYPTION_KEY**
+
+```typescript
+ENCRYPTION_KEY: z.string().length(64, "Must be 256 bits, 64 hex characters");
+```
+
+**Purpose:** Optional 256-bit key for encrypting sensitive database fields.
+
+**Generate:** `openssl rand -hex 32`
+
+---
+
+## Best Practices
+
+### 1. Always Import from env.mjs/env.ts
+
+```typescript
+// ❌ NEVER DO THIS
+const dbUrl = process.env.DATABASE_URL;
+
+// ✅ ALWAYS DO THIS
+import { env } from "@/src/env.mjs";
+const dbUrl = env.DATABASE_URL; // Type-safe, validated
+```
+
+### 2. Use Appropriate Import Path
+
+```typescript
+// In web package
+import { env } from "@/src/env.mjs";
+
+// In worker package
+import { env } from "./env";
+
+// In shared package
+import { env } from "@langfuse/shared/src/env";
+```
+
+### 3. Client Variables Must Start with NEXT*PUBLIC*
+
+```typescript
+// ❌ Won't work in browser
+API_KEY: z.string(); // in server config
+
+// ✅ Accessible in browser
+NEXT_PUBLIC_API_KEY: z.string(); // in client config
+```
+
+### 4. Provide Sensible Defaults for Development
+
+```typescript
+PORT: z.coerce.number().positive().default(3030),
+NODE_ENV: z.enum(["development", "test", "production"]).default("development"),
+REDIS_PORT: z.coerce.number().positive().default(6379),
+```
+
+### 5. Use Coercion for Numbers
+
+```typescript
+// .env files are always strings
+PORT: z.coerce.number(); // Converts "3000" to 3000
+```
+
+### 6. Transform Complex Values
+
+```typescript
+// Split comma-separated values
+LANGFUSE_LOG_PROPAGATED_HEADERS: z.string().optional().transform((s) =>
+  s ? s.split(",").map((s) => s.toLowerCase().trim()) : []
+),
+
+// Parse project:rate pairs
+LANGFUSE_INGESTION_PROCESSING_SAMPLED_PROJECTS: z.string().optional().transform((val) => {
+  const map = new Map<string, number>();
+  val?.split(",").forEach(part => {
+    const [projectId, rate] = part.split(":");
+    map.set(projectId, parseFloat(rate));
+  });
+  return map;
+}),
+```
+
+### 7. Validation at Startup
+
+All environment variables are validated when the application starts. Invalid configuration will cause immediate failure with clear error messages:
+
+```bash
+❌ Validation error:
+  - SALT: Required
+  - CLICKHOUSE_URL: Invalid url
+  - PORT: Number must be less than or equal to 65536
+```
+
+### 8. Skip Validation in Docker Builds
+
+Always include the Docker build escape hatch:
+
+```typescript
+export const env =
+  process.env.DOCKER_BUILD === "1"
+    ? (process.env as any)
+    : EnvSchema.parse(removeEmptyEnvVariables(process.env));
+```
+
+### 9. Use removeEmptyEnvVariables Helper
+
+Treats empty strings as undefined:
+
+```typescript
+import { removeEmptyEnvVariables } from "@langfuse/shared";
+
+EnvSchema.parse(removeEmptyEnvVariables(process.env));
+```
+
+This prevents errors from `.env` files with empty values:
+
+```bash
+# .env
+OPTIONAL_VAR=    # Treated as undefined, not empty string
+```
+
+---
+
+## Configuration File Locations
+
+```
+langfuse/
+├── .env                          # Local development overrides
+├── .env.dev.example              # Example dev configuration
+├── web/src/env.mjs               # Web app env validation
+├── worker/src/env.ts             # Worker env validation
+├── packages/shared/src/env.ts    # Shared env validation
+└── ee/src/env.ts                 # EE env validation
+```
+
+**DO NOT commit:**
+
+- `.env`
+- `.env.local`
+- `.env.production`
+
+---
+
+**Related Files:**
+
+- [../SKILL.md](../SKILL.md) - Main guide
+- [architecture-overview.md](architecture-overview.md) - Architecture patterns

.agents/skills/backend-dev-guidelines/references/database-patterns.md

@@ -0,0 +1,691 @@
+# Database Patterns - PostgreSQL & ClickHouse
+
+Complete guide to database access patterns in Langfuse using PostgreSQL (Prisma ORM) and ClickHouse (direct client).
+
+## Table of Contents
+
+- [Database Architecture Overview](#database-architecture-overview)
+- [PostgreSQL with Prisma](#postgresql-with-prisma)
+- [ClickHouse with Direct Client](#clickhouse-with-direct-client)
+- [Repository Pattern](#repository-pattern)
+- [When to Use Which Database](#when-to-use-which-database)
+- [Error Handling](#error-handling)
+
+---
+
+## Database Architecture Overview
+
+Langfuse uses a **dual database architecture**:
+
+| Database       | Technology        | Purpose                                                       | Access Pattern                         |
+| -------------- | ----------------- | ------------------------------------------------------------- | -------------------------------------- |
+| **PostgreSQL** | Prisma ORM        | Transactional data, relational data, CRUD operations          | Type-safe ORM with migrations          |
+| **ClickHouse** | Direct SQL client | Analytics data, high-volume traces/observations, aggregations | Raw SQL queries with streaming support |
+| **Redis**      | ioredis           | Queues (BullMQ), caching, rate limiting                       | Direct client access                   |
+
+**Key Principle**: Use PostgreSQL for transactional data and relationships. Use ClickHouse for high-volume analytics and time-series data.
+
+**⚠️ Important**: All queries must filter by `project_id` (or `projectId`) to ensure proper data isolation between tenants. This is essential for the multi-tenant architecture.
+
+---
+
+## PostgreSQL with Prisma
+
+### Import Pattern
+
+```typescript
+import { prisma } from "@langfuse/shared/src/db";
+
+// Direct access to Prisma client
+const user = await prisma.user.findUnique({ where: { id } });
+```
+
+**Important**: Always import from `@langfuse/shared/src/db`, not `@prisma/client` directly.
+
+### Common CRUD Operations
+
+**⚠️ ALWAYS include `projectId` in WHERE clauses** for project-scoped data:
+
+```typescript
+// Create
+const project = await prisma.project.create({
+  data: {
+    name: "My Project",
+    orgId: organizationId,
+  },
+});
+
+// ✅ GOOD: Read with projectId filter
+const trace = await prisma.trace.findUnique({
+  where: { id: traceId, projectId }, // ← Always include projectId for tenant isolation
+  include: {
+    scores: true,
+    project: { select: { id: true, name: true } },
+  },
+});
+
+// ❌ BAD: Missing projectId filter
+// const trace = await prisma.trace.findUnique({
+//   where: { id: traceId },  // ← Missing projectId!
+// });
+
+// Update
+await prisma.user.update({
+  where: { id: userId },
+  data: { lastLogin: new Date() },
+});
+
+// ✅ GOOD: Delete with projectId
+await prisma.apiKey.delete({
+  where: { id: apiKeyId, projectId }, // ← Always include projectId
+});
+
+// ✅ GOOD: Count with projectId
+const traceCount = await prisma.trace.count({
+  where: { projectId, userId }, // ← Always include projectId
+});
+```
+
+### Transactions
+
+Use Prisma interactive transactions for operations that must be atomic:
+
+```typescript
+const result = await prisma.$transaction(async (tx) => {
+  const user = await tx.user.create({ data: userData });
+
+  const project = await tx.project.create({
+    data: {
+      name: "Default Project",
+      orgId: user.id,
+    },
+  });
+
+  await tx.projectMembership.create({
+    data: {
+      userId: user.id,
+      projectId: project.id,
+      role: "OWNER",
+    },
+  });
+
+  return { user, project };
+});
+```
+
+**Transaction options:**
+
+```typescript
+await prisma.$transaction(
+  async (tx) => {
+    // Transaction logic
+  },
+  {
+    maxWait: 5000, // Max time to wait for transaction to start (ms)
+    timeout: 10000, // Max time transaction can run (ms)
+  },
+);
+```
+
+### Query Optimization
+
+**Use `select` to limit fields:**
+
+```typescript
+// ❌ Fetches all fields (including large JSON columns)
+const traces = await prisma.trace.findMany({ where: { projectId } });
+
+// ✅ Only fetch needed fields
+const traces = await prisma.trace.findMany({
+  where: { projectId },
+  select: {
+    id: true,
+    name: true,
+    timestamp: true,
+    userId: true,
+  },
+});
+```
+
+**Prevent N+1 queries with `include`:**
+
+```typescript
+// ❌ N+1 Query Problem
+const projects = await prisma.project.findMany();
+for (const project of projects) {
+  // N additional queries
+  const memberCount = await prisma.projectMembership.count({
+    where: { projectId: project.id },
+  });
+}
+
+// ✅ Use include or aggregation
+const projects = await prisma.project.findMany({
+  include: {
+    members: { select: { userId: true, role: true } },
+  },
+});
+```
+
+**Pagination:**
+
+```typescript
+const PAGE_SIZE = 50;
+
+const traces = await prisma.trace.findMany({
+  where: { projectId },
+  orderBy: { timestamp: "desc" },
+  take: PAGE_SIZE,
+  skip: page * PAGE_SIZE,
+});
+```
+
+## ClickHouse with Direct Client
+
+### Import Pattern
+
+```typescript
+import { queryClickhouse } from "@langfuse/shared/src/server/repositories/clickhouse";
+import { clickhouseClient } from "@langfuse/shared/src/server/clickhouse/client";
+```
+
+### ClickHouse Client Singleton
+
+ClickHouse uses a singleton client manager that reuses connections:
+
+```typescript
+import { clickhouseClient } from "@langfuse/shared/src/server/clickhouse/client";
+
+// Get client (automatically reuses existing connection)
+const client = clickhouseClient();
+
+// For read-only queries (uses read replica if configured)
+const client = clickhouseClient(undefined, "ReadOnly");
+```
+
+### Query Patterns
+
+ClickHouse queries use **raw SQL** with parameterized queries. Parameters use `{paramName: Type}` syntax:
+
+**⚠️ Important**: All ClickHouse queries must include `project_id` filter to ensure proper tenant isolation.
+
+**Simple query:**
+
+```typescript
+import { queryClickhouse } from "@langfuse/shared/src/server/repositories/clickhouse";
+
+// ✅ GOOD: Always filter by project_id
+const rows = await queryClickhouse<{ id: string; name: string }>({
+  query: `
+    SELECT id, name, timestamp
+    FROM traces
+    WHERE project_id = {projectId: String}  -- ← REQUIRED: Always filter by project_id
+    AND timestamp >= {startTime: DateTime64(3)}
+    ORDER BY timestamp DESC
+    LIMIT {limit: UInt32}
+  `,
+  params: {
+    projectId, // ← Required for tenant isolation
+    startTime: convertDateToClickhouseDateTime(startDate),
+    limit: 100,
+  },
+  tags: { feature: "tracing", type: "trace" },
+});
+
+// ❌ BAD: Missing project_id filter
+// const rows = await queryClickhouse({
+//   query: `SELECT * FROM traces WHERE timestamp >= {startTime: DateTime64(3)}`,
+//   params: { startTime },
+// });
+```
+
+**Streaming query (for large result sets):**
+
+```typescript
+import { queryClickhouseStream } from "@langfuse/shared/src/server/repositories/clickhouse";
+
+// Stream results to avoid loading all rows in memory
+for await (const row of queryClickhouseStream<ObservationRecordReadType>({
+  query: `
+    SELECT *
+    FROM observations
+    WHERE project_id = {projectId: String}
+    AND start_time >= {startTime: DateTime64(3)}
+  `,
+  params: { projectId, startTime },
+})) {
+  // Process row by row
+  await processObservation(row);
+}
+```
+
+**Upsert (insert) operation:**
+
+```typescript
+import { upsertClickhouse } from "@langfuse/shared/src/server/repositories/clickhouse";
+
+await upsertClickhouse({
+  table: "traces",
+  records: [
+    {
+      id: traceId,
+      project_id: projectId,
+      timestamp: new Date(),
+      name: "API Call",
+      user_id: userId,
+      // ... other fields
+    },
+  ],
+  eventBodyMapper: (record) => ({
+    // Transform record for event log
+    id: record.id,
+    name: record.name,
+    // ... other fields
+  }),
+  tags: { feature: "ingestion", type: "trace" },
+});
+```
+
+**DDL/Administrative commands:**
+
+```typescript
+import { commandClickhouse } from "@langfuse/shared/src/server/repositories/clickhouse";
+
+// Create table, alter schema, etc.
+await commandClickhouse({
+  query: `
+    ALTER TABLE traces
+    ADD COLUMN IF NOT EXISTS new_field String
+  `,
+  tags: { feature: "migration" },
+});
+```
+
+### ClickHouse Type Mapping
+
+| JavaScript Type | ClickHouse Param Type                                     |
+| --------------- | --------------------------------------------------------- |
+| `string`        | `String`                                                  |
+| `number`        | `UInt32`, `Int64`, `Float64`                              |
+| `Date`          | `DateTime64(3)` (use `convertDateToClickhouseDateTime()`) |
+| `boolean`       | `UInt8` (0 or 1)                                          |
+| `string[]`      | `Array(String)`                                           |
+
+**Date handling:**
+
+```typescript
+import { convertDateToClickhouseDateTime } from "@langfuse/shared/src/server/clickhouse/client";
+
+const params = {
+  startTime: convertDateToClickhouseDateTime(new Date()),
+};
+```
+
+### ClickHouse Query Best Practices
+
+**1. Always filter by `project_id` for tenant isolation:**
+
+```typescript
+// ✅ CORRECT: project_id filter is required
+const query = `
+  SELECT *
+  FROM traces
+  WHERE project_id = {projectId: String}  -- ← Required for tenant isolation
+  AND timestamp >= {startTime: DateTime64(3)}
+`;
+
+// ❌ WRONG: Missing project_id filter
+// const query = `
+//   SELECT * FROM traces WHERE timestamp >= {startTime: DateTime64(3)}
+// `;
+```
+
+**Why this is important:**
+
+- Langfuse is multi-tenant - each project's data must be isolated
+- The `project_id` filter ensures queries only access data from the intended tenant
+- All queries on project-scoped tables (traces, observations, scores, sessions, etc.) must filter by `project_id`
+
+**2. Use LIMIT BY for deduplication:**
+
+```typescript
+// Get latest version of each trace
+const query = `
+  SELECT *
+  FROM traces
+  WHERE project_id = {projectId: String}  -- ← Always include project_id
+  ORDER BY event_ts DESC
+  LIMIT 1 BY id, project_id
+`;
+```
+
+**`is_deleted` on `traces`, `observations`, and `scores` is dormant — avoid new filters.**
+
+These three tables are declared as
+`ReplacingMergeTree(event_ts, is_deleted)`, but no production
+code writes `is_deleted = 1` for them — all deletes use
+ClickHouse's lightweight `DELETE FROM` mutation (e.g.
+`deleteObservationsByTraceIds`,
+`deleteObservationsByProjectId`,
+`deleteObservationsOlderThanDays`), which marks rows via the
+engine-managed `_row_exists` column. `_row_exists` is handled
+transparently by the read path; no special query handling is
+needed.
+
+What this means for query authors:
+
+- **`WHERE is_deleted = 0` filters on these three tables are
+  dead weight in practice.** A few legacy reads still carry
+  them (e.g. `web/src/features/score-analytics/server/`); new
+  code should not add them unless soft-delete writes have
+  actually been introduced.
+
+**Separate case: `blob_storage_file_log`.** This table is also
+a `ReplacingMergeTree` but **does** use soft-delete
+intentionally — `ingestionFileDeletion.ts` writes
+`is_deleted: "1"`, `batch-project-blob-cleaner` reads with
+`countIf(is_deleted = 1)`. The guidance above does not apply
+to it.
+
+**3. Use time-based filtering for performance:**
+
+```typescript
+// Combine project_id filter with timestamp for optimal performance
+const query = `
+  SELECT *
+  FROM observations
+  WHERE project_id = {projectId: String}  -- ← Required for tenant isolation
+  AND start_time >= {startTime: DateTime64(3)}  -- ← Improves performance
+  AND start_time < {endTime: DateTime64(3)}
+`;
+```
+
+**4. Use CTEs for complex queries (still require `project_id`):**
+
+```typescript
+const query = `
+  WITH observations_agg AS (
+    SELECT
+      trace_id,
+      count() as observation_count,
+      sum(total_cost) as total_cost
+    FROM observations
+    WHERE project_id = {projectId: String}  -- ← Filter in CTE
+    GROUP BY trace_id
+  )
+  SELECT
+    t.id,
+    t.name,
+    o.observation_count,
+    o.total_cost
+  FROM traces t
+  LEFT JOIN observations_agg o ON t.id = o.trace_id
+  WHERE t.project_id = {projectId: String}  -- ← Filter in main query
+`;
+```
+
+**Note**: When using CTEs or subqueries, ensure `project_id` filter is applied at each level.
+
+**Error handling with retries:**
+
+ClickHouse queries automatically retry on network errors (socket hang up). Custom error handling for resource limits:
+
+```typescript
+import {
+  queryClickhouse,
+  ClickHouseResourceError,
+} from "@langfuse/shared/src/server/repositories/clickhouse";
+
+try {
+  const rows = await queryClickhouse({ query, params });
+} catch (error) {
+  if (error instanceof ClickHouseResourceError) {
+    // Memory limit, timeout, or overcommit error
+    throw new Error(ClickHouseResourceError.ERROR_ADVICE_MESSAGE);
+  }
+  throw error;
+}
+```
+
+---
+
+## Repository Pattern
+
+Langfuse uses repositories in `packages/shared/src/server/repositories/` for complex data access patterns.
+
+### When to Use Repositories
+
+✅ **Use repositories when:**
+
+- Complex ClickHouse queries with CTEs, aggregations, or joins
+- Query used in multiple places (DRY principle)
+- Need data transformation/converters (DB → domain models)
+- Building reusable query logic with filters
+
+❌ **Use direct Prisma/ClickHouse for:**
+
+- Simple CRUD operations
+- One-off queries
+- Prototyping (refactor to repository later)
+
+### Repository Examples
+
+**Trace repository (ClickHouse):**
+
+```typescript
+// packages/shared/src/server/repositories/traces.ts
+export const getTracesByIds = async (
+  projectId: string,
+  traceIds: string[],
+): Promise<TraceRecordReadType[]> => {
+  const rows = await queryClickhouse<TraceRecordReadType>({
+    query: `
+      SELECT *
+      FROM traces
+      WHERE project_id = {projectId: String}
+      AND id IN ({traceIds: Array(String)})
+      ORDER BY event_ts DESC
+      LIMIT 1 BY id, project_id
+    `,
+    params: { projectId, traceIds },
+    tags: { feature: "tracing", type: "trace" },
+  });
+
+  return rows.map(convertClickhouseToDomain);
+};
+```
+
+**Score repository (PostgreSQL + ClickHouse):**
+
+```typescript
+// Repositories can query both databases
+export const getScoresByTraceId = async (
+  projectId: string,
+  traceId: string,
+) => {
+  // Use ClickHouse for analytics
+  const clickhouseScores = await queryClickhouse<ScoreRecordReadType>({
+    query: `
+      SELECT *
+      FROM scores
+      WHERE project_id = {projectId: String}
+      AND trace_id = {traceId: String}
+    `,
+    params: { projectId, traceId },
+  });
+
+  // Use Prisma for config data
+  const scoreConfigs = await prisma.scoreConfig.findMany({
+    where: { projectId },
+  });
+
+  return enrichScoresWithConfigs(clickhouseScores, scoreConfigs);
+};
+```
+
+---
+
+## When to Use Which Database
+
+| Use Case                               | Database   | Reasoning                                  |
+| -------------------------------------- | ---------- | ------------------------------------------ |
+| User accounts, projects, API keys      | PostgreSQL | Transactional data with strong consistency |
+| Prompt management, dataset definitions | PostgreSQL | Configuration data with relations          |
+| Project settings, RBAC permissions     | PostgreSQL | Small, frequently updated data             |
+| Traces, observations, events           | ClickHouse | High-volume time-series data               |
+| Score aggregations, analytics queries  | ClickHouse | Fast aggregations over millions of rows    |
+| Usage metrics, cost calculations       | ClickHouse | Analytical queries with GROUP BY           |
+| Exports, large dataset queries         | ClickHouse | Streaming support for large result sets    |
+
+**Decision flow:**
+
+1. Is it high-volume time-series data? → **ClickHouse**
+2. Does it need aggregation over millions of rows? → **ClickHouse**
+3. Is it transactional data with relationships? → **PostgreSQL**
+4. Is it configuration or user data? → **PostgreSQL**
+5. Is it frequently updated? → **PostgreSQL**
+6. Is it append-only analytics data? → **ClickHouse**
+
+### Project-Scoped vs Global Tables
+
+**Project-scoped tables (MUST filter by `project_id`):**
+
+- `traces` - All trace queries require `project_id`
+- `observations` - All observation queries require `project_id`
+- `scores` - All score queries require `project_id`
+- `events` - All event queries require `project_id`
+- `dataset_run_items_rmt` - All dataset run queries require `project_id`
+
+**Global tables (no `project_id` filter needed):**
+
+- `users` - User management (use `id` for filtering)
+- `organizations` - Organization data (use `id` for filtering)
+- System configuration tables
+
+**Example of correct filtering:**
+
+```typescript
+// ✅ CORRECT: Project-scoped query
+const traces = await queryClickhouse({
+  query: `
+    SELECT * FROM traces
+    WHERE project_id = {projectId: String}
+    AND timestamp >= {startTime: DateTime64(3)}
+  `,
+  params: { projectId, startTime },
+});
+
+// ✅ CORRECT: Global table query (no project_id needed)
+const user = await prisma.user.findUnique({
+  where: { id: userId },
+});
+
+// ❌ WRONG: Project-scoped query without project_id filter
+// const traces = await queryClickhouse({
+//   query: `SELECT * FROM traces WHERE timestamp >= {startTime: DateTime64(3)}`,
+// });
+```
+
+---
+
+## Error Handling
+
+### PostgreSQL (Prisma) Errors
+
+```typescript
+import { Prisma } from "@prisma/client";
+import { prisma } from "@langfuse/shared/src/db";
+
+try {
+  await prisma.user.create({ data: userData });
+} catch (error) {
+  if (error instanceof Prisma.PrismaClientKnownRequestError) {
+    // Unique constraint violation
+    if (error.code === "P2002") {
+      const target = error.meta?.target as string[];
+      throw new ConflictError(`${target?.join(", ")} already exists`);
+    }
+
+    // Foreign key constraint
+    if (error.code === "P2003") {
+      throw new ValidationError("Invalid reference");
+    }
+
+    // Record not found
+    if (error.code === "P2025") {
+      throw new NotFoundError("Record not found");
+    }
+
+    // Record required to connect not found
+    if (error.code === "P2018") {
+      throw new ValidationError("Related record not found");
+    }
+  }
+
+  // Unknown error
+  logger.error("Prisma error", { error });
+  throw error;
+}
+```
+
+**Common Prisma error codes:**
+
+| Code    | Meaning                     | Typical Cause                          |
+| ------- | --------------------------- | -------------------------------------- |
+| `P2002` | Unique constraint violation | Duplicate email, API key, etc.         |
+| `P2003` | Foreign key constraint      | Referenced record doesn't exist        |
+| `P2025` | Record not found            | Update/delete of non-existent record   |
+| `P2018` | Required relation not found | Connect to non-existent related record |
+
+### ClickHouse Errors
+
+```typescript
+import {
+  queryClickhouse,
+  ClickHouseResourceError,
+} from "@langfuse/shared/src/server/repositories/clickhouse";
+
+try {
+  const rows = await queryClickhouse({ query, params });
+} catch (error) {
+  // ClickHouse resource errors (memory limit, timeout, overcommit)
+  if (error instanceof ClickHouseResourceError) {
+    logger.warn("ClickHouse resource error", {
+      errorType: error.errorType, // "MEMORY_LIMIT" | "OVERCOMMIT" | "TIMEOUT"
+      message: error.message,
+    });
+
+    // User-friendly error message
+    throw new BadRequestError(ClickHouseResourceError.ERROR_ADVICE_MESSAGE);
+  }
+
+  // Network/connection errors are automatically retried
+  logger.error("ClickHouse error", { error });
+  throw error;
+}
+```
+
+**ClickHouse error types:**
+
+| Error Type     | Discriminator           | Meaning                     | Solution                                          |
+| -------------- | ----------------------- | --------------------------- | ------------------------------------------------- |
+| `MEMORY_LIMIT` | "memory limit exceeded" | Query used too much memory  | Use more specific filters or shorter time range   |
+| `OVERCOMMIT`   | "OvercommitTracker"     | Memory overcommit limit hit | Reduce query complexity or result set size        |
+| `TIMEOUT`      | "Timeout", "timed out"  | Query took too long         | Add filters, reduce time range, or optimize query |
+
+**ClickHouse retries:**
+
+ClickHouse queries automatically retry network errors (socket hang up) with exponential backoff. Configure retry behavior:
+
+```typescript
+// In packages/shared/src/env.ts
+LANGFUSE_CLICKHOUSE_QUERY_MAX_ATTEMPTS: z.coerce.number().positive().default(3);
+```
+
+---
+
+**Related Files:**
+
+- [../SKILL.md](../SKILL.md) - Main backend development guidelines
+- [architecture-overview.md](architecture-overview.md) - System architecture
+- [configuration.md](configuration.md) - Environment variable configuration

.agents/skills/backend-dev-guidelines/references/middleware-guide.md

@@ -0,0 +1,793 @@
+# Middleware Guide - tRPC & Public API Patterns
+
+Complete guide to middleware patterns in Langfuse's Next.js + tRPC architecture.
+
+## Table of Contents
+
+- [tRPC Middleware](#trpc-middleware)
+- [Public API Middleware](#public-api-middleware)
+- [Authentication Patterns](#authentication-patterns)
+- [Error Handling Middleware](#error-handling-middleware)
+- [OpenTelemetry Instrumentation](#opentelemetry-instrumentation)
+- [Composable Procedures](#composable-procedures)
+
+---
+
+## tRPC Middleware
+
+**File:** `web/src/server/api/trpc.ts`
+
+tRPC middleware in Langfuse is composable and type-safe. Each middleware enriches the context and provides guarantees to subsequent middleware.
+
+### Core tRPC Middlewares
+
+**1. Error Handling Middleware (`withErrorHandling`)**
+
+Intercepts all errors and transforms them into user-friendly tRPC errors:
+
+```typescript
+const withErrorHandling = t.middleware(async ({ ctx, next }) => {
+  const res = await next({ ctx });
+
+  if (!res.ok) {
+    if (res.error.cause instanceof ClickHouseResourceError) {
+      // Surface ClickHouse resource errors with advice message
+      res.error = new TRPCError({
+        code: "SERVICE_UNAVAILABLE",
+        message: ClickHouseResourceError.ERROR_ADVICE_MESSAGE,
+      });
+    } else {
+      // Transform 5xx errors to not expose internals
+      const { code, httpStatus } = resolveError(res.error);
+      const isSafeToExpose = httpStatus >= 400 && httpStatus < 500;
+
+      res.error = new TRPCError({
+        code,
+        cause: null, // do not expose stack traces
+        message: isSafeToExpose
+          ? res.error.message
+          : "Internal error. We have been notified and are working on it.",
+      });
+    }
+  }
+
+  return res;
+});
+```
+
+**2. OpenTelemetry Instrumentation (`withOtelInstrumentation`)**
+
+Propagates OpenTelemetry context with Langfuse-specific baggage:
+
+```typescript
+const withOtelInstrumentation = t.middleware(async (opts) => {
+  const actualInput = await opts.getRawInput();
+
+  const baggageCtx = contextWithLangfuseProps({
+    headers: opts.ctx.headers,
+    userId: opts.ctx.session?.user?.id,
+    projectId: (actualInput as Record<string, string>)?.projectId,
+  });
+
+  return opentelemetry.context.with(baggageCtx, () => opts.next());
+});
+```
+
+**3. Authentication Middleware (`enforceUserIsAuthed`)**
+
+Ensures user is logged in via NextAuth session:
+
+```typescript
+const enforceUserIsAuthed = t.middleware(({ ctx, next }) => {
+  if (!ctx.session || !ctx.session.user) {
+    throw new TRPCError({ code: "UNAUTHORIZED" });
+  }
+
+  return next({
+    ctx: {
+      // infers the `session` as non-nullable
+      session: { ...ctx.session, user: ctx.session.user },
+    },
+  });
+});
+```
+
+**4. Project Membership Middleware (`enforceUserIsAuthedAndProjectMember`)**
+
+Validates that the user is a member of the project specified in input:
+
+```typescript
+const enforceUserIsAuthedAndProjectMember = t.middleware(async (opts) => {
+  const { ctx, next } = opts;
+
+  if (!ctx.session || !ctx.session.user) {
+    throw new TRPCError({ code: "UNAUTHORIZED" });
+  }
+
+  const actualInput = await opts.getRawInput();
+  const parsedInput = inputProjectSchema.safeParse(actualInput);
+
+  if (!parsedInput.success) {
+    throw new TRPCError({
+      code: "BAD_REQUEST",
+      message: "Invalid input, projectId is required",
+    });
+  }
+
+  const projectId = parsedInput.data.projectId;
+  const sessionProject = ctx.session.user.organizations
+    .flatMap((org) =>
+      org.projects.map((project) => ({ ...project, organization: org })),
+    )
+    .find((project) => project.id === projectId);
+
+  if (!sessionProject) {
+    throw new TRPCError({
+      code: "UNAUTHORIZED",
+      message: "User is not a member of this project",
+    });
+  }
+
+  return next({
+    ctx: {
+      session: {
+        ...ctx.session,
+        user: ctx.session.user,
+        orgId: sessionProject.organization.id,
+        orgRole: sessionProject.organization.role,
+        projectId: projectId,
+        projectRole: sessionProject.role,
+      },
+    },
+  });
+});
+```
+
+**5. Organization Membership Middleware (`enforceIsAuthedAndOrgMember`)**
+
+Validates organization membership:
+
+```typescript
+const enforceIsAuthedAndOrgMember = t.middleware(async (opts) => {
+  const { ctx, next } = opts;
+
+  if (!ctx.session || !ctx.session.user) {
+    throw new TRPCError({ code: "UNAUTHORIZED" });
+  }
+
+  const actualInput = await opts.getRawInput();
+  const result = inputOrganizationSchema.safeParse(actualInput);
+
+  if (!result.success) {
+    throw new TRPCError({
+      code: "BAD_REQUEST",
+      message: "Invalid input, orgId is required",
+    });
+  }
+
+  const orgId = result.data.orgId;
+  const sessionOrg = ctx.session.user.organizations.find(
+    (org) => org.id === orgId,
+  );
+
+  if (!sessionOrg && ctx.session.user.admin !== true) {
+    throw new TRPCError({
+      code: "UNAUTHORIZED",
+      message: "User is not a member of this organization",
+    });
+  }
+
+  return next({
+    ctx: {
+      session: {
+        ...ctx.session,
+        user: ctx.session.user,
+        orgId: orgId,
+        orgRole:
+          ctx.session.user.admin === true ? Role.OWNER : sessionOrg!.role,
+      },
+    },
+  });
+});
+```
+
+**6. Trace Access Middleware (`enforceTraceAccess`)**
+
+Special middleware for trace-level routes that supports public traces:
+
+```typescript
+const enforceTraceAccess = t.middleware(async (opts) => {
+  const { ctx, next } = opts;
+  const actualInput = await opts.getRawInput();
+  const result = inputTraceSchema.safeParse(actualInput);
+
+  if (!result.success) {
+    throw new TRPCError({ code: "BAD_REQUEST", message: "Invalid input" });
+  }
+
+  const trace = await getTraceById({
+    traceId: result.data.traceId,
+    projectId: result.data.projectId,
+    timestamp: result.data.timestamp ?? undefined,
+  });
+
+  if (!trace) {
+    throw new TRPCError({ code: "NOT_FOUND", message: "Trace not found" });
+  }
+
+  const sessionProject = ctx.session?.user?.organizations
+    .flatMap((org) => org.projects)
+    .find(({ id }) => id === result.data.projectId);
+
+  // Allow access if:
+  // 1. User is a project member
+  // 2. Trace is public
+  // 3. User is admin
+  if (!trace.public && !sessionProject && ctx.session?.user?.admin !== true) {
+    throw new TRPCError({
+      code: "UNAUTHORIZED",
+      message:
+        "User is not a member of this project and this trace is not public",
+    });
+  }
+
+  return next({
+    ctx: {
+      session: { ...ctx.session, projectRole: sessionProject?.role },
+      trace: trace, // pass the trace to avoid refetching
+    },
+  });
+});
+```
+
+### tRPC Procedure Types
+
+Langfuse exports composed procedures with middleware chains:
+
+```typescript
+// 1. Public procedure (no auth required)
+export const publicProcedure = withOtelTracingProcedure.use(withErrorHandling);
+
+// 2. Authenticated procedure (NextAuth session required)
+export const authenticatedProcedure = withOtelTracingProcedure
+  .use(withErrorHandling)
+  .use(enforceUserIsAuthed);
+
+// 3. Project-scoped procedure (project membership required)
+export const protectedProjectProcedure = withOtelTracingProcedure
+  .use(withErrorHandling)
+  .use(enforceUserIsAuthedAndProjectMember);
+
+// 4. Organization-scoped procedure
+export const protectedOrganizationProcedure = withOtelTracingProcedure
+  .use(withErrorHandling)
+  .use(enforceIsAuthedAndOrgMember);
+
+// 5. Trace access procedure (public traces supported)
+export const protectedGetTraceProcedure = withOtelTracingProcedure
+  .use(withErrorHandling)
+  .use(enforceTraceAccess);
+
+// 6. Session access procedure
+export const protectedGetSessionProcedure = withOtelTracingProcedure
+  .use(withErrorHandling)
+  .use(enforceSessionAccess);
+
+// 7. Admin API key procedure (for admin operations)
+export const adminProcedure = withOtelTracingProcedure
+  .use(withErrorHandling)
+  .use(enforceAdminAuth);
+```
+
+---
+
+## Public API Middleware
+
+**Files:** `web/src/features/public-api/server/`
+
+### withMiddlewares Pattern
+
+Wraps all public API routes with CORS, error handling, and OpenTelemetry:
+
+```typescript
+export function withMiddlewares(handlers: Handlers) {
+  return async (req: NextApiRequest, res: NextApiResponse) => {
+    const ctx = contextWithLangfuseProps({ headers: req.headers });
+
+    return opentelemetry.context.with(ctx, async () => {
+      try {
+        // 1. CORS middleware
+        await runMiddleware(req, res, cors);
+
+        // 2. HTTP method routing
+        const method = req.method as HttpMethod;
+        if (!handlers[method]) throw new MethodNotAllowedError();
+
+        // 3. Execute handler
+        return await handlers[method](req, res);
+      } catch (error) {
+        // 4. Error handling
+        if (error instanceof BaseError) {
+          if (error.httpCode >= 500) traceException(error);
+
+          return res.status(error.httpCode).json({
+            message: error.message,
+            error: error.name,
+          });
+        }
+
+        if (error instanceof ClickHouseResourceError) {
+          return res.status(524).json({
+            message: ClickHouseResourceError.ERROR_ADVICE_MESSAGE,
+            error: "Request is taking too long to process.",
+          });
+        }
+
+        if (isZodError(error)) {
+          return res.status(400).json({
+            message: "Invalid request data",
+            error: error.issues,
+          });
+        }
+
+        traceException(error);
+        return res.status(500).json({
+          message: "Internal Server Error",
+          error: error instanceof Error ? error.message : "Unknown error",
+        });
+      }
+    });
+  };
+}
+```
+
+**Usage:**
+
+```typescript
+// web/src/pages/api/public/datasets/[datasetName]/items.ts
+export default withMiddlewares({
+  GET: getDatasetItemsHandler,
+  POST: createDatasetItemHandler,
+});
+```
+
+### createAuthedProjectAPIRoute Pattern
+
+Factory function for authenticated public API routes with:
+
+- Authentication (Basic auth or Admin API key)
+- Rate limiting
+- Input/output validation (Zod)
+- OpenTelemetry context
+
+```typescript
+export const createAuthedProjectAPIRoute = <TQuery, TBody, TResponse>(
+  routeConfig: RouteConfig<TQuery, TBody, TResponse>,
+) => {
+  return async (req: NextApiRequest, res: NextApiResponse) => {
+    // 1. Authentication (verifyAuth)
+    const auth = await verifyAuth(
+      req,
+      routeConfig.isAdminApiKeyAuthAllowed || false,
+    );
+
+    // 2. Rate limiting
+    const rateLimitResponse =
+      await RateLimitService.getInstance().rateLimitRequest(
+        auth.scope,
+        routeConfig.rateLimitResource || "public-api",
+      );
+
+    if (rateLimitResponse?.isRateLimited()) {
+      return rateLimitResponse.sendRestResponseIfLimited(res);
+    }
+
+    // 3. Input validation
+    const query = routeConfig.querySchema
+      ? routeConfig.querySchema.parse(req.query)
+      : {};
+    const body = routeConfig.bodySchema
+      ? routeConfig.bodySchema.parse(req.body)
+      : {};
+
+    // 4. Execute with OpenTelemetry context
+    const ctx = contextWithLangfuseProps({
+      headers: req.headers,
+      projectId: auth.scope.projectId,
+    });
+
+    return opentelemetry.context.with(ctx, async () => {
+      const response = await routeConfig.fn({ query, body, req, res, auth });
+
+      // 5. Response validation (dev only)
+      if (env.NODE_ENV === "development" && routeConfig.responseSchema) {
+        const parsingResult = routeConfig.responseSchema.safeParse(response);
+        if (!parsingResult.success) {
+          logger.error("Response validation failed:", parsingResult.error);
+        }
+      }
+
+      res.status(routeConfig.successStatusCode || 200).json(response);
+    });
+  };
+};
+```
+
+**Usage:**
+
+```typescript
+// web/src/pages/api/public/traces/[traceId].ts
+export default createAuthedProjectAPIRoute({
+  name: "Get Trace",
+  querySchema: GetTraceV1Query,
+  responseSchema: GetTraceV1Response,
+  fn: async ({ query, auth }) => {
+    const trace = await getTraceById({
+      traceId: query.traceId,
+      projectId: auth.scope.projectId,
+    });
+
+    return transformTraceToApiResponse(trace);
+  },
+});
+```
+
+---
+
+## Authentication Patterns
+
+### tRPC Authentication (NextAuth)
+
+tRPC uses NextAuth sessions stored in JWT cookies:
+
+```typescript
+// Context creation with session
+export const createTRPCContext = async (opts: CreateNextContextOptions) => {
+  const { req, res } = opts;
+  const session = await getServerAuthSession({ req, res });
+
+  addUserToSpan({
+    userId: session?.user?.id,
+    email: session?.user?.email ?? undefined,
+  });
+
+  return {
+    session,
+    headers: req.headers,
+    prisma,
+    DB,
+  };
+};
+```
+
+**Session types:**
+
+```typescript
+// Base authenticated context
+export type AuthedContext = {
+  session: { user: NonNullable<Session["user"]> };
+};
+
+// Project-scoped context
+export type ProjectAuthedContext = {
+  session: AuthedContext["session"] & {
+    orgId: string;
+    orgRole: Role;
+    projectId: string;
+    projectRole: Role;
+  };
+};
+```
+
+### Public API Authentication
+
+Public APIs use **Basic Auth** with API keys:
+
+```typescript
+async function verifyBasicAuth(authHeader: string | undefined) {
+  const regularAuth = await new ApiAuthService(
+    prisma,
+    redis,
+  ).verifyAuthHeaderAndReturnScope(authHeader);
+
+  if (!regularAuth.validKey) {
+    throw { status: 401, message: regularAuth.error };
+  }
+
+  if (regularAuth.scope.accessLevel !== "project") {
+    throw {
+      status: 401,
+      message: "Access denied - need basic auth with secret key",
+    };
+  }
+
+  return regularAuth;
+}
+```
+
+**Admin API Key Authentication** (self-hosted only):
+
+```typescript
+async function verifyAdminApiKeyAuth(req: NextApiRequest) {
+  // Requires:
+  // 1. Authorization: Bearer <ADMIN_API_KEY>
+  // 2. x-langfuse-admin-api-key: <ADMIN_API_KEY>
+  // 3. x-langfuse-project-id: <project-id>
+
+  if (env.NEXT_PUBLIC_LANGFUSE_CLOUD_REGION) {
+    throw {
+      status: 403,
+      message: "Admin API key auth not available on Langfuse Cloud",
+    };
+  }
+
+  const adminApiKey = env.ADMIN_API_KEY;
+  const bearerToken = req.headers.authorization?.replace("Bearer ", "");
+  const adminApiKeyHeader = req.headers["x-langfuse-admin-api-key"];
+
+  // Timing-safe comparison
+  const isValid =
+    crypto.timingSafeEqual(
+      Buffer.from(bearerToken),
+      Buffer.from(adminApiKey),
+    ) &&
+    crypto.timingSafeEqual(
+      Buffer.from(adminApiKeyHeader),
+      Buffer.from(adminApiKey),
+    );
+
+  if (!isValid) throw { status: 401, message: "Invalid admin API key" };
+
+  const projectId = req.headers["x-langfuse-project-id"];
+  const project = await prisma.project.findUnique({ where: { id: projectId } });
+
+  if (!project) throw { status: 404, message: "Project not found" };
+
+  return { validKey: true, scope: { projectId, accessLevel: "project" } };
+}
+```
+
+---
+
+## Error Handling Middleware
+
+### tRPC Error Transformation
+
+All tRPC errors go through `withErrorHandling` middleware:
+
+**Error types handled:**
+
+1. **ClickHouseResourceError** → `SERVICE_UNAVAILABLE` (524)
+2. **BaseError** → Preserves httpCode and message
+3. **5xx errors** → Sanitized as "Internal error" (hides stack traces)
+4. **4xx errors** → Original error message preserved
+
+**Example:**
+
+```typescript
+if (!res.ok) {
+  if (res.error.cause instanceof ClickHouseResourceError) {
+    res.error = new TRPCError({
+      code: "SERVICE_UNAVAILABLE",
+      message: ClickHouseResourceError.ERROR_ADVICE_MESSAGE,
+    });
+  } else {
+    const { code, httpStatus } = resolveError(res.error);
+    const isSafeToExpose = httpStatus >= 400 && httpStatus < 500;
+
+    res.error = new TRPCError({
+      code,
+      cause: null,
+      message: isSafeToExpose ? res.error.message : "Internal error.",
+    });
+  }
+}
+```
+
+### Public API Error Handling
+
+Public API uses `withMiddlewares` for error handling:
+
+```typescript
+catch (error) {
+  // 1. BaseError (custom application errors)
+  if (error instanceof BaseError) {
+    if (error.httpCode >= 500) traceException(error);
+    return res.status(error.httpCode).json({
+      message: error.message,
+      error: error.name,
+    });
+  }
+
+  // 2. ClickHouseResourceError (query timeouts, memory limits)
+  if (error instanceof ClickHouseResourceError) {
+    return res.status(524).json({
+      message: ClickHouseResourceError.ERROR_ADVICE_MESSAGE,
+      error: "Request is taking too long to process.",
+    });
+  }
+
+  // 3. Zod validation errors
+  if (isZodError(error)) {
+    return res.status(400).json({
+      message: "Invalid request data",
+      error: error.issues,
+    });
+  }
+
+  // 4. Prisma errors
+  if (isPrismaException(error)) {
+    traceException(error);
+    return res.status(500).json({
+      message: "Internal Server Error",
+      error: "An unknown error occurred",
+    });
+  }
+
+  // 5. Unknown errors
+  traceException(error);
+  return res.status(500).json({
+    message: "Internal Server Error",
+    error: error instanceof Error ? error.message : "Unknown error",
+  });
+}
+```
+
+---
+
+## OpenTelemetry Instrumentation
+
+All requests (tRPC and public API) propagate OpenTelemetry context with Langfuse-specific baggage.
+
+### Context Propagation Pattern
+
+```typescript
+import { contextWithLangfuseProps } from "@langfuse/shared/src/server";
+import * as opentelemetry from "@opentelemetry/api";
+
+// Create context with Langfuse baggage
+const ctx = contextWithLangfuseProps({
+  headers: req.headers,
+  userId: session?.user?.id,
+  projectId: input?.projectId,
+});
+
+// Execute with context
+return opentelemetry.context.with(ctx, async () => {
+  // All instrumented code inside here will have access to baggage
+  return await handler();
+});
+```
+
+**Baggage includes:**
+
+- `userId` - User ID from session
+- `projectId` - Project ID from input
+- `headers` - Request headers for trace propagation
+
+### tRPC Instrumentation
+
+```typescript
+const withOtelInstrumentation = t.middleware(async (opts) => {
+  const actualInput = await opts.getRawInput();
+
+  const baggageCtx = contextWithLangfuseProps({
+    headers: opts.ctx.headers,
+    userId: opts.ctx.session?.user?.id,
+    projectId: (actualInput as Record<string, string>)?.projectId,
+  });
+
+  return opentelemetry.context.with(baggageCtx, () => opts.next());
+});
+
+// Used with Baselime tracing
+const withOtelTracingProcedure = t.procedure
+  .use(withOtelInstrumentation)
+  .use(tracing({ collectInput: true, collectResult: true }));
+```
+
+---
+
+## Composable Procedures
+
+tRPC procedures are composed by chaining middleware:
+
+### Composition Pattern
+
+```typescript
+// Base procedure with tracing + error handling
+const baseProcedure = withOtelTracingProcedure.use(withErrorHandling);
+
+// Add authentication
+const authedProcedure = baseProcedure.use(enforceUserIsAuthed);
+
+// Add project scoping
+const projectProcedure = authedProcedure.use(
+  enforceUserIsAuthedAndProjectMember,
+);
+```
+
+### Using Procedures in Routers
+
+```typescript
+import { protectedProjectProcedure } from "@/src/server/api/trpc";
+
+export const tracesRouter = createTRPCRouter({
+  // Input automatically validated against Zod schema
+  all: protectedProjectProcedure
+    .input(
+      z.object({
+        projectId: z.string(),
+        page: z.number().optional(),
+        limit: z.number().optional(),
+      }),
+    )
+    .query(async ({ input, ctx }) => {
+      // ctx.session.projectId is guaranteed to exist
+      // ctx.session.projectRole contains user's role
+
+      const traces = await getTraces({
+        projectId: input.projectId,
+        page: input.page ?? 0,
+        limit: input.limit ?? 50,
+      });
+
+      return traces;
+    }),
+
+  byId: protectedGetTraceProcedure
+    .input(
+      z.object({
+        traceId: z.string(),
+        projectId: z.string(),
+        timestamp: z.date().nullish(),
+      }),
+    )
+    .query(async ({ input, ctx }) => {
+      // ctx.trace is guaranteed to exist (fetched by middleware)
+      // No need to refetch
+      return ctx.trace;
+    }),
+});
+```
+
+### Middleware Execution Order
+
+Middleware executes in the order it's chained:
+
+```typescript
+protectedProjectProcedure
+  .use(withErrorHandling) // 1. Wraps entire execution
+  .use(enforceUserIsAuthed) // 2. Validates session exists
+  .use(enforceUserIsAuthedAndProjectMember) // 3. Validates project membership
+  .input(schema) // 4. Validates input
+  .query(async ({ input, ctx }) => {
+    // 5. Executes query
+    // ...
+  });
+```
+
+**Context enrichment:**
+
+Each middleware can enrich the context:
+
+```typescript
+// After enforceUserIsAuthed:
+ctx.session.user; // NonNullable<User>
+
+// After enforceUserIsAuthedAndProjectMember:
+ctx.session.projectId; // string
+ctx.session.projectRole; // Role (OWNER | ADMIN | MEMBER | VIEWER)
+ctx.session.orgId; // string
+ctx.session.orgRole; // Role
+
+// After enforceTraceAccess:
+ctx.trace; // TraceRecord (pre-fetched)
+```
+
+---
+
+**Related Files:**
+
+- [../SKILL.md](../SKILL.md) - Main backend development guidelines
+- [architecture-overview.md](architecture-overview.md) - System architecture

.agents/skills/backend-dev-guidelines/references/routing-and-controllers.md

@@ -0,0 +1,908 @@
+# Routing Patterns - Next.js & tRPC
+
+Complete guide to routing and separation of concerns in Langfuse's Next.js + tRPC architecture.
+
+## Table of Contents
+
+- [Architecture Overview](#architecture-overview)
+- [tRPC Routers](#trpc-routers)
+- [Public REST API Routes](#public-rest-api-routes)
+- [Fern API Definitions](#fern-api-definitions)
+- [Service Layer](#service-layer)
+- [Repository Layer](#repository-layer)
+- [Separation of Concerns](#separation-of-concerns)
+- [Anti-Patterns](#anti-patterns)
+
+---
+
+## Architecture Overview
+
+Langfuse uses a **layered architecture** with clear separation of concerns:
+
+```
+┌─────────────────────────────────────────────────────────────┐
+│                        ENTRY POINTS                          │
+│  ┌──────────────────────┐    ┌─────────────────────────┐   │
+│  │   tRPC Procedures    │    │  Public REST API Routes │   │
+│  │  (Internal UI API)   │    │   (SDK/External API)    │   │
+│  └──────────────────────┘    └─────────────────────────┘   │
+└─────────────────────────────────────────────────────────────┘
+                              ▼
+┌─────────────────────────────────────────────────────────────┐
+│                       SERVICE LAYER                          │
+│         Business logic, orchestration, validation            │
+│   web/src/features/*/server/ or packages/shared/services/   │
+└─────────────────────────────────────────────────────────────┘
+                              ▼
+┌─────────────────────────────────────────────────────────────┐
+│                     REPOSITORY LAYER                         │
+│         Complex queries, data transformation                 │
+│         packages/shared/src/server/repositories/             │
+└─────────────────────────────────────────────────────────────┘
+                              ▼
+┌─────────────────────────────────────────────────────────────┐
+│                      DATABASE LAYER                          │
+│       PostgreSQL (Prisma) + ClickHouse (Direct Client)      │
+└─────────────────────────────────────────────────────────────┘
+```
+
+### Key Principles
+
+**Entry Points (Routes/Procedures):**
+
+- ✅ Define routing and procedure signatures
+- ✅ Handle authentication/authorization (via middleware)
+- ✅ Validate input (Zod schemas)
+- ✅ Delegate to services
+- ✅ Return responses
+
+**Entry Points should NEVER:**
+
+- ❌ Contain business logic
+- ❌ Access database directly
+- ❌ Perform complex data transformations
+- ❌ Make direct repository calls (use services)
+
+**Services:**
+
+- ✅ Contain business logic
+- ✅ Orchestrate multiple operations
+- ✅ Call repositories or Prisma/ClickHouse
+- ✅ Handle complex workflows
+- ❌ Should NOT know about HTTP, tRPC, or request/response objects
+
+**Repositories:**
+
+- ✅ Complex database queries
+- ✅ Data transformation (DB → domain models)
+- ✅ ClickHouse query builders
+- ✅ Reusable query logic
+- ❌ Should NOT contain business logic
+
+---
+
+## tRPC Routers
+
+**Location:** `web/src/server/api/routers/`
+
+tRPC routers define type-safe procedures for the internal UI. Each router groups related operations.
+
+### Router Structure
+
+**File:** `web/src/server/api/routers/scores.ts`
+
+```typescript
+import { z } from "zod/v4";
+import {
+  createTRPCRouter,
+  protectedProjectProcedure,
+} from "@/src/server/api/trpc";
+import { paginationZod, singleFilter, orderBy } from "@langfuse/shared";
+import {
+  getScoresUiTable,
+  getScoresUiCount,
+  upsertScore,
+} from "@langfuse/shared/src/server";
+
+const ScoreAllOptions = z.object({
+  projectId: z.string(),
+  filter: z.array(singleFilter),
+  orderBy: orderBy,
+  ...paginationZod,
+});
+
+export const scoresRouter = createTRPCRouter({
+  /**
+   * Get all scores for a project
+   */
+  all: protectedProjectProcedure
+    .input(ScoreAllOptions)
+    .query(async ({ input, ctx }) => {
+      // Delegate to repository for data fetching
+      const clickhouseScoreData = await getScoresUiTable({
+        projectId: input.projectId,
+        filter: input.filter ?? [],
+        orderBy: input.orderBy,
+        limit: input.limit,
+        offset: input.page * input.limit,
+      });
+
+      // Delegate to Prisma for related data
+      const [jobExecutions, users] = await Promise.all([
+        ctx.prisma.jobExecution.findMany({
+          where: {
+            jobOutputScoreId: {
+              in: clickhouseScoreData.map((score) => score.id),
+            },
+          },
+        }),
+        ctx.prisma.user.findMany({
+          where: {
+            id: {
+              in: clickhouseScoreData
+                .map((s) => s.authorUserId)
+                .filter((id): id is string => id !== null),
+            },
+          },
+        }),
+      ]);
+
+      // Transform and combine data
+      return clickhouseScoreData.map((score) => ({
+        ...score,
+        jobConfigurationId:
+          jobExecutions.find((j) => j.jobOutputScoreId === score.id)
+            ?.jobConfigurationId ?? null,
+        authorUserImage:
+          users.find((u) => u.id === score.authorUserId)?.image ?? null,
+        authorUserName:
+          users.find((u) => u.id === score.authorUserId)?.name ?? null,
+      }));
+    }),
+
+  /**
+   * Create or update score
+   */
+  createAnnotationScore: protectedProjectProcedure
+    .input(CreateAnnotationScoreData)
+    .mutation(async ({ input, ctx }) => {
+      // Validation
+      validateConfigAgainstBody(input);
+
+      // Delegate to repository
+      await upsertScore({
+        id: input.id ?? randomUUID(),
+        traceId: input.traceId,
+        projectId: input.projectId,
+        name: input.name,
+        value: input.value,
+        source: ScoreSource.ANNOTATION,
+        authorUserId: ctx.session.user.id,
+        comment: input.comment,
+      });
+
+      // Audit log
+      await auditLog({
+        session: ctx.session,
+        resourceType: "score",
+        resourceId: input.id,
+        action: "create",
+      });
+
+      return { success: true };
+    }),
+});
+```
+
+**Key Points:**
+
+- Use appropriate procedure type (`protectedProjectProcedure`, `authenticatedProcedure`, etc.)
+- Define input schema with Zod (`.input()`)
+- Use `.query()` for reads, `.mutation()` for writes
+- Delegate to services/repositories for data access
+- Keep procedures thin - no business logic
+- Type-safe throughout (TypeScript infers types from Zod schemas)
+
+### Registering Routers
+
+**File:** `web/src/server/api/root.ts`
+
+```typescript
+import { createTRPCRouter } from "@/src/server/api/trpc";
+import { scoresRouter } from "./routers/scores";
+import { tracesRouter } from "./routers/traces";
+import { dashboardRouter } from "@/src/features/dashboard/server/dashboard-router";
+
+export const appRouter = createTRPCRouter({
+  scores: scoresRouter,
+  traces: tracesRouter,
+  dashboard: dashboardRouter,
+  // ... other routers
+});
+
+export type AppRouter = typeof appRouter;
+```
+
+**Calling from frontend:**
+
+```typescript
+// Type-safe client call
+const { data, isLoading } = api.scores.all.useQuery({
+  projectId: "proj_123",
+  page: 0,
+  limit: 50,
+  filter: [],
+  orderBy: null,
+});
+```
+
+---
+
+## Public REST API Routes
+
+**Location:** `web/src/pages/api/public/`
+
+Public API routes use **Next.js file-based routing** and provide REST endpoints for SDKs and external integrations.
+
+### File-based Routing
+
+Next.js uses file system for routing:
+
+```
+web/src/pages/api/public/
+├── scores/
+│   ├── index.ts          → GET/POST /api/public/scores
+│   └── [scoreId].ts      → GET/PATCH/DELETE /api/public/scores/:scoreId
+├── traces/
+│   ├── index.ts          → GET /api/public/traces
+│   └── [traceId].ts      → GET /api/public/traces/:traceId
+└── datasets/
+    └── [name]/
+        ├── index.ts      → GET/POST /api/public/datasets/:name
+        └── items/
+            └── index.ts  → GET /api/public/datasets/:name/items
+```
+
+**Dynamic routes:**
+
+- `[param].ts` → Single dynamic segment (e.g., `/api/public/scores/[scoreId].ts`)
+- `[...param].ts` → Catch-all route (e.g., `/api/public/[...path].ts`)
+
+### REST API Pattern
+
+**File:** `web/src/pages/api/public/scores/index.ts`
+
+```typescript
+import { v4 } from "uuid";
+import { createAuthedProjectAPIRoute } from "@/src/features/public-api/server/createAuthedProjectAPIRoute";
+import { withMiddlewares } from "@/src/features/public-api/server/withMiddlewares";
+import {
+  GetScoresQueryV1,
+  GetScoresResponseV1,
+  PostScoresBodyV1,
+  PostScoresResponseV1,
+} from "@langfuse/shared";
+import { eventTypes, processEventBatch } from "@langfuse/shared/src/server";
+import { ScoresApiService } from "@/src/features/public-api/server/scores-api-service";
+
+export default withMiddlewares({
+  // POST /api/public/scores
+  POST: createAuthedProjectAPIRoute({
+    name: "Create Score",
+    bodySchema: PostScoresBodyV1,
+    responseSchema: PostScoresResponseV1,
+    fn: async ({ body, auth, res }) => {
+      const event = {
+        id: v4(),
+        type: eventTypes.SCORE_CREATE,
+        timestamp: new Date().toISOString(),
+        body,
+      };
+
+      if (!event.body.id) {
+        event.body.id = v4();
+      }
+
+      const result = await processEventBatch([event], auth);
+
+      if (result.errors.length > 0) {
+        const error = result.errors[0];
+        res.status(error.status).json({
+          message: error.error ?? error.message,
+        });
+        return { id: "" };
+      }
+
+      return { id: event.body.id };
+    },
+  }),
+
+  // GET /api/public/scores
+  GET: createAuthedProjectAPIRoute({
+    name: "Get Scores",
+    querySchema: GetScoresQueryV1,
+    responseSchema: GetScoresResponseV1,
+    fn: async ({ query, auth }) => {
+      const scoresApiService = new ScoresApiService("v1");
+
+      const [items, count] = await Promise.all([
+        scoresApiService.generateScoresForPublicApi({
+          projectId: auth.scope.projectId,
+          page: query.page,
+          limit: query.limit,
+          userId: query.userId,
+          name: query.name,
+        }),
+        scoresApiService.getScoresCountForPublicApi({
+          projectId: auth.scope.projectId,
+          userId: query.userId,
+          name: query.name,
+        }),
+      ]);
+
+      return {
+        data: items,
+        meta: {
+          page: query.page,
+          limit: query.limit,
+          totalItems: count,
+          totalPages: Math.ceil(count / query.limit),
+        },
+      };
+    },
+  }),
+});
+```
+
+**Key Points:**
+
+- Use `withMiddlewares` for all public API routes (provides CORS, error handling, OpenTelemetry)
+- Use `createAuthedProjectAPIRoute` for authenticated endpoints (handles auth, rate limiting, validation)
+- Define separate handlers for each HTTP method
+- Input/output validated with Zod schemas
+- Delegate to services for business logic
+
+### Versioned API Type Location
+
+When a feature has versioned public API types, place them in `packages/shared/`
+under the feature's `interfaces/api/` folder, one subdirectory per version.
+The scores feature (`packages/shared/src/features/scores/interfaces/api/`) is
+the canonical example — the `GetScoresQueryV1` / `GetScoresResponseV1` symbols
+imported from `@langfuse/shared` originate there.
+
+Do not create a flat file (e.g. `<domain>-api-v2.ts`) and do not place
+versioned types in `web/src/features/public-api/types/`.
+
+```
+packages/shared/src/features/<domain>/interfaces/api/
+├── v1/
+│   ├── schemas.ts     # Zod schemas for request/response shapes
+│   ├── endpoints.ts   # Composed request/response types
+│   └── validation.ts  # Cross-field validation helpers
+├── v2/
+│   └── ...
+└── vN/
+    └── ...
+```
+
+### Fern API Definitions
+
+When modifying public API types in `web/src/features/public-api/types/` or under
+`packages/shared/src/features/<domain>/interfaces/api/`, update the matching
+Fern API definitions in `fern/apis/server/definition/`.
+
+**Zod to Fern Type Mapping:**
+
+| Zod Type       | Fern Type               | Example                                                   |
+| -------------- | ----------------------- | --------------------------------------------------------- |
+| `.nullish()`   | `optional<nullable<T>>` | `z.string().nullish()` -> `optional<nullable<string>>`    |
+| `.nullable()`  | `nullable<T>`           | `z.string().nullable()` -> `nullable<string>`             |
+| `.optional()`  | `optional<T>`           | `z.string().optional()` -> `optional<string>`             |
+| Always present | `T`                     | `z.string()` -> `string`                                  |
+
+Add a source comment at the top of each Fern type that references the
+TypeScript source:
+
+```yaml
+# Source: web/src/features/public-api/types/traces.ts - APITrace
+Trace:
+  properties:
+    id: string
+    name:
+      type: nullable<string>
+```
+
+### Simple Public Routes
+
+For routes that don't need authentication:
+
+```typescript
+// web/src/pages/api/public/health.ts
+import { withMiddlewares } from "@/src/features/public-api/server/withMiddlewares";
+
+export default withMiddlewares({
+  GET: async (req, res) => {
+    res.status(200).json({ status: "ok" });
+  },
+});
+```
+
+---
+
+## Service Layer
+
+**Location:** `web/src/features/*/server/` or `packages/shared/src/server/services/`
+
+Services contain business logic and orchestrate operations. They're called by tRPC procedures and API routes.
+
+### Service Pattern
+
+**File:** `web/src/features/public-api/server/scores-api-service.ts`
+
+```typescript
+import {
+  _handleGenerateScoresForPublicApi,
+  _handleGetScoresCountForPublicApi,
+  type ScoreQueryType,
+} from "@/src/features/public-api/server/scores";
+import { _handleGetScoreById } from "@langfuse/shared/src/server";
+
+export class ScoresApiService {
+  constructor(private readonly apiVersion: "v1" | "v2") {}
+
+  /**
+   * Get a specific score by ID
+   */
+  async getScoreById({
+    projectId,
+    scoreId,
+    source,
+  }: {
+    projectId: string;
+    scoreId: string;
+    source?: ScoreSourceType;
+  }) {
+    return _handleGetScoreById({
+      projectId,
+      scoreId,
+      source,
+      scoreScope: this.apiVersion === "v1" ? "traces_only" : "all",
+      preferredClickhouseService: "ReadOnly",
+    });
+  }
+
+  /**
+   * Get list of scores with version-aware filtering
+   */
+  async generateScoresForPublicApi(props: ScoreQueryType) {
+    return _handleGenerateScoresForPublicApi({
+      props,
+      scoreScope: this.apiVersion === "v1" ? "traces_only" : "all",
+    });
+  }
+
+  /**
+   * Get count of scores with version-aware filtering
+   */
+  async getScoresCountForPublicApi(props: ScoreQueryType) {
+    return _handleGetScoresCountForPublicApi({
+      props,
+      scoreScope: this.apiVersion === "v1" ? "traces_only" : "all",
+    });
+  }
+}
+```
+
+**Key Points:**
+
+- Services contain business logic, not routing logic
+- Services should NOT import tRPC or Next.js types
+- Services can call repositories, Prisma, ClickHouse directly
+- Services orchestrate multiple operations
+- Services are reusable across tRPC and public API
+
+### Where to Put Services
+
+**Feature-specific services:**
+
+```
+web/src/features/
+├── datasets/
+│   └── server/
+│       └── dataset-service.ts
+├── evals/
+│   └── server/
+│       └── eval-service.ts
+└── public-api/
+    └── server/
+        └── scores-api-service.ts
+```
+
+**Shared services:**
+
+```
+packages/shared/src/server/services/
+├── SlackService.ts
+├── DashboardService/
+├── StorageService.ts
+└── DefaultEvaluationModelService/
+```
+
+---
+
+## Repository Layer
+
+**Location:** `packages/shared/src/server/repositories/`
+
+Repositories handle complex database queries, data transformation, and provide reusable query logic.
+
+### Repository Structure
+
+```
+packages/shared/src/server/repositories/
+├── traces.ts              # Trace queries (ClickHouse)
+├── observations.ts        # Observation queries (ClickHouse)
+├── scores.ts              # Score queries (ClickHouse)
+├── clickhouse.ts          # Core ClickHouse helpers
+└── definitions.ts         # Type definitions
+```
+
+### Repository Pattern
+
+**File:** `packages/shared/src/server/repositories/traces.ts`
+
+```typescript
+import { queryClickhouse, upsertClickhouse } from "./clickhouse";
+import { TraceRecordReadType } from "./definitions";
+import { convertClickhouseToDomain } from "./traces_converters";
+
+/**
+ * Get traces by IDs
+ */
+export const getTracesByIds = async (
+  projectId: string,
+  traceIds: string[],
+): Promise<TraceRecordReadType[]> => {
+  const rows = await queryClickhouse<TraceRecordReadType>({
+    query: `
+      SELECT *
+      FROM traces
+      WHERE project_id = {projectId: String}
+      AND id IN ({traceIds: Array(String)})
+      ORDER BY event_ts DESC
+      LIMIT 1 BY id, project_id
+    `,
+    params: { projectId, traceIds },
+    tags: { feature: "tracing", type: "trace" },
+  });
+
+  return rows.map(convertClickhouseToDomain);
+};
+
+/**
+ * Upsert trace to ClickHouse
+ */
+export const upsertTrace = async (
+  trace: TraceRecordInsertType,
+): Promise<void> => {
+  await upsertClickhouse({
+    table: "traces",
+    records: [trace],
+    eventBodyMapper: (body) => ({
+      id: body.id,
+      name: body.name,
+      user_id: body.user_id,
+      // ... map fields
+    }),
+    tags: { feature: "ingestion", type: "trace" },
+  });
+};
+```
+
+**Key Points:**
+
+- Use `queryClickhouse` for SELECT queries
+- Use `upsertClickhouse` for INSERT/UPDATE
+- Use `commandClickhouse` for DDL (ALTER TABLE, etc.)
+- Include data converters (`convertClickhouseToDomain`)
+- Add OpenTelemetry tags for observability
+- Repositories should NOT contain business logic
+
+### When to Use Repositories
+
+✅ **Use repositories for:**
+
+- Complex ClickHouse queries with CTEs, joins, aggregations
+- Queries used in multiple places (DRY principle)
+- Data transformation from DB types to domain models
+- Streaming large result sets
+
+❌ **Use direct Prisma/ClickHouse for:**
+
+- Simple CRUD operations
+- One-off queries
+- Prototyping (can refactor to repository later)
+
+---
+
+## Separation of Concerns
+
+### ✅ Good Example: Proper Layering
+
+**tRPC Procedure (Entry Point):**
+
+```typescript
+// web/src/server/api/routers/scores.ts
+export const scoresRouter = createTRPCRouter({
+  all: protectedProjectProcedure
+    .input(ScoreFilterOptions)
+    .query(async ({ input }) => {
+      // ✅ Thin procedure - delegates to repository
+      return await getScoresUiTable({
+        projectId: input.projectId,
+        filter: input.filter,
+        orderBy: input.orderBy,
+      });
+    }),
+
+  create: protectedProjectProcedure
+    .input(CreateScoreInput)
+    .mutation(async ({ input, ctx }) => {
+      // ✅ Delegates to service for orchestration
+      return await createScoreWithValidation({
+        scoreData: input,
+        userId: ctx.session.user.id,
+        projectId: ctx.session.projectId,
+      });
+    }),
+});
+```
+
+**Service (Business Logic):**
+
+```typescript
+// web/src/features/scores/server/score-service.ts
+export async function createScoreWithValidation({
+  scoreData,
+  userId,
+  projectId,
+}: {
+  scoreData: CreateScoreInput;
+  userId: string;
+  projectId: string;
+}) {
+  // ✅ Business logic: validation
+  const config = await prisma.scoreConfig.findUnique({
+    where: { id: scoreData.configId },
+  });
+
+  if (!config) {
+    throw new LangfuseNotFoundError("Score config not found");
+  }
+
+  validateConfigAgainstBody(config, scoreData);
+
+  // ✅ Business logic: orchestration
+  const scoreId = randomUUID();
+
+  await Promise.all([
+    // Create score in ClickHouse
+    upsertScore({
+      id: scoreId,
+      projectId,
+      traceId: scoreData.traceId,
+      name: scoreData.name,
+      value: scoreData.value,
+      authorUserId: userId,
+    }),
+    // Audit log in PostgreSQL
+    auditLog({
+      userId,
+      resourceType: "score",
+      resourceId: scoreId,
+      action: "create",
+    }),
+  ]);
+
+  return { id: scoreId };
+}
+```
+
+**Repository (Data Access):**
+
+```typescript
+// packages/shared/src/server/repositories/scores.ts
+export const upsertScore = async (score: ScoreInsertType): Promise<void> => {
+  // ✅ Pure data access - no business logic
+  await upsertClickhouse({
+    table: "scores",
+    records: [score],
+    eventBodyMapper: (body) => ({
+      id: body.id,
+      trace_id: body.traceId,
+      name: body.name,
+      value: body.value,
+      author_user_id: body.authorUserId,
+    }),
+    tags: { feature: "scoring" },
+  });
+};
+```
+
+### Why This Works
+
+1. **tRPC Procedure**: Thin, delegates to service
+2. **Service**: Contains all business logic (validation, orchestration)
+3. **Repository**: Pure data access, reusable
+4. **Service is protocol-agnostic**: Can be called from tRPC, public API, or worker
+5. **Clear separation**: Easy to test, maintain, extend
+
+---
+
+## Anti-Patterns
+
+### ❌ Anti-Pattern 1: Business Logic in Routes
+
+**Bad:**
+
+```typescript
+// ❌ BAD: Business logic in tRPC procedure
+export const scoresRouter = createTRPCRouter({
+  create: protectedProjectProcedure
+    .input(CreateScoreInput)
+    .mutation(async ({ input, ctx }) => {
+      // ❌ Validation logic in route
+      const config = await ctx.prisma.scoreConfig.findUnique({
+        where: { id: input.configId },
+      });
+
+      if (!config) {
+        throw new TRPCError({ code: "NOT_FOUND" });
+      }
+
+      if (config.dataType === "NUMERIC" && typeof input.value !== "number") {
+        throw new TRPCError({ code: "BAD_REQUEST" });
+      }
+
+      // ❌ Direct database access
+      await ctx.prisma.score.create({
+        data: {
+          id: randomUUID(),
+          projectId: ctx.session.projectId,
+          traceId: input.traceId,
+          name: input.name,
+          value: input.value,
+        },
+      });
+
+      // ❌ More business logic
+      await auditLog({ ... });
+
+      return { success: true };
+    }),
+});
+```
+
+**Why it's bad:**
+
+- Business logic tied to tRPC (can't reuse in public API)
+- Hard to test (need to mock tRPC context)
+- No separation of concerns
+- Difficult to maintain
+
+**Good:**
+
+```typescript
+// ✅ GOOD: Thin procedure, delegates to service
+export const scoresRouter = createTRPCRouter({
+  create: protectedProjectProcedure
+    .input(CreateScoreInput)
+    .mutation(async ({ input, ctx }) => {
+      return await createScoreWithValidation({
+        scoreData: input,
+        userId: ctx.session.user.id,
+        projectId: ctx.session.projectId,
+      });
+    }),
+});
+```
+
+### ❌ Anti-Pattern 2: Database Calls in Routes
+
+**Bad:**
+
+```typescript
+// ❌ BAD: Direct database access in route
+export default withMiddlewares({
+  GET: createAuthedProjectAPIRoute({
+    name: "Get Scores",
+    fn: async ({ auth }) => {
+      // ❌ Direct ClickHouse query in route
+      const scores = await queryClickhouse({
+        query: "SELECT * FROM scores WHERE project_id = {projectId: String}",
+        params: { projectId: auth.scope.projectId },
+      });
+
+      return { data: scores };
+    },
+  }),
+});
+```
+
+**Good:**
+
+```typescript
+// ✅ GOOD: Delegates to service or repository
+export default withMiddlewares({
+  GET: createAuthedProjectAPIRoute({
+    name: "Get Scores",
+    fn: async ({ auth, query }) => {
+      const scoresService = new ScoresApiService("v1");
+
+      return await scoresService.generateScoresForPublicApi({
+        projectId: auth.scope.projectId,
+        page: query.page,
+        limit: query.limit,
+      });
+    },
+  }),
+});
+```
+
+### ❌ Anti-Pattern 3: Business Logic in Repositories
+
+**Bad:**
+
+```typescript
+// ❌ BAD: Business logic in repository
+export const upsertScore = async (
+  score: ScoreInsertType
+): Promise<void> => {
+  // ❌ Validation in repository
+  if (!score.name) {
+    throw new Error("Score name is required");
+  }
+
+  // ❌ Authorization check in repository
+  const project = await prisma.project.findUnique({
+    where: { id: score.projectId },
+  });
+
+  if (!project) {
+    throw new Error("Project not found");
+  }
+
+  // ❌ Side effects in repository
+  await auditLog({ ... });
+
+  await upsertClickhouse({ ... });
+};
+```
+
+**Good:**
+
+```typescript
+// ✅ GOOD: Pure data access, no business logic
+export const upsertScore = async (score: ScoreInsertType): Promise<void> => {
+  await upsertClickhouse({
+    table: "scores",
+    records: [score],
+    eventBodyMapper: (body) => ({
+      id: body.id,
+      trace_id: body.traceId,
+      name: body.name,
+      value: body.value,
+    }),
+    tags: { feature: "scoring" },
+  });
+};
+```
+
+---
+
+**Related Files:**
+
+- [../SKILL.md](../SKILL.md) - Main backend development guidelines
+- [architecture-overview.md](architecture-overview.md) - System architecture
+- [middleware-guide.md](middleware-guide.md) - Middleware patterns
+- [database-patterns.md](database-patterns.md) - Database access patterns

.agents/skills/backend-dev-guidelines/references/services-and-repositories.md

@@ -0,0 +1,650 @@
+# Services and Repositories - Business Logic Layer
+
+Complete guide to organizing business logic with services and data access with repositories.
+
+## Table of Contents
+
+- [Service Layer Overview](#service-layer-overview)
+- [Dependency Injection Pattern](#dependency-injection-pattern)
+- [Singleton Pattern](#singleton-pattern)
+- [Repository Pattern](#repository-pattern)
+- [Service Design Principles](#service-design-principles)
+- [Caching Strategies](#caching-strategies)
+- [Testing Services](#testing-services)
+
+---
+
+## Service Layer Overview
+
+### Purpose of Services
+
+**Services contain business logic** - the 'what' and 'why' of your application:
+
+```
+Controller asks: "Should I do this?"
+Service answers: "Yes/No, here's why, and here's what happens"
+Repository executes: "Here's the data you requested"
+```
+
+**Services are responsible for:**
+
+- ✅ Business rules enforcement
+- ✅ Orchestrating multiple repositories
+- ✅ Transaction management
+- ✅ Complex calculations
+- ✅ External service integration
+- ✅ Business validations
+
+**Services should NOT:**
+
+- ❌ Know about HTTP (Request/Response)
+- ❌ Direct Prisma access (use repositories)
+- ❌ Handle route-specific logic
+- ❌ Format HTTP responses
+
+---
+
+## Dependency Injection Pattern
+
+### Why Dependency Injection?
+
+**Benefits:**
+
+- Easy to test (inject mocks)
+- Clear dependencies
+- Flexible configuration
+- Promotes loose coupling
+
+### Excellent Example: NotificationService
+
+**File:** `/blog-api/src/services/NotificationService.ts`
+
+```typescript
+// Define dependencies interface for clarity
+export interface NotificationServiceDependencies {
+  prisma: PrismaClient;
+  batchingService: BatchingService;
+  emailComposer: EmailComposer;
+}
+
+// Service with dependency injection
+export class NotificationService {
+  private prisma: PrismaClient;
+  private batchingService: BatchingService;
+  private emailComposer: EmailComposer;
+  private preferencesCache: Map<
+    string,
+    { preferences: UserPreference; timestamp: number }
+  > = new Map();
+  private CACHE_TTL =
+    (notificationConfig.preferenceCacheTTLMinutes || 5) * 60 * 1000;
+
+  // Dependencies injected via constructor
+  constructor(dependencies: NotificationServiceDependencies) {
+    this.prisma = dependencies.prisma;
+    this.batchingService = dependencies.batchingService;
+    this.emailComposer = dependencies.emailComposer;
+  }
+
+  /**
+   * Create a notification and route it appropriately
+   */
+  async createNotification(params: CreateNotificationParams) {
+    const {
+      recipientID,
+      type,
+      title,
+      message,
+      link,
+      context = {},
+      channel = "both",
+      priority = NotificationPriority.NORMAL,
+    } = params;
+
+    try {
+      // Get template and render content
+      const template = getNotificationTemplate(type);
+      const rendered = renderNotificationContent(template, context);
+
+      // Create in-app notification record
+      const notificationId = await createNotificationRecord({
+        instanceId: parseInt(context.instanceId || "0", 10),
+        template: type,
+        recipientUserId: recipientID,
+        channel: channel === "email" ? "email" : "inApp",
+        contextData: context,
+        title: finalTitle,
+        message: finalMessage,
+        link: finalLink,
+      });
+
+      // Route notification based on channel
+      if (channel === "email" || channel === "both") {
+        await this.routeNotification({
+          notificationId,
+          userId: recipientID,
+          type,
+          priority,
+          title: finalTitle,
+          message: finalMessage,
+          link: finalLink,
+          context,
+        });
+      }
+
+      return notification;
+    } catch (error) {
+      ErrorLogger.log(error, {
+        context: {
+          "[NotificationService] createNotification": {
+            type: params.type,
+            recipientID: params.recipientID,
+          },
+        },
+      });
+      throw error;
+    }
+  }
+
+  /**
+   * Route notification based on user preferences
+   */
+  private async routeNotification(params: {
+    notificationId: number;
+    userId: string;
+    type: string;
+    priority: NotificationPriority;
+    title: string;
+    message: string;
+    link?: string;
+    context?: Record<string, any>;
+  }) {
+    // Get user preferences with caching
+    const preferences = await this.getUserPreferences(params.userId);
+
+    // Check if we should batch or send immediately
+    if (this.shouldBatchEmail(preferences, params.type, params.priority)) {
+      await this.batchingService.queueNotificationForBatch({
+        notificationId: params.notificationId,
+        userId: params.userId,
+        userPreference: preferences,
+        priority: params.priority,
+      });
+    } else {
+      // Send immediately via EmailComposer
+      await this.sendImmediateEmail({
+        userId: params.userId,
+        title: params.title,
+        message: params.message,
+        link: params.link,
+        context: params.context,
+        type: params.type,
+      });
+    }
+  }
+
+  /**
+   * Determine if email should be batched
+   */
+  shouldBatchEmail(
+    preferences: UserPreference,
+    notificationType: string,
+    priority: NotificationPriority,
+  ): boolean {
+    // HIGH priority always immediate
+    if (priority === NotificationPriority.HIGH) {
+      return false;
+    }
+
+    // Check batch mode
+    const batchMode = preferences.emailBatchMode || BatchMode.IMMEDIATE;
+    return batchMode !== BatchMode.IMMEDIATE;
+  }
+
+  /**
+   * Get user preferences with caching
+   */
+  async getUserPreferences(userId: string): Promise<UserPreference> {
+    // Check cache first
+    const cached = this.preferencesCache.get(userId);
+    if (cached && Date.now() - cached.timestamp < this.CACHE_TTL) {
+      return cached.preferences;
+    }
+
+    const preference = await this.prisma.userPreference.findUnique({
+      where: { userID: userId },
+    });
+
+    const finalPreferences = preference || DEFAULT_PREFERENCES;
+
+    // Update cache
+    this.preferencesCache.set(userId, {
+      preferences: finalPreferences,
+      timestamp: Date.now(),
+    });
+
+    return finalPreferences;
+  }
+}
+```
+
+**Usage in Controller:**
+
+```typescript
+// Instantiate with dependencies
+const notificationService = new NotificationService({
+  prisma: PrismaService.main,
+  batchingService: new BatchingService(PrismaService.main),
+  emailComposer: new EmailComposer(),
+});
+
+// Use in controller
+const notification = await notificationService.createNotification({
+  recipientID: "user-123",
+  type: "AFRLWorkflowNotification",
+  context: { workflowName: "AFRL Monthly Report" },
+});
+```
+
+**Key Takeaways:**
+
+- Dependencies passed via constructor
+- Clear interface defines required dependencies
+- Easy to test (inject mocks)
+- Encapsulated caching logic
+- Business rules isolated from HTTP
+
+---
+
+## Singleton Pattern
+
+### When to Use Singletons
+
+**Use for:**
+
+- Services with expensive initialization
+- Services with shared state (caching)
+- Services accessed from many places
+- Permission services
+- Configuration services
+
+### Example: PermissionService (Singleton)
+
+**File:** `/blog-api/src/services/permissionService.ts`
+
+```typescript
+import { PrismaClient } from "@prisma/client";
+
+class PermissionService {
+  private static instance: PermissionService;
+  private prisma: PrismaClient;
+  private permissionCache: Map<
+    string,
+    { canAccess: boolean; timestamp: number }
+  > = new Map();
+  private CACHE_TTL = 5 * 60 * 1000; // 5 minutes
+
+  // Private constructor prevents direct instantiation
+  private constructor() {
+    this.prisma = PrismaService.main;
+  }
+
+  // Get singleton instance
+  public static getInstance(): PermissionService {
+    if (!PermissionService.instance) {
+      PermissionService.instance = new PermissionService();
+    }
+    return PermissionService.instance;
+  }
+
+  /**
+   * Check if user can complete a workflow step
+   */
+  async canCompleteStep(
+    userId: string,
+    stepInstanceId: number,
+  ): Promise<boolean> {
+    const cacheKey = `${userId}:${stepInstanceId}`;
+
+    // Check cache
+    const cached = this.permissionCache.get(cacheKey);
+    if (cached && Date.now() - cached.timestamp < this.CACHE_TTL) {
+      return cached.canAccess;
+    }
+
+    try {
+      const post = await this.prisma.post.findUnique({
+        where: { id: postId },
+        include: {
+          author: true,
+          comments: {
+            include: {
+              user: true,
+            },
+          },
+        },
+      });
+
+      if (!post) {
+        return false;
+      }
+
+      // Check if user has permission
+      const canEdit =
+        post.authorId === userId || (await this.isUserAdmin(userId));
+
+      // Cache result
+      this.permissionCache.set(cacheKey, {
+        canAccess: isAssigned,
+        timestamp: Date.now(),
+      });
+
+      return isAssigned;
+    } catch (error) {
+      console.error(
+        "[PermissionService] Error checking step permission:",
+        error,
+      );
+      return false;
+    }
+  }
+
+  /**
+   * Clear cache for user
+   */
+  clearUserCache(userId: string): void {
+    for (const [key] of this.permissionCache) {
+      if (key.startsWith(`${userId}:`)) {
+        this.permissionCache.delete(key);
+      }
+    }
+  }
+
+  /**
+   * Clear all cache
+   */
+  clearCache(): void {
+    this.permissionCache.clear();
+  }
+}
+
+// Export singleton instance
+export const permissionService = PermissionService.getInstance();
+```
+
+**Usage:**
+
+```typescript
+import { permissionService } from "../services/permissionService";
+
+// Use anywhere in the codebase
+const canComplete = await permissionService.canCompleteStep(userId, stepId);
+
+if (!canComplete) {
+  throw new ForbiddenError("You do not have permission to complete this step");
+}
+```
+
+---
+
+## Repository Pattern
+
+### Purpose of Repositories
+
+**Repositories abstract data access** - the 'how' of data operations:
+
+```
+Service: "Get me all active users sorted by name"
+Repository: "Here's the Prisma query that does that"
+```
+
+**Repositories are responsible for:**
+
+- ✅ All Prisma operations
+- ✅ Query construction
+- ✅ Query optimization (select, include)
+- ✅ Database error handling
+- ✅ Caching database results
+
+**Repositories should NOT:**
+
+- ❌ Contain business logic
+- ❌ Know about HTTP
+- ❌ Make decisions (that's service layer)
+
+### Langfuse Repository Examples
+
+Use these current Langfuse files as repository templates:
+
+- PostgreSQL repository with project-scoped filters:
+  `packages/shared/src/server/repositories/comments.ts`
+- ClickHouse repository with project-scoped filters and query helpers:
+  `packages/shared/src/server/repositories/traces.ts`
+- Repository tests:
+  `web/src/__tests__/server/repositories/event-repository.servertest.ts`
+
+Keep data-access concerns in repositories and business decisions in services.
+Project-scoped queries must include `projectId` or `project_id` filters.
+
+---
+
+## Service Design Principles
+
+### 1. Single Responsibility
+
+Each service should have ONE clear purpose:
+
+```typescript
+// ✅ GOOD - Single responsibility
+class UserService {
+  async createUser() {}
+  async updateUser() {}
+  async deleteUser() {}
+}
+
+class EmailService {
+  async sendEmail() {}
+  async sendBulkEmails() {}
+}
+
+// ❌ BAD - Too many responsibilities
+class UserService {
+  async createUser() {}
+  async sendWelcomeEmail() {} // Should be EmailService
+  async logUserActivity() {} // Should be AuditService
+  async processPayment() {} // Should be PaymentService
+}
+```
+
+### 2. Clear Method Names
+
+Method names should describe WHAT they do:
+
+```typescript
+// ✅ GOOD - Clear intent
+async createNotification()
+async getUserPreferences()
+async shouldBatchEmail()
+async routeNotification()
+
+// ❌ BAD - Vague or misleading
+async process()
+async handle()
+async doIt()
+async execute()
+```
+
+### 3. Use Params Objects for Multiple Arguments
+
+When a function receives multiple arguments, use a single params object instead of positional arguments:
+
+```typescript
+// ❌ BAD - Positional arguments are unclear and can be swapped
+async function createTrace(
+  projectId: string,
+  userId: string,
+  sessionId: string,
+  name: string,
+) {}
+
+// Call site - which string is which?
+await createTrace(projectId, userId, sessionId, name);
+
+// ✅ GOOD - Params object makes intent clear
+async function createTrace(params: {
+  projectId: string;
+  userId: string;
+  sessionId: string;
+  name: string;
+}) {}
+
+// Call site - clear and prevents argument swapping bugs
+await createTrace({ projectId, userId, sessionId, name });
+```
+
+**Benefits:**
+
+- More readable at call sites
+- Prevents bugs when positional arguments of the same type are accidentally swapped
+- Easier to add optional parameters later
+- Self-documenting code
+
+### 4. Return Types
+
+Always use explicit return types:
+
+```typescript
+// ✅ GOOD - Explicit types
+async createUser(data: CreateUserDTO): Promise<User> {}
+async findUsers(): Promise<User[]> {}
+async deleteUser(id: string): Promise<void> {}
+
+// ❌ BAD - Implicit any
+async createUser(data) {}  // No types!
+```
+
+### 5. Error Handling
+
+Services should throw meaningful errors:
+
+```typescript
+// ✅ GOOD - Meaningful errors
+if (!user) {
+  throw new NotFoundError(`User not found: ${userId}`);
+}
+
+if (emailExists) {
+  throw new ConflictError("Email already exists");
+}
+
+// ❌ BAD - Generic errors
+if (!user) {
+  throw new Error("Error"); // What error?
+}
+```
+
+### 6. Avoid God Services
+
+Don't create services that do everything:
+
+```typescript
+// ❌ BAD - God service
+class WorkflowService {
+  async startWorkflow() {}
+  async completeStep() {}
+  async assignRoles() {}
+  async sendNotifications() {} // Should be NotificationService
+  async validatePermissions() {} // Should be PermissionService
+  async logAuditTrail() {} // Should be AuditService
+  // ... 50 more methods
+}
+
+// ✅ GOOD - Focused services
+class WorkflowService {
+  constructor(
+    private notificationService: NotificationService,
+    private permissionService: PermissionService,
+    private auditService: AuditService,
+  ) {}
+
+  async startWorkflow() {
+    // Orchestrate other services
+    await this.permissionService.checkPermission();
+    await this.workflowRepository.create();
+    await this.notificationService.notify();
+    await this.auditService.log();
+  }
+}
+```
+
+---
+
+## Caching Strategies
+
+### 1. In-Memory Caching
+
+```typescript
+class UserService {
+  private cache: Map<string, { user: User; timestamp: number }> = new Map();
+  private CACHE_TTL = 5 * 60 * 1000; // 5 minutes
+
+  async getUser(userId: string): Promise<User> {
+    // Check cache
+    const cached = this.cache.get(userId);
+    if (cached && Date.now() - cached.timestamp < this.CACHE_TTL) {
+      return cached.user;
+    }
+
+    // Fetch from database
+    const user = await userRepository.findById(userId);
+
+    // Update cache
+    if (user) {
+      this.cache.set(userId, { user, timestamp: Date.now() });
+    }
+
+    return user;
+  }
+
+  clearUserCache(userId: string): void {
+    this.cache.delete(userId);
+  }
+}
+```
+
+### 2. Cache Invalidation
+
+```typescript
+class UserService {
+  async updateUser(userId: string, data: UpdateUserDTO): Promise<User> {
+    // Update in database
+    const user = await userRepository.update(userId, data);
+
+    // Invalidate cache
+    this.clearUserCache(userId);
+
+    return user;
+  }
+}
+```
+
+---
+
+## Testing Services
+
+Use `testing-guide.md` for backend test patterns. Prefer current Langfuse tests
+over invented examples:
+
+- Repository tests:
+  `web/src/__tests__/server/repositories/event-repository.servertest.ts`
+- Pure service unit tests:
+  `web/src/__tests__/server/unit/`
+
+---
+
+**Related Files:**
+
+- [../SKILL.md](../SKILL.md) - Main guide
+- [routing-and-controllers.md](routing-and-controllers.md) - Controllers that use services
+- [database-patterns.md](database-patterns.md) - Prisma and repository patterns
+- [testing-guide.md](testing-guide.md) - Testing service and repository code

.agents/skills/backend-dev-guidelines/references/testing-guide.md

@@ -0,0 +1,543 @@
+# Testing Guide - Backend Testing Strategies
+
+Complete guide to testing Langfuse backend services across web, worker, and shared packages.
+
+## Table of Contents
+
+- [Key Testing Principles](#key-testing-principles)
+- [Test Types Overview](#test-types-overview)
+- [Integration Tests (Public API)](#integration-tests-public-api)
+- [Service-Level Tests (Repository/Service)](#service-level-tests-repositoryservice)
+- [tRPC Tests (Procedure Testing)](#trpc-tests-procedure-testing)
+- [Worker Tests (Queue Processing)](#worker-tests-queue-processing)
+- [Running Tests](#running-tests)
+
+---
+
+## Key Testing Principles
+
+### General Principles
+
+1. **Test Isolation**: Each test should be independent and runnable in any order
+2. **Unique IDs**: Use `randomUUID()` or unique project IDs to avoid test interference
+3. **Cleanup**: Always clean up test data in service tests (or use unique project IDs)
+4. **Avoid Global Resets**: Prefer scoped cleanup or unique project IDs over global reset helpers
+5. **Flags and Fallbacks**: When code branches on env flags, feature flags, or fallback data paths, test both branches and ensure fixtures are written to the same store the branch reads from
+
+### By Test Type
+
+| Test Type       | Key Principles                                                                   |
+| --------------- | -------------------------------------------------------------------------------- |
+| **Integration** | Test HTTP endpoints, validate status codes and response shapes                   |
+| **tRPC**        | Use `createInnerTRPCContext` and `appRouter.createCaller`, test auth/permissions |
+| **Service**     | Test individual functions with isolated data, always cleanup                     |
+| **Worker**      | Use vitest, test streams with async iteration, test filtering logic              |
+
+### Test Data Management
+
+```typescript
+// ✅ GOOD: Use unique IDs
+const projectId = randomUUID();
+const traceId = randomUUID();
+
+// ✅ GOOD: Cleanup in service tests
+afterAll(async () => {
+  await prisma.model.delete({ where: { id: modelId } });
+});
+
+// ✅ GOOD: Use unique projects (no cleanup needed)
+const { projectId } = await createOrgProjectAndApiKey();
+
+// ❌ BAD: Shared test data between tests
+const projectId = "7a88fb47-b4e2-43b8-a06c-a5ce950dc53a";
+```
+
+---
+
+## Test Types Overview
+
+Langfuse uses multiple testing strategies for different layers:
+
+| Test Type   | Framework | Location                                 | Purpose                             |
+| ----------- | --------- | ---------------------------------------- | ----------------------------------- |
+| Integration | Vitest    | `web/src/__tests__/server/`              | Full API endpoint testing           |
+| tRPC        | Vitest    | `web/src/__tests__/server/`              | tRPC procedure testing with auth    |
+| Service     | Vitest    | `web/src/__tests__/server/repositories/` | Repository/service function testing |
+| Worker      | Vitest    | `worker/src/__tests__/`                  | Queue processors and streams        |
+
+---
+
+## Integration Tests (Public API)
+
+Test full REST API endpoints end-to-end using HTTP requests.
+
+**File location:** `web/src/__tests__/server/datasets-api.servertest.ts`
+
+```typescript
+import { makeZodVerifiedAPICall } from "../helpers";
+import { PostDatasetsV1Response } from "@/src/features/public-api/types/datasets";
+
+describe("Dataset API", () => {
+  it("should create dataset", async () => {
+    const res = await makeZodVerifiedAPICall(
+      PostDatasetsV1Response,
+      "POST",
+      "/api/public/datasets",
+      { name: "test-dataset" },
+      auth,
+    );
+    expect(res.status).toBe(200);
+  });
+
+  it("should validate input", async () => {
+    const res = await makeZodVerifiedAPICall(
+      PostDatasetsV1Response,
+      "POST",
+      "/api/public/datasets",
+      { name: "" }, // Invalid empty name
+      auth,
+    );
+    expect(res.status).toBe(400);
+  });
+});
+```
+
+**Key Points:**
+
+- Uses `makeZodVerifiedAPICall` for type-safe API testing
+- Tests HTTP status codes and response validation
+- Tests both success and error cases
+
+---
+
+## Service-Level Tests (Repository/Service)
+
+Test individual repository/service functions with isolated data.
+
+**File location:** `web/src/__tests__/server/repositories/event-repository.servertest.ts`
+
+```typescript
+import {
+  createEvent,
+  createEventsCh,
+  getObservationsWithModelDataFromEventsTable,
+} from "@langfuse/shared/src/server";
+import { prisma } from "@langfuse/shared/src/db";
+import { randomUUID } from "crypto";
+
+describe("Event Repository Tests", () => {
+  it("should return observations with model data", async () => {
+    const traceId = randomUUID();
+    const generationId = randomUUID();
+    const modelId = randomUUID();
+
+    // Create test data
+    await prisma.model.create({
+      data: {
+        id: modelId,
+        projectId,
+        modelName: `gpt-4-${modelId}`,
+        matchPattern: `(?i)^(gpt-?4-${modelId})$`,
+        startDate: new Date("2023-01-01"),
+        unit: "TOKENS",
+        Price: {
+          create: [
+            { usageType: "input", price: 0.03 },
+            { usageType: "output", price: 0.06 },
+          ],
+        },
+      },
+    });
+
+    const event = createEvent({
+      id: generationId,
+      span_id: generationId,
+      project_id: projectId,
+      trace_id: traceId,
+      type: "GENERATION",
+      name: `test-generation-${generationId}`,
+      model_id: modelId,
+    });
+
+    await createEventsCh([event]);
+
+    // Test the service function
+    const result = await getObservationsWithModelDataFromEventsTable({
+      projectId,
+      filter: [
+        { type: "string", column: "id", operator: "=", value: generationId },
+      ],
+      limit: 1000,
+      offset: 0,
+    });
+
+    expect(result.length).toBeGreaterThan(0);
+    const observation = result.find((o) => o.id === generationId);
+    expect(observation?.internalModelId).toBe(modelId);
+    expect(Number(observation?.inputPrice)).toBeCloseTo(0.03, 5);
+
+    // Cleanup
+    await prisma.model.delete({ where: { id: modelId } });
+  });
+
+  it("should handle filters correctly", async () => {
+    const projectId = randomUUID();
+    const traceId = randomUUID();
+
+    const observations = [
+      createEvent({
+        id: randomUUID(),
+        project_id: projectId,
+        trace_id: traceId,
+        type: "GENERATION",
+        name: "test1",
+      }),
+      createEvent({
+        id: randomUUID(),
+        project_id: projectId,
+        trace_id: traceId,
+        type: "SPAN",
+        name: "test2",
+      }),
+    ];
+
+    await createEventsCh(observations);
+
+    const result = await getObservationsWithModelDataFromEventsTable({
+      projectId,
+      filter: [
+        {
+          type: "stringOptions",
+          column: "type",
+          operator: "any of",
+          value: ["GENERATION"],
+        },
+      ],
+      limit: 1000,
+      offset: 0,
+    });
+
+    expect(result.every((o) => o.type === "GENERATION")).toBe(true);
+  });
+});
+```
+
+**Key Points:**
+
+- Tests service/repository functions directly
+- Uses ClickHouse and Prisma test data
+- Always cleanup test data after tests
+- Use unique IDs to avoid test interference
+
+---
+
+## tRPC Tests (Procedure Testing)
+
+Test tRPC procedures with caller pattern and auth context.
+
+**File location:** `web/src/__tests__/server/automations-trpc.servertest.ts`
+
+```typescript
+import { appRouter } from "@/src/server/api/root";
+import { createInnerTRPCContext } from "@/src/server/api/trpc";
+import { prisma } from "@langfuse/shared/src/db";
+import { createOrgProjectAndApiKey } from "@langfuse/shared/src/server";
+import type { Session } from "next-auth";
+import { v4 } from "uuid";
+import { JobConfigState } from "@langfuse/shared";
+
+async function prepare() {
+  const { project, org } = await createOrgProjectAndApiKey();
+
+  const session: Session = {
+    expires: "1",
+    user: {
+      id: "user-1",
+      name: "Demo User",
+      organizations: [
+        {
+          id: org.id,
+          name: org.name,
+          role: "OWNER",
+          projects: [
+            {
+              id: project.id,
+              role: "ADMIN",
+              name: project.name,
+            },
+          ],
+        },
+      ],
+    },
+  };
+
+  const ctx = createInnerTRPCContext({ session, headers: {} });
+  const caller = appRouter.createCaller({ ...ctx, prisma });
+
+  return { project, org, session, ctx, caller };
+}
+
+describe("automations trpc", () => {
+  it("should retrieve all automations for a project", async () => {
+    const { project, caller } = await prepare();
+
+    // Create test trigger
+    const trigger = await prisma.trigger.create({
+      data: {
+        id: v4(),
+        projectId: project.id,
+        eventSource: "prompt",
+        eventActions: ["created"],
+        filter: [],
+        status: JobConfigState.ACTIVE,
+      },
+    });
+
+    // Create test action
+    const action = await prisma.action.create({
+      data: {
+        id: v4(),
+        projectId: project.id,
+        type: "WEBHOOK",
+        config: {
+          type: "WEBHOOK",
+          url: "https://example.com/webhook",
+          headers: { "Content-Type": "application/json" },
+        },
+      },
+    });
+
+    // Link trigger to action
+    await prisma.automation.create({
+      data: {
+        projectId: project.id,
+        triggerId: trigger.id,
+        actionId: action.id,
+        name: "Test Automation",
+      },
+    });
+
+    // Call tRPC procedure
+    const response = await caller.automations.getAutomations({
+      projectId: project.id,
+    });
+
+    expect(response).toHaveLength(1);
+    expect(response[0]).toMatchObject({
+      name: "Test Automation",
+      trigger: expect.objectContaining({
+        id: trigger.id,
+        eventSource: "prompt",
+      }),
+    });
+  });
+
+  it("should throw error when user lacks permissions", async () => {
+    const { project, session } = await prepare();
+
+    // Create limited session
+    const limitedSession: Session = {
+      ...session,
+      user: {
+        ...session.user!,
+        organizations: [
+          {
+            ...session.user!.organizations[0],
+            projects: [
+              {
+                ...session.user!.organizations[0].projects[0],
+                role: "VIEWER", // VIEWER can't create automations
+              },
+            ],
+          },
+        ],
+      },
+    };
+
+    const limitedCtx = createInnerTRPCContext({
+      session: limitedSession,
+      headers: {},
+    });
+    const limitedCaller = appRouter.createCaller({ ...limitedCtx, prisma });
+
+    await expect(
+      limitedCaller.automations.createAutomation({
+        projectId: project.id,
+        name: "Unauthorized",
+        eventSource: "prompt",
+        eventAction: ["created"],
+        filter: [],
+        status: JobConfigState.ACTIVE,
+        actionType: "WEBHOOK",
+        actionConfig: {
+          type: "WEBHOOK",
+          url: "https://example.com/webhook",
+          requestHeaders: {},
+          apiVersion: { prompt: "v1" },
+        },
+      }),
+    ).rejects.toThrow("User does not have access");
+  });
+});
+```
+
+**Key Points:**
+
+- Uses `prepare()` helper to set up test context
+- Creates authenticated caller with `appRouter.createCaller`
+- Tests both success and permission error cases
+- Can test different user roles and permissions
+
+---
+
+## Worker Tests (Queue Processing)
+
+Test queue processors and stream functions using vitest.
+
+**File location:** `worker/src/__tests__/batchExport.test.ts`
+
+```typescript
+import { randomUUID } from "crypto";
+import { expect, describe, it } from "vitest";
+import {
+  createObservation,
+  createObservationsCh,
+  createOrgProjectAndApiKey,
+  createTraceScore,
+  createScoresCh,
+  createTrace,
+  createTracesCh,
+} from "@langfuse/shared/src/server";
+import { getObservationStream } from "../features/database-read-stream/observation-stream";
+
+describe("batch export test suite", () => {
+  it("should export observations", async () => {
+    const { projectId } = await createOrgProjectAndApiKey();
+
+    const traceId = randomUUID();
+    const trace = createTrace({
+      project_id: projectId,
+      id: traceId,
+    });
+
+    await createTracesCh([trace]);
+
+    const observations = [
+      createObservation({
+        project_id: projectId,
+        trace_id: traceId,
+        type: "SPAN",
+      }),
+      createObservation({
+        project_id: projectId,
+        trace_id: randomUUID(),
+        type: "GENERATION",
+      }),
+    ];
+
+    const score = createTraceScore({
+      project_id: projectId,
+      trace_id: traceId,
+      observation_id: observations[0].id,
+      name: "test",
+      value: 123,
+    });
+
+    await createScoresCh([score]);
+    await createObservationsCh(observations);
+
+    // Test the stream function
+    const stream = await getObservationStream({
+      projectId: projectId,
+      cutoffCreatedAt: new Date(Date.now() + 1000 * 60 * 60 * 24),
+      filter: [],
+    });
+
+    const rows: any[] = [];
+    for await (const chunk of stream) {
+      rows.push(chunk);
+    }
+
+    expect(rows).toHaveLength(2);
+    expect(rows).toEqual(
+      expect.arrayContaining([
+        expect.objectContaining({
+          id: observations[0].id,
+          type: observations[0].type,
+          test: [score.value],
+        }),
+      ]),
+    );
+  });
+
+  it("should export with filters", async () => {
+    const { projectId } = await createOrgProjectAndApiKey();
+
+    const observations = [
+      createObservation({
+        project_id: projectId,
+        trace_id: randomUUID(),
+        type: "GENERATION",
+        name: "test1",
+      }),
+      createObservation({
+        project_id: projectId,
+        trace_id: randomUUID(),
+        type: "SPAN",
+        name: "test2",
+      }),
+    ];
+
+    await createObservationsCh(observations);
+
+    const stream = await getObservationStream({
+      projectId: projectId,
+      cutoffCreatedAt: new Date(Date.now() + 1000 * 60 * 60 * 24),
+      filter: [
+        {
+          type: "stringOptions",
+          operator: "any of",
+          column: "name",
+          value: ["test1"],
+        },
+      ],
+    });
+
+    const rows: any[] = [];
+    for await (const chunk of stream) {
+      rows.push(chunk);
+    }
+
+    expect(rows).toHaveLength(1);
+    expect(rows[0].name).toBe("test1");
+  });
+});
+```
+
+**Key Points:**
+
+- Uses vitest (not Jest) for worker tests
+- Tests stream functions with async iteration
+- Creates isolated test data per test
+- Use unique project IDs to avoid interference
+
+---
+
+## Running Tests
+
+Use the nearest package `AGENTS.md` as the source of truth for current test
+commands.
+
+Common targeted forms:
+
+- Web server tests: `pnpm --filter web run test <file-or-pattern>`
+- Web client tests: `pnpm --filter web run test-client <file-or-pattern>`
+- Worker tests: `pnpm --filter worker run test <file-or-pattern>`
+
+---
+
+**Related Files:**
+
+- [../SKILL.md](../SKILL.md) - Main backend guidelines
+- [architecture-overview.md](architecture-overview.md) - Architecture patterns
+- [services-and-repositories.md](services-and-repositories.md) - Service and repository examples

.agents/skills/changelog-writing/SKILL.md

@@ -0,0 +1,48 @@
+---
+name: changelog-writing
+description: |
+  Shared workflow for writing Langfuse changelog entries after a feature is complete.
+  Use when a branch is ready for merge and a changelog entry or changelog draft is needed.
+---
+
+# Changelog Writing
+
+Use this skill when a completed feature branch needs a changelog entry.
+
+## Workflow
+
+1. Understand the change set.
+2. Study recent changelog patterns in `../langfuse-docs/pages/changelog`.
+3. Find related documentation links in `../langfuse-docs/pages`.
+4. Draft a user-focused changelog entry.
+5. Recommend whether an image or screenshot should be added.
+
+## What To Gather
+
+- The branch diff relative to `main`
+- The Linear issue, if the branch name includes an `lfe-XXXX` identifier
+- The affected product areas
+- Relevant docs pages to link or create
+
+## Writing Rules
+
+- Write for users, not internal implementation detail
+- Prefer second person: "you can now..."
+- Focus on what changed, why it matters, and how to use it
+- Match the structure and tone of recent changelog posts
+- Keep technical detail only where it improves user understanding
+
+## Output Format
+
+Provide:
+
+1. A short summary of what changed
+2. The complete changelog post content
+3. Whether an image should be added and what it should show
+4. Any docs pages that should be linked or created
+
+## Reference Files
+
+- Changelog destination: `../langfuse-docs/pages/changelog`
+- Recent changelog examples: inspect 3-5 recent files in that directory
+- Existing docs: `../langfuse-docs/pages`

.agents/skills/clickhouse-best-practices/README.md

@@ -0,0 +1,51 @@
+# ClickHouse Best Practices
+
+Agent skill providing comprehensive ClickHouse guidance for schema design, query optimization, and data ingestion.
+
+## Installation
+
+```bash
+npx skills add ClickHouse/clickhouse-agent-skills
+```
+
+## What's Included
+
+**28 atomic rules** organized by prefix:
+
+| Prefix               | Count | Coverage                                    |
+| -------------------- | ----- | ------------------------------------------- |
+| `schema-pk-*`        | 4     | PRIMARY KEY selection, cardinality ordering |
+| `schema-types-*`     | 5     | Data types, LowCardinality, Nullable        |
+| `schema-partition-*` | 4     | Partitioning strategy, lifecycle management |
+| `schema-json-*`      | 1     | JSON type usage                             |
+| `query-join-*`       | 5     | JOIN algorithms, filtering, alternatives    |
+| `query-index-*`      | 1     | Data skipping indices                       |
+| `query-mv-*`         | 2     | Incremental and refreshable MVs             |
+| `insert-batch-*`     | 1     | Batch sizing (10K-100K rows)                |
+| `insert-async-*`     | 2     | Async inserts, data formats                 |
+| `insert-mutation-*`  | 2     | Mutation avoidance                          |
+| `insert-optimize-*`  | 1     | OPTIMIZE FINAL avoidance                    |
+
+## Trigger Phrases
+
+This skill activates when you:
+
+- "Create a table for..."
+- "Optimize this query..."
+- "Design a schema for..."
+- "Why is this query slow?"
+- "How should I insert data into..."
+- "Should I use UPDATE or..."
+
+## Files
+
+| File         | Purpose                                                  |
+| ------------ | -------------------------------------------------------- |
+| `SKILL.md`   | Review workflow, quick reference, and rule-selection entrypoint |
+| `rules/*.md` | Individual rule definitions                              |
+
+## Related Documentation
+
+All rules link to official ClickHouse documentation:
+
+- [ClickHouse Best Practices](https://clickhouse.com/docs/best-practices)

.agents/skills/clickhouse-best-practices/SKILL.md

@@ -0,0 +1,241 @@
+---
+name: clickhouse-best-practices
+description: MUST USE when reviewing ClickHouse schemas, queries, or configurations. Contains 28 rules that MUST be checked before providing recommendations. Always read relevant rule files and cite specific rules in responses.
+license: Apache-2.0
+metadata:
+  author: ClickHouse Inc
+  version: "0.3.0"
+---
+
+# ClickHouse Best Practices
+
+Comprehensive guidance for ClickHouse covering schema design, query optimization, and data ingestion. Contains 28 rules across 3 main categories (schema, query, insert), prioritized by impact.
+
+> **Official docs:** [ClickHouse Best Practices](https://clickhouse.com/docs/best-practices)
+
+## IMPORTANT: How to Apply This Skill
+
+**Before answering ClickHouse questions, follow this priority order:**
+
+1. **Check for applicable rules** in the `rules/` directory
+2. **If rules exist:** Apply them and cite them in your response using "Per `rule-name`..."
+3. **If no rule exists:** Use the LLM's ClickHouse knowledge or search documentation
+4. **If uncertain:** Use web search for current best practices
+5. **Always cite your source:** rule name, "general ClickHouse guidance", or URL
+
+**Why rules take priority:** ClickHouse has specific behaviors (columnar storage, sparse indexes, merge tree mechanics) where general database intuition can be misleading. The rules encode validated, ClickHouse-specific guidance.
+
+## Langfuse-Specific Rules
+
+- Use `packages/shared/src/server/queries/clickhouse-sql/event-query-builder.ts`
+  for queries against the `events` table. Do not hand-roll `events` SQL unless
+  you first confirm the query builder cannot express the query.
+- Never use `FINAL` on the `events` table; it is designed so `FINAL` is not
+  required and the keyword hurts performance.
+- Any migration in `packages/shared/clickhouse/migrations/clustered/**` with
+  more than one `ALTER` on the same table must end every metadata `ALTER`
+  (`ADD/DROP/MODIFY COLUMN`, `ADD/DROP INDEX`) with `SETTINGS alter_sync = 2`,
+  and every mutation-creating `ALTER` (`MATERIALIZE …`, `UPDATE`, `DELETE`)
+  with `SETTINGS mutations_sync = 2`. The matching `unclustered/` file runs
+  against plain `MergeTree` and does not need (and should not duplicate)
+  these settings.
+
+---
+
+## Review Procedures
+
+### For Schema Reviews (CREATE TABLE, ALTER TABLE)
+
+**Read these rule files in order:**
+
+1. `rules/schema-pk-plan-before-creation.md` - ORDER BY is immutable
+2. `rules/schema-pk-cardinality-order.md` - Column ordering in keys
+3. `rules/schema-pk-prioritize-filters.md` - Filter column inclusion
+4. `rules/schema-types-native-types.md` - Proper type selection
+5. `rules/schema-types-minimize-bitwidth.md` - Numeric type sizing
+6. `rules/schema-types-lowcardinality.md` - LowCardinality usage
+7. `rules/schema-types-avoid-nullable.md` - Nullable vs DEFAULT
+8. `rules/schema-partition-low-cardinality.md` - Partition count limits
+9. `rules/schema-partition-lifecycle.md` - Partitioning purpose
+
+**Check for:**
+
+- [ ] PRIMARY KEY / ORDER BY column order (low-to-high cardinality)
+- [ ] Data types match actual data ranges
+- [ ] LowCardinality applied to appropriate string columns
+- [ ] Partition key cardinality bounded (100-1,000 values)
+- [ ] ReplacingMergeTree has version column if used
+- [ ] Clustered migration files with multiple ALTERs on the same table use `SETTINGS alter_sync = 2` (metadata) and `SETTINGS mutations_sync = 2` (`MATERIALIZE …`, `UPDATE`, `DELETE`); unclustered mirror has none
+
+### For Query Reviews (SELECT, JOIN, aggregations)
+
+**Read these rule files:**
+
+1. `rules/query-join-choose-algorithm.md` - Algorithm selection
+2. `rules/query-join-filter-before.md` - Pre-join filtering
+3. `rules/query-join-use-any.md` - ANY vs regular JOIN
+4. `rules/query-index-skipping-indices.md` - Secondary index usage
+5. `rules/schema-pk-filter-on-orderby.md` - Filter alignment with ORDER BY
+
+**Check for:**
+
+- [ ] Filters use ORDER BY prefix columns
+- [ ] JOINs filter tables before joining (not after)
+- [ ] Correct JOIN algorithm for table sizes
+- [ ] Skipping indices for non-ORDER BY filter columns
+
+### For Insert Strategy Reviews (data ingestion, updates, deletes)
+
+**Read these rule files:**
+
+1. `rules/insert-batch-size.md` - Batch sizing requirements
+2. `rules/insert-mutation-avoid-update.md` - UPDATE alternatives
+3. `rules/insert-mutation-avoid-delete.md` - DELETE alternatives
+4. `rules/insert-async-small-batches.md` - Async insert usage
+5. `rules/insert-optimize-avoid-final.md` - OPTIMIZE TABLE risks
+
+**Check for:**
+
+- [ ] Batch size 10K-100K rows per INSERT
+- [ ] No ALTER TABLE UPDATE for frequent changes
+- [ ] ReplacingMergeTree or CollapsingMergeTree for update patterns
+- [ ] Async inserts enabled for high-frequency small batches
+
+---
+
+## Output Format
+
+Structure your response as follows:
+
+```
+## Rules Checked
+- `rule-name-1` - Compliant / Violation found
+- `rule-name-2` - Compliant / Violation found
+...
+
+## Findings
+
+### Violations
+- **`rule-name`**: Description of the issue
+  - Current: [what the code does]
+  - Required: [what it should do]
+  - Fix: [specific correction]
+
+### Compliant
+- `rule-name`: Brief note on why it's correct
+
+## Recommendations
+[Prioritized list of changes, citing rules]
+```
+
+---
+
+## Rule Categories by Priority
+
+| Priority | Category              | Impact   | Prefix              | Rule Count |
+| -------- | --------------------- | -------- | ------------------- | ---------- |
+| 1        | Primary Key Selection | CRITICAL | `schema-pk-`        | 4          |
+| 2        | Data Type Selection   | CRITICAL | `schema-types-`     | 5          |
+| 3        | JOIN Optimization     | CRITICAL | `query-join-`       | 5          |
+| 4        | Insert Batching       | CRITICAL | `insert-batch-`     | 1          |
+| 5        | Mutation Avoidance    | CRITICAL | `insert-mutation-`  | 2          |
+| 6        | Partitioning Strategy | HIGH     | `schema-partition-` | 4          |
+| 7        | Skipping Indices      | HIGH     | `query-index-`      | 1          |
+| 8        | Materialized Views    | HIGH     | `query-mv-`         | 2          |
+| 9        | Async Inserts         | HIGH     | `insert-async-`     | 2          |
+| 10       | OPTIMIZE Avoidance    | HIGH     | `insert-optimize-`  | 1          |
+| 11       | JSON Usage            | MEDIUM   | `schema-json-`      | 1          |
+
+---
+
+## Quick Reference
+
+### Schema Design - Primary Key (CRITICAL)
+
+- `schema-pk-plan-before-creation` - Plan ORDER BY before table creation (immutable)
+- `schema-pk-cardinality-order` - Order columns low-to-high cardinality
+- `schema-pk-prioritize-filters` - Include frequently filtered columns
+- `schema-pk-filter-on-orderby` - Query filters must use ORDER BY prefix
+
+### Schema Design - Data Types (CRITICAL)
+
+- `schema-types-native-types` - Use native types, not String for everything
+- `schema-types-minimize-bitwidth` - Use smallest numeric type that fits
+- `schema-types-lowcardinality` - LowCardinality for <10K unique strings
+- `schema-types-enum` - Enum for finite value sets with validation
+- `schema-types-avoid-nullable` - Avoid Nullable; use DEFAULT instead
+
+### Schema Design - Partitioning (HIGH)
+
+- `schema-partition-low-cardinality` - Keep partition count 100-1,000
+- `schema-partition-lifecycle` - Use partitioning for data lifecycle, not queries
+- `schema-partition-query-tradeoffs` - Understand partition pruning trade-offs
+- `schema-partition-start-without` - Consider starting without partitioning
+
+### Schema Design - JSON (MEDIUM)
+
+- `schema-json-when-to-use` - JSON for dynamic schemas; typed columns for known
+
+### Query Optimization - JOINs (CRITICAL)
+
+- `query-join-choose-algorithm` - Select algorithm based on table sizes
+- `query-join-use-any` - ANY JOIN when only one match needed
+- `query-join-filter-before` - Filter tables before joining
+- `query-join-consider-alternatives` - Dictionaries/denormalization vs JOIN
+- `query-join-null-handling` - join_use_nulls=0 for default values
+
+### Query Optimization - Indices (HIGH)
+
+- `query-index-skipping-indices` - Skipping indices for non-ORDER BY filters
+
+### Query Optimization - Materialized Views (HIGH)
+
+- `query-mv-incremental` - Incremental MVs for real-time aggregations
+- `query-mv-refreshable` - Refreshable MVs for complex joins
+
+### Insert Strategy - Batching (CRITICAL)
+
+- `insert-batch-size` - Batch 10K-100K rows per INSERT
+
+### Insert Strategy - Async (HIGH)
+
+- `insert-async-small-batches` - Async inserts for high-frequency small batches
+- `insert-format-native` - Native format for best performance
+
+### Insert Strategy - Mutations (CRITICAL)
+
+- `insert-mutation-avoid-update` - ReplacingMergeTree instead of ALTER UPDATE
+- `insert-mutation-avoid-delete` - Lightweight DELETE or DROP PARTITION
+
+### Insert Strategy - Optimization (HIGH)
+
+- `insert-optimize-avoid-final` - Let background merges work
+
+---
+
+## When to Apply
+
+This skill activates when you encounter:
+
+- `CREATE TABLE` statements
+- `ALTER TABLE` modifications
+- `ORDER BY` or `PRIMARY KEY` discussions
+- Data type selection questions
+- Slow query troubleshooting
+- JOIN optimization requests
+- Data ingestion pipeline design
+- Update/delete strategy questions
+- ReplacingMergeTree or other specialized engine usage
+- Partitioning strategy decisions
+
+---
+
+## Rule File Structure
+
+Each rule file in `rules/` contains:
+
+- **YAML frontmatter**: title, impact level, tags
+- **Brief explanation**: Why this rule matters
+- **Incorrect example**: Anti-pattern with explanation
+- **Correct example**: Best practice with explanation
+- **Additional context**: Trade-offs, when to apply, references

.agents/skills/clickhouse-best-practices/rules/_sections.md

@@ -0,0 +1,24 @@
+# Sections
+
+This file defines all sections, their ordering, impact levels, and descriptions.
+The section ID (in parentheses) is the filename prefix used to group rules.
+
+---
+
+## 1. Schema Design (schema)
+
+**Impact:** CRITICAL
+
+**Description:** Proper schema design is foundational to ClickHouse performance. ORDER BY is immutable after table creation; wrong choices require full data migration. Includes primary key selection, data types, partitioning strategy, and JSON usage. Column types and ordering can impact query speed by orders of magnitude.
+
+## 2. Query Optimization (query)
+
+**Impact:** CRITICAL
+
+**Description:** Query patterns dramatically affect performance. JOIN algorithms, filtering strategies, skipping indices, and materialized views can reduce query time from minutes to milliseconds. Pre-computed aggregations read thousands of rows instead of billions.
+
+## 3. Insert Strategy (insert)
+
+**Impact:** CRITICAL
+
+**Description:** Each INSERT creates a data part. Single-row inserts overwhelm the merge process. Proper batching (10K-100K rows), async inserts for high-frequency writes, mutation avoidance, and letting background merges work are essential for stable cluster performance.

.agents/skills/clickhouse-best-practices/rules/_template.md

@@ -0,0 +1,28 @@
+---
+title: Rule Title Here
+impact: CRITICAL | HIGH | MEDIUM | LOW
+impactDescription: "Quantified improvement (e.g., 10x faster queries)"
+tags: [tag1, tag2]
+---
+
+## Rule Title Here
+
+**Impact: CRITICAL** (optional description)
+
+Brief explanation of the rule and why it matters. This should be clear and concise, explaining the performance implications.
+
+**Incorrect (description of what's wrong):**
+
+```sql
+-- Bad: description
+SELECT * FROM table;
+```
+
+**Correct (description of what's right):**
+
+```sql
+-- Good: description
+SELECT * FROM table;
+```
+
+Reference: [Official Docs](https://clickhouse.com/docs/best-practices/...)

.agents/skills/clickhouse-best-practices/rules/insert-async-small-batches.md

@@ -0,0 +1,55 @@
+---
+title: Use Async Inserts for High-Frequency Small Batches
+impact: HIGH
+impactDescription: "Server-side buffering when client batching isn't practical"
+tags: [insert, async, buffering, small-batches]
+---
+
+## Use Async Inserts for High-Frequency Small Batches
+
+**Impact: HIGH**
+
+When client-side batching isn't practical, async inserts buffer server-side and create larger parts automatically.
+
+**Incorrect (small batches without async):**
+
+```python
+# Small batches without async_insert - creates too many parts
+for batch in chunks(events, 100):
+    client.execute("INSERT INTO events VALUES", batch)
+```
+
+**Correct (enable async inserts):**
+
+```python
+# Enable async_insert with safe defaults
+client.execute("SET async_insert = 1")
+client.execute("SET wait_for_async_insert = 1")  # Confirms durability
+
+for batch in chunks(events, 100):
+    client.execute("INSERT INTO events VALUES", batch)
+# Server buffers and creates larger parts automatically
+```
+
+```sql
+-- Configure server-side for specific users
+ALTER USER my_app_user SETTINGS
+    async_insert = 1,
+    wait_for_async_insert = 1,
+    async_insert_max_data_size = 10000000,  -- Flush at 10MB
+    async_insert_busy_timeout_ms = 1000;    -- Flush after 1s
+```
+
+**Flush conditions (whichever occurs first):**
+- Buffer reaches `async_insert_max_data_size`
+- Time threshold `async_insert_busy_timeout_ms` elapses
+- Maximum insert queries accumulate
+
+**Return modes:**
+
+| Setting | Behavior | Use Case |
+|---------|----------|----------|
+| `wait_for_async_insert=1` | Waits for flush, confirms durability | **Recommended** |
+| `wait_for_async_insert=0` | Fire-and-forget, unaware of errors | **Risky** - only if you accept data loss |
+
+Reference: [Selecting an Insert Strategy](https://clickhouse.com/docs/best-practices/selecting-an-insert-strategy)

.agents/skills/clickhouse-best-practices/rules/insert-batch-size.md

@@ -0,0 +1,54 @@
+---
+title: Batch Inserts Appropriately (10K-100K rows)
+impact: CRITICAL
+impactDescription: "Each INSERT creates a part; single-row inserts overwhelm merge process"
+tags: [insert, batching, parts, performance]
+---
+
+## Batch Inserts Appropriately (10K-100K rows)
+
+**Impact: CRITICAL**
+
+Each INSERT creates a new data part. Single-row or small-batch inserts create thousands of tiny parts, overwhelming the merge process and causing cluster instability.
+
+**Incorrect (single-row or tiny batches):**
+
+```python
+# Single-row inserts - creates 10,000 parts!
+for event in events:
+    client.execute("INSERT INTO events VALUES", [event])
+
+# Tiny batches - still too many parts
+for batch in chunks(events, 100):  # 100 rows per INSERT
+    client.execute("INSERT INTO events VALUES", batch)
+```
+
+**Correct (proper batch size):**
+
+```python
+# Ideal batch size: 10,000-100,000 rows
+BATCH_SIZE = 10_000
+for batch in chunks(events, BATCH_SIZE):
+    client.execute("INSERT INTO events VALUES", batch)
+```
+
+**Recommended batch sizes:**
+
+| Threshold | Value |
+|-----------|-------|
+| Minimum | 1,000 rows |
+| Ideal range | 10,000-100,000 rows |
+| Insert rate (sync) | ~1 insert per second |
+
+**Validation:**
+
+```sql
+-- Monitor part count (>3000 per partition blocks inserts)
+SELECT table, count() as parts, sum(rows) as total_rows
+FROM system.parts
+WHERE active AND database = 'default'
+GROUP BY table
+ORDER BY parts DESC;
+```
+
+Reference: [Selecting an Insert Strategy](https://clickhouse.com/docs/best-practices/selecting-an-insert-strategy)

.agents/skills/clickhouse-best-practices/rules/insert-format-native.md

@@ -0,0 +1,29 @@
+---
+title: Use Native Format for Best Insert Performance
+impact: MEDIUM
+impactDescription: "Native format is most efficient; JSONEachRow is expensive to parse"
+tags: [insert, format, Native, performance]
+---
+
+## Use Native Format for Best Insert Performance
+
+**Impact: MEDIUM**
+
+Data format affects insert performance. Native format is column-oriented with minimal parsing overhead.
+
+**Performance Ranking (fastest to slowest):**
+
+| Format | Notes |
+|--------|-------|
+| **Native** | Most efficient. Column-oriented, minimal parsing. Recommended. |
+| **RowBinary** | Efficient row-based alternative |
+| **JSONEachRow** | Easier to use but expensive to parse |
+
+**Example:**
+
+```python
+# Use Native format for best performance
+client.execute("INSERT INTO events VALUES", data, settings={'input_format': 'Native'})
+```
+
+Reference: [Selecting an Insert Strategy](https://clickhouse.com/docs/best-practices/selecting-an-insert-strategy)

.agents/skills/clickhouse-best-practices/rules/insert-mutation-avoid-delete.md

@@ -0,0 +1,74 @@
+---
+title: Avoid ALTER TABLE DELETE
+impact: CRITICAL
+impactDescription: "Use lightweight DELETE, CollapsingMergeTree, or DROP PARTITION instead"
+tags: [insert, mutation, DELETE, CollapsingMergeTree]
+---
+
+## Avoid ALTER TABLE DELETE
+
+**Impact: CRITICAL**
+
+`ALTER TABLE DELETE` is a mutation that rewrites entire data parts. Use alternatives like lightweight DELETE, CollapsingMergeTree, or DROP PARTITION.
+
+**Incorrect (mutation delete):**
+
+```sql
+-- Mutation delete for cleanup
+ALTER TABLE orders DELETE WHERE status = 'cancelled';
+
+-- Time-based cleanup via mutation (very expensive)
+ALTER TABLE sessions DELETE WHERE created_at < now() - INTERVAL 7 DAY;
+```
+
+**Correct - CollapsingMergeTree:**
+
+```sql
+CREATE TABLE orders (
+    order_id UInt64,
+    customer_id UInt64,
+    total Decimal(10,2),
+    sign Int8  -- 1 = active, -1 = deleted
+)
+ENGINE = CollapsingMergeTree(sign)
+ORDER BY order_id;
+
+-- Insert order
+INSERT INTO orders VALUES (123, 456, 99.99, 1);
+
+-- "Delete" by inserting with sign = -1
+INSERT INTO orders VALUES (123, 456, 99.99, -1);
+
+-- Query collapses +1 and -1 pairs
+SELECT order_id, sum(total * sign) as total
+FROM orders GROUP BY order_id HAVING sum(sign) > 0;
+```
+
+**Correct - Lightweight Deletes (23.3+):**
+
+```sql
+-- Marks rows, doesn't rewrite immediately
+DELETE FROM orders WHERE status = 'cancelled';
+-- Physical deletion happens during normal merges
+```
+
+**Correct - DROP PARTITION for Bulk Deletion:**
+
+```sql
+-- Instant deletion of old data
+ALTER TABLE events DROP PARTITION '202301';
+
+-- Much faster than:
+ALTER TABLE events DELETE WHERE toYYYYMM(timestamp) = 202301;
+```
+
+**Delete strategy comparison:**
+
+| Method | Speed | When to Use |
+|--------|-------|-------------|
+| ALTER DELETE | Slow | Rare corrections only |
+| CollapsingMergeTree | Fast | Frequent soft deletes |
+| Lightweight DELETE | Medium | Occasional deletes |
+| DROP PARTITION | Instant | Bulk deletion by partition |
+
+Reference: [Avoid Mutations](https://clickhouse.com/docs/best-practices/avoid-mutations)

.agents/skills/clickhouse-best-practices/rules/insert-mutation-avoid-update.md

@@ -0,0 +1,58 @@
+---
+title: Avoid ALTER TABLE UPDATE
+impact: CRITICAL
+impactDescription: "Mutations rewrite entire parts; use ReplacingMergeTree instead"
+tags: [insert, mutation, UPDATE, ReplacingMergeTree]
+---
+
+## Avoid ALTER TABLE UPDATE
+
+**Impact: CRITICAL**
+
+`ALTER TABLE UPDATE` is a mutation - an asynchronous background process that rewrites entire data parts affected by the change. This is extremely expensive for frequent or large-scale operations.
+
+**Why mutations are problematic:**
+- **Write amplification:** Rewrite complete parts even for minor changes
+- **Disk I/O spike:** Degrades overall cluster performance
+- **No rollback:** Cannot be rolled back after submission
+- **Inconsistent reads:** SELECT may read mix of mutated and unmutated parts
+
+**Incorrect (mutation for updates):**
+
+```sql
+-- Rewrites potentially huge amounts of data
+ALTER TABLE users UPDATE status = 'inactive'
+WHERE last_login < now() - INTERVAL 90 DAY;
+
+-- Frequent row updates via mutation
+ALTER TABLE inventory UPDATE quantity = quantity - 1
+WHERE product_id = 123;
+-- If product exists across 100 parts, rewrites ALL 100 parts
+```
+
+**Correct (ReplacingMergeTree):**
+
+```sql
+-- Table design for updates
+CREATE TABLE users (
+    user_id UInt64,
+    name String,
+    status LowCardinality(String),
+    updated_at DateTime DEFAULT now()
+)
+ENGINE = ReplacingMergeTree(updated_at)
+ORDER BY user_id;
+
+-- "Update" by inserting new version
+INSERT INTO users (user_id, name, status)
+VALUES (123, 'John', 'inactive');
+
+-- Query with FINAL to get latest version
+SELECT * FROM users FINAL WHERE user_id = 123;
+
+-- Or use aggregation
+SELECT user_id, argMax(status, updated_at) as status
+FROM users GROUP BY user_id;
+```
+
+Reference: [Avoid Mutations](https://clickhouse.com/docs/best-practices/avoid-mutations)

.agents/skills/clickhouse-best-practices/rules/insert-optimize-avoid-final.md

@@ -0,0 +1,57 @@
+---
+title: Avoid OPTIMIZE TABLE FINAL
+impact: HIGH
+impactDescription: "Forces expensive merge of all parts; let background merges work"
+tags: [insert, OPTIMIZE, merge, performance]
+---
+
+## Avoid OPTIMIZE TABLE FINAL
+
+**Impact: HIGH**
+
+`OPTIMIZE TABLE ... FINAL` forces immediate merge of all parts into one part per partition. This is resource-intensive and rarely necessary. ClickHouse already performs smart background merges.
+
+**Note:** `OPTIMIZE FINAL` is not the same as `FINAL`. The `FINAL` modifier in SELECT queries may be necessary for deduplicated results in ReplacingMergeTree and is generally fine to use.
+
+**Incorrect (OPTIMIZE FINAL after inserts):**
+
+```sql
+-- Running OPTIMIZE FINAL after every batch insert
+INSERT INTO events SELECT * FROM staging_events;
+OPTIMIZE TABLE events FINAL;  -- Expensive and unnecessary!
+
+-- Scheduled OPTIMIZE FINAL jobs
+-- Cron: 0 * * * * clickhouse-client -q "OPTIMIZE TABLE events FINAL"
+```
+
+**Correct (let background merges work):**
+
+```sql
+-- Let background merges handle optimization
+INSERT INTO events SELECT * FROM staging_events;
+-- Done! ClickHouse merges automatically
+
+-- For ReplacingMergeTree deduplication, use FINAL in queries
+SELECT * FROM events FINAL WHERE user_id = 123;
+-- Instead of running OPTIMIZE FINAL to deduplicate
+```
+
+**Problems with OPTIMIZE FINAL:**
+- Rewrites entire partition regardless of need
+- Ignores the ~150 GB part size safeguard
+- Can cause memory pressure or OOM errors
+- Lengthy execution time for large datasets
+
+**When OPTIMIZE FINAL may be acceptable:**
+- Finalizing data before table freezing
+- Preparing data for export operations
+- One-time operations, not regular workflows
+
+**Better alternatives:**
+
+| Need | Alternative |
+|------|-------------|
+| Deduplicate ReplacingMergeTree | Use `FINAL` modifier in SELECT |
+| Reduce part count | Rely on background merges |
+
+Reference: [Avoid OPTIMIZE FINAL](https://clickhouse.com/docs/best-practices/avoid-optimize-final)

.agents/skills/clickhouse-best-practices/rules/query-index-skipping-indices.md

@@ -0,0 +1,77 @@
+---
+title: Use Data Skipping Indices for Non-ORDER BY Filters
+impact: HIGH
+impactDescription: "Up to 60x faster queries by skipping irrelevant granules"
+tags: [query, index, skipping, bloom_filter]
+---
+
+## Use Data Skipping Indices for Non-ORDER BY Filters
+
+**Impact: HIGH**
+
+Queries filtering on columns not in ORDER BY cannot use the primary index and result in full scans. Data skipping indices store metadata about blocks and skip granules that definitely don't match.
+
+**Important:** Skip indices should be considered **after** optimizing data types, primary key selection, and materialized views.
+
+**When to use:**
+- High overall cardinality but low cardinality within blocks
+- Rare values critical for search (error codes, specific IDs)
+- Column correlates with primary key
+
+**When NOT to use:**
+- As a first optimization step
+- Matching values scattered across many blocks
+- Without testing on real data
+
+**Incorrect (filtering on non-ORDER BY column):**
+
+```sql
+CREATE TABLE events (
+    event_type LowCardinality(String),
+    timestamp DateTime,
+    user_id UInt64    -- Not in ORDER BY
+)
+ENGINE = MergeTree()
+ORDER BY (event_type, toDate(timestamp));
+
+-- Query filters on user_id - scans all matching event_type
+SELECT * FROM events
+WHERE event_type = 'click' AND user_id = 12345;
+```
+
+**Correct (add skipping index):**
+
+```sql
+CREATE TABLE events (
+    event_type LowCardinality(String),
+    timestamp DateTime,
+    user_id UInt64,
+    INDEX idx_user_id user_id TYPE bloom_filter GRANULARITY 4
+)
+ENGINE = MergeTree()
+ORDER BY (event_type, toDate(timestamp));
+
+-- Or add to existing table
+ALTER TABLE events ADD INDEX idx_user_id user_id TYPE bloom_filter GRANULARITY 4;
+ALTER TABLE events MATERIALIZE INDEX idx_user_id;
+```
+
+**Index types:**
+
+| Type | Best For | Example Filter |
+|------|----------|----------------|
+| `bloom_filter` | Equality on high-cardinality | `WHERE user_id = 123` |
+| `set(N)` | Low cardinality (N unique values) | `WHERE status IN ('a','b')` |
+| `minmax` | Range queries | `WHERE amount > 1000` |
+| `ngrambf_v1` | Text search | `WHERE text LIKE '%term%'` |
+| `tokenbf_v1` | Token search | `WHERE hasToken(text, 'word')` |
+
+**Validation:**
+
+```sql
+EXPLAIN indexes = 1
+SELECT * FROM events WHERE user_id = 12345;
+-- Look for "Skip" in output showing granules skipped
+```
+
+Reference: [Use Data Skipping Indices Where Appropriate](https://clickhouse.com/docs/best-practices/use-data-skipping-indices-where-appropriate)

.agents/skills/clickhouse-best-practices/rules/query-join-choose-algorithm.md

@@ -0,0 +1,43 @@
+---
+title: Choose the Right JOIN Algorithm
+impact: CRITICAL
+impactDescription: "Wrong algorithm causes OOM; right algorithm handles large tables efficiently"
+tags: [query, JOIN, algorithm, memory]
+---
+
+## Choose the Right JOIN Algorithm
+
+**Impact: CRITICAL**
+
+ClickHouse's default hash join loads the RIGHT table entirely into memory. Choose the right algorithm based on table sizes and constraints.
+
+**Algorithm selection:**
+
+| Algorithm | Best For | Trade-off |
+|-----------|----------|-----------|
+| `parallel_hash` | Small-to-medium in-memory tables | Default since 24.11; fast, concurrent |
+| `hash` | General purpose, all join types | Single-threaded hash table build |
+| `direct` | Dictionary lookups (INNER/LEFT only) | Fastest; no hash table construction |
+| `full_sorting_merge` | Tables already sorted on join key | Skips sort if pre-ordered; low memory |
+| `partial_merge` | Large tables, memory-constrained | Minimized memory; slower execution |
+| `grace_hash` | Large datasets, tunable memory | Flexible; disk-spilling capability |
+| `auto` | Adaptive algorithm selection | Tries hash first, falls back on memory pressure |
+
+**Example usage:**
+
+```sql
+-- Let ClickHouse choose automatically
+SET join_algorithm = 'auto';
+
+-- For large-to-large joins where memory is constrained
+SET join_algorithm = 'partial_merge';
+SELECT * FROM large_a JOIN large_b ON large_b.id = large_a.id;
+
+-- When joining by primary key columns, sort-merge skips sorting step
+SET join_algorithm = 'full_sorting_merge';
+SELECT * FROM table_a a JOIN table_b b ON b.pk_col = a.pk_col;
+```
+
+**Note:** ClickHouse 24.12+ automatically positions smaller tables on the right side. For earlier versions, manually ensure the smaller table is on the RIGHT.
+
+Reference: [Minimize and Optimize JOINs](https://clickhouse.com/docs/best-practices/minimize-optimize-joins)

.agents/skills/clickhouse-best-practices/rules/query-join-consider-alternatives.md

@@ -0,0 +1,72 @@
+---
+title: Consider Alternatives to JOINs
+impact: CRITICAL
+impactDescription: "Dictionaries and denormalization shift work from query time to insert time"
+tags: [query, JOIN, dictionary, denormalization]
+---
+
+## Consider Alternatives to JOINs
+
+**Impact: CRITICAL**
+
+Repeated JOINs to dimension tables add overhead. Dictionaries or denormalization shift computational work from query time to insert/pre-processing time.
+
+**Incorrect (JOIN on every query):**
+
+```sql
+-- JOIN on every query
+SELECT o.order_id, c.name, c.email
+FROM orders o
+JOIN customers c ON c.id = o.customer_id
+WHERE o.created_at > '2024-01-01';
+```
+
+**Correct - Dictionary Lookup:**
+
+```sql
+-- Create dictionary
+CREATE DICTIONARY customer_dict (
+    id UInt64,
+    name String,
+    email String
+)
+PRIMARY KEY id
+SOURCE(CLICKHOUSE(TABLE 'customers'))
+LAYOUT(HASHED())
+LIFETIME(MIN 300 MAX 360);
+
+-- Use dictGet instead of JOIN (uses direct join algorithm - fastest)
+SELECT
+    order_id,
+    dictGet('customer_dict', 'name', customer_id) as customer_name,
+    dictGet('customer_dict', 'email', customer_id) as customer_email
+FROM orders
+WHERE created_at > '2024-01-01';
+```
+
+**Correct - Denormalization:**
+
+```sql
+-- Denormalized table with materialized view
+CREATE MATERIALIZED VIEW orders_enriched_mv TO orders_enriched AS
+SELECT
+    o.order_id, o.customer_id,
+    c.name as customer_name,
+    c.email as customer_email,
+    o.total, o.created_at
+FROM orders o
+JOIN customers c ON c.id = o.customer_id;
+```
+
+**Approach comparison:**
+
+| Approach | Use Case | Performance |
+|----------|----------|-------------|
+| Dictionary | Frequent lookups to small dimension | Fastest (in-memory) |
+| Denormalization | Analytics always need enriched data | Fast (no join at query) |
+| IN subquery | Existence filtering | Often faster than JOIN |
+| JOIN | Infrequent or complex joins | Acceptable |
+
+**Critical dictionary caveat:** Dictionaries silently deduplicate duplicate keys, retaining only the final value. Only use when source has unique keys.
+
+Reference: [Minimize and Optimize JOINs](https://clickhouse.com/docs/best-practices/minimize-optimize-joins)

.agents/skills/clickhouse-best-practices/rules/query-join-filter-before.md

@@ -0,0 +1,54 @@
+---
+title: Filter Tables Before Joining
+impact: CRITICAL
+impactDescription: "Joining full tables then filtering wastes resources"
+tags: [query, JOIN, filtering, subquery]
+---
+
+## Filter Tables Before Joining
+
+**Impact: CRITICAL**
+
+Joining full tables then filtering wastes resources. Add filtering in `WHERE` or `JOIN ON` clauses. If automatic pushdown fails, restructure as a subquery.
+
+**Incorrect (join then filter):**
+
+```sql
+-- Joins entire tables, then filters
+SELECT o.order_id, c.name, o.total
+FROM orders o
+JOIN customers c ON c.id = o.customer_id
+WHERE o.created_at > '2024-01-01' AND c.country = 'US';
+```
+
+**Correct (filter in subqueries before joining):**
+
+```sql
+-- Filter in subqueries before joining
+SELECT o.order_id, c.name, o.total
+FROM (
+    SELECT order_id, customer_id, total
+    FROM orders
+    WHERE created_at > '2024-01-01'
+) o
+JOIN (
+    SELECT id, name
+    FROM customers
+    WHERE country = 'US'
+) c ON c.id = o.customer_id;
+```
+
+**Even better - aggregate before joining:**
+
+```sql
+SELECT c.country, o.total_revenue
+FROM (
+    SELECT customer_id, sum(total) as total_revenue
+    FROM orders
+    WHERE created_at > '2024-01-01'
+    GROUP BY customer_id
+) o
+JOIN customers c ON c.id = o.customer_id;
+```
+
+Reference: [Minimize and Optimize JOINs](https://clickhouse.com/docs/best-practices/minimize-optimize-joins)

.agents/skills/clickhouse-best-practices/rules/query-join-null-handling.md

@@ -0,0 +1,33 @@
+---
+title: Optimize NULL Handling in Outer JOINs
+impact: MEDIUM
+impactDescription: "Default values instead of NULL reduces memory overhead"
+tags: [query, JOIN, NULL, memory]
+---
+
+## Optimize NULL Handling in Outer JOINs
+
+**Impact: MEDIUM**
+
+Set `join_use_nulls = 0` to use default column values instead of NULL markers, reducing memory overhead compared to Nullable wrappers.
+
+**Example:**
+
+```sql
+-- Use default values instead of NULLs for non-matching rows
+SET join_use_nulls = 0;
+
+SELECT o.order_id, c.name
+FROM orders o
+LEFT JOIN customers c ON c.id = o.customer_id;
+-- Non-matching rows get '' for name instead of NULL
+```
+
+**When to use:**
+
+| Setting | Behavior | Use Case |
+|---------|----------|----------|
+| `join_use_nulls = 0` | Default values (empty string, 0) for non-matches | When you can handle default values |
+| `join_use_nulls = 1` (default) | NULL for non-matches | When you need to distinguish "no match" from "matched with default" |
+
+Reference: [Minimize and Optimize JOINs](https://clickhouse.com/docs/best-practices/minimize-optimize-joins)

.agents/skills/clickhouse-best-practices/rules/query-join-use-any.md

@@ -0,0 +1,40 @@
+---
+title: Use ANY JOIN When Only One Match Needed
+impact: HIGH
+impactDescription: "Returns first match only; less memory and faster execution"
+tags: [query, JOIN, ANY, performance]
+---
+
+## Use ANY JOIN When Only One Match Needed
+
+**Impact: HIGH**
+
+Use `ANY` JOINs when you only need a single match rather than all matches. They consume less memory and execute faster.
+
+**Incorrect (returns all matches):**
+
+```sql
+-- Returns all matching rows, uses more memory
+SELECT o.order_id, c.name
+FROM orders o
+LEFT JOIN customers c ON c.id = o.customer_id;
+```
+
+**Correct (returns first match only):**
+
+```sql
+-- Returns only first match per row, faster and less memory
+SELECT o.order_id, c.name
+FROM orders o
+LEFT ANY JOIN customers c ON c.id = o.customer_id;
+```
+
+**ANY JOIN types:**
+
+| Type | Behavior |
+|------|----------|
+| `LEFT ANY JOIN` | At most one match from right table |
+| `INNER ANY JOIN` | At most one match, only matching rows |
+| `RIGHT ANY JOIN` | At most one match from left table |
+
+Reference: [Minimize and Optimize JOINs](https://clickhouse.com/docs/best-practices/minimize-optimize-joins)

.agents/skills/clickhouse-best-practices/rules/query-mv-incremental.md

@@ -0,0 +1,68 @@
+---
+title: Use Incremental MVs for Real-Time Aggregations
+impact: HIGH
+impactDescription: "Read thousands of rows instead of billions; minimal cluster overhead"
+tags: [query, materialized-view, aggregation, real-time]
+---
+
+## Use Incremental MVs for Real-Time Aggregations
+
+**Impact: HIGH**
+
+Incremental MVs automatically apply the view's query to new data blocks at insert time. Results are written to a target table and partial results merge over time.
+
+**Incorrect (full aggregation on every query):**
+
+```sql
+-- Full aggregation on every dashboard load
+SELECT
+    event_type,
+    toStartOfHour(timestamp) as hour,
+    count() as events,
+    uniq(user_id) as unique_users
+FROM events
+WHERE timestamp >= now() - INTERVAL 7 DAY
+GROUP BY event_type, hour;
+-- Scans 7 days of data every time (billions of rows)
+```
+
+**Correct (incremental MV with pre-aggregation):**
+
+```sql
+-- Create target table for aggregated data
+CREATE TABLE events_hourly (
+    event_type LowCardinality(String),
+    hour DateTime,
+    events AggregateFunction(count),
+    unique_users AggregateFunction(uniq, UInt64)
+)
+ENGINE = AggregatingMergeTree()
+ORDER BY (event_type, hour);
+
+-- Create materialized view to populate incrementally
+CREATE MATERIALIZED VIEW events_hourly_mv TO events_hourly AS
+SELECT
+    event_type,
+    toStartOfHour(timestamp) as hour,
+    countState() as events,
+    uniqState(user_id) as unique_users
+FROM events
+GROUP BY event_type, hour;
+
+-- Query the pre-aggregated data
+SELECT
+    event_type, hour,
+    countMerge(events) as events,
+    uniqMerge(unique_users) as unique_users
+FROM events_hourly
+WHERE hour >= now() - INTERVAL 7 DAY
+GROUP BY event_type, hour;
+-- Reads thousands of rows instead of billions
+```
+
+**Key points:**
+- Use `-State` functions in MV, `-Merge` functions in query
+- Incremental - existing data not automatically included (backfill separately)
+- Minimal cluster overhead at insert time
+
+Reference: [Use Materialized Views](https://clickhouse.com/docs/best-practices/use-materialized-views)

.agents/skills/clickhouse-best-practices/rules/query-mv-refreshable.md

@@ -0,0 +1,64 @@
+---
+title: Use Refreshable MVs for Complex Joins and Batch Workflows
+impact: HIGH
+impactDescription: "Sub-millisecond queries with periodic refresh; ideal for complex joins"
+tags: [query, materialized-view, refresh, batch]
+---
+
+## Use Refreshable MVs for Complex Joins and Batch Workflows
+
+**Impact: HIGH**
+
+Refreshable MVs execute queries periodically on a schedule. The full query re-executes and overwrites (or appends to) the target table.
+
+**Best for:**
+- Sub-millisecond latency where minor staleness is acceptable
+- Caching "top N" results or lookup tables
+- Complex multi-table joins requiring denormalization
+- Batch workflows and DAG dependencies
+
+**Incorrect (expensive join on every request):**
+
+```sql
+-- Complex join executed on every request
+SELECT
+    o.order_id, o.total,
+    c.name as customer_name,
+    p.name as product_name
+FROM orders o
+JOIN customers c ON o.customer_id = c.id
+JOIN products p ON o.product_id = p.id
+WHERE o.created_at >= now() - INTERVAL 1 DAY;
+```
+
+**Correct (refreshable MV):**
+
+```sql
+-- Create refreshable MV that runs every 5 minutes
+CREATE MATERIALIZED VIEW orders_denormalized
+REFRESH EVERY 5 MINUTE
+ENGINE = MergeTree()
+ORDER BY (created_at, order_id)
+AS SELECT
+    o.order_id, o.created_at, o.total,
+    c.name as customer_name, c.segment,
+    p.name as product_name
+FROM orders o
+JOIN customers c ON o.customer_id = c.id
+JOIN products p ON o.product_id = p.id
+WHERE o.created_at >= now() - INTERVAL 1 DAY;
+
+-- Query the pre-joined data (sub-millisecond)
+SELECT * FROM orders_denormalized WHERE segment = 'enterprise';
+```
+
+**APPEND vs REPLACE modes:**
+
+| Mode | Behavior | Use Case |
+|------|----------|----------|
+| `REPLACE` (default) | Overwrites previous contents | Current state, lookup tables |
+| `APPEND` | Adds new rows to existing data | Periodic snapshots, historical accumulation |
+
+**Critical warning:** Query should run quickly compared to refresh interval. Don't schedule every 10 seconds if the query takes 10+ seconds.
+
+Reference: [Use Materialized Views](https://clickhouse.com/docs/best-practices/use-materialized-views)

.agents/skills/clickhouse-best-practices/rules/schema-json-when-to-use.md

@@ -0,0 +1,76 @@
+---
+title: Use JSON Type for Dynamic Schemas
+impact: MEDIUM
+impactDescription: "Field-level querying for semi-structured data; use typed columns for known schemas"
+tags: [schema, JSON, semi-structured, flexibility]
+---
+
+## Use JSON Type for Dynamic Schemas
+
+**Impact: MEDIUM**
+
+ClickHouse's JSON type splits JSON objects into separate sub-columns, enabling field-level query optimization. Use it for truly dynamic data, not everything.
+
+**Incorrect (schema bloat or opaque String):**
+
+```sql
+-- BAD: Hundreds of nullable columns for event properties
+CREATE TABLE events (
+    event_id UUID,
+    prop_page_url Nullable(String),
+    prop_button_id Nullable(String),
+    -- ... 100 more nullable columns
+)
+
+-- BAD: JSON as String when you need field queries
+CREATE TABLE events (
+    event_id UUID,
+    properties String  -- No field-level optimization
+)
+```
+
+**Correct (JSON for dynamic, typed for known):**
+
+```sql
+-- Use JSON type for dynamic properties
+CREATE TABLE events (
+    event_id UUID DEFAULT generateUUIDv4(),
+    event_type LowCardinality(String),
+    timestamp DateTime DEFAULT now(),
+    properties JSON  -- Flexible schema with type inference
+)
+ENGINE = MergeTree()
+ORDER BY (event_type, timestamp);
+
+-- Query JSON paths directly
+SELECT
+    event_type,
+    properties.url as page_url,
+    properties.amount as purchase_amount
+FROM events
+WHERE event_type = 'page_view' AND properties.url = '/home';
+```
+
+**When to use JSON:**
+
+| Scenario | Use JSON? |
+|----------|-----------|
+| Data structure varies unpredictably | Yes |
+| Field types/schemas change over time | Yes |
+| Need field-level querying | Yes |
+| Fixed, known schema | No (use typed columns) |
+| JSON as opaque blob (no field queries) | No (use String) |
+
+**Optimization: specify types for known paths:**
+
+```sql
+CREATE TABLE events (
+    properties JSON(
+        url String,
+        amount Float64,
+        product_id UInt64
+    )
+)
+```
+
+Reference: [Use JSON Where Appropriate](https://clickhouse.com/docs/best-practices/use-json-where-appropriate)

.agents/skills/clickhouse-best-practices/rules/schema-partition-lifecycle.md

@@ -0,0 +1,50 @@
+---
+title: Use Partitioning for Data Lifecycle Management
+impact: HIGH
+impactDescription: "DROP PARTITION is instant; DELETE is expensive row-by-row scan"
+tags: [schema, partitioning, TTL, data-management]
+---
+
+## Use Partitioning for Data Lifecycle Management
+
+**Impact: HIGH**
+
+Partitioning is **primarily a data management technique, not a query optimization tool**. It excels at:
+- **Dropping data**: Remove entire partitions as single metadata operations
+- **TTL retention**: Implement time-based retention policies efficiently
+- **Tiered storage**: Move old partitions to cold storage
+- **Archiving**: Move partitions between tables
+
+**Incorrect (no time alignment for lifecycle):**
+
+```sql
+-- Cannot efficiently drop old data by time
+CREATE TABLE events (...)
+ENGINE = MergeTree()
+PARTITION BY event_type  -- No time alignment
+ORDER BY (timestamp);
+
+-- Slow: must scan and delete row by row
+DELETE FROM events WHERE timestamp < '2023-01-01';
+```
+
+**Correct (time-based for lifecycle):**
+
+```sql
+CREATE TABLE events (
+    timestamp DateTime,
+    event_type LowCardinality(String)
+)
+ENGINE = MergeTree()
+PARTITION BY toStartOfMonth(timestamp)
+ORDER BY (event_type, timestamp)
+TTL timestamp + INTERVAL 1 YEAR DELETE;  -- Drops whole partitions
+
+-- Fast: metadata-only operation
+ALTER TABLE events DROP PARTITION '202301';
+
+-- Archive to cold storage
+ALTER TABLE events_archive ATTACH PARTITION '202301' FROM events;
+```
+
+Reference: [Choosing a Partitioning Key](https://clickhouse.com/docs/best-practices/choosing-a-partitioning-key)

.agents/skills/clickhouse-best-practices/rules/schema-partition-low-cardinality.md

@@ -0,0 +1,61 @@
+---
+title: Keep Partition Cardinality Low (100-1,000 Values)
+impact: HIGH
+impactDescription: "Too many partitions cause part explosion and 'too many parts' errors"
+tags: [schema, partitioning, parts]
+---
+
+## Keep Partition Cardinality Low (100-1,000 Values)
+
+**Impact: HIGH**
+
+Too many distinct partition values create excessive data parts, eventually triggering "too many parts" errors. ClickHouse enforces limits via `max_parts_in_total` and `parts_to_throw_insert` settings.
+
+**Incorrect (high cardinality partitioning):**
+
+```sql
+-- High cardinality = too many partitions
+CREATE TABLE events (...)
+ENGINE = MergeTree()
+PARTITION BY user_id  -- Millions of partitions!
+ORDER BY (timestamp);
+
+-- Daily partitions can grow unbounded over years
+CREATE TABLE logs (...)
+ENGINE = MergeTree()
+PARTITION BY toDate(timestamp)  -- 3650 partitions over 10 years
+ORDER BY (service, timestamp);
+```
+
+**Correct (bounded cardinality):**
+
+```sql
+-- Monthly partitions = 12 per year, bounded cardinality
+CREATE TABLE events (
+    timestamp DateTime,
+    event_type LowCardinality(String),
+    user_id UInt64
+)
+ENGINE = MergeTree()
+PARTITION BY toStartOfMonth(timestamp)
+ORDER BY (event_type, timestamp);
+```
+
+**Validation:**
+
+```sql
+-- Check partition count and health
+SELECT
+    partition,
+    count() as parts,
+    sum(rows) as rows,
+    formatReadableSize(sum(bytes_on_disk)) as size
+FROM system.parts
+WHERE table = 'events' AND active
+GROUP BY partition
+ORDER BY partition;
+
+-- Warning signs: hundreds or thousands of partitions
+```
+
+Reference: [Choosing a Partitioning Key](https://clickhouse.com/docs/best-practices/choosing-a-partitioning-key)

.agents/skills/clickhouse-best-practices/rules/schema-partition-query-tradeoffs.md

@@ -0,0 +1,35 @@
+---
+title: Understand Partition Query Performance Trade-offs
+impact: MEDIUM
+impactDescription: "Partition pruning helps some queries; spanning many partitions hurts others"
+tags: [schema, partitioning, query, performance]
+---
+
+## Understand Partition Query Performance Trade-offs
+
+**Impact: MEDIUM**
+
+Partitioning can help or hurt query performance:
+- **Potential improvement**: Queries filtering by partition key may benefit from partition pruning
+- **Potential degradation**: Queries spanning many partitions increase total parts scanned
+
+ClickHouse automatically builds **MinMax indexes** on partition columns. Data merges occur **within partitions only**, not across them.
+
+**Incorrect (query scans all partitions):**
+
+```sql
+-- Query must scan all partitions
+SELECT count(*) FROM events
+WHERE event_type = 'click';  -- No partition pruning
+```
+
+**Correct (query prunes to single partition):**
+
+```sql
+-- Query prunes to single partition
+SELECT count(*) FROM events
+WHERE timestamp >= '2024-01-01' AND timestamp < '2024-02-01'
+  AND event_type = 'click';
+```
+
+Reference: [Choosing a Partitioning Key](https://clickhouse.com/docs/best-practices/choosing-a-partitioning-key)

.agents/skills/clickhouse-best-practices/rules/schema-partition-start-without.md

@@ -0,0 +1,42 @@
+---
+title: Consider Starting Without Partitioning
+impact: MEDIUM
+impactDescription: "Add partitioning later when you have clear lifecycle requirements"
+tags: [schema, partitioning, simplicity]
+---
+
+## Consider Starting Without Partitioning
+
+**Impact: MEDIUM**
+
+Start without partitioning and add it later only if:
+- You have clear data lifecycle requirements (retention, archiving)
+- Your access patterns clearly benefit from partition pruning
+- You understand the cardinality implications
+
+**Example (start simple):**
+
+```sql
+-- Start simple, no partitioning
+CREATE TABLE events (
+    timestamp DateTime,
+    event_type LowCardinality(String),
+    user_id UInt64
+)
+ENGINE = MergeTree()
+ORDER BY (event_type, timestamp);
+
+-- Add partitioning later if needed for lifecycle management
+-- (requires table recreation or materialized view migration)
+```
+
+**When to add partitioning:**
+
+| Need | Add Partitioning? |
+|------|-------------------|
+| Time-based data retention | Yes |
+| Archive old data to cold storage | Yes |
+| Query performance on time ranges | Maybe (test first) |
+| No specific lifecycle needs | No |
+
+Reference: [Choosing a Partitioning Key](https://clickhouse.com/docs/best-practices/choosing-a-partitioning-key)

.agents/skills/clickhouse-best-practices/rules/schema-pk-cardinality-order.md

@@ -0,0 +1,45 @@
+---
+title: Order Columns by Cardinality (Low to High)
+impact: CRITICAL
+impactDescription: "Enables granule skipping; high-cardinality first prevents index pruning"
+tags: [schema, primary-key, cardinality, ORDER BY]
+---
+
+## Order Columns by Cardinality (Low to High)
+
+**Impact: CRITICAL**
+
+Since the sparse primary index operates on data blocks (granules) rather than individual rows, low-cardinality leading columns create more useful index entries that can skip entire blocks. Place lower-cardinality columns before higher-cardinality ones in the ordering key.
+
+**Incorrect (high cardinality first):**
+
+```sql
+-- UUID first means no pruning benefit
+CREATE TABLE events (...)
+ENGINE = MergeTree()
+ORDER BY (event_id, event_type, timestamp);
+-- Every granule has different event_id values, index can't skip anything
+```
+
+**Correct (low cardinality first):**
+
+```sql
+-- Low cardinality first enables pruning
+CREATE TABLE events (...)
+ENGINE = MergeTree()
+ORDER BY (event_type, event_date, event_id);
+-- Index can skip entire event_type groups
+```
+
+**Column Order Guidelines:**
+
+| Position | Cardinality | Examples |
+|----------|-------------|----------|
+| 1st | Low (few distinct values) | event_type, status, country |
+| 2nd | Date (coarse granularity) | toDate(timestamp) |
+| 3rd+ | Medium-High | user_id, session_id |
+| Last | High (if needed) | event_id, uuid |
+
+**Tip:** Use `toDate(timestamp)` instead of raw `DateTime` columns when day-level filtering suffices - this reduces index size from 32-bit to 16-bit representations.
+
+Reference: [Choosing a Primary Key](https://clickhouse.com/docs/best-practices/choosing-a-primary-key)

.agents/skills/clickhouse-best-practices/rules/schema-pk-filter-on-orderby.md

@@ -0,0 +1,52 @@
+---
+title: Filter on ORDER BY Columns in Queries
+impact: CRITICAL
+impactDescription: "Skipping prefix columns prevents index usage"
+tags: [schema, primary-key, WHERE, query]
+---
+
+## Filter on ORDER BY Columns in Queries
+
+**Impact: CRITICAL**
+
+Even with good schema design, queries must use ORDER BY columns to benefit. Skipping prefix columns or filtering on non-ORDER BY columns prevents index usage.
+
+**Incorrect (skips prefix or uses non-ORDER BY columns):**
+
+```sql
+-- Given: ORDER BY (tenant_id, event_type, timestamp)
+
+-- Skips prefix columns - can't use index effectively
+SELECT * FROM events WHERE event_type = 'click';
+
+-- Filter on column not in ORDER BY - full table scan
+SELECT * FROM events WHERE user_agent LIKE '%Chrome%';
+```
+
+**Correct (uses ORDER BY prefix):**
+
+```sql
+-- Given: ORDER BY (tenant_id, event_type, timestamp)
+
+-- Full prefix match - best performance
+SELECT * FROM events
+WHERE tenant_id = 123 AND event_type = 'click';
+
+-- Partial prefix - still uses index
+SELECT * FROM events WHERE tenant_id = 123;
+
+-- Range on later column after equality on earlier
+SELECT * FROM events
+WHERE tenant_id = 123 AND event_type = 'click' AND timestamp >= '2024-01-01';
+```
+
+**Index usage reference:**
+
+| Filter | Index Used? |
+|--------|-------------|
+| `WHERE tenant_id = 123` | Full |
+| `WHERE tenant_id = 123 AND event_type = 'click'` | Full |
+| `WHERE event_type = 'click'` | None (skipped prefix) |
+| `WHERE timestamp > '2024-01-01'` | None (skipped both) |
+
+Reference: [Choosing a Primary Key](https://clickhouse.com/docs/best-practices/choosing-a-primary-key)

.agents/skills/clickhouse-best-practices/rules/schema-pk-plan-before-creation.md

@@ -0,0 +1,64 @@
+---
+title: Plan PRIMARY KEY Before Table Creation
+impact: CRITICAL
+impactDescription: "ORDER BY is immutable; wrong choice requires full data migration"
+tags: [schema, primary-key, ORDER BY]
+---
+
+## Plan PRIMARY KEY Before Table Creation
+
+**Impact: CRITICAL** (immutable after creation)
+
+ClickHouse's ORDER BY clause defines physical data ordering and the sparse index. Unlike other databases, **ORDER BY cannot be modified after table creation**. A wrong choice requires creating a new table and migrating all data.
+
+**Incorrect (arbitrary ORDER BY without query analysis):**
+
+```sql
+-- Creating table without analyzing query patterns
+CREATE TABLE events (
+    event_id UUID,
+    user_id UInt64,
+    timestamp DateTime
+)
+ENGINE = MergeTree()
+ORDER BY (event_id);  -- Chosen arbitrarily
+
+-- Later: "Most queries filter by user_id!"
+-- Cannot fix with: ALTER TABLE events MODIFY ORDER BY (user_id, timestamp)
+-- ERROR: Cannot modify ORDER BY
+```
+
+**Correct (query-driven ORDER BY selection):**
+
+```sql
+-- Step 1: Document query patterns BEFORE creating table
+/*
+Query Analysis:
+- 60% of queries: WHERE user_id = ? AND timestamp BETWEEN ? AND ?
+- 25% of queries: WHERE event_type = ? AND timestamp > ?
+- 15% of queries: WHERE event_id = ?
+
+Conclusion: user_id and event_type are primary filters
+*/
+
+-- Step 2: Create table with correct ORDER BY
+CREATE TABLE events (
+    event_id UUID DEFAULT generateUUIDv4(),
+    user_id UInt64,
+    event_type LowCardinality(String),
+    timestamp DateTime,
+    event_date Date DEFAULT toDate(timestamp)
+)
+ENGINE = MergeTree()
+PARTITION BY toYYYYMM(event_date)
+ORDER BY (user_id, event_date, event_id);
+```
+
+**Pre-creation checklist:**
+- [ ] Listed top 5-10 query patterns
+- [ ] Identified columns in WHERE clauses with frequency
+- [ ] Prioritized columns that exclude large numbers of rows
+- [ ] Ordered columns by cardinality (low first, high last)
+- [ ] Limited to 4-5 key columns (typically sufficient)
+
+Reference: [Choosing a Primary Key](https://clickhouse.com/docs/best-practices/choosing-a-primary-key)

.agents/skills/clickhouse-best-practices/rules/schema-pk-prioritize-filters.md

@@ -0,0 +1,44 @@
+---
+title: Prioritize Filter Columns in ORDER BY
+impact: CRITICAL
+impactDescription: "Columns not in ORDER BY cause full table scans"
+tags: [schema, primary-key, WHERE, filtering]
+---
+
+## Prioritize Filter Columns in ORDER BY
+
+**Impact: CRITICAL**
+
+Prioritize columns frequently used in query filters (WHERE clause), especially those that exclude large numbers of rows. Queries filtering on columns not in ORDER BY result in full table scans.
+
+**Incorrect (ORDER BY doesn't match query patterns):**
+
+```sql
+-- If most queries filter by tenant_id:
+CREATE TABLE events (...)
+ENGINE = MergeTree()
+ORDER BY (event_id);  -- Queries by tenant_id will full-scan!
+```
+
+**Correct (ORDER BY matches filter patterns):**
+
+```sql
+-- ORDER BY matches query filter patterns
+CREATE TABLE events (...)
+ENGINE = MergeTree()
+ORDER BY (tenant_id, event_date, event_id);
+
+-- Query now uses primary index:
+SELECT * FROM events WHERE tenant_id = 123 AND event_date >= '2024-01-01';
+```
+
+**Validation:**
+
+```sql
+-- Verify index usage
+EXPLAIN indexes = 1
+SELECT * FROM events WHERE tenant_id = 123;
+-- Look for "PrimaryKey" with Key Condition
+```
+
+Reference: [Choosing a Primary Key](https://clickhouse.com/docs/best-practices/choosing-a-primary-key)

.agents/skills/clickhouse-best-practices/rules/schema-types-avoid-nullable.md

@@ -0,0 +1,55 @@
+---
+title: Avoid Nullable Unless Semantically Required
+impact: HIGH
+impactDescription: "Nullable adds storage overhead; use DEFAULT values instead"
+tags: [schema, data-types, Nullable, DEFAULT]
+---
+
+## Avoid Nullable Unless Semantically Required
+
+**Impact: HIGH**
+
+Nullable columns maintain a separate UInt8 column for tracking null values, increasing storage and degrading performance. Use DEFAULT values instead when feasible.
+
+**Incorrect (Nullable everywhere):**
+
+```sql
+CREATE TABLE users (
+    id Nullable(UInt64),              -- IDs should never be null
+    name Nullable(String),            -- Empty string is fine
+    age Nullable(UInt8),              -- 0 is a valid default
+    login_count Nullable(UInt32)      -- 0 is a valid default
+)
+```
+
+**Correct (DEFAULT values, Nullable only when semantic):**
+
+```sql
+CREATE TABLE users (
+    id UInt64,                                    -- Never null
+    name String DEFAULT '',                       -- Empty = unknown
+    age UInt8 DEFAULT 0,                          -- 0 = unknown
+    login_count UInt32 DEFAULT 0,                 -- 0 = never logged in
+    deleted_at Nullable(DateTime),                -- NULL = not deleted (semantic!)
+    parent_id Nullable(UInt64)                    -- NULL = no parent (semantic!)
+)
+```
+
+**When Nullable IS appropriate:**
+
+| Use Case | Why |
+|----------|-----|
+| `deleted_at` | NULL = "not deleted", timestamp = "deleted at X" |
+| `parent_id` | NULL = "no parent", value = "has parent" |
+| `discount_percent` | NULL = "no discount", 0 = "0% discount" |
+
+**Defaults instead of Nullable:**
+
+| Type | Default |
+|------|---------|
+| String | `''` (empty string) |
+| UInt*/Int* | `0` |
+| DateTime | `now()` or `toDateTime(0)` |
+| UUID | `generateUUIDv4()` |
+
+Reference: [Select Data Types](https://clickhouse.com/docs/best-practices/select-data-types)

.agents/skills/clickhouse-best-practices/rules/schema-types-enum.md

@@ -0,0 +1,58 @@
+---
+title: Use Enum for Finite Value Sets
+impact: MEDIUM
+impactDescription: "Insert-time validation and natural ordering; 1-2 bytes storage"
+tags: [schema, data-types, Enum, validation]
+---
+
+## Use Enum for Finite Value Sets
+
+**Impact: MEDIUM**
+
+Enum types provide validation at insert time and enable queries that exploit natural ordering. Use Enum8 (up to 256 values) or Enum16 (up to 65,536 values).
+
+**Incorrect (String without validation):**
+
+```sql
+CREATE TABLE orders (
+    status String    -- No validation, typos like "shiped" allowed
+)
+
+-- Ordering requires CASE statements
+SELECT * FROM orders ORDER BY
+    CASE status
+        WHEN 'pending' THEN 1
+        WHEN 'processing' THEN 2
+        WHEN 'shipped' THEN 3
+    END;
+```
+
+**Correct (Enum with validation and ordering):**
+
+```sql
+CREATE TABLE orders (
+    status Enum8('pending' = 1, 'processing' = 2, 'shipped' = 3, 'delivered' = 4)
+)
+
+-- Insert validation: invalid values rejected
+INSERT INTO orders VALUES ('shiped');  -- ERROR: Unknown element 'shiped'
+
+-- Natural ordering works automatically
+SELECT * FROM orders ORDER BY status;  -- Orders by enum value (1, 2, 3, 4)
+
+-- Comparisons use natural order
+SELECT * FROM orders WHERE status > 'processing';  -- shipped and delivered
+```
+
+**Enum Guidelines:**
+
+| Scenario | Use |
+|----------|-----|
+| Fixed set of values known at schema time | Enum8/Enum16 |
+| Values may change frequently | LowCardinality(String) |
+| Need insert-time validation | Enum |
+| Need natural ordering in queries | Enum |
+| < 256 distinct values | Enum8 (1 byte) |
+| 256-65,536 distinct values | Enum16 (2 bytes) |
+
+Reference: [Select Data Types](https://clickhouse.com/docs/best-practices/select-data-types)

.agents/skills/clickhouse-best-practices/rules/schema-types-lowcardinality.md

@@ -0,0 +1,58 @@
+---
+title: Use LowCardinality for Repeated Strings
+impact: HIGH
+impactDescription: "Dictionary encoding for <10K unique values; significant storage reduction"
+tags: [schema, data-types, LowCardinality, storage]
+---
+
+## Use LowCardinality for Repeated Strings
+
+**Impact: HIGH**
+
+String columns with repeated values store each value repeatedly. LowCardinality uses dictionary encoding for significant storage reduction.
+
+**Incorrect (plain String for repeated values):**
+
+```sql
+CREATE TABLE events (
+    country String,       -- "United States" stored 500M times
+    browser String,       -- "Chrome" stored 300M times
+    event_type String     -- "page_view" stored 800M times
+)
+```
+
+**Correct (LowCardinality for low unique counts):**
+
+```sql
+CREATE TABLE events (
+    country LowCardinality(String),      -- ~200 unique values
+    browser LowCardinality(String),      -- ~50 unique values
+    event_type LowCardinality(String)    -- ~100 unique values
+)
+```
+
+**When to use LowCardinality:**
+
+| Unique Values | Recommendation |
+|---------------|----------------|
+| < 10,000 | Use LowCardinality |
+| > 10,000 | Use regular String |
+
+```sql
+-- Check cardinality before deciding
+SELECT uniq(column_name) FROM table_name;
+```
+
+**LowCardinality vs FixedString:**
+
+Reserve `FixedString` for strictly fixed-length data (e.g., 2-char country codes). For most low-cardinality text, `LowCardinality(String)` outperforms `FixedString`.
+
+```sql
+-- FixedString: Only for truly fixed-length data
+country_code FixedString(2),    -- "US", "DE", "JP" - always 2 chars
+
+-- LowCardinality: For variable-length low-cardinality strings
+country_name LowCardinality(String),  -- "United States", "Germany"
+```
+
+Reference: [Select Data Types](https://clickhouse.com/docs/best-practices/select-data-types)

.agents/skills/clickhouse-best-practices/rules/schema-types-minimize-bitwidth.md

@@ -0,0 +1,49 @@
+---
+title: Minimize Bit-Width for Numeric Types
+impact: HIGH
+impactDescription: "Smaller types reduce storage and improve cache efficiency"
+tags: [schema, data-types, numeric, storage]
+---
+
+## Minimize Bit-Width for Numeric Types
+
+**Impact: HIGH**
+
+Select the smallest numeric type that accommodates your data range. Prefer unsigned types when negative values aren't needed.
+
+**Incorrect (oversized types):**
+
+```sql
+CREATE TABLE metrics (
+    status_code Int64,        -- HTTP codes are 100-599
+    age Int64,                -- Human age fits in UInt8
+    year Int64,               -- Years fit in UInt16
+    item_count Int64          -- Often small numbers
+)
+```
+
+**Correct (right-sized types):**
+
+```sql
+CREATE TABLE metrics (
+    status_code UInt16,       -- 0-65,535 (HTTP codes fit easily)
+    age UInt8,                -- 0-255 (sufficient for age)
+    year UInt16,              -- 0-65,535 (sufficient for years)
+    item_count UInt32         -- 0-4 billion (adjust based on actual max)
+)
+```
+
+**Numeric Type Reference:**
+
+| Type | Range | Bytes |
+|------|-------|-------|
+| UInt8 | 0 to 255 | 1 |
+| UInt16 | 0 to 65,535 | 2 |
+| UInt32 | 0 to 4.3 billion | 4 |
+| UInt64 | 0 to 18 quintillion | 8 |
+| Int8 | -128 to 127 | 1 |
+| Int16 | -32,768 to 32,767 | 2 |
+| Int32 | -2.1 billion to 2.1 billion | 4 |
+| Int64 | -9 quintillion to 9 quintillion | 8 |
+
+Reference: [Select Data Types](https://clickhouse.com/docs/best-practices/select-data-types)

.agents/skills/clickhouse-best-practices/rules/schema-types-native-types.md

@@ -0,0 +1,51 @@
+---
+title: Use Native Types Instead of String
+impact: CRITICAL
+impactDescription: "2-10x storage reduction; enables compression and correct semantics"
+tags: [schema, data-types, storage]
+---
+
+## Use Native Types Instead of String
+
+**Impact: CRITICAL**
+
+Using String for all data wastes storage, prevents compression optimization, and makes comparisons slower. ClickHouse's column-oriented architecture benefits directly from optimal type selection.
+
+**Incorrect (String for everything):**
+
+```sql
+CREATE TABLE events (
+    event_id String,        -- "550e8400-e29b-41d4-a716-446655440000" = 36 bytes
+    user_id String,         -- "12345" = 5 bytes (no numeric operations)
+    created_at String,      -- "2024-01-15 10:30:00" = 19 bytes
+    count String,           -- "42" - can't do math!
+    is_active String        -- "true" = 4 bytes
+)
+```
+
+**Correct (native types):**
+
+```sql
+CREATE TABLE events (
+    event_id UUID DEFAULT generateUUIDv4(),     -- 16 bytes (vs 36)
+    user_id UInt64,                              -- 8 bytes, numeric ops
+    created_at DateTime DEFAULT now(),           -- 4 bytes (vs 19)
+    count UInt32 DEFAULT 0,                      -- 4 bytes, math works
+    is_active Bool DEFAULT true                  -- 1 byte (vs 4)
+)
+```
+
+**Type Selection Quick Reference:**
+
+| Data | Use | Avoid |
+|------|-----|-------|
+| Sequential IDs | UInt32/UInt64 | String |
+| UUIDs | UUID | String |
+| Status/Category | Enum8 or LowCardinality(String) | String |
+| Timestamps | DateTime | DateTime64, String |
+| Dates only | Date or Date32 | DateTime, String |
+| Counts | UInt8/16/32 (smallest that fits) | Int64, String |
+| Money | Decimal(P,S) or Int64 (cents) | Float64, String |
+| Booleans | Bool or UInt8 | String |
+
+Reference: [Select Data Types](https://clickhouse.com/docs/best-practices/select-data-types)

.agents/skills/code-review/SKILL.md

@@ -0,0 +1,59 @@
+---
+name: code-review
+description: |
+  Shared code review workflow for Langfuse. Use when reviewing a PR, branch, diff,
+  or local changes for correctness, regressions, risk, and missing tests.
+  Start with references/review-checklist.md for repo-specific review rules and
+  use package AGENTS.md files plus any matching shared skills when the change
+  touches those areas.
+---
+
+# Code Review
+
+Use this skill when the task is to review code changes rather than implement a
+feature.
+
+## Start Here
+
+- Read [`references/review-checklist.md`](references/review-checklist.md) for
+  the repo's canonical review rules.
+- Read root [`AGENTS.md`](../../../AGENTS.md) and the nearest package
+  `AGENTS.md` for the files under review.
+- If the review touches ClickHouse, also use the shared
+  `clickhouse-best-practices` skill.
+- If the review touches backend code, also use the shared
+  `backend-dev-guidelines` skill where relevant.
+- If the change accepts a user-supplied URL, adds outbound HTTP, introduces a
+  new integration, or touches secrets, RBAC, or redirect handling, also use
+  the shared [`security-review`](../security-review/SKILL.md) skill. Run its
+  [`references/checklist.md`](../security-review/references/checklist.md)
+  before signoff.
+
+## Review Priorities
+
+Focus on:
+
+- correctness bugs
+- behavioral regressions
+- security and tenant-isolation risks
+- performance issues with real impact
+- missing or weak tests for risky changes
+
+## Output Expectations
+
+- Findings first, ordered by severity
+- File and line references for each finding
+- Short summary only after findings
+- If no findings, say so explicitly and mention any residual risk or coverage gaps
+
+## Scope Guidance
+
+Use `references/review-checklist.md` for Langfuse-specific checks such as:
+
+- ClickHouse and Postgres migration expectations
+- project-scoped tenant isolation checks
+- API/Fern consistency
+- banner-offset UI positioning
+- environment variable access patterns
+
+Do not duplicate those rules in ad hoc prompts or tool-specific command files.

.agents/skills/code-review/references/review-checklist.md

@@ -0,0 +1,73 @@
+# Langfuse Review Checklist
+
+This is the canonical shared review checklist for Langfuse.
+
+## Database Migrations
+
+### ClickHouse
+
+- ClickHouse migrations in the `packages/shared/clickhouse/migrations/clustered` directory should include `ON CLUSTER default` and should use `Replicated` merge tree table types.
+  - E.g. `ReplacingMergeTree` is likely an error while `ReplicatedReplacingMergeTree` would be correct in most cases.
+- ClickHouse migrations in the `packages/shared/clickhouse/migrations/unclustered` directory must not include `ON CLUSTER` statements and must not use `Replicated` merge tree table types.
+- Migrations in `packages/shared/clickhouse/migrations/clustered` should match their counterparts in `packages/shared/clickhouse/migrations/unclustered` aside from the restrictions listed above.
+- When adding new indexes on ClickHouse, ensure that there is a corresponding `MATERIALIZE INDEX` statement in the same migration. The materialization can use `SETTINGS mutations_sync = 2` if they operate on smaller tables, but may timeout otherwise.
+- All ClickHouse queries on project-scoped tables (traces, observations, scores, events, sessions, etc.) must include `WHERE project_id = {projectId: String}` filter to ensure proper tenant isolation and that queries only access data from the intended project.
+- For operations on the `events` table, you must never use the `FINAL` keyword as it kills performance. `events` is built so that `FINAL` is never required.
+
+### Postgres
+
+- Most `schema.prisma` changes should produce a change in `packages/shared/prisma/migrations`.
+- All Prisma queries on project-scoped tables must include `projectId` in the WHERE clause (e.g., `where: { id: traceId, projectId }`) to ensure proper tenant isolation and that queries only access data from the intended project.
+
+### Environment Variables
+
+- Environment variables should be imported from the `env.mjs/ts` file of the respective package and not from `process.env.*` to ensure validation and typing.
+
+## Redis Invocations
+
+- Highlight usage of `redis.call` invocations. Those may have suboptimal redis cluster routing and will raise errors. Instead, use the native call patterns.
+  Example: `await redis?.call("SET", key, "1", "NX", "EX", TTLSeconds);` should use `await redis?.set(key, "1", "EX", TTLSeconds, "NX");` instead.
+
+## Langfuse Cloud
+
+- When attempting to confirm if the current environment is Langfuse Cloud in the frontend, use the `useLangfuseCloudRegion` hook and never environment variables directly.
+
+## Banner Height System
+
+- Use `top-banner-offset` instead of `top-0` for any elements that are positioned `sticky`, `fixed`, or `absolute` with a global reference point (e.g., `top-0`). This ensures proper spacing when system banners (payment, maintenance, etc.) are displayed.
+- The banner height is managed through CSS variables (`--banner-height` and `--banner-offset`) defined in `web/src/styles/globals.css`.
+- Banner components (like PaymentBanner) dynamically update `--banner-height` using ResizeObserver to track their actual height, ensuring accurate positioning even when banners resize (e.g., on mobile wrapping).
+- Available Tailwind utilities:
+  - `top-banner-offset` / `pt-banner-offset` - For sticky/fixed/absolute positioning and padding
+  - `h-screen-with-banner` / `min-h-screen-with-banner` - For full-height containers accounting for banners
+
+## Security
+
+- For changes that accept a user-supplied URL, host, `endpoint`, `baseURL`,
+  or webhook target, or that issue a new outbound HTTP request, run the
+  shared [`security-review`](../../security-review/SKILL.md) skill and treat
+  its [`outbound-url-validation.md`](../../security-review/references/outbound-url-validation.md)
+  defenses (save-time validation + use-time / connection-time validation +
+  redirect-time validation) as required. Plain `fetch(<userUrl>)` or SDK
+  init with `endpoint: <userUrl>` without one of the canonical validators
+  (`validateLlmConnectionBaseURL`, `validateWebhookURL`,
+  `validateBlobStorageEndpoint`, or a new wrapper around
+  `validateOutboundUrlHost`) is a finding.
+- For changes that add a new integration, secret-bearing field, redirect
+  follower, or RBAC scope, run the rest of the
+  [`security-review/references/checklist.md`](../../security-review/references/checklist.md).
+
+## JavaScript / TypeScript Style
+
+- use concat instead of spread to avoid stack overflow with large arrays
+
+## Seeder
+
+- make sure that for new features with data model changes, the database seeder is adjusted.
+
+## API Documentation
+
+- Whenever a file in `web/src/features/public-api/types` changes, the `fern/apis` definition probably needs to be adjusted, too.
+- `nullish` types should map to `optional<nullable<T>>` in fern.
+- `nullable` types should map to `nullable<T>` in fern.
+- `optional` types should map to `optional<T>` in fern.

.agents/skills/datadog-query-recipes/SKILL.md

@@ -0,0 +1,74 @@
+---
+name: datadog-query-recipes
+description: |
+  Langfuse-specific Datadog query recipes for production telemetry research.
+  Use when asked to investigate tenant or project activity, public API endpoint
+  usage, queue consumer behavior, spans, logs, metrics, or ad hoc production
+  questions across prod-us, prod-eu, prod-hipaa, and prod-jp. This skill is for
+  reusable query shapes and measured research; pair it with
+  debug-issue-with-datadog when the task is an incident or root-cause analysis.
+---
+
+# Datadog Query Recipes
+
+Use this skill for Langfuse production telemetry research where the main work is
+finding the right Datadog data path. Keep findings evidence-based and include
+the exact Datadog links or query shapes that support the answer.
+
+## Required Scope
+
+Unless the user explicitly narrows the scope, cover every production
+environment:
+
+- `prod-us`
+- `prod-eu`
+- `prod-hipaa`
+- `prod-jp`
+
+Query both Datadog sites when needed. Default to the EU site for `prod-eu` and
+the US site for the other prod environments, but verify with a small count or
+facet query before concluding an environment has no data.
+
+Before querying live Datadog, load the relevant Datadog MCP guidance for the
+data domain you need: traces, logs, metrics, and visualizations.
+
+## Workflow
+
+1. Identify the entity and signal: tenant ID, org ID, project ID, route, queue,
+   service, error class, or metric.
+2. Read only the relevant reference:
+   - Prod environment/site routing:
+     [`references/environments.md`](references/environments.md)
+   - Public API tenant or legacy endpoint usage:
+     [`references/public-api-tenant-usage.md`](references/public-api-tenant-usage.md)
+   - Queue inventory, queue consumers, and queue metrics:
+     [`references/queue-consumers.md`](references/queue-consumers.md)
+3. Start with aggregate queries, grouped by environment, service, route,
+   queue, project, org, status, or error facets as appropriate.
+4. Fetch raw spans, logs, or traces only after aggregation identifies the
+   cluster or sample you need.
+5. For tenant-specific HTTP usage, prefer trace correlation over single-span
+   queries when tenant tags and route tags live on different spans.
+6. Report the windows, environments, sites, query links, and any sampling or
+   missing-data caveats.
+
+## When To Use Other Skills
+
+- Use [`debug-issue-with-datadog`](../debug-issue-with-datadog/SKILL.md) when a
+  Linear issue, GitHub issue, incident report, or monitor needs root-cause
+  analysis and patch recommendations.
+- Use [`weekly-production-review`](../weekly-production-review/SKILL.md) when
+  the user asks for a weekly engineering overview of production bugs, pages,
+  and incidents.
+- Use [`linear-bug-triage`](../linear-bug-triage/SKILL.md) only after a human
+  approves sharing measured findings in Linear.
+
+## Output Expectations
+
+Summarize what was checked, including:
+
+- Datadog site and `env` values covered.
+- Time windows.
+- Core filters or metrics used.
+- Count, rate, latency, queue depth, trace sample, or "No measurements found".
+- Datadog links or trace IDs that let the human rerun the query.

.agents/skills/datadog-query-recipes/agents/openai.yaml

@@ -0,0 +1,4 @@
+interface:
+  display_name: "Datadog Query Recipes"
+  short_description: "Langfuse production telemetry queries"
+  default_prompt: "Use $datadog-query-recipes to investigate Langfuse production telemetry for a tenant, endpoint, queue, or regression."

.agents/skills/datadog-query-recipes/references/environments.md

@@ -0,0 +1,45 @@
+# Production Environments
+
+Langfuse production deploys cover these environments and services. The deploy
+matrix is defined in `.github/workflows/deploy.yml`.
+
+| Environment | Primary Datadog site | Common services |
+| --- | --- | --- |
+| `prod-us` | US, `datadoghq.com` | `web`, `web-ingestion`, `web-iso`, `worker`, `worker-cpu` |
+| `prod-eu` | EU, `datadoghq.eu` | `web`, `web-ingestion`, `web-iso`, `worker`, `worker-cpu` |
+| `prod-hipaa` | US, `datadoghq.com` | `web`, `web-ingestion`, `web-iso`, `worker`, `worker-cpu` |
+| `prod-jp` | US, `datadoghq.com` | `web`, `web-ingestion`, `web-iso`, `worker`, `worker-cpu` |
+
+The site mapping is a starting point, not proof. For cross-region research, run
+a small count or facet query on both Datadog sites before saying an environment
+has no data.
+
+## Starter Filters
+
+Use these as first-pass filters, then add the subsystem-specific route, queue,
+tenant, or error facets.
+
+```text
+env:prod-us
+env:prod-eu
+env:prod-hipaa
+env:prod-jp
+```
+
+```text
+env:<env> (service:web OR service:web-ingestion OR service:web-iso)
+env:<env> (service:worker OR service:worker-cpu)
+```
+
+For HTTP routes, start with `service:web` or `service:web-ingestion` depending
+on the endpoint. For queue consumers, start with `service:worker` and
+`service:worker-cpu`.
+
+## Cross-Site Rule
+
+When a query returns zero:
+
+1. Check the time window and spelling of `env`, `service`, and route/resource.
+2. Query facets on the same site for `env` and `service`.
+3. Repeat a small count query on the other Datadog site.
+4. Only then report "No measurements found".

.agents/skills/datadog-query-recipes/references/public-api-tenant-usage.md

@@ -0,0 +1,99 @@
+# Public API Tenant Usage
+
+Use this recipe when checking whether an org or project calls a public API
+route, including legacy endpoints such as:
+
+- `/api/public/traces`
+- `/api/public/traces/<traceId>` (`/api/public/traces/[traceId]` in Next.js)
+- `/api/public/observations`
+- `/api/public/observations/<observationId>`
+  (`/api/public/observations/[observationId]` in Next.js)
+- `/api/public/metrics`
+
+Migrated endpoints commonly include:
+
+- `/api/public/v2/observations`
+- `/api/public/v2/metrics`
+- `/api/public/otel/v1/traces`
+
+## Why Correlation Is Needed
+
+Public API auth attaches tenant attributes to the `api-auth-verify` child span:
+
+- `@langfuse.project.id`: project-scoped API key target.
+- `@langfuse.org.id`: owning org or org-scoped key target.
+- `@langfuse.org.plan`: plan when present.
+
+The HTTP route is on the request root span, usually in `http.path_group`,
+`http.route`, `http.target`, or `resource_name`.
+
+Because tenant tags and route tags often live on different spans in the same
+trace, do not expect this single-span query to work:
+
+```text
+@langfuse.project.id:<id> @http.path_group:/api/public/observations
+```
+
+## Per-Tenant Endpoint Recipe
+
+1. Aggregate auth spans for the tenant to confirm the identifier and active
+   projects.
+
+```text
+env:<env> @langfuse.org.id:<orgId> resource_name:api-auth-verify
+env:<env> @langfuse.project.id:<projectId> resource_name:api-auth-verify
+```
+
+Group by `service`, `@langfuse.project.id`, and optionally a daily interval.
+
+2. Fetch representative matching auth spans with `search_datadog_spans`.
+   Include custom attributes such as `langfuse.*`, then copy representative
+   `traceid` values.
+
+3. Open those traces with `get_datadog_trace`. Request service-entry spans and
+   include HTTP and Langfuse attributes:
+
+```text
+only_service_entry_spans: true
+extra_fields: ["http.*", "next.*", "langfuse.*"]
+```
+
+4. Read the request root span's `http.path_group`, `http.route`,
+   `http.target`, and `resource_name` to identify the endpoint.
+   ID lookup routes may appear with the concrete ID in `http.target` or with
+   the normalized Next.js route, such as
+   `/api/public/traces/[traceId]` or
+   `/api/public/observations/[observationId]`.
+
+5. Repeat across all relevant prod environments and both Datadog sites when the
+   user asks for a global answer.
+
+Treat per-tenant endpoint results as sampled unless you have an unsampled
+metric or log source with tenant and route on the same event.
+
+## Fleet-Level Endpoint Volume
+
+For endpoint volume without tenant scoping, aggregate request spans directly:
+
+```text
+env:<env> service:web resource_name:"GET /api/public/observations*"
+env:<env> service:web resource_name:"GET /api/public/observations/*"
+env:<env> service:web resource_name:"POST /api/public/traces*"
+env:<env> service:web resource_name:"GET /api/public/traces/*"
+env:<env> service:web resource_name:"POST /api/public/metrics*"
+```
+
+Use route facets where available:
+
+```text
+env:<env> service:web @http.path_group:/api/public/observations
+env:<env> service:web @http.path_group:/api/public/observations/*
+env:<env> service:web @http.path_group:/api/public/observations/[observationId]
+env:<env> service:web @http.route:/api/public/observations
+env:<env> service:web @http.route:/api/public/observations/[observationId]
+env:<env> service:web @http.path_group:/api/public/traces/[traceId]
+env:<env> service:web @http.route:/api/public/traces/[traceId]
+```
+
+If route facets and resource names disagree, fetch a few traces and inspect the
+root span before reporting the result.

.agents/skills/datadog-query-recipes/references/queue-consumers.md

@@ -0,0 +1,234 @@
+# Queue Consumers
+
+Use this reference when a task asks which Langfuse queues exist, whether a
+consumer is running, how much work a queue has, or how to query queue processor
+spans.
+
+## Source Of Truth
+
+- Queue names and job names:
+  `packages/shared/src/server/queues.ts` (`QueueName`, `QueueJobs`).
+- Queue producer classes and shard naming:
+  `packages/shared/src/server/redis/*.ts`.
+- Worker consumer registration and feature gates:
+  `worker/src/app.ts`.
+- Worker consumer env vars:
+  `worker/src/env.ts` (`QUEUE_CONSUMER_*_IS_ENABLED` plus feature-specific
+  gates).
+- Worker registration, request/error counters, wait/processing time, and
+  sampled old-style depth metrics:
+  `worker/src/queues/workerManager.ts`.
+- Queue depth background reporter:
+  `worker/src/features/queue-metrics-runner/index.ts`.
+- Metric name conversion:
+  `packages/shared/src/server/instrumentation/index.ts`
+  (`convertQueueNameToMetricName`).
+- Sharded queue registry:
+  `worker/src/queues/shardedQueueRegistry.ts`.
+- BullMQ tracing setup:
+  `worker/src/instrumentation.ts` (`BullMQInstrumentation`).
+
+## Queue Inventory
+
+Current `QueueName` values:
+
+| Queue | Notes |
+| --- | --- |
+| `trace-upsert` | Sharded. Registers all `TraceUpsertQueue` shards. |
+| `trace-delete` | Delete traces from storage. |
+| `project-delete` | Project deletion cleanup. |
+| `evaluation-execution-queue` | Sharded eval execution. |
+| `secondary-evaluation-execution-queue` | Sharded secondary eval execution. |
+| `llm-as-a-judge-execution-queue` | Sharded observation-based eval execution. |
+| `dataset-run-item-upsert-queue` | Dataset run item upserts. |
+| `batch-export-queue` | Batch exports. |
+| `otel-ingestion-queue` | Sharded OTel ingestion. |
+| `secondary-otel-ingestion-queue` | Sharded secondary OTel ingestion. |
+| `ingestion-queue` | Sharded single-event ingestion. |
+| `secondary-ingestion-queue` | Sharded secondary single-event ingestion. |
+| `cloud-usage-metering-queue` | Cloud-only, Stripe-gated. |
+| `cloud-spend-alert-queue` | Cloud-only, Stripe-gated. |
+| `cloud-free-tier-usage-threshold-queue` | Cloud-only, Stripe-gated. |
+| `experiment-create-queue` | Experiment creation. |
+| `posthog-integration-queue` | Schedules PostHog integration jobs. |
+| `posthog-integration-processing-queue` | Processes PostHog projects. |
+| `mixpanel-integration-queue` | Schedules Mixpanel integration jobs. |
+| `mixpanel-integration-processing-queue` | Processes Mixpanel projects. |
+| `blobstorage-integration-queue` | Schedules blob storage jobs. |
+| `blobstorage-integration-processing-queue` | Processes blob storage projects. |
+| `core-data-s3-export-queue` | Cloud export feature gate. |
+| `metering-data-postgres-export-queue` | Cloud export feature gate. |
+| `data-retention-queue` | Schedules data retention jobs. |
+| `data-retention-processing-queue` | Processes data retention projects. |
+| `batch-action-queue` | Batch actions. |
+| `create-eval-queue` | Eval job creation. |
+| `score-delete` | Score deletion cleanup. |
+| `dataset-delete-queue` | Dataset deletion cleanup. |
+| `dead-letter-retry-queue` | Dead letter retry worker. |
+| `webhook-queue` | Webhook delivery. |
+| `entity-change-queue` | Entity change propagation. |
+| `event-propagation-queue` | Experiment event propagation gate. |
+| `notification-queue` | Notifications. |
+
+Sharded queues use the base queue for shard 0 and append `-1`, `-2`, etc. for
+additional shards. The sharded base queues are:
+
+- `trace-upsert`
+- `evaluation-execution-queue`
+- `secondary-evaluation-execution-queue`
+- `llm-as-a-judge-execution-queue`
+- `otel-ingestion-queue`
+- `secondary-otel-ingestion-queue`
+- `ingestion-queue`
+- `secondary-ingestion-queue`
+
+## Consumer Gates
+
+Consumer registration is in `worker/src/app.ts`. Some gates register multiple
+queues or every shard for a sharded queue.
+
+| Gate | Queues registered |
+| --- | --- |
+| `QUEUE_CONSUMER_TRACE_UPSERT_QUEUE_IS_ENABLED` | `trace-upsert` shards |
+| `QUEUE_CONSUMER_CREATE_EVAL_QUEUE_IS_ENABLED` | `create-eval-queue` |
+| `LANGFUSE_S3_CORE_DATA_EXPORT_IS_ENABLED` | `core-data-s3-export-queue` |
+| `LANGFUSE_POSTGRES_METERING_DATA_EXPORT_IS_ENABLED` | `metering-data-postgres-export-queue` |
+| `QUEUE_CONSUMER_TRACE_DELETE_QUEUE_IS_ENABLED` | `trace-delete` |
+| `QUEUE_CONSUMER_SCORE_DELETE_QUEUE_IS_ENABLED` | `score-delete` |
+| `QUEUE_CONSUMER_DATASET_DELETE_QUEUE_IS_ENABLED` | `dataset-delete-queue` |
+| `QUEUE_CONSUMER_PROJECT_DELETE_QUEUE_IS_ENABLED` | `project-delete` |
+| `QUEUE_CONSUMER_DATASET_RUN_ITEM_UPSERT_QUEUE_IS_ENABLED` | `dataset-run-item-upsert-queue` |
+| `QUEUE_CONSUMER_EVAL_EXECUTION_QUEUE_IS_ENABLED` | `evaluation-execution-queue` shards, `llm-as-a-judge-execution-queue` shards |
+| `QUEUE_CONSUMER_EVAL_EXECUTION_SECONDARY_QUEUE_IS_ENABLED` | `secondary-evaluation-execution-queue` shards |
+| `QUEUE_CONSUMER_BATCH_EXPORT_QUEUE_IS_ENABLED` | `batch-export-queue` |
+| `QUEUE_CONSUMER_BATCH_ACTION_QUEUE_IS_ENABLED` | `batch-action-queue` |
+| `QUEUE_CONSUMER_OTEL_INGESTION_QUEUE_IS_ENABLED` | `otel-ingestion-queue` shards |
+| `QUEUE_CONSUMER_OTEL_INGESTION_SECONDARY_QUEUE_IS_ENABLED` | `secondary-otel-ingestion-queue` shards |
+| `QUEUE_CONSUMER_INGESTION_QUEUE_IS_ENABLED` | `ingestion-queue` shards |
+| `QUEUE_CONSUMER_INGESTION_SECONDARY_QUEUE_IS_ENABLED` | `secondary-ingestion-queue` shards |
+| `QUEUE_CONSUMER_CLOUD_USAGE_METERING_QUEUE_IS_ENABLED` plus `STRIPE_SECRET_KEY` | `cloud-usage-metering-queue` |
+| `QUEUE_CONSUMER_CLOUD_SPEND_ALERT_QUEUE_IS_ENABLED` plus `STRIPE_SECRET_KEY` | `cloud-spend-alert-queue` |
+| `QUEUE_CONSUMER_FREE_TIER_USAGE_THRESHOLD_QUEUE_IS_ENABLED` plus cloud region and Stripe gates | `cloud-free-tier-usage-threshold-queue` |
+| `QUEUE_CONSUMER_EXPERIMENT_CREATE_QUEUE_IS_ENABLED` | `experiment-create-queue` |
+| `QUEUE_CONSUMER_POSTHOG_INTEGRATION_QUEUE_IS_ENABLED` | `posthog-integration-queue`, `posthog-integration-processing-queue` |
+| `QUEUE_CONSUMER_MIXPANEL_INTEGRATION_QUEUE_IS_ENABLED` | `mixpanel-integration-queue`, `mixpanel-integration-processing-queue` |
+| `QUEUE_CONSUMER_BLOB_STORAGE_INTEGRATION_QUEUE_IS_ENABLED` | `blobstorage-integration-queue`, `blobstorage-integration-processing-queue` |
+| `QUEUE_CONSUMER_DATA_RETENTION_QUEUE_IS_ENABLED` | `data-retention-queue`, `data-retention-processing-queue` |
+| `QUEUE_CONSUMER_DEAD_LETTER_RETRY_QUEUE_IS_ENABLED` | `dead-letter-retry-queue` |
+| `QUEUE_CONSUMER_WEBHOOK_QUEUE_IS_ENABLED` | `webhook-queue` |
+| `QUEUE_CONSUMER_ENTITY_CHANGE_QUEUE_IS_ENABLED` | `entity-change-queue` |
+| `QUEUE_CONSUMER_EVENT_PROPAGATION_QUEUE_IS_ENABLED` plus events-table experiment gate | `event-propagation-queue` |
+| `QUEUE_CONSUMER_NOTIFICATION_QUEUE_IS_ENABLED` | `notification-queue` |
+
+## Query Consumer Spans
+
+Start with aggregate spans on worker services:
+
+```text
+env:<env> (service:worker OR service:worker-cpu) operation_name:bullmq.process
+```
+
+Then group by `resource_name`, queue facets such as `bullmq.queue` or
+`messaging.*`, and error fields. Facet names can differ between Datadog sites,
+so inspect one sample span before relying on a specific facet.
+
+Queue-specific starter query:
+
+```text
+env:<env> (service:worker OR service:worker-cpu) operation_name:bullmq.process (resource_name:"process otel-ingestion-queue" OR resource_name:"Worker.run otel-ingestion-queue" OR bullmq.queue:otel-ingestion-queue)
+```
+
+For sharded queues, query the base queue and shard suffixes:
+
+```text
+env:<env> (service:worker OR service:worker-cpu) operation_name:bullmq.process resource_name:"*otel-ingestion-queue*"
+```
+
+If a queue file wraps the handler with `instrumentAsync`, also search the
+domain-specific resource name. Examples:
+
+| Subsystem | Resource name |
+| --- | --- |
+| PostHog project processing | `process posthog-integration-project` |
+| Mixpanel project processing | `process mixpanel-integration-project` |
+| Blob storage project processing | `process blob-storage-project` |
+| Data retention project processing | `process data-retention-project` |
+| Event propagation | `process event-propagation` |
+| Cloud usage metering | `process cloud-usage-metering` |
+| Free-tier usage threshold | `process cloud-free-tier-usage-threshold` |
+
+Useful aggregations:
+
+- Count by `env`, `service`, and `resource_name`.
+- Count by queue facet and status.
+- Count by `error.type` and `error.message`.
+- p50, p95, and p99 duration by queue or shard.
+- Count by `messaging.bullmq.job.input.projectId` when the processor attaches
+  project IDs to the current span.
+
+## Query Queue Metrics
+
+Metric base names come from `convertQueueNameToMetricName(queueName)`:
+
+```text
+langfuse.queue.<queue-name-with-hyphens-replaced-by-underscores-and-trailing-_queue-removed>
+```
+
+Examples:
+
+| Queue | Metric base |
+| --- | --- |
+| `ingestion-queue` | `langfuse.queue.ingestion` |
+| `otel-ingestion-queue` | `langfuse.queue.otel_ingestion` |
+| `secondary-otel-ingestion-queue` | `langfuse.queue.secondary_otel_ingestion` |
+| `evaluation-execution-queue` | `langfuse.queue.evaluation_execution` |
+| `trace-upsert` | `langfuse.queue.trace_upsert` |
+| `batch-export-queue` | `langfuse.queue.batch_export` |
+
+Prefer the newer tagged metrics:
+
+```text
+<metric_base>.depth{env:<env>,type:waiting}
+<metric_base>.depth{env:<env>,type:failed}
+<metric_base>.depth{env:<env>,type:active}
+<metric_base>.rate{env:<env>,type:request}
+<metric_base>.rate{env:<env>,type:failed}
+<metric_base>.rate{env:<env>,type:error}
+<metric_base>.time{env:<env>,type:wait}
+<metric_base>.time{env:<env>,type:processing}
+```
+
+For sharded queues, use the `shard` tag when present. `shard:all` is emitted by
+the depth runner for aggregate depth across shards.
+
+Backward-compatible metrics may still appear:
+
+```text
+<metric_base>.length
+<metric_base>.dlq_length
+<metric_base>.active
+<metric_base>.request
+<metric_base>.failed
+<metric_base>.error
+<metric_base>.wait_time
+<metric_base>.processing_time
+```
+
+For non-BullMQ internal write buffering, `ClickhouseWriter` emits
+`langfuse.queue.clickhouse_writer.*` metrics, but it is not a `QueueName`
+consumer.
+
+## Consumer Running Checklist
+
+To establish whether a consumer is running in production:
+
+1. Check queue depth metrics for waiting, failed, and active counts.
+2. Check `rate{type:request}` or old `.request` metrics for recent processing.
+3. Search BullMQ processor spans on `worker` and `worker-cpu`.
+4. Search worker logs for the queue name or processor-specific log prefix.
+5. If all signals are empty, verify the relevant `QUEUE_CONSUMER_*_IS_ENABLED`
+   gate and any feature-specific gates in `worker/src/app.ts`.
+
+The queue metrics runner only polls queues with registered workers. Missing
+depth metrics can mean the consumer is not registered on that worker, queue
+metrics are disabled, or the data is on the other Datadog site.

.agents/skills/debug-issue-with-datadog/SKILL.md

@@ -0,0 +1,121 @@
+---
+name: debug-issue-with-datadog
+description: |
+  Debug a user-reported issue, Linear ticket, or incident report by combining
+  Datadog (APM, logs, metrics) with the Langfuse repo to establish a
+  root cause. Use when given a Linear issue URL/ID (e.g. LFE-XXXX), a GitHub
+  issue, or a pasted error/report and asked to investigate, root-cause, or
+  triage. Produces a structured analysis — error breakdown, hypothesis-by-class,
+  suggested patches with code references.
+---
+
+# Debug Issue with Datadog
+
+Use this skill whenever the task is **investigative** rather than
+implementational: a user, customer, or oncall has surfaced a problem and you
+need to figure out *what is actually happening in production* and *where in the
+code it lives*. The deliverable is an analysis, not a patch — though the
+analysis should make the right patch obvious.
+
+## When to Apply
+
+- A Linear issue (typically with an `LFE-XXXX` ID) describes a production
+  failure, error spike, or customer report.
+- A GitHub issue or pasted incident/error report needs triage.
+- A monitor alerted and you need to understand *why* before deciding what to
+  fix.
+- Existing tickets under the "Make monitoring useful again" project (parent
+  `LFE-8837`) and similar — these expect the structured analysis output below.
+
+If the task is "implement this fix" rather than "figure out what's broken",
+this is the wrong skill — go to `backend-dev-guidelines` or the relevant
+package guide.
+
+## Workflow
+
+Read the inputs first, then plan the Datadog sweep, then read the code, then
+write the analysis. Do not skip ahead to suggested patches before the data
+supports them.
+
+1. **Intake.** Pull every signal already available in the report. See
+   [`references/intake.md`](references/intake.md). For a Linear URL/ID, fetch
+   the issue *and* its comments via the Linear MCP — the description is often
+   updated inline as triage proceeds. For a GitHub issue, use `gh issue view`.
+   For pasted text, treat it as the description.
+
+2. **Scope the sweep.** From the intake, pick the affected subsystem and time
+   window. Use [`references/repo-debug-map.md`](references/repo-debug-map.md)
+   to translate "PostHog integration", "ingestion failures", "evals stuck",
+   etc. into the Datadog filters and source files you should be looking at.
+
+3. **Run the broad Datadog sweep.** Default to the full sweep in
+   [`references/datadog-playbook.md`](references/datadog-playbook.md): APM
+   spans, error logs, metrics, and monitors — split across `prod-eu`
+   and `prod-us` (and `prod-hipaa` / `prod-jp` when relevant). Always check
+   regional disparity first; it usually rules whole hypotheses in or out.
+   Use [`datadog-query-recipes`](../datadog-query-recipes/SKILL.md) for
+   reusable tenant, public API, queue consumer, and cross-environment query
+   shapes.
+
+4. **Cluster the errors.** Group by `(projectId, error.message)` or
+   `(error.type, error.message)`. Treat each distinct cluster as its own
+   hypothesis — Langfuse incidents commonly have *multiple* coexisting root
+   causes, not one.
+
+5. **Map clusters to code.** For each cluster, open the relevant handler file
+   from the repo-debug map and read enough of it to confirm or refute the
+   hypothesis. Cite specific files and line ranges in the output.
+
+6. **Write the analysis** using
+   [`references/output-template.md`](references/output-template.md).
+
+7. **Deliver.** Default: print the analysis in chat. If the user asked for it,
+   also save under the workflow they specified (file, Linear comment via, etc.).
+
+## Datadog MCP Usage Notes
+
+Two Datadog MCP servers are typically available — one bound to the EU site
+(`datadoghq.eu`) and one to the US site (`datadoghq.com`). Always run
+region-relevant queries against **both** unless intake clearly localizes the
+incident. The `prod-eu` / `prod-us` env tags live on each side respectively.
+
+- Span search filter pattern:
+  `service:worker resource_name:"process posthog-integration-project" status:error`
+- Log search filter pattern:
+  `service:worker env:prod-eu @langfuse.project.id:cm1r6u… status:error`
+- For high-volume queries, prefer `aggregate_spans` / `aggregate_events`
+  grouped by `(error.message, projectId)` over fetching individual traces.
+- Always link to the Datadog UI for the queries you ran (final section of the
+  output template).
+
+See [`references/datadog-playbook.md`](references/datadog-playbook.md) for the
+full set of starter queries and parameter shapes.
+
+## Output Expectations
+
+From the output template:
+
+- Header: data source, time window, region split (EU vs US table).
+- Hotspots: per-`projectId` (or per-cluster) error counts.
+- Root cause by error class: each cluster gets a short hypothesis with
+  reasoning, distinguishing primary causes from symptoms.
+- Suggested patches: P0/P1/P2 grouped, with concrete file paths and short code
+  sketches. Reference the actual handler in `worker/src/features/**` or
+  `web/src/**`.
+- Dashboards: paste the Datadog query URLs at the end.
+
+Findings come first, recommendations last. If the data is thin, say so
+explicitly and propose what would need to be true to confirm each hypothesis —
+do not invent root causes.
+
+## Cross-References
+
+- Production telemetry query recipes, tenant/public API usage, and queue
+  consumer measurements:
+  [`datadog-query-recipes`](../datadog-query-recipes/SKILL.md)
+- Backend layout, queue contracts, instrumentation patterns:
+  [`backend-dev-guidelines`](../backend-dev-guidelines/SKILL.md)
+- ClickHouse-related findings (memory ceilings, JOIN spills, slow queries):
+  [`clickhouse-best-practices`](../clickhouse-best-practices/SKILL.md)
+- Once a fix is identified and you switch to implementation, hand off to the
+  package `AGENTS.md` for the affected directory.

.agents/skills/debug-issue-with-datadog/references/datadog-playbook.md

@@ -0,0 +1,141 @@
+# Datadog Query Playbook
+
+The default for this skill is the **broad sweep**: APM spans + logs + metrics
++ monitors + incidents, run against both EU and US sites unless intake clearly
+localizes. Cluster results before drilling in.
+
+Two MCP servers are typically connected — one for `datadoghq.eu`, one for
+`datadoghq.com`. Run the same query on both and compare. The contrast itself
+is often the most informative finding (e.g. LFE-9475's 23.5% EU vs 0.7% US
+error rate immediately ruled out PostHog Cloud as the global cause).
+
+## Tag Vocabulary
+
+- **Site:** `datadoghq.eu` for EU, `datadoghq.com` for US.
+- **Region tag (`env`):** `prod-eu`, `prod-us`, `prod-hipaa`, `prod-jp`.
+- **Service:** `worker` (default in `worker/src/env.ts`) or `web` (default in
+  `web/src/env.mjs`). Some deployments override to `langfuse`.
+- **Span resource names** for worker async jobs follow the pattern
+  `process <queue-name>` — see `repo-debug-map.md`.
+
+## 1. APM Span Sweep — find the failing handler
+
+Use `aggregate_spans` first; only fetch individual traces once a cluster is
+identified.
+
+Starter shape (rename the resource for the relevant subsystem):
+
+```text
+service:worker resource_name:"process posthog-integration-project" status:error
+```
+
+Aggregations to run, in order:
+
+1. Count by `env` — confirms region split.
+2. Count by `error.message` — primary error classes.
+3. Count by `(projectId, error.message)` — which tenants are affected, by
+   class. `projectId` lives on span tags as `@projectId` for log search and as
+   a tag for spans (depends on instrumentation site).
+4. p50 / p95 / p99 duration by region — fingerprints timeouts vs. crashes vs.
+   slow successes.
+
+If `aggregate_spans` returns no results, check:
+
+- the resource name is right (case-sensitive, see `repo-debug-map.md`);
+- the time window covers when the issue was actually firing;
+- the region tag matches reality (the EU MCP only sees EU traces).
+
+### Public API tenant / legacy-endpoint usage
+
+For tenant-specific public API route usage, use
+[`../../datadog-query-recipes/references/public-api-tenant-usage.md`](../../datadog-query-recipes/references/public-api-tenant-usage.md).
+The key gotcha is that tenant tags usually live on the `api-auth-verify` child
+span while the HTTP route lives on the request root span, so correlate by
+`traceid` rather than relying on one combined span filter.
+
+## 2. Log Sweep — read what the handler said
+
+Logs are the right tool for *messages* the handler emitted. Spans are the
+right tool for *which handler invocations failed*.
+
+Starter shapes:
+
+```text
+service:worker env:prod-eu @projectId:cm1r6u1iq00ccfvrkoy8vg3ms status:error
+service:worker env:prod-eu "[POSTHOG]" status:error
+```
+
+Useful log facets:
+
+- `@projectId` or `@langfuse.project.id` — Langfuse project (cuid).
+- `@error.kind` / `@error.message` / `@error.stack` — when the Winston logger
+  serialized an Error.
+- `@queue` / `@jobName` — when set by BullMQ instrumentation.
+
+For high-volume subsystems (`ingestion-queue`, `otel-ingestion-queue`),
+prefer `analyze_datadog_logs` with grouping over `search_datadog_logs` — the
+raw matches are too noisy.
+
+## 3. Metric Sweep — confirm the trend
+
+Pick 2–3 metrics that match the subsystem. Common ones:
+
+- `trace.bullmq.process.errors` and `trace.bullmq.process.duration` —
+  per-queue health from the BullMQ OTel instrumentation. Filter by
+  `resource_name:"process <queue-name>"`.
+- `trace.http_request.errors` and `trace.http_request.duration` for HTTP
+  handlers (`service:web`).
+- ClickHouse: cluster-level `clickhouse.query.duration`,
+  `clickhouse.memory_usage` — the worker doesn't emit these directly, they
+  come from the `clickhouse` integration in the infra repo.
+- Postgres: `aurora.databaseconnections`, `aurora.deadlocks` — relevant when
+  the symptom is `connection_limit` / `connection pool` errors.
+
+If the subsystem isn't already known, run `search_datadog_metrics` for the
+subsystem name and pick the obvious counter / gauge / histogram triplet.
+
+## 4. Monitors & Incidents
+
+- `search_datadog_monitors` for the subsystem name — tells you what alerts
+  *would* have fired and what their thresholds are. A muted monitor on the
+  affected subsystem is itself a finding (see LFE-9475: "EU alert muted for
+  a week").
+- `search_datadog_incidents` for the time window — links any pre-existing
+  incident the user may not have referenced.
+
+## 5. RUM / Frontend (only when the symptom is user-facing)
+
+Skip unless the issue is "page broken" / "slow load". Then:
+
+- `search_datadog_rum_events` filtered by `@view.url:` patterns matching the
+  affected route.
+- Cross-reference with `service:web` API errors at the same time.
+
+## 6. Trace Drill-Down
+
+Once a cluster is identified, fetch one or two representative traces with
+`get_datadog_trace` to read the actual stack and confirm where in the handler
+the throw originates. This is what lets you point at a specific file and
+line range in the analysis.
+
+## Anti-Patterns
+
+- Don't fetch individual logs/traces before aggregating. You'll burn context
+  on noise and miss the cluster pattern.
+- Don't trust a single-region query as global. Always compare EU and US.
+- Don't read an `error.message` literally if it goes through a custom error
+  wrapper — `validateWebhookURL` rejections, for example, are re-logged as
+  "DNS lookup failed" but are actually validator rejections.
+- Don't assume monitors are firing just because errors exist — check if the
+  monitor is muted.
+
+## Linking Out
+
+End the analysis with the actual Datadog UI URLs you queried, e.g.:
+
+```text
+https://app.datadoghq.eu/apm/traces?query=resource_name%3A%22process+posthog-integration-project%22+status%3Aerror
+https://app.datadoghq.com/apm/traces?query=resource_name%3A%22process+posthog-integration-project%22+status%3Aerror
+```
+
+so the human reader can re-run the same query.

.agents/skills/debug-issue-with-datadog/references/intake.md

@@ -0,0 +1,72 @@
+# Intake — What Are We Actually Investigating?
+
+Goal: extract every signal from the input *before* you touch Datadog. Cheap
+to do, expensive to skip — the wrong time window or wrong service tag can
+hide the real cause.
+
+## Source-by-Source Recipes
+
+### Linear issue (URL or `LFE-XXXX` ID)
+
+1. Fetch the issue via the Linear MCP:
+   `mcp__d39d26f2-…__get_issue` with `id: "LFE-XXXX"` and
+   `includeRelations: true`.
+2. Fetch comments separately:
+   `mcp__d39d26f2-…__list_comments` with the same `issueId`.
+3. Note: Langfuse triages **inside the issue description**. The description
+   often grows reaction sections (`projectId: <one-line note>`) as oncall
+   investigates. Treat both description and comments as authoritative state.
+4. Pull `attachments` for linked PRs/commits — these show what's already been
+   tried. Reading the diff of a half-merged fix often reveals the original
+   theory of the bug.
+5. Pull labels (`integration-posthog`, `feat-exports`, etc.) — they map
+   directly to the subsystem clusters in `repo-debug-map.md`.
+
+### GitHub issue
+
+1. `gh issue view <url-or-number> --json title,body,labels,comments,assignees`.
+2. If there's a stack trace in the issue body, copy the top frames into your
+   notes — those frames usually map straight to a handler file.
+
+### Pasted error / incident text
+
+1. Treat as the issue description. If it's a stack trace, identify:
+   - the throwing module (handler vs. SDK vs. infra),
+   - whether it looks like a *user-input* failure (HTTP 4xx, validation, auth)
+     or an *infra* failure (5xx, timeout, OOM, DNS, pool exhaustion),
+   - the error class (`Error`, `TypeError`, `PrismaClientKnownRequestError`,
+     etc.).
+2. If a `projectId` appears, that's gold — anchor every Datadog query on it.
+3. If a `traceId` (Datadog's, not Langfuse's) appears, jump straight to
+   `get_datadog_trace`.
+
+## Extract The Following Before Querying
+
+Build a small notes block. If a value is missing, mark it `?` rather than
+guessing — Datadog will tell you what's missing.
+
+- **Subsystem.** PostHog integration, blob-storage export, evaluation
+  execution, OTel ingestion, batch action, webhook delivery, etc. Map to
+  `repo-debug-map.md`.
+- **Region(s).** `prod-eu` / `prod-us` / `prod-hipaa` / `prod-jp`. If unknown,
+  query both EU and US.
+- **Service.** Most issues are `service:worker`; UI/API timeouts are
+  `service:web`. Check for both when unsure.
+- **Time window.** Default to 7 days back from the issue's `createdAt`. If
+  the issue references a specific incident or alert, use that window ±1 day.
+- **`projectId`(s).** Project IDs in Langfuse are `cuid`-shaped
+  (`cl…` / `cm…`, 25 chars). The reaction blocks in Linear descriptions
+  often *are* lists of affected project IDs.
+- **Error message fragments.** Exact substrings to grep for in DD logs:
+  `Header overflow`, `Timeout error.`, `HTTP 403`, `DNS lookup failed`,
+  `Cannot write to canceled buffer`, `connection pool`, etc.
+- **Already-attempted fixes.** Linked PRs/commits on the issue. Read their
+  diffs — your analysis must not re-recommend something that's already
+  shipped.
+
+## Output Of The Intake Step
+
+A short bullet list (not yet formatted as the final analysis) with each of
+the above filled in. The remaining steps key off this — `datadog-playbook.md`
+expects subsystem + region + window, `repo-debug-map.md` expects subsystem,
+and the output template wants the affected projects.

.agents/skills/debug-issue-with-datadog/references/output-template.md

@@ -0,0 +1,127 @@
+# Output Template
+
+The analysis should be structured so it can be pasted directly as the first
+investigative comment on the Linear issue. The example to anchor on is the
+first comment on `LFE-9475` (PostHog Integration Processing Failures).
+
+Findings come first, recommendations last. If the data doesn't support a
+hypothesis, say so — do not invent root causes to fill the template.
+
+## Section Order
+
+1. **Header** — data source, time window, scope of sweep.
+2. **Volume & error-rate split** — table by region (always).
+3. **Hotspots** — table by `(projectId, dominant cause)` or by cluster.
+4. **Root cause by error class** — one numbered subsection per cluster.
+5. **Suggested patches** — P0 / P1 / P2, each with file paths and a short
+   code sketch.
+6. **Dashboards** — Datadog UI URLs for the queries you ran.
+
+## Skeleton (fill in with your findings)
+
+````markdown
+## Datadog APM + log analysis (<N>-day window, <YYYY-MM-DD> → <YYYY-MM-DD>)
+
+Source: APM spans with `resource_name:"process <queue-name>"` across EU and US.
+
+### Volume & error rate — <one-line summary of regional split>
+
+| Region | Total spans | Errors | Error rate |
+|---|---|---|---|
+| EU (`prod-eu`) | <n> | <n> | **<pct>%** |
+| US (`prod-us`) | <n> | <n> | **<pct>%** |
+
+<One sentence explaining where the noise actually lives.>
+
+### Hotspots — concentrated on ~<N> <region> projects
+
+<Region> errors break down by `(projectId, error.message)`:
+
+| ProjectId | Errors | Dominant cause |
+|---|---|---|
+| `<projectId>` | <n> | `<error message>` (<n>) + others |
+| ...           | ... | ...                              |
+
+<Optional: contrast with another subsystem if relevant — e.g.
+"Unlike blob storage, PostHog has multiple distinct root causes — not one
+hotspot pattern.">
+
+## Root cause by error class
+
+### 1. `<error message>` — <n> errors, <n> projects
+<2–4 sentences explaining what this error class actually is at the
+implementation level (which library, which call site). Then list candidate
+causes in order of likelihood. Mark which ones are confirmed by the data
+vs. speculative.>
+
+### 2. `<error message>` — <n> errors, mostly <n> projects
+<Same pattern.>
+
+### 3. <next class>
+<...>
+
+### <N>. <Symptom of upstream failure>
+<Use this slot when a class is a *symptom* of another class rather than an
+independent bug — call it out so suggested patches don't double-count.>
+
+## Suggested patches
+
+### P0 — <one-line summary, e.g. "Auto-disable integrations on persistent
+auth failures">
+<Why this is P0 — what noise it kills, what data it stops corrupting, what
+unblocks downstream work.>
+
+```ts
+// <relative path from repo root>
+// Short code sketch (5–20 lines). It does not need to compile —
+// it must communicate the shape of the change.
+```
+
+### P0 — <next P0>
+<...>
+
+### P1 — <smaller / less urgent fix>
+<Same shape.>
+
+### P2 — <separate-but-surfaced finding>
+<E.g. a Prisma pool sizing issue surfaced incidentally by this analysis but
+not the original bug. Call it out with its own section so it doesn't get
+lost.>
+
+### Regional split explanation (only if relevant)
+<One paragraph explaining why EU vs. US asymmetry exists — usually not an
+infra bug, just where the affected tenants happen to live.>
+
+Dashboards:
+- EU APM: <url>
+- US APM: <url>
+- (logs / metrics / monitor links as relevant)
+````
+
+## Style Rules
+
+- Lead with numbers, not adjectives. "23.5% error rate" beats "very noisy".
+- Distinguish **primary causes** from **symptoms** explicitly. Symptoms
+  shouldn't get their own P0 patch.
+- Always cite specific files when proposing a code change. A patch
+  recommendation without `worker/src/features/<…>/<file>.ts` is unfinished.
+- Code sketches are illustrative — clearly mark them as sketches if they
+  hand-wave types. The next agent / human will write the real diff.
+- If the analysis surfaces a finding *outside* the original ticket scope
+  (e.g. a Prisma pool issue while debugging PostHog), include it as a P2
+  with a sentence explaining it's separate.
+- If the data refuses to converge on a single root cause, say so. The
+  template handles N classes — use as many subsections as the data warrants.
+
+## When To Skip Sections
+
+- **Single-region deployments:** if the issue clearly affects only one
+  region, you can replace the "Volume & error rate" table with a single-row
+  variant, but still note that the other region was checked and clean.
+- **No code change recommended:** if the only finding is "the affected
+  tenants have misconfigured credentials and we should reach out", the
+  Suggested-patches section can be a single sentence — but still include
+  the dashboards.
+- **Aborted investigation:** if Datadog access fails or the data is
+  insufficient, write what you tried, what was missing, and what would let
+  the next investigator pick it up.

.agents/skills/debug-issue-with-datadog/references/repo-debug-map.md

@@ -0,0 +1,100 @@
+# Repo Debug Map — Subsystem → Code → Datadog Filters
+
+For each subsystem we ship monitors and incidents on, this is the canonical
+map between the symptom, the Datadog query that surfaces it, and the source
+files where the bug almost certainly lives.
+
+When intake gives you a subsystem (PostHog, evals, exports, etc.), start
+here to pick the right Datadog filters and the right files to read.
+
+## Worker Async Jobs
+
+Worker handlers are wrapped by `instrumentAsync` in their queue file. The
+span resource name follows the pattern `process <queue-name>`. Queue and job
+name constants live in
+`packages/shared/src/server/queues.ts`
+(`QueueName` and `QueueJobs` enums).
+
+| Subsystem | Queue file | Handler dir | Span `resource_name` | Log prefix |
+| --- | --- | --- | --- | --- |
+| PostHog integration | `worker/src/queues/postHogIntegrationQueue.ts` | `worker/src/features/posthog/` | `process posthog-integration-project` | `[POSTHOG]` |
+| Mixpanel integration | `worker/src/queues/mixpanelIntegrationQueue.ts` | `worker/src/features/mixpanel/` | `process mixpanel-integration-project` | `[MIXPANEL]` |
+| Blob storage export | `worker/src/queues/blobStorageIntegrationQueue.ts` | `worker/src/features/blobstorage/` | `process blob-storage-project` | `[BLOBSTORAGE]` |
+| Data retention | `worker/src/queues/dataRetentionQueue.ts` | `worker/src/features/batch-data-retention-cleaner/` | `process data-retention-project` | n/a |
+| Event propagation | `worker/src/queues/eventPropagationQueue.ts` | `worker/src/features/eventPropagation/` | `process event-propagation` | n/a |
+| Cloud usage metering | `worker/src/queues/cloudUsageMeteringQueue.ts` | `worker/src/ee/` (cloud-only) | `process cloud-usage-metering` | n/a |
+| Free-tier usage threshold | `worker/src/queues/cloudFreeTierUsageThresholdQueue.ts` | `worker/src/ee/usageThresholds/` | `process cloud-free-tier-usage-threshold` | n/a |
+| Ingestion (single event) | `worker/src/queues/ingestionQueue.ts` | `worker/src/features/ingestion/` (and `IngestionService`) | BullMQ default span | n/a |
+| OTel ingestion | `worker/src/queues/otelIngestionQueue.ts` | `worker/src/features/otel/` | BullMQ default span | n/a |
+| Evaluation execution | `worker/src/queues/evalQueue.ts` | `worker/src/features/evaluation/` | BullMQ default span | n/a |
+| Batch export | `worker/src/queues/batchExportQueue.ts` | `worker/src/features/batchExport/` | BullMQ default span | n/a |
+| Webhook delivery | `worker/src/queues/webhooks.ts` | `worker/src/features/webhooks/` | BullMQ default span | n/a |
+| Trace / score / dataset / project delete | `worker/src/queues/{traceDelete,scoreDelete,datasetDelete,projectDelete}.ts` | `worker/src/features/traces/`, `…/scores/`, `…/datasets/` | BullMQ default span | n/a |
+
+For queues using BullMQ default spans (no `instrumentAsync` wrapper), search
+APM with `service:worker operation_name:bullmq.process` filtered by
+`bullmq.queue:<queue-name>`. For queue inventory, sharded queue naming, and
+queue metric recipes, use
+[`../../datadog-query-recipes/references/queue-consumers.md`](../../datadog-query-recipes/references/queue-consumers.md).
+
+## Web (Next.js / tRPC / public API)
+
+| Subsystem | Code | Span / log filter |
+| --- | --- | --- |
+| Public REST API | `web/src/pages/api/public/**` | Request span: `service:web resource_name:"GET /api/public/<path>"`; tenant span: `resource_name:api-auth-verify` with `@langfuse.project.id` / `@langfuse.org.id` |
+| tRPC procedures | `web/src/server/api/routers/**` | `service:web resource_name:"POST /api/trpc/<router>.<proc>"` |
+| Auth / API key verification | `web/src/features/public-api/server/apiAuth.ts` | look for `verifyAuthHeaderAndReturnScope` spans |
+| Stripe billing | `web/src/ee/features/billing/server/stripeBillingService.ts` | wrapped in `instrumentAsync`; spans named after the method |
+
+For tenant-specific public API usage questions, first query
+`resource_name:api-auth-verify` by `@langfuse.project.id` or
+`@langfuse.org.id`, then open representative trace IDs and inspect the request
+root span for `http.path_group`, `http.route`, and `http.target`. The tenant
+tags and endpoint path are usually on different spans, so a single-span query
+combining both may return no results even when the trace proves usage.
+For the full reusable recipe, use
+[`../../datadog-query-recipes/references/public-api-tenant-usage.md`](../../datadog-query-recipes/references/public-api-tenant-usage.md).
+
+## Shared Layers
+
+These are not subsystems on their own, but are *frequently the actual cause*
+behind a worker subsystem failure.
+
+| Layer | Location | Common failure modes |
+| --- | --- | --- |
+| ClickHouse access | `packages/shared/src/server/clickhouse/`, `packages/shared/src/server/repositories/` | OOM (`Code: 241`), buffer cancel (`Code: 734`), JOIN spills, slow queries on un-pre-filtered traces |
+| Prisma access | `packages/shared/src/db.ts` and per-feature repos | `connection pool timeout` (worker default `connection_limit=5`), N+1 queries |
+| Queue contracts | `packages/shared/src/server/queues.ts` | wrong queue name, missing schema validation |
+| Logger / instrumentation | `packages/shared/src/server/logger.ts`, `packages/shared/src/server/instrumentation.ts` | log silently dropped because `LANGFUSE_LOG_LEVEL` set wrong, or span missing because handler doesn't call `instrumentAsync` |
+| Webhook URL validation | `packages/shared/src/server/validateWebhookURL.ts` | rejects with messages that *look* like DNS errors but are SSRF guard rejections |
+| Encryption | `packages/shared/encryption` | bad keys → 403/auth-style failures masquerading as upstream errors |
+
+## Common Symptoms → First Files To Read
+
+- **"403 from upstream":** check the per-integration credentials table in
+  Postgres (`PostHogIntegration`, `BlobStorageIntegration`, `WebhookConfig`,
+  etc.) and the encryption layer.
+- **"Timeout":** check the SDK timeout default and the per-stream
+  flush/batch size in the handler. Worker async jobs default to long-running
+  but the upstream SDK does not.
+- **"DNS lookup failed":** distinguish actual DNS from `validateWebhookURL`
+  rejection. The error message wrapping is misleading on purpose.
+- **"Cannot write to canceled buffer" (CH):** ClickHouse stream wasn't
+  aborted when the downstream consumer threw. Look for an `AbortController`
+  threaded through the handler.
+- **"Connection pool timeout" (Prisma):** worker `connection_limit` is set
+  in the connection string; jobs doing per-row `findFirst()` exhaust it.
+  Check whether the integration row could be cached in closure scope.
+- **"memory limit exceeded" (CH):** look for unbounded JOINs without a
+  pre-filter CTE, especially in analytics integrations.
+- **"Header overflow":** Node HTTP parser's default 80 KB ceiling. Either
+  raise `--max-http-header-size` for the worker, or replace the SDK's HTTP
+  client.
+
+## Where to Look for Already-Shipped Fixes
+
+Before recommending a patch, confirm it isn't already merged or in flight:
+
+- `attachments` on the Linear issue (PRs and commits are auto-linked).
+- `git log --oneline --since=<recent-window> -- <handler-path>`.
+- Open PRs touching the file via `gh pr list --search "<filename>"`.

.agents/skills/frontend-browser-review/SKILL.md

@@ -0,0 +1,60 @@
+---
+name: frontend-browser-review
+description: |
+  Shared workflow for browser-based review of user-visible frontend changes in Langfuse.
+  Use when a change affects UI behavior, layout, styling, navigation, or browser-visible
+  regressions and should be checked with the Playwright MCP server before signoff.
+---
+
+# Frontend Browser Review
+
+Use this skill when a change affects what users see or do in the browser.
+
+## Start Here
+
+- Read [`../../../web/AGENTS.md`](../../../web/AGENTS.md) for web-specific
+  entry points and test commands.
+- Use the workspace `playwright` MCP server configured from the repo-owned
+  shared agent setup.
+
+## When To Use It
+
+- UI changes in `web/**`
+- Layout, styling, or responsive behavior changes
+- Changes to navigation or page flows
+- Bug fixes where the failure mode is visible in the browser
+- Final signoff for user-visible frontend work
+
+## Review Loop
+
+1. Start the app with `pnpm run dev:web` unless an existing local server is
+   already running.
+2. Install Chromium with `pnpm run playwright:install` if Playwright has not
+   been set up on the machine yet.
+3. Open the primary changed flow with the Playwright MCP server.
+4. Exercise the main happy path affected by the change.
+5. Check for obvious visual regressions:
+   - broken layout or spacing
+   - banner overlap or viewport anchoring issues
+   - missing loading, empty, or error states
+   - broken responsive behavior on narrow widths
+6. If the page changed materially, inspect the resulting UI state and compare
+   it against the intended behavior from the task or existing patterns.
+7. If the browser session fails, inspect traces and artifacts under
+   `/tmp/playwright-mcp`.
+
+## Output Expectations
+
+Report:
+
+1. What flow you reviewed
+2. Whether the primary flow worked
+3. Any visible regressions or follow-up risks
+4. If review was blocked, exactly what prevented browser verification
+
+## Scope Notes
+
+- This skill complements, not replaces, targeted tests and linting.
+- For implementation details, stay in `web/AGENTS.md` and package-local skills.
+- Use this as the browser-signoff workflow, not as a generic frontend coding
+  guide.

.agents/skills/frontend-large-feature-architecture/SKILL.md

@@ -0,0 +1,74 @@
+---
+name: frontend-large-feature-architecture
+description: |
+  Use when building, changing, or refactoring large Langfuse frontend features,
+  virtualized lists, large tables, controller components, local feature state,
+  Zustand stores, row selection, high-frequency UI state, or
+  rendering-performance issues.
+---
+
+# Frontend Large Feature Architecture
+
+Use this skill when building, changing, or refactoring a large frontend
+surface.
+
+In this skill, "controller" means a component or hook that owns feature logic:
+data fetching, view state, table/list state, effects, actions, and expensive
+rendering. The problem is not the name; it is one place owning too many
+changing responsibilities.
+
+## Big Feature Rules
+
+When a feature grows, split rendering from logic. Most components should be
+view-only. Data preparation should be pure. Complex user actions should live in
+external async functions or local-store actions. Effects are integration
+boundaries, not the normal way to derive state.
+
+For the full rules, read
+[`references/big-feature-rules.md`](references/big-feature-rules.md).
+
+## Required Model
+
+- The page/view owns lifecycle and creates feature-scoped dependencies.
+- Server/query state stays in tRPC/React Query; route state stays in the
+  router/filter hooks.
+- High-frequency feature UI state belongs in a per-mount local vanilla Zustand
+  store. Create it with lazy `useState`, not `useMemo`.
+- Global stores are only for truly cross-feature, cross-route product state.
+- React context may provide a stable store or action owner. Do not put
+  frequently changing state directly in provider values.
+- Rendered rows/cells/items should be view-only or narrow containers. Put
+  effects, subscriptions, data loading, and workflows outside expensive views.
+- Shared `src/components/*` exports should stay context-free or receive
+  explicit props. Put view-scoped Zustand consumers in `src/features/*`.
+- Large feature folders should have a concise `README.md` owner map.
+- Migrate real features through small PRs that improve one state boundary,
+  action workflow, data-preparation seam, or render boundary.
+
+## Local Store Default
+
+Prefer a local vanilla Zustand store for large or high-frequency feature state.
+The store is created by the page/view and destroyed on unmount.
+
+Use selectors that return primitives or stable references. If a component needs
+multiple values, use shallow selector helpers or split subscriptions so one
+changing field does not rerender unrelated UI.
+
+Complex user workflows should live in `actions/*.ts` files or store actions.
+The component wires hooks and passes dependencies; the action owns the workflow.
+
+## When To Read References
+
+- For the core big-feature rules and migration reality, read
+  [`references/big-feature-rules.md`](references/big-feature-rules.md).
+- For virtualized lists, translated DOM, row measurement, and scroll rerenders,
+  read [`references/virtualized-lists.md`](references/virtualized-lists.md).
+- For local feature stores and splitting large components, read
+  [`references/local-feature-state.md`](references/local-feature-state.md).
+- For feature `README.md` owner maps, read
+  [`references/feature-readmes.md`](references/feature-readmes.md).
+- For a step-by-step migration from a controller component to a managed feature
+  pattern, read
+  [`references/controller-migration.md`](references/controller-migration.md).
+  This is the default reference for traces/observations tables, sessions,
+  experiments, prompts, evals, datasets, and other controller-heavy surfaces.

.agents/skills/frontend-large-feature-architecture/agents/openai.yaml

@@ -0,0 +1,7 @@
+interface:
+  display_name: "Frontend Feature Architecture"
+  short_description: "Build, change, or refactor React features"
+  default_prompt: "Use $frontend-large-feature-architecture to build, change, or refactor this large frontend surface with clear state ownership, stable subscriptions, and view-only render boundaries."
+
+policy:
+  allow_implicit_invocation: true

.agents/skills/frontend-large-feature-architecture/references/big-feature-rules.md

@@ -0,0 +1,98 @@
+# Big Feature Rules
+
+Use these rules when a feature keeps growing and the instinct is to add one
+more state variable, prop, callback, or effect to the same component.
+
+Here, "controller" means a component or hook that owns feature logic: state,
+effects, data loading, subscriptions, actions, and derived data. Some
+controller code is necessary. The failure mode is one controller owning too many
+changing responsibilities and waking too much UI.
+
+## Ownership Baseline
+
+- The page/view owns lifecycle and creates feature-scoped dependencies.
+- Server/query state stays in tRPC/React Query.
+- Route state stays in router/filter hooks.
+- High-frequency local UI state belongs in a per-mount feature store.
+- Global stores are only for product state shared across routes or features.
+- Pure data preparation turns fetched data into UI data before rendering.
+- Complex workflows live in `actions/*.ts` or store actions.
+- Effects are integration boundaries, not ordinary state derivation.
+- Expensive rows/cells/items should be view-only behind narrow containers.
+
+## Hard Rules
+
+1. Growth usually means React rendering and feature logic need to be separated.
+   A bigger component is not an architecture.
+2. Most components should be view-only. They render what they are given, or
+   they read a small selected value from a context-available feature store.
+3. Complicated data preparation belongs in separate pure functions. Backend
+   data should flow one way: fetched data -> compiled UI data -> render.
+4. User actions that trigger complex logic should live outside render
+   components as named async functions or store actions. If the action needs a
+   lot of context, pass the feature store instance and let the action read what
+   it needs.
+5. A feature with both local store state and React Query data needs an explicit
+   bridge in a custom hook or named action.
+6. Prefer view-scoped local feature stores over global stores for page-instance
+   state. Filters, selected rows, expanded rows, lazy load state, drawers, and
+   local actions usually belong to one mounted page instance.
+7. Create local store instances with a lazy `useState` initializer, not
+   `useMemo`. The store is a per-mount instance, not a render-time derived
+   value.
+8. Effects should not be normal state mutators. Before adding an effect, try
+   pure data preparation, a store action, or an explicit event handler.
+9. Do not copy existing large Langfuse feature components as examples. Treat
+   them as legacy unless they follow this skill.
+10. Prefer small reliable PRs over a large rewrite. Each PR should improve one
+    state boundary, action workflow, data-preparation seam, or render boundary,
+    then update the feature README or migration note with the next slice.
+
+## Migration Reality
+
+Most large frontend features are not yet in the ideal shape. Traces,
+observations, experiments, prompts, evals, datasets, and session views all have
+some controller-heavy surfaces. Do not copy an existing large component just
+because it works today. Treat it as a migration candidate and move it one
+state/action/data-preparation boundary at a time.
+
+For the step-by-step path, read `controller-migration.md`.
+
+## Small PR Policy
+
+A good big-feature PR is intentionally incomplete. It should change one
+semantic interaction and keep behavior stable: one selection boundary, one
+action workflow, one pure data-preparation helper, or one render boundary.
+
+Do not bundle file reorganization, state extraction, visual changes, and
+behavior fixes in the same PR unless the user explicitly asks for that scope.
+When a reorganization is needed, prefer a rename-only PR first or a later
+follow-up PR.
+
+## Effects And Actions
+
+Effects are for subscriptions, observers, imperative third-party APIs,
+one-time initialization, and cleanup. Keep them in containers or feature hooks,
+not view components. If an effect writes state repeatedly, make the store action
+idempotent.
+
+Complex actions should be callable without rendering a component. Put workflows
+in `actions/*.ts` or named store actions. Components wire hooks and pass
+dependencies; actions own the workflow.
+
+```ts
+await applyBulkAction({
+  store,
+  queryClient,
+  projectId,
+});
+```
+
+Actions must not call React hooks. Pass hook results, query helpers, the local
+store instance, or narrow callback dependencies into the action. If an action
+needs substantial data preparation, export a pure helper next to it so the
+transformation can be tested independently.
+
+Do not pass twenty props through the tree so a button can do feature-level work,
+and do not leave complex workflows inline in a page controller because that
+controller happened to have all dependencies in scope.

.agents/skills/frontend-large-feature-architecture/references/controller-migration.md

@@ -0,0 +1,123 @@
+# Controller Migration Guide
+
+Use this when a frontend surface has grown into one component or hook that owns
+data fetching, route glue, filters, table/list state, selection, drawers,
+actions, and expensive rendering.
+
+The target is clear ownership, not "everything in a store." Start from the
+ownership baseline in `big-feature-rules.md`.
+
+Most existing features are not there yet. Migrate in narrow slices and keep
+behavior stable.
+
+## Realistic Migration Strategy
+
+Do not start by designing the perfect final feature. Start by making the next
+change safer than the previous one.
+
+For each migration PR, write down:
+
+- the current controller problem being targeted
+- the single state/action/data-preparation/render boundary being improved
+- what behavior must remain unchanged
+- what instrumentation or tests prove the slice worked
+- what still remains spread across the feature
+- the next recommended atomic slice
+
+This is how large features become managed features without review-hostile
+rewrites. The feature README is the living owner map; update it in place as the
+feature moves forward.
+
+## Step-by-Step Path
+
+1. **Map the controller.** List every state group, query, derived value, effect,
+   callback, action, and expensive child render owned by the component.
+2. **Classify state.** Separate server/query, route, persisted browser,
+   high-frequency local UI, derived view data, imperative integration, and
+   one-off modal/form state.
+3. **Instrument one symptom.** Pick a concrete interaction such as row
+   selection, scroll, filter change, drawer open, form step change, or
+   saved-view change. Measure what rerenders, remounts, refetches, or
+   recalculates.
+4. **Choose one boundary.** Start with the smallest high-value boundary:
+   selection, lazy-row state, batch action workflow, filter-target mapping,
+   column/view state, wizard step state, or pure data preparation. Do not
+   migrate everything at once.
+5. **Choose the lightest tool.** A pure helper or action extraction may be the
+   right first PR. Add a local store only when selective subscriptions or
+   per-mount persistence are needed.
+6. **Create a local store instance when needed.** Use lazy `useState` in the
+   page/view:
+
+   ```ts
+   const [store] = useState(() => createFeatureStore(initialState));
+   ```
+
+   Provide only this stable store instance through context.
+
+7. **Move mutations into named actions.** Put state-changing logic in store
+   actions or external action functions. Keep components responsible for user
+   events, not workflows.
+8. **Split containers from views.** Containers may subscribe, call hooks, or
+   fetch data. Views should render props or tiny selected values.
+9. **Move data preparation out of render.** Put expensive or complicated
+   transformations in pure functions. Backend data should flow into compiled UI
+   data, then into rendering.
+10. **Bridge query state explicitly.** If local store decisions depend on React
+    Query data, use a named feature hook or action to express that relationship.
+11. **Isolate imperative integration.** Virtualizers, observers, keyboard
+    listeners, third-party DOM mutation handling, and timers belong in narrow
+    integration hooks.
+12. **Update the feature README.** Record what this PR improved, desired
+    boundaries, known spread state, and the next extraction target.
+13. **Remove debug instrumentation.** Temporary logs are useful during
+    migration, but should not survive the slice.
+14. **Repeat.** Each slice should make one semantic interaction narrower and
+    easier to reason about.
+
+## Feature-Specific First Slices
+
+Use the feature's current shape to pick the first slice:
+
+- **Traces and observations tables**: start with row selection, select-all,
+  batch actions, or expensive cell wrappers.
+- **Session detail and session events**: isolate virtualization, lazy-row load
+  state, and dynamic measurement from rendered row content.
+- **Experiment result tables**: start with selected-row state, filter-target
+  mapping, run/evaluation batch actions, or pure helpers for comparison column
+  construction.
+- **Experiment creation wizards**: separate submitted form data from display
+  state such as active step, selected prompt labels, schema display names, and
+  evaluator selection.
+- **Prompt management**: split prompt detail route/query state, label/version
+  selection, prompt history data preparation, and mutation workflows.
+- **Eval template and evaluator forms**: extract form defaults, model/provider
+  preparation, validation helpers, and submit workflows before adding a store.
+- **Datasets and dataset runs**: isolate active-cell/compare-field state,
+  table selection, run comparison preparation, and upload/import workflows.
+
+If a feature already has hooks for part of this work, treat them as partial
+migration, not proof that the whole surface is healthy.
+
+## Acceptance Criteria
+
+- A local state change wakes only components that selected that state.
+- Page components no longer rebuild columns, filters, row wrappers, and action
+  callbacks for unrelated row-level changes.
+- Expensive rows/cells are view-only behind narrow containers.
+- Effects are integration boundaries or one-time initialization, not ordinary
+  data derivation.
+- Complex workflows can be called without rendering the page.
+- The feature README tells the next developer what has improved, what remains
+  spread, and what the next small PR should target.
+
+Avoid:
+
+- Replacing a giant component with a giant global store.
+- Moving all state at once without measured acceptance criteria.
+- Treating memoization as the architecture.
+- Putting provider-coupled store hooks into shared `src/components/*` exports.
+- Leaving a workflow inline in the page because the page had every dependency in
+  scope.
+- Using the README as a victory statement while hiding remaining controller
+  state. Be explicit about remaining debt.

.agents/skills/frontend-large-feature-architecture/references/feature-readmes.md

@@ -0,0 +1,89 @@
+# Feature READMEs
+
+Large frontend feature folders should have a short `README.md` that acts as an
+owner map for humans and agents. Prefer `README.md` over `FEATURE.md` to match
+the existing `web/src/features/*` convention. Use `FEATURE.md` only if a folder
+already has a user-facing or generated README.
+
+The README is not a changelog. It should describe durable boundaries and point
+to deeper migration notes.
+
+## Required Sections
+
+- **Surface**: what product surface the folder owns.
+- **Entry Points**: route files or parent components that mount the feature,
+  and the page/view lifecycle owner files they call.
+- **Structure**: what each subfolder owns. Use root `components/` only for
+  components reused across surfaces inside the feature. Use surface folders such
+  as `detail/` for page controllers, local stores, `actions/`, integration
+  hooks, and surface-private containers.
+- **External Consumers**: other features that import these components. This
+  keeps shared exports context-free and prevents accidental provider coupling.
+- **State Ownership**: where server/query state, route state, local feature
+  state, global product state, DOM integration state, and view-only props live.
+- **Performance And Stability Boundaries**: which interactions are
+  high-frequency or externally unstable, and which components are allowed to
+  rerender or measure because of them.
+- **Migration State**: what has already been improved, what state/actions are
+  still spread, and the next one or two atomic slices.
+- **Development Context**: agent skills, migration notes, or issue docs to read
+  before extending the feature.
+
+## State Ownership Rules
+
+Use the ownership baseline in `big-feature-rules.md`. The README only needs to
+name where each state category lives in this feature and where known spread
+state remains.
+
+## Performance And Stability Map
+
+Every large feature README should name the high-frequency interactions that
+must stay narrow. Typical examples:
+
+- scroll and virtualization updates
+- row selection, hover, expansion, and lazy-loading
+- filter, saved-view, and column-state changes
+- drawers, peek navigation, and keyboard navigation
+- browser translation or other third-party DOM mutation
+- resize and dynamic row measurement
+
+For each interaction, state which boundary should update. The page component
+should not rerun expensive data preparation, recreate column/config objects, or
+rerender unchanged expensive cells for unrelated state changes.
+
+## Current-State Honesty
+
+If a feature is mid-migration, say so. The README should make the desired
+boundaries clear while naming known spread state as debt. Do not present a
+partially migrated feature as the final pattern.
+
+Use an update-in-place style rather than a changelog. For example:
+
+- **Improved in current shape**: local store owns row selection; export action
+  moved to `actions/exportFeatureData.ts`; row view no longer subscribes to
+  filter state.
+- **Still spread**: saved-view state remains in the page controller; filter
+  option preparation is still inline; mutation workflows still close over page
+  hooks.
+- **Next slice**: extract filter-option preparation into pure helpers; move
+  batch action workflow into an action file; split route/query glue from view
+  components.
+
+This makes small PRs reviewable while keeping the feature migration plan
+visible. Do not wait for a perfect reorganization before documenting the
+current state.
+
+## PR-Scale Guidance
+
+Feature README updates should match the PR size:
+
+- For a small state/action extraction, add or update only the relevant
+  migration-state bullets.
+- For a new feature folder structure, include the owner map and external
+  consumers before moving logic into the folder.
+- For a rename-only PR, document intended structure but avoid changing behavior.
+- For behavior PRs, avoid unrelated file moves unless they are required for the
+  boundary being improved.
+
+The README should help the next contributor avoid falling back into the same
+large component, not argue that the migration is complete.

.agents/skills/frontend-large-feature-architecture/references/local-feature-state.md

@@ -0,0 +1,162 @@
+# Local Feature State
+
+Large frontend features should not let one component or hook own everything.
+That shape makes one checkbox, hover, or row-selection change rerun the code
+that also builds filters, columns, data wrappers, expensive cells, drawers,
+actions, and routing glue.
+
+The goal is to make each state change wake only the UI and actions that
+semantically depend on it.
+
+## Default Pattern
+
+Start from the ownership baseline in `big-feature-rules.md`, then add a local
+vanilla Zustand store only when state is high-frequency, shared across multiple
+subtrees in the mounted feature, or must survive row/item remounts.
+
+Use this shape:
+
+1. The page/view creates one store instance with lazy `useState`.
+2. Context provides only the stable store instance.
+3. Components subscribe to the smallest useful slice.
+4. Mutations live in named store actions.
+5. Complex user workflows live in `actions/*.ts` or store actions.
+6. Expensive cells/rows stay view-only behind narrow containers.
+7. Large feature roots have a short `README.md` owner map.
+
+## Feature README
+
+For a concrete owner-map template, read `feature-readmes.md`. Do not use the
+README as a changelog; record durable ownership facts and the next migration
+slice.
+
+## Why Local, Not Global
+
+Langfuse pages are often local-state heavy: filters, saved views, selected rows,
+expanded rows, drawers, peek navigation, lazy row load state, and view-local
+actions. Those states usually belong to one mounted page instance, not the whole
+application.
+
+Use local feature stores for this state. Create the store in the page/view and
+destroy it on unmount. This keeps multiple mounted instances independent, avoids
+cross-route state leaks, and makes ownership visible.
+
+Global state is reserved for product state that is genuinely shared across
+features or routes. Do not promote state globally to avoid prop drilling or to
+make a large component smaller.
+
+Do not add a store just to make a PR look architectural. If the immediate
+problem is an inline export workflow, duplicated filter option shaping, or a
+large column builder, first extract an action or pure helper. Use the store when
+state needs selective subscriptions or must survive row remounts within one
+mounted feature instance.
+
+## Creating The Store
+
+Prefer lazy `useState` for local store instances:
+
+```ts
+const [store] = useState(() =>
+  createFeatureStore({
+    initialProjectId: projectId,
+  }),
+);
+```
+
+This expresses the real lifecycle: one store instance for the committed mounted
+view. The unused setter is acceptable. `useMemo` is the wrong default because it
+is a render-time cache for derived values, not an ownership boundary for an
+external store instance. A local store is stateful infrastructure, so treat its
+identity as state.
+
+`useRef` can also hold a stable instance, but prefer `useState` unless a ref is
+needed for imperative setup. Keep store creation pure; sync changing route/query
+inputs into named store actions such as `resetForFeature(...)` or `init(...)`.
+
+## Independent Actions
+
+Complex workflows should be independent functions. Put surface-specific actions
+in `actions/*.ts`, or make them named store actions when they are tightly coupled
+to local feature state.
+
+The component wires hooks, route params, stores, analytics, and query helpers.
+The action owns the workflow: refetching through callbacks, reading the passed
+store if needed, calling pure helpers, performing browser side effects, and
+emitting analytics.
+
+Actions must not call React hooks. If a workflow needs a lot of context, pass
+the local store instance or a small dependency object rather than threading long
+prop chains through view components.
+
+Example:
+
+```ts
+await exportFeatureData({
+  capture,
+  fetchDetails,
+  projectId,
+  refetchSummary,
+  selectedIds,
+});
+```
+
+For substantial data shaping, export a pure helper next to the action so the
+transformation can be tested without rendering the page.
+
+## Store Shape
+
+Use immutable plain objects for keyed state that must be selector-friendly:
+
+```ts
+type FeatureStoreState = {
+  selectedIds: Record<string, true>;
+  activeId: string | null;
+  actions: {
+    toggleSelected: (id: string, selected: boolean) => void;
+    setActiveId: (id: string | null) => void;
+  };
+};
+```
+
+Avoid mutating `Set` or `Map` in place. If you use them, replace the whole
+instance when updating.
+
+## Anti-Patterns
+
+- A table/list component owns selection, filters, columns, routing, peek state,
+  batch actions, local dialogs, and expensive rendered cells.
+- Context provider `value` changes on row selection, hover, scroll, active row,
+  expanded row, or other high-frequency state.
+- Local feature state is promoted to a global store even though it only belongs
+  to one mounted page instance.
+- `useMemo` is used to own a local external store instance.
+- A shared `src/components/*` component calls a feature-scoped store hook. That
+  silently breaks other callers that do not mount the feature provider.
+- Memoization is the only fix. `memo` helps, but it does not fix a bad state
+  boundary.
+- A callback depends on an inline config object and changes identity on every
+  render.
+- Components subscribe to large objects when they only need a boolean.
+- `useEffect` derives ordinary UI state from fetched data.
+- A component passes a long chain of feature context through props just so a
+  button can perform an action.
+- A page component keeps a complex async workflow inline because all the hooks
+  happen to be in scope there.
+
+## Migration Steps
+
+1. Instrument first: identify which semantic state change causes broad renders.
+2. Choose the smallest useful boundary: store state, action workflow, pure data
+   preparation, or imperative integration hook.
+3. Extract that one state group into a local store only when selective
+   subscriptions are needed.
+4. Replace broad props with selector subscriptions at the smallest UI boundary.
+5. Move related mutations into named actions.
+6. Stabilize callbacks and data wrappers.
+7. Move expensive data preparation into pure functions.
+8. Move complex user actions into store actions or external functions under
+   `actions/*.ts`.
+9. Update the feature README with what improved, what remains spread, and the
+   next atomic slice.
+10. Remove debug instrumentation.
+11. Repeat for the next state group.

.agents/skills/frontend-large-feature-architecture/references/virtualized-lists.md

@@ -0,0 +1,81 @@
+# Virtualized Lists
+
+Virtualized lists are render-boundary infrastructure. They should calculate
+which item shells are visible and position those shells. They should not make
+each row own feature state, effects, subscriptions, data loading, and workflows.
+
+Langfuse uses `@tanstack/react-virtual` for virtualization. Find current
+callsites before changing a virtualized surface, then apply the same
+state-boundary rules as any large feature.
+
+## Smartness Trap
+
+The broken shape is:
+
+- virtualizer rerenders on scroll
+- parent recreates callbacks, config, row wrappers, or data objects
+- row components receive changed props even though the semantic row did not
+  change
+- row-local effects/load state reset or refire
+- dynamic measurement observes DOM changed by an external mutator such as Google
+  Translate
+- measurement updates virtualizer state, which rerenders the same rows again
+
+That is not a small memoization bug. It is leaked state ownership.
+
+The fix is to make scroll and measurement state update the smallest possible
+integration boundary. If scrolling changes a virtual item offset, unchanged row
+content should not receive new semantic props, refire effects, or recreate
+expensive derived data.
+
+## Google Translate DOM Behavior
+
+Google Translate mutates rendered DOM after React has committed it. It can wrap
+text nodes, replace text, and change element dimensions outside React's data
+flow. React and TanStack Virtual do not know whether the changed DOM represents
+stable translated content or a transient mutation.
+
+Do not opt product UI out of translation with `translate="no"` unless product
+explicitly chooses that. Langfuse must work under browser translation.
+
+## Measurement Rules
+
+- Always put the correct `data-index` on the row element TanStack treats as the
+  item.
+- Do not combine live `measureElement` with externally mutated translated DOM
+  in text-heavy rows.
+- Prefer fixed estimates plus overscan for simple rows.
+- For dynamic text-heavy rows, use controlled measurement:
+  - `ResizeObserver` reads the row shell.
+  - Debounce commits.
+  - Do not commit while actively scrolling.
+  - Round heights to avoid sub-pixel churn.
+  - Call `virtualizer.resizeItem(index, height)`.
+  - If a row alternates between two heights repeatedly, clamp to a minimum
+    height that still allows later legitimate growth.
+
+## Row Rules
+
+- The virtualizer owns positioning only.
+- State that must survive remounts lives outside the row instance.
+- Expensive row content should be a memoized view component.
+- Narrow row containers may subscribe to local store slices and queries.
+- View components should receive stable props and perform no effects.
+- Feature-scoped row containers belong under `src/features/*`; shared
+  `src/components/*` row exports should be context-free.
+- Scrolling may rerender the virtualizer. It should not rerender unchanged
+  expensive row content.
+- Do not use a global store to preserve row state across virtualization. Use a
+  view-scoped store owned by the mounted list/page instance.
+
+## Migration Steps
+
+1. Add temporary logs to identify whether scroll causes remounts, prop changes,
+   measurement loops, or query refetches.
+2. Remove logs before shipping.
+3. Move row-local state that must survive virtualization into a local store.
+4. Stabilize callbacks and config objects passed to rows.
+5. Replace live `measureElement` with fixed estimates or controlled measurement.
+6. Move row logic into pure helpers or feature-local containers.
+7. Verify with browser translation enabled, horizontal resize, and small
+   vertical scroll deltas.

.agents/skills/git-workflow/SKILL.md

@@ -0,0 +1,51 @@
+---
+name: git-workflow
+description: |
+  Langfuse repo Git, GitHub, commit, branch, pull request, issue search,
+  release, and production-promotion workflow. Use when staging, committing,
+  pushing, opening PRs, searching GitHub issues, or changing release/promotion
+  behavior.
+---
+
+# Git Workflow
+
+Use this skill for repo-specific Git, GitHub, pull request, and release
+operations.
+
+## Safety
+
+- Inspect `git status` before staging or committing.
+- Do not stage unrelated working-tree changes.
+- Do not revert unrelated working-tree changes.
+- Do not use destructive commands such as `git reset --hard` or
+  `git checkout --` unless explicitly requested.
+- Keep commits focused and atomic.
+- Never add secrets or credentials to the repo.
+
+## Commits and Pull Requests
+
+- Commit messages and PR titles must follow Conventional Commits:
+  `type(scope): description` or `type: description`.
+- Use `feat` for new features and `fix` for bug fixes.
+- Use a scope when it clarifies the affected area, for example
+  `fix(api): handle missing trace id`.
+- Mark breaking changes with `!` in the type/scope or a `BREAKING CHANGE:`
+  footer.
+- PR titles are validated by `.github/workflows/validate-pr-title.yml`.
+- In PR descriptions, list impacted packages and executed verification
+  commands.
+
+## GitHub
+
+- Use `gh search issues` for GitHub issue search.
+- Prefer non-interactive Git and GitHub commands where possible.
+- Keep PRs narrow enough to review without unrelated refactors.
+
+## Release
+
+- Release workflow is managed at root with `pnpm run release`.
+- Promote `main` to `production` via
+  `.github/workflows/promote-main-to-production.yml` or
+  `pnpm run release:cloud`.
+- Do not change release/versioning flow without updating this skill and the
+  impacted package guides.

.agents/skills/linear-bug-triage/SKILL.md

@@ -0,0 +1,116 @@
+---
+name: linear-bug-triage
+description: |
+  Deduplicate measured bug or regression evidence against Linear, then either
+  add evidence comments to related existing issues or create concise Linear bug
+  issues in Triage. Use when Codex has confirmed evidence from Datadog,
+  benchmarks, traces, timings, flamegraphs, logs, or production measurements and
+  needs Linear search, comments, labels, Datadog links, or bug ticket creation
+  without fix suggestions, but only after a human has approved sharing the
+  findings in Linear.
+---
+
+# Linear Bug Triage
+
+Use this skill after a bug or regression candidate has measured evidence. This
+skill owns Linear search, deduplication, evidence comments, and ticket creation;
+the calling skill owns deciding whether the signal is issue-worthy.
+
+## Human Approval Gate
+
+Before doing anything in Linear, first show the findings to the human in a
+compact markdown table and ask for explicit permission to share them in Linear.
+The table should include one row per candidate with:
+
+- Candidate / cluster name.
+- Environments.
+- Service and route/resource.
+- Recent window measurement.
+- Baseline measurement.
+- Delta / regression summary.
+- Key Datadog evidence links.
+- Proposed Linear action (`comment existing`, `create new`, or `none`).
+
+If the human does not explicitly approve, stop after presenting the table. Do
+not search Linear, do not comment on issues, and do not create issues.
+
+If a calling workflow already showed the findings table and obtained explicit
+human approval for a Linear handoff, skip this gate and proceed directly to
+deduplication.
+
+## Required Evidence
+
+For each candidate, gather:
+
+- Recent window and baseline window as absolute time ranges with timezone.
+- Measured signal: counts, rates, p50/p95/p99 latency, trace samples,
+  flamegraphs, monitor thresholds, or benchmark deltas.
+- Affected environments, services, routes/resources, status codes, and top error
+  messages.
+- Datadog links for logs, spans, traces, metrics, dashboards, or flamegraphs
+  used as evidence.
+- The exact text `No measurements found` for requested measurements that are
+  unavailable.
+
+Do not create or comment based on guesses, unsupported impact claims, or missing
+measurements alone.
+
+## Deduplication
+
+After the human explicitly approves, before creating a new issue:
+
+1. Search Linear for related open issues using exact error text, route/resource,
+   service, environment, monitor name, and Datadog link keywords.
+2. Search recently closed or canceled issues if the error is recurring or the
+   wording is distinctive.
+3. If a related issue exists, add a concise evidence comment instead of creating
+   a duplicate.
+4. If no related issue exists, create one Linear issue in the `Triage` state for
+   each distinct bug cluster.
+
+## Existing Issue Comments
+
+For related existing issues, add only:
+
+- Recent window and baseline window.
+- Measured delta or `No measurements found` for unavailable signals.
+- Affected environments, services, routes/resources, and top error messages.
+- Datadog links.
+
+Do not add fix suggestions, root-cause guesses, implementation notes, owner
+assignments, or next steps.
+
+## New Issue Format
+
+Create new issues with:
+
+- State/status `Triage`; pass the Linear state explicitly on creation and do not
+  rely on workspace defaults.
+- Label `bug`.
+- Additional existing labels that match the evidence, such as affected service,
+  environment, API, ingestion, latency, ClickHouse, Postgres, integrations, or
+  observability labels. Query labels first and use the repository/team's exact
+  label names.
+- Concise title: `bug: <service or route> <measured symptom> in <envs>`.
+- Concise body, evidence-only:
+
+```markdown
+Recent window: <absolute time range and timezone>
+Baseline: <absolute time range and timezone>
+
+Signal:
+- <count/rate/latency delta with env/service/route>
+- <"No measurements found" for missing requested measurements>
+
+Evidence:
+- Datadog logs: <url>
+- Datadog spans/traces: <url>
+- Datadog metrics, dashboard, or latency graph: <url>
+
+Related Linear search:
+- <brief search terms used and result>
+```
+
+Do not include fix suggestions, root-cause guesses, implementation notes, owner
+assignments, or next steps unless the user explicitly asks outside the Linear
+issue or comment.

.agents/skills/linear-bug-triage/agents/openai.yaml

@@ -0,0 +1,4 @@
+interface:
+  display_name: "Linear Bug Triage"
+  short_description: "Deduplicate and file Linear bug evidence"
+  default_prompt: "Use $linear-bug-triage to deduplicate measured bug evidence and create or comment Linear triage issues."

.agents/skills/pnpm-upgrade-package/SKILL.md

@@ -0,0 +1,74 @@
+---
+name: pnpm-upgrade-package
+description: >-
+  Upgrade pnpm workspace dependencies to target/latest versions:
+  direct/transitive bumps, release-age checks, temporary overrides,
+  minimumReleaseAgeExclude, lockfile/dedupe verification.
+---
+
+# PNPM Upgrade Package
+
+Use this skill for interactive dependency bumps in Langfuse.
+
+## Read Order
+
+- Use this `SKILL.md` for the end-to-end workflow.
+- Run the main helper once at the start of the upgrade:
+  `node .agents/skills/pnpm-upgrade-package/scripts/check-release-age-window.mjs <package> [targetVersion]`
+
+## Apply This Skill
+
+- Ask for the package name if the user did not provide one.
+- Ask for the target version if the user did not provide one.
+- Run the main helper once as the first analysis step and use that single
+  output for scope, exclusion decisions, and the final bump.
+- If the target package is not directly declared anywhere, run
+  `pnpm why -r <package>` to find which direct dependency brings it in, then
+  inspect whether the current top-level parent already allows the requested
+  transitive version via its dependency range.
+- If the current parent range already covers the requested transitive version,
+  prefer a lockfile refresh / reinstall path over bumping the parent manifest.
+- If the current parent range does not cover the requested transitive version,
+  upgrade that parent dependency instead of adding the target package directly
+  unless the user explicitly wants that.
+- If pnpm will not move an already-allowed transitive version, a scoped
+  `overrides` entry in `pnpm-workspace.yaml` may be used as a temporary
+  resolution tool. Before finishing, prove whether the override is still
+  required: remove it, run `pnpm install`, then run `pnpm dedupe`. Inspect the
+  diff after each generated change. If the target version remains without the
+  override, do not keep the override; keep or restore it only when pnpm reverts
+  or drifts from the requested version without it.
+- Never manually edit `pnpm-lock.yaml`; regenerate lockfile changes with
+  `pnpm` commands only. If a lockfile-only refresh causes unrelated churn,
+  adjust the pnpm command and rerun instead of patching the lockfile by hand.
+- After fixing or upgrading a package, run `pnpm dedupe`. Always inspect the
+  diff after dedupe and revert that generated attempt if it introduces
+  unrelated churn.
+- Resolve the registry latest version, but do not silently upgrade to latest
+  unless the user asked for latest.
+- Compare the target version with the latest version installable under the
+  current `minimumReleaseAge` window.
+- Ask before adding `minimumReleaseAgeExclude` entries for the target package,
+  exact dependency companions from `dependencies` or `optionalDependencies`, or
+  locally installed exact peer dependencies.
+- Finish with `pnpm why -r <package>` to confirm that only the intended version
+  remains in the workspace.
+
+## Quick Commands
+
+- Analysis pass:
+  `node .agents/skills/pnpm-upgrade-package/scripts/check-release-age-window.mjs <package> <targetVersion>`
+- Transitive provenance / final graph verification:
+  `pnpm why -r <package>`
+- Inspect a current parent manifest on the registry:
+  `npm view <parent>@<installedVersion> dependencies peerDependencies optionalDependencies --json`
+- Optional lockfile cleanup:
+  `pnpm dedupe`
+- Bump in the root workspace:
+  `pnpm -w up <package>@<version>`
+- Bump in one workspace:
+  `pnpm --filter web up <package>@<version>`
+- Bump everywhere that should move together:
+  `pnpm -r up <package>@<version>`
+- Verify temporary override removal:
+  remove the override, then run `pnpm install` and `pnpm dedupe`

.agents/skills/pnpm-upgrade-package/agents/openai.yaml

@@ -0,0 +1,4 @@
+interface:
+  display_name: "PNPM Upgrade Package"
+  short_description: "Interactive pnpm package bump workflow"
+  default_prompt: "Use $pnpm-upgrade-package to upgrade a package in this pnpm workspace, asking me for the package or version if I did not provide them."

.agents/skills/pnpm-upgrade-package/scripts/check-release-age-window.mjs

@@ -0,0 +1,418 @@
+#!/usr/bin/env node
+
+import { join } from "node:path";
+import {
+  entryCoversVersion,
+  findLocalPackageReferences,
+  formatWorkspaceReference,
+  getRootPnpmControls,
+  readWorkspaceConfig,
+} from "./lib/workspace-utils.mjs";
+
+const args = process.argv.slice(2);
+const asJson = args.includes("--json");
+const positional = args.filter((arg) => !arg.startsWith("--"));
+const packageName = positional[0];
+const requestedTargetVersion = positional[1] ?? null;
+
+if (!packageName) {
+  console.error(
+    "Usage: node .agents/skills/pnpm-upgrade-package/scripts/check-release-age-window.mjs <package> [targetVersion] [--json]",
+  );
+  process.exit(1);
+}
+
+const repoRoot = process.cwd();
+const workspaceConfig = readWorkspaceConfig(join(repoRoot, "pnpm-workspace.yaml"));
+const minimumReleaseAgeMinutes = workspaceConfig.minimumReleaseAge ?? 0;
+const thresholdMs = Date.now() - minimumReleaseAgeMinutes * 60 * 1000;
+const REGISTRY_FETCH_TIMEOUT_MS = 30_000;
+const registryCache = new Map();
+const workspaceReferenceCache = new Map();
+
+const getWorkspaceReferences = (name) => {
+  if (!workspaceReferenceCache.has(name)) {
+    workspaceReferenceCache.set(name, findLocalPackageReferences(repoRoot, name));
+  }
+
+  return workspaceReferenceCache.get(name);
+};
+
+function printSectionHeader(title) {
+  console.log("");
+  console.log(title);
+}
+
+function isPrerelease(version) {
+  return version.includes("-");
+}
+
+function isExactVersion(spec) {
+  return /^\d+\.\d+\.\d+(?:[-+][0-9A-Za-z.-]+)?$/.test(spec.trim());
+}
+
+function getMatchingExcludeEntries(name, version) {
+  return workspaceConfig.minimumReleaseAgeExclude.filter((entry) =>
+    entryCoversVersion(entry, name, version),
+  );
+}
+
+async function fetchRegistryPackage(name) {
+  if (registryCache.has(name)) return registryCache.get(name);
+
+  const abortController = new AbortController();
+  const timeoutId = setTimeout(
+    () => abortController.abort(),
+    REGISTRY_FETCH_TIMEOUT_MS,
+  );
+  timeoutId.unref?.();
+
+  try {
+    const response = await fetch(
+      `https://registry.npmjs.org/${encodeURIComponent(name)}`,
+      {
+        headers: {
+          accept: "application/json",
+          "user-agent": "langfuse-pnpm-upgrade-package-skill",
+        },
+        signal: abortController.signal,
+      },
+    );
+
+    if (!response.ok) {
+      throw new Error(
+        `Failed to fetch ${name} from npm registry: ${response.status}`,
+      );
+    }
+
+    const metadata = await response.json();
+    registryCache.set(name, metadata);
+    return metadata;
+  } catch (error) {
+    if (error?.name === "AbortError") {
+      throw new Error(
+        `Timed out fetching ${name} from npm registry after ${REGISTRY_FETCH_TIMEOUT_MS}ms`,
+        { cause: error },
+      );
+    }
+
+    throw error;
+  } finally {
+    clearTimeout(timeoutId);
+  }
+}
+
+function getInstallability(metadata, name, version) {
+  const publishedAt = metadata.time?.[version] ?? null;
+  const publishedAtMs = publishedAt ? Date.parse(publishedAt) : null;
+  const matchingExcludeEntries = getMatchingExcludeEntries(name, version);
+  const isYoungerThanMinimumReleaseAge =
+    publishedAtMs != null ? publishedAtMs > thresholdMs : null;
+  const isInstallableWithoutNewExclude =
+    isYoungerThanMinimumReleaseAge == null
+      ? null
+      : !isYoungerThanMinimumReleaseAge || matchingExcludeEntries.length > 0;
+
+  return {
+    name,
+    version,
+    publishedAt,
+    isYoungerThanMinimumReleaseAge,
+    isInstallableWithoutNewExclude,
+    matchingExcludeEntries,
+    suggestedExclude:
+      isInstallableWithoutNewExclude === false ? `${name}@${version}` : null,
+  };
+}
+
+function selectLatestInstallableVersion(metadata, name) {
+  const times = metadata.time ?? {};
+
+  return (
+    Object.keys(metadata.versions ?? {})
+      .filter((version) => times[version] && !isPrerelease(version))
+      .sort((left, right) => Date.parse(times[right]) - Date.parse(times[left]))
+      .map((version) => getInstallability(metadata, name, version))
+      .find((candidate) => candidate.isInstallableWithoutNewExclude) ?? null
+  );
+}
+
+function collectManifestEntries(manifest, fields) {
+  const merged = new Map();
+
+  for (const field of fields) {
+    for (const [name, spec] of Object.entries(manifest[field] ?? {})) {
+      const key = `${name}:${spec}`;
+      const entry = merged.get(key);
+
+      if (entry) {
+        entry.fields.push(field);
+        continue;
+      }
+
+      merged.set(key, { name, spec, fields: [field] });
+    }
+  }
+
+  return [...merged.values()].sort((left, right) =>
+    left.name.localeCompare(right.name),
+  );
+}
+
+async function analyzeManifestEntries(entries, { includeWorkspace = false } = {}) {
+  const exact = [];
+  const range = [];
+
+  for (const entry of entries) {
+    const workspaceReferences = includeWorkspace
+      ? getWorkspaceReferences(entry.name)
+      : null;
+
+    if (!isExactVersion(entry.spec)) {
+      range.push({
+        ...entry,
+        ...(includeWorkspace ? { workspaceReferences } : {}),
+      });
+      continue;
+    }
+
+    const metadata = await fetchRegistryPackage(entry.name);
+    const installability = getInstallability(metadata, entry.name, entry.spec);
+
+    exact.push({
+      ...entry,
+      ...installability,
+      ...(includeWorkspace
+        ? {
+            workspaceReferences,
+            isInstalledInWorkspace: workspaceReferences.length > 0,
+          }
+        : {}),
+      suggestedExclude:
+        includeWorkspace && workspaceReferences.length === 0
+          ? null
+          : installability.suggestedExclude,
+    });
+  }
+
+  return { exact, range };
+}
+
+function printWorkspaceReferences(title, references) {
+  printSectionHeader(title);
+  if (references.length === 0) {
+    console.log("- none");
+    return;
+  }
+
+  for (const reference of references) {
+    console.log(`- ${formatWorkspaceReference(reference)}`);
+  }
+}
+
+function printRootPnpmControls(rootPnpm) {
+  printSectionHeader("Root pnpm controls:");
+  if (
+    rootPnpm.overrideMatches.length === 0 &&
+    rootPnpm.patchedDependencyMatches.length === 0
+  ) {
+    console.log("- none");
+    return;
+  }
+
+  for (const match of rootPnpm.overrideMatches) {
+    console.log(`- override ${match.selector}: ${match.value}`);
+  }
+  for (const match of rootPnpm.patchedDependencyMatches) {
+    console.log(`- patched dependency ${match.selector}: ${match.value}`);
+  }
+}
+
+function printVersionEntries(title, entries, { includeWorkspace = false } = {}) {
+  printSectionHeader(title);
+  if (entries.length === 0) {
+    console.log("- none");
+    return;
+  }
+
+  for (const entry of entries) {
+    const status =
+      entry.isInstallableWithoutNewExclude == null
+        ? "unknown"
+        : entry.isInstallableWithoutNewExclude
+          ? "installable now"
+          : "needs exclude";
+
+    console.log(
+      `- ${entry.name}@${entry.version} (${status}; via ${entry.fields.join(", ")})`,
+    );
+    if (entry.publishedAt) {
+      console.log(`  published at: ${entry.publishedAt}`);
+    }
+    if (includeWorkspace) {
+      console.log(
+        `  installed in workspace: ${entry.isInstalledInWorkspace ? "yes" : "no"}`,
+      );
+      for (const reference of entry.workspaceReferences) {
+        console.log(`  workspace reference: ${formatWorkspaceReference(reference)}`);
+      }
+    }
+    if (entry.matchingExcludeEntries.length > 0) {
+      console.log(
+        `  matching exclude entries: ${entry.matchingExcludeEntries.join(", ")}`,
+      );
+    }
+    if (entry.suggestedExclude) {
+      console.log(`  suggested exclude: ${entry.suggestedExclude}`);
+    }
+  }
+}
+
+function printRangeEntries(title, entries) {
+  printSectionHeader(title);
+  if (entries.length === 0) {
+    console.log("- none");
+    return;
+  }
+
+  for (const entry of entries) {
+    console.log(
+      `- ${entry.name}: ${entry.spec} (manual review; via ${entry.fields.join(", ")})`,
+    );
+    for (const reference of entry.workspaceReferences ?? []) {
+      console.log(`  workspace reference: ${formatWorkspaceReference(reference)}`);
+    }
+  }
+}
+
+const packageMetadata = await fetchRegistryPackage(packageName);
+const latestVersion = packageMetadata["dist-tags"]?.latest ?? null;
+const targetVersion = requestedTargetVersion ?? latestVersion;
+
+if (!targetVersion) {
+  console.error(`Could not resolve a target version for ${packageName}.`);
+  process.exit(1);
+}
+
+if (!packageMetadata.versions?.[targetVersion]) {
+  console.error(`Version ${targetVersion} was not found for ${packageName}.`);
+  process.exit(1);
+}
+
+const packageWorkspaceReferences = getWorkspaceReferences(packageName);
+const rootPnpm = getRootPnpmControls(repoRoot, packageName);
+const latestInstallableWithoutNewExclude = selectLatestInstallableVersion(
+  packageMetadata,
+  packageName,
+);
+const targetInstallability = getInstallability(
+  packageMetadata,
+  packageName,
+  targetVersion,
+);
+const targetManifest = packageMetadata.versions[targetVersion];
+
+const dependencyCompanions = await analyzeManifestEntries(
+  collectManifestEntries(targetManifest, [
+    "dependencies",
+    "optionalDependencies",
+  ]),
+);
+const peerDependencies = await analyzeManifestEntries(
+  collectManifestEntries(targetManifest, ["peerDependencies"]),
+  { includeWorkspace: true },
+);
+
+const result = {
+  packageName,
+  targetVersion,
+  targetWasExplicitlyProvided: requestedTargetVersion != null,
+  packageWorkspaceReferences,
+  rootPnpm,
+  minimumReleaseAgeMinutes,
+  thresholdIso: new Date(thresholdMs).toISOString(),
+  latestRegistryVersion: latestVersion,
+  latestRegistryPublishedAt:
+    latestVersion != null ? packageMetadata.time?.[latestVersion] ?? null : null,
+  latestInstallableWithoutNewExclude,
+  targetPublishedAt: targetInstallability.publishedAt,
+  targetIsYoungerThanMinimumReleaseAge:
+    targetInstallability.isYoungerThanMinimumReleaseAge,
+  targetIsInstallableWithoutNewExclude:
+    targetInstallability.isInstallableWithoutNewExclude,
+  matchingPackageExcludeEntries: targetInstallability.matchingExcludeEntries,
+  suggestedPackageExclude: targetInstallability.suggestedExclude,
+  exactDependencyCompanions: dependencyCompanions.exact,
+  rangeDependencyCompanions: dependencyCompanions.range,
+  exactPeerDependencies: peerDependencies.exact,
+  rangePeerDependencies: peerDependencies.range,
+};
+
+if (asJson) {
+  console.log(JSON.stringify(result, null, 2));
+  process.exit(0);
+}
+
+console.log(`Package: ${packageName}`);
+console.log(
+  `Target version: ${targetVersion}${
+    requestedTargetVersion
+      ? ""
+      : " (resolved latest; still ask before bumping if version was omitted)"
+  }`,
+);
+
+printWorkspaceReferences(
+  "Target package workspace references:",
+  packageWorkspaceReferences,
+);
+printRootPnpmControls(rootPnpm);
+
+printSectionHeader("Release-age window:");
+console.log(`minimumReleaseAge: ${minimumReleaseAgeMinutes} minutes`);
+console.log(`Threshold: ${result.thresholdIso}`);
+console.log(`Latest registry version: ${result.latestRegistryVersion ?? "unknown"}`);
+if (result.latestRegistryPublishedAt) {
+  console.log(`Latest registry published at: ${result.latestRegistryPublishedAt}`);
+}
+if (latestInstallableWithoutNewExclude) {
+  console.log(
+    `Latest installable without new exclude: ${latestInstallableWithoutNewExclude.version} (${latestInstallableWithoutNewExclude.publishedAt})`,
+  );
+} else {
+  console.log("Latest installable without new exclude: none found");
+}
+console.log(`Target published at: ${result.targetPublishedAt ?? "unknown"}`);
+console.log(
+  `Target installable without new exclude: ${
+    result.targetIsInstallableWithoutNewExclude == null
+      ? "unknown"
+      : result.targetIsInstallableWithoutNewExclude
+        ? "yes"
+        : "no"
+  }`,
+);
+if (result.matchingPackageExcludeEntries.length > 0) {
+  console.log("Matching package exclude entries:");
+  for (const entry of result.matchingPackageExcludeEntries) {
+    console.log(`- ${entry}`);
+  }
+} else {
+  console.log("Matching package exclude entries: none");
+}
+if (result.suggestedPackageExclude) {
+  console.log(`Suggested package exclude: ${result.suggestedPackageExclude}`);
+}
+
+printVersionEntries(
+  "Exact dependency companions (dependencies + optionalDependencies):",
+  result.exactDependencyCompanions,
+);
+printRangeEntries(
+  "Range dependency companions (dependencies + optionalDependencies):",
+  result.rangeDependencyCompanions,
+);
+printVersionEntries("Exact peer dependencies:", result.exactPeerDependencies, {
+  includeWorkspace: true,
+});
+printRangeEntries("Range peer dependencies:", result.rangePeerDependencies);

.agents/skills/pnpm-upgrade-package/scripts/lib/workspace-utils.mjs

@@ -0,0 +1,244 @@
+import { existsSync, readFileSync, readdirSync } from "node:fs";
+import { join, relative } from "node:path";
+
+const packageFields = [
+  "dependencies",
+  "devDependencies",
+  "peerDependencies",
+  "optionalDependencies",
+];
+
+export function formatWorkspaceReference(reference) {
+  const label = reference.workspaceName
+    ? `${reference.path} (${reference.workspaceName})`
+    : reference.path;
+  const specs = reference.matches
+    .map((match) => `${match.field}: ${match.spec}`)
+    .join(", ");
+
+  return `${label} -> ${specs}`;
+}
+
+export function readJson(path) {
+  return JSON.parse(readFileSync(path, "utf8"));
+}
+
+function stripInlineComment(line) {
+  let quote = null;
+  let escaped = false;
+
+  for (let index = 0; index < line.length; index += 1) {
+    const char = line[index];
+
+    if (escaped) {
+      escaped = false;
+      continue;
+    }
+
+    if (quote) {
+      if (char === "\\") {
+        escaped = true;
+        continue;
+      }
+
+      if (char === quote) {
+        quote = null;
+      }
+      continue;
+    }
+
+    if (char === "'" || char === '"') {
+      quote = char;
+      continue;
+    }
+
+    if (char === "#") {
+      return line.slice(0, index).trimEnd();
+    }
+  }
+
+  return line;
+}
+
+function unquoteYamlScalar(value) {
+  return value.trim().replace(/^['"]|['"]$/g, "");
+}
+
+export function readWorkspaceConfig(path) {
+  const raw = readFileSync(path, "utf8");
+  const lines = raw.split(/\r?\n/);
+  let minimumReleaseAge = 0;
+  const minimumReleaseAgeExclude = [];
+  const overrides = {};
+  const patchedDependencies = {};
+  let activeBlock = null;
+
+  for (const line of lines) {
+    const uncommented = stripInlineComment(line);
+    const trimmed = uncommented.trim();
+
+    if (!trimmed || trimmed.startsWith("#")) continue;
+
+    const ageMatch = trimmed.match(/^minimumReleaseAge:\s*(\d+)\s*$/);
+    if (ageMatch) {
+      minimumReleaseAge = Number(ageMatch[1]);
+      activeBlock = null;
+      continue;
+    }
+
+    if (/^minimumReleaseAgeExclude:\s*$/.test(trimmed)) {
+      activeBlock = "minimumReleaseAgeExclude";
+      continue;
+    }
+
+    if (/^overrides:\s*$/.test(trimmed)) {
+      activeBlock = "overrides";
+      continue;
+    }
+
+    if (/^patchedDependencies:\s*$/.test(trimmed)) {
+      activeBlock = "patchedDependencies";
+      continue;
+    }
+
+    if (/^\S/.test(uncommented)) {
+      activeBlock = null;
+      continue;
+    }
+
+    if (activeBlock === "minimumReleaseAgeExclude") {
+      const excludeMatch = uncommented.match(/^\s*-\s+(.+?)\s*$/);
+      if (excludeMatch) {
+        minimumReleaseAgeExclude.push(unquoteYamlScalar(excludeMatch[1]));
+      }
+      continue;
+    }
+
+    if (activeBlock === "overrides" || activeBlock === "patchedDependencies") {
+      const entryMatch = uncommented.match(/^\s+(.+?):\s+(.+?)\s*$/);
+      if (!entryMatch) continue;
+
+      const selector = unquoteYamlScalar(entryMatch[1]);
+      const value = unquoteYamlScalar(entryMatch[2]);
+      if (activeBlock === "overrides") {
+        overrides[selector] = value;
+      } else {
+        patchedDependencies[selector] = value;
+      }
+    }
+  }
+
+  return {
+    minimumReleaseAge,
+    minimumReleaseAgeExclude,
+    overrides,
+    patchedDependencies,
+  };
+}
+
+export function collectPackageJsonPaths(repoRoot) {
+  const paths = [
+    "package.json",
+    "web/package.json",
+    "worker/package.json",
+    "ee/package.json",
+  ];
+  const packagesRoot = join(repoRoot, "packages");
+
+  if (!existsSync(packagesRoot)) return paths;
+
+  const stack = [packagesRoot];
+  while (stack.length > 0) {
+    const current = stack.pop();
+    for (const entry of readdirSync(current, { withFileTypes: true })) {
+      if (
+        entry.name === "node_modules" ||
+        entry.name === "dist" ||
+        entry.name === ".git"
+      ) {
+        continue;
+      }
+
+      const nextPath = join(current, entry.name);
+      if (entry.isDirectory()) {
+        stack.push(nextPath);
+        continue;
+      }
+
+      if (entry.isFile() && entry.name === "package.json") {
+        paths.push(relative(repoRoot, nextPath));
+      }
+    }
+  }
+
+  return [...new Set(paths)];
+}
+
+export function matchesPackageSelector(selector, wantedPackage) {
+  if (selector === wantedPackage) return true;
+  if (selector.startsWith(`${wantedPackage}@`)) return true;
+  if (selector.endsWith(`>${wantedPackage}`)) return true;
+  if (selector.includes(`>${wantedPackage}@`)) return true;
+  if (selector.endsWith("/*")) {
+    const prefix = selector.slice(0, -1);
+    return wantedPackage.startsWith(prefix);
+  }
+  return false;
+}
+
+export function entryCoversVersion(entry, wantedPackage, wantedVersion) {
+  if (entry === wantedPackage) return true;
+  if (entry.endsWith("/*")) {
+    const prefix = entry.slice(0, -1);
+    return wantedPackage.startsWith(prefix);
+  }
+  if (!entry.startsWith(`${wantedPackage}@`)) return false;
+
+  return entry
+    .slice(wantedPackage.length + 1)
+    .split("||")
+    .map((part) => part.trim())
+    .includes(wantedVersion);
+}
+
+export function findLocalPackageReferences(repoRoot, wantedPackage) {
+  const results = [];
+
+  for (const packageJsonPath of collectPackageJsonPaths(repoRoot)) {
+    const json = readJson(join(repoRoot, packageJsonPath));
+    const matches = [];
+
+    for (const field of packageFields) {
+      if (json[field]?.[wantedPackage]) {
+        matches.push({ field, spec: json[field][wantedPackage] });
+      }
+    }
+
+    if (matches.length > 0) {
+      results.push({
+        path: packageJsonPath,
+        workspaceName: json.name ?? null,
+        matches,
+      });
+    }
+  }
+
+  return results;
+}
+
+export function getRootPnpmControls(repoRoot, packageName) {
+  const workspaceConfig = readWorkspaceConfig(
+    join(repoRoot, "pnpm-workspace.yaml"),
+  );
+
+  return {
+    overrideMatches: Object.entries(workspaceConfig.overrides)
+      .filter(([selector]) => matchesPackageSelector(selector, packageName))
+      .map(([selector, value]) => ({ selector, value })),
+    patchedDependencyMatches: Object.entries(
+      workspaceConfig.patchedDependencies,
+    )
+      .filter(([selector]) => matchesPackageSelector(selector, packageName))
+      .map(([selector, value]) => ({ selector, value })),
+  };
+}

.agents/skills/security-review/SKILL.md

@@ -0,0 +1,101 @@
+---
+name: security-review
+description: Security review patterns for Langfuse. Use during code review, design, or planning whenever a change accepts user-supplied URLs, host/endpoint/baseURL fields, secrets, cross-tenant data, new outbound HTTP requests, new integrations (webhooks, blob storage, LLM connections, image proxies), redirect-following behavior, or new auth/permission scopes. Covers SSRF/outbound URL validation today and is intentionally extensible to other recurring security findings (tenant isolation, secret handling, redirect mishandling, file upload, RBAC scope drift).
+---
+
+# Security Review
+
+Use this skill when reviewing or planning code that touches a security-sensitive
+surface in Langfuse. It collects the recurring findings the team has seen in
+external security reports so that future agents catch them at design and review
+time rather than after the fact.
+
+## When to Apply
+
+Apply this skill when the change touches any of:
+
+- a user-supplied URL, host, endpoint, `baseURL`, or webhook target
+- a new outbound HTTP request (`fetch`, `axios`, AWS SDK client init with a
+  custom `endpoint`, OpenAI/Anthropic/Bedrock client init with a custom
+  `baseURL`, etc.)
+- a new integration form under Settings -> Integrations or any
+  admin-configurable network destination
+- a new tRPC procedure or public API route that mutates project-scoped data or
+  changes who can access it
+- secrets, API keys, signing secrets, or encryption-at-rest fields
+- redirect-following or cross-origin header handling
+- file uploads, image proxies, or other binary data flowing in or out
+
+Apply this skill during **plan mode** when designing a new integration so the
+correct validation surfaces land in the plan, not in a follow-up CVE.
+
+## How to Read This Skill
+
+1. Open [references/checklist.md](references/checklist.md) and run the mental
+   sweep against the change.
+2. For each bullet that fires, open the matching topic reference.
+
+| Topic | Open when | File |
+| --- | --- | --- |
+| SSRF and outbound URL validation | The change accepts or fetches a user-supplied URL, host, or endpoint | [references/outbound-url-validation.md](references/outbound-url-validation.md) |
+
+The catalog is intentionally short today. New topic files are added as new
+finding classes recur (see "Extending This Skill").
+
+## Output Expectations (Review Mode)
+
+When this skill is used during code review:
+
+- List findings first, ordered by severity, with file and line references.
+- For each finding, name the canonical helper or known-good call site the
+  author should copy.
+- For SSRF-class findings, point at [references/outbound-url-validation.md](references/outbound-url-validation.md)
+  rather than re-deriving the fix.
+- Call out missing **negative tests** (private-IP, cross-tenant, missing-scope)
+  as findings, not as nice-to-haves.
+
+## Output Expectations (Design / Plan Mode)
+
+When this skill is used while planning:
+
+- Restate which surfaces the new feature exposes (forms, public API routes,
+  worker entrypoints).
+- For each surface that matches a checklist trigger, name the validator or
+  helper that must be invoked and at which layer (save-time, use-time,
+  connection-time, redirect-time).
+- Treat "we will validate later" as a design defect: validation belongs in the
+  same change that introduces the surface.
+
+## Extending This Skill
+
+Add a new `references/<topic>.md` whenever a security finding recurs across
+features or PR reviews. Keep each reference narrow and concrete:
+
+1. Threat in plain language (one paragraph).
+2. Canonical helpers in this repo, with paths.
+3. Known-good call sites that can be copied.
+4. Required defenses (save-time, use-time, transport-time, etc.).
+5. Anti-patterns to flag in review.
+
+Then add a one-line trigger to [references/checklist.md](references/checklist.md)
+pointing at the new topic file, and add a row to the table above.
+
+Candidates for future references (do not add until a real finding recurs):
+
+- Tenant isolation (`projectId` filters across Prisma and ClickHouse)
+- Secret handling and encryption-at-rest read paths
+- Redirect mishandling and sensitive-header propagation
+- File upload validation and content-type sniffing
+- RBAC scope drift on new tRPC/public API endpoints
+- Signed URL scoping (expiry, path, method)
+- Public API rate limiting and auth boundary checks
+
+## Integration With Other Skills
+
+- The shared `code-review` skill should defer here for any change that matches
+  the triggers above; see [code-review/SKILL.md](../code-review/SKILL.md).
+- The shared `backend-dev-guidelines` skill should defer here when adding
+  outbound HTTP, integration config, or URL-accepting procedures; see
+  [backend-dev-guidelines/SKILL.md](../backend-dev-guidelines/SKILL.md).
+- Confirmed issues with reproduction evidence go through `linear-bug-triage`
+  for Linear handoff.

.agents/skills/security-review/references/checklist.md

@@ -0,0 +1,74 @@
+# Security Review Checklist
+
+Run this list mentally on every relevant change. For each bullet, either find
+that it does not apply, or confirm the listed mitigation is present in the
+diff. Each bullet links to the topic reference that owns the detail.
+
+## User-Supplied URLs and Outbound Requests
+
+- Does the change accept a URL, host, `endpoint`, `baseURL`, webhook target,
+  or any field that becomes the destination of an outbound HTTP request?
+  - If yes, see [outbound-url-validation.md](outbound-url-validation.md).
+  - Required: a save-time validator on the mutation/API route **and** use-time
+    or connection-time validation when the request is issued, including
+    redirects. Raw `fetch(userUrl)` or SDK init with `endpoint: userUrl`
+    without that wiring is a finding.
+
+- Does the change add a new "integration" (Settings -> Integrations, new
+  webhook destination, new storage backend, new LLM provider, new image
+  proxy) that lets an admin configure a host?
+  - If yes, see [outbound-url-validation.md](outbound-url-validation.md).
+  - Required: new env-driven allowlist trio (host / IPs / IP segments) for
+    self-hosted, plus strict Cloud enforcement.
+
+- Does the change follow redirects on a user-controlled URL with plain
+  `fetch()`?
+  - If yes, see [outbound-url-validation.md](outbound-url-validation.md)
+    (Redirect Handling). Required: `fetchWithSecureRedirects` with the
+    matching validator.
+
+## Tenant Isolation
+
+- Does every new Prisma query on a project-scoped table include `projectId`
+  in the `where` clause?
+- Does every new ClickHouse query on a project-scoped table include
+  `project_id = {projectId: String}`?
+- Does every new tRPC procedure use `protectedProjectProcedure` (or
+  equivalent) and call `throwIfNoProjectAccess` with the right scope?
+- Does every new public API route go through `withMiddlewares` +
+  `createAuthedProjectAPIRoute` (or the equivalent organization variant) with
+  the right scope?
+
+These are enforced today by the repo-wide review checklist in
+[../../code-review/references/review-checklist.md](../../code-review/references/review-checklist.md);
+restate them here so the security sweep stays self-contained.
+
+## Secrets and Credentials
+
+- Are new secrets stored encrypted at rest (e.g. via `encrypt` / `decrypt`
+  from `@langfuse/shared/encryption`) and never returned through `get`/list
+  endpoints?
+- Are secrets omitted from API responses (`select` / `omit` in Prisma) and
+  from logs?
+- Are new env vars added to `.env*.example` files and validated via the
+  package `env.mjs/ts`, not read from `process.env` directly?
+
+## Audit Logging
+
+- Do sensitive mutations (integration config changes, role or scope changes,
+  exports, deletions) emit `auditLog` with the right `action` and
+  `resourceType`? Match existing wiring in
+  `web/src/features/blobstorage-integration/blobstorage-integration-router.ts`
+  and similar routers.
+
+## Negative Tests
+
+- Does the change include tests that prove **blocked** inputs are rejected
+  (private IPs, cross-tenant IDs, missing scope, http: on Cloud)? Missing
+  negative coverage on a security-sensitive surface is a finding.
+
+## Extending
+
+When a new finding class starts to recur in PR reviews or security reports,
+add a one-line bullet here pointing at a new `references/<topic>.md`. Do not
+restate the detail in this checklist; keep it as the trigger surface.

.agents/skills/security-review/references/outbound-url-validation.md

@@ -0,0 +1,165 @@
+# Outbound URL Validation (SSRF)
+
+## Threat
+
+Any Langfuse code path that issues an outbound HTTP request to a URL derived
+from user input (mutation form, public API field, integration config, image
+proxy) can be coerced into Server-Side Request Forgery. The high-value
+internal targets in Langfuse's deployment topology include:
+
+- Cloud instance metadata services (`169.254.169.254`, IMDSv2 endpoints)
+- Internal Postgres, ClickHouse, Redis, S3/MinIO, queue admin UIs
+- Loopback admin interfaces (`127.0.0.1`, `localhost`, Docker API on
+  `2375/2376`)
+- Kubernetes API server and other in-cluster control planes
+- Any RFC1918 / RFC6598 / IPv6 ULA range routable from the pod or container
+
+Even when the surface requires `integrations:CRUD` or admin scope, the
+attacker model assumes the credentialed user is malicious or compromised;
+SSRF lets them pivot from app-level admin to network-level access that the
+deployment topology otherwise denies.
+
+## Canonical Helpers
+
+All under `packages/shared/src/server/outbound-url/`:
+
+- [`parseOutboundUrl(urlString)`](../../../../packages/shared/src/server/outbound-url/validation.ts)
+  — safe parse. Rejects embedded credentials, invalid encoding, and bad URL
+  syntax. **Use this instead of `new URL(...)` for any user-supplied URL.**
+- [`validateOutboundUrlHost({ url, whitelist, logContext, shouldSkipDnsCheckForLiteralIps })`](../../../../packages/shared/src/server/outbound-url/validation.ts)
+  — checks hostname blocklist, IP literal blocklist, and forward DNS
+  resolution against blocked CIDRs (defends DNS rebinding by resolving every
+  A/AAAA plus the local `getaddrinfo` view).
+- [`addSecureOutboundConnectionValidation(options, ...)`](../../../../packages/shared/src/server/outbound-url/connection.ts)
+  — attaches connect-time IP validation to a `fetch` request so the TCP peer
+  is re-validated after DNS resolution, not just at save time.
+- [`fetchWithSecureRedirects(...)`](../../../../packages/shared/src/server/outbound-url/fetch.ts)
+  — manual redirect handling. Validates each `Location` hop with the
+  caller-supplied validator and strips sensitive headers (`Authorization`,
+  `Cookie`, signing headers) on cross-origin redirects.
+
+Surface-specific wrappers (reuse these rather than rolling your own):
+
+- LLM base URL:
+  [`validateLlmConnectionBaseURL`](../../../../packages/shared/src/server/llm/baseUrlValidation.ts)
+- Webhook URL:
+  [`validateWebhookURL`](../../../../packages/shared/src/server/webhooks/validation.ts)
+- Blob storage endpoint:
+  [`validateBlobStorageEndpoint`](../../../../packages/shared/src/server/services/blobStorageEndpointValidation.ts)
+  and the companion
+  [`blobStorageEndpointConnectionValidationOptions`](../../../../packages/shared/src/server/services/blobStorageEndpointValidation.ts)
+  for connect-time enforcement through `StorageServiceFactory`.
+
+## Required Defenses
+
+Every outbound-URL surface MUST apply **all three** of:
+
+1. **Save-time validation** in the mutation, tRPC procedure, or public API
+   route that persists the URL. Reject the write if validation fails.
+2. **Use-time / connection-time validation** when the request is actually
+   issued (worker job, lazy validation endpoint, processor). DNS can change
+   between save and use; the SDK that ultimately makes the call may resolve a
+   different IP than the save-time check did. Plumb
+   `addSecureOutboundConnectionValidation` (or the SDK's equivalent hook)
+   through the request.
+3. **Redirect-time validation** if the request can be redirected. Plain
+   `fetch()` defaults to `redirect: 'follow'` and will silently chase a
+   redirect into the loopback range. Use `fetchWithSecureRedirects` with the
+   matching validator instead.
+
+## Known-Good Call Sites (Copy These)
+
+- LLM base URL save:
+  `web/src/features/llm-api-key/server/router.ts` (`update` mutation calls
+  `validateLlmConnectionBaseURL` before persisting).
+- LLM base URL through public API:
+  `web/src/pages/api/public/llm-connections/index.ts`.
+- Webhook URL save + use:
+  `packages/shared/src/server/webhooks/validation.ts` is wired into both the
+  automation form and the worker-side webhook sender.
+- Blob storage endpoint:
+  `web/src/features/blobstorage-integration/blobstorage-integration-router.ts`
+  (`validate` mutation calls `validateBlobStorageEndpoint`); connection-time
+  enforcement flows through `StorageServiceFactory.getInstance({
+  connectionValidation: blobStorageEndpointConnectionValidationOptions() })`.
+
+## When Adding a New Outbound URL Surface
+
+1. Identify the user-input layers: form mutation, public API route, env import,
+   anywhere the URL can be supplied by a tenant.
+2. Decide whether an existing wrapper fits. If yes, reuse it. If not, add a
+   new wrapper under `packages/shared/src/server/...` that delegates to
+   `validateOutboundUrlHost` so blocklist behavior, DNS rebinding handling,
+   and credential checks stay centralized.
+3. Define the env allowlist trio for self-hosted users who legitimately point
+   at private network targets (mirroring
+   `LANGFUSE_WEBHOOK_WHITELISTED_HOST/IPS/IP_SEGMENTS`,
+   `LANGFUSE_LLM_CONNECTION_WHITELISTED_HOST/IPS/IP_SEGMENTS`, and
+   `LANGFUSE_BLOB_STORAGE_ENDPOINT_WHITELISTED_HOST/IPS/IP_SEGMENTS`). Do
+   not share another surface's allowlist; each surface keeps its own.
+4. Add the env vars to `.env*.example` and the package `env.mjs/ts`.
+5. Call the wrapper from:
+   - every tRPC mutation that writes the URL,
+   - every public API route that accepts the URL,
+   - the worker/processor that issues the request (use connect-time
+     validation if the underlying SDK does not expose a host-validation hook).
+6. If the request can redirect, use `fetchWithSecureRedirects` with the same
+   wrapper as the validator.
+7. Add server-side tests that prove blocked targets fail validation:
+   `127.0.0.1`, `169.254.169.254`, an RFC1918 literal, an RFC1918 hostname
+   (DNS rebinding), `http://` on Cloud, and a URL containing
+   `user:pass@host`.
+
+## Anti-Patterns to Flag in Review
+
+- `fetch(<user-supplied-url>)` (or `axios`, `got`, etc.) without an upstream
+  call to a `validate*URL` helper, or without
+  `addSecureOutboundConnectionValidation` on the request options.
+- A tRPC mutation that persists a `host` / `endpoint` / `baseURL` / `webhookUrl`
+  field without invoking the matching validator before the write.
+- `StorageServiceFactory.getInstance({ endpoint })` (or any SDK client init
+  that takes a user-controlled URL) without `connectionValidation` plumbed
+  through.
+- Custom URL parsing via `new URL(userInput)` instead of
+  `parseOutboundUrl(userInput)`. The latter rejects embedded credentials and
+  bad encoding, both of which are recurring SSRF/credential-leak vectors.
+- Following redirects with `fetch(url)` (default `redirect: 'follow'`) on a
+  user-controlled URL. Switch to `fetchWithSecureRedirects`.
+- A new integration UI that validates only on the client side. Save-time
+  validation must run server-side.
+- Save-time validation present, use-time validation missing (or vice versa).
+  Both layers are required; the worker may issue the request hours after the
+  save and DNS will have moved.
+
+## Env Allowlist Behavior
+
+- Cloud (`NEXT_PUBLIC_LANGFUSE_CLOUD_REGION` set) forces strict mode.
+  Allowlist env vars are ignored on Cloud, and HTTPS is enforced for surfaces
+  that require it (LLM base URL, blob storage endpoint).
+- Self-hosted reads the per-surface env trio:
+  - `LANGFUSE_LLM_CONNECTION_WHITELISTED_HOST/IPS/IP_SEGMENTS`
+  - `LANGFUSE_WEBHOOK_WHITELISTED_HOST/IPS/IP_SEGMENTS`
+  - `LANGFUSE_BLOB_STORAGE_ENDPOINT_WHITELISTED_HOST/IPS/IP_SEGMENTS`
+- Blob storage endpoint validation is **opt-in** today on self-hosted (the
+  helper is a no-op until the operator configures one of the allowlist env
+  vars). There is a `TODO(next major)` in
+  `blobStorageEndpointValidation.ts` to flip the default; until then, do not
+  rely on blob storage validation for new surfaces — wire your own wrapper
+  that defaults to strict.
+
+## Negative Tests (Required)
+
+A change that adds a new outbound URL surface MUST include server-side tests
+that assert each of the following fails validation:
+
+- Loopback literal (`http://127.0.0.1`, `http://[::1]`)
+- Cloud metadata literal (`http://169.254.169.254`)
+- RFC1918 literal (`http://10.0.0.1`)
+- Hostname that resolves to a private IP (DNS rebinding sanity check)
+- URL with embedded credentials (`http://user:pass@host`)
+- `http://` on Cloud (where HTTPS is required)
+- An empty allowlist permits no internal targets on self-hosted
+
+Pattern reference:
+`worker/src/__tests__/llm-base-url-validation.test.ts` and the test files
+adjacent to the wrappers above.

.agents/skills/skill-creator/SKILL.md

@@ -0,0 +1,456 @@
+---
+name: skill-creator
+description: Guide for creating effective skills. This skill should be used when users want to create a new skill (or update an existing skill) that extends Codex's capabilities with specialized knowledge, workflows, or tool integrations.
+metadata:
+  short-description: Create or update Codex skills
+---
+
+# Skill Creator
+
+This skill provides guidance for creating effective skills.
+
+## About Skills
+
+Skills are modular, self-contained folders that extend Codex's capabilities by providing
+specialized knowledge, workflows, and tools. Think of them as "onboarding guides" for specific
+domains or tasks—they transform Codex from a general-purpose agent into a specialized agent
+equipped with procedural knowledge that no model can fully possess.
+
+### What Skills Provide
+
+1. Specialized workflows - Multi-step procedures for specific domains
+2. Tool integrations - Instructions for working with specific file formats or APIs
+3. Domain expertise - Company-specific knowledge, schemas, business logic
+4. Bundled resources - Scripts, references, and assets for complex and repetitive tasks
+
+## Core Principles
+
+### Concise is Key
+
+The context window is a public good. Skills share the context window with everything else Codex needs: system prompt, conversation history, other Skills' metadata, and the actual user request.
+
+**Default assumption: Codex is already very smart.** Only add context Codex doesn't already have. Challenge each piece of information: "Does Codex really need this explanation?" and "Does this paragraph justify its token cost?"
+
+Prefer concise examples over verbose explanations.
+
+### Set Appropriate Degrees of Freedom
+
+Match the level of specificity to the task's fragility and variability:
+
+**High freedom (text-based instructions)**: Use when multiple approaches are valid, decisions depend on context, or heuristics guide the approach.
+
+**Medium freedom (pseudocode or scripts with parameters)**: Use when a preferred pattern exists, some variation is acceptable, or configuration affects behavior.
+
+**Low freedom (specific scripts, few parameters)**: Use when operations are fragile and error-prone, consistency is critical, or a specific sequence must be followed.
+
+Think of Codex as exploring a path: a narrow bridge with cliffs needs specific guardrails (low freedom), while an open field allows many routes (high freedom).
+
+### Require Human Review for Ticket Writes
+
+For skills that create Linear tickets, update Linear tickets, or add evidence to
+existing tickets, require human review before any write. The skill must present
+all findings in a table, ask the human which findings to create or update in
+Linear, and wait for an explicit selection before making changes.
+
+Use this table structure unless the domain needs additional columns:
+
+| ID | Finding | Evidence | Impact / Scope | Existing Ticket Match | Proposed Linear Action | Confidence | Human Decision |
+| --- | --- | --- | --- | --- | --- | --- | --- |
+| F1 | Concise symptom or bug claim | Measured counts, deltas, links, traces, logs, or "No measurements found" | Affected env, service, route, customer segment, or blast radius supported by evidence | Existing issue key/link, duplicate candidate, or "None found" | Create new ticket, add evidence comment, update status/labels, or no action | High/medium/low plus one short reason | Leave blank for the human to choose |
+
+In the skill instructions, state that Codex must not create tickets, comment on
+tickets, edit ticket fields, or add evidence until the human chooses one or more
+row IDs and actions. If the human asks for an automated sweep, still pause at
+this review table before writing to Linear.
+
+### Protect Validation Integrity
+
+You may use subagents during iteration to validate whether a skill works on realistic tasks or whether a suspected problem is real. This is most useful when you want an independent pass on the skill's behavior, outputs, or failure modes after a revision.  Only do this when it is possible to start new subagents.
+
+When using subagents for validation, treat that as an evaluation surface. The goal is to learn whether the skill generalizes, not whether another agent can reconstruct the answer from leaked context.
+
+Prefer raw artifacts such as example prompts, outputs, diffs, logs, or traces. Give the minimum task-local context needed to perform the validation. Avoid passing the intended answer, suspected bug, intended fix, or your prior conclusions unless the validation explicitly requires them.
+
+### Require Valid Markdown Output
+
+When a skill instructs Codex to return Markdown, require the final output to be
+valid Markdown, not just Markdown-like text.
+
+In the skill instructions, explicitly require Codex to:
+
+- use valid Markdown syntax for headings, lists, links, tables, and code fences;
+- include a space after list markers such as `-`, `*`, and `1.`;
+- close every Markdown link and parenthesis correctly;
+- avoid malformed tables, dangling backticks, or partially opened fenced code
+  blocks;
+- prefer plain paragraphs over complex formatting when the structure would be
+  fragile.
+
+If a skill produces structured reports, include a short output-format section
+that says the response must be valid Markdown and should be checked for basic
+syntax mistakes before returning it.
+
+### Anatomy of a Skill
+
+Every skill consists of a required SKILL.md file and optional bundled resources:
+
+```
+skill-name/
+├── SKILL.md (required)
+│   ├── YAML frontmatter metadata (required)
+│   │   ├── name: (required)
+│   │   └── description: (required)
+│   └── Markdown instructions (required)
+├── agents/ (recommended)
+│   └── openai.yaml - UI metadata for skill lists and chips
+└── Bundled Resources (optional)
+    ├── scripts/          - Executable code (Python/Bash/etc.)
+    ├── references/       - Documentation intended to be loaded into context as needed
+    └── assets/           - Files used in output (templates, icons, fonts, etc.)
+```
+
+#### SKILL.md (required)
+
+Every SKILL.md consists of:
+
+- **Frontmatter** (YAML): Contains `name` and `description` fields. These are the only fields that Codex reads to determine when the skill gets used, thus it is very important to be clear and comprehensive in describing what the skill is, and when it should be used.
+- **Body** (Markdown): Instructions and guidance for using the skill. Only loaded AFTER the skill triggers (if at all).
+
+#### Agents metadata (recommended)
+
+- UI-facing metadata for skill lists and chips
+- Read references/openai_yaml.md before generating values and follow its descriptions and constraints
+- Create: human-facing `display_name`, `short_description`, and `default_prompt` by reading the skill
+- Generate deterministically by passing the values as `--interface key=value` to `scripts/generate_openai_yaml.py` or `scripts/init_skill.py`
+- On updates: validate `agents/openai.yaml` still matches SKILL.md; regenerate if stale
+- Only include other optional interface fields (icons, brand color) if explicitly provided
+- See references/openai_yaml.md for field definitions and examples
+
+#### Bundled Resources (optional)
+
+##### Scripts (`scripts/`)
+
+Executable code (Python/Bash/etc.) for tasks that require deterministic reliability or are repeatedly rewritten.
+
+- **When to include**: When the same code is being rewritten repeatedly or deterministic reliability is needed
+- **Example**: `scripts/rotate_pdf.py` for PDF rotation tasks
+- **Benefits**: Token efficient, deterministic, may be executed without loading into context
+- **Note**: Scripts may still need to be read by Codex for patching or environment-specific adjustments
+
+##### References (`references/`)
+
+Documentation and reference material intended to be loaded as needed into context to inform Codex's process and thinking.
+
+- **When to include**: For documentation that Codex should reference while working
+- **Examples**: `references/finance.md` for financial schemas, `references/mnda.md` for company NDA template, `references/policies.md` for company policies, `references/api_docs.md` for API specifications
+- **Use cases**: Database schemas, API documentation, domain knowledge, company policies, detailed workflow guides
+- **Benefits**: Keeps SKILL.md lean, loaded only when Codex determines it's needed
+- **Best practice**: If files are large (>10k words), include grep search patterns in SKILL.md
+- **Avoid duplication**: Information should live in either SKILL.md or references files, not both. Prefer references files for detailed information unless it's truly core to the skill—this keeps SKILL.md lean while making information discoverable without hogging the context window. Keep only essential procedural instructions and workflow guidance in SKILL.md; move detailed reference material, schemas, and examples to references files.
+
+##### Assets (`assets/`)
+
+Files not intended to be loaded into context, but rather used within the output Codex produces.
+
+- **When to include**: When the skill needs files that will be used in the final output
+- **Examples**: `assets/logo.png` for brand assets, `assets/slides.pptx` for PowerPoint templates, `assets/frontend-template/` for HTML/React boilerplate, `assets/font.ttf` for typography
+- **Use cases**: Templates, images, icons, boilerplate code, fonts, sample documents that get copied or modified
+- **Benefits**: Separates output resources from documentation, enables Codex to use files without loading them into context
+
+#### What to Not Include in a Skill
+
+A skill should only contain essential files that directly support its functionality. Do NOT create extraneous documentation or auxiliary files, including:
+
+- README.md
+- INSTALLATION_GUIDE.md
+- QUICK_REFERENCE.md
+- CHANGELOG.md
+- etc.
+
+The skill should only contain the information needed for an AI agent to do the job at hand. It should not contain auxiliary context about the process that went into creating it, setup and testing procedures, user-facing documentation, etc. Creating additional documentation files just adds clutter and confusion.
+
+### Progressive Disclosure Design Principle
+
+Skills use a three-level loading system to manage context efficiently:
+
+1. **Metadata (name + description)** - Always in context (~100 words)
+2. **SKILL.md body** - When skill triggers (<5k words)
+3. **Bundled resources** - As needed by Codex (Unlimited because scripts can be executed without reading into context window)
+
+#### Progressive Disclosure Patterns
+
+Keep SKILL.md body to the essentials and under 500 lines to minimize context bloat. Split content into separate files when approaching this limit. When splitting out content into other files, it is very important to reference them from SKILL.md and describe clearly when to read them, to ensure the reader of the skill knows they exist and when to use them.
+
+**Key principle:** When a skill supports multiple variations, frameworks, or options, keep only the core workflow and selection guidance in SKILL.md. Move variant-specific details (patterns, examples, configuration) into separate reference files.
+
+**Pattern 1: High-level guide with references**
+
+```markdown
+# PDF Processing
+
+## Quick start
+
+Extract text with pdfplumber:
+[code example]
+
+## Advanced features
+
+- **Form filling**: See [FORMS.md](FORMS.md) for complete guide
+- **API reference**: See [REFERENCE.md](REFERENCE.md) for all methods
+- **Examples**: See [EXAMPLES.md](EXAMPLES.md) for common patterns
+```
+
+Codex loads FORMS.md, REFERENCE.md, or EXAMPLES.md only when needed.
+
+**Pattern 2: Domain-specific organization**
+
+For Skills with multiple domains, organize content by domain to avoid loading irrelevant context:
+
+```
+bigquery-skill/
+├── SKILL.md (overview and navigation)
+└── reference/
+    ├── finance.md (revenue, billing metrics)
+    ├── sales.md (opportunities, pipeline)
+    ├── product.md (API usage, features)
+    └── marketing.md (campaigns, attribution)
+```
+
+When a user asks about sales metrics, Codex only reads sales.md.
+
+Similarly, for skills supporting multiple frameworks or variants, organize by variant:
+
+```
+cloud-deploy/
+├── SKILL.md (workflow + provider selection)
+└── references/
+    ├── aws.md (AWS deployment patterns)
+    ├── gcp.md (GCP deployment patterns)
+    └── azure.md (Azure deployment patterns)
+```
+
+When the user chooses AWS, Codex only reads aws.md.
+
+**Pattern 3: Conditional details**
+
+Show basic content, link to advanced content:
+
+```markdown
+# DOCX Processing
+
+## Creating documents
+
+Use docx-js for new documents. See [DOCX-JS.md](DOCX-JS.md).
+
+## Editing documents
+
+For simple edits, modify the XML directly.
+
+**For tracked changes**: See [REDLINING.md](REDLINING.md)
+**For OOXML details**: See [OOXML.md](OOXML.md)
+```
+
+Codex reads REDLINING.md or OOXML.md only when the user needs those features.
+
+**Important guidelines:**
+
+- **Avoid deeply nested references** - Keep references one level deep from SKILL.md. All reference files should link directly from SKILL.md.
+- **Structure longer reference files** - For files longer than 100 lines, include a table of contents at the top so Codex can see the full scope when previewing.
+
+## Skill Creation Process
+
+Skill creation involves these steps:
+
+1. Understand the skill with concrete examples
+2. Plan reusable skill contents (scripts, references, assets)
+3. Initialize the skill (run init_skill.py)
+4. Edit the skill (implement resources and write SKILL.md)
+5. Validate the skill (run quick_validate.py)
+6. Iterate based on real usage and forward-test complex skills.
+
+Follow these steps in order, skipping only if there is a clear reason why they are not applicable.
+
+### Skill Naming
+
+- Use lowercase letters, digits, and hyphens only; normalize user-provided titles to hyphen-case (e.g., "Plan Mode" -> `plan-mode`).
+- When generating names, generate a name under 64 characters (letters, digits, hyphens).
+- Prefer short, verb-led phrases that describe the action.
+- Namespace by tool when it improves clarity or triggering (e.g., `gh-address-comments`, `linear-address-issue`).
+- Name the skill folder exactly after the skill name.
+
+### Step 1: Understanding the Skill with Concrete Examples
+
+Skip this step only when the skill's usage patterns are already clearly understood. It remains valuable even when working with an existing skill.
+
+To create an effective skill, clearly understand concrete examples of how the skill will be used. This understanding can come from either direct user examples or generated examples that are validated with user feedback.
+
+For example, when building an image-editor skill, relevant questions include:
+
+- "What functionality should the image-editor skill support? Editing, rotating, anything else?"
+- "Can you give some examples of how this skill would be used?"
+- "I can imagine users asking for things like 'Remove the red-eye from this image' or 'Rotate this image'. Are there other ways you imagine this skill being used?"
+- "What would a user say that should trigger this skill?"
+- "Where should I create this skill? If you do not have a preference, I will place it in `$CODEX_HOME/skills` (or `~/.codex/skills` when `CODEX_HOME` is unset) so Codex can discover it automatically."
+
+To avoid overwhelming users, avoid asking too many questions in a single message. Start with the most important questions and follow up as needed for better effectiveness.
+
+Conclude this step when there is a clear sense of the functionality the skill should support.
+
+### Step 2: Planning the Reusable Skill Contents
+
+To turn concrete examples into an effective skill, analyze each example by:
+
+1. Considering how to execute on the example from scratch
+2. Identifying what scripts, references, and assets would be helpful when executing these workflows repeatedly
+
+Example: When building a `pdf-editor` skill to handle queries like "Help me rotate this PDF," the analysis shows:
+
+1. Rotating a PDF requires re-writing the same code each time
+2. A `scripts/rotate_pdf.py` script would be helpful to store in the skill
+
+Example: When designing a `frontend-webapp-builder` skill for queries like "Build me a todo app" or "Build me a dashboard to track my steps," the analysis shows:
+
+1. Writing a frontend webapp requires the same boilerplate HTML/React each time
+2. An `assets/hello-world/` template containing the boilerplate HTML/React project files would be helpful to store in the skill
+
+Example: When building a `big-query` skill to handle queries like "How many users have logged in today?" the analysis shows:
+
+1. Querying BigQuery requires re-discovering the table schemas and relationships each time
+2. A `references/schema.md` file documenting the table schemas would be helpful to store in the skill
+
+To establish the skill's contents, analyze each concrete example to create a list of the reusable resources to include: scripts, references, and assets.
+
+### Step 3: Initializing the Skill
+
+At this point, it is time to actually create the skill.
+
+Skip this step only if the skill being developed already exists. In this case, continue to the next step.
+
+Before running `init_skill.py`, ask where the user wants the skill created. If they do not specify a location, default to `$CODEX_HOME/skills`; when `CODEX_HOME` is unset, fall back to `~/.codex/skills` so the skill is auto-discovered.
+
+When creating a new skill from scratch, always run the `init_skill.py` script. The script conveniently generates a new template skill directory that automatically includes everything a skill requires, making the skill creation process much more efficient and reliable.
+
+Usage:
+
+```bash
+scripts/init_skill.py <skill-name> --path <output-directory> [--resources scripts,references,assets] [--examples]
+```
+
+Examples:
+
+```bash
+scripts/init_skill.py my-skill --path "${CODEX_HOME:-$HOME/.codex}/skills"
+scripts/init_skill.py my-skill --path "${CODEX_HOME:-$HOME/.codex}/skills" --resources scripts,references
+scripts/init_skill.py my-skill --path ~/work/skills --resources scripts --examples
+```
+
+The script:
+
+- Creates the skill directory at the specified path
+- Generates a SKILL.md template with proper frontmatter and TODO placeholders
+- Creates `agents/openai.yaml` using agent-generated `display_name`, `short_description`, and `default_prompt` passed via `--interface key=value`
+- Optionally creates resource directories based on `--resources`
+- Optionally adds example files when `--examples` is set
+
+After initialization, customize the SKILL.md and add resources as needed. If you used `--examples`, replace or delete placeholder files.
+
+Generate `display_name`, `short_description`, and `default_prompt` by reading the skill, then pass them as `--interface key=value` to `init_skill.py` or regenerate with:
+
+```bash
+scripts/generate_openai_yaml.py <path/to/skill-folder> --interface key=value
+```
+
+Only include other optional interface fields when the user explicitly provides them. For full field descriptions and examples, see references/openai_yaml.md.
+
+### Step 4: Edit the Skill
+
+When editing the (newly-generated or existing) skill, remember that the skill is being created for another instance of Codex to use. Include information that would be beneficial and non-obvious to Codex. Consider what procedural knowledge, domain-specific details, or reusable assets would help another Codex instance execute these tasks more effectively.
+
+After substantial revisions, or if the skill is particularly tricky, you should use subagents to forward-test the skill on realistic tasks or artifacts. When doing so, pass the artifact under validation rather than your diagnosis of what is wrong, and keep the prompt generic enough that success depends on transferable reasoning rather than hidden ground truth.
+
+#### Start with Reusable Skill Contents
+
+To begin implementation, start with the reusable resources identified above: `scripts/`, `references/`, and `assets/` files. Note that this step may require user input. For example, when implementing a `brand-guidelines` skill, the user may need to provide brand assets or templates to store in `assets/`, or documentation to store in `references/`.
+
+Added scripts must be tested by actually running them to ensure there are no bugs and that the output matches what is expected. If there are many similar scripts, only a representative sample needs to be tested to ensure confidence that they all work while balancing time to completion.
+
+If you used `--examples`, delete any placeholder files that are not needed for the skill. Only create resource directories that are actually required.
+
+#### Update SKILL.md
+
+**Writing Guidelines:** Always use imperative/infinitive form.
+
+##### Frontmatter
+
+Write the YAML frontmatter with `name` and `description`:
+
+- `name`: The skill name
+- `description`: This is the primary triggering mechanism for your skill, and helps Codex understand when to use the skill.
+  - Include both what the Skill does and specific triggers/contexts for when to use it.
+  - Include all "when to use" information here - Not in the body. The body is only loaded after triggering, so "When to Use This Skill" sections in the body are not helpful to Codex.
+  - Example description for a `docx` skill: "Comprehensive document creation, editing, and analysis with support for tracked changes, comments, formatting preservation, and text extraction. Use when Codex needs to work with professional documents (.docx files) for: (1) Creating new documents, (2) Modifying or editing content, (3) Working with tracked changes, (4) Adding comments, or any other document tasks"
+
+Do not include any other fields in YAML frontmatter.
+
+##### Body
+
+Write instructions for using the skill and its bundled resources.
+
+If the skill expects Markdown output, add an explicit output-format section that
+requires valid Markdown syntax in the final response.
+
+### Step 5: Validate the Skill
+
+Once development of the skill is complete, validate the skill folder to catch basic issues early:
+
+```bash
+scripts/quick_validate.py <path/to/skill-folder>
+```
+
+The validation script checks YAML frontmatter format, required fields, and naming rules. If validation fails, fix the reported issues and run the command again.
+
+### Step 6: Iterate
+
+After testing the skill, you may detect the skill is complex enough that it requires forward-testing; or users may request improvements.
+
+User testing often this happens right after using the skill, with fresh context of how the skill performed.
+
+**Forward-testing and iteration workflow:**
+
+1. Use the skill on real tasks
+2. Notice struggles or inefficiencies
+3. Identify how SKILL.md or bundled resources should be updated
+4. Implement changes and test again
+5. Forward-test if it is reasonable and appropriate
+
+## Forward-testing
+
+To forward-test, launch subagents as a way to stress test the skill with minimal context.
+Subagents should *not* know that they are being asked to test the skill.  They should be treated as
+an agent asked to perform a task by the user.  Prompts to subagents should look like:
+  `Use $skill-x at /path/to/skill-x to solve problem y`
+Not:
+  `Review the skill at /path/to/skill-x; pretend a user asks you to...`
+
+Decision rule for forward-testing:
+  - Err on the side of forward-testing
+  - Ask for approval if you think there's a risk that forward-testing would:
+    * take a long time,
+    * require additional approvals from the user, or
+    * modify live production systems
+
+  In these cases, show the user your proposed prompt and request (1) a yes/no decision, and
+  (2) any suggested modifictions.
+
+Considerations when forward-testing:
+   - use fresh threads for independent passes
+   - pass the skill, and a request in a similar way the user would.
+   - pass raw artifacts, not your conclusions
+   - avoid showing expected answers or intended fixes
+   - rebuild context from source artifacts after each iteration
+   - review the subagent's output and reasoning and emitted artifacts
+   - avoid leaving artifacts the agent can find on disk between iterations;
+     clean up subagents' artifacts to avoid additional contamination.
+
+If forward-testing only succeeds when subagents see leaked context, tighten the skill or the
+forward-testing setup before trusting the result.

.agents/skills/skill-creator/agents/openai.yaml

@@ -0,0 +1,6 @@
+interface:
+  display_name: "Skill Creator"
+  short_description: "Create or update Codex skills"
+  icon_small: "./assets/skill-creator-small.svg"
+  icon_large: "./assets/skill-creator.png"
+  default_prompt: "Use $skill-creator to create or refine a concise Codex skill."

.agents/skills/skill-creator/assets/skill-creator-small.svg

@@ -0,0 +1,3 @@
+<svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" fill="currentColor" viewBox="0 0 20 20">
+  <path fill="#0D0D0D" d="M12.03 4.113a3.612 3.612 0 0 1 5.108 5.108l-6.292 6.29c-.324.324-.56.561-.791.752l-.235.176c-.205.14-.422.261-.65.36l-.229.093a4.136 4.136 0 0 1-.586.16l-.764.134-2.394.4c-.142.024-.294.05-.423.06-.098.007-.232.01-.378-.026l-.149-.05a1.081 1.081 0 0 1-.521-.474l-.046-.093a1.104 1.104 0 0 1-.075-.527c.01-.129.035-.28.06-.422l.398-2.394c.1-.602.162-.987.295-1.35l.093-.23c.1-.228.22-.445.36-.65l.176-.235c.19-.232.428-.467.751-.79l6.292-6.292Zm-5.35 7.232c-.35.35-.534.535-.66.688l-.11.147a2.67 2.67 0 0 0-.24.433l-.062.154c-.08.22-.124.462-.232 1.112l-.398 2.394-.001.001h.003l2.393-.399.717-.126a2.63 2.63 0 0 0 .394-.105l.154-.063a2.65 2.65 0 0 0 .433-.24l.147-.11c.153-.126.339-.31.688-.66l4.988-4.988-3.227-3.226-4.987 4.988Zm9.517-6.291a2.281 2.281 0 0 0-3.225 0l-.364.362 3.226 3.227.363-.364c.89-.89.89-2.334 0-3.225ZM4.583 1.783a.3.3 0 0 1 .294.241c.117.585.347 1.092.707 1.48.357.385.859.668 1.549.783a.3.3 0 0 1 0 .592c-.69.115-1.192.398-1.549.783-.315.34-.53.77-.657 1.265l-.05.215a.3.3 0 0 1-.588 0c-.117-.585-.347-1.092-.707-1.48-.357-.384-.859-.668-1.549-.783a.3.3 0 0 1 0-.592c.69-.115 1.192-.398 1.549-.783.36-.388.59-.895.707-1.48l.015-.05a.3.3 0 0 1 .279-.19Z"/>
+</svg>

.agents/skills/skill-creator/license.txt

@@ -0,0 +1,202 @@
+
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [yyyy] [name of copyright owner]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

.agents/skills/skill-creator/references/openai_yaml.md

@@ -0,0 +1,49 @@
+# openai.yaml fields (full example + descriptions)
+
+`agents/openai.yaml` is an extended, product-specific config intended for the machine/harness to read, not the agent. Other product-specific config can also live in the `agents/` folder.
+
+## Full example
+
+```yaml
+interface:
+  display_name: "Optional user-facing name"
+  short_description: "Optional user-facing description"
+  icon_small: "./assets/small-400px.png"
+  icon_large: "./assets/large-logo.svg"
+  brand_color: "#3B82F6"
+  default_prompt: "Optional surrounding prompt to use the skill with"
+
+dependencies:
+  tools:
+    - type: "mcp"
+      value: "github"
+      description: "GitHub MCP server"
+      transport: "streamable_http"
+      url: "https://api.githubcopilot.com/mcp/"
+
+policy:
+  allow_implicit_invocation: true
+```
+
+## Field descriptions and constraints
+
+Top-level constraints:
+
+- Quote all string values.
+- Keep keys unquoted.
+- For `interface.default_prompt`: generate a helpful, short (typically 1 sentence) example starting prompt based on the skill. It must explicitly mention the skill as `$skill-name` (e.g., "Use $skill-name-here to draft a concise weekly status update.").
+
+- `interface.display_name`: Human-facing title shown in UI skill lists and chips.
+- `interface.short_description`: Human-facing short UI blurb (25–64 chars) for quick scanning.
+- `interface.icon_small`: Path to a small icon asset (relative to skill dir). Default to `./assets/` and place icons in the skill's `assets/` folder.
+- `interface.icon_large`: Path to a larger logo asset (relative to skill dir). Default to `./assets/` and place icons in the skill's `assets/` folder.
+- `interface.brand_color`: Hex color used for UI accents (e.g., badges).
+- `interface.default_prompt`: Default prompt snippet inserted when invoking the skill.
+- `dependencies.tools[].type`: Dependency category. Only `mcp` is supported for now.
+- `dependencies.tools[].value`: Identifier of the tool or dependency.
+- `dependencies.tools[].description`: Human-readable explanation of the dependency.
+- `dependencies.tools[].transport`: Connection type when `type` is `mcp`.
+- `dependencies.tools[].url`: MCP server URL when `type` is `mcp`.
+- `policy.allow_implicit_invocation`: When false, the skill is not injected into
+  the model context by default, but can still be invoked explicitly via `$skill`.
+  Defaults to true.

.agents/skills/skill-creator/scripts/generate_openai_yaml.py

@@ -0,0 +1,226 @@
+#!/usr/bin/env python3
+"""
+OpenAI YAML Generator - Creates agents/openai.yaml for a skill folder.
+
+Usage:
+    generate_openai_yaml.py <skill_dir> [--name <skill_name>] [--interface key=value]
+"""
+
+import argparse
+import re
+import sys
+from pathlib import Path
+
+ACRONYMS = {
+    "GH",
+    "MCP",
+    "API",
+    "CI",
+    "CLI",
+    "LLM",
+    "PDF",
+    "PR",
+    "UI",
+    "URL",
+    "SQL",
+}
+
+BRANDS = {
+    "openai": "OpenAI",
+    "openapi": "OpenAPI",
+    "github": "GitHub",
+    "pagerduty": "PagerDuty",
+    "datadog": "DataDog",
+    "sqlite": "SQLite",
+    "fastapi": "FastAPI",
+}
+
+SMALL_WORDS = {"and", "or", "to", "up", "with"}
+
+ALLOWED_INTERFACE_KEYS = {
+    "display_name",
+    "short_description",
+    "icon_small",
+    "icon_large",
+    "brand_color",
+    "default_prompt",
+}
+
+
+def yaml_quote(value):
+    escaped = value.replace("\\", "\\\\").replace('"', '\\"').replace("\n", "\\n")
+    return f'"{escaped}"'
+
+
+def format_display_name(skill_name):
+    words = [word for word in skill_name.split("-") if word]
+    formatted = []
+    for index, word in enumerate(words):
+        lower = word.lower()
+        upper = word.upper()
+        if upper in ACRONYMS:
+            formatted.append(upper)
+            continue
+        if lower in BRANDS:
+            formatted.append(BRANDS[lower])
+            continue
+        if index > 0 and lower in SMALL_WORDS:
+            formatted.append(lower)
+            continue
+        formatted.append(word.capitalize())
+    return " ".join(formatted)
+
+
+def generate_short_description(display_name):
+    description = f"Help with {display_name} tasks"
+
+    if len(description) < 25:
+        description = f"Help with {display_name} tasks and workflows"
+    if len(description) < 25:
+        description = f"Help with {display_name} tasks with guidance"
+
+    if len(description) > 64:
+        description = f"Help with {display_name}"
+    if len(description) > 64:
+        description = f"{display_name} helper"
+    if len(description) > 64:
+        description = f"{display_name} tools"
+    if len(description) > 64:
+        suffix = " helper"
+        max_name_length = 64 - len(suffix)
+        trimmed = display_name[:max_name_length].rstrip()
+        description = f"{trimmed}{suffix}"
+    if len(description) > 64:
+        description = description[:64].rstrip()
+
+    if len(description) < 25:
+        description = f"{description} workflows"
+        if len(description) > 64:
+            description = description[:64].rstrip()
+
+    return description
+
+
+def read_frontmatter_name(skill_dir):
+    skill_md = Path(skill_dir) / "SKILL.md"
+    if not skill_md.exists():
+        print(f"[ERROR] SKILL.md not found in {skill_dir}")
+        return None
+    content = skill_md.read_text()
+    match = re.match(r"^---\n(.*?)\n---", content, re.DOTALL)
+    if not match:
+        print("[ERROR] Invalid SKILL.md frontmatter format.")
+        return None
+    frontmatter_text = match.group(1)
+
+    import yaml
+
+    try:
+        frontmatter = yaml.safe_load(frontmatter_text)
+    except yaml.YAMLError as exc:
+        print(f"[ERROR] Invalid YAML frontmatter: {exc}")
+        return None
+    if not isinstance(frontmatter, dict):
+        print("[ERROR] Frontmatter must be a YAML dictionary.")
+        return None
+    name = frontmatter.get("name", "")
+    if not isinstance(name, str) or not name.strip():
+        print("[ERROR] Frontmatter 'name' is missing or invalid.")
+        return None
+    return name.strip()
+
+
+def parse_interface_overrides(raw_overrides):
+    overrides = {}
+    optional_order = []
+    for item in raw_overrides:
+        if "=" not in item:
+            print(f"[ERROR] Invalid interface override '{item}'. Use key=value.")
+            return None, None
+        key, value = item.split("=", 1)
+        key = key.strip()
+        value = value.strip()
+        if not key:
+            print(f"[ERROR] Invalid interface override '{item}'. Key is empty.")
+            return None, None
+        if key not in ALLOWED_INTERFACE_KEYS:
+            allowed = ", ".join(sorted(ALLOWED_INTERFACE_KEYS))
+            print(f"[ERROR] Unknown interface field '{key}'. Allowed: {allowed}")
+            return None, None
+        overrides[key] = value
+        if key not in ("display_name", "short_description") and key not in optional_order:
+            optional_order.append(key)
+    return overrides, optional_order
+
+
+def write_openai_yaml(skill_dir, skill_name, raw_overrides):
+    overrides, optional_order = parse_interface_overrides(raw_overrides)
+    if overrides is None:
+        return None
+
+    display_name = overrides.get("display_name") or format_display_name(skill_name)
+    short_description = overrides.get("short_description") or generate_short_description(display_name)
+
+    if not (25 <= len(short_description) <= 64):
+        print(
+            "[ERROR] short_description must be 25-64 characters "
+            f"(got {len(short_description)})."
+        )
+        return None
+
+    interface_lines = [
+        "interface:",
+        f"  display_name: {yaml_quote(display_name)}",
+        f"  short_description: {yaml_quote(short_description)}",
+    ]
+
+    for key in optional_order:
+        value = overrides.get(key)
+        if value is not None:
+            interface_lines.append(f"  {key}: {yaml_quote(value)}")
+
+    agents_dir = Path(skill_dir) / "agents"
+    agents_dir.mkdir(parents=True, exist_ok=True)
+    output_path = agents_dir / "openai.yaml"
+    output_path.write_text("\n".join(interface_lines) + "\n")
+    print(f"[OK] Created agents/openai.yaml")
+    return output_path
+
+
+def main():
+    parser = argparse.ArgumentParser(
+        description="Create agents/openai.yaml for a skill directory.",
+    )
+    parser.add_argument("skill_dir", help="Path to the skill directory")
+    parser.add_argument(
+        "--name",
+        help="Skill name override (defaults to SKILL.md frontmatter)",
+    )
+    parser.add_argument(
+        "--interface",
+        action="append",
+        default=[],
+        help="Interface override in key=value format (repeatable)",
+    )
+    args = parser.parse_args()
+
+    skill_dir = Path(args.skill_dir).resolve()
+    if not skill_dir.exists():
+        print(f"[ERROR] Skill directory not found: {skill_dir}")
+        sys.exit(1)
+    if not skill_dir.is_dir():
+        print(f"[ERROR] Path is not a directory: {skill_dir}")
+        sys.exit(1)
+
+    skill_name = args.name or read_frontmatter_name(skill_dir)
+    if not skill_name:
+        sys.exit(1)
+
+    result = write_openai_yaml(skill_dir, skill_name, args.interface)
+    if result:
+        sys.exit(0)
+    sys.exit(1)
+
+
+if __name__ == "__main__":
+    main()

.agents/skills/skill-creator/scripts/init_skill.py

@@ -0,0 +1,400 @@
+#!/usr/bin/env python3
+"""
+Skill Initializer - Creates a new skill from template
+
+Usage:
+    init_skill.py <skill-name> --path <path> [--resources scripts,references,assets] [--examples] [--interface key=value]
+
+Examples:
+    init_skill.py my-new-skill --path skills/public
+    init_skill.py my-new-skill --path skills/public --resources scripts,references
+    init_skill.py my-api-helper --path skills/private --resources scripts --examples
+    init_skill.py custom-skill --path /custom/location
+    init_skill.py my-skill --path skills/public --interface short_description="Short UI label"
+"""
+
+import argparse
+import re
+import sys
+from pathlib import Path
+
+from generate_openai_yaml import write_openai_yaml
+
+MAX_SKILL_NAME_LENGTH = 64
+ALLOWED_RESOURCES = {"scripts", "references", "assets"}
+
+SKILL_TEMPLATE = """---
+name: {skill_name}
+description: [TODO: Complete and informative explanation of what the skill does and when to use it. Include WHEN to use this skill - specific scenarios, file types, or tasks that trigger it.]
+---
+
+# {skill_title}
+
+## Overview
+
+[TODO: 1-2 sentences explaining what this skill enables]
+
+## Structuring This Skill
+
+[TODO: Choose the structure that best fits this skill's purpose. Common patterns:
+
+**1. Workflow-Based** (best for sequential processes)
+- Works well when there are clear step-by-step procedures
+- Example: DOCX skill with "Workflow Decision Tree" -> "Reading" -> "Creating" -> "Editing"
+- Structure: ## Overview -> ## Workflow Decision Tree -> ## Step 1 -> ## Step 2...
+
+**2. Task-Based** (best for tool collections)
+- Works well when the skill offers different operations/capabilities
+- Example: PDF skill with "Quick Start" -> "Merge PDFs" -> "Split PDFs" -> "Extract Text"
+- Structure: ## Overview -> ## Quick Start -> ## Task Category 1 -> ## Task Category 2...
+
+**3. Reference/Guidelines** (best for standards or specifications)
+- Works well for brand guidelines, coding standards, or requirements
+- Example: Brand styling with "Brand Guidelines" -> "Colors" -> "Typography" -> "Features"
+- Structure: ## Overview -> ## Guidelines -> ## Specifications -> ## Usage...
+
+**4. Capabilities-Based** (best for integrated systems)
+- Works well when the skill provides multiple interrelated features
+- Example: Product Management with "Core Capabilities" -> numbered capability list
+- Structure: ## Overview -> ## Core Capabilities -> ### 1. Feature -> ### 2. Feature...
+
+Patterns can be mixed and matched as needed. Most skills combine patterns (e.g., start with task-based, add workflow for complex operations).
+
+Delete this entire "Structuring This Skill" section when done - it's just guidance.]
+
+## [TODO: Replace with the first main section based on chosen structure]
+
+[TODO: Add content here. See examples in existing skills:
+- Code samples for technical skills
+- Decision trees for complex workflows
+- Concrete examples with realistic user requests
+- References to scripts/templates/references as needed]
+
+## Resources (optional)
+
+Create only the resource directories this skill actually needs. Delete this section if no resources are required.
+
+### scripts/
+Executable code (Python/Bash/etc.) that can be run directly to perform specific operations.
+
+**Examples from other skills:**
+- PDF skill: `fill_fillable_fields.py`, `extract_form_field_info.py` - utilities for PDF manipulation
+- DOCX skill: `document.py`, `utilities.py` - Python modules for document processing
+
+**Appropriate for:** Python scripts, shell scripts, or any executable code that performs automation, data processing, or specific operations.
+
+**Note:** Scripts may be executed without loading into context, but can still be read by Codex for patching or environment adjustments.
+
+### references/
+Documentation and reference material intended to be loaded into context to inform Codex's process and thinking.
+
+**Examples from other skills:**
+- Product management: `communication.md`, `context_building.md` - detailed workflow guides
+- BigQuery: API reference documentation and query examples
+- Finance: Schema documentation, company policies
+
+**Appropriate for:** In-depth documentation, API references, database schemas, comprehensive guides, or any detailed information that Codex should reference while working.
+
+### assets/
+Files not intended to be loaded into context, but rather used within the output Codex produces.
+
+**Examples from other skills:**
+- Brand styling: PowerPoint template files (.pptx), logo files
+- Frontend builder: HTML/React boilerplate project directories
+- Typography: Font files (.ttf, .woff2)
+
+**Appropriate for:** Templates, boilerplate code, document templates, images, icons, fonts, or any files meant to be copied or used in the final output.
+
+---
+
+**Not every skill requires all three types of resources.**
+"""
+
+EXAMPLE_SCRIPT = '''#!/usr/bin/env python3
+"""
+Example helper script for {skill_name}
+
+This is a placeholder script that can be executed directly.
+Replace with actual implementation or delete if not needed.
+
+Example real scripts from other skills:
+- pdf/scripts/fill_fillable_fields.py - Fills PDF form fields
+- pdf/scripts/convert_pdf_to_images.py - Converts PDF pages to images
+"""
+
+def main():
+    print("This is an example script for {skill_name}")
+    # TODO: Add actual script logic here
+    # This could be data processing, file conversion, API calls, etc.
+
+if __name__ == "__main__":
+    main()
+'''
+
+EXAMPLE_REFERENCE = """# Reference Documentation for {skill_title}
+
+This is a placeholder for detailed reference documentation.
+Replace with actual reference content or delete if not needed.
+
+Example real reference docs from other skills:
+- product-management/references/communication.md - Comprehensive guide for status updates
+- product-management/references/context_building.md - Deep-dive on gathering context
+- bigquery/references/ - API references and query examples
+
+## When Reference Docs Are Useful
+
+Reference docs are ideal for:
+- Comprehensive API documentation
+- Detailed workflow guides
+- Complex multi-step processes
+- Information too lengthy for main SKILL.md
+- Content that's only needed for specific use cases
+
+## Structure Suggestions
+
+### API Reference Example
+- Overview
+- Authentication
+- Endpoints with examples
+- Error codes
+- Rate limits
+
+### Workflow Guide Example
+- Prerequisites
+- Step-by-step instructions
+- Common patterns
+- Troubleshooting
+- Best practices
+"""
+
+EXAMPLE_ASSET = """# Example Asset File
+
+This placeholder represents where asset files would be stored.
+Replace with actual asset files (templates, images, fonts, etc.) or delete if not needed.
+
+Asset files are NOT intended to be loaded into context, but rather used within
+the output Codex produces.
+
+Example asset files from other skills:
+- Brand guidelines: logo.png, slides_template.pptx
+- Frontend builder: hello-world/ directory with HTML/React boilerplate
+- Typography: custom-font.ttf, font-family.woff2
+- Data: sample_data.csv, test_dataset.json
+
+## Common Asset Types
+
+- Templates: .pptx, .docx, boilerplate directories
+- Images: .png, .jpg, .svg, .gif
+- Fonts: .ttf, .otf, .woff, .woff2
+- Boilerplate code: Project directories, starter files
+- Icons: .ico, .svg
+- Data files: .csv, .json, .xml, .yaml
+
+Note: This is a text placeholder. Actual assets can be any file type.
+"""
+
+
+def normalize_skill_name(skill_name):
+    """Normalize a skill name to lowercase hyphen-case."""
+    normalized = skill_name.strip().lower()
+    normalized = re.sub(r"[^a-z0-9]+", "-", normalized)
+    normalized = normalized.strip("-")
+    normalized = re.sub(r"-{2,}", "-", normalized)
+    return normalized
+
+
+def title_case_skill_name(skill_name):
+    """Convert hyphenated skill name to Title Case for display."""
+    return " ".join(word.capitalize() for word in skill_name.split("-"))
+
+
+def parse_resources(raw_resources):
+    if not raw_resources:
+        return []
+    resources = [item.strip() for item in raw_resources.split(",") if item.strip()]
+    invalid = sorted({item for item in resources if item not in ALLOWED_RESOURCES})
+    if invalid:
+        allowed = ", ".join(sorted(ALLOWED_RESOURCES))
+        print(f"[ERROR] Unknown resource type(s): {', '.join(invalid)}")
+        print(f"   Allowed: {allowed}")
+        sys.exit(1)
+    deduped = []
+    seen = set()
+    for resource in resources:
+        if resource not in seen:
+            deduped.append(resource)
+            seen.add(resource)
+    return deduped
+
+
+def create_resource_dirs(skill_dir, skill_name, skill_title, resources, include_examples):
+    for resource in resources:
+        resource_dir = skill_dir / resource
+        resource_dir.mkdir(exist_ok=True)
+        if resource == "scripts":
+            if include_examples:
+                example_script = resource_dir / "example.py"
+                example_script.write_text(EXAMPLE_SCRIPT.format(skill_name=skill_name))
+                example_script.chmod(0o755)
+                print("[OK] Created scripts/example.py")
+            else:
+                print("[OK] Created scripts/")
+        elif resource == "references":
+            if include_examples:
+                example_reference = resource_dir / "api_reference.md"
+                example_reference.write_text(EXAMPLE_REFERENCE.format(skill_title=skill_title))
+                print("[OK] Created references/api_reference.md")
+            else:
+                print("[OK] Created references/")
+        elif resource == "assets":
+            if include_examples:
+                example_asset = resource_dir / "example_asset.txt"
+                example_asset.write_text(EXAMPLE_ASSET)
+                print("[OK] Created assets/example_asset.txt")
+            else:
+                print("[OK] Created assets/")
+
+
+def init_skill(skill_name, path, resources, include_examples, interface_overrides):
+    """
+    Initialize a new skill directory with template SKILL.md.
+
+    Args:
+        skill_name: Name of the skill
+        path: Path where the skill directory should be created
+        resources: Resource directories to create
+        include_examples: Whether to create example files in resource directories
+
+    Returns:
+        Path to created skill directory, or None if error
+    """
+    # Determine skill directory path
+    skill_dir = Path(path).resolve() / skill_name
+
+    # Check if directory already exists
+    if skill_dir.exists():
+        print(f"[ERROR] Skill directory already exists: {skill_dir}")
+        return None
+
+    # Create skill directory
+    try:
+        skill_dir.mkdir(parents=True, exist_ok=False)
+        print(f"[OK] Created skill directory: {skill_dir}")
+    except Exception as e:
+        print(f"[ERROR] Error creating directory: {e}")
+        return None
+
+    # Create SKILL.md from template
+    skill_title = title_case_skill_name(skill_name)
+    skill_content = SKILL_TEMPLATE.format(skill_name=skill_name, skill_title=skill_title)
+
+    skill_md_path = skill_dir / "SKILL.md"
+    try:
+        skill_md_path.write_text(skill_content)
+        print("[OK] Created SKILL.md")
+    except Exception as e:
+        print(f"[ERROR] Error creating SKILL.md: {e}")
+        return None
+
+    # Create agents/openai.yaml
+    try:
+        result = write_openai_yaml(skill_dir, skill_name, interface_overrides)
+        if not result:
+            return None
+    except Exception as e:
+        print(f"[ERROR] Error creating agents/openai.yaml: {e}")
+        return None
+
+    # Create resource directories if requested
+    if resources:
+        try:
+            create_resource_dirs(skill_dir, skill_name, skill_title, resources, include_examples)
+        except Exception as e:
+            print(f"[ERROR] Error creating resource directories: {e}")
+            return None
+
+    # Print next steps
+    print(f"\n[OK] Skill '{skill_name}' initialized successfully at {skill_dir}")
+    print("\nNext steps:")
+    print("1. Edit SKILL.md to complete the TODO items and update the description")
+    if resources:
+        if include_examples:
+            print("2. Customize or delete the example files in scripts/, references/, and assets/")
+        else:
+            print("2. Add resources to scripts/, references/, and assets/ as needed")
+    else:
+        print("2. Create resource directories only if needed (scripts/, references/, assets/)")
+    print("3. Update agents/openai.yaml if the UI metadata should differ")
+    print("4. Run the validator when ready to check the skill structure")
+    print(
+        "5. Forward-test complex skills with realistic user requests to ensure they work as intended"
+    )
+
+    return skill_dir
+
+
+def main():
+    parser = argparse.ArgumentParser(
+        description="Create a new skill directory with a SKILL.md template.",
+    )
+    parser.add_argument("skill_name", help="Skill name (normalized to hyphen-case)")
+    parser.add_argument("--path", required=True, help="Output directory for the skill")
+    parser.add_argument(
+        "--resources",
+        default="",
+        help="Comma-separated list: scripts,references,assets",
+    )
+    parser.add_argument(
+        "--examples",
+        action="store_true",
+        help="Create example files inside the selected resource directories",
+    )
+    parser.add_argument(
+        "--interface",
+        action="append",
+        default=[],
+        help="Interface override in key=value format (repeatable)",
+    )
+    args = parser.parse_args()
+
+    raw_skill_name = args.skill_name
+    skill_name = normalize_skill_name(raw_skill_name)
+    if not skill_name:
+        print("[ERROR] Skill name must include at least one letter or digit.")
+        sys.exit(1)
+    if len(skill_name) > MAX_SKILL_NAME_LENGTH:
+        print(
+            f"[ERROR] Skill name '{skill_name}' is too long ({len(skill_name)} characters). "
+            f"Maximum is {MAX_SKILL_NAME_LENGTH} characters."
+        )
+        sys.exit(1)
+    if skill_name != raw_skill_name:
+        print(f"Note: Normalized skill name from '{raw_skill_name}' to '{skill_name}'.")
+
+    resources = parse_resources(args.resources)
+    if args.examples and not resources:
+        print("[ERROR] --examples requires --resources to be set.")
+        sys.exit(1)
+
+    path = args.path
+
+    print(f"Initializing skill: {skill_name}")
+    print(f"   Location: {path}")
+    if resources:
+        print(f"   Resources: {', '.join(resources)}")
+        if args.examples:
+            print("   Examples: enabled")
+    else:
+        print("   Resources: none (create as needed)")
+    print()
+
+    result = init_skill(skill_name, path, resources, args.examples, args.interface)
+
+    if result:
+        sys.exit(0)
+    else:
+        sys.exit(1)
+
+
+if __name__ == "__main__":
+    main()

.agents/skills/skill-creator/scripts/quick_validate.py

@@ -0,0 +1,101 @@
+#!/usr/bin/env python3
+"""
+Quick validation script for skills - minimal version
+"""
+
+import re
+import sys
+from pathlib import Path
+
+import yaml
+
+MAX_SKILL_NAME_LENGTH = 64
+
+
+def validate_skill(skill_path):
+    """Basic validation of a skill"""
+    skill_path = Path(skill_path)
+
+    skill_md = skill_path / "SKILL.md"
+    if not skill_md.exists():
+        return False, "SKILL.md not found"
+
+    content = skill_md.read_text()
+    if not content.startswith("---"):
+        return False, "No YAML frontmatter found"
+
+    match = re.match(r"^---\n(.*?)\n---", content, re.DOTALL)
+    if not match:
+        return False, "Invalid frontmatter format"
+
+    frontmatter_text = match.group(1)
+
+    try:
+        frontmatter = yaml.safe_load(frontmatter_text)
+        if not isinstance(frontmatter, dict):
+            return False, "Frontmatter must be a YAML dictionary"
+    except yaml.YAMLError as e:
+        return False, f"Invalid YAML in frontmatter: {e}"
+
+    allowed_properties = {"name", "description", "license", "allowed-tools", "metadata"}
+
+    unexpected_keys = set(frontmatter.keys()) - allowed_properties
+    if unexpected_keys:
+        allowed = ", ".join(sorted(allowed_properties))
+        unexpected = ", ".join(sorted(unexpected_keys))
+        return (
+            False,
+            f"Unexpected key(s) in SKILL.md frontmatter: {unexpected}. Allowed properties are: {allowed}",
+        )
+
+    if "name" not in frontmatter:
+        return False, "Missing 'name' in frontmatter"
+    if "description" not in frontmatter:
+        return False, "Missing 'description' in frontmatter"
+
+    name = frontmatter.get("name", "")
+    if not isinstance(name, str):
+        return False, f"Name must be a string, got {type(name).__name__}"
+    name = name.strip()
+    if name:
+        if not re.match(r"^[a-z0-9-]+$", name):
+            return (
+                False,
+                f"Name '{name}' should be hyphen-case (lowercase letters, digits, and hyphens only)",
+            )
+        if name.startswith("-") or name.endswith("-") or "--" in name:
+            return (
+                False,
+                f"Name '{name}' cannot start/end with hyphen or contain consecutive hyphens",
+            )
+        if len(name) > MAX_SKILL_NAME_LENGTH:
+            return (
+                False,
+                f"Name is too long ({len(name)} characters). "
+                f"Maximum is {MAX_SKILL_NAME_LENGTH} characters.",
+            )
+
+    description = frontmatter.get("description", "")
+    if not isinstance(description, str):
+        return False, f"Description must be a string, got {type(description).__name__}"
+    description = description.strip()
+    if description:
+        if "<" in description or ">" in description:
+            return False, "Description cannot contain angle brackets (< or >)"
+        if len(description) > 1024:
+            return (
+                False,
+                f"Description is too long ({len(description)} characters). Maximum is 1024 characters.",
+            )
+
+    return True, "Skill is valid!"
+
+
+if __name__ == "__main__":
+    if len(sys.argv) != 2:
+        print("Usage: python quick_validate.py <skill_directory>")
+        sys.exit(1)
+
+    valid, message = validate_skill(sys.argv[1])
+    print(message)
+    sys.exit(0 if valid else 1)

.agents/skills/storybook/SKILL.md

@@ -0,0 +1,61 @@
+---
+name: storybook
+description: Use when writing or reviewing Storybook stories (`.stories.tsx`) for React components.
+---
+
+# Storybook Component Stories
+
+## Which Components Can Have Stories?
+
+Only create stories for components that **do not** do any of the following:
+
+- Depend on context.
+- Fetch data via any private API, including Langfuse’s own tRPC API.
+
+Stories should follow the `ComponentName.stories.tsx` filename pattern. Components covered by stories should have exactly one exported or public component.
+
+If this is not the case, suggest splitting up the file that includes the component to be covered first.
+
+Be mindful of the breadth a story covers. A story should show a component in isolation. Page-level compositions should be rare and intentional.
+
+## What to Do If a Component Violates the Criteria
+
+Suggest abstracting a presentational component that does not violate the criteria and receives relevant data via props.
+
+Make sure the props are well-defined using TypeScript.
+
+Keep the existing component, but update it to use the newly created component for rendering. These presentational components are easier to test and easier to reuse.
+
+## How Stories Should Be Written
+
+- Use "CSF Next" format by default.
+- Cover only the relevant component by default.
+- Avoid custom render functions by default.
+- Use `satisfies` and typed Storybook metadata so invalid args, decorators, and play functions are type-checked.
+- Use play functions to test user-relevant interactions after render, not to compensate for complex setup or hidden dependencies.
+- Name stories after the state they represent, not the implementation. Also do not include the component name in the story name.
+  - Prefer: `Default`, `Empty`, `WithLongName`, `Error`, `Disabled`, `Loading`
+  - Avoid: `Test1`, `CustomRenderExample`, `ButtonWithLongNameAndIcon`
+- Set callbacks up as Storybook Actions by default:
+
+  ```ts
+  import { fn } from "storybook/test";
+  ```
+
+- Avoid large fixtures.
+- Use the smallest meaningful data shape needed to render the state.
+- If fixtures are required and may be shared, check whether a reusable helper function exists. Otherwise, create one for defining the fixture.
+
+## Variant and Design Showcase Stories
+
+If a component has many variants, and the point of the story is to showcase the design of a component rather than its functionality, stories may render the component multiple times.
+
+For example, a `Button` with a `size: "sm" | "md" | "lg"` prop may have a story that shows three buttons side by side.
+
+If the button also has a `variant: "primary" | "secondary"` prop, consider using a matrix-like UI that showcases all possible combinations.
+
+These compositional stories **should not** contain Storybook play functions. They should also not allow the Storybook user to customize the predefined args, such as `size` and `variant`, via Storybook args. Having an arg for non-bound props, such as `text`, may be acceptable.
+
+## Additional Information
+
+- We do not use MSW and are not planning to add it.

.agents/skills/turborepo/SKILL.md

@@ -0,0 +1,951 @@
+---
+name: turborepo
+description: |
+  Turborepo monorepo build system guidance. Triggers on: turbo.json, task pipelines,
+  dependsOn, caching, remote cache, the "turbo" CLI, --filter, --affected, CI optimization, environment
+  variables, internal packages, monorepo structure/best practices, and boundaries.
+
+  Use when user: configures tasks/workflows/pipelines, creates packages, sets up
+  monorepo, shares code between apps, runs changed/affected packages, debugs cache,
+  or has apps/packages directories.
+metadata:
+  version: 2.8.21-canary.9
+---
+
+# Turborepo Skill
+
+Build system for JavaScript/TypeScript monorepos. Turborepo caches task outputs and runs tasks in parallel based on dependency graph.
+
+## IMPORTANT: Package Tasks, Not Root Tasks
+
+**DO NOT create Root Tasks. ALWAYS create package tasks.**
+
+When creating tasks/scripts/pipelines, you MUST:
+
+1. Add the script to each relevant package's `package.json`
+2. Register the task in root `turbo.json`
+3. Root `package.json` only delegates via `turbo run <task>`
+
+**DO NOT** put task logic in root `package.json`. This defeats Turborepo's parallelization.
+
+```json
+// DO THIS: Scripts in each package
+// apps/web/package.json
+{ "scripts": { "build": "next build", "lint": "eslint .", "test": "vitest" } }
+
+// apps/api/package.json
+{ "scripts": { "build": "tsc", "lint": "eslint .", "test": "vitest" } }
+
+// packages/ui/package.json
+{ "scripts": { "build": "tsc", "lint": "eslint .", "test": "vitest" } }
+```
+
+```json
+// turbo.json - register tasks
+{
+  "tasks": {
+    "build": { "dependsOn": ["^build"], "outputs": ["dist/**"] },
+    "lint": {},
+    "test": { "dependsOn": ["build"] }
+  }
+}
+```
+
+```json
+// Root package.json - ONLY delegates, no task logic
+{
+  "scripts": {
+    "build": "turbo run build",
+    "lint": "turbo run lint",
+    "test": "turbo run test"
+  }
+}
+```
+
+```json
+// DO NOT DO THIS - defeats parallelization
+// Root package.json
+{
+  "scripts": {
+    "build": "cd apps/web && next build && cd ../api && tsc",
+    "lint": "eslint apps/ packages/",
+    "test": "vitest"
+  }
+}
+```
+
+Root Tasks (`//#taskname`) are ONLY for tasks that truly cannot exist in packages (rare).
+
+## Secondary Rule: `turbo run` vs `turbo`
+
+**Always use `turbo run` when the command is written into code:**
+
+```json
+// package.json - ALWAYS "turbo run"
+{
+  "scripts": {
+    "build": "turbo run build"
+  }
+}
+```
+
+```yaml
+# CI workflows - ALWAYS "turbo run"
+- run: turbo run build --affected
+```
+
+**The shorthand `turbo <tasks>` is ONLY for one-off terminal commands** typed directly by humans or agents. Never write `turbo build` into package.json, CI, or scripts.
+
+## Quick Decision Trees
+
+### "I need to configure a task"
+
+```
+Configure a task?
+├─ Define task dependencies → references/configuration/tasks.md
+├─ Lint/check-types (parallel + caching) → Use Transit Nodes pattern (see below)
+├─ Specify build outputs → references/configuration/tasks.md#outputs
+├─ Handle environment variables → references/environment/RULE.md
+├─ Set up dev/watch tasks → references/configuration/tasks.md#persistent
+├─ Package-specific config → references/configuration/RULE.md#package-configurations
+└─ Global settings (cacheDir, daemon) → references/configuration/global-options.md
+```
+
+### "My cache isn't working"
+
+```
+Cache problems?
+├─ Tasks run but outputs not restored → Missing `outputs` key
+├─ Cache misses unexpectedly → references/caching/gotchas.md
+├─ Need to debug hash inputs → Use --summarize or --dry
+├─ Want to skip cache entirely → Use --force or cache: false
+├─ Remote cache not working → references/caching/remote-cache.md
+└─ Environment causing misses → references/environment/gotchas.md
+```
+
+### "I want to run only changed packages"
+
+```
+Run only what changed?
+├─ Changed packages + dependents (RECOMMENDED) → turbo run build --affected
+├─ Custom base branch → --affected --affected-base=origin/develop
+├─ Manual git comparison → --filter=...[origin/main]
+└─ See all filter options → references/filtering/RULE.md
+```
+
+**`--affected` is the primary way to run only changed packages.** It automatically compares against the default branch and includes dependents.
+
+### "I want to filter packages"
+
+```
+Filter packages?
+├─ Only changed packages → --affected (see above)
+├─ By package name → --filter=web
+├─ By directory → --filter=./apps/*
+├─ Package + dependencies → --filter=web...
+├─ Package + dependents → --filter=...web
+└─ Complex combinations → references/filtering/patterns.md
+```
+
+### "Environment variables aren't working"
+
+```
+Environment issues?
+├─ Vars not available at runtime → Strict mode filtering (default)
+├─ Cache hits with wrong env → Var not in `env` key
+├─ .env changes not causing rebuilds → .env not in `inputs`
+├─ CI variables missing → references/environment/gotchas.md
+└─ Framework vars (NEXT_PUBLIC_*) → Auto-included via inference
+```
+
+### "I need to set up CI"
+
+```
+CI setup?
+├─ GitHub Actions → references/ci/github-actions.md
+├─ Vercel deployment → references/ci/vercel.md
+├─ Remote cache in CI → references/caching/remote-cache.md
+├─ Only build changed packages → --affected flag
+├─ Skip unnecessary builds → turbo-ignore (references/cli/commands.md)
+└─ Skip container setup when no changes → turbo-ignore
+```
+
+### "I want to watch for changes during development"
+
+```
+Watch mode?
+├─ Re-run tasks on change → turbo watch (references/watch/RULE.md)
+├─ Dev servers with dependencies → Use `with` key (references/configuration/tasks.md#with)
+├─ Restart dev server on dep change → Use `interruptible: true`
+└─ Persistent dev tasks → Use `persistent: true`
+```
+
+### "I need to create/structure a package"
+
+```
+Package creation/structure?
+├─ Create an internal package → references/best-practices/packages.md
+├─ Repository structure → references/best-practices/structure.md
+├─ Dependency management → references/best-practices/dependencies.md
+├─ Best practices overview → references/best-practices/RULE.md
+├─ JIT vs Compiled packages → references/best-practices/packages.md#compilation-strategies
+└─ Sharing code between apps → references/best-practices/RULE.md#package-types
+```
+
+### "How should I structure my monorepo?"
+
+```
+Monorepo structure?
+├─ Standard layout (apps/, packages/) → references/best-practices/RULE.md
+├─ Package types (apps vs libraries) → references/best-practices/RULE.md#package-types
+├─ Creating internal packages → references/best-practices/packages.md
+├─ TypeScript configuration → references/best-practices/structure.md#typescript-configuration
+├─ ESLint configuration → references/best-practices/structure.md#eslint-configuration
+├─ Dependency management → references/best-practices/dependencies.md
+└─ Enforce package boundaries → references/boundaries/RULE.md
+```
+
+### "I want to enforce architectural boundaries"
+
+```
+Enforce boundaries?
+├─ Check for violations → turbo boundaries
+├─ Tag packages → references/boundaries/RULE.md#tags
+├─ Restrict which packages can import others → references/boundaries/RULE.md#rule-types
+└─ Prevent cross-package file imports → references/boundaries/RULE.md
+```
+
+## Critical Anti-Patterns
+
+### Using `turbo` Shorthand in Code
+
+**`turbo run` is recommended in package.json scripts and CI pipelines.** The shorthand `turbo <task>` is intended for interactive terminal use.
+
+```json
+// WRONG - using shorthand in package.json
+{
+  "scripts": {
+    "build": "turbo build",
+    "dev": "turbo dev"
+  }
+}
+
+// CORRECT
+{
+  "scripts": {
+    "build": "turbo run build",
+    "dev": "turbo run dev"
+  }
+}
+```
+
+```yaml
+# WRONG - using shorthand in CI
+- run: turbo build --affected
+
+# CORRECT
+- run: turbo run build --affected
+```
+
+### Root Scripts Bypassing Turbo
+
+Root `package.json` scripts MUST delegate to `turbo run`, not run tasks directly.
+
+```json
+// WRONG - bypasses turbo entirely
+{
+  "scripts": {
+    "build": "bun build",
+    "dev": "bun dev"
+  }
+}
+
+// CORRECT - delegates to turbo
+{
+  "scripts": {
+    "build": "turbo run build",
+    "dev": "turbo run dev"
+  }
+}
+```
+
+### Using `&&` to Chain Turbo Tasks
+
+Don't chain turbo tasks with `&&`. Let turbo orchestrate.
+
+```json
+// WRONG - turbo task not using turbo run
+{
+  "scripts": {
+    "changeset:publish": "bun build && changeset publish"
+  }
+}
+
+// CORRECT
+{
+  "scripts": {
+    "changeset:publish": "turbo run build && changeset publish"
+  }
+}
+```
+
+### `prebuild` Scripts That Manually Build Dependencies
+
+Scripts like `prebuild` that manually build other packages bypass Turborepo's dependency graph.
+
+```json
+// WRONG - manually building dependencies
+{
+  "scripts": {
+    "prebuild": "cd ../../packages/types && bun run build && cd ../utils && bun run build",
+    "build": "next build"
+  }
+}
+```
+
+**However, the fix depends on whether workspace dependencies are declared:**
+
+1. **If dependencies ARE declared** (e.g., `"@repo/types": "workspace:*"` in package.json), remove the `prebuild` script. Turbo's `dependsOn: ["^build"]` handles this automatically.
+
+2. **If dependencies are NOT declared**, the `prebuild` exists because `^build` won't trigger without a dependency relationship. The fix is to:
+   - Add the dependency to package.json: `"@repo/types": "workspace:*"`
+   - Then remove the `prebuild` script
+
+```json
+// CORRECT - declare dependency, let turbo handle build order
+// package.json
+{
+  "dependencies": {
+    "@repo/types": "workspace:*",
+    "@repo/utils": "workspace:*"
+  },
+  "scripts": {
+    "build": "next build"
+  }
+}
+
+// turbo.json
+{
+  "tasks": {
+    "build": {
+      "dependsOn": ["^build"]
+    }
+  }
+}
+```
+
+**Key insight:** `^build` only runs build in packages listed as dependencies. No dependency declaration = no automatic build ordering.
+
+### Overly Broad `globalDependencies`
+
+`globalDependencies` affects ALL tasks in ALL packages via the **global hash** — tasks cannot opt out of specific files, even with negation globs in `inputs`. Be specific.
+
+```json
+// WRONG - heavy hammer, affects all hashes
+{
+  "globalDependencies": ["**/.env.*local"]
+}
+
+// BETTER - move to task-level inputs
+{
+  "globalDependencies": [".env"],
+  "tasks": {
+    "build": {
+      "inputs": ["$TURBO_DEFAULT$", ".env*"],
+      "outputs": ["dist/**"]
+    }
+  }
+}
+```
+
+With `futureFlags.globalConfiguration`, this problem is reduced because `global.inputs` files are folded into each task's inputs (not the global hash). Tasks can exclude specific files:
+
+```json
+// BEST - global.inputs with per-task exclusion
+{
+  "futureFlags": { "globalConfiguration": true },
+  "global": {
+    "inputs": [".env"]
+  },
+  "tasks": {
+    "build": { "outputs": ["dist/**"] },
+    "lint": {
+      "inputs": ["$TURBO_DEFAULT$", "!$TURBO_ROOT$/.env"]
+    }
+  }
+}
+```
+
+### Repetitive Task Configuration
+
+Look for repeated configuration across tasks that can be collapsed. Turborepo supports shared configuration patterns.
+
+```json
+// WRONG - repetitive env and inputs across tasks
+{
+  "tasks": {
+    "build": {
+      "env": ["API_URL", "DATABASE_URL"],
+      "inputs": ["$TURBO_DEFAULT$", ".env*"]
+    },
+    "test": {
+      "env": ["API_URL", "DATABASE_URL"],
+      "inputs": ["$TURBO_DEFAULT$", ".env*"]
+    },
+    "dev": {
+      "env": ["API_URL", "DATABASE_URL"],
+      "inputs": ["$TURBO_DEFAULT$", ".env*"],
+      "cache": false,
+      "persistent": true
+    }
+  }
+}
+
+// BETTER - use globalEnv and globalDependencies for shared config
+{
+  "globalEnv": ["API_URL", "DATABASE_URL"],
+  "globalDependencies": [".env*"],
+  "tasks": {
+    "build": {},
+    "test": {},
+    "dev": {
+      "cache": false,
+      "persistent": true
+    }
+  }
+}
+```
+
+**When to use global vs task-level:**
+
+- `globalEnv` / `globalDependencies` - affects ALL tasks, use for truly shared config
+- Task-level `env` / `inputs` - use when only specific tasks need it
+
+### NOT an Anti-Pattern: Large `env` Arrays
+
+A large `env` array (even 50+ variables) is **not** a problem. It usually means the user was thorough about declaring their build's environment dependencies. Do not flag this as an issue.
+
+### Using `--parallel` Flag
+
+The `--parallel` flag bypasses Turborepo's dependency graph. If tasks need parallel execution, configure `dependsOn` correctly instead.
+
+```bash
+# WRONG - bypasses dependency graph
+turbo run lint --parallel
+
+# CORRECT - configure tasks to allow parallel execution
+# In turbo.json, set dependsOn appropriately (or use transit nodes)
+turbo run lint
+```
+
+### Package-Specific Task Overrides in Root turbo.json
+
+When multiple packages need different task configurations, use **Package Configurations** (`turbo.json` in each package) instead of cluttering root `turbo.json` with `package#task` overrides.
+
+```json
+// WRONG - root turbo.json with many package-specific overrides
+{
+  "tasks": {
+    "test": { "dependsOn": ["build"] },
+    "@repo/web#test": { "outputs": ["coverage/**"] },
+    "@repo/api#test": { "outputs": ["coverage/**"] },
+    "@repo/utils#test": { "outputs": [] },
+    "@repo/cli#test": { "outputs": [] },
+    "@repo/core#test": { "outputs": [] }
+  }
+}
+
+// CORRECT - use Package Configurations
+// Root turbo.json - base config only
+{
+  "tasks": {
+    "test": { "dependsOn": ["build"] }
+  }
+}
+
+// packages/web/turbo.json - package-specific override
+{
+  "extends": ["//"],
+  "tasks": {
+    "test": { "outputs": ["coverage/**"] }
+  }
+}
+
+// packages/api/turbo.json
+{
+  "extends": ["//"],
+  "tasks": {
+    "test": { "outputs": ["coverage/**"] }
+  }
+}
+```
+
+**Benefits of Package Configurations:**
+
+- Keeps configuration close to the code it affects
+- Root turbo.json stays clean and focused on base patterns
+- Easier to understand what's special about each package
+- Works with `$TURBO_EXTENDS$` to inherit + extend arrays
+
+**When to use `package#task` in root:**
+
+- Single package needs a unique dependency (e.g., `"deploy": { "dependsOn": ["web#build"] }`)
+- Temporary override while migrating
+
+See `references/configuration/RULE.md#package-configurations` for full details.
+
+### Using `../` to Traverse Out of Package in `inputs`
+
+Don't use relative paths like `../` to reference files outside the package. Use `$TURBO_ROOT$` instead.
+
+```json
+// WRONG - traversing out of package
+{
+  "tasks": {
+    "build": {
+      "inputs": ["$TURBO_DEFAULT$", "../shared-config.json"]
+    }
+  }
+}
+
+// CORRECT - use $TURBO_ROOT$ for repo root
+{
+  "tasks": {
+    "build": {
+      "inputs": ["$TURBO_DEFAULT$", "$TURBO_ROOT$/shared-config.json"]
+    }
+  }
+}
+```
+
+### Missing `outputs` for File-Producing Tasks
+
+**Before flagging missing `outputs`, check what the task actually produces:**
+
+1. Read the package's script (e.g., `"build": "tsc"`, `"test": "vitest"`)
+2. Determine if it writes files to disk or only outputs to stdout
+3. Only flag if the task produces files that should be cached
+
+```json
+// WRONG: build produces files but they're not cached
+{
+  "tasks": {
+    "build": {
+      "dependsOn": ["^build"]
+    }
+  }
+}
+
+// CORRECT: build outputs are cached
+{
+  "tasks": {
+    "build": {
+      "dependsOn": ["^build"],
+      "outputs": ["dist/**"]
+    }
+  }
+}
+```
+
+Common outputs by framework:
+
+- Next.js: `[".next/**", "!.next/cache/**"]`
+- Vite/Rollup: `["dist/**"]`
+- tsc: `["dist/**"]` or custom `outDir`
+
+**TypeScript `--noEmit` can still produce cache files:**
+
+When `incremental: true` in tsconfig.json, `tsc --noEmit` writes `.tsbuildinfo` files even without emitting JS. Check the tsconfig before assuming no outputs:
+
+```json
+// If tsconfig has incremental: true, tsc --noEmit produces cache files
+{
+  "tasks": {
+    "typecheck": {
+      "outputs": ["node_modules/.cache/tsbuildinfo.json"] // or wherever tsBuildInfoFile points
+    }
+  }
+}
+```
+
+To determine correct outputs for TypeScript tasks:
+
+1. Check if `incremental` or `composite` is enabled in tsconfig
+2. Check `tsBuildInfoFile` for custom cache location (default: alongside `outDir` or in project root)
+3. If no incremental mode, `tsc --noEmit` produces no files
+
+### `^build` vs `build` Confusion
+
+```json
+{
+  "tasks": {
+    // ^build = run build in DEPENDENCIES first (other packages this one imports)
+    "build": {
+      "dependsOn": ["^build"]
+    },
+    // build (no ^) = run build in SAME PACKAGE first
+    "test": {
+      "dependsOn": ["build"]
+    },
+    // pkg#task = specific package's task
+    "deploy": {
+      "dependsOn": ["web#build"]
+    }
+  }
+}
+```
+
+### Environment Variables Not Hashed
+
+```json
+// WRONG: API_URL changes won't cause rebuilds
+{
+  "tasks": {
+    "build": {
+      "outputs": ["dist/**"]
+    }
+  }
+}
+
+// CORRECT: API_URL changes invalidate cache
+{
+  "tasks": {
+    "build": {
+      "outputs": ["dist/**"],
+      "env": ["API_URL", "API_KEY"]
+    }
+  }
+}
+```
+
+### `.env` Files Not in Inputs
+
+Turbo does NOT load `.env` files - your framework does. But Turbo needs to know about changes:
+
+```json
+// WRONG: .env changes don't invalidate cache
+{
+  "tasks": {
+    "build": {
+      "env": ["API_URL"]
+    }
+  }
+}
+
+// CORRECT: .env file changes invalidate cache
+{
+  "tasks": {
+    "build": {
+      "env": ["API_URL"],
+      "inputs": ["$TURBO_DEFAULT$", ".env", ".env.*"]
+    }
+  }
+}
+```
+
+### Root `.env` File in Monorepo
+
+A `.env` file at the repo root is an anti-pattern — even for small monorepos or starter templates. It creates implicit coupling between packages and makes it unclear which packages depend on which variables.
+
+```
+// WRONG - root .env affects all packages implicitly
+my-monorepo/
+├── .env              # Which packages use this?
+├── apps/
+│   ├── web/
+│   └── api/
+└── packages/
+
+// CORRECT - .env files in packages that need them
+my-monorepo/
+├── apps/
+│   ├── web/
+│   │   └── .env      # Clear: web needs DATABASE_URL
+│   └── api/
+│       └── .env      # Clear: api needs API_KEY
+└── packages/
+```
+
+**Problems with root `.env`:**
+
+- Unclear which packages consume which variables
+- All packages get all variables (even ones they don't need)
+- Cache invalidation is coarse-grained (root .env change invalidates everything)
+- Security risk: packages may accidentally access sensitive vars meant for others
+- Bad habits start small — starter templates should model correct patterns
+
+**If you must share variables**, use `globalEnv` to be explicit about what's shared, and document why.
+
+### Strict Mode Filtering CI Variables
+
+By default, Turborepo filters environment variables to only those in `env`/`globalEnv`. CI variables may be missing:
+
+```json
+// If CI scripts need GITHUB_TOKEN but it's not in env:
+{
+  "globalPassThroughEnv": ["GITHUB_TOKEN", "CI"],
+  "tasks": { ... }
+}
+```
+
+Or use `--env-mode=loose` (not recommended for production).
+
+### Shared Code in Apps (Should Be a Package)
+
+```
+// WRONG: Shared code inside an app
+apps/
+  web/
+    shared/          # This breaks monorepo principles!
+      utils.ts
+
+// CORRECT: Extract to a package
+packages/
+  utils/
+    src/utils.ts
+```
+
+### Accessing Files Across Package Boundaries
+
+```typescript
+// WRONG: Reaching into another package's internals
+import { Button } from "../../packages/ui/src/button";
+
+// CORRECT: Install and import properly
+import { Button } from "@repo/ui/button";
+```
+
+### Too Many Root Dependencies
+
+```json
+// WRONG: App dependencies in root
+{
+  "dependencies": {
+    "react": "^18",
+    "next": "^14"
+  }
+}
+
+// CORRECT: Only repo tools in root
+{
+  "devDependencies": {
+    "turbo": "latest"
+  }
+}
+```
+
+## Common Task Configurations
+
+### Standard Build Pipeline
+
+```json
+{
+  "$schema": "https://v2-8-21-canary-9.turborepo.dev/schema.json",
+  "tasks": {
+    "build": {
+      "dependsOn": ["^build"],
+      "outputs": ["dist/**", ".next/**", "!.next/cache/**"]
+    },
+    "dev": {
+      "cache": false,
+      "persistent": true
+    }
+  }
+}
+```
+
+Add a `transit` task if you have tasks that need parallel execution with cache invalidation (see below).
+
+### Dev Task with `^dev` Pattern (for `turbo watch`)
+
+A `dev` task with `dependsOn: ["^dev"]` and `persistent: false` in root turbo.json may look unusual but is **correct for `turbo watch` workflows**:
+
+```json
+// Root turbo.json
+{
+  "tasks": {
+    "dev": {
+      "dependsOn": ["^dev"],
+      "cache": false,
+      "persistent": false  // Packages have one-shot dev scripts
+    }
+  }
+}
+
+// Package turbo.json (apps/web/turbo.json)
+{
+  "extends": ["//"],
+  "tasks": {
+    "dev": {
+      "persistent": true  // Apps run long-running dev servers
+    }
+  }
+}
+```
+
+**Why this works:**
+
+- **Packages** (e.g., `@acme/db`, `@acme/validators`) have `"dev": "tsc"` — one-shot type generation that completes quickly
+- **Apps** override with `persistent: true` for actual dev servers (Next.js, etc.)
+- **`turbo watch`** re-runs the one-shot package `dev` scripts when source files change, keeping types in sync
+
+**Intended usage:** Run `turbo watch dev` (not `turbo run dev`). Watch mode re-executes one-shot tasks on file changes while keeping persistent tasks running.
+
+**Alternative pattern:** Use a separate task name like `prepare` or `generate` for one-shot dependency builds to make the intent clearer:
+
+```json
+{
+  "tasks": {
+    "prepare": {
+      "dependsOn": ["^prepare"],
+      "outputs": ["dist/**"]
+    },
+    "dev": {
+      "dependsOn": ["prepare"],
+      "cache": false,
+      "persistent": true
+    }
+  }
+}
+```
+
+### Transit Nodes for Parallel Tasks with Cache Invalidation
+
+Some tasks can run in parallel (don't need built output from dependencies) but must invalidate cache when dependency source code changes.
+
+**The problem with `dependsOn: ["^taskname"]`:**
+
+- Forces sequential execution (slow)
+
+**The problem with `dependsOn: []` (no dependencies):**
+
+- Allows parallel execution (fast)
+- But cache is INCORRECT - changing dependency source won't invalidate cache
+
+**Transit Nodes solve both:**
+
+```json
+{
+  "tasks": {
+    "transit": { "dependsOn": ["^transit"] },
+    "my-task": { "dependsOn": ["transit"] }
+  }
+}
+```
+
+The `transit` task creates dependency relationships without matching any actual script, so tasks run in parallel with correct cache invalidation.
+
+**How to identify tasks that need this pattern:** Look for tasks that read source files from dependencies but don't need their build outputs.
+
+### With Environment Variables
+
+```json
+{
+  "globalEnv": ["NODE_ENV"],
+  "globalDependencies": [".env"],
+  "tasks": {
+    "build": {
+      "dependsOn": ["^build"],
+      "outputs": ["dist/**"],
+      "env": ["API_URL", "DATABASE_URL"]
+    }
+  }
+}
+```
+
+With `futureFlags.globalConfiguration`, the same config moves global settings under `global` — and `.env` becomes a per-task input instead of a global hash input:
+
+```json
+{
+  "futureFlags": { "globalConfiguration": true },
+  "global": {
+    "env": ["NODE_ENV"],
+    "inputs": [".env"]
+  },
+  "tasks": {
+    "build": {
+      "dependsOn": ["^build"],
+      "outputs": ["dist/**"],
+      "env": ["API_URL", "DATABASE_URL"]
+    }
+  }
+}
+```
+
+## Reference Index
+
+### Configuration
+
+| File                                                                            | Purpose                                                                   |
+| ------------------------------------------------------------------------------- | ------------------------------------------------------------------------- |
+| [configuration/RULE.md](./references/configuration/RULE.md)                     | turbo.json overview, Package Configurations                               |
+| [configuration/tasks.md](./references/configuration/tasks.md)                   | dependsOn, outputs, inputs, env, cache, persistent                        |
+| [configuration/global-options.md](./references/configuration/global-options.md) | globalEnv, globalDependencies, global key, futureFlags, cacheDir, envMode |
+| [configuration/gotchas.md](./references/configuration/gotchas.md)               | Common configuration mistakes                                             |
+
+### Caching
+
+| File                                                            | Purpose                                      |
+| --------------------------------------------------------------- | -------------------------------------------- |
+| [caching/RULE.md](./references/caching/RULE.md)                 | How caching works, hash inputs               |
+| [caching/remote-cache.md](./references/caching/remote-cache.md) | Vercel Remote Cache, self-hosted, login/link |
+| [caching/gotchas.md](./references/caching/gotchas.md)           | Debugging cache misses, --summarize, --dry   |
+
+### Environment Variables
+
+| File                                                          | Purpose                                   |
+| ------------------------------------------------------------- | ----------------------------------------- |
+| [environment/RULE.md](./references/environment/RULE.md)       | env, globalEnv, passThroughEnv            |
+| [environment/modes.md](./references/environment/modes.md)     | Strict vs Loose mode, framework inference |
+| [environment/gotchas.md](./references/environment/gotchas.md) | .env files, CI issues                     |
+
+### Filtering
+
+| File                                                        | Purpose                  |
+| ----------------------------------------------------------- | ------------------------ |
+| [filtering/RULE.md](./references/filtering/RULE.md)         | --filter syntax overview |
+| [filtering/patterns.md](./references/filtering/patterns.md) | Common filter patterns   |
+
+### CI/CD
+
+| File                                                      | Purpose                         |
+| --------------------------------------------------------- | ------------------------------- |
+| [ci/RULE.md](./references/ci/RULE.md)                     | General CI principles           |
+| [ci/github-actions.md](./references/ci/github-actions.md) | Complete GitHub Actions setup   |
+| [ci/vercel.md](./references/ci/vercel.md)                 | Vercel deployment, turbo-ignore |
+| [ci/patterns.md](./references/ci/patterns.md)             | --affected, caching strategies  |
+
+### CLI
+
+| File                                            | Purpose                                       |
+| ----------------------------------------------- | --------------------------------------------- |
+| [cli/RULE.md](./references/cli/RULE.md)         | turbo run basics                              |
+| [cli/commands.md](./references/cli/commands.md) | turbo run flags, turbo-ignore, other commands |
+
+### Best Practices
+
+| File                                                                          | Purpose                                                         |
+| ----------------------------------------------------------------------------- | --------------------------------------------------------------- |
+| [best-practices/RULE.md](./references/best-practices/RULE.md)                 | Monorepo best practices overview                                |
+| [best-practices/structure.md](./references/best-practices/structure.md)       | Repository structure, workspace config, TypeScript/ESLint setup |
+| [best-practices/packages.md](./references/best-practices/packages.md)         | Creating internal packages, JIT vs Compiled, exports            |
+| [best-practices/dependencies.md](./references/best-practices/dependencies.md) | Dependency management, installing, version sync                 |
+
+### Watch Mode
+
+| File                                        | Purpose                                         |
+| ------------------------------------------- | ----------------------------------------------- |
+| [watch/RULE.md](./references/watch/RULE.md) | turbo watch, interruptible tasks, dev workflows |
+
+### Boundaries (Experimental)
+
+| File                                                  | Purpose                                               |
+| ----------------------------------------------------- | ----------------------------------------------------- |
+| [boundaries/RULE.md](./references/boundaries/RULE.md) | Enforce package isolation, tag-based dependency rules |
+
+## Source Documentation
+
+This skill is based on the official Turborepo documentation at:
+
+- Source: `apps/docs/content/docs/` in the Turborepo repository
+- Live: https://turborepo.dev/docs

.agents/skills/turborepo/command/turborepo.md

@@ -0,0 +1,70 @@
+---
+description: Load Turborepo skill for creating workflows, tasks, and pipelines in monorepos. Use when users ask to "create a workflow", "make a task", "generate a pipeline", or set up build orchestration.
+---
+
+Load the Turborepo skill and help with monorepo task orchestration: creating workflows, configuring tasks, setting up pipelines, and optimizing builds.
+
+## Workflow
+
+### Step 1: Load turborepo skill
+
+```
+skill({ name: 'turborepo' })
+```
+
+### Step 2: Identify task type from user request
+
+Analyze $ARGUMENTS to determine:
+
+- **Topic**: configuration, caching, filtering, environment, CI, or CLI
+- **Task type**: new setup, debugging, optimization, or implementation
+
+Use decision trees in SKILL.md to select the relevant reference files.
+
+### Step 3: Read relevant reference files
+
+Based on task type, read from `references/<topic>/`:
+
+| Task                 | Files to Read                                           |
+| -------------------- | ------------------------------------------------------- |
+| Configure turbo.json | `configuration/RULE.md` + `configuration/tasks.md`      |
+| Debug cache issues   | `caching/gotchas.md`                                    |
+| Set up remote cache  | `caching/remote-cache.md`                               |
+| Filter packages      | `filtering/RULE.md` + `filtering/patterns.md`           |
+| Environment problems | `environment/gotchas.md` + `environment/modes.md`       |
+| Set up CI            | `ci/RULE.md` + `ci/github-actions.md` or `ci/vercel.md` |
+| CLI usage            | `cli/commands.md`                                       |
+
+### Step 4: Execute task
+
+Apply Turborepo-specific patterns from references to complete the user's request.
+
+**CRITICAL - When creating tasks/scripts/pipelines:**
+
+1. **DO NOT create Root Tasks** - Always create package tasks
+2. Add scripts to each relevant package's `package.json` (e.g., `apps/web/package.json`, `packages/ui/package.json`)
+3. Register the task in root `turbo.json`
+4. Root `package.json` only contains `turbo run <task>` - never actual task logic
+
+**Other things to verify:**
+
+- `outputs` defined for cacheable tasks
+- `dependsOn` uses correct syntax (`^task` vs `task`)
+- Environment variables in `env` key
+- `.env` files in `inputs` if used
+- Use `turbo run` (not `turbo`) in package.json and CI
+
+### Step 5: Summarize
+
+```
+=== Turborepo Task Complete ===
+
+Topic: <configuration|caching|filtering|environment|ci|cli>
+Files referenced: <reference files consulted>
+
+<brief summary of what was done>
+```
+
+<user-request>
+$ARGUMENTS
+</user-request>

.agents/skills/turborepo/references/best-practices/RULE.md

@@ -0,0 +1,241 @@
+# Monorepo Best Practices
+
+Essential patterns for structuring and maintaining a healthy Turborepo monorepo.
+
+## Repository Structure
+
+### Standard Layout
+
+```
+my-monorepo/
+├── apps/                    # Application packages (deployable)
+│   ├── web/
+│   ├── docs/
+│   └── api/
+├── packages/                # Library packages (shared code)
+│   ├── ui/
+│   ├── utils/
+│   └── config-*/           # Shared configs (eslint, typescript, etc.)
+├── package.json            # Root package.json (minimal deps)
+├── turbo.json              # Turborepo configuration
+├── pnpm-workspace.yaml     # (pnpm) or workspaces in package.json
+└── pnpm-lock.yaml          # Lockfile (required)
+```
+
+### Key Principles
+
+1. **`apps/` for deployables**: Next.js sites, APIs, CLIs - things that get deployed
+2. **`packages/` for libraries**: Shared code consumed by apps or other packages
+3. **One purpose per package**: Each package should do one thing well
+4. **No nested packages**: Don't put packages inside packages
+
+## Package Types
+
+### Application Packages (`apps/`)
+
+- **Deployable**: These are the "endpoints" of your package graph
+- **Not installed by other packages**: Apps shouldn't be dependencies of other packages
+- **No shared code**: If code needs sharing, extract to `packages/`
+
+```json
+// apps/web/package.json
+{
+  "name": "web",
+  "private": true,
+  "dependencies": {
+    "@repo/ui": "workspace:*",
+    "next": "latest"
+  }
+}
+```
+
+### Library Packages (`packages/`)
+
+- **Shared code**: Utilities, components, configs
+- **Namespaced names**: Use `@repo/` or `@yourorg/` prefix
+- **Clear exports**: Define what the package exposes
+
+```json
+// packages/ui/package.json
+{
+  "name": "@repo/ui",
+  "exports": {
+    "./button": "./src/button.tsx",
+    "./card": "./src/card.tsx"
+  }
+}
+```
+
+## Package Compilation Strategies
+
+### Just-in-Time (Simplest)
+
+Export TypeScript directly; let the app's bundler compile it.
+
+```json
+{
+  "name": "@repo/ui",
+  "exports": {
+    "./button": "./src/button.tsx"
+  }
+}
+```
+
+**Pros**: Zero build config, instant changes
+**Cons**: Can't cache builds, requires app bundler support
+
+### Compiled (Recommended for Libraries)
+
+Package compiles itself with `tsc` or bundler.
+
+```json
+{
+  "name": "@repo/ui",
+  "exports": {
+    "./button": {
+      "types": "./src/button.tsx",
+      "default": "./dist/button.js"
+    }
+  },
+  "scripts": {
+    "build": "tsc"
+  }
+}
+```
+
+**Pros**: Cacheable by Turborepo, works everywhere
+**Cons**: More configuration
+
+## Dependency Management
+
+### Install Where Used
+
+Install dependencies in the package that uses them, not the root.
+
+```bash
+# Good: Install in the package that needs it
+pnpm add lodash --filter=@repo/utils
+
+# Avoid: Installing everything at root
+pnpm add lodash -w  # Only for repo-level tools
+```
+
+### Root Dependencies
+
+Only these belong in root `package.json`:
+
+- `turbo` - The build system
+- `husky`, `lint-staged` - Git hooks
+- Repository-level tooling
+
+### Internal Dependencies
+
+Use workspace protocol for internal packages:
+
+```json
+// pnpm/bun
+{ "@repo/ui": "workspace:*" }
+
+// npm/yarn
+{ "@repo/ui": "*" }
+```
+
+## Exports Best Practices
+
+### Use `exports` Field (Not `main`)
+
+```json
+{
+  "exports": {
+    ".": "./src/index.ts",
+    "./button": "./src/button.tsx",
+    "./utils": "./src/utils.ts"
+  }
+}
+```
+
+### Avoid Barrel Files
+
+Don't create `index.ts` files that re-export everything:
+
+```typescript
+// BAD: packages/ui/src/index.ts
+export * from './button';
+export * from './card';
+export * from './modal';
+// ... imports everything even if you need one thing
+
+// GOOD: Direct exports in package.json
+{
+  "exports": {
+    "./button": "./src/button.tsx",
+    "./card": "./src/card.tsx"
+  }
+}
+```
+
+### Namespace Your Packages
+
+```json
+// Good
+{ "name": "@repo/ui" }
+{ "name": "@acme/utils" }
+
+// Avoid (conflicts with npm registry)
+{ "name": "ui" }
+{ "name": "utils" }
+```
+
+## Common Anti-Patterns
+
+### Accessing Files Across Package Boundaries
+
+```typescript
+// BAD: Reaching into another package
+import { Button } from "../../packages/ui/src/button";
+
+// GOOD: Install and import properly
+import { Button } from "@repo/ui/button";
+```
+
+### Shared Code in Apps
+
+```
+// BAD
+apps/
+  web/
+    shared/        # This should be a package!
+      utils.ts
+
+// GOOD
+packages/
+  utils/           # Proper shared package
+    src/utils.ts
+```
+
+### Too Many Root Dependencies
+
+```json
+// BAD: Root has app dependencies
+{
+  "dependencies": {
+    "react": "^18",
+    "next": "^14",
+    "lodash": "^4"
+  }
+}
+
+// GOOD: Root only has repo tools
+{
+  "devDependencies": {
+    "turbo": "latest",
+    "husky": "latest"
+  }
+}
+```
+
+## See Also
+
+- [structure.md](./structure.md) - Detailed repository structure patterns
+- [packages.md](./packages.md) - Creating and managing internal packages
+- [dependencies.md](./dependencies.md) - Dependency management strategies

.agents/skills/turborepo/references/best-practices/dependencies.md

@@ -0,0 +1,246 @@
+# Dependency Management
+
+Best practices for managing dependencies in a Turborepo monorepo.
+
+## Core Principle: Install Where Used
+
+Dependencies belong in the package that uses them, not the root.
+
+```bash
+# Good: Install in specific package
+pnpm add react --filter=@repo/ui
+pnpm add next --filter=web
+
+# Avoid: Installing in root
+pnpm add react -w  # Only for repo-level tools!
+```
+
+## Benefits of Local Installation
+
+### 1. Clarity
+
+Each package's `package.json` lists exactly what it needs:
+
+```json
+// packages/ui/package.json
+{
+  "dependencies": {
+    "react": "^18.0.0",
+    "class-variance-authority": "^0.7.0"
+  }
+}
+```
+
+### 2. Flexibility
+
+Different packages can use different versions when needed:
+
+```json
+// packages/legacy-ui/package.json
+{ "dependencies": { "react": "^17.0.0" } }
+
+// packages/ui/package.json
+{ "dependencies": { "react": "^18.0.0" } }
+```
+
+### 3. Better Caching
+
+Installing in root changes workspace lockfile, invalidating all caches.
+
+### 4. Pruning Support
+
+`turbo prune` can remove unused dependencies for Docker images.
+
+## What Belongs in Root
+
+Only repository-level tools:
+
+```json
+// Root package.json
+{
+  "devDependencies": {
+    "turbo": "latest",
+    "husky": "^8.0.0",
+    "lint-staged": "^15.0.0"
+  }
+}
+```
+
+**NOT** application dependencies:
+
+- react, next, express
+- lodash, axios, zod
+- Testing libraries (unless truly repo-wide)
+
+## Installing Dependencies
+
+### Single Package
+
+```bash
+# pnpm
+pnpm add lodash --filter=@repo/utils
+
+# npm
+npm install lodash --workspace=@repo/utils
+
+# yarn
+yarn workspace @repo/utils add lodash
+
+# bun
+cd packages/utils && bun add lodash
+```
+
+### Multiple Packages
+
+```bash
+# pnpm
+pnpm add vitest --save-dev --filter=web --filter=@repo/ui
+
+# npm
+npm install vitest --save-dev --workspace=web --workspace=@repo/ui
+
+# yarn (v2+)
+yarn workspaces foreach -R --from '{web,@repo/ui}' add vitest --dev
+```
+
+### Internal Packages
+
+```bash
+# pnpm
+pnpm add @repo/ui --filter=web
+
+# This updates package.json:
+{
+  "dependencies": {
+    "@repo/ui": "workspace:*"
+  }
+}
+```
+
+## Keeping Versions in Sync
+
+### Option 1: Tooling
+
+```bash
+# syncpack - Check and fix version mismatches
+npx syncpack list-mismatches
+npx syncpack fix-mismatches
+
+# manypkg - Similar functionality
+npx @manypkg/cli check
+npx @manypkg/cli fix
+
+# sherif - Rust-based, very fast
+npx sherif
+```
+
+### Option 2: Package Manager Commands
+
+```bash
+# pnpm - Update everywhere
+pnpm up --recursive typescript@latest
+
+# npm - Update in all workspaces
+npm install typescript@latest --workspaces
+```
+
+### Option 3: pnpm Catalogs (pnpm 9.5+)
+
+```yaml
+# pnpm-workspace.yaml
+packages:
+  - "apps/*"
+  - "packages/*"
+
+catalog:
+  react: ^18.2.0
+  typescript: ^5.3.0
+```
+
+```json
+// Any package.json
+{
+  "dependencies": {
+    "react": "catalog:" // Uses version from catalog
+  }
+}
+```
+
+## Internal vs External Dependencies
+
+### Internal (Workspace)
+
+```json
+// pnpm/bun
+{ "@repo/ui": "workspace:*" }
+
+// npm/yarn
+{ "@repo/ui": "*" }
+```
+
+Turborepo understands these relationships and orders builds accordingly.
+
+### External (npm Registry)
+
+```json
+{ "lodash": "^4.17.21" }
+```
+
+Standard semver versioning from npm.
+
+## Peer Dependencies
+
+For library packages that expect the consumer to provide dependencies:
+
+```json
+// packages/ui/package.json
+{
+  "peerDependencies": {
+    "react": "^18.0.0",
+    "react-dom": "^18.0.0"
+  },
+  "devDependencies": {
+    "react": "^18.0.0", // For development/testing
+    "react-dom": "^18.0.0"
+  }
+}
+```
+
+## Common Issues
+
+### "Module not found"
+
+1. Check the dependency is installed in the right package
+2. Run `pnpm install` / `npm install` to update lockfile
+3. Check exports are defined in the package
+
+### Version Conflicts
+
+Packages can use different versions - this is a feature, not a bug. But if you need consistency:
+
+1. Use tooling (syncpack, manypkg)
+2. Use pnpm catalogs
+3. Create a lint rule
+
+### Hoisting Issues
+
+Some tools expect dependencies in specific locations. Use package manager config:
+
+```yaml
+# .npmrc (pnpm)
+public-hoist-pattern[]=*eslint*
+public-hoist-pattern[]=*prettier*
+```
+
+## Lockfile
+
+**Required** for:
+
+- Reproducible builds
+- Turborepo dependency analysis
+- Cache correctness
+
+```bash
+# Commit your lockfile!
+git add pnpm-lock.yaml  # or package-lock.json, yarn.lock
+```