chore: release v3.187.0

fix(blob-storage): surface SDK error cause in validation failure messages (#14276 )
The validate button's error handler only returned the wrapper message ("Failed to upload file to S3 or generate signed URL") without the underlying SDK cause. Users had no actionable info about what was wrong. Unwrap one level of error.cause to include the SDK message (e.g. "The AWS Access Key Id you provided does not exist in our records"), capped at 500 chars to limit exposure. Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-06-16 12:23:53 +02:00 · 2026-06-16 09:59:35 +00:00 · 2026-06-16 08:44:41 +00:00 · 2026-06-16 08:32:03 +00:00 · 2026-06-16 08:31:52 +00:00 · 2026-06-16 07:59:56 +00:00
1858 changed files with 166132 additions and 43922 deletions
@@ -1,70 +1,39 @@
 # Agent Guidelines for Langfuse

-This is the canonical root agent guide for the repo. The root `AGENTS.md`
-should remain only as a discovery symlink so tools that require that filename
-continue to work while `.agents/` stays the source of truth.
-
 Langfuse is an open source LLM engineering platform for developing, monitoring,
 evaluating, and debugging AI applications.

-## Maintenance Contract
+## How To Work

- `AGENTS.md` is a living document.
- Keep this file concise and router-like. Push narrow or conditional workflows
-  into package-local `AGENTS.md` files or shared skills under `skills/`.
- Update this file in the same PR when monorepo-level architecture, workflows,
-  dependency boundaries, mandatory verification commands, or release/security
-  processes materially change.
- Update this file and the relevant shared skills when user feedback introduces
-  a durable repo-level default for future agents. Do not edit this file for
-  one-off task preferences.
- For package-local material changes, update the nearest package `AGENTS.md` in
-  the same PR.
-
-## Start Here By Task
-
- Repo-wide agent setup, `.agents/**`, provider shims, or MCP/bootstrap config:
-  [`README.md`](README.md),
-  [`skills/agent-setup-maintenance/SKILL.md`](skills/agent-setup-maintenance/SKILL.md)
- Backend/API work in `web/src/server/**`, `web/src/pages/api/public/**`,
-  `worker/src/**`, or `packages/shared/src/**`:
-  [`skills/backend-dev-guidelines/SKILL.md`](skills/backend-dev-guidelines/SKILL.md)
- Model pricing work in `worker/src/constants/default-model-prices.json`,
-  `packages/shared/src/server/llm/types.ts`, or related pricing files:
-  [`skills/add-model-price/SKILL.md`](skills/add-model-price/SKILL.md)
- Code review tasks:
-  [`skills/code-review/SKILL.md`](skills/code-review/SKILL.md)
- Changelog drafting for completed feature branches:
-  [`skills/changelog-writing/SKILL.md`](skills/changelog-writing/SKILL.md)
- ClickHouse schema/query review:
-  [`skills/clickhouse-best-practices/SKILL.md`](skills/clickhouse-best-practices/SKILL.md)
- Monorepo/Turbo task graph changes:
-  [`skills/turborepo/SKILL.md`](skills/turborepo/SKILL.md)
- User-visible frontend changes, Playwright review, or browser signoff:
-  [`skills/frontend-browser-review/SKILL.md`](skills/frontend-browser-review/SKILL.md)
- Web UI and frontend entry points:
-  `../web/AGENTS.md`
- Worker queues and processors:
-  `../worker/AGENTS.md`
- Shared contracts, exports, schema, and migrations:
-  `../packages/shared/AGENTS.md`
- EE-only work:
-  `../ee/AGENTS.md`
-
-Read the minimal set required for the task. More-specific package guides and
-shared skills take precedence over this root file for their scoped areas.
+- Read the minimal local context required for the task.
+- Keep changes scoped and avoid unrelated refactors.
+- For bug fixes, write the failing test first, confirm it fails, then fix the
+  bug. If the bug depends on a data shape, pause and ask: can
+  `pnpm run seed` prefill that shape locally? If not, consider extending a
+  seeder scenario so the bug stays cheaply reproducible
+  (`packages/shared/scripts/seeder/AGENTS.md`), or note why a seed cannot
+  express it.
+- For user-visible frontend changes in `web/**`, review the affected flow in a
+  real browser before signoff. Prefill the data the flow needs with the seed
+  CLI (`pnpm run seed -- list` shows scenarios; runs print UI deep links) —
+  never with ad-hoc scripts or raw ClickHouse inserts.
+- For documentation screenshots in Markdown, avoid fixed `height` on `<img>`
+  tags; prefer Markdown images or width-only HTML so previews preserve aspect
+  ratio.
+- Never commit secrets or credentials. Keep `.env*.example` files in
+  sync with required env vars.

 ## Project Structure

 ```text
 langfuse/
-├─ web/                     # Next.js app (UI + tRPC + public REST)
-├─ worker/                  # Queue consumers and background processing
-├─ packages/shared/         # Shared domain, DB, queue contracts, repositories
-├─ ee/                      # Enterprise package consumed by web
-├─ generated/               # Generated API clients (do not hand-edit)
-├─ fern/                    # API definition sources
-└─ scripts/                 # Repo scripts
+|- web/                     # Next.js app (UI + tRPC + public REST)
+|- worker/                  # Queue consumers and background processing
+|- packages/shared/         # Shared domain, DB, queue contracts, repositories
+|- ee/                      # Enterprise package consumed by web
+|- generated/               # Generated API clients (do not hand-edit)
+|- fern/                    # API definition sources
+`- scripts/                 # Repo scripts
 ```

 - Dependency direction:
@@ -79,10 +48,7 @@ langfuse/
  - Postgres schema: `packages/shared/prisma/schema.prisma`
  - ClickHouse migrations:
    `packages/shared/clickhouse/migrations/{clustered,unclustered}/*.sql`
- Architecture handbook:
-  [langfuse.com/handbook/product-engineering/architecture](https://langfuse.com/handbook/product-engineering/architecture)
-  with source markdown in
-  `../langfuse-docs/content/handbook/product-engineering/architecture.mdx`
+- Architecture principles live in `.agents/ARCHITECTURE_PRINCIPLES.md`.

 ## Core Commands

@@ -94,82 +60,48 @@ langfuse/
 - Typecheck all: `pnpm run typecheck` / `pnpm tc`
 - Build check: `pnpm run build:check`
 - Full build: `pnpm run build`
- Full reset/bootstrap (destructive): `pnpm run dx`
- Codex environment bootstrap: `bash scripts/codex/setup.sh`
- Codex environment maintenance: `bash scripts/codex/maintenance.sh`
- Install Playwright Chromium for agent browser review: `pnpm run playwright:install`
+- Worktree bootstrap: `bash scripts/codex/setup.sh`
+- Worktree maintenance: `bash scripts/codex/maintenance.sh`
+- Install Playwright Chromium: `pnpm run playwright:install`

-Minimum verification matrix:
+## Verification

-| Change scope | Minimum verification |
-| --- | --- |
-| `web/**` only | `pnpm --filter web run lint` + targeted web tests |
-| `worker/**` only | `pnpm --filter worker run lint` + targeted worker tests |
-| `packages/shared/**` (non-schema) | `pnpm --filter @langfuse/shared run lint` + one targeted web check + one targeted worker check |
-| `packages/shared/prisma/**` or `packages/shared/clickhouse/**` | `pnpm --filter @langfuse/shared run lint` + `pnpm run db:generate` + targeted web/worker regressions |
-| Public API contract (`web/src/pages/api/public/**`, `web/src/features/public-api/types/**`, `fern/apis/**`) | web lint + targeted server API tests + Fern update/regeneration; never hand-edit `generated/**` |
-| Cross-package refactor (`web` + `worker` + `shared`) | `pnpm run lint` + `pnpm run typecheck` + targeted tests per impacted package |
+- `web/**`: `pnpm run lint` plus targeted web tests.
+- `worker/**`: `pnpm run lint` plus targeted worker tests.
+- `packages/shared/**` non-schema changes:
+  `pnpm run lint` plus one targeted web check and one targeted worker check.
+- `packages/shared/prisma/**` or `packages/shared/clickhouse/**`:
+  `pnpm run lint`, `pnpm run db:generate`, and targeted web/worker
+  regressions.
+- Public API contracts in `web/src/pages/api/public/**`,
+  `web/src/features/public-api/types/**`, or `fern/apis/**`: `pnpm run lint`,
+  targeted server API tests, and Fern update/regeneration.
+- Cross-package refactors: `pnpm run lint`, `pnpm run typecheck`, and targeted
+  tests for impacted packages.

-## Repo Rules
+## Generated Files

- Keep changes scoped; avoid unrelated refactors.
- Prefer package-local implementation details in package `AGENTS.md` files.
- Do not hand-edit generated/build artifacts:
-  - `generated/*`
-  - `web/.next/*`
-  - `web/.next-check/*`
-  - `*/dist/*`
-  - `packages/shared/prisma/generated/*`
- Public API contract changes must update Fern sources in `fern/apis/**` and
-  regenerated outputs; never hand-edit `generated/**`.
- Keep tests independent and parallel-safe.
- For bug fixes, write the failing test first, confirm it fails, then fix the
-  bug.
- For user-visible frontend changes in `web/**`, review the affected flow in a
-  real browser with the Playwright MCP server before signoff. Use
-  `skills/frontend-browser-review/SKILL.md` and `../web/AGENTS.md` for the
-  browser-review loop.
- Never commit secrets or credentials. Keep `.env*.example` files in sync with
-  required env vars.
+Do not hand-edit generated or build artifacts:
+
+- `generated/*`
+- `web/.next/*`
+- `web/.next-check/*`
+- `*/dist/*`
+- `packages/shared/prisma/generated/*`
+
+Public API contract changes must update Fern sources in `fern/apis/**` and
+regenerated outputs. Never hand-edit `generated/**`.

 ## Shared Agent Setup

 - `.agents/AGENTS.md` is the canonical root guide.
 - Root `AGENTS.md` is a symlink to `.agents/AGENTS.md`.
 - Root `CLAUDE.md` is a compatibility symlink to `AGENTS.md`.
- Shared agent/tool config lives in `config.json` and shared skills live in
-  `skills/`.
- Project-scoped provider discovery files are generated local artifacts. Edit
-  the canonical files under `.agents/` instead of editing generated tool
-  directories by hand.
- If you change `.agents/config.json`, `skills/**`, or the shim-generation
-  workflow, run:
-  - `pnpm run agents:sync`
-  - `pnpm run agents:check`
- Do not commit generated provider config or shim outputs under `.claude/`,
-  `.cursor/`, `.codex/`, `.vscode/`, or `.mcp.json`.
- Durable cross-tool guidance belongs in root/package `AGENTS.md` files or
-  `skills/**`, not only in tool-specific config directories.
-
-## Commit, PR, and Release Rules
-
- Commit messages and PR titles must follow Conventional Commits:
-  `type(scope): description` or `type: description`.
- PR titles are validated by `.github/workflows/validate-pr-title.yml`.
- In PR descriptions, list impacted packages and executed verification commands.
- Release workflow is managed at root with `pnpm run release`.
- Promote `main` to `production` via
-  `.github/workflows/promote-main-to-production.yml` or
-  `pnpm run release:cloud`.
- Do not change release/versioning flow without updating this file and impacted
-  package guides.
-
-## Git and Tooling Notes
-
- Use `gh search issues` for GitHub issue search.
- Do not use destructive git commands such as `reset --hard` unless explicitly
-  requested.
- Do not revert unrelated working-tree changes.
- Keep commits focused and atomic.
- Remaining `.cursor/rules/*.mdc` files should stay thin wrappers around shared
-  docs or skills rather than owning durable repo guidance directly.
+- When creating or editing `.agents/skills/**`, use
+  `.agents/skills/skill-creator/SKILL.md`; keep skills concise with
+  progressive disclosure.
+- After changing shared agent setup, run `pnpm run agents:sync` and
+  `pnpm run agents:check`.
+- Generated provider config and shim outputs under `.claude/`, `.cursor/`,
+  `.codex/`, `.vscode/`, or `.mcp.json` are local artifacts, not source of
+  truth files.
@@ -0,0 +1,49 @@
+# Underlying Architecture Principles
+
+Langfuse architecture should optimize for high-scale, exploratory observability
+on wide, structured event data. These principles are grounded in current
+production scale and the reference material below.
+
+## Reference Posts
+
+- [Simplifying Langfuse for Scale](https://langfuse.com/blog/2026-03-10-simplify-langfuse-for-scale)
+- [Charity Majors on Observability 2.0](https://charity.wtf/tag/observability-2-0/)
+- [All you need is Wide Events, not "Metrics, Logs and Traces"](https://isburmistrov.substack.com/p/all-you-need-is-wide-events-not-metrics)
+
+## Principles
+
+- Model observations as the primary analytical unit. A trace is a correlation
+  handle that links related observations, not the only useful entry point.
+- Prefer wide, richly attributed events over fragmented metrics, logs, and trace
+  records that require later reconstruction.
+- Preserve high-cardinality context so users can slice, group, filter, and debug
+  unknown unknowns without predefining every future question.
+- Favor immutable or append-oriented event records for high-volume telemetry.
+  Updates that force read-time deduplication create hidden query costs at scale.
+- Denormalize carefully when it removes hot-path joins and makes common filters
+  into direct column predicates.
+- Design storage and query paths around columnar access patterns: narrow field
+  selection, time-bounded scans, useful ordering keys, and data pruning.
+- Keep list, dashboard, and aggregate views on compact query-optimized
+  representations. Fetch large raw payloads only for focused detail views.
+- Make API contracts scale-aware: require time windows where needed, expose field
+  selection, use token pagination, and avoid defaults that can scan all history.
+- Treat cost and operational simplicity as architectural constraints. Extra
+  databases, queues, materialized views, and migrations must earn their long-term
+  operational burden.
+- Preserve real-time or near-real-time debugging workflows. Batch processing can
+  help, but it should not make fresh production behavior invisible.
+
+## Practical Defaults For Agents
+
+- Before adding a metric, ask whether the same question is better answered from
+  wide event data.
+- Before adding a join, ask whether the attribute should be propagated or
+  denormalized onto the observation path.
+- Before reading large fields, ask whether the view needs them or can defer them
+  until a single-record fetch.
+- Before adding an update-heavy design, ask whether immutable events plus
+  derived representations would be simpler at production scale.
+- Before documenting public behavior, separate stable public contracts from
+  private production topology, account details, secret names, and incident
+  runbooks.
@@ -10,6 +10,8 @@ or `.vscode/`.
 ## Layout

 - `AGENTS.md`: canonical shared root instructions
+- `ARCHITECTURE_PRINCIPLES.md`: architecture principles for high-scale
+  observability
 - `config.json`: shared bootstrap and MCP configuration used to generate
  tool-specific shims
 - `skills/`: shared, tool-neutral implementation guidance for recurring
@@ -38,15 +40,42 @@ Current shape:
    "playwright": {
      "transport": "stdio",
      "command": "npx",
-      "args": ["-y", "@playwright/mcp@latest"]
+      "args": [
+        "-y",
+        "@playwright/mcp@latest",
+        "--isolated",
+        "--save-session",
+        "--output-dir",
+        "/tmp/playwright-mcp",
+        "--test-id-attribute",
+        "data-testid"
+      ]
    },
-    "datadog": {
+    "langfuse-docs": {
      "transport": "http",
-      "url": "https://mcp.datadoghq.com/api/unstable/mcp-server/mcp"
+      "url": "https://langfuse.com/api/mcp"
+    },
+    "linear": {
+      "transport": "http",
+      "url": "https://mcp.linear.app/mcp"
    }
  },
  "claude": {
-    "settings": {}
+    "settings": {
+      "permissions": {
+        "allow": [
+          "Bash(find:*)",
+          "Bash(rg:*)",
+          "Bash(grep:*)",
+          "Bash(ls:*)",
+          "Bash(cat:*)",
+          "Bash(head:*)",
+          "Bash(tail:*)"
+        ],
+        "deny": []
+      },
+      "enableAllProjectMcpServers": true
+    }
  },
  "codex": {
    "environment": {
@@ -175,7 +204,6 @@ Use them for durable, reusable guidance such as:

 Do not use skills for one-off task notes or tool runtime configuration.

+Use `skills/skill-creator/SKILL.md` when creating or editing shared skills.
 `pnpm run agents:sync` projects the shared skills into `.claude/skills/` so
 Claude can discover the same repo-owned skills.
-
-For the skill authoring workflow, see [skills/README.md](skills/README.md).
@@ -12,9 +12,9 @@
        "-y",
        "@playwright/mcp@latest",
        "--isolated",
-        "--save-trace",
+        "--save-session",
        "--output-dir",
-        ".playwright-mcp",
+        "/tmp/playwright-mcp",
        "--test-id-attribute",
        "data-testid"
      ]
@@ -1,109 +0,0 @@
-# Shared Skills
-
-Shared repo skills for any coding agent working in Langfuse.
-
-Use these from `AGENTS.md`. Claude Code reaches the same shared instructions via
-the root `CLAUDE.md` compatibility symlink. Shared skills should stay focused on
-reusable implementation guidance rather than runtime automation.
-
-For the shared agent config and generated shim model, start with
-[`../README.md`](../README.md).
-
-Claude discovers these shared skills through symlinks under `.claude/skills/`.
-Those discovery links are created and verified by `pnpm run agents:sync` and
-`pnpm run agents:check`.
-
-Shared skills should use progressive disclosure:
-
- `SKILL.md` is the short entrypoint with trigger guidance and navigation.
- `AGENTS.md` is optional and should stay concise when it exists.
- `references/` holds focused prose references that agents should open only
-  when the task needs them.
- `scripts/` holds deterministic helpers for repetitive or fragile steps.
-
-## Available Skills
-
-### agent-setup-maintenance
-
-Use for:
- `.agents/config.json`, `.agents/AGENTS.md`, or `.agents/README.md`
- shared skill additions or shared skill routing changes
- generated shim behavior in `scripts/agents/sync-agent-shims.mjs`
- install-time agent sync behavior and provider discovery paths
-
-Open: [agent-setup-maintenance/SKILL.md](agent-setup-maintenance/SKILL.md)
-
-### frontend-browser-review
-
-Use for:
- user-visible changes in `web/**`
- Playwright MCP browser review before signoff
- checking visible regressions in layout, styling, navigation, or responsive behavior
-
-Open: [frontend-browser-review/SKILL.md](frontend-browser-review/SKILL.md)
-
-### backend-dev-guidelines
-
-Use for:
- tRPC routers and procedures
- public API endpoints
- worker queue processors
- Prisma and ClickHouse backed services
- backend auth, validation, observability, and tests
-
-Open: [backend-dev-guidelines/SKILL.md](backend-dev-guidelines/SKILL.md)
-
-### add-model-price
-
-Use for:
- `worker/src/constants/default-model-prices.json`
- `packages/shared/src/server/llm/types.ts`
- pricing tiers, tokenizer IDs, and model `matchPattern` changes
-
-Open: [add-model-price/SKILL.md](add-model-price/SKILL.md)
-
-### code-review
-
-Use for:
- PR or branch review
- correctness, regression, and risk-focused review tasks
- applying the repo-specific review policy in
-  `code-review/references/review-checklist.md`
-
-Open: [code-review/SKILL.md](code-review/SKILL.md)
-
-### changelog-writing
-
-Use for:
- changelog entries for completed features
- drafting user-facing release notes
- checking related docs links for changelog posts
-
-Open: [changelog-writing/SKILL.md](changelog-writing/SKILL.md)
-
-## Adding a New Shared Skill
-
-1. Codex may create or refine shared skills under `.agents/skills/` when a
-   repo-specific workflow becomes repeated enough to justify durable guidance.
-2. Create a concise `.agents/skills/<skill-name>/SKILL.md`.
-3. Add `.agents/skills/<skill-name>/AGENTS.md` only when the skill benefits
-   from a short router or checklist on top of `SKILL.md`.
-4. Prefer `references/` for detailed prose and `scripts/` for deterministic
-   execution helpers.
-5. Keep the skill tightly scoped to one domain or workflow.
-6. Link the skill from `AGENTS.md` if it is relevant across the repo.
-7. Run `pnpm run agents:sync` and `pnpm run agents:check` so Claude's projected
-   `.claude/skills/` view stays in sync.
-8. Update `AGENTS.md` or package-local `AGENTS.md` if the new skill changes the
-   default reusable workflow for future agents.
-9. Run the relevant verification for the package or workflow the skill affects.
-
-## Skill Design Rules
-
- Keep the skill tool-neutral.
- Use `SKILL.md` as the short entrypoint, not the full knowledge dump.
- Prefer `references/` for deeper docs and `scripts/` for deterministic helpers.
- Avoid copying large sections of repo docs into the skill when a stable link is
-  enough.
- If the skill is web- or package-specific, link the nearest package
-  `AGENTS.md` or package docs instead of restating them.
@@ -1,544 +0,0 @@
-# Add Model Price
-
-Guide for adding or updating model pricing entries in Langfuse. Use this when
-editing `worker/src/constants/default-model-prices.json`,
-`packages/shared/src/server/llm/types.ts`, model `matchPattern` values,
-tokenizer IDs, or pricing tiers.
-
-## Purpose
-
-This guide keeps model pricing changes consistent across providers and runtime
-surfaces so Langfuse can calculate token costs accurately.
-
-## How to Use This Skill
-
-1. Read [references/schema-and-tiers.md](references/schema-and-tiers.md) for
-   the JSON shape and pricing-tier rules.
-2. Read
-   [references/provider-sources-and-price-keys.md](references/provider-sources-and-price-keys.md)
-   for official pricing URLs, per-token conversion, and provider-specific usage
-   keys.
-3. Read [references/match-patterns.md](references/match-patterns.md) when you
-   need to add or expand regex coverage.
-4. Read
-   [references/workflow-and-validation.md](references/workflow-and-validation.md)
-   for the end-to-end edit workflow, validation rules, and common mistakes.
-
-## Deterministic Helpers
-
- Validate the pricing file:
-  `node .agents/skills/add-model-price/scripts/validate-pricing-file.mjs`
- Test a regex directly:
-  `node .agents/skills/add-model-price/scripts/test-match-pattern.mjs --pattern '(?i)^(openai/)?(gpt-4o)$' --accept gpt-4o openai/gpt-4o --reject gpt-4o-mini`
- Test the regex for an existing model entry:
-  `node .agents/skills/add-model-price/scripts/test-match-pattern.mjs --model gpt-4o --accept gpt-4o openai/gpt-4o --reject gpt-4o-mini`
-
-## Quick Start Checklist
-
-### Adding a New Model
-
- [ ] Gather official pricing from the provider documentation
- [ ] Generate a lowercase UUID for the model entry
- [ ] Create a `matchPattern` that covers supported provider formats
- [ ] Add at least one default pricing tier
- [ ] Insert the pricing entry into
-      `worker/src/constants/default-model-prices.json`
- [ ] Update `packages/shared/src/server/llm/types.ts` if the model should be
-      selectable in playground or evaluation flows
- [ ] Validate the JSON after editing
-
-### Updating an Existing Model
-
- [ ] Update the relevant prices, keys, tiers, or regexes
- [ ] Refresh `updatedAt` to today's ISO-8601 timestamp
- [ ] Validate the JSON after editing
-
-## Target Files
-
- Pricing data:
-  `worker/src/constants/default-model-prices.json`
- Shared model types:
-  `packages/shared/src/server/llm/types.ts`
- Validation logic:
-  `packages/shared/src/features/model-pricing/validation.ts`
- Matching logic:
-  `packages/shared/src/server/pricing-tiers/matcher.ts`
- Tests:
-  `worker/src/__tests__/pricing-tier-matcher.test.ts`
-
-## Data Structure
-
-### Complete Model Entry Schema
-
-```json
-{
-  "id": "uuid-generated-with-uuidgen",
-  "modelName": "model-name-identifier",
-  "matchPattern": "(?i)^regex-pattern$",
-  "createdAt": "ISO-8601-timestamp",
-  "updatedAt": "ISO-8601-timestamp",
-  "tokenizerConfig": null,
-  "tokenizerId": "claude|openai|null",
-  "pricingTiers": [
-    {
-      "id": "model-uuid_tier_default",
-      "name": "Standard",
-      "isDefault": true,
-      "priority": 0,
-      "conditions": [],
-      "prices": {
-        "input": 0.000005,
-        "output": 0.000025
-      }
-    }
-  ]
-}
-```
-
-### Required Fields
-
-| Field | Type | Description |
-| --- | --- | --- |
-| `id` | string | Unique lowercase UUID |
-| `modelName` | string | Primary model identifier |
-| `matchPattern` | string | Regex for matching model names |
-| `createdAt` | string | ISO-8601 timestamp set on creation |
-| `updatedAt` | string | ISO-8601 timestamp refreshed whenever the entry changes |
-| `pricingTiers` | array | At least one pricing tier |
-
-### Optional Fields
-
-| Field | Type | Default | Description |
-| --- | --- | --- | --- |
-| `tokenizerId` | string | `null` | `"claude"`, `"openai"`, or `null` |
-| `tokenizerConfig` | object | `null` | Custom tokenizer settings |
-
-## Pricing Tier Structure
-
-### Default Tier
-
-Every model must have exactly one default tier:
-
-```json
-{
-  "id": "{model-id}_tier_default",
-  "name": "Standard",
-  "isDefault": true,
-  "priority": 0,
-  "conditions": [],
-  "prices": {}
-}
-```
-
-Rules for the default tier:
-
- `isDefault` must be `true`
- `priority` must be `0`
- `conditions` must be `[]`
-
-### Additional Tiers
-
-Use extra tiers for context-window or usage-based pricing:
-
-```json
-{
-  "id": "uuid-for-tier",
-  "name": "Large Context (>200K)",
-  "isDefault": false,
-  "priority": 1,
-  "conditions": [
-    {
-      "usageDetailPattern": "(input|prompt|cached)",
-      "operator": "gt",
-      "value": 200000,
-      "caseSensitive": false
-    }
-  ],
-  "prices": {}
-}
-```
-
-Supported condition operators: `gt`, `gte`, `lt`, `lte`, `eq`, `neq`
-
-## Official Pricing Sources
-
-Always fetch pricing from the provider's official docs before editing. Do not
-infer or estimate missing values.
-
-| Provider | Source |
-| --- | --- |
-| Anthropic Claude | `https://platform.claude.com/docs/en/about-claude/pricing` |
-| OpenAI | `https://openai.com/api/pricing/` |
-| Google Gemini | `https://ai.google.dev/pricing` |
-| AWS Bedrock | `https://aws.amazon.com/bedrock/pricing/` |
-| Azure OpenAI | `https://azure.microsoft.com/pricing/details/cognitive-services/openai-service/` |
-
-Gather:
-
-1. Base input token price per million tokens
-2. Output token price per million tokens
-3. Cache write price when supported
-4. Cache read price when supported
-5. Any long-context pricing tiers
-6. All model ID formats that Langfuse should match
-
-## Price Conversion
-
-Values in `default-model-prices.json` are per token, not per million tokens.
-
-| Provider Price | JSON Value |
-| --- | --- |
-| `$5 / MTok` | `5e-6` |
-| `$25 / MTok` | `25e-6` |
-| `$0.50 / MTok` | `0.5e-6` |
-| `$6.25 / MTok` | `6.25e-6` |
-
-Formula:
-
-```text
-price_per_token = price_per_mtok / 1_000_000
-```
-
-## Common Price Keys by Provider
-
-### Anthropic Claude Models
-
-```json
-{
-  "input": "<base_input_price>",
-  "input_tokens": "<base_input_price>",
-  "output": "<output_price>",
-  "output_tokens": "<output_price>",
-  "cache_creation_input_tokens": "<cache_write_price>",
-  "input_cache_creation": "<cache_write_price>",
-  "cache_read_input_tokens": "<cache_read_price>",
-  "input_cache_read": "<cache_read_price>"
-}
-```
-
-### OpenAI Models
-
-```json
-{
-  "input": "<input_price>",
-  "input_cached_tokens": "<cached_input_price>",
-  "input_cache_read": "<cached_input_price>",
-  "output": "<output_price>"
-}
-```
-
-### Google Gemini Models
-
-```json
-{
-  "input": "<input_price>",
-  "input_modality_1": "<input_price>",
-  "prompt_token_count": "<input_price>",
-  "promptTokenCount": "<input_price>",
-  "input_cached_tokens": "<cached_price>",
-  "cached_content_token_count": "<cached_price>",
-  "output": "<output_price>",
-  "output_modality_1": "<output_price>",
-  "candidates_token_count": "<output_price>",
-  "candidatesTokenCount": "<output_price>"
-}
-```
-
-## Match Pattern Examples
-
-### Anthropic Claude: API + Bedrock + Vertex
-
-```regex
-(?i)^(anthropic\/)?(claude-opus-4-6|(eu\\.|us\\.|apac\\.)?anthropic\\.claude-opus-4-6-v1(:0)?|claude-opus-4-6)$
-```
-
-Matches:
-
- `claude-opus-4-6`
- `anthropic/claude-opus-4-6`
- `anthropic.claude-opus-4-6-v1:0`
- `us.anthropic.claude-opus-4-6-v1:0`
- `claude-opus-4-6`
-
-### With Version Date
-
-```regex
-(?i)^(anthropic\/)?(claude-opus-4-5-20251101|(eu\\.|us\\.|apac\\.)?anthropic\\.claude-opus-4-5-20251101-v1:0|claude-opus-4-5@20251101)$
-```
-
-### OpenAI
-
-```regex
-(?i)^(openai\/)?(gpt-4o)$
-```
-
-### Google Gemini
-
-```regex
-(?i)^(google\/)?(gemini-2.5-pro)$
-```
-
-### Pattern Components
-
-| Component | Purpose | Example |
-| --- | --- | --- |
-| `(?i)` | Case-insensitive match | `gpt-4o` and `GPT-4O` |
-| `^...$` | Full-string match | Avoids partial matches |
-| `(provider\/)?` | Optional provider prefix | `openai/gpt-4o` |
-| `(eu\\.|us\\.|apac\\.)?` | Optional AWS region prefix | `us.anthropic.model` |
-| `(:0)?` | Optional version suffix | Bedrock model versions |
-| `@date` | Vertex AI version format | `claude-3-5-sonnet@20240620` |
-
-## Step-by-Step Workflow
-
-### 1. Fetch Official Pricing
-
-Open the official provider pricing page and capture the model's input, output,
-cache write, and cache read prices.
-
-### 2. Generate a Lowercase UUID
-
-```bash
-uuidgen
-```
-
-Convert the output to lowercase before using it.
-
-### 3. Create the JSON Entry
-
-Example for a model with $5 input, $25 output, $6.25 cache write, and
-$0.50 cache read:
-
-```json
-{
-  "id": "13458bc0-1c20-44c2-8753-172f54b67647",
-  "modelName": "claude-opus-4-6",
-  "matchPattern": "(?i)^(anthropic\/)?(claude-opus-4-6|(eu\\.|us\\.|apac\\.)?anthropic\\.claude-opus-4-6-v1(:0)?|claude-opus-4-6)$",
-  "createdAt": "2026-03-09T00:00:00.000Z",
-  "updatedAt": "2026-03-09T00:00:00.000Z",
-  "tokenizerConfig": null,
-  "tokenizerId": "claude",
-  "pricingTiers": [
-    {
-      "id": "13458bc0-1c20-44c2-8753-172f54b67647_tier_default",
-      "name": "Standard",
-      "isDefault": true,
-      "priority": 0,
-      "conditions": [],
-      "prices": {
-        "input": 5e-6,
-        "input_tokens": 5e-6,
-        "output": 25e-6,
-        "output_tokens": 25e-6,
-        "cache_creation_input_tokens": 6.25e-6,
-        "input_cache_creation": 6.25e-6,
-        "cache_read_input_tokens": 0.5e-6,
-        "input_cache_read": 0.5e-6
-      }
-    }
-  ]
-}
-```
-
-### 4. Insert the Entry
-
-Add the entry to the JSON array in
-`worker/src/constants/default-model-prices.json`. Keep related models grouped
-together.
-
-### 5. Update Shared Model Types When Needed
-
-If the model should be available in the playground or LLM-as-judge flows, add
-it to the correct array in `packages/shared/src/server/llm/types.ts`.
-
-Model arrays include:
-
- `anthropicModels`
- `openAIModels`
- `vertexAIModels`
- `googleAIStudioModels`
-
-Do not add a new model as the first entry in one of these arrays. The first
-entry is used as a default model in some test or evaluation paths and newer
-models may not be available to all users yet.
-
-### 6. Validate the Change
-
-```bash
-jq . worker/src/constants/default-model-prices.json > /dev/null
-```
-
-You can also inspect a specific entry:
-
-```bash
-jq '.[] | select(.modelName == "claude-opus-4-6")' worker/src/constants/default-model-prices.json
-```
-
-## Multi-Tier Example
-
-For models with long-context pricing:
-
-```json
-{
-  "id": "uuid-here",
-  "modelName": "model-name",
-  "matchPattern": "...",
-  "pricingTiers": [
-    {
-      "id": "uuid-here_tier_default",
-      "name": "Standard",
-      "isDefault": true,
-      "priority": 0,
-      "conditions": [],
-      "prices": {
-        "input": 5e-6,
-        "output": 25e-6
-      }
-    },
-    {
-      "id": "uuid-for-large-context-tier",
-      "name": "Large Context (>200K)",
-      "isDefault": false,
-      "priority": 1,
-      "conditions": [
-        {
-          "usageDetailPattern": "(input|prompt|cached)",
-          "operator": "gt",
-          "value": 200000,
-          "caseSensitive": false
-        }
-      ],
-      "prices": {
-        "input": 10e-6,
-        "output": 37.5e-6
-      }
-    }
-  ]
-}
-```
-
-## Validation Rules
-
-1. Exactly one default tier must have `isDefault: true`
-2. The default tier must have `priority: 0`
-3. The default tier must have `conditions: []`
-4. Non-default tiers must have `priority > 0`
-5. Non-default tiers must have at least one condition
-6. Priorities must be unique within a model
-7. Tier names must be unique within a model
-8. Each tier must contain at least one price
-9. All tiers must expose the same usage-type keys
-10. Regex patterns must be valid and safe
-
-## Common Mistakes
-
-### Guessing Instead of Using Official Pricing
-
-Wrong:
-
-```json
-{
-  "cache_creation_input_tokens": "input_price * 1.25"
-}
-```
-
-Correct:
-
-```json
-{
-  "cache_creation_input_tokens": 6.25e-6
-}
-```
-
-### Using MTok Values Directly
-
-Wrong:
-
-```json
-{
-  "input": 5
-}
-```
-
-Correct:
-
-```json
-{
-  "input": 5e-6
-}
-```
-
-### Missing the Default Tier Suffix
-
-Wrong:
-
-```json
-{
-  "id": "some-uuid"
-}
-```
-
-Correct:
-
-```json
-{
-  "id": "model-uuid_tier_default"
-}
-```
-
-### Invalid Regex Escaping
-
-Wrong:
-
-```json
-{
-  "matchPattern": "anthropic.claude"
-}
-```
-
-Correct:
-
-```json
-{
-  "matchPattern": "anthropic\\.claude"
-}
-```
-
-### Forgetting to Update `updatedAt`
-
-Wrong:
-
-```json
-{
-  "updatedAt": "2025-12-12T15:00:06.513Z"
-}
-```
-
-Correct:
-
-```json
-{
-  "updatedAt": "2026-03-09T00:00:00.000Z"
-}
-```
-
-## Testing Model Matching
-
-After adding a model, verify that the regex matches the intended provider
-variants:
-
-```javascript
-const pattern = new RegExp(matchPattern);
-console.log(pattern.test("claude-opus-4-6")); // true
-console.log(pattern.test("anthropic/claude-opus-4-6")); // true
-console.log(pattern.test("anthropic.claude-opus-4-6-v1:0")); // true
-console.log(pattern.test("us.anthropic.claude-opus-4-6-v1:0")); // true
-```
-
-## Existing Model Templates
-
-Use nearby entries as templates:
-
- `claude-opus-4-5-20251101` for Anthropic multi-provider patterns
- `gpt-4o` for a simple OpenAI pattern
- `gemini-2.5-pro` for a multi-tier Gemini entry
@@ -19,18 +19,36 @@ updates in `packages/shared/`.

 ## How to Read This Skill

- Start with [AGENTS.md](AGENTS.md) for the high-level workflow and helper
-  scripts.
- Then open only the specific reference file that matches the task.
+- Use this `SKILL.md` as the high-level workflow and helper index.
+- Open only the specific reference file that matches the task.
+
+## Quick Start Checklist
+
+### Adding a New Model
+
+- Gather official pricing from the provider documentation.
+- Generate a lowercase UUID for the model entry.
+- Create a `matchPattern` that covers supported provider formats.
+- Add at least one default pricing tier.
+- Insert the pricing entry into `worker/src/constants/default-model-prices.json`.
+- Update `packages/shared/src/server/llm/types.ts` if the model should be
+  selectable in playground or evaluation flows.
+- Validate the JSON after editing.
+
+### Updating an Existing Model
+
+- Update the relevant prices, keys, tiers, or regexes.
+- Refresh `updatedAt` to today's ISO-8601 timestamp.
+- Validate the JSON after editing.

 ## Reference Map

-| Topic | Read this when | File |
-| --- | --- | --- |
-| Schema and tier rules | You need the entry shape or pricing-tier invariants | [references/schema-and-tiers.md](references/schema-and-tiers.md) |
+| Topic                           | Read this when                                                                        | File                                                                                           |
+| ------------------------------- | ------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------- |
+| Schema and tier rules           | You need the entry shape or pricing-tier invariants                                   | [references/schema-and-tiers.md](references/schema-and-tiers.md)                               |
 | Provider sources and price keys | You need official pricing URLs, per-token conversion, or provider-specific usage keys | [references/provider-sources-and-price-keys.md](references/provider-sources-and-price-keys.md) |
-| Match patterns | You are editing `matchPattern` regexes or provider coverage | [references/match-patterns.md](references/match-patterns.md) |
-| Workflow and validation | You are applying the end-to-end edit process or checking common mistakes | [references/workflow-and-validation.md](references/workflow-and-validation.md) |
+| Match patterns                  | You are editing `matchPattern` regexes or provider coverage                           | [references/match-patterns.md](references/match-patterns.md)                                   |
+| Workflow and validation         | You are applying the end-to-end edit process or checking common mistakes              | [references/workflow-and-validation.md](references/workflow-and-validation.md)                 |

 ## Deterministic Helpers

@@ -38,3 +56,5 @@ updates in `packages/shared/`.
  `node .agents/skills/add-model-price/scripts/validate-pricing-file.mjs`
 - Match-pattern tester:
  `node .agents/skills/add-model-price/scripts/test-match-pattern.mjs --model <modelName> --accept <sample...> --reject <sample...>`
+- Direct regex tester:
+  `node .agents/skills/add-model-price/scripts/test-match-pattern.mjs --pattern '(?i)^(openai/)?(gpt-4o)$' --accept gpt-4o openai/gpt-4o --reject gpt-4o-mini`
@@ -94,3 +94,45 @@ Use extra tiers for context-window or usage-based pricing:
 ```

 Supported operators: `gt`, `gte`, `lt`, `lte`, `eq`, `neq`
+
+## Multi-Tier Example
+
+Use multiple tiers when a provider changes pricing by context window or usage
+class. Keep exactly one default tier and assign higher priorities to conditional
+tiers:
+
+```json
+{
+  "pricingTiers": [
+    {
+      "id": "{model-id}_tier_default",
+      "name": "Standard",
+      "isDefault": true,
+      "priority": 0,
+      "conditions": [],
+      "prices": {
+        "input": 0.000001,
+        "output": 0.000002
+      }
+    },
+    {
+      "id": "uuid-for-large-context-tier",
+      "name": "Large Context (>200K)",
+      "isDefault": false,
+      "priority": 1,
+      "conditions": [
+        {
+          "usageDetailPattern": "(input|prompt|cached)",
+          "operator": "gt",
+          "value": 200000,
+          "caseSensitive": false
+        }
+      ],
+      "prices": {
+        "input": 0.000002,
+        "output": 0.000004
+      }
+    }
+  ]
+}
+```
@@ -22,10 +22,58 @@ template, then:

 - add the new entry near related models
 - refresh `updatedAt` when editing an existing entry
- update `packages/shared/src/server/llm/types.ts` when the model should be
-  selectable in product flows

-### 4. Validate the Result
+Example for a model with `$5` input, `$25` output, `$6.25` cache write, and
+`$0.50` cache read per million tokens:
+
+```json
+{
+  "id": "13458bc0-1c20-44c2-8753-172f54b67647",
+  "modelName": "claude-opus-4-6",
+  "matchPattern": "(?i)^(anthropic\/)?(claude-opus-4-6|(eu\\.|us\\.|apac\\.)?anthropic\\.claude-opus-4-6-v1(:0)?)$",
+  "createdAt": "2026-03-09T00:00:00.000Z",
+  "updatedAt": "2026-03-09T00:00:00.000Z",
+  "tokenizerConfig": null,
+  "tokenizerId": "claude",
+  "pricingTiers": [
+    {
+      "id": "13458bc0-1c20-44c2-8753-172f54b67647_tier_default",
+      "name": "Standard",
+      "isDefault": true,
+      "priority": 0,
+      "conditions": [],
+      "prices": {
+        "input": 5e-6,
+        "input_tokens": 5e-6,
+        "output": 25e-6,
+        "output_tokens": 25e-6,
+        "cache_creation_input_tokens": 6.25e-6,
+        "input_cache_creation": 6.25e-6,
+        "cache_read_input_tokens": 0.5e-6,
+        "input_cache_read": 0.5e-6
+      }
+    }
+  ]
+}
+```
+
+### 4. Update Shared Model Types When Needed
+
+If the model should be available in playground or LLM-as-judge flows, add it to
+the correct array in `packages/shared/src/server/llm/types.ts`.
+
+Common arrays include:
+
+- `anthropicModels`
+- `openAIModels`
+- `vertexAIModels`
+- `googleAIStudioModels`
+
+Do not add a new model as the first entry in one of these arrays. The first
+entry is used as a default model in some test or evaluation paths, and newer
+models may not be available to all users yet.
+
+### 5. Validate the Result

 Run the bundled validator:

@@ -33,6 +81,12 @@ Run the bundled validator:
 node .agents/skills/add-model-price/scripts/validate-pricing-file.mjs
 ```

+For quick manual inspection, use `jq`:
+
+```bash
+jq '.[] | select(.modelName == "claude-opus-4-6")' worker/src/constants/default-model-prices.json
+```
+
 ## Validation Rules

 1. Exactly one default tier must have `isDefault: true`
@@ -46,6 +100,17 @@ node .agents/skills/add-model-price/scripts/validate-pricing-file.mjs
 9. All tiers must expose the same usage-type keys
 10. Regex patterns must be valid

+## Testing Model Matching
+
+Use the bundled tester before finishing any `matchPattern` change:
+
+```bash
+node .agents/skills/add-model-price/scripts/test-match-pattern.mjs --model <modelName> --accept <sample...> --reject <sample...>
+```
+
+Use representative accepted and rejected model IDs for every provider format the
+regex is intended to cover.
+
 ## Common Mistakes

 - Guessing prices instead of using official provider docs
@@ -14,6 +14,9 @@ Use this skill when changing the shared agent setup for the repository.

 - Read [`../../README.md`](../../README.md) for the shared config and shim model.
 - Read root [`../../AGENTS.md`](../../AGENTS.md) for repo-level expectations.
+- When adding or editing shared skills, use
+  [`../skill-creator/SKILL.md`](../skill-creator/SKILL.md), then apply the
+  repo-specific checks in this skill.
 - Inspect [`../../../scripts/agents/sync-agent-shims.mjs`](../../../scripts/agents/sync-agent-shims.mjs)
  before changing generated outputs or provider discovery behavior.
 - Inspect [`../../../scripts/postinstall.sh`](../../../scripts/postinstall.sh)
@@ -28,11 +31,14 @@ Use this skill when changing the shared agent setup for the repository.
 3. Treat tool-specific directories such as `.claude/`, `.cursor/`, `.codex/`,
   `.vscode/`, and `.mcp.json` as generated discovery surfaces unless the tool
   requires a truly tool-specific feature.
-4. Keep root `AGENTS.md` concise and router-like. Move detailed or conditional
-   workflows into shared skills or package `AGENTS.md` files.
-5. When adding or changing a shared skill, update `skills/README.md` and link
-   it from root `AGENTS.md` if it changes the default reusable workflow.
-6. When shared setup behavior changes materially, update `README.md` and
+4. Keep root `AGENTS.md` concise. Move detailed or conditional workflows into
+   shared skills or package `AGENTS.md` files.
+5. Treat developer feedback as a learning loop: when a task reveals a durable
+   repo convention, recurring pitfall, reusable workflow, or verification
+   pattern, update the smallest relevant `AGENTS.md` or shared skill.
+6. When adding or changing a shared skill, keep `SKILL.md` as the entrypoint; do
+   not add skill-by-skill links to root `AGENTS.md`.
+7. When shared setup behavior changes materially, update `README.md` and
   contributor-facing docs in the same PR.

 ## Docker / Install-Time Constraint
@@ -0,0 +1,62 @@
+---
+name: analyze-cloud-costs
+description: |
+  Analyze Langfuse Cloud infrastructure cost structure using Metabase cost
+  marts. Use when asked about cloud spend, AWS versus ClickHouse cost splits,
+  cost drivers by provider/service/usage type/account, daily cost per tracing
+  event, infra cost dashboards, or cost regressions visible in Metabase.
+---
+
+# Analyze Cloud Costs
+
+## Overview
+
+Use this skill for evidence-backed Langfuse Cloud cost analysis. The primary
+source is the Metabase infra cost dashboard and its production cost marts; the
+deliverable should name the time window, query grain, top drivers, and caveats.
+
+## Workflow
+
+1. Clarify the question and choose the grain:
+   - Headline daily totals: total, AWS, ClickHouse, tracing events, and cost per
+     100k events.
+   - Cost structure: provider, service, usage type, operation, account, and day.
+   - Driver or regression analysis: compare a recent complete-day window against
+     a prior baseline.
+2. Load [`references/cost-marts.md`](references/cost-marts.md) for table IDs,
+   field IDs, query examples, and caveats.
+3. Use the Metabase MCP. If the Metabase tools are not visible, discover them
+   with tool search before falling back to manual interpretation.
+4. Prefer complete UTC days. Avoid treating current-day AWS cost as final
+   because AWS CUR rows can arrive late.
+5. Start broad, then drill down:
+   - Provider split.
+   - Service split within the dominant provider.
+   - Usage type, operation, and account split for the top services.
+   - Daily trend when explaining change over time.
+6. Report only what the queried data supports. If a requested slice is absent,
+   say that no rows were found for that slice instead of inventing a driver.
+
+## Query Rules
+
+- Use `mcp__metabase__.query` for quick reads. Use
+  `construct_query` plus `execute_query` when you need to inspect or reuse the
+  opaque query.
+- Pass `filters`, `aggregations`, `group_by`, and `fields` as JSON arrays. Some
+  tool schemas may display these as strings; if that happens, serialize the same
+  arrays without changing their shape.
+- Keep limits explicit and small enough for analysis. Use pagination only when
+  the continuation token is needed.
+- Include the Metabase dashboard link or query result context in the final
+  answer when useful.
+
+## Output Expectations
+
+Summarize:
+
+- Time window and whether it uses complete UTC days.
+- Total cost and provider split when relevant.
+- Top cost drivers by service, usage type, operation, or account.
+- Trend or baseline comparison when the user asks "why did this change?"
+- Caveats, especially incomplete current-day AWS data and ClickHouse credit
+  labeling in the unified mart.
@@ -0,0 +1,4 @@
+interface:
+  display_name: "Analyze Cloud Costs"
+  short_description: "Analyze Langfuse Cloud cost structure"
+  default_prompt: "Use $analyze-cloud-costs to explain recent Langfuse Cloud cost drivers from Metabase."
@@ -0,0 +1,137 @@
+# Langfuse Cloud Cost Marts
+
+Use this reference when querying or explaining Langfuse Cloud cost structure.
+
+Dashboard:
+
+- https://langfuse.metabaseapp.com/dashboard/22-infra-cost?account=&date=past90days&tab=20-tab-1
+
+## Primary Tables
+
+| Purpose | Table | ID |
+| --- | --- | --- |
+| Unified AWS and ClickHouse cost rows by provider, service, usage type, account, and day | `langfuse_prod.mart_daily_cost_chart` | `739` |
+| Daily headline totals plus tracing event counts and cost per 100k events | `langfuse_prod.mart_daily_cost_with_events` | `784` |
+| Detailed AWS CUR summary by product, operation, account, and usage type | `langfuse_prod.mart_aws_cost_daily_by_service` | `610` |
+| Detailed ClickHouse costs by entity and metric | `langfuse_prod.mart_clickhouse_daily_cost` | `689` |
+
+Prefer table `739` for structural breakdowns. Prefer table `784` for daily
+headline totals.
+
+## Field IDs
+
+### `mart_daily_cost_chart` (`739`)
+
+| Field ID | Field |
+| --- | --- |
+| `t739-0` | `usage_date` |
+| `t739-1` | `service_provider` |
+| `t739-2` | `service_name` |
+| `t739-3` | `operation` |
+| `t739-4` | `usage_type` |
+| `t739-5` | `account_name` |
+| `t739-6` | `cost_usd` |
+
+### `mart_daily_cost_with_events` (`784`)
+
+| Field ID | Field |
+| --- | --- |
+| `t784-0` | `usage_date` |
+| `t784-1` | `total_cost_usd` |
+| `t784-2` | `clickhouse_cost_usd` |
+| `t784-3` | `aws_cost_usd` |
+| `t784-5` | `s3_api_operations_cost_usd` |
+| `t784-6` | `total_tracing_events` |
+| `t784-7` | `total_cost_per_100k_events` |
+
+## Metabase MCP Patterns
+
+The Metabase MCP supports `query` for direct reads and
+`construct_query` plus `execute_query` for reusable opaque queries. In practice,
+pass `filters`, `aggregations`, `group_by`, and `fields` as JSON arrays:
+
+```json
+{
+  "table_id": 739,
+  "filters": [
+    {
+      "field_id": "t739-0",
+      "operation": "greater-than-or-equal",
+      "value": "2026-05-09"
+    }
+  ],
+  "aggregations": [
+    {
+      "function": "sum",
+      "field_id": "t739-6"
+    }
+  ],
+  "group_by": [
+    { "field_id": "t739-1" },
+    { "field_id": "t739-2" }
+  ],
+  "limit": "200"
+}
+```
+
+If a tool surface insists on strings for those parameters, serialize the same
+arrays as JSON strings.
+
+## Common Breakdowns
+
+Provider split:
+
+- Table `739`
+- Aggregate `sum(t739-6)`
+- Group by `t739-1`
+
+Service split:
+
+- Table `739`
+- Aggregate `sum(t739-6)`
+- Group by `t739-1`, `t739-2`
+
+Usage type split:
+
+- Table `739`
+- Aggregate `sum(t739-6)`
+- Group by `t739-1`, `t739-4`
+
+Environment/account split:
+
+- Table `739`
+- Aggregate `sum(t739-6)`
+- Group by `t739-1`, `t739-5`
+
+Daily headline totals:
+
+- Table `784`
+- Filter `t784-0` by date.
+- Read `total_cost_usd`, `clickhouse_cost_usd`, `aws_cost_usd`,
+  `total_tracing_events`, and `total_cost_per_100k_events`.
+
+Daily trend by provider:
+
+- Table `739`
+- Aggregate `sum(t739-6)`
+- Group by `t739-0`, `t739-1`
+
+Drilldown sequence for a cost spike:
+
+1. Compare total daily cost in table `784`.
+2. Split the same days by provider in table `739`.
+3. Split the dominant provider by service.
+4. Split the dominant service by usage type, operation, and account.
+
+## Caveats
+
+- Current-day AWS cost can be incomplete because AWS CUR data may not have
+  landed yet.
+- For stable recent analysis, prefer the last complete UTC days rather than
+  including today.
+- ClickHouse cost rows are labeled `cost_usd` in the unified mart, but the
+  source metric is ClickHouse credits. Mention this when precision or billing
+  interpretation matters.
+- Field IDs can change if Metabase models are rebuilt. If a query fails, search
+  Metabase for the table name and inspect the returned metadata before
+  changing the analysis.
@@ -1,577 +0,0 @@
-# Backend Development Guidelines
-
-## Purpose
-
-Establish consistency and best practices across Langfuse's backend packages (web, worker, packages/shared) using Next.js 14, tRPC, BullMQ, and TypeScript patterns.
-
-## When to Use This Skill
-
-Use this guide when working on:
-
- Creating or modifying tRPC routers and procedures
- Creating or modifying public API endpoints (REST)
- Creating or modifying BullMQ queue consumers and producers
- Building services with business logic
- Authenticating API requests
- Accessing resources based on entitlements
- Implementing middleware (tRPC, NextAuth, public API)
- Database operations with Prisma (PostgreSQL) or ClickHouse
- Observability with OpenTelemetry, DataDog, logger, and traceException
- Input validation with Zod v4
- Environment configuration from env variables
- Backend testing and refactoring
-
---
-
-## Quick Start
-
-### UI: New tRPC Feature Checklist (Web)
-
- [ ] **Router**: Define in `features/[feature]/server/*Router.ts`
- [ ] **Procedures**: Use appropriate procedure type (protected, public)
- [ ] **Authentication**: Use JWT authorization via middlewares.
- [ ] **Entitlement check**: Access resources based on resource and role
- [ ] **Validation**: Zod v4 schema for input
- [ ] **Service**: Business logic in service file
- [ ] **Error handling**: Use traceException wrapper
- [ ] **Tests**: Unit + integration tests in `__tests__/`
- [ ] **Config**: Access via env.mjs
-
-### SDKs: New Public API Endpoint Checklist (Web)
-
- [ ] **Route file**: Create in `pages/api/public/`
- [ ] **Wrapper**: Use `withMiddlewares` + `createAuthedProjectAPIRoute`
- [ ] **Types**: Define in `features/public-api/types/`
- [ ] **Authentication**: Authorization via basic auth
- [ ] **Validation**: Zod schemas for query/body/response
- [ ] **Versioning**: Versioning in API path and Zod schemas for query/body/response
- [ ] **Fern API Docs**: Update `fern/apis/server/definition/` to match TypeScript types
- [ ] **Tests**: Add end-to-end test in `__tests__/async/`
-
-### New Queue Processor Checklist (Worker)
-
- [ ] **Processor**: Create in `worker/src/queues/`
- [ ] **Queue types**: Create queue types in `packages/shared/src/server/queues`
- [ ] **Service**: Business logic in `features/` or `worker/src/features/`
- [ ] **Error handling**: Distinguish between errors which should fail queue processing and errors which should result in a succeeded event.
- [ ] **Queue registration**: Add to WorkerManager in app.ts
- [ ] **Tests**: Add vitest tests in worker
-
---
-
-## Architecture Overview
-
-### Layered Architecture
-
-```
-# Web Package (Next.js 14)
-
-┌─ tRPC API ──────────────────┐   ┌── Public REST API ──────────┐
-│                             │   │                             │
-│  HTTP Request               │   │  HTTP Request               │
-│      ↓                      │   │      ↓                      │
-│  tRPC Procedure             │   │  withMiddlewares +          │
-│  (protectedProjectProcedure)│   │  createAuthedProjectAPIRoute│
-│      ↓                      │   │      ↓                      │
-│  Service (business logic)   │   │  Service (business logic)   │
-│      ↓                      │   │      ↓                      │
-│  Prisma / ClickHouse        │   │  Prisma / ClickHouse        │
-│                             │   │                             │
-└─────────────────────────────┘   └─────────────────────────────┘
-                 ↓
-            [optional]: Publish to Redis BullMQ queue
-                 ↓
-┌─ Worker Package (Express) ──────────────────────────────────┐
-│                                                             │
-│  BullMQ Queue Job                                           │
-│      ↓                                                      │
-│  Queue Processor (handles job)                              │
-│      ↓                                                      │
-│  Service (business logic)                                   │
-│      ↓                                                      │
-│  Prisma / ClickHouse                                        │
-│                                                             │
-└─────────────────────────────────────────────────────────────┘
-```
-
-**Key Principles:**
-
- **Web**: tRPC procedures for UI OR public API routes for SDKs → Services → Database
- **Worker**: Queue processors → Services → Database
- **packages/shared**: Shared code for Web and Worker
-
-See [references/architecture-overview.md](references/architecture-overview.md)
-for complete details.
-
---
-
-## Directory Structure
-
-### Web Package (`/web/`)
-
-```
-web/src/
-├── features/                # Feature-organized code
-│   ├── [feature-name]/
-│   │   ├── server/          # Backend logic
-│   │   │   ├── *Router.ts   # tRPC router
-│   │   │   └── service.ts   # Business logic
-│   │   ├── components/      # React components
-│   │   └── types/           # Feature types
-├── server/
-│   ├── api/
-│   │   ├── routers/         # tRPC routers
-│   │   ├── trpc.ts          # tRPC setup & middleware
-│   │   └── root.ts          # Main router
-│   ├── auth.ts              # NextAuth.js config
-│   └── db.ts                # Database client
-├── pages/
-│   ├── api/
-│   │   ├── public/          # Public REST APIs
-│   │   └── trpc/            # tRPC endpoint
-│   └── [routes].tsx         # Next.js pages
-├── __tests__/               # Jest tests
-│   └── async/               # Integration tests
-├── instrumentation.ts       # OpenTelemetry (FIRST IMPORT)
-└── env.mjs                  # Environment config
-```
-
-### Worker Package (`/worker/`)
-
-```
-worker/src/
-├── queues/                  # BullMQ processors
-│   ├── evalQueue.ts
-│   ├── ingestionQueue.ts
-│   └── workerManager.ts
-├── features/                # Business logic
-│   └── [feature]/
-│       └── service.ts
-├── instrumentation.ts       # OpenTelemetry (FIRST IMPORT)
-├── app.ts                   # Express setup + queue registration
-├── env.ts                   # Environment config
-└── index.ts                 # Server start
-```
-
-### Shared Package (`/packages/shared/`)
-
-```
-shared/src/
-├── server/                  # Server utilities
-│   ├── auth/                # Authentication helpers
-│   ├── clickhouse/          # ClickHouse client & schema
-│   ├── instrumentation/     # OpenTelemetry helpers
-│   ├── llm/                 # LLM integration utilities
-│   ├── redis/               # Redis queues & cache
-│   ├── repositories/        # Data repositories
-│   ├── services/            # Shared services
-│   ├── utils/               # Server utilities
-│   ├── logger.ts
-│   └── queues.ts
-├── encryption/              # Encryption utilities
-├── features/                # Feature-specific code
-├── tableDefinitions/        # Table schemas
-├── utils/                   # Shared utilities
-├── constants.ts
-├── db.ts                    # Prisma client
-├── env.ts                   # Environment config
-└── index.ts                 # Main exports
-```
-
-**Import Paths (package.json exports):**
-
-The shared package exposes specific import paths for different use cases:
-
-| Import Path                                | Maps To                           | Use For                                                         |
-| ------------------------------------------ | --------------------------------- | --------------------------------------------------------------- |
-| `@langfuse/shared`                         | `dist/src/index.js`               | General types, schemas, utilities, constants                    |
-| `@langfuse/shared/src/db`                  | `dist/src/db.js`                  | Prisma client and database types                                |
-| `@langfuse/shared/src/server`              | `dist/src/server/index.js`        | Server-side utilities (queues, auth, services, instrumentation) |
-| `@langfuse/shared/src/server/auth/apiKeys` | `dist/src/server/auth/apiKeys.js` | API key management utilities                                    |
-| `@langfuse/shared/encryption`              | `dist/src/encryption/index.js`    | Encryption and signature utilities                              |
-
-**Usage Examples:**
-
-```typescript
-// General imports - types, schemas, constants, interfaces
-import {
-  CloudConfigSchema,
-  StringNoHTML,
-  AnnotationQueueObjectType,
-  type APIScoreV2,
-  type ColumnDefinition,
-  Role,
-} from "@langfuse/shared";
-
-// Database - Prisma client and types
-import { prisma, Prisma, JobExecutionStatus } from "@langfuse/shared/src/db";
-import { type DB as Database } from "@langfuse/shared";
-
-// Server utilities - queues, services, auth, instrumentation
-import {
-  logger,
-  instrumentAsync,
-  traceException,
-  redis,
-  getTracesTable,
-  StorageService,
-  sendMembershipInvitationEmail,
-  invalidateApiKeysForProject,
-  recordIncrement,
-  recordHistogram,
-} from "@langfuse/shared/src/server";
-
-// API key management (specific path)
-import { createAndAddApiKeysToDb } from "@langfuse/shared/src/server/auth/apiKeys";
-
-// Encryption utilities
-import { encrypt, decrypt, sign, verify } from "@langfuse/shared/encryption";
-```
-
-**What Goes Where:**
-
-The shared package provides types, utilities, and server code used by both web and worker packages. It has **5 export paths** that control frontend vs backend access:
-
-| Import Path                                | Usage                 | What's Included                                                                    |
-| ------------------------------------------ | --------------------- | ---------------------------------------------------------------------------------- |
-| `@langfuse/shared`                         | ✅ Frontend + Backend | Prisma types, Zod schemas, constants, table definitions, domain models, utilities  |
-| `@langfuse/shared/src/db`                  | 🔒 Backend only       | Prisma client instance                                                             |
-| `@langfuse/shared/src/server`              | 🔒 Backend only       | Services, repositories, queues, auth, ClickHouse, LLM integration, instrumentation |
-| `@langfuse/shared/src/server/auth/apiKeys` | 🔒 Backend only       | API key management (separated to avoid circular deps)                              |
-| `@langfuse/shared/encryption`              | 🔒 Backend only       | Database field encryption/decryption                                               |
-
-**Naming Conventions:**
-
- tRPC Routers: `camelCaseRouter.ts` - `datasetRouter.ts`
- Services: `service.ts` in feature directory
- Queue Processors: `camelCaseQueue.ts` - `evalQueue.ts`
- Public APIs: `kebab-case.ts` - `dataset-items.ts`
-
---
-
-## Core Principles
-
-### 1. tRPC Procedures Delegate to Services
-
-```typescript
-// ❌ NEVER: Business logic in procedures
-export const traceRouter = createTRPCRouter({
-  byId: protectedProjectProcedure
-    .input(z.object({ traceId: z.string() }))
-    .query(async ({ input, ctx }) => {
-      // 200 lines of logic here
-    }),
-});
-
-// ✅ ALWAYS: Delegate to service
-export const traceRouter = createTRPCRouter({
-  byId: protectedProjectProcedure
-    .input(z.object({ traceId: z.string() }))
-    .query(async ({ input, ctx }) => {
-      return await getTraceById(input.traceId);
-    }),
-});
-```
-
-### 2. Access Config via env.mjs, NEVER process.env
-
-```typescript
-// ❌ NEVER (except in env.mjs itself)
-const dbUrl = process.env.DATABASE_URL;
-
-// ✅ ALWAYS
-import { env } from "@/src/env.mjs";
-const dbUrl = env.DATABASE_URL;
-```
-
-### 3. Validate ALL Input with Zod v4
-
-```typescript
-import { z } from "zod/v4";
-
-const schema = z.object({
-  email: z.string().email(),
-  projectId: z.string(),
-});
-const validated = schema.parse(input);
-```
-
-### 4. Services Use Prisma Directly for Simple CRUD or Repositories for Complex Queries
-
-```typescript
-// Services use Prisma directly for simple CRUD
-import { prisma } from "@langfuse/shared/src/db";
-
-const dataset = await prisma.dataset.findUnique({
-  where: { id: datasetId, projectId }, // Always filter by projectId for tenant isolation
-});
-
-// Or use repositories for complex queries (traces, observations, scores)
-import { getTracesTable } from "@langfuse/shared/src/server";
-
-const traces = await getTracesTable({
-  projectId,
-  filter: [...],
-  limit: 1000,
-});
-```
-
-### 6. Observability: OpenTelemetry + DataDog (Not Sentry for Backend)
-
-**Langfuse uses OpenTelemetry for backend observability, with traces and logs sent to DataDog.**
-
-```typescript
-// Import observability utilities
-import {
-  logger,          // Winston logger with OpenTelemetry/DataDog context
-  traceException,  // Record exceptions to OpenTelemetry spans
-  instrumentAsync, // Create instrumented spans
-} from "@langfuse/shared/src/server";
-
-// Structured logging (includes trace_id, span_id, dd.trace_id)
-logger.info("Processing dataset", { datasetId, projectId });
-logger.error("Failed to create dataset", { error: err.message });
-
-// Record exceptions to OpenTelemetry (sent to DataDog)
-try {
-  await operation();
-} catch (error) {
-  traceException(error); // Records to current span
-  throw error;
-}
-
-// Instrument critical operations (all API routes auto-instrumented)
-const result = await instrumentAsync(
-  { name: "dataset.create" },
-  async (span) => {
-    span.setAttributes({ datasetId, projectId });
-    // Operation here
-    return dataset;
-  },
-);
-```
-
-**Note**: Frontend uses Sentry, but backend (tRPC, API routes, services, worker) uses OpenTelemetry + DataDog.
-
-### 7. Comprehensive Testing Required
-
-Write tests for all new features and bug fixes. See [testing-guide.md](references/testing-guide.md) for detailed examples.
-
-**Test Types:**
-
-| Type        | Framework | Location                                | Purpose                      |
-| ----------- | --------- | --------------------------------------- | ---------------------------- |
-| Integration | Jest      | `web/src/__tests__/async/`              | Full API endpoint testing    |
-| tRPC        | Jest      | `web/src/__tests__/async/`              | tRPC procedures with auth    |
-| Service     | Jest      | `web/src/__tests__/async/repositories/` | Repository/service functions |
-| Worker      | Vitest    | `worker/src/__tests__/`                 | Queue processors & streams   |
-
-**Quick Examples:**
-
-```typescript
-// Integration Test (Public API)
-const res = await makeZodVerifiedAPICall(
-  PostDatasetsV1Response, "POST", "/api/public/datasets",
-  { name: "test-dataset" }, auth
-);
-expect(res.status).toBe(200);
-
-// tRPC Test
-const { caller } = await prepare(); // Creates session + caller
-const response = await caller.automations.getAutomations({ projectId });
-expect(response).toHaveLength(1);
-
-// Service Test
-const result = await getObservationsWithModelDataFromEventsTable({
-  projectId, filter: [...], limit: 1000, offset: 0
-});
-expect(result.length).toBeGreaterThan(0);
-
-// Worker Test (vitest)
-const stream = await getObservationStream({ projectId, filter: [] });
-const rows = [];
-for await (const chunk of stream) rows.push(chunk);
-expect(rows).toHaveLength(2);
-```
-
-**Key Principles:**
-
- Use unique IDs (`randomUUID()`) to avoid test interference
- Clean up test data or use unique project IDs
- Tests must be independent and runnable in any order
- Prefer scoped cleanup or unique project IDs over global reset helpers
-
-### 8. Always Filter by projectId for Tenant Isolation
-
-```typescript
-// ✅ CORRECT: Filter by projectId for tenant isolation
-const trace = await prisma.trace.findUnique({
-  where: { id: traceId, projectId }, // Required for multi-tenant data isolation
-});
-
-// ✅ CORRECT: ClickHouse queries also require projectId
-const traces = await queryClickhouse({
-  query: `
-    SELECT * FROM traces
-    WHERE project_id = {projectId: String}
-    AND timestamp >= {startTime: DateTime64(3)}
-  `,
-  params: { projectId, startTime },
-});
-```
-
-### 9. Keep Fern API Definitions in Sync with TypeScript Types
-
-When modifying public API types in `web/src/features/public-api/types/`, the corresponding Fern API definitions in `fern/apis/server/definition/` must be updated to match.
-
-**Zod to Fern Type Mapping:**
-
-| Zod Type | Fern Type | Example |
-| -------- | --------- | ------- |
-| `.nullish()` | `optional<nullable<T>>` | `z.string().nullish()` → `optional<nullable<string>>` |
-| `.nullable()` | `nullable<T>` | `z.string().nullable()` → `nullable<string>` |
-| `.optional()` | `optional<T>` | `z.string().optional()` → `optional<string>` |
-| Always present | `T` | `z.string()` → `string` |
-
-**Source References:**
-
-Add a comment at the top of each Fern type referencing the TypeScript source file:
-
-```yaml
-# Source: web/src/features/public-api/types/traces.ts - APITrace
-Trace:
-  properties:
-    id: string
-    name:
-      type: nullable<string>
-```
-
---
-
-## Common Imports
-
-```typescript
-// tRPC (Web)
-import { z } from "zod/v4";
-import {
-  createTRPCRouter,
-  protectedProjectProcedure,
-} from "@/src/server/api/trpc";
-import { TRPCError } from "@trpc/server";
-
-// Database
-import { prisma } from "@langfuse/shared/src/db";
-import type { Prisma } from "@prisma/client";
-
-// ClickHouse
-import {
-  queryClickhouse,
-  queryClickhouseStream,
-  upsertClickhouse,
-} from "@langfuse/shared/src/server";
-
-// Observability - OpenTelemetry + DataDog (NOT Sentry for backend)
-import {
-  logger,          // Winston logger with OTEL/DataDog trace context
-  traceException,  // Record exceptions to OpenTelemetry spans
-  instrumentAsync, // Create instrumented spans for operations
-} from "@langfuse/shared/src/server";
-
-// Config
-import { env } from "@/src/env.mjs"; // web
-// or
-import { env } from "./env"; // worker
-
-// Public API (Web)
-import { withMiddlewares } from "@/src/features/public-api/server/withMiddlewares";
-import { createAuthedProjectAPIRoute } from "@/src/features/public-api/server/createAuthedProjectAPIRoute";
-
-// Queue Processing (Worker)
-import { Job } from "bullmq";
-import { QueueName, TQueueJobTypes } from "@langfuse/shared/src/server";
-```
-
---
-
-## Quick Reference
-
-### HTTP Status Codes
-
-| Code | Use Case     |
-| ---- | ------------ |
-| 200  | Success      |
-| 201  | Created      |
-| 400  | Bad Request  |
-| 401  | Unauthorized |
-| 403  | Forbidden    |
-| 404  | Not Found    |
-| 500  | Server Error |
-
-### Example Features to Reference
-
-Reference existing Langfuse features for implementation patterns:
- **Datasets** (`web/src/features/datasets/`) - Complete feature with tRPC router, public API, and service
- **Prompts** (`web/src/features/prompts/`) - Feature with versioning and templates
- **Evaluations** (`web/src/features/evals/`) - Complex feature with worker integration
- **Public API** (`web/src/features/public-api/`) - Middleware and route patterns
-
---
-
-## Anti-Patterns to Avoid
-
-❌ Business logic in routes/procedures
-❌ Direct process.env usage (always use env.mjs/env.ts)
-❌ Missing error handling
-❌ No input validation (always use Zod v4)
-❌ Missing projectId filter on tenant-scoped queries
-❌ console.log instead of logger/traceException (OpenTelemetry)
-
---
-
-## Navigation Guide
-
-| Need to...                | Read this                                                    |
-| ------------------------- | ------------------------------------------------------------ |
-| Understand architecture   | [architecture-overview.md](references/architecture-overview.md)         |
-| Create routes/controllers | [routing-and-controllers.md](references/routing-and-controllers.md)     |
-| Organize business logic   | [services-and-repositories.md](references/services-and-repositories.md) |
-| Create middleware         | [middleware-guide.md](references/middleware-guide.md)                   |
-| Database access           | [database-patterns.md](references/database-patterns.md)                 |
-| Manage config             | [configuration.md](references/configuration.md)                         |
-| Write tests               | [testing-guide.md](references/testing-guide.md)                         |
-
---
-
-## Reference Files
-
-### [architecture-overview.md](references/architecture-overview.md)
-
-Three-layer architecture (tRPC/Public API → Services → Data Access), request lifecycle for tRPC/Public API/Worker, Next.js 14 directory structure, dual database system (PostgreSQL + ClickHouse), separation of concerns, repository pattern for complex queries
-
-### [routing-and-controllers.md](references/routing-and-controllers.md)
-
-Next.js file-based routing, tRPC router patterns, Public REST API routes, layered architecture (Entry Points → Services → Repositories → Database), service layer organization, anti-patterns to avoid
-
-### [services-and-repositories.md](references/services-and-repositories.md)
-
-Service layer overview, dependency injection patterns, singleton patterns, repository pattern for data access, service design principles, caching strategies, testing services
-
-### [middleware-guide.md](references/middleware-guide.md)
-
-tRPC middleware (withErrorHandling, withOtelInstrumentation, enforceUserIsAuthed), seven tRPC procedure types (publicProcedure, authenticatedProcedure, protectedProjectProcedure, etc.), Public API middleware (withMiddlewares, createAuthedProjectAPIRoute), authentication patterns (NextAuth for tRPC, Basic Auth for Public API)
-
-### [database-patterns.md](references/database-patterns.md)
-
-Dual database architecture (PostgreSQL via Prisma + ClickHouse via direct client), PostgreSQL CRUD operations, ClickHouse query patterns (queryClickhouse, queryClickhouseStream, upsertClickhouse), repository pattern for complex queries, tenant isolation with projectId filtering, when to use which database
-
-### [configuration.md](references/configuration.md)
-
-Environment variable validation with Zod, package-specific configs (web/env.mjs with t3-oss/env-nextjs, worker/env.ts, shared/env.ts), NEXT_PUBLIC_LANGFUSE_CLOUD_REGION usage, LANGFUSE_EE_LICENSE_KEY for enterprise features, best practices for env management
-
-### [testing-guide.md](references/testing-guide.md)
-
-Integration tests (Public API with makeZodVerifiedAPICall), tRPC tests (createInnerTRPCContext, appRouter.createCaller), service-level tests (repository/service functions), worker tests (vitest with streams), test isolation principles, running tests (Jest for web, vitest for worker)
-
-**Skill Status**: COMPLETE ✅
-**Line Count**: ~540 lines
-**Progressive Disclosure**: 7 reference files ✅
@@ -1,6 +1,6 @@
 ---
 name: backend-dev-guidelines
-description: Shared backend guide for Langfuse's Next.js 14, tRPC, BullMQ, and TypeScript monorepo. Use when creating or reviewing tRPC routers, public REST endpoints, BullMQ queue processors, backend services, middleware, Prisma or ClickHouse data access, OpenTelemetry instrumentation, Zod validation, env configuration, or backend tests across web, worker, or packages/shared.
+description: Shared backend guide for Langfuse's Next.js, tRPC, BullMQ, and TypeScript monorepo. Use when creating or reviewing tRPC routers, public REST endpoints, BullMQ queue processors, backend services, middleware, Prisma or ClickHouse data access, OpenTelemetry instrumentation, Zod validation, env configuration, or backend tests across web, worker, or packages/shared.
 ---

 # Backend Development Guidelines
@@ -20,24 +20,97 @@ Use this skill for backend and API work across `web/`, `worker/`, and

 ## How to Read This Skill

- Start with [AGENTS.md](AGENTS.md) when the task spans multiple backend areas
-  or you need the end-to-end checklists.
+- Use this `SKILL.md` when the task spans multiple backend areas or you need the
+  end-to-end reference map.
 - Read only the specific reference file that matches the work when the scope is
  narrower.
+- If the task introduces a user-supplied URL, an outbound HTTP request, a new
+  integration, or touches secrets, RBAC, or redirect handling, also load the
+  shared [`security-review`](../security-review/SKILL.md) skill before
+  designing or implementing the change.
+
+## Quick Start Checklists
+
+### UI: New tRPC Feature
+
+- Define the router in `features/[feature]/server/*Router.ts`.
+- Use the appropriate protected or public procedure.
+- Authenticate with JWT-aware middleware.
+- Check project/resource access and entitlements.
+- Validate input with Zod v4.
+- Put business logic in a service file.
+- Use `traceException` for error handling where relevant.
+- Add unit or integration tests in `__tests__/`.
+- Access config via `env.mjs`.
+
+### SDKs: New Public API Endpoint
+
+- Create the route in `pages/api/public/`.
+- Wrap it with `withMiddlewares` and `createAuthedProjectAPIRoute`.
+- Define types in `features/public-api/types/`.
+- Authenticate with basic auth.
+- Validate query, body, and response with Zod schemas.
+- Include API versioning in paths and schemas.
+- Update Fern API definitions to match TypeScript types.
+- Add end-to-end tests in `__tests__/async/`.
+
+### Worker: New Queue Processor
+
+- Create the processor in `worker/src/queues/`.
+- Define queue types in `packages/shared/src/server/queues`.
+- Place business logic in `features/` or `worker/src/features/`.
+- Distinguish failed jobs from jobs that should succeed with a recorded error.
+- Register the queue in `WorkerManager` in `app.ts`.
+- Add worker vitest coverage.
+
+## Core Principles
+
+- tRPC procedures, public API routes, and queue processors delegate business
+  logic to services.
+- Access configuration through `env.mjs`; do not read `process.env` directly
+  outside env setup.
+- Validate all external input with Zod v4.
+- Use Prisma directly for simple CRUD and repositories for complex query access.
+- Use OpenTelemetry and DataDog for backend observability.
+- Always filter project-scoped database queries by `projectId`.
+- Keep Fern API definitions in sync with public TypeScript API contracts.
+- Keep backend tests independent and parallel-safe.
+
+## Live Examples
+
+- tRPC router with project auth and Zod input:
+  `web/src/features/events/server/eventsRouter.ts`.
+- Public API route with middleware and typed request/response schemas:
+  `web/src/pages/api/public/datasets/index.ts`.
+- Worker queue processor with typed jobs, logging, and retry behavior:
+  `worker/src/queues/evalQueue.ts`.
+- Tenant filters for Prisma and ClickHouse:
+  `references/database-patterns.md`.
+
+## Naming Conventions
+
+- tRPC routers: `camelCaseRouter.ts`, for example `datasetRouter.ts`.
+- Services: `service.ts` in the feature server directory.
+- Queue processors: `camelCaseQueue.ts`, for example `evalQueue.ts`.
+- Public API routes: kebab-case filenames, for example `dataset-items.ts`.
+
+## Anti-Patterns to Avoid
+
+- Business logic in routes or procedures.
+- Direct `process.env` usage instead of `env.mjs` / `env.ts`.
+- Missing error handling.
+- Missing input validation.
+- Missing `projectId` filters on tenant-scoped queries.
+- `console.log` instead of `logger` / `traceException`.

 ## Reference Map

-| Topic | Read this when | File |
-| --- | --- | --- |
-| Architecture and package boundaries | You need the web/worker/shared split, request flow, or queue lifecycle | [references/architecture-overview.md](references/architecture-overview.md) |
-| Routing and controllers | You are writing tRPC procedures, public API routes, or queue entrypoints | [references/routing-and-controllers.md](references/routing-and-controllers.md) |
-| Middleware and auth | You are changing request auth, permissions, or middleware composition | [references/middleware-guide.md](references/middleware-guide.md) |
-| Services and repositories | You are placing business logic, repository code, or DI patterns | [references/services-and-repositories.md](references/services-and-repositories.md) |
-| Database access | You are touching Prisma, ClickHouse, tenant filters, or query patterns | [references/database-patterns.md](references/database-patterns.md) |
-| Configuration | You are adding env vars, startup config, or runtime toggles | [references/configuration.md](references/configuration.md) |
-| Testing | You are adding or updating backend tests | [references/testing-guide.md](references/testing-guide.md) |
-
-## Full Compiled Guide
-
-Read [AGENTS.md](AGENTS.md) for the complete backend guide with checklists,
-directory conventions, imports, architecture, and cross-cutting practices.
+| Topic                               | Read this when                                                           | File                                                                               |
+| ----------------------------------- | ------------------------------------------------------------------------ | ---------------------------------------------------------------------------------- |
+| Architecture and package boundaries | You need the web/worker/shared split, request flow, or queue lifecycle   | [references/architecture-overview.md](references/architecture-overview.md)         |
+| Routing and controllers             | You are writing tRPC procedures, public API routes, or queue entrypoints | [references/routing-and-controllers.md](references/routing-and-controllers.md)     |
+| Middleware and auth                 | You are changing request auth, permissions, or middleware composition    | [references/middleware-guide.md](references/middleware-guide.md)                   |
+| Services and repositories           | You are placing business logic, repository code, or DI patterns          | [references/services-and-repositories.md](references/services-and-repositories.md) |
+| Database access                     | You are touching Prisma, ClickHouse, tenant filters, or query patterns   | [references/database-patterns.md](references/database-patterns.md)                 |
+| Configuration                       | You are adding env vars, startup config, or runtime toggles              | [references/configuration.md](references/configuration.md)                         |
+| Testing                             | You are adding or updating backend tests                                 | [references/testing-guide.md](references/testing-guide.md)                         |
@@ -1,6 +1,6 @@
 # Architecture Overview - Langfuse Backend

-Complete guide to the layered architecture pattern used in Langfuse's Next.js 14/tRPC/Express monorepo.
+Complete guide to the layered architecture pattern used in Langfuse's Next.js/tRPC/Express monorepo. Check package manifests such as `web/package.json` for current framework versions before version-sensitive work.

 ## Table of Contents

@@ -20,7 +20,7 @@ Langfuse uses a **three-layer architecture** with two primary entry points (tRPC
 ### The Three Layers

 ```
-# Web Package (Next.js 14)
+# Web Package (Next.js)

 ┌─ tRPC API ──────────────────┐   ┌── Public REST API ──────────┐
 │                             │   │                             │
@@ -163,7 +163,7 @@ Two types of entry points:
 ```typescript
 1. HTTP POST /api/public/datasets
   ↓
-2. Next.js API route handler (pages/api/public/datasets.ts)
+2. Next.js API route handler (pages/api/public/datasets/index.ts)
   ↓
 3. withMiddlewares wrapper executes:
   - Basic auth verification
@@ -571,7 +571,7 @@ export async function createDataset(data: {
 **Public API (Alternative Entry Point):**

 ```typescript
-// web/src/pages/api/public/datasets.ts
+// web/src/pages/api/public/datasets/index.ts
 import { withMiddlewares } from "@/src/features/public-api/server/withMiddlewares";
 import { createAuthedProjectAPIRoute } from "@/src/features/public-api/server/createAuthedProjectAPIRoute";
 import { createDataset } from "@/src/features/datasets/server/service";
@@ -864,7 +864,7 @@ const validated = bodySchema.parse(req.body);

 **Related Files:**

- [../AGENTS.md](../AGENTS.md) - Main guide
+- [../SKILL.md](../SKILL.md) - Main guide
 - [routing-and-controllers.md](routing-and-controllers.md) - tRPC and Public API details
 - [services-and-repositories.md](services-and-repositories.md) - Service patterns
 - [testing-guide.md](testing-guide.md) - Testing strategies
@@ -286,7 +286,7 @@ const licenseKey = env.LANGFUSE_EE_LICENSE_KEY;
 **When Set:**

 | Environment              | Value                  | Purpose                                        |
-|--------------------------|------------------------|------------------------------------------------|
+| ------------------------ | ---------------------- | ---------------------------------------------- |
 | **Developer Laptop**     | `"DEV"` or `"STAGING"` | Local development against cloud infrastructure |
 | **Langfuse Cloud US**    | `"US"`                 | Production US region                           |
 | **Langfuse Cloud EU**    | `"EU"`                 | Production EU region                           |
@@ -559,5 +559,5 @@ langfuse/

 **Related Files:**

- [../AGENTS.md](../AGENTS.md) - Main guide
+- [../SKILL.md](../SKILL.md) - Main guide
 - [architecture-overview.md](architecture-overview.md) - Architecture patterns
@@ -57,7 +57,7 @@ const project = await prisma.project.create({

 // ✅ GOOD: Read with projectId filter
 const trace = await prisma.trace.findUnique({
-  where: { id: traceId, projectId },  // ← Always include projectId for tenant isolation
+  where: { id: traceId, projectId }, // ← Always include projectId for tenant isolation
  include: {
    scores: true,
    project: { select: { id: true, name: true } },
@@ -77,12 +77,12 @@ await prisma.user.update({

 // ✅ GOOD: Delete with projectId
 await prisma.apiKey.delete({
-  where: { id: apiKeyId, projectId },  // ← Always include projectId
+  where: { id: apiKeyId, projectId }, // ← Always include projectId
 });

 // ✅ GOOD: Count with projectId
 const traceCount = await prisma.trace.count({
-  where: { projectId, userId },  // ← Always include projectId
+  where: { projectId, userId }, // ← Always include projectId
 });
 ```

@@ -225,7 +225,7 @@ const rows = await queryClickhouse<{ id: string; name: string }>({
    LIMIT {limit: UInt32}
  `,
  params: {
-    projectId,  // ← Required for tenant isolation
+    projectId, // ← Required for tenant isolation
    startTime: convertDateToClickhouseDateTime(startDate),
    limit: 100,
  },
@@ -341,6 +341,7 @@ const query = `
 ```

 **Why this is important:**
+
 - Langfuse is multi-tenant - each project's data must be isolated
 - The `project_id` filter ensures queries only access data from the intended tenant
 - All queries on project-scoped tables (traces, observations, scores, sessions, etc.) must filter by `project_id`
@@ -358,6 +359,34 @@ const query = `
 `;
 ```

+**`is_deleted` on `traces`, `observations`, `scores`, and `dataset_run_items_rmt` is dormant — avoid new filters.**
+
+These four tables are declared as
+`ReplacingMergeTree(event_ts, is_deleted)`, but no production
+code writes `is_deleted = 1` for them — all deletes use
+ClickHouse's lightweight `DELETE FROM` mutation (e.g.
+`deleteObservationsByTraceIds`,
+`deleteObservationsByProjectId`,
+`deleteObservationsOlderThanDays`), which marks rows via the
+engine-managed `_row_exists` column. `_row_exists` is handled
+transparently by the read path; no special query handling is
+needed.
+
+What this means for query authors:
+
+- **`WHERE is_deleted = 0` filters on these four tables are
+  dead weight in practice.** A few legacy reads still carry
+  them (e.g. `web/src/features/score-analytics/server/`); new
+  code should not add them unless soft-delete writes have
+  actually been introduced.
+
+**Separate case: `blob_storage_file_log`.** This table is also
+a `ReplacingMergeTree` but **does** use soft-delete
+intentionally — `ingestionFileDeletion.ts` writes
+`is_deleted: "1"`, `batch-project-blob-cleaner` reads with
+`countIf(is_deleted = 1)`. The guidance above does not apply
+to it.
+
 **3. Use time-based filtering for performance:**

 ```typescript
@@ -520,6 +549,7 @@ export const getScoresByTraceId = async (
 ### Project-Scoped vs Global Tables

 **Project-scoped tables (MUST filter by `project_id`):**
+
 - `traces` - All trace queries require `project_id`
 - `observations` - All observation queries require `project_id`
 - `scores` - All score queries require `project_id`
@@ -527,6 +557,7 @@ export const getScoresByTraceId = async (
 - `dataset_run_items_rmt` - All dataset run queries require `project_id`

 **Global tables (no `project_id` filter needed):**
+
 - `users` - User management (use `id` for filtering)
 - `organizations` - Organization data (use `id` for filtering)
 - System configuration tables
@@ -599,12 +630,12 @@ try {

 **Common Prisma error codes:**

-| Code     | Meaning                      | Typical Cause                         |
-| -------- | ---------------------------- | ------------------------------------- |
-| `P2002`  | Unique constraint violation  | Duplicate email, API key, etc.        |
-| `P2003`  | Foreign key constraint       | Referenced record doesn't exist       |
-| `P2025`  | Record not found             | Update/delete of non-existent record  |
-| `P2018`  | Required relation not found  | Connect to non-existent related record |
+| Code    | Meaning                     | Typical Cause                          |
+| ------- | --------------------------- | -------------------------------------- |
+| `P2002` | Unique constraint violation | Duplicate email, API key, etc.         |
+| `P2003` | Foreign key constraint      | Referenced record doesn't exist        |
+| `P2025` | Record not found            | Update/delete of non-existent record   |
+| `P2018` | Required relation not found | Connect to non-existent related record |

 ### ClickHouse Errors

@@ -636,11 +667,11 @@ try {

 **ClickHouse error types:**

-| Error Type      | Discriminator           | Meaning                      | Solution                                           |
-| --------------- | ----------------------- | ---------------------------- | -------------------------------------------------- |
-| `MEMORY_LIMIT`  | "memory limit exceeded" | Query used too much memory   | Use more specific filters or shorter time range    |
-| `OVERCOMMIT`    | "OvercommitTracker"     | Memory overcommit limit hit  | Reduce query complexity or result set size         |
-| `TIMEOUT`       | "Timeout", "timed out"  | Query took too long          | Add filters, reduce time range, or optimize query  |
+| Error Type     | Discriminator           | Meaning                     | Solution                                          |
+| -------------- | ----------------------- | --------------------------- | ------------------------------------------------- |
+| `MEMORY_LIMIT` | "memory limit exceeded" | Query used too much memory  | Use more specific filters or shorter time range   |
+| `OVERCOMMIT`   | "OvercommitTracker"     | Memory overcommit limit hit | Reduce query complexity or result set size        |
+| `TIMEOUT`      | "Timeout", "timed out"  | Query took too long         | Add filters, reduce time range, or optimize query |

 **ClickHouse retries:**

@@ -648,13 +679,13 @@ ClickHouse queries automatically retry network errors (socket hang up) with expo

 ```typescript
 // In packages/shared/src/env.ts
-LANGFUSE_CLICKHOUSE_QUERY_MAX_ATTEMPTS: z.coerce.number().positive().default(3)
+LANGFUSE_CLICKHOUSE_QUERY_MAX_ATTEMPTS: z.coerce.number().positive().default(3);
 ```

 ---

 **Related Files:**

- [../AGENTS.md](../AGENTS.md) - Main backend development guidelines
+- [../SKILL.md](../SKILL.md) - Main backend development guidelines
 - [architecture-overview.md](architecture-overview.md) - System architecture
 - [configuration.md](configuration.md) - Environment variable configuration
@@ -116,7 +116,9 @@ const enforceUserIsAuthedAndProjectMember = t.middleware(async (opts) => {

  const projectId = parsedInput.data.projectId;
  const sessionProject = ctx.session.user.organizations
-    .flatMap((org) => org.projects.map((project) => ({ ...project, organization: org })))
+    .flatMap((org) =>
+      org.projects.map((project) => ({ ...project, organization: org })),
+    )
    .find((project) => project.id === projectId);

  if (!sessionProject) {
@@ -164,7 +166,9 @@ const enforceIsAuthedAndOrgMember = t.middleware(async (opts) => {
  }

  const orgId = result.data.orgId;
-  const sessionOrg = ctx.session.user.organizations.find((org) => org.id === orgId);
+  const sessionOrg = ctx.session.user.organizations.find(
+    (org) => org.id === orgId,
+  );

  if (!sessionOrg && ctx.session.user.admin !== true) {
    throw new TRPCError({
@@ -179,7 +183,8 @@ const enforceIsAuthedAndOrgMember = t.middleware(async (opts) => {
        ...ctx.session,
        user: ctx.session.user,
        orgId: orgId,
-        orgRole: ctx.session.user.admin === true ? Role.OWNER : sessionOrg!.role,
+        orgRole:
+          ctx.session.user.admin === true ? Role.OWNER : sessionOrg!.role,
      },
    },
  });
@@ -221,7 +226,8 @@ const enforceTraceAccess = t.middleware(async (opts) => {
  if (!trace.public && !sessionProject && ctx.session?.user?.admin !== true) {
    throw new TRPCError({
      code: "UNAUTHORIZED",
-      message: "User is not a member of this project and this trace is not public",
+      message:
+        "User is not a member of this project and this trace is not public",
    });
  }

@@ -348,6 +354,7 @@ export default withMiddlewares({
 ### createAuthedProjectAPIRoute Pattern

 Factory function for authenticated public API routes with:
+
 - Authentication (Basic auth or Admin API key)
 - Rate limiting
 - Input/output validation (Zod)
@@ -355,17 +362,21 @@ Factory function for authenticated public API routes with:

 ```typescript
 export const createAuthedProjectAPIRoute = <TQuery, TBody, TResponse>(
-  routeConfig: RouteConfig<TQuery, TBody, TResponse>
+  routeConfig: RouteConfig<TQuery, TBody, TResponse>,
 ) => {
  return async (req: NextApiRequest, res: NextApiResponse) => {
    // 1. Authentication (verifyAuth)
-    const auth = await verifyAuth(req, routeConfig.isAdminApiKeyAuthAllowed || false);
+    const auth = await verifyAuth(
+      req,
+      routeConfig.isAdminApiKeyAuthAllowed || false,
+    );

    // 2. Rate limiting
-    const rateLimitResponse = await RateLimitService.getInstance().rateLimitRequest(
-      auth.scope,
-      routeConfig.rateLimitResource || "public-api"
-    );
+    const rateLimitResponse =
+      await RateLimitService.getInstance().rateLimitRequest(
+        auth.scope,
+        routeConfig.rateLimitResource || "public-api",
+      );

    if (rateLimitResponse?.isRateLimited()) {
      return rateLimitResponse.sendRestResponseIfLimited(res);
@@ -474,15 +485,20 @@ Public APIs use **Basic Auth** with API keys:

 ```typescript
 async function verifyBasicAuth(authHeader: string | undefined) {
-  const regularAuth = await new ApiAuthService(prisma, redis)
-    .verifyAuthHeaderAndReturnScope(authHeader);
+  const regularAuth = await new ApiAuthService(
+    prisma,
+    redis,
+  ).verifyAuthHeaderAndReturnScope(authHeader);

  if (!regularAuth.validKey) {
    throw { status: 401, message: regularAuth.error };
  }

  if (regularAuth.scope.accessLevel !== "project") {
-    throw { status: 401, message: "Access denied - need basic auth with secret key" };
+    throw {
+      status: 401,
+      message: "Access denied - need basic auth with secret key",
+    };
  }

  return regularAuth;
@@ -499,7 +515,10 @@ async function verifyAdminApiKeyAuth(req: NextApiRequest) {
  // 3. x-langfuse-project-id: <project-id>

  if (env.NEXT_PUBLIC_LANGFUSE_CLOUD_REGION) {
-    throw { status: 403, message: "Admin API key auth not available on Langfuse Cloud" };
+    throw {
+      status: 403,
+      message: "Admin API key auth not available on Langfuse Cloud",
+    };
  }

  const adminApiKey = env.ADMIN_API_KEY;
@@ -508,8 +527,14 @@ async function verifyAdminApiKeyAuth(req: NextApiRequest) {

  // Timing-safe comparison
  const isValid =
-    crypto.timingSafeEqual(Buffer.from(bearerToken), Buffer.from(adminApiKey)) &&
-    crypto.timingSafeEqual(Buffer.from(adminApiKeyHeader), Buffer.from(adminApiKey));
+    crypto.timingSafeEqual(
+      Buffer.from(bearerToken),
+      Buffer.from(adminApiKey),
+    ) &&
+    crypto.timingSafeEqual(
+      Buffer.from(adminApiKeyHeader),
+      Buffer.from(adminApiKey),
+    );

  if (!isValid) throw { status: 401, message: "Invalid admin API key" };

@@ -635,6 +660,7 @@ return opentelemetry.context.with(ctx, async () => {
 ```

 **Baggage includes:**
+
 - `userId` - User ID from session
 - `projectId` - Project ID from input
 - `headers` - Request headers for trace propagation
@@ -676,7 +702,9 @@ const baseProcedure = withOtelTracingProcedure.use(withErrorHandling);
 const authedProcedure = baseProcedure.use(enforceUserIsAuthed);

 // Add project scoping
-const projectProcedure = authedProcedure.use(enforceUserIsAuthedAndProjectMember);
+const projectProcedure = authedProcedure.use(
+  enforceUserIsAuthedAndProjectMember,
+);
 ```

 ### Using Procedures in Routers
@@ -692,7 +720,7 @@ export const tracesRouter = createTRPCRouter({
        projectId: z.string(),
        page: z.number().optional(),
        limit: z.number().optional(),
-      })
+      }),
    )
    .query(async ({ input, ctx }) => {
      // ctx.session.projectId is guaranteed to exist
@@ -713,7 +741,7 @@ export const tracesRouter = createTRPCRouter({
        traceId: z.string(),
        projectId: z.string(),
        timestamp: z.date().nullish(),
-      })
+      }),
    )
    .query(async ({ input, ctx }) => {
      // ctx.trace is guaranteed to exist (fetched by middleware)
@@ -729,11 +757,12 @@ Middleware executes in the order it's chained:

 ```typescript
 protectedProjectProcedure
-  .use(withErrorHandling)              // 1. Wraps entire execution
-  .use(enforceUserIsAuthed)            // 2. Validates session exists
-  .use(enforceUserIsAuthedAndProjectMember)  // 3. Validates project membership
-  .input(schema)                       // 4. Validates input
-  .query(async ({ input, ctx }) => {   // 5. Executes query
+  .use(withErrorHandling) // 1. Wraps entire execution
+  .use(enforceUserIsAuthed) // 2. Validates session exists
+  .use(enforceUserIsAuthedAndProjectMember) // 3. Validates project membership
+  .input(schema) // 4. Validates input
+  .query(async ({ input, ctx }) => {
+    // 5. Executes query
    // ...
  });
 ```
@@ -744,22 +773,21 @@ Each middleware can enrich the context:

 ```typescript
 // After enforceUserIsAuthed:
-ctx.session.user // NonNullable<User>
+ctx.session.user; // NonNullable<User>

 // After enforceUserIsAuthedAndProjectMember:
-ctx.session.projectId   // string
-ctx.session.projectRole // Role (OWNER | ADMIN | MEMBER | VIEWER)
-ctx.session.orgId       // string
-ctx.session.orgRole     // Role
+ctx.session.projectId; // string
+ctx.session.projectRole; // Role (OWNER | ADMIN | MEMBER | VIEWER)
+ctx.session.orgId; // string
+ctx.session.orgRole; // Role

 // After enforceTraceAccess:
-ctx.trace // TraceRecord (pre-fetched)
+ctx.trace; // TraceRecord (pre-fetched)
 ```

 ---

 **Related Files:**

- [../AGENTS.md](../AGENTS.md) - Main backend development guidelines
+- [../SKILL.md](../SKILL.md) - Main backend development guidelines
 - [architecture-overview.md](architecture-overview.md) - System architecture
- [../AGENTS.md](../AGENTS.md) - Error handling patterns and traceException guidance
@@ -7,6 +7,7 @@ Complete guide to routing and separation of concerns in Langfuse's Next.js + tRP
 - [Architecture Overview](#architecture-overview)
 - [tRPC Routers](#trpc-routers)
 - [Public REST API Routes](#public-rest-api-routes)
+- [Fern API Definitions](#fern-api-definitions)
 - [Service Layer](#service-layer)
 - [Repository Layer](#repository-layer)
 - [Separation of Concerns](#separation-of-concerns)
@@ -48,6 +49,7 @@ Langfuse uses a **layered architecture** with clear separation of concerns:
 ### Key Principles

 **Entry Points (Routes/Procedures):**
+
 - ✅ Define routing and procedure signatures
 - ✅ Handle authentication/authorization (via middleware)
 - ✅ Validate input (Zod schemas)
@@ -55,12 +57,14 @@ Langfuse uses a **layered architecture** with clear separation of concerns:
 - ✅ Return responses

 **Entry Points should NEVER:**
+
 - ❌ Contain business logic
 - ❌ Access database directly
 - ❌ Perform complex data transformations
 - ❌ Make direct repository calls (use services)

 **Services:**
+
 - ✅ Contain business logic
 - ✅ Orchestrate multiple operations
 - ✅ Call repositories or Prisma/ClickHouse
@@ -68,6 +72,7 @@ Langfuse uses a **layered architecture** with clear separation of concerns:
 - ❌ Should NOT know about HTTP, tRPC, or request/response objects

 **Repositories:**
+
 - ✅ Complex database queries
 - ✅ Data transformation (DB → domain models)
 - ✅ ClickHouse query builders
@@ -88,7 +93,10 @@ tRPC routers define type-safe procedures for the internal UI. Each router groups

 ```typescript
 import { z } from "zod/v4";
-import { createTRPCRouter, protectedProjectProcedure } from "@/src/server/api/trpc";
+import {
+  createTRPCRouter,
+  protectedProjectProcedure,
+} from "@/src/server/api/trpc";
 import { paginationZod, singleFilter, orderBy } from "@langfuse/shared";
 import {
  getScoresUiTable,
@@ -187,6 +195,7 @@ export const scoresRouter = createTRPCRouter({
 ```

 **Key Points:**
+
 - Use appropriate procedure type (`protectedProjectProcedure`, `authenticatedProcedure`, etc.)
 - Define input schema with Zod (`.input()`)
 - Use `.query()` for reads, `.mutation()` for writes
@@ -255,6 +264,7 @@ web/src/pages/api/public/
 ```

 **Dynamic routes:**
+
 - `[param].ts` → Single dynamic segment (e.g., `/api/public/scores/[scoreId].ts`)
 - `[...param].ts` → Catch-all route (e.g., `/api/public/[...path].ts`)

@@ -345,12 +355,63 @@ export default withMiddlewares({
 ```

 **Key Points:**
+
 - Use `withMiddlewares` for all public API routes (provides CORS, error handling, OpenTelemetry)
 - Use `createAuthedProjectAPIRoute` for authenticated endpoints (handles auth, rate limiting, validation)
 - Define separate handlers for each HTTP method
 - Input/output validated with Zod schemas
 - Delegate to services for business logic

+### Versioned API Type Location
+
+When a feature has versioned public API types, place them in `packages/shared/`
+under the feature's `interfaces/api/` folder, one subdirectory per version.
+The scores feature (`packages/shared/src/features/scores/interfaces/api/`) is
+the canonical example — the `GetScoresQueryV1` / `GetScoresResponseV1` symbols
+imported from `@langfuse/shared` originate there.
+
+Do not create a flat file (e.g. `<domain>-api-v2.ts`) and do not place
+versioned types in `web/src/features/public-api/types/`.
+
+```
+packages/shared/src/features/<domain>/interfaces/api/
+├── v1/
+│   ├── schemas.ts     # Zod schemas for request/response shapes
+│   ├── endpoints.ts   # Composed request/response types
+│   └── validation.ts  # Cross-field validation helpers
+├── v2/
+│   └── ...
+└── vN/
+    └── ...
+```
+
+### Fern API Definitions
+
+When modifying public API types in `web/src/features/public-api/types/` or under
+`packages/shared/src/features/<domain>/interfaces/api/`, update the matching
+Fern API definitions in `fern/apis/server/definition/`.
+
+**Zod to Fern Type Mapping:**
+
+| Zod Type       | Fern Type               | Example                                                   |
+| -------------- | ----------------------- | --------------------------------------------------------- |
+| `.nullish()`   | `optional<nullable<T>>` | `z.string().nullish()` -> `optional<nullable<string>>`    |
+| `.nullable()`  | `nullable<T>`           | `z.string().nullable()` -> `nullable<string>`             |
+| `.optional()`  | `optional<T>`           | `z.string().optional()` -> `optional<string>`             |
+| Always present | `T`                     | `z.string()` -> `string`                                  |
+
+Add a source comment at the top of each Fern type that references the
+TypeScript source:
+
+```yaml
+# Source: web/src/features/public-api/types/traces.ts - APITrace
+Trace:
+  properties:
+    id: string
+    name:
+      type: nullable<string>
+```
+
 ### Simple Public Routes

 For routes that don't need authentication:
@@ -433,6 +494,7 @@ export class ScoresApiService {
 ```

 **Key Points:**
+
 - Services contain business logic, not routing logic
 - Services should NOT import tRPC or Next.js types
 - Services can call repositories, Prisma, ClickHouse directly
@@ -442,6 +504,7 @@ export class ScoresApiService {
 ### Where to Put Services

 **Feature-specific services:**
+
 ```
 web/src/features/
 ├── datasets/
@@ -456,6 +519,7 @@ web/src/features/
 ```

 **Shared services:**
+
 ```
 packages/shared/src/server/services/
 ├── SlackService.ts
@@ -497,7 +561,7 @@ import { convertClickhouseToDomain } from "./traces_converters";
 */
 export const getTracesByIds = async (
  projectId: string,
-  traceIds: string[]
+  traceIds: string[],
 ): Promise<TraceRecordReadType[]> => {
  const rows = await queryClickhouse<TraceRecordReadType>({
    query: `
@@ -519,7 +583,7 @@ export const getTracesByIds = async (
 * Upsert trace to ClickHouse
 */
 export const upsertTrace = async (
-  trace: TraceRecordInsertType
+  trace: TraceRecordInsertType,
 ): Promise<void> => {
  await upsertClickhouse({
    table: "traces",
@@ -536,6 +600,7 @@ export const upsertTrace = async (
 ```

 **Key Points:**
+
 - Use `queryClickhouse` for SELECT queries
 - Use `upsertClickhouse` for INSERT/UPDATE
 - Use `commandClickhouse` for DDL (ALTER TABLE, etc.)
@@ -546,12 +611,14 @@ export const upsertTrace = async (
 ### When to Use Repositories

 ✅ **Use repositories for:**
+
 - Complex ClickHouse queries with CTEs, joins, aggregations
 - Queries used in multiple places (DRY principle)
 - Data transformation from DB types to domain models
 - Streaming large result sets

 ❌ **Use direct Prisma/ClickHouse for:**
+
 - Simple CRUD operations
 - One-off queries
 - Prototyping (can refactor to repository later)
@@ -645,9 +712,7 @@ export async function createScoreWithValidation({

 ```typescript
 // packages/shared/src/server/repositories/scores.ts
-export const upsertScore = async (
-  score: ScoreInsertType
-): Promise<void> => {
+export const upsertScore = async (score: ScoreInsertType): Promise<void> => {
  // ✅ Pure data access - no business logic
  await upsertClickhouse({
    table: "scores",
@@ -719,6 +784,7 @@ export const scoresRouter = createTRPCRouter({
 ```

 **Why it's bad:**
+
 - Business logic tied to tRPC (can't reuse in public API)
 - Hard to test (need to mock tRPC context)
 - No separation of concerns
@@ -817,9 +883,7 @@ export const upsertScore = async (

 ```typescript
 // ✅ GOOD: Pure data access, no business logic
-export const upsertScore = async (
-  score: ScoreInsertType
-): Promise<void> => {
+export const upsertScore = async (score: ScoreInsertType): Promise<void> => {
  await upsertClickhouse({
    table: "scores",
    records: [score],
@@ -838,7 +902,7 @@ export const upsertScore = async (

 **Related Files:**

- [../AGENTS.md](../AGENTS.md) - Main backend development guidelines
+- [../SKILL.md](../SKILL.md) - Main backend development guidelines
 - [architecture-overview.md](architecture-overview.md) - System architecture
 - [middleware-guide.md](middleware-guide.md) - Middleware patterns
 - [database-patterns.md](database-patterns.md) - Database access patterns
@@ -412,190 +412,19 @@ Repository: "Here's the Prisma query that does that"
 - ❌ Know about HTTP
 - ❌ Make decisions (that's service layer)

-### Repository Template
+### Langfuse Repository Examples

-```typescript
-// repositories/UserRepository.ts
-import { PrismaService } from "@project-lifecycle-portal/database";
-import type { User, Prisma } from "@project-lifecycle-portal/database";
+Use these current Langfuse files as repository templates:

-export class UserRepository {
-  /**
-   * Find user by ID with optimized query
-   */
-  async findById(userId: string): Promise<User | null> {
-    try {
-      return await PrismaService.main.user.findUnique({
-        where: { userID: userId },
-        select: {
-          userID: true,
-          email: true,
-          name: true,
-          isActive: true,
-          roles: true,
-          createdAt: true,
-          updatedAt: true,
-        },
-      });
-    } catch (error) {
-      console.error("[UserRepository] Error finding user by ID:", error);
-      throw new Error(`Failed to find user: ${userId}`);
-    }
-  }
+- PostgreSQL repository with project-scoped filters:
+  `packages/shared/src/server/repositories/comments.ts`
+- ClickHouse repository with project-scoped filters and query helpers:
+  `packages/shared/src/server/repositories/traces.ts`
+- Repository tests:
+  `web/src/__tests__/server/repositories/event-repository.servertest.ts`

-  /**
-   * Find all active users
-   */
-  async findActive(options?: {
-    orderBy?: Prisma.UserOrderByWithRelationInput;
-  }): Promise<User[]> {
-    try {
-      return await PrismaService.main.user.findMany({
-        where: { isActive: true },
-        orderBy: options?.orderBy || { name: "asc" },
-        select: {
-          userID: true,
-          email: true,
-          name: true,
-          roles: true,
-        },
-      });
-    } catch (error) {
-      console.error("[UserRepository] Error finding active users:", error);
-      throw new Error("Failed to find active users");
-    }
-  }
-
-  /**
-   * Find user by email
-   */
-  async findByEmail(email: string): Promise<User | null> {
-    try {
-      return await PrismaService.main.user.findUnique({
-        where: { email },
-      });
-    } catch (error) {
-      console.error("[UserRepository] Error finding user by email:", error);
-      throw new Error(`Failed to find user with email: ${email}`);
-    }
-  }
-
-  /**
-   * Create new user
-   */
-  async create(data: Prisma.UserCreateInput): Promise<User> {
-    try {
-      return await PrismaService.main.user.create({ data });
-    } catch (error) {
-      console.error("[UserRepository] Error creating user:", error);
-      throw new Error("Failed to create user");
-    }
-  }
-
-  /**
-   * Update user
-   */
-  async update(userId: string, data: Prisma.UserUpdateInput): Promise<User> {
-    try {
-      return await PrismaService.main.user.update({
-        where: { userID: userId },
-        data,
-      });
-    } catch (error) {
-      console.error("[UserRepository] Error updating user:", error);
-      throw new Error(`Failed to update user: ${userId}`);
-    }
-  }
-
-  /**
-   * Delete user (soft delete by setting isActive = false)
-   */
-  async delete(userId: string): Promise<User> {
-    try {
-      return await PrismaService.main.user.update({
-        where: { userID: userId },
-        data: { isActive: false },
-      });
-    } catch (error) {
-      console.error("[UserRepository] Error deleting user:", error);
-      throw new Error(`Failed to delete user: ${userId}`);
-    }
-  }
-
-  /**
-   * Check if email exists
-   */
-  async emailExists(email: string): Promise<boolean> {
-    try {
-      const count = await PrismaService.main.user.count({
-        where: { email },
-      });
-      return count > 0;
-    } catch (error) {
-      console.error("[UserRepository] Error checking email exists:", error);
-      throw new Error("Failed to check if email exists");
-    }
-  }
-}
-
-// Export singleton instance
-export const userRepository = new UserRepository();
-```
-
-**Using Repository in Service:**
-
-```typescript
-// services/userService.ts
-import { userRepository } from "../repositories/UserRepository";
-import { ConflictError, NotFoundError } from "../utils/errors";
-
-export class UserService {
-  /**
-   * Create new user with business rules
-   */
-  async createUser(data: {
-    email: string;
-    name: string;
-    roles: string[];
-  }): Promise<User> {
-    // Business rule: Check if email already exists
-    const emailExists = await userRepository.emailExists(data.email);
-    if (emailExists) {
-      throw new ConflictError("Email already exists");
-    }
-
-    // Business rule: Validate roles
-    const validRoles = ["admin", "operations", "user"];
-    const invalidRoles = data.roles.filter(
-      (role) => !validRoles.includes(role),
-    );
-    if (invalidRoles.length > 0) {
-      throw new ValidationError(`Invalid roles: ${invalidRoles.join(", ")}`);
-    }
-
-    // Create user via repository
-    return await userRepository.create({
-      email: data.email,
-      name: data.name,
-      roles: data.roles,
-      isActive: true,
-    });
-  }
-
-  /**
-   * Get user by ID
-   */
-  async getUser(userId: string): Promise<User> {
-    const user = await userRepository.findById(userId);
-
-    if (!user) {
-      throw new NotFoundError(`User not found: ${userId}`);
-    }
-
-    return user;
-  }
-}
-```
+Keep data-access concerns in repositories and business decisions in services.
+Project-scoped queries must include `projectId` or `project_id` filters.

 ---

@@ -803,75 +632,19 @@ class UserService {

 ## Testing Services

-### Unit Tests
+Use `testing-guide.md` for backend test patterns. Prefer current Langfuse tests
+over invented examples:

-```typescript
-// tests/userService.test.ts
-import { UserService } from "../services/userService";
-import { userRepository } from "../repositories/UserRepository";
-import { ConflictError } from "../utils/errors";
-
-// Mock repository
-jest.mock("../repositories/UserRepository");
-
-describe("UserService", () => {
-  let userService: UserService;
-
-  beforeEach(() => {
-    userService = new UserService();
-    jest.clearAllMocks();
-  });
-
-  describe("createUser", () => {
-    it("should create user when email does not exist", async () => {
-      // Arrange
-      const userData = {
-        email: "test@example.com",
-        name: "Test User",
-        roles: ["user"],
-      };
-
-      (userRepository.emailExists as jest.Mock).mockResolvedValue(false);
-      (userRepository.create as jest.Mock).mockResolvedValue({
-        userID: "123",
-        ...userData,
-      });
-
-      // Act
-      const user = await userService.createUser(userData);
-
-      // Assert
-      expect(user).toBeDefined();
-      expect(user.email).toBe(userData.email);
-      expect(userRepository.emailExists).toHaveBeenCalledWith(userData.email);
-      expect(userRepository.create).toHaveBeenCalled();
-    });
-
-    it("should throw ConflictError when email exists", async () => {
-      // Arrange
-      const userData = {
-        email: "existing@example.com",
-        name: "Test User",
-        roles: ["user"],
-      };
-
-      (userRepository.emailExists as jest.Mock).mockResolvedValue(true);
-
-      // Act & Assert
-      await expect(userService.createUser(userData)).rejects.toThrow(
-        ConflictError,
-      );
-      expect(userRepository.create).not.toHaveBeenCalled();
-    });
-  });
-});
-```
+- Repository tests:
+  `web/src/__tests__/server/repositories/event-repository.servertest.ts`
+- Pure service unit tests:
+  `web/src/__tests__/server/unit/`

 ---

 **Related Files:**

- [../AGENTS.md](../AGENTS.md) - Main guide
+- [../SKILL.md](../SKILL.md) - Main guide
 - [routing-and-controllers.md](routing-and-controllers.md) - Controllers that use services
 - [database-patterns.md](database-patterns.md) - Prisma and repository patterns
 - [testing-guide.md](testing-guide.md) - Testing service and repository code
@@ -4,26 +4,66 @@ Complete guide to testing Langfuse backend services across web, worker, and shar

 ## Table of Contents

+- [Key Testing Principles](#key-testing-principles)
 - [Test Types Overview](#test-types-overview)
 - [Integration Tests (Public API)](#integration-tests-public-api)
 - [Service-Level Tests (Repository/Service)](#service-level-tests-repositoryservice)
 - [tRPC Tests (Procedure Testing)](#trpc-tests-procedure-testing)
 - [Worker Tests (Queue Processing)](#worker-tests-queue-processing)
- [Key Testing Principles](#key-testing-principles)
 - [Running Tests](#running-tests)

 ---

+## Key Testing Principles
+
+### General Principles
+
+1. **Test Isolation**: Each test should be independent and runnable in any order
+2. **Unique IDs**: Use `randomUUID()` or unique project IDs to avoid test interference
+3. **Cleanup**: Always clean up test data in service tests (or use unique project IDs)
+4. **Avoid Global Resets**: Prefer scoped cleanup or unique project IDs over global reset helpers
+5. **Flags and Fallbacks**: When code branches on env flags, feature flags, or fallback data paths, test both branches and ensure fixtures are written to the same store the branch reads from
+
+### By Test Type
+
+| Test Type       | Key Principles                                                                   |
+| --------------- | -------------------------------------------------------------------------------- |
+| **Integration** | Test HTTP endpoints, validate status codes and response shapes                   |
+| **tRPC**        | Use `createInnerTRPCContext` and `appRouter.createCaller`, test auth/permissions |
+| **Service**     | Test individual functions with isolated data, always cleanup                     |
+| **Worker**      | Use vitest, test streams with async iteration, test filtering logic              |
+
+### Test Data Management
+
+```typescript
+// ✅ GOOD: Use unique IDs
+const projectId = randomUUID();
+const traceId = randomUUID();
+
+// ✅ GOOD: Cleanup in service tests
+afterAll(async () => {
+  await prisma.model.delete({ where: { id: modelId } });
+});
+
+// ✅ GOOD: Use unique projects (no cleanup needed)
+const { projectId } = await createOrgProjectAndApiKey();
+
+// ❌ BAD: Shared test data between tests
+const projectId = "7a88fb47-b4e2-43b8-a06c-a5ce950dc53a";
+```
+
+---
+
 ## Test Types Overview

 Langfuse uses multiple testing strategies for different layers:

-| Test Type | Framework | Location | Purpose |
-|-----------|-----------|----------|---------|
-| Integration | Jest | `web/src/__tests__/async/` | Full API endpoint testing |
-| tRPC | Jest | `web/src/__tests__/async/` | tRPC procedure testing with auth |
-| Service | Jest | `web/src/__tests__/async/repositories/` | Repository/service function testing |
-| Worker | Vitest | `worker/src/__tests__/` | Queue processors and streams |
+| Test Type   | Framework | Location                                 | Purpose                             |
+| ----------- | --------- | ---------------------------------------- | ----------------------------------- |
+| Integration | Vitest    | `web/src/__tests__/server/`              | Full API endpoint testing           |
+| tRPC        | Vitest    | `web/src/__tests__/server/`              | tRPC procedure testing with auth    |
+| Service     | Vitest    | `web/src/__tests__/server/repositories/` | Repository/service function testing |
+| Worker      | Vitest    | `worker/src/__tests__/`                  | Queue processors and streams        |

 ---

@@ -31,7 +71,7 @@ Langfuse uses multiple testing strategies for different layers:

 Test full REST API endpoints end-to-end using HTTP requests.

-**File location:** `web/src/__tests__/async/datasets-api.servertest.ts`
+**File location:** `web/src/__tests__/server/datasets-api.servertest.ts`

 ```typescript
 import { makeZodVerifiedAPICall } from "../helpers";
@@ -63,6 +103,7 @@ describe("Dataset API", () => {
 ```

 **Key Points:**
+
 - Uses `makeZodVerifiedAPICall` for type-safe API testing
 - Tests HTTP status codes and response validation
 - Tests both success and error cases
@@ -73,7 +114,7 @@ describe("Dataset API", () => {

 Test individual repository/service functions with isolated data.

-**File location:** `web/src/__tests__/async/repositories/event-repository.servertest.ts`
+**File location:** `web/src/__tests__/server/repositories/event-repository.servertest.ts`

 ```typescript
 import {
@@ -123,7 +164,9 @@ describe("Event Repository Tests", () => {
    // Test the service function
    const result = await getObservationsWithModelDataFromEventsTable({
      projectId,
-      filter: [{ type: "string", column: "id", operator: "=", value: generationId }],
+      filter: [
+        { type: "string", column: "id", operator: "=", value: generationId },
+      ],
      limit: 1000,
      offset: 0,
    });
@@ -163,18 +206,24 @@ describe("Event Repository Tests", () => {
    const result = await getObservationsWithModelDataFromEventsTable({
      projectId,
      filter: [
-        { type: "stringOptions", column: "type", operator: "any of", value: ["GENERATION"] }
+        {
+          type: "stringOptions",
+          column: "type",
+          operator: "any of",
+          value: ["GENERATION"],
+        },
      ],
      limit: 1000,
      offset: 0,
    });

-    expect(result.every(o => o.type === "GENERATION")).toBe(true);
+    expect(result.every((o) => o.type === "GENERATION")).toBe(true);
  });
 });
 ```

 **Key Points:**
+
 - Tests service/repository functions directly
 - Uses ClickHouse and Prisma test data
 - Always cleanup test data after tests
@@ -186,7 +235,7 @@ describe("Event Repository Tests", () => {

 Test tRPC procedures with caller pattern and auth context.

-**File location:** `web/src/__tests__/async/automations-trpc.servertest.ts`
+**File location:** `web/src/__tests__/server/automations-trpc.servertest.ts`

 ```typescript
 import { appRouter } from "@/src/server/api/root";
@@ -205,16 +254,20 @@ async function prepare() {
    user: {
      id: "user-1",
      name: "Demo User",
-      organizations: [{
-        id: org.id,
-        name: org.name,
-        role: "OWNER",
-        projects: [{
-          id: project.id,
-          role: "ADMIN",
-          name: project.name,
-        }],
-      }],
+      organizations: [
+        {
+          id: org.id,
+          name: org.name,
+          role: "OWNER",
+          projects: [
+            {
+              id: project.id,
+              role: "ADMIN",
+              name: project.name,
+            },
+          ],
+        },
+      ],
    },
  };

@@ -287,13 +340,17 @@ describe("automations trpc", () => {
      ...session,
      user: {
        ...session.user!,
-        organizations: [{
-          ...session.user!.organizations[0],
-          projects: [{
-            ...session.user!.organizations[0].projects[0],
-            role: "VIEWER", // VIEWER can't create automations
-          }],
-        }],
+        organizations: [
+          {
+            ...session.user!.organizations[0],
+            projects: [
+              {
+                ...session.user!.organizations[0].projects[0],
+                role: "VIEWER", // VIEWER can't create automations
+              },
+            ],
+          },
+        ],
      },
    };

@@ -325,6 +382,7 @@ describe("automations trpc", () => {
 ```

 **Key Points:**
+
 - Uses `prepare()` helper to set up test context
 - Creates authenticated caller with `appRouter.createCaller`
 - Tests both success and permission error cases
@@ -457,6 +515,7 @@ describe("batch export test suite", () => {
 ```

 **Key Points:**
+
 - Uses vitest (not Jest) for worker tests
 - Tests stream functions with async iteration
 - Creates isolated test data per test
@@ -464,92 +523,21 @@ describe("batch export test suite", () => {

 ---

-## Key Testing Principles
-
-### General Principles
-
-1. **Test Isolation**: Each test should be independent and runnable in any order
-2. **Unique IDs**: Use `randomUUID()` or unique project IDs to avoid test interference
-3. **Cleanup**: Always clean up test data in service tests (or use unique project IDs)
-4. **Avoid Global Resets**: Prefer scoped cleanup or unique project IDs over global reset helpers
-
-### By Test Type
-
-| Test Type | Key Principles |
-|-----------|----------------|
-| **Integration** | Test HTTP endpoints, validate status codes and response shapes |
-| **tRPC** | Use `createInnerTRPCContext` and `appRouter.createCaller`, test auth/permissions |
-| **Service** | Test individual functions with isolated data, always cleanup |
-| **Worker** | Use vitest, test streams with async iteration, test filtering logic |
-
-### Test Data Management
-
-```typescript
-// ✅ GOOD: Use unique IDs
-const projectId = randomUUID();
-const traceId = randomUUID();
-
-// ✅ GOOD: Cleanup in service tests
-afterAll(async () => {
-  await prisma.model.delete({ where: { id: modelId } });
-});
-
-// ✅ GOOD: Use unique projects (no cleanup needed)
-const { projectId } = await createOrgProjectAndApiKey();
-
-// ❌ BAD: Shared test data between tests
-const projectId = "7a88fb47-b4e2-43b8-a06c-a5ce950dc53a";
-```
-
---
-
 ## Running Tests

-### Web Tests (Jest)
+Use the nearest package `AGENTS.md` as the source of truth for current test
+commands.

-```bash
-# Run all tests
-pnpm test
+Common targeted forms:

-# Run sync tests
-pnpm test-sync
-
-# Run async tests
-pnpm test -- --testPathPattern="async"
-
-# Run specific test file
-pnpm test -- --testPathPattern="datasets-api"
-
-# Run specific test
-pnpm test -- --testPathPattern="datasets-api" --testNamePattern="should create dataset"
-```
-
-### Worker Tests (Vitest)
-
-```bash
-# Run all worker tests
-pnpm run test --filter=worker
-
-# Run specific test file
-pnpm run test --filter=worker -- batchExport
-
-# Run specific test
-pnpm run test --filter=worker -- batchExport -t "should export observations"
-```
-
-### Coverage
-
-```bash
-# Web coverage
-pnpm test -- --coverage
-
-# Worker coverage
-pnpm run test --filter=worker -- --coverage
-```
+- Web server tests: `pnpm --filter web run test <file-or-pattern>`
+- Web client tests: `pnpm --filter web run test-client <file-or-pattern>`
+- Worker tests: `pnpm --filter worker run test <file-or-pattern>`

 ---

 **Related Files:**
- [../AGENTS.md](../AGENTS.md) - Main backend guidelines
+
+- [../SKILL.md](../SKILL.md) - Main backend guidelines
 - [architecture-overview.md](architecture-overview.md) - Architecture patterns
 - [services-and-repositories.md](services-and-repositories.md) - Service and repository examples
@@ -12,23 +12,24 @@ npx skills add ClickHouse/clickhouse-agent-skills

 **28 atomic rules** organized by prefix:

-| Prefix | Count | Coverage |
-|--------|-------|----------|
-| `schema-pk-*` | 4 | PRIMARY KEY selection, cardinality ordering |
-| `schema-types-*` | 5 | Data types, LowCardinality, Nullable |
-| `schema-partition-*` | 4 | Partitioning strategy, lifecycle management |
-| `schema-json-*` | 1 | JSON type usage |
-| `query-join-*` | 5 | JOIN algorithms, filtering, alternatives |
-| `query-index-*` | 1 | Data skipping indices |
-| `query-mv-*` | 2 | Incremental and refreshable MVs |
-| `insert-batch-*` | 1 | Batch sizing (10K-100K rows) |
-| `insert-async-*` | 2 | Async inserts, data formats |
-| `insert-mutation-*` | 2 | Mutation avoidance |
-| `insert-optimize-*` | 1 | OPTIMIZE FINAL avoidance |
+| Prefix               | Count | Coverage                                    |
+| -------------------- | ----- | ------------------------------------------- |
+| `schema-pk-*`        | 4     | PRIMARY KEY selection, cardinality ordering |
+| `schema-types-*`     | 5     | Data types, LowCardinality, Nullable        |
+| `schema-partition-*` | 4     | Partitioning strategy, lifecycle management |
+| `schema-json-*`      | 1     | JSON type usage                             |
+| `query-join-*`       | 5     | JOIN algorithms, filtering, alternatives    |
+| `query-index-*`      | 1     | Data skipping indices                       |
+| `query-mv-*`         | 2     | Incremental and refreshable MVs             |
+| `insert-batch-*`     | 1     | Batch sizing (10K-100K rows)                |
+| `insert-async-*`     | 2     | Async inserts, data formats                 |
+| `insert-mutation-*`  | 2     | Mutation avoidance                          |
+| `insert-optimize-*`  | 1     | OPTIMIZE FINAL avoidance                    |

 ## Trigger Phrases

 This skill activates when you:
+
 - "Create a table for..."
 - "Optimize this query..."
 - "Design a schema for..."
@@ -38,13 +39,13 @@ This skill activates when you:

 ## Files

-| File | Purpose |
-|------|---------|
-| `SKILL.md` | Quick reference and decision frameworks |
-| `AGENTS.md` | Complete rule reference (auto-generated) |
-| `rules/*.md` | Individual rule definitions |
+| File         | Purpose                                                  |
+| ------------ | -------------------------------------------------------- |
+| `SKILL.md`   | Review workflow, quick reference, and rule-selection entrypoint |
+| `rules/*.md` | Individual rule definitions                              |

 ## Related Documentation

 All rules link to official ClickHouse documentation:
+
 - [ClickHouse Best Practices](https://clickhouse.com/docs/best-practices)
@@ -25,9 +25,20 @@ Comprehensive guidance for ClickHouse covering schema design, query optimization

 **Why rules take priority:** ClickHouse has specific behaviors (columnar storage, sparse indexes, merge tree mechanics) where general database intuition can be misleading. The rules encode validated, ClickHouse-specific guidance.

-### For Formal Reviews
+## Langfuse-Specific Rules

-When performing a formal review of schemas, queries, or data ingestion:
+- Use `packages/shared/src/server/queries/clickhouse-sql/event-query-builder.ts`
+  for queries against the `events` table. Do not hand-roll `events` SQL unless
+  you first confirm the query builder cannot express the query.
+- Never use `FINAL` on the `events` table; it is designed so `FINAL` is not
+  required and the keyword hurts performance.
+- Any migration in `packages/shared/clickhouse/migrations/clustered/**` with
+  more than one `ALTER` on the same table must end every metadata `ALTER`
+  (`ADD/DROP/MODIFY COLUMN`, `ADD/DROP INDEX`) with `SETTINGS alter_sync = 2`,
+  and every mutation-creating `ALTER` (`MATERIALIZE …`, `UPDATE`, `DELETE`)
+  with `SETTINGS mutations_sync = 2`. The matching `unclustered/` file runs
+  against plain `MergeTree` and does not need (and should not duplicate)
+  these settings.

 ---

@@ -48,11 +59,13 @@ When performing a formal review of schemas, queries, or data ingestion:
 9. `rules/schema-partition-lifecycle.md` - Partitioning purpose

 **Check for:**
+
 - [ ] PRIMARY KEY / ORDER BY column order (low-to-high cardinality)
 - [ ] Data types match actual data ranges
 - [ ] LowCardinality applied to appropriate string columns
 - [ ] Partition key cardinality bounded (100-1,000 values)
 - [ ] ReplacingMergeTree has version column if used
+- [ ] Clustered migration files with multiple ALTERs on the same table use `SETTINGS alter_sync = 2` (metadata) and `SETTINGS mutations_sync = 2` (`MATERIALIZE …`, `UPDATE`, `DELETE`); unclustered mirror has none

 ### For Query Reviews (SELECT, JOIN, aggregations)

@@ -65,6 +78,7 @@ When performing a formal review of schemas, queries, or data ingestion:
 5. `rules/schema-pk-filter-on-orderby.md` - Filter alignment with ORDER BY

 **Check for:**
+
 - [ ] Filters use ORDER BY prefix columns
 - [ ] JOINs filter tables before joining (not after)
 - [ ] Correct JOIN algorithm for table sizes
@@ -81,6 +95,7 @@ When performing a formal review of schemas, queries, or data ingestion:
 5. `rules/insert-optimize-avoid-final.md` - OPTIMIZE TABLE risks

 **Check for:**
+
 - [ ] Batch size 10K-100K rows per INSERT
 - [ ] No ALTER TABLE UPDATE for frequent changes
 - [ ] ReplacingMergeTree or CollapsingMergeTree for update patterns
@@ -117,19 +132,19 @@ Structure your response as follows:

 ## Rule Categories by Priority

-| Priority | Category | Impact | Prefix | Rule Count |
-|----------|----------|--------|--------|------------|
-| 1 | Primary Key Selection | CRITICAL | `schema-pk-` | 4 |
-| 2 | Data Type Selection | CRITICAL | `schema-types-` | 5 |
-| 3 | JOIN Optimization | CRITICAL | `query-join-` | 5 |
-| 4 | Insert Batching | CRITICAL | `insert-batch-` | 1 |
-| 5 | Mutation Avoidance | CRITICAL | `insert-mutation-` | 2 |
-| 6 | Partitioning Strategy | HIGH | `schema-partition-` | 4 |
-| 7 | Skipping Indices | HIGH | `query-index-` | 1 |
-| 8 | Materialized Views | HIGH | `query-mv-` | 2 |
-| 9 | Async Inserts | HIGH | `insert-async-` | 2 |
-| 10 | OPTIMIZE Avoidance | HIGH | `insert-optimize-` | 1 |
-| 11 | JSON Usage | MEDIUM | `schema-json-` | 1 |
+| Priority | Category              | Impact   | Prefix              | Rule Count |
+| -------- | --------------------- | -------- | ------------------- | ---------- |
+| 1        | Primary Key Selection | CRITICAL | `schema-pk-`        | 4          |
+| 2        | Data Type Selection   | CRITICAL | `schema-types-`     | 5          |
+| 3        | JOIN Optimization     | CRITICAL | `query-join-`       | 5          |
+| 4        | Insert Batching       | CRITICAL | `insert-batch-`     | 1          |
+| 5        | Mutation Avoidance    | CRITICAL | `insert-mutation-`  | 2          |
+| 6        | Partitioning Strategy | HIGH     | `schema-partition-` | 4          |
+| 7        | Skipping Indices      | HIGH     | `query-index-`      | 1          |
+| 8        | Materialized Views    | HIGH     | `query-mv-`         | 2          |
+| 9        | Async Inserts         | HIGH     | `insert-async-`     | 2          |
+| 10       | OPTIMIZE Avoidance    | HIGH     | `insert-optimize-`  | 1          |
+| 11       | JSON Usage            | MEDIUM   | `schema-json-`      | 1          |

 ---

@@ -224,11 +239,3 @@ Each rule file in `rules/` contains:
 - **Incorrect example**: Anti-pattern with explanation
 - **Correct example**: Best practice with explanation
 - **Additional context**: Trade-offs, when to apply, references
-
---
-
-## Full Compiled Document
-
-For the complete guide with all rules expanded inline: `AGENTS.md`
-
-Use `AGENTS.md` when you need to check multiple rules quickly without reading individual files.
@@ -23,6 +23,11 @@ feature.
  `clickhouse-best-practices` skill.
 - If the review touches backend code, also use the shared
  `backend-dev-guidelines` skill where relevant.
+- If the change accepts a user-supplied URL, adds outbound HTTP, introduces a
+  new integration, or touches secrets, RBAC, or redirect handling, also use
+  the shared [`security-review`](../security-review/SKILL.md) skill. Run its
+  [`references/checklist.md`](../security-review/references/checklist.md)
+  before signoff.

 ## Review Priorities

@@ -41,6 +41,22 @@ This is the canonical shared review checklist for Langfuse.
  - `top-banner-offset` / `pt-banner-offset` - For sticky/fixed/absolute positioning and padding
  - `h-screen-with-banner` / `min-h-screen-with-banner` - For full-height containers accounting for banners

+## Security
+
+- For changes that accept a user-supplied URL, host, `endpoint`, `baseURL`,
+  or webhook target, or that issue a new outbound HTTP request, run the
+  shared [`security-review`](../../security-review/SKILL.md) skill and treat
+  its [`outbound-url-validation.md`](../../security-review/references/outbound-url-validation.md)
+  defenses (save-time validation + use-time / connection-time validation +
+  redirect-time validation) as required. Plain `fetch(<userUrl>)` or SDK
+  init with `endpoint: <userUrl>` without one of the canonical validators
+  (`validateLlmConnectionBaseURL`, `validateWebhookURL`,
+  `validateBlobStorageEndpoint`, or a new wrapper around
+  `validateOutboundUrlHost`) is a finding.
+- For changes that add a new integration, secret-bearing field, redirect
+  follower, or RBAC scope, run the rest of the
+  [`security-review/references/checklist.md`](../../security-review/references/checklist.md).
+
 ## JavaScript / TypeScript Style

 - use concat instead of spread to avoid stack overflow with large arrays
@@ -0,0 +1,74 @@
+---
+name: datadog-query-recipes
+description: |
+  Langfuse-specific Datadog query recipes for production telemetry research.
+  Use when asked to investigate tenant or project activity, public API endpoint
+  usage, queue consumer behavior, spans, logs, metrics, or ad hoc production
+  questions across prod-us, prod-eu, prod-hipaa, and prod-jp. This skill is for
+  reusable query shapes and measured research; pair it with
+  debug-issue-with-datadog when the task is an incident or root-cause analysis.
+---
+
+# Datadog Query Recipes
+
+Use this skill for Langfuse production telemetry research where the main work is
+finding the right Datadog data path. Keep findings evidence-based and include
+the exact Datadog links or query shapes that support the answer.
+
+## Required Scope
+
+Unless the user explicitly narrows the scope, cover every production
+environment:
+
+- `prod-us`
+- `prod-eu`
+- `prod-hipaa`
+- `prod-jp`
+
+Query both Datadog sites when needed. Default to the EU site for `prod-eu` and
+the US site for the other prod environments, but verify with a small count or
+facet query before concluding an environment has no data.
+
+Before querying live Datadog, load the relevant Datadog MCP guidance for the
+data domain you need: traces, logs, metrics, and visualizations.
+
+## Workflow
+
+1. Identify the entity and signal: tenant ID, org ID, project ID, route, queue,
+   service, error class, or metric.
+2. Read only the relevant reference:
+   - Prod environment/site routing:
+     [`references/environments.md`](references/environments.md)
+   - Public API tenant or legacy endpoint usage:
+     [`references/public-api-tenant-usage.md`](references/public-api-tenant-usage.md)
+   - Queue inventory, queue consumers, and queue metrics:
+     [`references/queue-consumers.md`](references/queue-consumers.md)
+3. Start with aggregate queries, grouped by environment, service, route,
+   queue, project, org, status, or error facets as appropriate.
+4. Fetch raw spans, logs, or traces only after aggregation identifies the
+   cluster or sample you need.
+5. For tenant-specific HTTP usage, prefer trace correlation over single-span
+   queries when tenant tags and route tags live on different spans.
+6. Report the windows, environments, sites, query links, and any sampling or
+   missing-data caveats.
+
+## When To Use Other Skills
+
+- Use [`debug-issue-with-datadog`](../debug-issue-with-datadog/SKILL.md) when a
+  Linear issue, GitHub issue, incident report, or monitor needs root-cause
+  analysis and patch recommendations.
+- Use [`weekly-production-review`](../weekly-production-review/SKILL.md) when
+  the user asks for a weekly engineering overview of production bugs, pages,
+  and incidents.
+- Use [`linear-bug-triage`](../linear-bug-triage/SKILL.md) only after a human
+  approves sharing measured findings in Linear.
+
+## Output Expectations
+
+Summarize what was checked, including:
+
+- Datadog site and `env` values covered.
+- Time windows.
+- Core filters or metrics used.
+- Count, rate, latency, queue depth, trace sample, or "No measurements found".
+- Datadog links or trace IDs that let the human rerun the query.
@@ -0,0 +1,4 @@
+interface:
+  display_name: "Datadog Query Recipes"
+  short_description: "Langfuse production telemetry queries"
+  default_prompt: "Use $datadog-query-recipes to investigate Langfuse production telemetry for a tenant, endpoint, queue, or regression."
@@ -0,0 +1,45 @@
+# Production Environments
+
+Langfuse production deploys cover these environments and services. The deploy
+matrix is defined in `.github/workflows/deploy.yml`.
+
+| Environment | Primary Datadog site | Common services |
+| --- | --- | --- |
+| `prod-us` | US, `datadoghq.com` | `web`, `web-ingestion`, `web-iso`, `worker`, `worker-cpu` |
+| `prod-eu` | EU, `datadoghq.eu` | `web`, `web-ingestion`, `web-iso`, `worker`, `worker-cpu` |
+| `prod-hipaa` | US, `datadoghq.com` | `web`, `web-ingestion`, `web-iso`, `worker`, `worker-cpu` |
+| `prod-jp` | US, `datadoghq.com` | `web`, `web-ingestion`, `web-iso`, `worker`, `worker-cpu` |
+
+The site mapping is a starting point, not proof. For cross-region research, run
+a small count or facet query on both Datadog sites before saying an environment
+has no data.
+
+## Starter Filters
+
+Use these as first-pass filters, then add the subsystem-specific route, queue,
+tenant, or error facets.
+
+```text
+env:prod-us
+env:prod-eu
+env:prod-hipaa
+env:prod-jp
+```
+
+```text
+env:<env> (service:web OR service:web-ingestion OR service:web-iso)
+env:<env> (service:worker OR service:worker-cpu)
+```
+
+For HTTP routes, start with `service:web` or `service:web-ingestion` depending
+on the endpoint. For queue consumers, start with `service:worker` and
+`service:worker-cpu`.
+
+## Cross-Site Rule
+
+When a query returns zero:
+
+1. Check the time window and spelling of `env`, `service`, and route/resource.
+2. Query facets on the same site for `env` and `service`.
+3. Repeat a small count query on the other Datadog site.
+4. Only then report "No measurements found".
@@ -0,0 +1,99 @@
+# Public API Tenant Usage
+
+Use this recipe when checking whether an org or project calls a public API
+route, including legacy endpoints such as:
+
+- `/api/public/traces`
+- `/api/public/traces/<traceId>` (`/api/public/traces/[traceId]` in Next.js)
+- `/api/public/observations`
+- `/api/public/observations/<observationId>`
+  (`/api/public/observations/[observationId]` in Next.js)
+- `/api/public/metrics`
+
+Migrated endpoints commonly include:
+
+- `/api/public/v2/observations`
+- `/api/public/v2/metrics`
+- `/api/public/otel/v1/traces`
+
+## Why Correlation Is Needed
+
+Public API auth attaches tenant attributes to the `api-auth-verify` child span:
+
+- `@langfuse.project.id`: project-scoped API key target.
+- `@langfuse.org.id`: owning org or org-scoped key target.
+- `@langfuse.org.plan`: plan when present.
+
+The HTTP route is on the request root span, usually in `http.path_group`,
+`http.route`, `http.target`, or `resource_name`.
+
+Because tenant tags and route tags often live on different spans in the same
+trace, do not expect this single-span query to work:
+
+```text
+@langfuse.project.id:<id> @http.path_group:/api/public/observations
+```
+
+## Per-Tenant Endpoint Recipe
+
+1. Aggregate auth spans for the tenant to confirm the identifier and active
+   projects.
+
+```text
+env:<env> @langfuse.org.id:<orgId> resource_name:api-auth-verify
+env:<env> @langfuse.project.id:<projectId> resource_name:api-auth-verify
+```
+
+Group by `service`, `@langfuse.project.id`, and optionally a daily interval.
+
+2. Fetch representative matching auth spans with `search_datadog_spans`.
+   Include custom attributes such as `langfuse.*`, then copy representative
+   `traceid` values.
+
+3. Open those traces with `get_datadog_trace`. Request service-entry spans and
+   include HTTP and Langfuse attributes:
+
+```text
+only_service_entry_spans: true
+extra_fields: ["http.*", "next.*", "langfuse.*"]
+```
+
+4. Read the request root span's `http.path_group`, `http.route`,
+   `http.target`, and `resource_name` to identify the endpoint.
+   ID lookup routes may appear with the concrete ID in `http.target` or with
+   the normalized Next.js route, such as
+   `/api/public/traces/[traceId]` or
+   `/api/public/observations/[observationId]`.
+
+5. Repeat across all relevant prod environments and both Datadog sites when the
+   user asks for a global answer.
+
+Treat per-tenant endpoint results as sampled unless you have an unsampled
+metric or log source with tenant and route on the same event.
+
+## Fleet-Level Endpoint Volume
+
+For endpoint volume without tenant scoping, aggregate request spans directly:
+
+```text
+env:<env> service:web resource_name:"GET /api/public/observations*"
+env:<env> service:web resource_name:"GET /api/public/observations/*"
+env:<env> service:web resource_name:"POST /api/public/traces*"
+env:<env> service:web resource_name:"GET /api/public/traces/*"
+env:<env> service:web resource_name:"POST /api/public/metrics*"
+```
+
+Use route facets where available:
+
+```text
+env:<env> service:web @http.path_group:/api/public/observations
+env:<env> service:web @http.path_group:/api/public/observations/*
+env:<env> service:web @http.path_group:/api/public/observations/[observationId]
+env:<env> service:web @http.route:/api/public/observations
+env:<env> service:web @http.route:/api/public/observations/[observationId]
+env:<env> service:web @http.path_group:/api/public/traces/[traceId]
+env:<env> service:web @http.route:/api/public/traces/[traceId]
+```
+
+If route facets and resource names disagree, fetch a few traces and inspect the
+root span before reporting the result.
@@ -0,0 +1,234 @@
+# Queue Consumers
+
+Use this reference when a task asks which Langfuse queues exist, whether a
+consumer is running, how much work a queue has, or how to query queue processor
+spans.
+
+## Source Of Truth
+
+- Queue names and job names:
+  `packages/shared/src/server/queues.ts` (`QueueName`, `QueueJobs`).
+- Queue producer classes and shard naming:
+  `packages/shared/src/server/redis/*.ts`.
+- Worker consumer registration and feature gates:
+  `worker/src/app.ts`.
+- Worker consumer env vars:
+  `worker/src/env.ts` (`QUEUE_CONSUMER_*_IS_ENABLED` plus feature-specific
+  gates).
+- Worker registration, request/error counters, wait/processing time, and
+  sampled old-style depth metrics:
+  `worker/src/queues/workerManager.ts`.
+- Queue depth background reporter:
+  `worker/src/features/queue-metrics-runner/index.ts`.
+- Metric name conversion:
+  `packages/shared/src/server/instrumentation/index.ts`
+  (`convertQueueNameToMetricName`).
+- Sharded queue registry:
+  `worker/src/queues/shardedQueueRegistry.ts`.
+- BullMQ tracing setup:
+  `worker/src/instrumentation.ts` (`BullMQInstrumentation`).
+
+## Queue Inventory
+
+Current `QueueName` values:
+
+| Queue | Notes |
+| --- | --- |
+| `trace-upsert` | Sharded. Registers all `TraceUpsertQueue` shards. |
+| `trace-delete` | Delete traces from storage. |
+| `project-delete` | Project deletion cleanup. |
+| `evaluation-execution-queue` | Sharded eval execution. |
+| `secondary-evaluation-execution-queue` | Sharded secondary eval execution. |
+| `llm-as-a-judge-execution-queue` | Sharded observation-based eval execution. |
+| `dataset-run-item-upsert-queue` | Dataset run item upserts. |
+| `batch-export-queue` | Batch exports. |
+| `otel-ingestion-queue` | Sharded OTel ingestion. |
+| `secondary-otel-ingestion-queue` | Sharded secondary OTel ingestion. |
+| `ingestion-queue` | Sharded single-event ingestion. |
+| `secondary-ingestion-queue` | Sharded secondary single-event ingestion. |
+| `cloud-usage-metering-queue` | Cloud-only, Stripe-gated. |
+| `cloud-spend-alert-queue` | Cloud-only, Stripe-gated. |
+| `cloud-free-tier-usage-threshold-queue` | Cloud-only, Stripe-gated. |
+| `experiment-create-queue` | Experiment creation. |
+| `posthog-integration-queue` | Schedules PostHog integration jobs. |
+| `posthog-integration-processing-queue` | Processes PostHog projects. |
+| `mixpanel-integration-queue` | Schedules Mixpanel integration jobs. |
+| `mixpanel-integration-processing-queue` | Processes Mixpanel projects. |
+| `blobstorage-integration-queue` | Schedules blob storage jobs. |
+| `blobstorage-integration-processing-queue` | Processes blob storage projects. |
+| `core-data-s3-export-queue` | Cloud export feature gate. |
+| `metering-data-postgres-export-queue` | Cloud export feature gate. |
+| `data-retention-queue` | Schedules data retention jobs. |
+| `data-retention-processing-queue` | Processes data retention projects. |
+| `batch-action-queue` | Batch actions. |
+| `create-eval-queue` | Eval job creation. |
+| `score-delete` | Score deletion cleanup. |
+| `dataset-delete-queue` | Dataset deletion cleanup. |
+| `dead-letter-retry-queue` | Dead letter retry worker. |
+| `webhook-queue` | Webhook delivery. |
+| `entity-change-queue` | Entity change propagation. |
+| `event-propagation-queue` | Experiment event propagation gate. |
+| `notification-queue` | Notifications. |
+
+Sharded queues use the base queue for shard 0 and append `-1`, `-2`, etc. for
+additional shards. The sharded base queues are:
+
+- `trace-upsert`
+- `evaluation-execution-queue`
+- `secondary-evaluation-execution-queue`
+- `llm-as-a-judge-execution-queue`
+- `otel-ingestion-queue`
+- `secondary-otel-ingestion-queue`
+- `ingestion-queue`
+- `secondary-ingestion-queue`
+
+## Consumer Gates
+
+Consumer registration is in `worker/src/app.ts`. Some gates register multiple
+queues or every shard for a sharded queue.
+
+| Gate | Queues registered |
+| --- | --- |
+| `QUEUE_CONSUMER_TRACE_UPSERT_QUEUE_IS_ENABLED` | `trace-upsert` shards |
+| `QUEUE_CONSUMER_CREATE_EVAL_QUEUE_IS_ENABLED` | `create-eval-queue` |
+| `LANGFUSE_S3_CORE_DATA_EXPORT_IS_ENABLED` | `core-data-s3-export-queue` |
+| `LANGFUSE_POSTGRES_METERING_DATA_EXPORT_IS_ENABLED` | `metering-data-postgres-export-queue` |
+| `QUEUE_CONSUMER_TRACE_DELETE_QUEUE_IS_ENABLED` | `trace-delete` |
+| `QUEUE_CONSUMER_SCORE_DELETE_QUEUE_IS_ENABLED` | `score-delete` |
+| `QUEUE_CONSUMER_DATASET_DELETE_QUEUE_IS_ENABLED` | `dataset-delete-queue` |
+| `QUEUE_CONSUMER_PROJECT_DELETE_QUEUE_IS_ENABLED` | `project-delete` |
+| `QUEUE_CONSUMER_DATASET_RUN_ITEM_UPSERT_QUEUE_IS_ENABLED` | `dataset-run-item-upsert-queue` |
+| `QUEUE_CONSUMER_EVAL_EXECUTION_QUEUE_IS_ENABLED` | `evaluation-execution-queue` shards, `llm-as-a-judge-execution-queue` shards |
+| `QUEUE_CONSUMER_EVAL_EXECUTION_SECONDARY_QUEUE_IS_ENABLED` | `secondary-evaluation-execution-queue` shards |
+| `QUEUE_CONSUMER_BATCH_EXPORT_QUEUE_IS_ENABLED` | `batch-export-queue` |
+| `QUEUE_CONSUMER_BATCH_ACTION_QUEUE_IS_ENABLED` | `batch-action-queue` |
+| `QUEUE_CONSUMER_OTEL_INGESTION_QUEUE_IS_ENABLED` | `otel-ingestion-queue` shards |
+| `QUEUE_CONSUMER_OTEL_INGESTION_SECONDARY_QUEUE_IS_ENABLED` | `secondary-otel-ingestion-queue` shards |
+| `QUEUE_CONSUMER_INGESTION_QUEUE_IS_ENABLED` | `ingestion-queue` shards |
+| `QUEUE_CONSUMER_INGESTION_SECONDARY_QUEUE_IS_ENABLED` | `secondary-ingestion-queue` shards |
+| `QUEUE_CONSUMER_CLOUD_USAGE_METERING_QUEUE_IS_ENABLED` plus `STRIPE_SECRET_KEY` | `cloud-usage-metering-queue` |
+| `QUEUE_CONSUMER_CLOUD_SPEND_ALERT_QUEUE_IS_ENABLED` plus `STRIPE_SECRET_KEY` | `cloud-spend-alert-queue` |
+| `QUEUE_CONSUMER_FREE_TIER_USAGE_THRESHOLD_QUEUE_IS_ENABLED` plus cloud region and Stripe gates | `cloud-free-tier-usage-threshold-queue` |
+| `QUEUE_CONSUMER_EXPERIMENT_CREATE_QUEUE_IS_ENABLED` | `experiment-create-queue` |
+| `QUEUE_CONSUMER_POSTHOG_INTEGRATION_QUEUE_IS_ENABLED` | `posthog-integration-queue`, `posthog-integration-processing-queue` |
+| `QUEUE_CONSUMER_MIXPANEL_INTEGRATION_QUEUE_IS_ENABLED` | `mixpanel-integration-queue`, `mixpanel-integration-processing-queue` |
+| `QUEUE_CONSUMER_BLOB_STORAGE_INTEGRATION_QUEUE_IS_ENABLED` | `blobstorage-integration-queue`, `blobstorage-integration-processing-queue` |
+| `QUEUE_CONSUMER_DATA_RETENTION_QUEUE_IS_ENABLED` | `data-retention-queue`, `data-retention-processing-queue` |
+| `QUEUE_CONSUMER_DEAD_LETTER_RETRY_QUEUE_IS_ENABLED` | `dead-letter-retry-queue` |
+| `QUEUE_CONSUMER_WEBHOOK_QUEUE_IS_ENABLED` | `webhook-queue` |
+| `QUEUE_CONSUMER_ENTITY_CHANGE_QUEUE_IS_ENABLED` | `entity-change-queue` |
+| `QUEUE_CONSUMER_EVENT_PROPAGATION_QUEUE_IS_ENABLED` plus events-table experiment gate | `event-propagation-queue` |
+| `QUEUE_CONSUMER_NOTIFICATION_QUEUE_IS_ENABLED` | `notification-queue` |
+
+## Query Consumer Spans
+
+Start with aggregate spans on worker services:
+
+```text
+env:<env> (service:worker OR service:worker-cpu) operation_name:bullmq.process
+```
+
+Then group by `resource_name`, queue facets such as `bullmq.queue` or
+`messaging.*`, and error fields. Facet names can differ between Datadog sites,
+so inspect one sample span before relying on a specific facet.
+
+Queue-specific starter query:
+
+```text
+env:<env> (service:worker OR service:worker-cpu) operation_name:bullmq.process (resource_name:"process otel-ingestion-queue" OR resource_name:"Worker.run otel-ingestion-queue" OR bullmq.queue:otel-ingestion-queue)
+```
+
+For sharded queues, query the base queue and shard suffixes:
+
+```text
+env:<env> (service:worker OR service:worker-cpu) operation_name:bullmq.process resource_name:"*otel-ingestion-queue*"
+```
+
+If a queue file wraps the handler with `instrumentAsync`, also search the
+domain-specific resource name. Examples:
+
+| Subsystem | Resource name |
+| --- | --- |
+| PostHog project processing | `process posthog-integration-project` |
+| Mixpanel project processing | `process mixpanel-integration-project` |
+| Blob storage project processing | `process blob-storage-project` |
+| Data retention project processing | `process data-retention-project` |
+| Event propagation | `process event-propagation` |
+| Cloud usage metering | `process cloud-usage-metering` |
+| Free-tier usage threshold | `process cloud-free-tier-usage-threshold` |
+
+Useful aggregations:
+
+- Count by `env`, `service`, and `resource_name`.
+- Count by queue facet and status.
+- Count by `error.type` and `error.message`.
+- p50, p95, and p99 duration by queue or shard.
+- Count by `messaging.bullmq.job.input.projectId` when the processor attaches
+  project IDs to the current span.
+
+## Query Queue Metrics
+
+Metric base names come from `convertQueueNameToMetricName(queueName)`:
+
+```text
+langfuse.queue.<queue-name-with-hyphens-replaced-by-underscores-and-trailing-_queue-removed>
+```
+
+Examples:
+
+| Queue | Metric base |
+| --- | --- |
+| `ingestion-queue` | `langfuse.queue.ingestion` |
+| `otel-ingestion-queue` | `langfuse.queue.otel_ingestion` |
+| `secondary-otel-ingestion-queue` | `langfuse.queue.secondary_otel_ingestion` |
+| `evaluation-execution-queue` | `langfuse.queue.evaluation_execution` |
+| `trace-upsert` | `langfuse.queue.trace_upsert` |
+| `batch-export-queue` | `langfuse.queue.batch_export` |
+
+Prefer the newer tagged metrics:
+
+```text
+<metric_base>.depth{env:<env>,type:waiting}
+<metric_base>.depth{env:<env>,type:failed}
+<metric_base>.depth{env:<env>,type:active}
+<metric_base>.rate{env:<env>,type:request}
+<metric_base>.rate{env:<env>,type:failed}
+<metric_base>.rate{env:<env>,type:error}
+<metric_base>.time{env:<env>,type:wait}
+<metric_base>.time{env:<env>,type:processing}
+```
+
+For sharded queues, use the `shard` tag when present. `shard:all` is emitted by
+the depth runner for aggregate depth across shards.
+
+Backward-compatible metrics may still appear:
+
+```text
+<metric_base>.length
+<metric_base>.dlq_length
+<metric_base>.active
+<metric_base>.request
+<metric_base>.failed
+<metric_base>.error
+<metric_base>.wait_time
+<metric_base>.processing_time
+```
+
+For non-BullMQ internal write buffering, `ClickhouseWriter` emits
+`langfuse.queue.clickhouse_writer.*` metrics, but it is not a `QueueName`
+consumer.
+
+## Consumer Running Checklist
+
+To establish whether a consumer is running in production:
+
+1. Check queue depth metrics for waiting, failed, and active counts.
+2. Check `rate{type:request}` or old `.request` metrics for recent processing.
+3. Search BullMQ processor spans on `worker` and `worker-cpu`.
+4. Search worker logs for the queue name or processor-specific log prefix.
+5. If all signals are empty, verify the relevant `QUEUE_CONSUMER_*_IS_ENABLED`
+   gate and any feature-specific gates in `worker/src/app.ts`.
+
+The queue metrics runner only polls queues with registered workers. Missing
+depth metrics can mean the consumer is not registered on that worker, queue
+metrics are disabled, or the data is on the other Datadog site.
@@ -0,0 +1,121 @@
+---
+name: debug-issue-with-datadog
+description: |
+  Debug a user-reported issue, Linear ticket, or incident report by combining
+  Datadog (APM, logs, metrics) with the Langfuse repo to establish a
+  root cause. Use when given a Linear issue URL/ID (e.g. LFE-XXXX), a GitHub
+  issue, or a pasted error/report and asked to investigate, root-cause, or
+  triage. Produces a structured analysis — error breakdown, hypothesis-by-class,
+  suggested patches with code references.
+---
+
+# Debug Issue with Datadog
+
+Use this skill whenever the task is **investigative** rather than
+implementational: a user, customer, or oncall has surfaced a problem and you
+need to figure out *what is actually happening in production* and *where in the
+code it lives*. The deliverable is an analysis, not a patch — though the
+analysis should make the right patch obvious.
+
+## When to Apply
+
+- A Linear issue (typically with an `LFE-XXXX` ID) describes a production
+  failure, error spike, or customer report.
+- A GitHub issue or pasted incident/error report needs triage.
+- A monitor alerted and you need to understand *why* before deciding what to
+  fix.
+- Existing tickets under the "Make monitoring useful again" project (parent
+  `LFE-8837`) and similar — these expect the structured analysis output below.
+
+If the task is "implement this fix" rather than "figure out what's broken",
+this is the wrong skill — go to `backend-dev-guidelines` or the relevant
+package guide.
+
+## Workflow
+
+Read the inputs first, then plan the Datadog sweep, then read the code, then
+write the analysis. Do not skip ahead to suggested patches before the data
+supports them.
+
+1. **Intake.** Pull every signal already available in the report. See
+   [`references/intake.md`](references/intake.md). For a Linear URL/ID, fetch
+   the issue *and* its comments via the Linear MCP — the description is often
+   updated inline as triage proceeds. For a GitHub issue, use `gh issue view`.
+   For pasted text, treat it as the description.
+
+2. **Scope the sweep.** From the intake, pick the affected subsystem and time
+   window. Use [`references/repo-debug-map.md`](references/repo-debug-map.md)
+   to translate "PostHog integration", "ingestion failures", "evals stuck",
+   etc. into the Datadog filters and source files you should be looking at.
+
+3. **Run the broad Datadog sweep.** Default to the full sweep in
+   [`references/datadog-playbook.md`](references/datadog-playbook.md): APM
+   spans, error logs, metrics, and monitors — split across `prod-eu`
+   and `prod-us` (and `prod-hipaa` / `prod-jp` when relevant). Always check
+   regional disparity first; it usually rules whole hypotheses in or out.
+   Use [`datadog-query-recipes`](../datadog-query-recipes/SKILL.md) for
+   reusable tenant, public API, queue consumer, and cross-environment query
+   shapes.
+
+4. **Cluster the errors.** Group by `(projectId, error.message)` or
+   `(error.type, error.message)`. Treat each distinct cluster as its own
+   hypothesis — Langfuse incidents commonly have *multiple* coexisting root
+   causes, not one.
+
+5. **Map clusters to code.** For each cluster, open the relevant handler file
+   from the repo-debug map and read enough of it to confirm or refute the
+   hypothesis. Cite specific files and line ranges in the output.
+
+6. **Write the analysis** using
+   [`references/output-template.md`](references/output-template.md).
+
+7. **Deliver.** Default: print the analysis in chat. If the user asked for it,
+   also save under the workflow they specified (file, Linear comment via, etc.).
+
+## Datadog MCP Usage Notes
+
+Two Datadog MCP servers are typically available — one bound to the EU site
+(`datadoghq.eu`) and one to the US site (`datadoghq.com`). Always run
+region-relevant queries against **both** unless intake clearly localizes the
+incident. The `prod-eu` / `prod-us` env tags live on each side respectively.
+
+- Span search filter pattern:
+  `service:worker resource_name:"process posthog-integration-project" status:error`
+- Log search filter pattern:
+  `service:worker env:prod-eu @langfuse.project.id:cm1r6u… status:error`
+- For high-volume queries, prefer `aggregate_spans` / `aggregate_events`
+  grouped by `(error.message, projectId)` over fetching individual traces.
+- Always link to the Datadog UI for the queries you ran (final section of the
+  output template).
+
+See [`references/datadog-playbook.md`](references/datadog-playbook.md) for the
+full set of starter queries and parameter shapes.
+
+## Output Expectations
+
+From the output template:
+
+- Header: data source, time window, region split (EU vs US table).
+- Hotspots: per-`projectId` (or per-cluster) error counts.
+- Root cause by error class: each cluster gets a short hypothesis with
+  reasoning, distinguishing primary causes from symptoms.
+- Suggested patches: P0/P1/P2 grouped, with concrete file paths and short code
+  sketches. Reference the actual handler in `worker/src/features/**` or
+  `web/src/**`.
+- Dashboards: paste the Datadog query URLs at the end.
+
+Findings come first, recommendations last. If the data is thin, say so
+explicitly and propose what would need to be true to confirm each hypothesis —
+do not invent root causes.
+
+## Cross-References
+
+- Production telemetry query recipes, tenant/public API usage, and queue
+  consumer measurements:
+  [`datadog-query-recipes`](../datadog-query-recipes/SKILL.md)
+- Backend layout, queue contracts, instrumentation patterns:
+  [`backend-dev-guidelines`](../backend-dev-guidelines/SKILL.md)
+- ClickHouse-related findings (memory ceilings, JOIN spills, slow queries):
+  [`clickhouse-best-practices`](../clickhouse-best-practices/SKILL.md)
+- Once a fix is identified and you switch to implementation, hand off to the
+  package `AGENTS.md` for the affected directory.
@@ -0,0 +1,141 @@
+# Datadog Query Playbook
+
+The default for this skill is the **broad sweep**: APM spans + logs + metrics
+ monitors + incidents, run against both EU and US sites unless intake clearly
+localizes. Cluster results before drilling in.
+
+Two MCP servers are typically connected — one for `datadoghq.eu`, one for
+`datadoghq.com`. Run the same query on both and compare. The contrast itself
+is often the most informative finding (e.g. LFE-9475's 23.5% EU vs 0.7% US
+error rate immediately ruled out PostHog Cloud as the global cause).
+
+## Tag Vocabulary
+
+- **Site:** `datadoghq.eu` for EU, `datadoghq.com` for US.
+- **Region tag (`env`):** `prod-eu`, `prod-us`, `prod-hipaa`, `prod-jp`.
+- **Service:** `worker` (default in `worker/src/env.ts`) or `web` (default in
+  `web/src/env.mjs`). Some deployments override to `langfuse`.
+- **Span resource names** for worker async jobs follow the pattern
+  `process <queue-name>` — see `repo-debug-map.md`.
+
+## 1. APM Span Sweep — find the failing handler
+
+Use `aggregate_spans` first; only fetch individual traces once a cluster is
+identified.
+
+Starter shape (rename the resource for the relevant subsystem):
+
+```text
+service:worker resource_name:"process posthog-integration-project" status:error
+```
+
+Aggregations to run, in order:
+
+1. Count by `env` — confirms region split.
+2. Count by `error.message` — primary error classes.
+3. Count by `(projectId, error.message)` — which tenants are affected, by
+   class. `projectId` lives on span tags as `@projectId` for log search and as
+   a tag for spans (depends on instrumentation site).
+4. p50 / p95 / p99 duration by region — fingerprints timeouts vs. crashes vs.
+   slow successes.
+
+If `aggregate_spans` returns no results, check:
+
+- the resource name is right (case-sensitive, see `repo-debug-map.md`);
+- the time window covers when the issue was actually firing;
+- the region tag matches reality (the EU MCP only sees EU traces).
+
+### Public API tenant / legacy-endpoint usage
+
+For tenant-specific public API route usage, use
+[`../../datadog-query-recipes/references/public-api-tenant-usage.md`](../../datadog-query-recipes/references/public-api-tenant-usage.md).
+The key gotcha is that tenant tags usually live on the `api-auth-verify` child
+span while the HTTP route lives on the request root span, so correlate by
+`traceid` rather than relying on one combined span filter.
+
+## 2. Log Sweep — read what the handler said
+
+Logs are the right tool for *messages* the handler emitted. Spans are the
+right tool for *which handler invocations failed*.
+
+Starter shapes:
+
+```text
+service:worker env:prod-eu @projectId:cm1r6u1iq00ccfvrkoy8vg3ms status:error
+service:worker env:prod-eu "[POSTHOG]" status:error
+```
+
+Useful log facets:
+
+- `@projectId` or `@langfuse.project.id` — Langfuse project (cuid).
+- `@error.kind` / `@error.message` / `@error.stack` — when the Winston logger
+  serialized an Error.
+- `@queue` / `@jobName` — when set by BullMQ instrumentation.
+
+For high-volume subsystems (`ingestion-queue`, `otel-ingestion-queue`),
+prefer `analyze_datadog_logs` with grouping over `search_datadog_logs` — the
+raw matches are too noisy.
+
+## 3. Metric Sweep — confirm the trend
+
+Pick 2–3 metrics that match the subsystem. Common ones:
+
+- `trace.bullmq.process.errors` and `trace.bullmq.process.duration` —
+  per-queue health from the BullMQ OTel instrumentation. Filter by
+  `resource_name:"process <queue-name>"`.
+- `trace.http_request.errors` and `trace.http_request.duration` for HTTP
+  handlers (`service:web`).
+- ClickHouse: cluster-level `clickhouse.query.duration`,
+  `clickhouse.memory_usage` — the worker doesn't emit these directly, they
+  come from the `clickhouse` integration in the infra repo.
+- Postgres: `aurora.databaseconnections`, `aurora.deadlocks` — relevant when
+  the symptom is `connection_limit` / `connection pool` errors.
+
+If the subsystem isn't already known, run `search_datadog_metrics` for the
+subsystem name and pick the obvious counter / gauge / histogram triplet.
+
+## 4. Monitors & Incidents
+
+- `search_datadog_monitors` for the subsystem name — tells you what alerts
+  *would* have fired and what their thresholds are. A muted monitor on the
+  affected subsystem is itself a finding (see LFE-9475: "EU alert muted for
+  a week").
+- `search_datadog_incidents` for the time window — links any pre-existing
+  incident the user may not have referenced.
+
+## 5. RUM / Frontend (only when the symptom is user-facing)
+
+Skip unless the issue is "page broken" / "slow load". Then:
+
+- `search_datadog_rum_events` filtered by `@view.url:` patterns matching the
+  affected route.
+- Cross-reference with `service:web` API errors at the same time.
+
+## 6. Trace Drill-Down
+
+Once a cluster is identified, fetch one or two representative traces with
+`get_datadog_trace` to read the actual stack and confirm where in the handler
+the throw originates. This is what lets you point at a specific file and
+line range in the analysis.
+
+## Anti-Patterns
+
+- Don't fetch individual logs/traces before aggregating. You'll burn context
+  on noise and miss the cluster pattern.
+- Don't trust a single-region query as global. Always compare EU and US.
+- Don't read an `error.message` literally if it goes through a custom error
+  wrapper — `validateWebhookURL` rejections, for example, are re-logged as
+  "DNS lookup failed" but are actually validator rejections.
+- Don't assume monitors are firing just because errors exist — check if the
+  monitor is muted.
+
+## Linking Out
+
+End the analysis with the actual Datadog UI URLs you queried, e.g.:
+
+```text
+https://app.datadoghq.eu/apm/traces?query=resource_name%3A%22process+posthog-integration-project%22+status%3Aerror
+https://app.datadoghq.com/apm/traces?query=resource_name%3A%22process+posthog-integration-project%22+status%3Aerror
+```
+
+so the human reader can re-run the same query.
@@ -0,0 +1,72 @@
+# Intake — What Are We Actually Investigating?
+
+Goal: extract every signal from the input *before* you touch Datadog. Cheap
+to do, expensive to skip — the wrong time window or wrong service tag can
+hide the real cause.
+
+## Source-by-Source Recipes
+
+### Linear issue (URL or `LFE-XXXX` ID)
+
+1. Fetch the issue via the Linear MCP:
+   `mcp__d39d26f2-…__get_issue` with `id: "LFE-XXXX"` and
+   `includeRelations: true`.
+2. Fetch comments separately:
+   `mcp__d39d26f2-…__list_comments` with the same `issueId`.
+3. Note: Langfuse triages **inside the issue description**. The description
+   often grows reaction sections (`projectId: <one-line note>`) as oncall
+   investigates. Treat both description and comments as authoritative state.
+4. Pull `attachments` for linked PRs/commits — these show what's already been
+   tried. Reading the diff of a half-merged fix often reveals the original
+   theory of the bug.
+5. Pull labels (`integration-posthog`, `feat-exports`, etc.) — they map
+   directly to the subsystem clusters in `repo-debug-map.md`.
+
+### GitHub issue
+
+1. `gh issue view <url-or-number> --json title,body,labels,comments,assignees`.
+2. If there's a stack trace in the issue body, copy the top frames into your
+   notes — those frames usually map straight to a handler file.
+
+### Pasted error / incident text
+
+1. Treat as the issue description. If it's a stack trace, identify:
+   - the throwing module (handler vs. SDK vs. infra),
+   - whether it looks like a *user-input* failure (HTTP 4xx, validation, auth)
+     or an *infra* failure (5xx, timeout, OOM, DNS, pool exhaustion),
+   - the error class (`Error`, `TypeError`, `PrismaClientKnownRequestError`,
+     etc.).
+2. If a `projectId` appears, that's gold — anchor every Datadog query on it.
+3. If a `traceId` (Datadog's, not Langfuse's) appears, jump straight to
+   `get_datadog_trace`.
+
+## Extract The Following Before Querying
+
+Build a small notes block. If a value is missing, mark it `?` rather than
+guessing — Datadog will tell you what's missing.
+
+- **Subsystem.** PostHog integration, blob-storage export, evaluation
+  execution, OTel ingestion, batch action, webhook delivery, etc. Map to
+  `repo-debug-map.md`.
+- **Region(s).** `prod-eu` / `prod-us` / `prod-hipaa` / `prod-jp`. If unknown,
+  query both EU and US.
+- **Service.** Most issues are `service:worker`; UI/API timeouts are
+  `service:web`. Check for both when unsure.
+- **Time window.** Default to 7 days back from the issue's `createdAt`. If
+  the issue references a specific incident or alert, use that window ±1 day.
+- **`projectId`(s).** Project IDs in Langfuse are `cuid`-shaped
+  (`cl…` / `cm…`, 25 chars). The reaction blocks in Linear descriptions
+  often *are* lists of affected project IDs.
+- **Error message fragments.** Exact substrings to grep for in DD logs:
+  `Header overflow`, `Timeout error.`, `HTTP 403`, `DNS lookup failed`,
+  `Cannot write to canceled buffer`, `connection pool`, etc.
+- **Already-attempted fixes.** Linked PRs/commits on the issue. Read their
+  diffs — your analysis must not re-recommend something that's already
+  shipped.
+
+## Output Of The Intake Step
+
+A short bullet list (not yet formatted as the final analysis) with each of
+the above filled in. The remaining steps key off this — `datadog-playbook.md`
+expects subsystem + region + window, `repo-debug-map.md` expects subsystem,
+and the output template wants the affected projects.
@@ -0,0 +1,127 @@
+# Output Template
+
+The analysis should be structured so it can be pasted directly as the first
+investigative comment on the Linear issue. The example to anchor on is the
+first comment on `LFE-9475` (PostHog Integration Processing Failures).
+
+Findings come first, recommendations last. If the data doesn't support a
+hypothesis, say so — do not invent root causes to fill the template.
+
+## Section Order
+
+1. **Header** — data source, time window, scope of sweep.
+2. **Volume & error-rate split** — table by region (always).
+3. **Hotspots** — table by `(projectId, dominant cause)` or by cluster.
+4. **Root cause by error class** — one numbered subsection per cluster.
+5. **Suggested patches** — P0 / P1 / P2, each with file paths and a short
+   code sketch.
+6. **Dashboards** — Datadog UI URLs for the queries you ran.
+
+## Skeleton (fill in with your findings)
+
+````markdown
+## Datadog APM + log analysis (<N>-day window, <YYYY-MM-DD> → <YYYY-MM-DD>)
+
+Source: APM spans with `resource_name:"process <queue-name>"` across EU and US.
+
+### Volume & error rate — <one-line summary of regional split>
+
+| Region | Total spans | Errors | Error rate |
+|---|---|---|---|
+| EU (`prod-eu`) | <n> | <n> | **<pct>%** |
+| US (`prod-us`) | <n> | <n> | **<pct>%** |
+
+<One sentence explaining where the noise actually lives.>
+
+### Hotspots — concentrated on ~<N> <region> projects
+
+<Region> errors break down by `(projectId, error.message)`:
+
+| ProjectId | Errors | Dominant cause |
+|---|---|---|
+| `<projectId>` | <n> | `<error message>` (<n>) + others |
+| ...           | ... | ...                              |
+
+<Optional: contrast with another subsystem if relevant — e.g.
+"Unlike blob storage, PostHog has multiple distinct root causes — not one
+hotspot pattern.">
+
+## Root cause by error class
+
+### 1. `<error message>` — <n> errors, <n> projects
+<2–4 sentences explaining what this error class actually is at the
+implementation level (which library, which call site). Then list candidate
+causes in order of likelihood. Mark which ones are confirmed by the data
+vs. speculative.>
+
+### 2. `<error message>` — <n> errors, mostly <n> projects
+<Same pattern.>
+
+### 3. <next class>
+<...>
+
+### <N>. <Symptom of upstream failure>
+<Use this slot when a class is a *symptom* of another class rather than an
+independent bug — call it out so suggested patches don't double-count.>
+
+## Suggested patches
+
+### P0 — <one-line summary, e.g. "Auto-disable integrations on persistent
+auth failures">
+<Why this is P0 — what noise it kills, what data it stops corrupting, what
+unblocks downstream work.>
+
+```ts
+// <relative path from repo root>
+// Short code sketch (5–20 lines). It does not need to compile —
+// it must communicate the shape of the change.
+```
+
+### P0 — <next P0>
+<...>
+
+### P1 — <smaller / less urgent fix>
+<Same shape.>
+
+### P2 — <separate-but-surfaced finding>
+<E.g. a Prisma pool sizing issue surfaced incidentally by this analysis but
+not the original bug. Call it out with its own section so it doesn't get
+lost.>
+
+### Regional split explanation (only if relevant)
+<One paragraph explaining why EU vs. US asymmetry exists — usually not an
+infra bug, just where the affected tenants happen to live.>
+
+Dashboards:
+- EU APM: <url>
+- US APM: <url>
+- (logs / metrics / monitor links as relevant)
+````
+
+## Style Rules
+
+- Lead with numbers, not adjectives. "23.5% error rate" beats "very noisy".
+- Distinguish **primary causes** from **symptoms** explicitly. Symptoms
+  shouldn't get their own P0 patch.
+- Always cite specific files when proposing a code change. A patch
+  recommendation without `worker/src/features/<…>/<file>.ts` is unfinished.
+- Code sketches are illustrative — clearly mark them as sketches if they
+  hand-wave types. The next agent / human will write the real diff.
+- If the analysis surfaces a finding *outside* the original ticket scope
+  (e.g. a Prisma pool issue while debugging PostHog), include it as a P2
+  with a sentence explaining it's separate.
+- If the data refuses to converge on a single root cause, say so. The
+  template handles N classes — use as many subsections as the data warrants.
+
+## When To Skip Sections
+
+- **Single-region deployments:** if the issue clearly affects only one
+  region, you can replace the "Volume & error rate" table with a single-row
+  variant, but still note that the other region was checked and clean.
+- **No code change recommended:** if the only finding is "the affected
+  tenants have misconfigured credentials and we should reach out", the
+  Suggested-patches section can be a single sentence — but still include
+  the dashboards.
+- **Aborted investigation:** if Datadog access fails or the data is
+  insufficient, write what you tried, what was missing, and what would let
+  the next investigator pick it up.
@@ -0,0 +1,100 @@
+# Repo Debug Map — Subsystem → Code → Datadog Filters
+
+For each subsystem we ship monitors and incidents on, this is the canonical
+map between the symptom, the Datadog query that surfaces it, and the source
+files where the bug almost certainly lives.
+
+When intake gives you a subsystem (PostHog, evals, exports, etc.), start
+here to pick the right Datadog filters and the right files to read.
+
+## Worker Async Jobs
+
+Worker handlers are wrapped by `instrumentAsync` in their queue file. The
+span resource name follows the pattern `process <queue-name>`. Queue and job
+name constants live in
+`packages/shared/src/server/queues.ts`
+(`QueueName` and `QueueJobs` enums).
+
+| Subsystem | Queue file | Handler dir | Span `resource_name` | Log prefix |
+| --- | --- | --- | --- | --- |
+| PostHog integration | `worker/src/queues/postHogIntegrationQueue.ts` | `worker/src/features/posthog/` | `process posthog-integration-project` | `[POSTHOG]` |
+| Mixpanel integration | `worker/src/queues/mixpanelIntegrationQueue.ts` | `worker/src/features/mixpanel/` | `process mixpanel-integration-project` | `[MIXPANEL]` |
+| Blob storage export | `worker/src/queues/blobStorageIntegrationQueue.ts` | `worker/src/features/blobstorage/` | `process blob-storage-project` | `[BLOBSTORAGE]` |
+| Data retention | `worker/src/queues/dataRetentionQueue.ts` | `worker/src/features/batch-data-retention-cleaner/` | `process data-retention-project` | n/a |
+| Event propagation | `worker/src/queues/eventPropagationQueue.ts` | `worker/src/features/eventPropagation/` | `process event-propagation` | n/a |
+| Cloud usage metering | `worker/src/queues/cloudUsageMeteringQueue.ts` | `worker/src/ee/` (cloud-only) | `process cloud-usage-metering` | n/a |
+| Free-tier usage threshold | `worker/src/queues/cloudFreeTierUsageThresholdQueue.ts` | `worker/src/ee/usageThresholds/` | `process cloud-free-tier-usage-threshold` | n/a |
+| Ingestion (single event) | `worker/src/queues/ingestionQueue.ts` | `worker/src/features/ingestion/` (and `IngestionService`) | BullMQ default span | n/a |
+| OTel ingestion | `worker/src/queues/otelIngestionQueue.ts` | `worker/src/features/otel/` | BullMQ default span | n/a |
+| Evaluation execution | `worker/src/queues/evalQueue.ts` | `worker/src/features/evaluation/` | BullMQ default span | n/a |
+| Batch export | `worker/src/queues/batchExportQueue.ts` | `worker/src/features/batchExport/` | BullMQ default span | n/a |
+| Webhook delivery | `worker/src/queues/webhooks.ts` | `worker/src/features/webhooks/` | BullMQ default span | n/a |
+| Trace / score / dataset / project delete | `worker/src/queues/{traceDelete,scoreDelete,datasetDelete,projectDelete}.ts` | `worker/src/features/traces/`, `…/scores/`, `…/datasets/` | BullMQ default span | n/a |
+
+For queues using BullMQ default spans (no `instrumentAsync` wrapper), search
+APM with `service:worker operation_name:bullmq.process` filtered by
+`bullmq.queue:<queue-name>`. For queue inventory, sharded queue naming, and
+queue metric recipes, use
+[`../../datadog-query-recipes/references/queue-consumers.md`](../../datadog-query-recipes/references/queue-consumers.md).
+
+## Web (Next.js / tRPC / public API)
+
+| Subsystem | Code | Span / log filter |
+| --- | --- | --- |
+| Public REST API | `web/src/pages/api/public/**` | Request span: `service:web resource_name:"GET /api/public/<path>"`; tenant span: `resource_name:api-auth-verify` with `@langfuse.project.id` / `@langfuse.org.id` |
+| tRPC procedures | `web/src/server/api/routers/**` | `service:web resource_name:"POST /api/trpc/<router>.<proc>"` |
+| Auth / API key verification | `web/src/features/public-api/server/apiAuth.ts` | look for `verifyAuthHeaderAndReturnScope` spans |
+| Stripe billing | `web/src/ee/features/billing/server/stripeBillingService.ts` | wrapped in `instrumentAsync`; spans named after the method |
+
+For tenant-specific public API usage questions, first query
+`resource_name:api-auth-verify` by `@langfuse.project.id` or
+`@langfuse.org.id`, then open representative trace IDs and inspect the request
+root span for `http.path_group`, `http.route`, and `http.target`. The tenant
+tags and endpoint path are usually on different spans, so a single-span query
+combining both may return no results even when the trace proves usage.
+For the full reusable recipe, use
+[`../../datadog-query-recipes/references/public-api-tenant-usage.md`](../../datadog-query-recipes/references/public-api-tenant-usage.md).
+
+## Shared Layers
+
+These are not subsystems on their own, but are *frequently the actual cause*
+behind a worker subsystem failure.
+
+| Layer | Location | Common failure modes |
+| --- | --- | --- |
+| ClickHouse access | `packages/shared/src/server/clickhouse/`, `packages/shared/src/server/repositories/` | OOM (`Code: 241`), buffer cancel (`Code: 734`), JOIN spills, slow queries on un-pre-filtered traces |
+| Prisma access | `packages/shared/src/db.ts` and per-feature repos | `connection pool timeout` (worker default `connection_limit=5`), N+1 queries |
+| Queue contracts | `packages/shared/src/server/queues.ts` | wrong queue name, missing schema validation |
+| Logger / instrumentation | `packages/shared/src/server/logger.ts`, `packages/shared/src/server/instrumentation.ts` | log silently dropped because `LANGFUSE_LOG_LEVEL` set wrong, or span missing because handler doesn't call `instrumentAsync` |
+| Webhook URL validation | `packages/shared/src/server/validateWebhookURL.ts` | rejects with messages that *look* like DNS errors but are SSRF guard rejections |
+| Encryption | `packages/shared/encryption` | bad keys → 403/auth-style failures masquerading as upstream errors |
+
+## Common Symptoms → First Files To Read
+
+- **"403 from upstream":** check the per-integration credentials table in
+  Postgres (`PostHogIntegration`, `BlobStorageIntegration`, `WebhookConfig`,
+  etc.) and the encryption layer.
+- **"Timeout":** check the SDK timeout default and the per-stream
+  flush/batch size in the handler. Worker async jobs default to long-running
+  but the upstream SDK does not.
+- **"DNS lookup failed":** distinguish actual DNS from `validateWebhookURL`
+  rejection. The error message wrapping is misleading on purpose.
+- **"Cannot write to canceled buffer" (CH):** ClickHouse stream wasn't
+  aborted when the downstream consumer threw. Look for an `AbortController`
+  threaded through the handler.
+- **"Connection pool timeout" (Prisma):** worker `connection_limit` is set
+  in the connection string; jobs doing per-row `findFirst()` exhaust it.
+  Check whether the integration row could be cached in closure scope.
+- **"memory limit exceeded" (CH):** look for unbounded JOINs without a
+  pre-filter CTE, especially in analytics integrations.
+- **"Header overflow":** Node HTTP parser's default 80 KB ceiling. Either
+  raise `--max-http-header-size` for the worker, or replace the SDK's HTTP
+  client.
+
+## Where to Look for Already-Shipped Fixes
+
+Before recommending a patch, confirm it isn't already merged or in flight:
+
+- `attachments` on the Linear issue (PRs and commits are auto-linked).
+- `git log --oneline --since=<recent-window> -- <handler-path>`.
+- Open PRs touching the file via `gh pr list --search "<filename>"`.
@@ -25,13 +25,29 @@ Use this skill when a change affects what users see or do in the browser.
 - Bug fixes where the failure mode is visible in the browser
 - Final signoff for user-visible frontend work

+## Prefill Test Data First
+
+Most flows are only reviewable against meaningful data. Before opening the
+browser, seed what the flow needs with the seed CLI (see the
+`seed-test-data` skill for the need→command table):
+
+- `pnpm run seed -- trace-tree --observations 5000 --v4` — complex
+  observation trees (v3 + v4 events)
+- `pnpm run seed -- long-session --traces 300` — heavy session views
+- `pnpm run seed -- many-traces --count 100000` — list/filter performance
+- `pnpm run seed -- doctor` — when the stack misbehaves
+
+Every run prints UI deep links — open those instead of navigating manually.
+Do not hand-write seed scripts or raw ClickHouse inserts.
+
 ## Review Loop

 1. Start the app with `pnpm run dev:web` unless an existing local server is
   already running.
 2. Install Chromium with `pnpm run playwright:install` if Playwright has not
   been set up on the machine yet.
-3. Open the primary changed flow with the Playwright MCP server.
+3. Open the primary changed flow with the Playwright MCP server, using the
+   deep links printed by the seed CLI when the flow needs seeded data.
 4. Exercise the main happy path affected by the change.
 5. Check for obvious visual regressions:
   - broken layout or spacing
@@ -41,7 +57,7 @@ Use this skill when a change affects what users see or do in the browser.
 6. If the page changed materially, inspect the resulting UI state and compare
   it against the intended behavior from the task or existing patterns.
 7. If the browser session fails, inspect traces and artifacts under
-   `.playwright-mcp/`.
+   `/tmp/playwright-mcp`.

 ## Output Expectations

@@ -0,0 +1,74 @@
+---
+name: frontend-large-feature-architecture
+description: |
+  Use when building, changing, or refactoring large Langfuse frontend features,
+  virtualized lists, large tables, controller components, local feature state,
+  Zustand stores, row selection, high-frequency UI state, or
+  rendering-performance issues.
+---
+
+# Frontend Large Feature Architecture
+
+Use this skill when building, changing, or refactoring a large frontend
+surface.
+
+In this skill, "controller" means a component or hook that owns feature logic:
+data fetching, view state, table/list state, effects, actions, and expensive
+rendering. The problem is not the name; it is one place owning too many
+changing responsibilities.
+
+## Big Feature Rules
+
+When a feature grows, split rendering from logic. Most components should be
+view-only. Data preparation should be pure. Complex user actions should live in
+external async functions or local-store actions. Effects are integration
+boundaries, not the normal way to derive state.
+
+For the full rules, read
+[`references/big-feature-rules.md`](references/big-feature-rules.md).
+
+## Required Model
+
+- The page/view owns lifecycle and creates feature-scoped dependencies.
+- Server/query state stays in tRPC/React Query; route state stays in the
+  router/filter hooks.
+- High-frequency feature UI state belongs in a per-mount local vanilla Zustand
+  store. Create it with lazy `useState`, not `useMemo`.
+- Global stores are only for truly cross-feature, cross-route product state.
+- React context may provide a stable store or action owner. Do not put
+  frequently changing state directly in provider values.
+- Rendered rows/cells/items should be view-only or narrow containers. Put
+  effects, subscriptions, data loading, and workflows outside expensive views.
+- Shared `src/components/*` exports should stay context-free or receive
+  explicit props. Put view-scoped Zustand consumers in `src/features/*`.
+- Large feature folders should have a concise `README.md` owner map.
+- Migrate real features through small PRs that improve one state boundary,
+  action workflow, data-preparation seam, or render boundary.
+
+## Local Store Default
+
+Prefer a local vanilla Zustand store for large or high-frequency feature state.
+The store is created by the page/view and destroyed on unmount.
+
+Use selectors that return primitives or stable references. If a component needs
+multiple values, use shallow selector helpers or split subscriptions so one
+changing field does not rerender unrelated UI.
+
+Complex user workflows should live in `actions/*.ts` files or store actions.
+The component wires hooks and passes dependencies; the action owns the workflow.
+
+## When To Read References
+
+- For the core big-feature rules and migration reality, read
+  [`references/big-feature-rules.md`](references/big-feature-rules.md).
+- For virtualized lists, translated DOM, row measurement, and scroll rerenders,
+  read [`references/virtualized-lists.md`](references/virtualized-lists.md).
+- For local feature stores and splitting large components, read
+  [`references/local-feature-state.md`](references/local-feature-state.md).
+- For feature `README.md` owner maps, read
+  [`references/feature-readmes.md`](references/feature-readmes.md).
+- For a step-by-step migration from a controller component to a managed feature
+  pattern, read
+  [`references/controller-migration.md`](references/controller-migration.md).
+  This is the default reference for traces/observations tables, sessions,
+  experiments, prompts, evals, datasets, and other controller-heavy surfaces.
@@ -0,0 +1,7 @@
+interface:
+  display_name: "Frontend Feature Architecture"
+  short_description: "Build, change, or refactor React features"
+  default_prompt: "Use $frontend-large-feature-architecture to build, change, or refactor this large frontend surface with clear state ownership, stable subscriptions, and view-only render boundaries."
+
+policy:
+  allow_implicit_invocation: true
@@ -0,0 +1,98 @@
+# Big Feature Rules
+
+Use these rules when a feature keeps growing and the instinct is to add one
+more state variable, prop, callback, or effect to the same component.
+
+Here, "controller" means a component or hook that owns feature logic: state,
+effects, data loading, subscriptions, actions, and derived data. Some
+controller code is necessary. The failure mode is one controller owning too many
+changing responsibilities and waking too much UI.
+
+## Ownership Baseline
+
+- The page/view owns lifecycle and creates feature-scoped dependencies.
+- Server/query state stays in tRPC/React Query.
+- Route state stays in router/filter hooks.
+- High-frequency local UI state belongs in a per-mount feature store.
+- Global stores are only for product state shared across routes or features.
+- Pure data preparation turns fetched data into UI data before rendering.
+- Complex workflows live in `actions/*.ts` or store actions.
+- Effects are integration boundaries, not ordinary state derivation.
+- Expensive rows/cells/items should be view-only behind narrow containers.
+
+## Hard Rules
+
+1. Growth usually means React rendering and feature logic need to be separated.
+   A bigger component is not an architecture.
+2. Most components should be view-only. They render what they are given, or
+   they read a small selected value from a context-available feature store.
+3. Complicated data preparation belongs in separate pure functions. Backend
+   data should flow one way: fetched data -> compiled UI data -> render.
+4. User actions that trigger complex logic should live outside render
+   components as named async functions or store actions. If the action needs a
+   lot of context, pass the feature store instance and let the action read what
+   it needs.
+5. A feature with both local store state and React Query data needs an explicit
+   bridge in a custom hook or named action.
+6. Prefer view-scoped local feature stores over global stores for page-instance
+   state. Filters, selected rows, expanded rows, lazy load state, drawers, and
+   local actions usually belong to one mounted page instance.
+7. Create local store instances with a lazy `useState` initializer, not
+   `useMemo`. The store is a per-mount instance, not a render-time derived
+   value.
+8. Effects should not be normal state mutators. Before adding an effect, try
+   pure data preparation, a store action, or an explicit event handler.
+9. Do not copy existing large Langfuse feature components as examples. Treat
+   them as legacy unless they follow this skill.
+10. Prefer small reliable PRs over a large rewrite. Each PR should improve one
+    state boundary, action workflow, data-preparation seam, or render boundary,
+    then update the feature README or migration note with the next slice.
+
+## Migration Reality
+
+Most large frontend features are not yet in the ideal shape. Traces,
+observations, experiments, prompts, evals, datasets, and session views all have
+some controller-heavy surfaces. Do not copy an existing large component just
+because it works today. Treat it as a migration candidate and move it one
+state/action/data-preparation boundary at a time.
+
+For the step-by-step path, read `controller-migration.md`.
+
+## Small PR Policy
+
+A good big-feature PR is intentionally incomplete. It should change one
+semantic interaction and keep behavior stable: one selection boundary, one
+action workflow, one pure data-preparation helper, or one render boundary.
+
+Do not bundle file reorganization, state extraction, visual changes, and
+behavior fixes in the same PR unless the user explicitly asks for that scope.
+When a reorganization is needed, prefer a rename-only PR first or a later
+follow-up PR.
+
+## Effects And Actions
+
+Effects are for subscriptions, observers, imperative third-party APIs,
+one-time initialization, and cleanup. Keep them in containers or feature hooks,
+not view components. If an effect writes state repeatedly, make the store action
+idempotent.
+
+Complex actions should be callable without rendering a component. Put workflows
+in `actions/*.ts` or named store actions. Components wire hooks and pass
+dependencies; actions own the workflow.
+
+```ts
+await applyBulkAction({
+  store,
+  queryClient,
+  projectId,
+});
+```
+
+Actions must not call React hooks. Pass hook results, query helpers, the local
+store instance, or narrow callback dependencies into the action. If an action
+needs substantial data preparation, export a pure helper next to it so the
+transformation can be tested independently.
+
+Do not pass twenty props through the tree so a button can do feature-level work,
+and do not leave complex workflows inline in a page controller because that
+controller happened to have all dependencies in scope.
@@ -0,0 +1,123 @@
+# Controller Migration Guide
+
+Use this when a frontend surface has grown into one component or hook that owns
+data fetching, route glue, filters, table/list state, selection, drawers,
+actions, and expensive rendering.
+
+The target is clear ownership, not "everything in a store." Start from the
+ownership baseline in `big-feature-rules.md`.
+
+Most existing features are not there yet. Migrate in narrow slices and keep
+behavior stable.
+
+## Realistic Migration Strategy
+
+Do not start by designing the perfect final feature. Start by making the next
+change safer than the previous one.
+
+For each migration PR, write down:
+
+- the current controller problem being targeted
+- the single state/action/data-preparation/render boundary being improved
+- what behavior must remain unchanged
+- what instrumentation or tests prove the slice worked
+- what still remains spread across the feature
+- the next recommended atomic slice
+
+This is how large features become managed features without review-hostile
+rewrites. The feature README is the living owner map; update it in place as the
+feature moves forward.
+
+## Step-by-Step Path
+
+1. **Map the controller.** List every state group, query, derived value, effect,
+   callback, action, and expensive child render owned by the component.
+2. **Classify state.** Separate server/query, route, persisted browser,
+   high-frequency local UI, derived view data, imperative integration, and
+   one-off modal/form state.
+3. **Instrument one symptom.** Pick a concrete interaction such as row
+   selection, scroll, filter change, drawer open, form step change, or
+   saved-view change. Measure what rerenders, remounts, refetches, or
+   recalculates.
+4. **Choose one boundary.** Start with the smallest high-value boundary:
+   selection, lazy-row state, batch action workflow, filter-target mapping,
+   column/view state, wizard step state, or pure data preparation. Do not
+   migrate everything at once.
+5. **Choose the lightest tool.** A pure helper or action extraction may be the
+   right first PR. Add a local store only when selective subscriptions or
+   per-mount persistence are needed.
+6. **Create a local store instance when needed.** Use lazy `useState` in the
+   page/view:
+
+   ```ts
+   const [store] = useState(() => createFeatureStore(initialState));
+   ```
+
+   Provide only this stable store instance through context.
+
+7. **Move mutations into named actions.** Put state-changing logic in store
+   actions or external action functions. Keep components responsible for user
+   events, not workflows.
+8. **Split containers from views.** Containers may subscribe, call hooks, or
+   fetch data. Views should render props or tiny selected values.
+9. **Move data preparation out of render.** Put expensive or complicated
+   transformations in pure functions. Backend data should flow into compiled UI
+   data, then into rendering.
+10. **Bridge query state explicitly.** If local store decisions depend on React
+    Query data, use a named feature hook or action to express that relationship.
+11. **Isolate imperative integration.** Virtualizers, observers, keyboard
+    listeners, third-party DOM mutation handling, and timers belong in narrow
+    integration hooks.
+12. **Update the feature README.** Record what this PR improved, desired
+    boundaries, known spread state, and the next extraction target.
+13. **Remove debug instrumentation.** Temporary logs are useful during
+    migration, but should not survive the slice.
+14. **Repeat.** Each slice should make one semantic interaction narrower and
+    easier to reason about.
+
+## Feature-Specific First Slices
+
+Use the feature's current shape to pick the first slice:
+
+- **Traces and observations tables**: start with row selection, select-all,
+  batch actions, or expensive cell wrappers.
+- **Session detail and session events**: isolate virtualization, lazy-row load
+  state, and dynamic measurement from rendered row content.
+- **Experiment result tables**: start with selected-row state, filter-target
+  mapping, run/evaluation batch actions, or pure helpers for comparison column
+  construction.
+- **Experiment creation wizards**: separate submitted form data from display
+  state such as active step, selected prompt labels, schema display names, and
+  evaluator selection.
+- **Prompt management**: split prompt detail route/query state, label/version
+  selection, prompt history data preparation, and mutation workflows.
+- **Eval template and evaluator forms**: extract form defaults, model/provider
+  preparation, validation helpers, and submit workflows before adding a store.
+- **Datasets and dataset runs**: isolate active-cell/compare-field state,
+  table selection, run comparison preparation, and upload/import workflows.
+
+If a feature already has hooks for part of this work, treat them as partial
+migration, not proof that the whole surface is healthy.
+
+## Acceptance Criteria
+
+- A local state change wakes only components that selected that state.
+- Page components no longer rebuild columns, filters, row wrappers, and action
+  callbacks for unrelated row-level changes.
+- Expensive rows/cells are view-only behind narrow containers.
+- Effects are integration boundaries or one-time initialization, not ordinary
+  data derivation.
+- Complex workflows can be called without rendering the page.
+- The feature README tells the next developer what has improved, what remains
+  spread, and what the next small PR should target.
+
+Avoid:
+
+- Replacing a giant component with a giant global store.
+- Moving all state at once without measured acceptance criteria.
+- Treating memoization as the architecture.
+- Putting provider-coupled store hooks into shared `src/components/*` exports.
+- Leaving a workflow inline in the page because the page had every dependency in
+  scope.
+- Using the README as a victory statement while hiding remaining controller
+  state. Be explicit about remaining debt.
@@ -0,0 +1,89 @@
+# Feature READMEs
+
+Large frontend feature folders should have a short `README.md` that acts as an
+owner map for humans and agents. Prefer `README.md` over `FEATURE.md` to match
+the existing `web/src/features/*` convention. Use `FEATURE.md` only if a folder
+already has a user-facing or generated README.
+
+The README is not a changelog. It should describe durable boundaries and point
+to deeper migration notes.
+
+## Required Sections
+
+- **Surface**: what product surface the folder owns.
+- **Entry Points**: route files or parent components that mount the feature,
+  and the page/view lifecycle owner files they call.
+- **Structure**: what each subfolder owns. Use root `components/` only for
+  components reused across surfaces inside the feature. Use surface folders such
+  as `detail/` for page controllers, local stores, `actions/`, integration
+  hooks, and surface-private containers.
+- **External Consumers**: other features that import these components. This
+  keeps shared exports context-free and prevents accidental provider coupling.
+- **State Ownership**: where server/query state, route state, local feature
+  state, global product state, DOM integration state, and view-only props live.
+- **Performance And Stability Boundaries**: which interactions are
+  high-frequency or externally unstable, and which components are allowed to
+  rerender or measure because of them.
+- **Migration State**: what has already been improved, what state/actions are
+  still spread, and the next one or two atomic slices.
+- **Development Context**: agent skills, migration notes, or issue docs to read
+  before extending the feature.
+
+## State Ownership Rules
+
+Use the ownership baseline in `big-feature-rules.md`. The README only needs to
+name where each state category lives in this feature and where known spread
+state remains.
+
+## Performance And Stability Map
+
+Every large feature README should name the high-frequency interactions that
+must stay narrow. Typical examples:
+
+- scroll and virtualization updates
+- row selection, hover, expansion, and lazy-loading
+- filter, saved-view, and column-state changes
+- drawers, peek navigation, and keyboard navigation
+- browser translation or other third-party DOM mutation
+- resize and dynamic row measurement
+
+For each interaction, state which boundary should update. The page component
+should not rerun expensive data preparation, recreate column/config objects, or
+rerender unchanged expensive cells for unrelated state changes.
+
+## Current-State Honesty
+
+If a feature is mid-migration, say so. The README should make the desired
+boundaries clear while naming known spread state as debt. Do not present a
+partially migrated feature as the final pattern.
+
+Use an update-in-place style rather than a changelog. For example:
+
+- **Improved in current shape**: local store owns row selection; export action
+  moved to `actions/exportFeatureData.ts`; row view no longer subscribes to
+  filter state.
+- **Still spread**: saved-view state remains in the page controller; filter
+  option preparation is still inline; mutation workflows still close over page
+  hooks.
+- **Next slice**: extract filter-option preparation into pure helpers; move
+  batch action workflow into an action file; split route/query glue from view
+  components.
+
+This makes small PRs reviewable while keeping the feature migration plan
+visible. Do not wait for a perfect reorganization before documenting the
+current state.
+
+## PR-Scale Guidance
+
+Feature README updates should match the PR size:
+
+- For a small state/action extraction, add or update only the relevant
+  migration-state bullets.
+- For a new feature folder structure, include the owner map and external
+  consumers before moving logic into the folder.
+- For a rename-only PR, document intended structure but avoid changing behavior.
+- For behavior PRs, avoid unrelated file moves unless they are required for the
+  boundary being improved.
+
+The README should help the next contributor avoid falling back into the same
+large component, not argue that the migration is complete.
@@ -0,0 +1,162 @@
+# Local Feature State
+
+Large frontend features should not let one component or hook own everything.
+That shape makes one checkbox, hover, or row-selection change rerun the code
+that also builds filters, columns, data wrappers, expensive cells, drawers,
+actions, and routing glue.
+
+The goal is to make each state change wake only the UI and actions that
+semantically depend on it.
+
+## Default Pattern
+
+Start from the ownership baseline in `big-feature-rules.md`, then add a local
+vanilla Zustand store only when state is high-frequency, shared across multiple
+subtrees in the mounted feature, or must survive row/item remounts.
+
+Use this shape:
+
+1. The page/view creates one store instance with lazy `useState`.
+2. Context provides only the stable store instance.
+3. Components subscribe to the smallest useful slice.
+4. Mutations live in named store actions.
+5. Complex user workflows live in `actions/*.ts` or store actions.
+6. Expensive cells/rows stay view-only behind narrow containers.
+7. Large feature roots have a short `README.md` owner map.
+
+## Feature README
+
+For a concrete owner-map template, read `feature-readmes.md`. Do not use the
+README as a changelog; record durable ownership facts and the next migration
+slice.
+
+## Why Local, Not Global
+
+Langfuse pages are often local-state heavy: filters, saved views, selected rows,
+expanded rows, drawers, peek navigation, lazy row load state, and view-local
+actions. Those states usually belong to one mounted page instance, not the whole
+application.
+
+Use local feature stores for this state. Create the store in the page/view and
+destroy it on unmount. This keeps multiple mounted instances independent, avoids
+cross-route state leaks, and makes ownership visible.
+
+Global state is reserved for product state that is genuinely shared across
+features or routes. Do not promote state globally to avoid prop drilling or to
+make a large component smaller.
+
+Do not add a store just to make a PR look architectural. If the immediate
+problem is an inline export workflow, duplicated filter option shaping, or a
+large column builder, first extract an action or pure helper. Use the store when
+state needs selective subscriptions or must survive row remounts within one
+mounted feature instance.
+
+## Creating The Store
+
+Prefer lazy `useState` for local store instances:
+
+```ts
+const [store] = useState(() =>
+  createFeatureStore({
+    initialProjectId: projectId,
+  }),
+);
+```
+
+This expresses the real lifecycle: one store instance for the committed mounted
+view. The unused setter is acceptable. `useMemo` is the wrong default because it
+is a render-time cache for derived values, not an ownership boundary for an
+external store instance. A local store is stateful infrastructure, so treat its
+identity as state.
+
+`useRef` can also hold a stable instance, but prefer `useState` unless a ref is
+needed for imperative setup. Keep store creation pure; sync changing route/query
+inputs into named store actions such as `resetForFeature(...)` or `init(...)`.
+
+## Independent Actions
+
+Complex workflows should be independent functions. Put surface-specific actions
+in `actions/*.ts`, or make them named store actions when they are tightly coupled
+to local feature state.
+
+The component wires hooks, route params, stores, analytics, and query helpers.
+The action owns the workflow: refetching through callbacks, reading the passed
+store if needed, calling pure helpers, performing browser side effects, and
+emitting analytics.
+
+Actions must not call React hooks. If a workflow needs a lot of context, pass
+the local store instance or a small dependency object rather than threading long
+prop chains through view components.
+
+Example:
+
+```ts
+await exportFeatureData({
+  capture,
+  fetchDetails,
+  projectId,
+  refetchSummary,
+  selectedIds,
+});
+```
+
+For substantial data shaping, export a pure helper next to the action so the
+transformation can be tested without rendering the page.
+
+## Store Shape
+
+Use immutable plain objects for keyed state that must be selector-friendly:
+
+```ts
+type FeatureStoreState = {
+  selectedIds: Record<string, true>;
+  activeId: string | null;
+  actions: {
+    toggleSelected: (id: string, selected: boolean) => void;
+    setActiveId: (id: string | null) => void;
+  };
+};
+```
+
+Avoid mutating `Set` or `Map` in place. If you use them, replace the whole
+instance when updating.
+
+## Anti-Patterns
+
+- A table/list component owns selection, filters, columns, routing, peek state,
+  batch actions, local dialogs, and expensive rendered cells.
+- Context provider `value` changes on row selection, hover, scroll, active row,
+  expanded row, or other high-frequency state.
+- Local feature state is promoted to a global store even though it only belongs
+  to one mounted page instance.
+- `useMemo` is used to own a local external store instance.
+- A shared `src/components/*` component calls a feature-scoped store hook. That
+  silently breaks other callers that do not mount the feature provider.
+- Memoization is the only fix. `memo` helps, but it does not fix a bad state
+  boundary.
+- A callback depends on an inline config object and changes identity on every
+  render.
+- Components subscribe to large objects when they only need a boolean.
+- `useEffect` derives ordinary UI state from fetched data.
+- A component passes a long chain of feature context through props just so a
+  button can perform an action.
+- A page component keeps a complex async workflow inline because all the hooks
+  happen to be in scope there.
+
+## Migration Steps
+
+1. Instrument first: identify which semantic state change causes broad renders.
+2. Choose the smallest useful boundary: store state, action workflow, pure data
+   preparation, or imperative integration hook.
+3. Extract that one state group into a local store only when selective
+   subscriptions are needed.
+4. Replace broad props with selector subscriptions at the smallest UI boundary.
+5. Move related mutations into named actions.
+6. Stabilize callbacks and data wrappers.
+7. Move expensive data preparation into pure functions.
+8. Move complex user actions into store actions or external functions under
+   `actions/*.ts`.
+9. Update the feature README with what improved, what remains spread, and the
+   next atomic slice.
+10. Remove debug instrumentation.
+11. Repeat for the next state group.
@@ -0,0 +1,81 @@
+# Virtualized Lists
+
+Virtualized lists are render-boundary infrastructure. They should calculate
+which item shells are visible and position those shells. They should not make
+each row own feature state, effects, subscriptions, data loading, and workflows.
+
+Langfuse uses `@tanstack/react-virtual` for virtualization. Find current
+callsites before changing a virtualized surface, then apply the same
+state-boundary rules as any large feature.
+
+## Smartness Trap
+
+The broken shape is:
+
+- virtualizer rerenders on scroll
+- parent recreates callbacks, config, row wrappers, or data objects
+- row components receive changed props even though the semantic row did not
+  change
+- row-local effects/load state reset or refire
+- dynamic measurement observes DOM changed by an external mutator such as Google
+  Translate
+- measurement updates virtualizer state, which rerenders the same rows again
+
+That is not a small memoization bug. It is leaked state ownership.
+
+The fix is to make scroll and measurement state update the smallest possible
+integration boundary. If scrolling changes a virtual item offset, unchanged row
+content should not receive new semantic props, refire effects, or recreate
+expensive derived data.
+
+## Google Translate DOM Behavior
+
+Google Translate mutates rendered DOM after React has committed it. It can wrap
+text nodes, replace text, and change element dimensions outside React's data
+flow. React and TanStack Virtual do not know whether the changed DOM represents
+stable translated content or a transient mutation.
+
+Do not opt product UI out of translation with `translate="no"` unless product
+explicitly chooses that. Langfuse must work under browser translation.
+
+## Measurement Rules
+
+- Always put the correct `data-index` on the row element TanStack treats as the
+  item.
+- Do not combine live `measureElement` with externally mutated translated DOM
+  in text-heavy rows.
+- Prefer fixed estimates plus overscan for simple rows.
+- For dynamic text-heavy rows, use controlled measurement:
+  - `ResizeObserver` reads the row shell.
+  - Debounce commits.
+  - Do not commit while actively scrolling.
+  - Round heights to avoid sub-pixel churn.
+  - Call `virtualizer.resizeItem(index, height)`.
+  - If a row alternates between two heights repeatedly, clamp to a minimum
+    height that still allows later legitimate growth.
+
+## Row Rules
+
+- The virtualizer owns positioning only.
+- State that must survive remounts lives outside the row instance.
+- Expensive row content should be a memoized view component.
+- Narrow row containers may subscribe to local store slices and queries.
+- View components should receive stable props and perform no effects.
+- Feature-scoped row containers belong under `src/features/*`; shared
+  `src/components/*` row exports should be context-free.
+- Scrolling may rerender the virtualizer. It should not rerender unchanged
+  expensive row content.
+- Do not use a global store to preserve row state across virtualization. Use a
+  view-scoped store owned by the mounted list/page instance.
+
+## Migration Steps
+
+1. Add temporary logs to identify whether scroll causes remounts, prop changes,
+   measurement loops, or query refetches.
+2. Remove logs before shipping.
+3. Move row-local state that must survive virtualization into a local store.
+4. Stabilize callbacks and config objects passed to rows.
+5. Replace live `measureElement` with fixed estimates or controlled measurement.
+6. Move row logic into pure helpers or feature-local containers.
+7. Verify with browser translation enabled, horizontal resize, and small
+   vertical scroll deltas.
@@ -0,0 +1,51 @@
+---
+name: git-workflow
+description: |
+  Langfuse repo Git, GitHub, commit, branch, pull request, issue search,
+  release, and production-promotion workflow. Use when staging, committing,
+  pushing, opening PRs, searching GitHub issues, or changing release/promotion
+  behavior.
+---
+
+# Git Workflow
+
+Use this skill for repo-specific Git, GitHub, pull request, and release
+operations.
+
+## Safety
+
+- Inspect `git status` before staging or committing.
+- Do not stage unrelated working-tree changes.
+- Do not revert unrelated working-tree changes.
+- Do not use destructive commands such as `git reset --hard` or
+  `git checkout --` unless explicitly requested.
+- Keep commits focused and atomic.
+- Never add secrets or credentials to the repo.
+
+## Commits and Pull Requests
+
+- Commit messages and PR titles must follow Conventional Commits:
+  `type(scope): description` or `type: description`.
+- Use `feat` for new features and `fix` for bug fixes.
+- Use a scope when it clarifies the affected area, for example
+  `fix(api): handle missing trace id`.
+- Mark breaking changes with `!` in the type/scope or a `BREAKING CHANGE:`
+  footer.
+- PR titles are validated by `.github/workflows/validate-pr-title.yml`.
+- In PR descriptions, list impacted packages and executed verification
+  commands.
+
+## GitHub
+
+- Use `gh search issues` for GitHub issue search.
+- Prefer non-interactive Git and GitHub commands where possible.
+- Keep PRs narrow enough to review without unrelated refactors.
+
+## Release
+
+- Release workflow is managed at root with `pnpm run release`.
+- Promote `main` to `production` via
+  `.github/workflows/promote-main-to-production.yml` or
+  `pnpm run release:cloud`.
+- Do not change release/versioning flow without updating this skill and the
+  impacted package guides.
@@ -0,0 +1,148 @@
+---
+name: housekeeping
+description: Review an engineer's recurring work queue across Linear, Pylon, and GitHub. Use when asked to triage assigned issues, waiting or urgent work, customer-support follow-ups, stale tickets, or pull requests where the engineer is a direct or team reviewer; return concrete action recommendations and gate all writes on explicit human approval.
+---
+
+# Housekeeping
+
+## Scope
+
+Review live work queues and recommend what the engineer should do next. Stay read-only until the human explicitly approves specific writes.
+
+Use this skill for:
+
+- Linear issue triage, assigned work, waiting issues, urgent-priority issues, and stale cleanup candidates.
+- Pylon customer issues that need an engineer response, engineering follow-up, linked-ticket check, or stale close recommendation.
+- GitHub pull requests where the engineer is a direct or team reviewer.
+
+Do not use this skill for deep root-cause debugging, implementation, incident response, or broad roadmap planning unless the user explicitly asks to continue from a recommendation into that work.
+
+## Required Behavior
+
+Every reviewed item must include an action recommendation. If evidence is insufficient, recommend the next inspection step rather than leaving the action blank.
+
+Use these action labels:
+
+- `act-now`: needs attention today.
+- `quick-win`: likely under 15 minutes.
+- `todo`: should be picked up in the next 1-2 weeks.
+- `backlog`: real but can wait months.
+- `waiting`: next move is outside the engineer or team.
+- `stale-candidate`: likely safe to close or cancel, but needs human confirmation.
+- `no-action`: no current action beyond monitoring; explain why.
+
+Include confidence for each recommendation: `high`, `medium`, `low`, or `unknown`.
+
+## Workflow
+
+1. Identify the current engineer in each system from the available connector, CLI, or authenticated API context. If identity cannot be determined for a system, say so and continue with the other systems.
+2. Gather live data before recommending action. Do not rely on memory for current queues.
+3. Inspect comments, latest activity, linked issues, checks, and review threads when status or ownership is ambiguous.
+4. Cluster duplicates or related items before prioritizing.
+5. Return recommendations ordered by urgency, then quick wins, then cleanup.
+6. Before any write, present a decision table and wait for explicit row IDs and actions.
+
+## Linear Review
+
+Fetch:
+
+- Issues assigned to the engineer in `Triage`, `In Progress`, and `Waiting`.
+- Active urgent-priority issues relevant to the engineer or team.
+- Recent comments for waiting, stale, urgent, or ambiguous issues.
+
+Recommend:
+
+- `act-now` for data loss, production regressions, security/privacy risk, billing/cost correctness, repeated customer pain, blocked teammate/release, or reporter waiting on the engineer.
+- `todo` for bounded fixes with current customer impact.
+- `backlog` for real but lower-impact work, product-design work, or upstream-dependent work.
+- `waiting` only when the latest evidence shows the next step is on the customer, upstream, another team, or an external dependency.
+- `stale-candidate` when the issue has no recent meaningful activity, no clear current customer blocker, and appears superseded, abandoned, solved, or duplicated.
+
+If a triage issue has an obvious low-risk fix, include the quick-fix path and whether it should be handled before broader prioritization.
+
+## Pylon Review
+
+Use a Pylon connector, MCP server, or authenticated API only if already available. Do not ask the user to paste secrets into chat. If Pylon is unavailable, report that limitation and continue.
+
+Review open issues assigned to the engineer or their team, plus urgent or high-priority issues where engineering appears to own the next step. Use Pylon's issue states:
+
+- `new`: no team response yet.
+- `waiting_on_you`: next action is on the team.
+- `waiting_on_customer`: next action is on the customer.
+- `on_hold`: pending external work, commonly an engineering fix.
+- `closed`: resolved; include only if it reopened or is linked from an active item.
+
+Inspect:
+
+- latest customer and internal activity;
+- priority, requester/account, assignee/team, source, and age;
+- linked Linear, GitHub, Jira, or other external issues;
+- whether the linked engineering issue is still open, completed, stale, or missing.
+
+Recommend:
+
+- `act-now` when the customer is waiting on the team, priority is urgent/high, an SLA looks at risk, or a linked engineering issue is complete and the customer needs an update.
+- `quick-win` for a short reply, clarification request, link repair, or status correction.
+- `waiting` when the latest customer-facing state correctly waits on the customer or an external ticket.
+- `todo` when engineering owns a real follow-up but it is not same-day urgent.
+- `stale-candidate` for old `waiting_on_customer` or `on_hold` issues with no meaningful recent activity, but never close them without human approval.
+
+For Pylon issues linked to Linear or GitHub, make the recommended action consistent across systems. Example: if a Linear issue is done and Pylon is still `on_hold`, recommend a customer update and status change instead of more engineering work.
+
+## GitHub Review
+
+Find open pull requests where the engineer is requested as a direct reviewer and where one of the engineer's teams is requested. Include PRs across relevant organization repositories, not only the current repository.
+
+Inspect:
+
+- PR title, repo, age, author, requested reviewer source, mergeability, review decision, and checks;
+- changed files and diff size;
+- unresolved comments, bot findings, requested changes, and author responses;
+- whether failures are code failures or external authorization/noise.
+
+Recommend:
+
+- `quick-win` with `approve` only for small focused diffs, acceptable checks, no unresolved material concerns, and tests or plainly trivial behavior.
+- `quick-win` with `comment` for small PRs needing one narrow author action such as rebase, CLA, missing test, or cleanup.
+- `act-now` for blocked releases, security fixes, production regressions, or PRs where the engineer is the bottleneck.
+- `todo` for meaningful PRs that need real review soon.
+- `stale-candidate` for old, conflicting, duplicate, or superseded PRs.
+- `no-action` when a PR is blocked on the author, failing CLA, merge conflicts, unresolved requested changes, or unrelated team ownership.
+
+Do not approve, request changes, comment, close, merge, or edit a PR without explicit human approval for that PR.
+
+## Human Gates
+
+All writes require explicit human confirmation by row ID. This includes:
+
+- Linear comments, status changes, priority changes, assignee changes, labels, cancellation, or customer-need changes.
+- Pylon replies, internal notes, status changes, assignment, tags, snoozes, closes, or external-issue links.
+- GitHub approvals, comments, requested changes, reviewer changes, closes, merges, labels, or branch actions.
+
+If the user says "do the quick ones", first show the exact proposed writes and ask for confirmation unless they already named the exact row IDs.
+
+Use this table before writes:
+
+| ID | System | Item | Recommended Action | Proposed Write | Confidence | Human Decision |
+| --- | --- | --- | --- | --- | --- | --- |
+
+## Output
+
+Return valid Markdown. Keep the overview concise and action-first.
+
+Default structure:
+
+1. `Top Actions`: the highest-priority items across all systems.
+2. `Quick Wins`: items likely under 15 minutes.
+3. `Full Queue`: grouped by Linear, Pylon, and GitHub.
+4. `Decisions Needed`: stale closes, cancellations, comments, approvals, or status changes that require approval.
+
+For each item include:
+
+- system and link or identifier;
+- title or short description;
+- evidence from current data;
+- action recommendation;
+- confidence.
+
+Use concrete dates for age and stale reasoning. Avoid vague phrases like "recently" when exact timestamps are available.
@@ -0,0 +1,4 @@
+interface:
+  display_name: "Housekeeping"
+  short_description: "Review issues, support, and PR queues"
+  default_prompt: "Use $housekeeping to review my Linear, Pylon, and GitHub review queues with recommended actions."
@@ -0,0 +1,116 @@
+---
+name: linear-bug-triage
+description: |
+  Deduplicate measured bug or regression evidence against Linear, then either
+  add evidence comments to related existing issues or create concise Linear bug
+  issues in Triage. Use when Codex has confirmed evidence from Datadog,
+  benchmarks, traces, timings, flamegraphs, logs, or production measurements and
+  needs Linear search, comments, labels, Datadog links, or bug ticket creation
+  without fix suggestions, but only after a human has approved sharing the
+  findings in Linear.
+---
+
+# Linear Bug Triage
+
+Use this skill after a bug or regression candidate has measured evidence. This
+skill owns Linear search, deduplication, evidence comments, and ticket creation;
+the calling skill owns deciding whether the signal is issue-worthy.
+
+## Human Approval Gate
+
+Before doing anything in Linear, first show the findings to the human in a
+compact markdown table and ask for explicit permission to share them in Linear.
+The table should include one row per candidate with:
+
+- Candidate / cluster name.
+- Environments.
+- Service and route/resource.
+- Recent window measurement.
+- Baseline measurement.
+- Delta / regression summary.
+- Key Datadog evidence links.
+- Proposed Linear action (`comment existing`, `create new`, or `none`).
+
+If the human does not explicitly approve, stop after presenting the table. Do
+not search Linear, do not comment on issues, and do not create issues.
+
+If a calling workflow already showed the findings table and obtained explicit
+human approval for a Linear handoff, skip this gate and proceed directly to
+deduplication.
+
+## Required Evidence
+
+For each candidate, gather:
+
+- Recent window and baseline window as absolute time ranges with timezone.
+- Measured signal: counts, rates, p50/p95/p99 latency, trace samples,
+  flamegraphs, monitor thresholds, or benchmark deltas.
+- Affected environments, services, routes/resources, status codes, and top error
+  messages.
+- Datadog links for logs, spans, traces, metrics, dashboards, or flamegraphs
+  used as evidence.
+- The exact text `No measurements found` for requested measurements that are
+  unavailable.
+
+Do not create or comment based on guesses, unsupported impact claims, or missing
+measurements alone.
+
+## Deduplication
+
+After the human explicitly approves, before creating a new issue:
+
+1. Search Linear for related open issues using exact error text, route/resource,
+   service, environment, monitor name, and Datadog link keywords.
+2. Search recently closed or canceled issues if the error is recurring or the
+   wording is distinctive.
+3. If a related issue exists, add a concise evidence comment instead of creating
+   a duplicate.
+4. If no related issue exists, create one Linear issue in the `Triage` state for
+   each distinct bug cluster.
+
+## Existing Issue Comments
+
+For related existing issues, add only:
+
+- Recent window and baseline window.
+- Measured delta or `No measurements found` for unavailable signals.
+- Affected environments, services, routes/resources, and top error messages.
+- Datadog links.
+
+Do not add fix suggestions, root-cause guesses, implementation notes, owner
+assignments, or next steps.
+
+## New Issue Format
+
+Create new issues with:
+
+- State/status `Triage`; pass the Linear state explicitly on creation and do not
+  rely on workspace defaults.
+- Label `bug`.
+- Additional existing labels that match the evidence, such as affected service,
+  environment, API, ingestion, latency, ClickHouse, Postgres, integrations, or
+  observability labels. Query labels first and use the repository/team's exact
+  label names.
+- Concise title: `bug: <service or route> <measured symptom> in <envs>`.
+- Concise body, evidence-only:
+
+```markdown
+Recent window: <absolute time range and timezone>
+Baseline: <absolute time range and timezone>
+
+Signal:
+- <count/rate/latency delta with env/service/route>
+- <"No measurements found" for missing requested measurements>
+
+Evidence:
+- Datadog logs: <url>
+- Datadog spans/traces: <url>
+- Datadog metrics, dashboard, or latency graph: <url>
+
+Related Linear search:
+- <brief search terms used and result>
+```
+
+Do not include fix suggestions, root-cause guesses, implementation notes, owner
+assignments, or next steps unless the user explicitly asks outside the Linear
+issue or comment.
@@ -0,0 +1,4 @@
+interface:
+  display_name: "Linear Bug Triage"
+  short_description: "Deduplicate and file Linear bug evidence"
+  default_prompt: "Use $linear-bug-triage to deduplicate measured bug evidence and create or comment Linear triage issues."
@@ -0,0 +1,74 @@
+---
+name: pnpm-upgrade-package
+description: >-
+  Upgrade pnpm workspace dependencies to target/latest versions:
+  direct/transitive bumps, release-age checks, temporary overrides,
+  minimumReleaseAgeExclude, lockfile/dedupe verification.
+---
+
+# PNPM Upgrade Package
+
+Use this skill for interactive dependency bumps in Langfuse.
+
+## Read Order
+
+- Use this `SKILL.md` for the end-to-end workflow.
+- Run the main helper once at the start of the upgrade:
+  `node .agents/skills/pnpm-upgrade-package/scripts/check-release-age-window.mjs <package> [targetVersion]`
+
+## Apply This Skill
+
+- Ask for the package name if the user did not provide one.
+- Ask for the target version if the user did not provide one.
+- Run the main helper once as the first analysis step and use that single
+  output for scope, exclusion decisions, and the final bump.
+- If the target package is not directly declared anywhere, run
+  `pnpm why -r <package>` to find which direct dependency brings it in, then
+  inspect whether the current top-level parent already allows the requested
+  transitive version via its dependency range.
+- If the current parent range already covers the requested transitive version,
+  prefer a lockfile refresh / reinstall path over bumping the parent manifest.
+- If the current parent range does not cover the requested transitive version,
+  upgrade that parent dependency instead of adding the target package directly
+  unless the user explicitly wants that.
+- If pnpm will not move an already-allowed transitive version, a scoped
+  `overrides` entry in `pnpm-workspace.yaml` may be used as a temporary
+  resolution tool. Before finishing, prove whether the override is still
+  required: remove it, run `pnpm install`, then run `pnpm dedupe`. Inspect the
+  diff after each generated change. If the target version remains without the
+  override, do not keep the override; keep or restore it only when pnpm reverts
+  or drifts from the requested version without it.
+- Never manually edit `pnpm-lock.yaml`; regenerate lockfile changes with
+  `pnpm` commands only. If a lockfile-only refresh causes unrelated churn,
+  adjust the pnpm command and rerun instead of patching the lockfile by hand.
+- After fixing or upgrading a package, run `pnpm dedupe`. Always inspect the
+  diff after dedupe and revert that generated attempt if it introduces
+  unrelated churn.
+- Resolve the registry latest version, but do not silently upgrade to latest
+  unless the user asked for latest.
+- Compare the target version with the latest version installable under the
+  current `minimumReleaseAge` window.
+- Ask before adding `minimumReleaseAgeExclude` entries for the target package,
+  exact dependency companions from `dependencies` or `optionalDependencies`, or
+  locally installed exact peer dependencies.
+- Finish with `pnpm why -r <package>` to confirm that only the intended version
+  remains in the workspace.
+
+## Quick Commands
+
+- Analysis pass:
+  `node .agents/skills/pnpm-upgrade-package/scripts/check-release-age-window.mjs <package> <targetVersion>`
+- Transitive provenance / final graph verification:
+  `pnpm why -r <package>`
+- Inspect a current parent manifest on the registry:
+  `npm view <parent>@<installedVersion> dependencies peerDependencies optionalDependencies --json`
+- Optional lockfile cleanup:
+  `pnpm dedupe`
+- Bump in the root workspace:
+  `pnpm -w up <package>@<version>`
+- Bump in one workspace:
+  `pnpm --filter web up <package>@<version>`
+- Bump everywhere that should move together:
+  `pnpm -r up <package>@<version>`
+- Verify temporary override removal:
+  remove the override, then run `pnpm install` and `pnpm dedupe`
@@ -0,0 +1,4 @@
+interface:
+  display_name: "PNPM Upgrade Package"
+  short_description: "Interactive pnpm package bump workflow"
+  default_prompt: "Use $pnpm-upgrade-package to upgrade a package in this pnpm workspace, asking me for the package or version if I did not provide them."
@@ -0,0 +1,418 @@
+#!/usr/bin/env node
+
+import { join } from "node:path";
+import {
+  entryCoversVersion,
+  findLocalPackageReferences,
+  formatWorkspaceReference,
+  getRootPnpmControls,
+  readWorkspaceConfig,
+} from "./lib/workspace-utils.mjs";
+
+const args = process.argv.slice(2);
+const asJson = args.includes("--json");
+const positional = args.filter((arg) => !arg.startsWith("--"));
+const packageName = positional[0];
+const requestedTargetVersion = positional[1] ?? null;
+
+if (!packageName) {
+  console.error(
+    "Usage: node .agents/skills/pnpm-upgrade-package/scripts/check-release-age-window.mjs <package> [targetVersion] [--json]",
+  );
+  process.exit(1);
+}
+
+const repoRoot = process.cwd();
+const workspaceConfig = readWorkspaceConfig(join(repoRoot, "pnpm-workspace.yaml"));
+const minimumReleaseAgeMinutes = workspaceConfig.minimumReleaseAge ?? 0;
+const thresholdMs = Date.now() - minimumReleaseAgeMinutes * 60 * 1000;
+const REGISTRY_FETCH_TIMEOUT_MS = 30_000;
+const registryCache = new Map();
+const workspaceReferenceCache = new Map();
+
+const getWorkspaceReferences = (name) => {
+  if (!workspaceReferenceCache.has(name)) {
+    workspaceReferenceCache.set(name, findLocalPackageReferences(repoRoot, name));
+  }
+
+  return workspaceReferenceCache.get(name);
+};
+
+function printSectionHeader(title) {
+  console.log("");
+  console.log(title);
+}
+
+function isPrerelease(version) {
+  return version.includes("-");
+}
+
+function isExactVersion(spec) {
+  return /^\d+\.\d+\.\d+(?:[-+][0-9A-Za-z.-]+)?$/.test(spec.trim());
+}
+
+function getMatchingExcludeEntries(name, version) {
+  return workspaceConfig.minimumReleaseAgeExclude.filter((entry) =>
+    entryCoversVersion(entry, name, version),
+  );
+}
+
+async function fetchRegistryPackage(name) {
+  if (registryCache.has(name)) return registryCache.get(name);
+
+  const abortController = new AbortController();
+  const timeoutId = setTimeout(
+    () => abortController.abort(),
+    REGISTRY_FETCH_TIMEOUT_MS,
+  );
+  timeoutId.unref?.();
+
+  try {
+    const response = await fetch(
+      `https://registry.npmjs.org/${encodeURIComponent(name)}`,
+      {
+        headers: {
+          accept: "application/json",
+          "user-agent": "langfuse-pnpm-upgrade-package-skill",
+        },
+        signal: abortController.signal,
+      },
+    );
+
+    if (!response.ok) {
+      throw new Error(
+        `Failed to fetch ${name} from npm registry: ${response.status}`,
+      );
+    }
+
+    const metadata = await response.json();
+    registryCache.set(name, metadata);
+    return metadata;
+  } catch (error) {
+    if (error?.name === "AbortError") {
+      throw new Error(
+        `Timed out fetching ${name} from npm registry after ${REGISTRY_FETCH_TIMEOUT_MS}ms`,
+        { cause: error },
+      );
+    }
+
+    throw error;
+  } finally {
+    clearTimeout(timeoutId);
+  }
+}
+
+function getInstallability(metadata, name, version) {
+  const publishedAt = metadata.time?.[version] ?? null;
+  const publishedAtMs = publishedAt ? Date.parse(publishedAt) : null;
+  const matchingExcludeEntries = getMatchingExcludeEntries(name, version);
+  const isYoungerThanMinimumReleaseAge =
+    publishedAtMs != null ? publishedAtMs > thresholdMs : null;
+  const isInstallableWithoutNewExclude =
+    isYoungerThanMinimumReleaseAge == null
+      ? null
+      : !isYoungerThanMinimumReleaseAge || matchingExcludeEntries.length > 0;
+
+  return {
+    name,
+    version,
+    publishedAt,
+    isYoungerThanMinimumReleaseAge,
+    isInstallableWithoutNewExclude,
+    matchingExcludeEntries,
+    suggestedExclude:
+      isInstallableWithoutNewExclude === false ? `${name}@${version}` : null,
+  };
+}
+
+function selectLatestInstallableVersion(metadata, name) {
+  const times = metadata.time ?? {};
+
+  return (
+    Object.keys(metadata.versions ?? {})
+      .filter((version) => times[version] && !isPrerelease(version))
+      .sort((left, right) => Date.parse(times[right]) - Date.parse(times[left]))
+      .map((version) => getInstallability(metadata, name, version))
+      .find((candidate) => candidate.isInstallableWithoutNewExclude) ?? null
+  );
+}
+
+function collectManifestEntries(manifest, fields) {
+  const merged = new Map();
+
+  for (const field of fields) {
+    for (const [name, spec] of Object.entries(manifest[field] ?? {})) {
+      const key = `${name}:${spec}`;
+      const entry = merged.get(key);
+
+      if (entry) {
+        entry.fields.push(field);
+        continue;
+      }
+
+      merged.set(key, { name, spec, fields: [field] });
+    }
+  }
+
+  return [...merged.values()].sort((left, right) =>
+    left.name.localeCompare(right.name),
+  );
+}
+
+async function analyzeManifestEntries(entries, { includeWorkspace = false } = {}) {
+  const exact = [];
+  const range = [];
+
+  for (const entry of entries) {
+    const workspaceReferences = includeWorkspace
+      ? getWorkspaceReferences(entry.name)
+      : null;
+
+    if (!isExactVersion(entry.spec)) {
+      range.push({
+        ...entry,
+        ...(includeWorkspace ? { workspaceReferences } : {}),
+      });
+      continue;
+    }
+
+    const metadata = await fetchRegistryPackage(entry.name);
+    const installability = getInstallability(metadata, entry.name, entry.spec);
+
+    exact.push({
+      ...entry,
+      ...installability,
+      ...(includeWorkspace
+        ? {
+            workspaceReferences,
+            isInstalledInWorkspace: workspaceReferences.length > 0,
+          }
+        : {}),
+      suggestedExclude:
+        includeWorkspace && workspaceReferences.length === 0
+          ? null
+          : installability.suggestedExclude,
+    });
+  }
+
+  return { exact, range };
+}
+
+function printWorkspaceReferences(title, references) {
+  printSectionHeader(title);
+  if (references.length === 0) {
+    console.log("- none");
+    return;
+  }
+
+  for (const reference of references) {
+    console.log(`- ${formatWorkspaceReference(reference)}`);
+  }
+}
+
+function printRootPnpmControls(rootPnpm) {
+  printSectionHeader("Root pnpm controls:");
+  if (
+    rootPnpm.overrideMatches.length === 0 &&
+    rootPnpm.patchedDependencyMatches.length === 0
+  ) {
+    console.log("- none");
+    return;
+  }
+
+  for (const match of rootPnpm.overrideMatches) {
+    console.log(`- override ${match.selector}: ${match.value}`);
+  }
+  for (const match of rootPnpm.patchedDependencyMatches) {
+    console.log(`- patched dependency ${match.selector}: ${match.value}`);
+  }
+}
+
+function printVersionEntries(title, entries, { includeWorkspace = false } = {}) {
+  printSectionHeader(title);
+  if (entries.length === 0) {
+    console.log("- none");
+    return;
+  }
+
+  for (const entry of entries) {
+    const status =
+      entry.isInstallableWithoutNewExclude == null
+        ? "unknown"
+        : entry.isInstallableWithoutNewExclude
+          ? "installable now"
+          : "needs exclude";
+
+    console.log(
+      `- ${entry.name}@${entry.version} (${status}; via ${entry.fields.join(", ")})`,
+    );
+    if (entry.publishedAt) {
+      console.log(`  published at: ${entry.publishedAt}`);
+    }
+    if (includeWorkspace) {
+      console.log(
+        `  installed in workspace: ${entry.isInstalledInWorkspace ? "yes" : "no"}`,
+      );
+      for (const reference of entry.workspaceReferences) {
+        console.log(`  workspace reference: ${formatWorkspaceReference(reference)}`);
+      }
+    }
+    if (entry.matchingExcludeEntries.length > 0) {
+      console.log(
+        `  matching exclude entries: ${entry.matchingExcludeEntries.join(", ")}`,
+      );
+    }
+    if (entry.suggestedExclude) {
+      console.log(`  suggested exclude: ${entry.suggestedExclude}`);
+    }
+  }
+}
+
+function printRangeEntries(title, entries) {
+  printSectionHeader(title);
+  if (entries.length === 0) {
+    console.log("- none");
+    return;
+  }
+
+  for (const entry of entries) {
+    console.log(
+      `- ${entry.name}: ${entry.spec} (manual review; via ${entry.fields.join(", ")})`,
+    );
+    for (const reference of entry.workspaceReferences ?? []) {
+      console.log(`  workspace reference: ${formatWorkspaceReference(reference)}`);
+    }
+  }
+}
+
+const packageMetadata = await fetchRegistryPackage(packageName);
+const latestVersion = packageMetadata["dist-tags"]?.latest ?? null;
+const targetVersion = requestedTargetVersion ?? latestVersion;
+
+if (!targetVersion) {
+  console.error(`Could not resolve a target version for ${packageName}.`);
+  process.exit(1);
+}
+
+if (!packageMetadata.versions?.[targetVersion]) {
+  console.error(`Version ${targetVersion} was not found for ${packageName}.`);
+  process.exit(1);
+}
+
+const packageWorkspaceReferences = getWorkspaceReferences(packageName);
+const rootPnpm = getRootPnpmControls(repoRoot, packageName);
+const latestInstallableWithoutNewExclude = selectLatestInstallableVersion(
+  packageMetadata,
+  packageName,
+);
+const targetInstallability = getInstallability(
+  packageMetadata,
+  packageName,
+  targetVersion,
+);
+const targetManifest = packageMetadata.versions[targetVersion];
+
+const dependencyCompanions = await analyzeManifestEntries(
+  collectManifestEntries(targetManifest, [
+    "dependencies",
+    "optionalDependencies",
+  ]),
+);
+const peerDependencies = await analyzeManifestEntries(
+  collectManifestEntries(targetManifest, ["peerDependencies"]),
+  { includeWorkspace: true },
+);
+
+const result = {
+  packageName,
+  targetVersion,
+  targetWasExplicitlyProvided: requestedTargetVersion != null,
+  packageWorkspaceReferences,
+  rootPnpm,
+  minimumReleaseAgeMinutes,
+  thresholdIso: new Date(thresholdMs).toISOString(),
+  latestRegistryVersion: latestVersion,
+  latestRegistryPublishedAt:
+    latestVersion != null ? packageMetadata.time?.[latestVersion] ?? null : null,
+  latestInstallableWithoutNewExclude,
+  targetPublishedAt: targetInstallability.publishedAt,
+  targetIsYoungerThanMinimumReleaseAge:
+    targetInstallability.isYoungerThanMinimumReleaseAge,
+  targetIsInstallableWithoutNewExclude:
+    targetInstallability.isInstallableWithoutNewExclude,
+  matchingPackageExcludeEntries: targetInstallability.matchingExcludeEntries,
+  suggestedPackageExclude: targetInstallability.suggestedExclude,
+  exactDependencyCompanions: dependencyCompanions.exact,
+  rangeDependencyCompanions: dependencyCompanions.range,
+  exactPeerDependencies: peerDependencies.exact,
+  rangePeerDependencies: peerDependencies.range,
+};
+
+if (asJson) {
+  console.log(JSON.stringify(result, null, 2));
+  process.exit(0);
+}
+
+console.log(`Package: ${packageName}`);
+console.log(
+  `Target version: ${targetVersion}${
+    requestedTargetVersion
+      ? ""
+      : " (resolved latest; still ask before bumping if version was omitted)"
+  }`,
+);
+
+printWorkspaceReferences(
+  "Target package workspace references:",
+  packageWorkspaceReferences,
+);
+printRootPnpmControls(rootPnpm);
+
+printSectionHeader("Release-age window:");
+console.log(`minimumReleaseAge: ${minimumReleaseAgeMinutes} minutes`);
+console.log(`Threshold: ${result.thresholdIso}`);
+console.log(`Latest registry version: ${result.latestRegistryVersion ?? "unknown"}`);
+if (result.latestRegistryPublishedAt) {
+  console.log(`Latest registry published at: ${result.latestRegistryPublishedAt}`);
+}
+if (latestInstallableWithoutNewExclude) {
+  console.log(
+    `Latest installable without new exclude: ${latestInstallableWithoutNewExclude.version} (${latestInstallableWithoutNewExclude.publishedAt})`,
+  );
+} else {
+  console.log("Latest installable without new exclude: none found");
+}
+console.log(`Target published at: ${result.targetPublishedAt ?? "unknown"}`);
+console.log(
+  `Target installable without new exclude: ${
+    result.targetIsInstallableWithoutNewExclude == null
+      ? "unknown"
+      : result.targetIsInstallableWithoutNewExclude
+        ? "yes"
+        : "no"
+  }`,
+);
+if (result.matchingPackageExcludeEntries.length > 0) {
+  console.log("Matching package exclude entries:");
+  for (const entry of result.matchingPackageExcludeEntries) {
+    console.log(`- ${entry}`);
+  }
+} else {
+  console.log("Matching package exclude entries: none");
+}
+if (result.suggestedPackageExclude) {
+  console.log(`Suggested package exclude: ${result.suggestedPackageExclude}`);
+}
+
+printVersionEntries(
+  "Exact dependency companions (dependencies + optionalDependencies):",
+  result.exactDependencyCompanions,
+);
+printRangeEntries(
+  "Range dependency companions (dependencies + optionalDependencies):",
+  result.rangeDependencyCompanions,
+);
+printVersionEntries("Exact peer dependencies:", result.exactPeerDependencies, {
+  includeWorkspace: true,
+});
+printRangeEntries("Range peer dependencies:", result.rangePeerDependencies);
@@ -0,0 +1,244 @@
+import { existsSync, readFileSync, readdirSync } from "node:fs";
+import { join, relative } from "node:path";
+
+const packageFields = [
+  "dependencies",
+  "devDependencies",
+  "peerDependencies",
+  "optionalDependencies",
+];
+
+export function formatWorkspaceReference(reference) {
+  const label = reference.workspaceName
+    ? `${reference.path} (${reference.workspaceName})`
+    : reference.path;
+  const specs = reference.matches
+    .map((match) => `${match.field}: ${match.spec}`)
+    .join(", ");
+
+  return `${label} -> ${specs}`;
+}
+
+export function readJson(path) {
+  return JSON.parse(readFileSync(path, "utf8"));
+}
+
+function stripInlineComment(line) {
+  let quote = null;
+  let escaped = false;
+
+  for (let index = 0; index < line.length; index += 1) {
+    const char = line[index];
+
+    if (escaped) {
+      escaped = false;
+      continue;
+    }
+
+    if (quote) {
+      if (char === "\\") {
+        escaped = true;
+        continue;
+      }
+
+      if (char === quote) {
+        quote = null;
+      }
+      continue;
+    }
+
+    if (char === "'" || char === '"') {
+      quote = char;
+      continue;
+    }
+
+    if (char === "#") {
+      return line.slice(0, index).trimEnd();
+    }
+  }
+
+  return line;
+}
+
+function unquoteYamlScalar(value) {
+  return value.trim().replace(/^['"]|['"]$/g, "");
+}
+
+export function readWorkspaceConfig(path) {
+  const raw = readFileSync(path, "utf8");
+  const lines = raw.split(/\r?\n/);
+  let minimumReleaseAge = 0;
+  const minimumReleaseAgeExclude = [];
+  const overrides = {};
+  const patchedDependencies = {};
+  let activeBlock = null;
+
+  for (const line of lines) {
+    const uncommented = stripInlineComment(line);
+    const trimmed = uncommented.trim();
+
+    if (!trimmed || trimmed.startsWith("#")) continue;
+
+    const ageMatch = trimmed.match(/^minimumReleaseAge:\s*(\d+)\s*$/);
+    if (ageMatch) {
+      minimumReleaseAge = Number(ageMatch[1]);
+      activeBlock = null;
+      continue;
+    }
+
+    if (/^minimumReleaseAgeExclude:\s*$/.test(trimmed)) {
+      activeBlock = "minimumReleaseAgeExclude";
+      continue;
+    }
+
+    if (/^overrides:\s*$/.test(trimmed)) {
+      activeBlock = "overrides";
+      continue;
+    }
+
+    if (/^patchedDependencies:\s*$/.test(trimmed)) {
+      activeBlock = "patchedDependencies";
+      continue;
+    }
+
+    if (/^\S/.test(uncommented)) {
+      activeBlock = null;
+      continue;
+    }
+
+    if (activeBlock === "minimumReleaseAgeExclude") {
+      const excludeMatch = uncommented.match(/^\s*-\s+(.+?)\s*$/);
+      if (excludeMatch) {
+        minimumReleaseAgeExclude.push(unquoteYamlScalar(excludeMatch[1]));
+      }
+      continue;
+    }
+
+    if (activeBlock === "overrides" || activeBlock === "patchedDependencies") {
+      const entryMatch = uncommented.match(/^\s+(.+?):\s+(.+?)\s*$/);
+      if (!entryMatch) continue;
+
+      const selector = unquoteYamlScalar(entryMatch[1]);
+      const value = unquoteYamlScalar(entryMatch[2]);
+      if (activeBlock === "overrides") {
+        overrides[selector] = value;
+      } else {
+        patchedDependencies[selector] = value;
+      }
+    }
+  }
+
+  return {
+    minimumReleaseAge,
+    minimumReleaseAgeExclude,
+    overrides,
+    patchedDependencies,
+  };
+}
+
+export function collectPackageJsonPaths(repoRoot) {
+  const paths = [
+    "package.json",
+    "web/package.json",
+    "worker/package.json",
+    "ee/package.json",
+  ];
+  const packagesRoot = join(repoRoot, "packages");
+
+  if (!existsSync(packagesRoot)) return paths;
+
+  const stack = [packagesRoot];
+  while (stack.length > 0) {
+    const current = stack.pop();
+    for (const entry of readdirSync(current, { withFileTypes: true })) {
+      if (
+        entry.name === "node_modules" ||
+        entry.name === "dist" ||
+        entry.name === ".git"
+      ) {
+        continue;
+      }
+
+      const nextPath = join(current, entry.name);
+      if (entry.isDirectory()) {
+        stack.push(nextPath);
+        continue;
+      }
+
+      if (entry.isFile() && entry.name === "package.json") {
+        paths.push(relative(repoRoot, nextPath));
+      }
+    }
+  }
+
+  return [...new Set(paths)];
+}
+
+export function matchesPackageSelector(selector, wantedPackage) {
+  if (selector === wantedPackage) return true;
+  if (selector.startsWith(`${wantedPackage}@`)) return true;
+  if (selector.endsWith(`>${wantedPackage}`)) return true;
+  if (selector.includes(`>${wantedPackage}@`)) return true;
+  if (selector.endsWith("/*")) {
+    const prefix = selector.slice(0, -1);
+    return wantedPackage.startsWith(prefix);
+  }
+  return false;
+}
+
+export function entryCoversVersion(entry, wantedPackage, wantedVersion) {
+  if (entry === wantedPackage) return true;
+  if (entry.endsWith("/*")) {
+    const prefix = entry.slice(0, -1);
+    return wantedPackage.startsWith(prefix);
+  }
+  if (!entry.startsWith(`${wantedPackage}@`)) return false;
+
+  return entry
+    .slice(wantedPackage.length + 1)
+    .split("||")
+    .map((part) => part.trim())
+    .includes(wantedVersion);
+}
+
+export function findLocalPackageReferences(repoRoot, wantedPackage) {
+  const results = [];
+
+  for (const packageJsonPath of collectPackageJsonPaths(repoRoot)) {
+    const json = readJson(join(repoRoot, packageJsonPath));
+    const matches = [];
+
+    for (const field of packageFields) {
+      if (json[field]?.[wantedPackage]) {
+        matches.push({ field, spec: json[field][wantedPackage] });
+      }
+    }
+
+    if (matches.length > 0) {
+      results.push({
+        path: packageJsonPath,
+        workspaceName: json.name ?? null,
+        matches,
+      });
+    }
+  }
+
+  return results;
+}
+
+export function getRootPnpmControls(repoRoot, packageName) {
+  const workspaceConfig = readWorkspaceConfig(
+    join(repoRoot, "pnpm-workspace.yaml"),
+  );
+
+  return {
+    overrideMatches: Object.entries(workspaceConfig.overrides)
+      .filter(([selector]) => matchesPackageSelector(selector, packageName))
+      .map(([selector, value]) => ({ selector, value })),
+    patchedDependencyMatches: Object.entries(
+      workspaceConfig.patchedDependencies,
+    )
+      .filter(([selector]) => matchesPackageSelector(selector, packageName))
+      .map(([selector, value]) => ({ selector, value })),
+  };
+}
@@ -0,0 +1,101 @@
+---
+name: security-review
+description: Security review patterns for Langfuse. Use during code review, design, or planning whenever a change accepts user-supplied URLs, host/endpoint/baseURL fields, secrets, cross-tenant data, new outbound HTTP requests, new integrations (webhooks, blob storage, LLM connections, image proxies), redirect-following behavior, or new auth/permission scopes. Covers SSRF/outbound URL validation today and is intentionally extensible to other recurring security findings (tenant isolation, secret handling, redirect mishandling, file upload, RBAC scope drift).
+---
+
+# Security Review
+
+Use this skill when reviewing or planning code that touches a security-sensitive
+surface in Langfuse. It collects the recurring findings the team has seen in
+external security reports so that future agents catch them at design and review
+time rather than after the fact.
+
+## When to Apply
+
+Apply this skill when the change touches any of:
+
+- a user-supplied URL, host, endpoint, `baseURL`, or webhook target
+- a new outbound HTTP request (`fetch`, `axios`, AWS SDK client init with a
+  custom `endpoint`, OpenAI/Anthropic/Bedrock client init with a custom
+  `baseURL`, etc.)
+- a new integration form under Settings -> Integrations or any
+  admin-configurable network destination
+- a new tRPC procedure or public API route that mutates project-scoped data or
+  changes who can access it
+- secrets, API keys, signing secrets, or encryption-at-rest fields
+- redirect-following or cross-origin header handling
+- file uploads, image proxies, or other binary data flowing in or out
+
+Apply this skill during **plan mode** when designing a new integration so the
+correct validation surfaces land in the plan, not in a follow-up CVE.
+
+## How to Read This Skill
+
+1. Open [references/checklist.md](references/checklist.md) and run the mental
+   sweep against the change.
+2. For each bullet that fires, open the matching topic reference.
+
+| Topic | Open when | File |
+| --- | --- | --- |
+| SSRF and outbound URL validation | The change accepts or fetches a user-supplied URL, host, or endpoint | [references/outbound-url-validation.md](references/outbound-url-validation.md) |
+
+The catalog is intentionally short today. New topic files are added as new
+finding classes recur (see "Extending This Skill").
+
+## Output Expectations (Review Mode)
+
+When this skill is used during code review:
+
+- List findings first, ordered by severity, with file and line references.
+- For each finding, name the canonical helper or known-good call site the
+  author should copy.
+- For SSRF-class findings, point at [references/outbound-url-validation.md](references/outbound-url-validation.md)
+  rather than re-deriving the fix.
+- Call out missing **negative tests** (private-IP, cross-tenant, missing-scope)
+  as findings, not as nice-to-haves.
+
+## Output Expectations (Design / Plan Mode)
+
+When this skill is used while planning:
+
+- Restate which surfaces the new feature exposes (forms, public API routes,
+  worker entrypoints).
+- For each surface that matches a checklist trigger, name the validator or
+  helper that must be invoked and at which layer (save-time, use-time,
+  connection-time, redirect-time).
+- Treat "we will validate later" as a design defect: validation belongs in the
+  same change that introduces the surface.
+
+## Extending This Skill
+
+Add a new `references/<topic>.md` whenever a security finding recurs across
+features or PR reviews. Keep each reference narrow and concrete:
+
+1. Threat in plain language (one paragraph).
+2. Canonical helpers in this repo, with paths.
+3. Known-good call sites that can be copied.
+4. Required defenses (save-time, use-time, transport-time, etc.).
+5. Anti-patterns to flag in review.
+
+Then add a one-line trigger to [references/checklist.md](references/checklist.md)
+pointing at the new topic file, and add a row to the table above.
+
+Candidates for future references (do not add until a real finding recurs):
+
+- Tenant isolation (`projectId` filters across Prisma and ClickHouse)
+- Secret handling and encryption-at-rest read paths
+- Redirect mishandling and sensitive-header propagation
+- File upload validation and content-type sniffing
+- RBAC scope drift on new tRPC/public API endpoints
+- Signed URL scoping (expiry, path, method)
+- Public API rate limiting and auth boundary checks
+
+## Integration With Other Skills
+
+- The shared `code-review` skill should defer here for any change that matches
+  the triggers above; see [code-review/SKILL.md](../code-review/SKILL.md).
+- The shared `backend-dev-guidelines` skill should defer here when adding
+  outbound HTTP, integration config, or URL-accepting procedures; see
+  [backend-dev-guidelines/SKILL.md](../backend-dev-guidelines/SKILL.md).
+- Confirmed issues with reproduction evidence go through `linear-bug-triage`
+  for Linear handoff.
@@ -0,0 +1,74 @@
+# Security Review Checklist
+
+Run this list mentally on every relevant change. For each bullet, either find
+that it does not apply, or confirm the listed mitigation is present in the
+diff. Each bullet links to the topic reference that owns the detail.
+
+## User-Supplied URLs and Outbound Requests
+
+- Does the change accept a URL, host, `endpoint`, `baseURL`, webhook target,
+  or any field that becomes the destination of an outbound HTTP request?
+  - If yes, see [outbound-url-validation.md](outbound-url-validation.md).
+  - Required: a save-time validator on the mutation/API route **and** use-time
+    or connection-time validation when the request is issued, including
+    redirects. Raw `fetch(userUrl)` or SDK init with `endpoint: userUrl`
+    without that wiring is a finding.
+
+- Does the change add a new "integration" (Settings -> Integrations, new
+  webhook destination, new storage backend, new LLM provider, new image
+  proxy) that lets an admin configure a host?
+  - If yes, see [outbound-url-validation.md](outbound-url-validation.md).
+  - Required: new env-driven allowlist trio (host / IPs / IP segments) for
+    self-hosted, plus strict Cloud enforcement.
+
+- Does the change follow redirects on a user-controlled URL with plain
+  `fetch()`?
+  - If yes, see [outbound-url-validation.md](outbound-url-validation.md)
+    (Redirect Handling). Required: `fetchWithSecureRedirects` with the
+    matching validator.
+
+## Tenant Isolation
+
+- Does every new Prisma query on a project-scoped table include `projectId`
+  in the `where` clause?
+- Does every new ClickHouse query on a project-scoped table include
+  `project_id = {projectId: String}`?
+- Does every new tRPC procedure use `protectedProjectProcedure` (or
+  equivalent) and call `throwIfNoProjectAccess` with the right scope?
+- Does every new public API route go through `withMiddlewares` +
+  `createAuthedProjectAPIRoute` (or the equivalent organization variant) with
+  the right scope?
+
+These are enforced today by the repo-wide review checklist in
+[../../code-review/references/review-checklist.md](../../code-review/references/review-checklist.md);
+restate them here so the security sweep stays self-contained.
+
+## Secrets and Credentials
+
+- Are new secrets stored encrypted at rest (e.g. via `encrypt` / `decrypt`
+  from `@langfuse/shared/encryption`) and never returned through `get`/list
+  endpoints?
+- Are secrets omitted from API responses (`select` / `omit` in Prisma) and
+  from logs?
+- Are new env vars added to `.env*.example` files and validated via the
+  package `env.mjs/ts`, not read from `process.env` directly?
+
+## Audit Logging
+
+- Do sensitive mutations (integration config changes, role or scope changes,
+  exports, deletions) emit `auditLog` with the right `action` and
+  `resourceType`? Match existing wiring in
+  `web/src/features/blobstorage-integration/blobstorage-integration-router.ts`
+  and similar routers.
+
+## Negative Tests
+
+- Does the change include tests that prove **blocked** inputs are rejected
+  (private IPs, cross-tenant IDs, missing scope, http: on Cloud)? Missing
+  negative coverage on a security-sensitive surface is a finding.
+
+## Extending
+
+When a new finding class starts to recur in PR reviews or security reports,
+add a one-line bullet here pointing at a new `references/<topic>.md`. Do not
+restate the detail in this checklist; keep it as the trigger surface.
@@ -0,0 +1,165 @@
+# Outbound URL Validation (SSRF)
+
+## Threat
+
+Any Langfuse code path that issues an outbound HTTP request to a URL derived
+from user input (mutation form, public API field, integration config, image
+proxy) can be coerced into Server-Side Request Forgery. The high-value
+internal targets in Langfuse's deployment topology include:
+
+- Cloud instance metadata services (`169.254.169.254`, IMDSv2 endpoints)
+- Internal Postgres, ClickHouse, Redis, S3/MinIO, queue admin UIs
+- Loopback admin interfaces (`127.0.0.1`, `localhost`, Docker API on
+  `2375/2376`)
+- Kubernetes API server and other in-cluster control planes
+- Any RFC1918 / RFC6598 / IPv6 ULA range routable from the pod or container
+
+Even when the surface requires `integrations:CRUD` or admin scope, the
+attacker model assumes the credentialed user is malicious or compromised;
+SSRF lets them pivot from app-level admin to network-level access that the
+deployment topology otherwise denies.
+
+## Canonical Helpers
+
+All under `packages/shared/src/server/outbound-url/`:
+
+- [`parseOutboundUrl(urlString)`](../../../../packages/shared/src/server/outbound-url/validation.ts)
+  — safe parse. Rejects embedded credentials, invalid encoding, and bad URL
+  syntax. **Use this instead of `new URL(...)` for any user-supplied URL.**
+- [`validateOutboundUrlHost({ url, whitelist, logContext, shouldSkipDnsCheckForLiteralIps })`](../../../../packages/shared/src/server/outbound-url/validation.ts)
+  — checks hostname blocklist, IP literal blocklist, and forward DNS
+  resolution against blocked CIDRs (defends DNS rebinding by resolving every
+  A/AAAA plus the local `getaddrinfo` view).
+- [`addSecureOutboundConnectionValidation(options, ...)`](../../../../packages/shared/src/server/outbound-url/connection.ts)
+  — attaches connect-time IP validation to a `fetch` request so the TCP peer
+  is re-validated after DNS resolution, not just at save time.
+- [`fetchWithSecureRedirects(...)`](../../../../packages/shared/src/server/outbound-url/fetch.ts)
+  — manual redirect handling. Validates each `Location` hop with the
+  caller-supplied validator and strips sensitive headers (`Authorization`,
+  `Cookie`, signing headers) on cross-origin redirects.
+
+Surface-specific wrappers (reuse these rather than rolling your own):
+
+- LLM base URL:
+  [`validateLlmConnectionBaseURL`](../../../../packages/shared/src/server/llm/baseUrlValidation.ts)
+- Webhook URL:
+  [`validateWebhookURL`](../../../../packages/shared/src/server/webhooks/validation.ts)
+- Blob storage endpoint:
+  [`validateBlobStorageEndpoint`](../../../../packages/shared/src/server/services/blobStorageEndpointValidation.ts)
+  and the companion
+  [`blobStorageEndpointConnectionValidationOptions`](../../../../packages/shared/src/server/services/blobStorageEndpointValidation.ts)
+  for connect-time enforcement through `StorageServiceFactory`.
+
+## Required Defenses
+
+Every outbound-URL surface MUST apply **all three** of:
+
+1. **Save-time validation** in the mutation, tRPC procedure, or public API
+   route that persists the URL. Reject the write if validation fails.
+2. **Use-time / connection-time validation** when the request is actually
+   issued (worker job, lazy validation endpoint, processor). DNS can change
+   between save and use; the SDK that ultimately makes the call may resolve a
+   different IP than the save-time check did. Plumb
+   `addSecureOutboundConnectionValidation` (or the SDK's equivalent hook)
+   through the request.
+3. **Redirect-time validation** if the request can be redirected. Plain
+   `fetch()` defaults to `redirect: 'follow'` and will silently chase a
+   redirect into the loopback range. Use `fetchWithSecureRedirects` with the
+   matching validator instead.
+
+## Known-Good Call Sites (Copy These)
+
+- LLM base URL save:
+  `web/src/features/llm-api-key/server/router.ts` (`update` mutation calls
+  `validateLlmConnectionBaseURL` before persisting).
+- LLM base URL through public API:
+  `web/src/pages/api/public/llm-connections/index.ts`.
+- Webhook URL save + use:
+  `packages/shared/src/server/webhooks/validation.ts` is wired into both the
+  automation form and the worker-side webhook sender.
+- Blob storage endpoint:
+  `web/src/features/blobstorage-integration/blobstorage-integration-router.ts`
+  (`validate` mutation calls `validateBlobStorageEndpoint`); connection-time
+  enforcement flows through `StorageServiceFactory.getInstance({
+  connectionValidation: blobStorageEndpointConnectionValidationOptions() })`.
+
+## When Adding a New Outbound URL Surface
+
+1. Identify the user-input layers: form mutation, public API route, env import,
+   anywhere the URL can be supplied by a tenant.
+2. Decide whether an existing wrapper fits. If yes, reuse it. If not, add a
+   new wrapper under `packages/shared/src/server/...` that delegates to
+   `validateOutboundUrlHost` so blocklist behavior, DNS rebinding handling,
+   and credential checks stay centralized.
+3. Define the env allowlist trio for self-hosted users who legitimately point
+   at private network targets (mirroring
+   `LANGFUSE_WEBHOOK_WHITELISTED_HOST/IPS/IP_SEGMENTS`,
+   `LANGFUSE_LLM_CONNECTION_WHITELISTED_HOST/IPS/IP_SEGMENTS`, and
+   `LANGFUSE_BLOB_STORAGE_ENDPOINT_WHITELISTED_HOST/IPS/IP_SEGMENTS`). Do
+   not share another surface's allowlist; each surface keeps its own.
+4. Add the env vars to `.env*.example` and the package `env.mjs/ts`.
+5. Call the wrapper from:
+   - every tRPC mutation that writes the URL,
+   - every public API route that accepts the URL,
+   - the worker/processor that issues the request (use connect-time
+     validation if the underlying SDK does not expose a host-validation hook).
+6. If the request can redirect, use `fetchWithSecureRedirects` with the same
+   wrapper as the validator.
+7. Add server-side tests that prove blocked targets fail validation:
+   `127.0.0.1`, `169.254.169.254`, an RFC1918 literal, an RFC1918 hostname
+   (DNS rebinding), `http://` on Cloud, and a URL containing
+   `user:pass@host`.
+
+## Anti-Patterns to Flag in Review
+
+- `fetch(<user-supplied-url>)` (or `axios`, `got`, etc.) without an upstream
+  call to a `validate*URL` helper, or without
+  `addSecureOutboundConnectionValidation` on the request options.
+- A tRPC mutation that persists a `host` / `endpoint` / `baseURL` / `webhookUrl`
+  field without invoking the matching validator before the write.
+- `StorageServiceFactory.getInstance({ endpoint })` (or any SDK client init
+  that takes a user-controlled URL) without `connectionValidation` plumbed
+  through.
+- Custom URL parsing via `new URL(userInput)` instead of
+  `parseOutboundUrl(userInput)`. The latter rejects embedded credentials and
+  bad encoding, both of which are recurring SSRF/credential-leak vectors.
+- Following redirects with `fetch(url)` (default `redirect: 'follow'`) on a
+  user-controlled URL. Switch to `fetchWithSecureRedirects`.
+- A new integration UI that validates only on the client side. Save-time
+  validation must run server-side.
+- Save-time validation present, use-time validation missing (or vice versa).
+  Both layers are required; the worker may issue the request hours after the
+  save and DNS will have moved.
+
+## Env Allowlist Behavior
+
+- Cloud (`NEXT_PUBLIC_LANGFUSE_CLOUD_REGION` set) forces strict mode.
+  Allowlist env vars are ignored on Cloud, and HTTPS is enforced for surfaces
+  that require it (LLM base URL, blob storage endpoint).
+- Self-hosted reads the per-surface env trio:
+  - `LANGFUSE_LLM_CONNECTION_WHITELISTED_HOST/IPS/IP_SEGMENTS`
+  - `LANGFUSE_WEBHOOK_WHITELISTED_HOST/IPS/IP_SEGMENTS`
+  - `LANGFUSE_BLOB_STORAGE_ENDPOINT_WHITELISTED_HOST/IPS/IP_SEGMENTS`
+- Blob storage endpoint validation is **opt-in** today on self-hosted (the
+  helper is a no-op until the operator configures one of the allowlist env
+  vars). There is a `TODO(next major)` in
+  `blobStorageEndpointValidation.ts` to flip the default; until then, do not
+  rely on blob storage validation for new surfaces — wire your own wrapper
+  that defaults to strict.
+
+## Negative Tests (Required)
+
+A change that adds a new outbound URL surface MUST include server-side tests
+that assert each of the following fails validation:
+
+- Loopback literal (`http://127.0.0.1`, `http://[::1]`)
+- Cloud metadata literal (`http://169.254.169.254`)
+- RFC1918 literal (`http://10.0.0.1`)
+- Hostname that resolves to a private IP (DNS rebinding sanity check)
+- URL with embedded credentials (`http://user:pass@host`)
+- `http://` on Cloud (where HTTPS is required)
+- An empty allowlist permits no internal targets on self-hosted
+
+Pattern reference:
+`worker/src/__tests__/llm-base-url-validation.test.ts` and the test files
+adjacent to the wrappers above.
@@ -0,0 +1,67 @@
+---
+name: seed-test-data
+description: |
+  Seed local Langfuse test data with one command: large/branching observation
+  trees (v3 and v4 events), long sessions, bulk traces for list performance.
+  Use whenever a task needs ClickHouse/Postgres test data — e.g. "seed a
+  complex trace", "make a tough session", "fill the trace list", "test v4
+  events UI", or when debugging trace/session/list rendering or performance.
+  Never write ad-hoc seed scripts or raw ClickHouse inserts.
+---
+
+# Seed Test Data
+
+One-shot deterministic test data for local Langfuse. The CLI handles env
+loading, preflight checks, ClickHouse/Postgres writes, readback verification,
+and prints UI deep links plus a machine-readable JSON summary (last stdout
+line).
+
+## If anything fails, run doctor first
+
+```bash
+pnpm run seed -- doctor
+```
+
+Prints PASS/WARN/FAIL per dependency (Postgres, migrations, project,
+ClickHouse, v4 dev tables, Redis, MinIO, web app) with the exact fix command
+for every failure. Do not debug Docker/ClickHouse manually before running
+this.
+
+## Need → command
+
+| I need... | Command |
+| --- | --- |
+| A very complex observation tree (v3) | `pnpm run seed -- trace-tree --observations 5000 --depth 12 --breadth 500` |
+| The same tree readable in the v4 events UI | add `--v4` (writes `events_full`; `events_core` fills via MV) |
+| A super tough session (v3 legacy session view) | `pnpm run seed -- long-session --traces 300 --observations-per-trace 8` |
+| Many traces for list/filter performance | `pnpm run seed -- many-traces --count 100000 --days 14` |
+| Huge/malformed/unicode payloads | `pnpm run seed -- trace-tree --payload-bytes 1000000 --payload-style malformed` (styles: json, text, malformed, unicode) |
+| See all scenarios and flags | `pnpm run seed -- list --json` |
+| Predict without writing | add `--dry-run` |
+
+## Contract
+
+- Last stdout line is a JSON summary: `traceIds`, `sessionIds`, `counts`,
+  `verified` (ClickHouse readback), `links` (UI deep links). Use `--json` to
+  suppress progress logs. Non-zero exit = data did not land; the error
+  includes a `fix:` line.
+- Deterministic: same `--seed` (default 42) and flags → same ids (ids never
+  contain dates), with timestamps anchored to the current UTC day. Re-running
+  within the same day overwrites in place; a later-day re-run updates the
+  same ids with re-anchored timestamps (the previous day's rows persist
+  under their old dates until then). Independent copies come only from
+  `--id-prefix`.
+- Default project is the seeded `7a88fb47-b4e2-43b8-a06c-a5ce950dc53a`
+  (login `demo@langfuse.com` / `password`); override with `--project`.
+- Open the printed `links` in the browser to verify visually. The v4
+  events-backed UI is the per-user "Fast (Preview)" sidebar toggle, or
+  `LANGFUSE_MIGRATION_V4_WRITE_MODE=events_only` server-side.
+
+## Extending
+
+Add a scenario in `packages/shared/scripts/seeder/scenarios/`: a plain
+function using the deterministic `Rng` (never `Math.random`), register it in
+`scenarios/index.ts`, and update the table in
+`packages/shared/scripts/seeder/AGENTS.md` and this skill. Scenario names,
+flags, and JSON keys are additive-only contracts. Design rationale:
+`packages/shared/scripts/seeder/README.md`.
@@ -0,0 +1,4 @@
+interface:
+  display_name: "Seed Test Data"
+  short_description: "One-command Langfuse test data seeding"
+  default_prompt: "Use $seed-test-data to seed the local Langfuse data this task needs (complex traces, long sessions, bulk traces, v4 events)."
@@ -0,0 +1,456 @@
+---
+name: skill-creator
+description: Guide for creating effective skills. This skill should be used when users want to create a new skill (or update an existing skill) that extends Codex's capabilities with specialized knowledge, workflows, or tool integrations.
+metadata:
+  short-description: Create or update Codex skills
+---
+
+# Skill Creator
+
+This skill provides guidance for creating effective skills.
+
+## About Skills
+
+Skills are modular, self-contained folders that extend Codex's capabilities by providing
+specialized knowledge, workflows, and tools. Think of them as "onboarding guides" for specific
+domains or tasks—they transform Codex from a general-purpose agent into a specialized agent
+equipped with procedural knowledge that no model can fully possess.
+
+### What Skills Provide
+
+1. Specialized workflows - Multi-step procedures for specific domains
+2. Tool integrations - Instructions for working with specific file formats or APIs
+3. Domain expertise - Company-specific knowledge, schemas, business logic
+4. Bundled resources - Scripts, references, and assets for complex and repetitive tasks
+
+## Core Principles
+
+### Concise is Key
+
+The context window is a public good. Skills share the context window with everything else Codex needs: system prompt, conversation history, other Skills' metadata, and the actual user request.
+
+**Default assumption: Codex is already very smart.** Only add context Codex doesn't already have. Challenge each piece of information: "Does Codex really need this explanation?" and "Does this paragraph justify its token cost?"
+
+Prefer concise examples over verbose explanations.
+
+### Set Appropriate Degrees of Freedom
+
+Match the level of specificity to the task's fragility and variability:
+
+**High freedom (text-based instructions)**: Use when multiple approaches are valid, decisions depend on context, or heuristics guide the approach.
+
+**Medium freedom (pseudocode or scripts with parameters)**: Use when a preferred pattern exists, some variation is acceptable, or configuration affects behavior.
+
+**Low freedom (specific scripts, few parameters)**: Use when operations are fragile and error-prone, consistency is critical, or a specific sequence must be followed.
+
+Think of Codex as exploring a path: a narrow bridge with cliffs needs specific guardrails (low freedom), while an open field allows many routes (high freedom).
+
+### Require Human Review for Ticket Writes
+
+For skills that create Linear tickets, update Linear tickets, or add evidence to
+existing tickets, require human review before any write. The skill must present
+all findings in a table, ask the human which findings to create or update in
+Linear, and wait for an explicit selection before making changes.
+
+Use this table structure unless the domain needs additional columns:
+
+| ID | Finding | Evidence | Impact / Scope | Existing Ticket Match | Proposed Linear Action | Confidence | Human Decision |
+| --- | --- | --- | --- | --- | --- | --- | --- |
+| F1 | Concise symptom or bug claim | Measured counts, deltas, links, traces, logs, or "No measurements found" | Affected env, service, route, customer segment, or blast radius supported by evidence | Existing issue key/link, duplicate candidate, or "None found" | Create new ticket, add evidence comment, update status/labels, or no action | High/medium/low plus one short reason | Leave blank for the human to choose |
+
+In the skill instructions, state that Codex must not create tickets, comment on
+tickets, edit ticket fields, or add evidence until the human chooses one or more
+row IDs and actions. If the human asks for an automated sweep, still pause at
+this review table before writing to Linear.
+
+### Protect Validation Integrity
+
+You may use subagents during iteration to validate whether a skill works on realistic tasks or whether a suspected problem is real. This is most useful when you want an independent pass on the skill's behavior, outputs, or failure modes after a revision.  Only do this when it is possible to start new subagents.
+
+When using subagents for validation, treat that as an evaluation surface. The goal is to learn whether the skill generalizes, not whether another agent can reconstruct the answer from leaked context.
+
+Prefer raw artifacts such as example prompts, outputs, diffs, logs, or traces. Give the minimum task-local context needed to perform the validation. Avoid passing the intended answer, suspected bug, intended fix, or your prior conclusions unless the validation explicitly requires them.
+
+### Require Valid Markdown Output
+
+When a skill instructs Codex to return Markdown, require the final output to be
+valid Markdown, not just Markdown-like text.
+
+In the skill instructions, explicitly require Codex to:
+
+- use valid Markdown syntax for headings, lists, links, tables, and code fences;
+- include a space after list markers such as `-`, `*`, and `1.`;
+- close every Markdown link and parenthesis correctly;
+- avoid malformed tables, dangling backticks, or partially opened fenced code
+  blocks;
+- prefer plain paragraphs over complex formatting when the structure would be
+  fragile.
+
+If a skill produces structured reports, include a short output-format section
+that says the response must be valid Markdown and should be checked for basic
+syntax mistakes before returning it.
+
+### Anatomy of a Skill
+
+Every skill consists of a required SKILL.md file and optional bundled resources:
+
+```
+skill-name/
+├── SKILL.md (required)
+│   ├── YAML frontmatter metadata (required)
+│   │   ├── name: (required)
+│   │   └── description: (required)
+│   └── Markdown instructions (required)
+├── agents/ (recommended)
+│   └── openai.yaml - UI metadata for skill lists and chips
+└── Bundled Resources (optional)
+    ├── scripts/          - Executable code (Python/Bash/etc.)
+    ├── references/       - Documentation intended to be loaded into context as needed
+    └── assets/           - Files used in output (templates, icons, fonts, etc.)
+```
+
+#### SKILL.md (required)
+
+Every SKILL.md consists of:
+
+- **Frontmatter** (YAML): Contains `name` and `description` fields. These are the only fields that Codex reads to determine when the skill gets used, thus it is very important to be clear and comprehensive in describing what the skill is, and when it should be used.
+- **Body** (Markdown): Instructions and guidance for using the skill. Only loaded AFTER the skill triggers (if at all).
+
+#### Agents metadata (recommended)
+
+- UI-facing metadata for skill lists and chips
+- Read references/openai_yaml.md before generating values and follow its descriptions and constraints
+- Create: human-facing `display_name`, `short_description`, and `default_prompt` by reading the skill
+- Generate deterministically by passing the values as `--interface key=value` to `scripts/generate_openai_yaml.py` or `scripts/init_skill.py`
+- On updates: validate `agents/openai.yaml` still matches SKILL.md; regenerate if stale
+- Only include other optional interface fields (icons, brand color) if explicitly provided
+- See references/openai_yaml.md for field definitions and examples
+
+#### Bundled Resources (optional)
+
+##### Scripts (`scripts/`)
+
+Executable code (Python/Bash/etc.) for tasks that require deterministic reliability or are repeatedly rewritten.
+
+- **When to include**: When the same code is being rewritten repeatedly or deterministic reliability is needed
+- **Example**: `scripts/rotate_pdf.py` for PDF rotation tasks
+- **Benefits**: Token efficient, deterministic, may be executed without loading into context
+- **Note**: Scripts may still need to be read by Codex for patching or environment-specific adjustments
+
+##### References (`references/`)
+
+Documentation and reference material intended to be loaded as needed into context to inform Codex's process and thinking.
+
+- **When to include**: For documentation that Codex should reference while working
+- **Examples**: `references/finance.md` for financial schemas, `references/mnda.md` for company NDA template, `references/policies.md` for company policies, `references/api_docs.md` for API specifications
+- **Use cases**: Database schemas, API documentation, domain knowledge, company policies, detailed workflow guides
+- **Benefits**: Keeps SKILL.md lean, loaded only when Codex determines it's needed
+- **Best practice**: If files are large (>10k words), include grep search patterns in SKILL.md
+- **Avoid duplication**: Information should live in either SKILL.md or references files, not both. Prefer references files for detailed information unless it's truly core to the skill—this keeps SKILL.md lean while making information discoverable without hogging the context window. Keep only essential procedural instructions and workflow guidance in SKILL.md; move detailed reference material, schemas, and examples to references files.
+
+##### Assets (`assets/`)
+
+Files not intended to be loaded into context, but rather used within the output Codex produces.
+
+- **When to include**: When the skill needs files that will be used in the final output
+- **Examples**: `assets/logo.png` for brand assets, `assets/slides.pptx` for PowerPoint templates, `assets/frontend-template/` for HTML/React boilerplate, `assets/font.ttf` for typography
+- **Use cases**: Templates, images, icons, boilerplate code, fonts, sample documents that get copied or modified
+- **Benefits**: Separates output resources from documentation, enables Codex to use files without loading them into context
+
+#### What to Not Include in a Skill
+
+A skill should only contain essential files that directly support its functionality. Do NOT create extraneous documentation or auxiliary files, including:
+
+- README.md
+- INSTALLATION_GUIDE.md
+- QUICK_REFERENCE.md
+- CHANGELOG.md
+- etc.
+
+The skill should only contain the information needed for an AI agent to do the job at hand. It should not contain auxiliary context about the process that went into creating it, setup and testing procedures, user-facing documentation, etc. Creating additional documentation files just adds clutter and confusion.
+
+### Progressive Disclosure Design Principle
+
+Skills use a three-level loading system to manage context efficiently:
+
+1. **Metadata (name + description)** - Always in context (~100 words)
+2. **SKILL.md body** - When skill triggers (<5k words)
+3. **Bundled resources** - As needed by Codex (Unlimited because scripts can be executed without reading into context window)
+
+#### Progressive Disclosure Patterns
+
+Keep SKILL.md body to the essentials and under 500 lines to minimize context bloat. Split content into separate files when approaching this limit. When splitting out content into other files, it is very important to reference them from SKILL.md and describe clearly when to read them, to ensure the reader of the skill knows they exist and when to use them.
+
+**Key principle:** When a skill supports multiple variations, frameworks, or options, keep only the core workflow and selection guidance in SKILL.md. Move variant-specific details (patterns, examples, configuration) into separate reference files.
+
+**Pattern 1: High-level guide with references**
+
+```markdown
+# PDF Processing
+
+## Quick start
+
+Extract text with pdfplumber:
+[code example]
+
+## Advanced features
+
+- **Form filling**: See [FORMS.md](FORMS.md) for complete guide
+- **API reference**: See [REFERENCE.md](REFERENCE.md) for all methods
+- **Examples**: See [EXAMPLES.md](EXAMPLES.md) for common patterns
+```
+
+Codex loads FORMS.md, REFERENCE.md, or EXAMPLES.md only when needed.
+
+**Pattern 2: Domain-specific organization**
+
+For Skills with multiple domains, organize content by domain to avoid loading irrelevant context:
+
+```
+bigquery-skill/
+├── SKILL.md (overview and navigation)
+└── reference/
+    ├── finance.md (revenue, billing metrics)
+    ├── sales.md (opportunities, pipeline)
+    ├── product.md (API usage, features)
+    └── marketing.md (campaigns, attribution)
+```
+
+When a user asks about sales metrics, Codex only reads sales.md.
+
+Similarly, for skills supporting multiple frameworks or variants, organize by variant:
+
+```
+cloud-deploy/
+├── SKILL.md (workflow + provider selection)
+└── references/
+    ├── aws.md (AWS deployment patterns)
+    ├── gcp.md (GCP deployment patterns)
+    └── azure.md (Azure deployment patterns)
+```
+
+When the user chooses AWS, Codex only reads aws.md.
+
+**Pattern 3: Conditional details**
+
+Show basic content, link to advanced content:
+
+```markdown
+# DOCX Processing
+
+## Creating documents
+
+Use docx-js for new documents. See [DOCX-JS.md](DOCX-JS.md).
+
+## Editing documents
+
+For simple edits, modify the XML directly.
+
+**For tracked changes**: See [REDLINING.md](REDLINING.md)
+**For OOXML details**: See [OOXML.md](OOXML.md)
+```
+
+Codex reads REDLINING.md or OOXML.md only when the user needs those features.
+
+**Important guidelines:**
+
+- **Avoid deeply nested references** - Keep references one level deep from SKILL.md. All reference files should link directly from SKILL.md.
+- **Structure longer reference files** - For files longer than 100 lines, include a table of contents at the top so Codex can see the full scope when previewing.
+
+## Skill Creation Process
+
+Skill creation involves these steps:
+
+1. Understand the skill with concrete examples
+2. Plan reusable skill contents (scripts, references, assets)
+3. Initialize the skill (run init_skill.py)
+4. Edit the skill (implement resources and write SKILL.md)
+5. Validate the skill (run quick_validate.py)
+6. Iterate based on real usage and forward-test complex skills.
+
+Follow these steps in order, skipping only if there is a clear reason why they are not applicable.
+
+### Skill Naming
+
+- Use lowercase letters, digits, and hyphens only; normalize user-provided titles to hyphen-case (e.g., "Plan Mode" -> `plan-mode`).
+- When generating names, generate a name under 64 characters (letters, digits, hyphens).
+- Prefer short, verb-led phrases that describe the action.
+- Namespace by tool when it improves clarity or triggering (e.g., `gh-address-comments`, `linear-address-issue`).
+- Name the skill folder exactly after the skill name.
+
+### Step 1: Understanding the Skill with Concrete Examples
+
+Skip this step only when the skill's usage patterns are already clearly understood. It remains valuable even when working with an existing skill.
+
+To create an effective skill, clearly understand concrete examples of how the skill will be used. This understanding can come from either direct user examples or generated examples that are validated with user feedback.
+
+For example, when building an image-editor skill, relevant questions include:
+
+- "What functionality should the image-editor skill support? Editing, rotating, anything else?"
+- "Can you give some examples of how this skill would be used?"
+- "I can imagine users asking for things like 'Remove the red-eye from this image' or 'Rotate this image'. Are there other ways you imagine this skill being used?"
+- "What would a user say that should trigger this skill?"
+- "Where should I create this skill? If you do not have a preference, I will place it in `$CODEX_HOME/skills` (or `~/.codex/skills` when `CODEX_HOME` is unset) so Codex can discover it automatically."
+
+To avoid overwhelming users, avoid asking too many questions in a single message. Start with the most important questions and follow up as needed for better effectiveness.
+
+Conclude this step when there is a clear sense of the functionality the skill should support.
+
+### Step 2: Planning the Reusable Skill Contents
+
+To turn concrete examples into an effective skill, analyze each example by:
+
+1. Considering how to execute on the example from scratch
+2. Identifying what scripts, references, and assets would be helpful when executing these workflows repeatedly
+
+Example: When building a `pdf-editor` skill to handle queries like "Help me rotate this PDF," the analysis shows:
+
+1. Rotating a PDF requires re-writing the same code each time
+2. A `scripts/rotate_pdf.py` script would be helpful to store in the skill
+
+Example: When designing a `frontend-webapp-builder` skill for queries like "Build me a todo app" or "Build me a dashboard to track my steps," the analysis shows:
+
+1. Writing a frontend webapp requires the same boilerplate HTML/React each time
+2. An `assets/hello-world/` template containing the boilerplate HTML/React project files would be helpful to store in the skill
+
+Example: When building a `big-query` skill to handle queries like "How many users have logged in today?" the analysis shows:
+
+1. Querying BigQuery requires re-discovering the table schemas and relationships each time
+2. A `references/schema.md` file documenting the table schemas would be helpful to store in the skill
+
+To establish the skill's contents, analyze each concrete example to create a list of the reusable resources to include: scripts, references, and assets.
+
+### Step 3: Initializing the Skill
+
+At this point, it is time to actually create the skill.
+
+Skip this step only if the skill being developed already exists. In this case, continue to the next step.
+
+Before running `init_skill.py`, ask where the user wants the skill created. If they do not specify a location, default to `$CODEX_HOME/skills`; when `CODEX_HOME` is unset, fall back to `~/.codex/skills` so the skill is auto-discovered.
+
+When creating a new skill from scratch, always run the `init_skill.py` script. The script conveniently generates a new template skill directory that automatically includes everything a skill requires, making the skill creation process much more efficient and reliable.
+
+Usage:
+
+```bash
+scripts/init_skill.py <skill-name> --path <output-directory> [--resources scripts,references,assets] [--examples]
+```
+
+Examples:
+
+```bash
+scripts/init_skill.py my-skill --path "${CODEX_HOME:-$HOME/.codex}/skills"
+scripts/init_skill.py my-skill --path "${CODEX_HOME:-$HOME/.codex}/skills" --resources scripts,references
+scripts/init_skill.py my-skill --path ~/work/skills --resources scripts --examples
+```
+
+The script:
+
+- Creates the skill directory at the specified path
+- Generates a SKILL.md template with proper frontmatter and TODO placeholders
+- Creates `agents/openai.yaml` using agent-generated `display_name`, `short_description`, and `default_prompt` passed via `--interface key=value`
+- Optionally creates resource directories based on `--resources`
+- Optionally adds example files when `--examples` is set
+
+After initialization, customize the SKILL.md and add resources as needed. If you used `--examples`, replace or delete placeholder files.
+
+Generate `display_name`, `short_description`, and `default_prompt` by reading the skill, then pass them as `--interface key=value` to `init_skill.py` or regenerate with:
+
+```bash
+scripts/generate_openai_yaml.py <path/to/skill-folder> --interface key=value
+```
+
+Only include other optional interface fields when the user explicitly provides them. For full field descriptions and examples, see references/openai_yaml.md.
+
+### Step 4: Edit the Skill
+
+When editing the (newly-generated or existing) skill, remember that the skill is being created for another instance of Codex to use. Include information that would be beneficial and non-obvious to Codex. Consider what procedural knowledge, domain-specific details, or reusable assets would help another Codex instance execute these tasks more effectively.
+
+After substantial revisions, or if the skill is particularly tricky, you should use subagents to forward-test the skill on realistic tasks or artifacts. When doing so, pass the artifact under validation rather than your diagnosis of what is wrong, and keep the prompt generic enough that success depends on transferable reasoning rather than hidden ground truth.
+
+#### Start with Reusable Skill Contents
+
+To begin implementation, start with the reusable resources identified above: `scripts/`, `references/`, and `assets/` files. Note that this step may require user input. For example, when implementing a `brand-guidelines` skill, the user may need to provide brand assets or templates to store in `assets/`, or documentation to store in `references/`.
+
+Added scripts must be tested by actually running them to ensure there are no bugs and that the output matches what is expected. If there are many similar scripts, only a representative sample needs to be tested to ensure confidence that they all work while balancing time to completion.
+
+If you used `--examples`, delete any placeholder files that are not needed for the skill. Only create resource directories that are actually required.
+
+#### Update SKILL.md
+
+**Writing Guidelines:** Always use imperative/infinitive form.
+
+##### Frontmatter
+
+Write the YAML frontmatter with `name` and `description`:
+
+- `name`: The skill name
+- `description`: This is the primary triggering mechanism for your skill, and helps Codex understand when to use the skill.
+  - Include both what the Skill does and specific triggers/contexts for when to use it.
+  - Include all "when to use" information here - Not in the body. The body is only loaded after triggering, so "When to Use This Skill" sections in the body are not helpful to Codex.
+  - Example description for a `docx` skill: "Comprehensive document creation, editing, and analysis with support for tracked changes, comments, formatting preservation, and text extraction. Use when Codex needs to work with professional documents (.docx files) for: (1) Creating new documents, (2) Modifying or editing content, (3) Working with tracked changes, (4) Adding comments, or any other document tasks"
+
+Do not include any other fields in YAML frontmatter.
+
+##### Body
+
+Write instructions for using the skill and its bundled resources.
+
+If the skill expects Markdown output, add an explicit output-format section that
+requires valid Markdown syntax in the final response.
+
+### Step 5: Validate the Skill
+
+Once development of the skill is complete, validate the skill folder to catch basic issues early:
+
+```bash
+scripts/quick_validate.py <path/to/skill-folder>
+```
+
+The validation script checks YAML frontmatter format, required fields, and naming rules. If validation fails, fix the reported issues and run the command again.
+
+### Step 6: Iterate
+
+After testing the skill, you may detect the skill is complex enough that it requires forward-testing; or users may request improvements.
+
+User testing often this happens right after using the skill, with fresh context of how the skill performed.
+
+**Forward-testing and iteration workflow:**
+
+1. Use the skill on real tasks
+2. Notice struggles or inefficiencies
+3. Identify how SKILL.md or bundled resources should be updated
+4. Implement changes and test again
+5. Forward-test if it is reasonable and appropriate
+
+## Forward-testing
+
+To forward-test, launch subagents as a way to stress test the skill with minimal context.
+Subagents should *not* know that they are being asked to test the skill.  They should be treated as
+an agent asked to perform a task by the user.  Prompts to subagents should look like:
+  `Use $skill-x at /path/to/skill-x to solve problem y`
+Not:
+  `Review the skill at /path/to/skill-x; pretend a user asks you to...`
+
+Decision rule for forward-testing:
+  - Err on the side of forward-testing
+  - Ask for approval if you think there's a risk that forward-testing would:
+    * take a long time,
+    * require additional approvals from the user, or
+    * modify live production systems
+
+  In these cases, show the user your proposed prompt and request (1) a yes/no decision, and
+  (2) any suggested modifictions.
+
+Considerations when forward-testing:
+   - use fresh threads for independent passes
+   - pass the skill, and a request in a similar way the user would.
+   - pass raw artifacts, not your conclusions
+   - avoid showing expected answers or intended fixes
+   - rebuild context from source artifacts after each iteration
+   - review the subagent's output and reasoning and emitted artifacts
+   - avoid leaving artifacts the agent can find on disk between iterations;
+     clean up subagents' artifacts to avoid additional contamination.
+
+If forward-testing only succeeds when subagents see leaked context, tighten the skill or the
+forward-testing setup before trusting the result.
@@ -0,0 +1,6 @@
+interface:
+  display_name: "Skill Creator"
+  short_description: "Create or update Codex skills"
+  icon_small: "./assets/skill-creator-small.svg"
+  icon_large: "./assets/skill-creator.png"
+  default_prompt: "Use $skill-creator to create or refine a concise Codex skill."
@@ -0,0 +1,3 @@
+<svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" fill="currentColor" viewBox="0 0 20 20">
+  <path fill="#0D0D0D" d="M12.03 4.113a3.612 3.612 0 0 1 5.108 5.108l-6.292 6.29c-.324.324-.56.561-.791.752l-.235.176c-.205.14-.422.261-.65.36l-.229.093a4.136 4.136 0 0 1-.586.16l-.764.134-2.394.4c-.142.024-.294.05-.423.06-.098.007-.232.01-.378-.026l-.149-.05a1.081 1.081 0 0 1-.521-.474l-.046-.093a1.104 1.104 0 0 1-.075-.527c.01-.129.035-.28.06-.422l.398-2.394c.1-.602.162-.987.295-1.35l.093-.23c.1-.228.22-.445.36-.65l.176-.235c.19-.232.428-.467.751-.79l6.292-6.292Zm-5.35 7.232c-.35.35-.534.535-.66.688l-.11.147a2.67 2.67 0 0 0-.24.433l-.062.154c-.08.22-.124.462-.232 1.112l-.398 2.394-.001.001h.003l2.393-.399.717-.126a2.63 2.63 0 0 0 .394-.105l.154-.063a2.65 2.65 0 0 0 .433-.24l.147-.11c.153-.126.339-.31.688-.66l4.988-4.988-3.227-3.226-4.987 4.988Zm9.517-6.291a2.281 2.281 0 0 0-3.225 0l-.364.362 3.226 3.227.363-.364c.89-.89.89-2.334 0-3.225ZM4.583 1.783a.3.3 0 0 1 .294.241c.117.585.347 1.092.707 1.48.357.385.859.668 1.549.783a.3.3 0 0 1 0 .592c-.69.115-1.192.398-1.549.783-.315.34-.53.77-.657 1.265l-.05.215a.3.3 0 0 1-.588 0c-.117-.585-.347-1.092-.707-1.48-.357-.384-.859-.668-1.549-.783a.3.3 0 0 1 0-.592c.69-.115 1.192-.398 1.549-.783.36-.388.59-.895.707-1.48l.015-.05a.3.3 0 0 1 .279-.19Z"/>
+</svg>
@@ -0,0 +1,202 @@
+
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [yyyy] [name of copyright owner]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
@@ -0,0 +1,49 @@
+# openai.yaml fields (full example + descriptions)
+
+`agents/openai.yaml` is an extended, product-specific config intended for the machine/harness to read, not the agent. Other product-specific config can also live in the `agents/` folder.
+
+## Full example
+
+```yaml
+interface:
+  display_name: "Optional user-facing name"
+  short_description: "Optional user-facing description"
+  icon_small: "./assets/small-400px.png"
+  icon_large: "./assets/large-logo.svg"
+  brand_color: "#3B82F6"
+  default_prompt: "Optional surrounding prompt to use the skill with"
+
+dependencies:
+  tools:
+    - type: "mcp"
+      value: "github"
+      description: "GitHub MCP server"
+      transport: "streamable_http"
+      url: "https://api.githubcopilot.com/mcp/"
+
+policy:
+  allow_implicit_invocation: true
+```
+
+## Field descriptions and constraints
+
+Top-level constraints:
+
+- Quote all string values.
+- Keep keys unquoted.
+- For `interface.default_prompt`: generate a helpful, short (typically 1 sentence) example starting prompt based on the skill. It must explicitly mention the skill as `$skill-name` (e.g., "Use $skill-name-here to draft a concise weekly status update.").
+
+- `interface.display_name`: Human-facing title shown in UI skill lists and chips.
+- `interface.short_description`: Human-facing short UI blurb (25–64 chars) for quick scanning.
+- `interface.icon_small`: Path to a small icon asset (relative to skill dir). Default to `./assets/` and place icons in the skill's `assets/` folder.
+- `interface.icon_large`: Path to a larger logo asset (relative to skill dir). Default to `./assets/` and place icons in the skill's `assets/` folder.
+- `interface.brand_color`: Hex color used for UI accents (e.g., badges).
+- `interface.default_prompt`: Default prompt snippet inserted when invoking the skill.
+- `dependencies.tools[].type`: Dependency category. Only `mcp` is supported for now.
+- `dependencies.tools[].value`: Identifier of the tool or dependency.
+- `dependencies.tools[].description`: Human-readable explanation of the dependency.
+- `dependencies.tools[].transport`: Connection type when `type` is `mcp`.
+- `dependencies.tools[].url`: MCP server URL when `type` is `mcp`.
+- `policy.allow_implicit_invocation`: When false, the skill is not injected into
+  the model context by default, but can still be invoked explicitly via `$skill`.
+  Defaults to true.
@@ -0,0 +1,226 @@
+#!/usr/bin/env python3
+"""
+OpenAI YAML Generator - Creates agents/openai.yaml for a skill folder.
+
+Usage:
+    generate_openai_yaml.py <skill_dir> [--name <skill_name>] [--interface key=value]
+"""
+
+import argparse
+import re
+import sys
+from pathlib import Path
+
+ACRONYMS = {
+    "GH",
+    "MCP",
+    "API",
+    "CI",
+    "CLI",
+    "LLM",
+    "PDF",
+    "PR",
+    "UI",
+    "URL",
+    "SQL",
+}
+
+BRANDS = {
+    "openai": "OpenAI",
+    "openapi": "OpenAPI",
+    "github": "GitHub",
+    "pagerduty": "PagerDuty",
+    "datadog": "DataDog",
+    "sqlite": "SQLite",
+    "fastapi": "FastAPI",
+}
+
+SMALL_WORDS = {"and", "or", "to", "up", "with"}
+
+ALLOWED_INTERFACE_KEYS = {
+    "display_name",
+    "short_description",
+    "icon_small",
+    "icon_large",
+    "brand_color",
+    "default_prompt",
+}
+
+
+def yaml_quote(value):
+    escaped = value.replace("\\", "\\\\").replace('"', '\\"').replace("\n", "\\n")
+    return f'"{escaped}"'
+
+
+def format_display_name(skill_name):
+    words = [word for word in skill_name.split("-") if word]
+    formatted = []
+    for index, word in enumerate(words):
+        lower = word.lower()
+        upper = word.upper()
+        if upper in ACRONYMS:
+            formatted.append(upper)
+            continue
+        if lower in BRANDS:
+            formatted.append(BRANDS[lower])
+            continue
+        if index > 0 and lower in SMALL_WORDS:
+            formatted.append(lower)
+            continue
+        formatted.append(word.capitalize())
+    return " ".join(formatted)
+
+
+def generate_short_description(display_name):
+    description = f"Help with {display_name} tasks"
+
+    if len(description) < 25:
+        description = f"Help with {display_name} tasks and workflows"
+    if len(description) < 25:
+        description = f"Help with {display_name} tasks with guidance"
+
+    if len(description) > 64:
+        description = f"Help with {display_name}"
+    if len(description) > 64:
+        description = f"{display_name} helper"
+    if len(description) > 64:
+        description = f"{display_name} tools"
+    if len(description) > 64:
+        suffix = " helper"
+        max_name_length = 64 - len(suffix)
+        trimmed = display_name[:max_name_length].rstrip()
+        description = f"{trimmed}{suffix}"
+    if len(description) > 64:
+        description = description[:64].rstrip()
+
+    if len(description) < 25:
+        description = f"{description} workflows"
+        if len(description) > 64:
+            description = description[:64].rstrip()
+
+    return description
+
+
+def read_frontmatter_name(skill_dir):
+    skill_md = Path(skill_dir) / "SKILL.md"
+    if not skill_md.exists():
+        print(f"[ERROR] SKILL.md not found in {skill_dir}")
+        return None
+    content = skill_md.read_text()
+    match = re.match(r"^---\n(.*?)\n---", content, re.DOTALL)
+    if not match:
+        print("[ERROR] Invalid SKILL.md frontmatter format.")
+        return None
+    frontmatter_text = match.group(1)
+
+    import yaml
+
+    try:
+        frontmatter = yaml.safe_load(frontmatter_text)
+    except yaml.YAMLError as exc:
+        print(f"[ERROR] Invalid YAML frontmatter: {exc}")
+        return None
+    if not isinstance(frontmatter, dict):
+        print("[ERROR] Frontmatter must be a YAML dictionary.")
+        return None
+    name = frontmatter.get("name", "")
+    if not isinstance(name, str) or not name.strip():
+        print("[ERROR] Frontmatter 'name' is missing or invalid.")
+        return None
+    return name.strip()
+
+
+def parse_interface_overrides(raw_overrides):
+    overrides = {}
+    optional_order = []
+    for item in raw_overrides:
+        if "=" not in item:
+            print(f"[ERROR] Invalid interface override '{item}'. Use key=value.")
+            return None, None
+        key, value = item.split("=", 1)
+        key = key.strip()
+        value = value.strip()
+        if not key:
+            print(f"[ERROR] Invalid interface override '{item}'. Key is empty.")
+            return None, None
+        if key not in ALLOWED_INTERFACE_KEYS:
+            allowed = ", ".join(sorted(ALLOWED_INTERFACE_KEYS))
+            print(f"[ERROR] Unknown interface field '{key}'. Allowed: {allowed}")
+            return None, None
+        overrides[key] = value
+        if key not in ("display_name", "short_description") and key not in optional_order:
+            optional_order.append(key)
+    return overrides, optional_order
+
+
+def write_openai_yaml(skill_dir, skill_name, raw_overrides):
+    overrides, optional_order = parse_interface_overrides(raw_overrides)
+    if overrides is None:
+        return None
+
+    display_name = overrides.get("display_name") or format_display_name(skill_name)
+    short_description = overrides.get("short_description") or generate_short_description(display_name)
+
+    if not (25 <= len(short_description) <= 64):
+        print(
+            "[ERROR] short_description must be 25-64 characters "
+            f"(got {len(short_description)})."
+        )
+        return None
+
+    interface_lines = [
+        "interface:",
+        f"  display_name: {yaml_quote(display_name)}",
+        f"  short_description: {yaml_quote(short_description)}",
+    ]
+
+    for key in optional_order:
+        value = overrides.get(key)
+        if value is not None:
+            interface_lines.append(f"  {key}: {yaml_quote(value)}")
+
+    agents_dir = Path(skill_dir) / "agents"
+    agents_dir.mkdir(parents=True, exist_ok=True)
+    output_path = agents_dir / "openai.yaml"
+    output_path.write_text("\n".join(interface_lines) + "\n")
+    print(f"[OK] Created agents/openai.yaml")
+    return output_path
+
+
+def main():
+    parser = argparse.ArgumentParser(
+        description="Create agents/openai.yaml for a skill directory.",
+    )
+    parser.add_argument("skill_dir", help="Path to the skill directory")
+    parser.add_argument(
+        "--name",
+        help="Skill name override (defaults to SKILL.md frontmatter)",
+    )
+    parser.add_argument(
+        "--interface",
+        action="append",
+        default=[],
+        help="Interface override in key=value format (repeatable)",
+    )
+    args = parser.parse_args()
+
+    skill_dir = Path(args.skill_dir).resolve()
+    if not skill_dir.exists():
+        print(f"[ERROR] Skill directory not found: {skill_dir}")
+        sys.exit(1)
+    if not skill_dir.is_dir():
+        print(f"[ERROR] Path is not a directory: {skill_dir}")
+        sys.exit(1)
+
+    skill_name = args.name or read_frontmatter_name(skill_dir)
+    if not skill_name:
+        sys.exit(1)
+
+    result = write_openai_yaml(skill_dir, skill_name, args.interface)
+    if result:
+        sys.exit(0)
+    sys.exit(1)
+
+
+if __name__ == "__main__":
+    main()
@@ -0,0 +1,400 @@
+#!/usr/bin/env python3
+"""
+Skill Initializer - Creates a new skill from template
+
+Usage:
+    init_skill.py <skill-name> --path <path> [--resources scripts,references,assets] [--examples] [--interface key=value]
+
+Examples:
+    init_skill.py my-new-skill --path skills/public
+    init_skill.py my-new-skill --path skills/public --resources scripts,references
+    init_skill.py my-api-helper --path skills/private --resources scripts --examples
+    init_skill.py custom-skill --path /custom/location
+    init_skill.py my-skill --path skills/public --interface short_description="Short UI label"
+"""
+
+import argparse
+import re
+import sys
+from pathlib import Path
+
+from generate_openai_yaml import write_openai_yaml
+
+MAX_SKILL_NAME_LENGTH = 64
+ALLOWED_RESOURCES = {"scripts", "references", "assets"}
+
+SKILL_TEMPLATE = """---
+name: {skill_name}
+description: [TODO: Complete and informative explanation of what the skill does and when to use it. Include WHEN to use this skill - specific scenarios, file types, or tasks that trigger it.]
+---
+
+# {skill_title}
+
+## Overview
+
+[TODO: 1-2 sentences explaining what this skill enables]
+
+## Structuring This Skill
+
+[TODO: Choose the structure that best fits this skill's purpose. Common patterns:
+
+**1. Workflow-Based** (best for sequential processes)
+- Works well when there are clear step-by-step procedures
+- Example: DOCX skill with "Workflow Decision Tree" -> "Reading" -> "Creating" -> "Editing"
+- Structure: ## Overview -> ## Workflow Decision Tree -> ## Step 1 -> ## Step 2...
+
+**2. Task-Based** (best for tool collections)
+- Works well when the skill offers different operations/capabilities
+- Example: PDF skill with "Quick Start" -> "Merge PDFs" -> "Split PDFs" -> "Extract Text"
+- Structure: ## Overview -> ## Quick Start -> ## Task Category 1 -> ## Task Category 2...
+
+**3. Reference/Guidelines** (best for standards or specifications)
+- Works well for brand guidelines, coding standards, or requirements
+- Example: Brand styling with "Brand Guidelines" -> "Colors" -> "Typography" -> "Features"
+- Structure: ## Overview -> ## Guidelines -> ## Specifications -> ## Usage...
+
+**4. Capabilities-Based** (best for integrated systems)
+- Works well when the skill provides multiple interrelated features
+- Example: Product Management with "Core Capabilities" -> numbered capability list
+- Structure: ## Overview -> ## Core Capabilities -> ### 1. Feature -> ### 2. Feature...
+
+Patterns can be mixed and matched as needed. Most skills combine patterns (e.g., start with task-based, add workflow for complex operations).
+
+Delete this entire "Structuring This Skill" section when done - it's just guidance.]
+
+## [TODO: Replace with the first main section based on chosen structure]
+
+[TODO: Add content here. See examples in existing skills:
+- Code samples for technical skills
+- Decision trees for complex workflows
+- Concrete examples with realistic user requests
+- References to scripts/templates/references as needed]
+
+## Resources (optional)
+
+Create only the resource directories this skill actually needs. Delete this section if no resources are required.
+
+### scripts/
+Executable code (Python/Bash/etc.) that can be run directly to perform specific operations.
+
+**Examples from other skills:**
+- PDF skill: `fill_fillable_fields.py`, `extract_form_field_info.py` - utilities for PDF manipulation
+- DOCX skill: `document.py`, `utilities.py` - Python modules for document processing
+
+**Appropriate for:** Python scripts, shell scripts, or any executable code that performs automation, data processing, or specific operations.
+
+**Note:** Scripts may be executed without loading into context, but can still be read by Codex for patching or environment adjustments.
+
+### references/
+Documentation and reference material intended to be loaded into context to inform Codex's process and thinking.
+
+**Examples from other skills:**
+- Product management: `communication.md`, `context_building.md` - detailed workflow guides
+- BigQuery: API reference documentation and query examples
+- Finance: Schema documentation, company policies
+
+**Appropriate for:** In-depth documentation, API references, database schemas, comprehensive guides, or any detailed information that Codex should reference while working.
+
+### assets/
+Files not intended to be loaded into context, but rather used within the output Codex produces.
+
+**Examples from other skills:**
+- Brand styling: PowerPoint template files (.pptx), logo files
+- Frontend builder: HTML/React boilerplate project directories
+- Typography: Font files (.ttf, .woff2)
+
+**Appropriate for:** Templates, boilerplate code, document templates, images, icons, fonts, or any files meant to be copied or used in the final output.
+
+---
+
+**Not every skill requires all three types of resources.**
+"""
+
+EXAMPLE_SCRIPT = '''#!/usr/bin/env python3
+"""
+Example helper script for {skill_name}
+
+This is a placeholder script that can be executed directly.
+Replace with actual implementation or delete if not needed.
+
+Example real scripts from other skills:
+- pdf/scripts/fill_fillable_fields.py - Fills PDF form fields
+- pdf/scripts/convert_pdf_to_images.py - Converts PDF pages to images
+"""
+
+def main():
+    print("This is an example script for {skill_name}")
+    # TODO: Add actual script logic here
+    # This could be data processing, file conversion, API calls, etc.
+
+if __name__ == "__main__":
+    main()
+'''
+
+EXAMPLE_REFERENCE = """# Reference Documentation for {skill_title}
+
+This is a placeholder for detailed reference documentation.
+Replace with actual reference content or delete if not needed.
+
+Example real reference docs from other skills:
+- product-management/references/communication.md - Comprehensive guide for status updates
+- product-management/references/context_building.md - Deep-dive on gathering context
+- bigquery/references/ - API references and query examples
+
+## When Reference Docs Are Useful
+
+Reference docs are ideal for:
+- Comprehensive API documentation
+- Detailed workflow guides
+- Complex multi-step processes
+- Information too lengthy for main SKILL.md
+- Content that's only needed for specific use cases
+
+## Structure Suggestions
+
+### API Reference Example
+- Overview
+- Authentication
+- Endpoints with examples
+- Error codes
+- Rate limits
+
+### Workflow Guide Example
+- Prerequisites
+- Step-by-step instructions
+- Common patterns
+- Troubleshooting
+- Best practices
+"""
+
+EXAMPLE_ASSET = """# Example Asset File
+
+This placeholder represents where asset files would be stored.
+Replace with actual asset files (templates, images, fonts, etc.) or delete if not needed.
+
+Asset files are NOT intended to be loaded into context, but rather used within
+the output Codex produces.
+
+Example asset files from other skills:
+- Brand guidelines: logo.png, slides_template.pptx
+- Frontend builder: hello-world/ directory with HTML/React boilerplate
+- Typography: custom-font.ttf, font-family.woff2
+- Data: sample_data.csv, test_dataset.json
+
+## Common Asset Types
+
+- Templates: .pptx, .docx, boilerplate directories
+- Images: .png, .jpg, .svg, .gif
+- Fonts: .ttf, .otf, .woff, .woff2
+- Boilerplate code: Project directories, starter files
+- Icons: .ico, .svg
+- Data files: .csv, .json, .xml, .yaml
+
+Note: This is a text placeholder. Actual assets can be any file type.
+"""
+
+
+def normalize_skill_name(skill_name):
+    """Normalize a skill name to lowercase hyphen-case."""
+    normalized = skill_name.strip().lower()
+    normalized = re.sub(r"[^a-z0-9]+", "-", normalized)
+    normalized = normalized.strip("-")
+    normalized = re.sub(r"-{2,}", "-", normalized)
+    return normalized
+
+
+def title_case_skill_name(skill_name):
+    """Convert hyphenated skill name to Title Case for display."""
+    return " ".join(word.capitalize() for word in skill_name.split("-"))
+
+
+def parse_resources(raw_resources):
+    if not raw_resources:
+        return []
+    resources = [item.strip() for item in raw_resources.split(",") if item.strip()]
+    invalid = sorted({item for item in resources if item not in ALLOWED_RESOURCES})
+    if invalid:
+        allowed = ", ".join(sorted(ALLOWED_RESOURCES))
+        print(f"[ERROR] Unknown resource type(s): {', '.join(invalid)}")
+        print(f"   Allowed: {allowed}")
+        sys.exit(1)
+    deduped = []
+    seen = set()
+    for resource in resources:
+        if resource not in seen:
+            deduped.append(resource)
+            seen.add(resource)
+    return deduped
+
+
+def create_resource_dirs(skill_dir, skill_name, skill_title, resources, include_examples):
+    for resource in resources:
+        resource_dir = skill_dir / resource
+        resource_dir.mkdir(exist_ok=True)
+        if resource == "scripts":
+            if include_examples:
+                example_script = resource_dir / "example.py"
+                example_script.write_text(EXAMPLE_SCRIPT.format(skill_name=skill_name))
+                example_script.chmod(0o755)
+                print("[OK] Created scripts/example.py")
+            else:
+                print("[OK] Created scripts/")
+        elif resource == "references":
+            if include_examples:
+                example_reference = resource_dir / "api_reference.md"
+                example_reference.write_text(EXAMPLE_REFERENCE.format(skill_title=skill_title))
+                print("[OK] Created references/api_reference.md")
+            else:
+                print("[OK] Created references/")
+        elif resource == "assets":
+            if include_examples:
+                example_asset = resource_dir / "example_asset.txt"
+                example_asset.write_text(EXAMPLE_ASSET)
+                print("[OK] Created assets/example_asset.txt")
+            else:
+                print("[OK] Created assets/")
+
+
+def init_skill(skill_name, path, resources, include_examples, interface_overrides):
+    """
+    Initialize a new skill directory with template SKILL.md.
+
+    Args:
+        skill_name: Name of the skill
+        path: Path where the skill directory should be created
+        resources: Resource directories to create
+        include_examples: Whether to create example files in resource directories
+
+    Returns:
+        Path to created skill directory, or None if error
+    """
+    # Determine skill directory path
+    skill_dir = Path(path).resolve() / skill_name
+
+    # Check if directory already exists
+    if skill_dir.exists():
+        print(f"[ERROR] Skill directory already exists: {skill_dir}")
+        return None
+
+    # Create skill directory
+    try:
+        skill_dir.mkdir(parents=True, exist_ok=False)
+        print(f"[OK] Created skill directory: {skill_dir}")
+    except Exception as e:
+        print(f"[ERROR] Error creating directory: {e}")
+        return None
+
+    # Create SKILL.md from template
+    skill_title = title_case_skill_name(skill_name)
+    skill_content = SKILL_TEMPLATE.format(skill_name=skill_name, skill_title=skill_title)
+
+    skill_md_path = skill_dir / "SKILL.md"
+    try:
+        skill_md_path.write_text(skill_content)
+        print("[OK] Created SKILL.md")
+    except Exception as e:
+        print(f"[ERROR] Error creating SKILL.md: {e}")
+        return None
+
+    # Create agents/openai.yaml
+    try:
+        result = write_openai_yaml(skill_dir, skill_name, interface_overrides)
+        if not result:
+            return None
+    except Exception as e:
+        print(f"[ERROR] Error creating agents/openai.yaml: {e}")
+        return None
+
+    # Create resource directories if requested
+    if resources:
+        try:
+            create_resource_dirs(skill_dir, skill_name, skill_title, resources, include_examples)
+        except Exception as e:
+            print(f"[ERROR] Error creating resource directories: {e}")
+            return None
+
+    # Print next steps
+    print(f"\n[OK] Skill '{skill_name}' initialized successfully at {skill_dir}")
+    print("\nNext steps:")
+    print("1. Edit SKILL.md to complete the TODO items and update the description")
+    if resources:
+        if include_examples:
+            print("2. Customize or delete the example files in scripts/, references/, and assets/")
+        else:
+            print("2. Add resources to scripts/, references/, and assets/ as needed")
+    else:
+        print("2. Create resource directories only if needed (scripts/, references/, assets/)")
+    print("3. Update agents/openai.yaml if the UI metadata should differ")
+    print("4. Run the validator when ready to check the skill structure")
+    print(
+        "5. Forward-test complex skills with realistic user requests to ensure they work as intended"
+    )
+
+    return skill_dir
+
+
+def main():
+    parser = argparse.ArgumentParser(
+        description="Create a new skill directory with a SKILL.md template.",
+    )
+    parser.add_argument("skill_name", help="Skill name (normalized to hyphen-case)")
+    parser.add_argument("--path", required=True, help="Output directory for the skill")
+    parser.add_argument(
+        "--resources",
+        default="",
+        help="Comma-separated list: scripts,references,assets",
+    )
+    parser.add_argument(
+        "--examples",
+        action="store_true",
+        help="Create example files inside the selected resource directories",
+    )
+    parser.add_argument(
+        "--interface",
+        action="append",
+        default=[],
+        help="Interface override in key=value format (repeatable)",
+    )
+    args = parser.parse_args()
+
+    raw_skill_name = args.skill_name
+    skill_name = normalize_skill_name(raw_skill_name)
+    if not skill_name:
+        print("[ERROR] Skill name must include at least one letter or digit.")
+        sys.exit(1)
+    if len(skill_name) > MAX_SKILL_NAME_LENGTH:
+        print(
+            f"[ERROR] Skill name '{skill_name}' is too long ({len(skill_name)} characters). "
+            f"Maximum is {MAX_SKILL_NAME_LENGTH} characters."
+        )
+        sys.exit(1)
+    if skill_name != raw_skill_name:
+        print(f"Note: Normalized skill name from '{raw_skill_name}' to '{skill_name}'.")
+
+    resources = parse_resources(args.resources)
+    if args.examples and not resources:
+        print("[ERROR] --examples requires --resources to be set.")
+        sys.exit(1)
+
+    path = args.path
+
+    print(f"Initializing skill: {skill_name}")
+    print(f"   Location: {path}")
+    if resources:
+        print(f"   Resources: {', '.join(resources)}")
+        if args.examples:
+            print("   Examples: enabled")
+    else:
+        print("   Resources: none (create as needed)")
+    print()
+
+    result = init_skill(skill_name, path, resources, args.examples, args.interface)
+
+    if result:
+        sys.exit(0)
+    else:
+        sys.exit(1)
+
+
+if __name__ == "__main__":
+    main()
@@ -0,0 +1,101 @@
+#!/usr/bin/env python3
+"""
+Quick validation script for skills - minimal version
+"""
+
+import re
+import sys
+from pathlib import Path
+
+import yaml
+
+MAX_SKILL_NAME_LENGTH = 64
+
+
+def validate_skill(skill_path):
+    """Basic validation of a skill"""
+    skill_path = Path(skill_path)
+
+    skill_md = skill_path / "SKILL.md"
+    if not skill_md.exists():
+        return False, "SKILL.md not found"
+
+    content = skill_md.read_text()
+    if not content.startswith("---"):
+        return False, "No YAML frontmatter found"
+
+    match = re.match(r"^---\n(.*?)\n---", content, re.DOTALL)
+    if not match:
+        return False, "Invalid frontmatter format"
+
+    frontmatter_text = match.group(1)
+
+    try:
+        frontmatter = yaml.safe_load(frontmatter_text)
+        if not isinstance(frontmatter, dict):
+            return False, "Frontmatter must be a YAML dictionary"
+    except yaml.YAMLError as e:
+        return False, f"Invalid YAML in frontmatter: {e}"
+
+    allowed_properties = {"name", "description", "license", "allowed-tools", "metadata"}
+
+    unexpected_keys = set(frontmatter.keys()) - allowed_properties
+    if unexpected_keys:
+        allowed = ", ".join(sorted(allowed_properties))
+        unexpected = ", ".join(sorted(unexpected_keys))
+        return (
+            False,
+            f"Unexpected key(s) in SKILL.md frontmatter: {unexpected}. Allowed properties are: {allowed}",
+        )
+
+    if "name" not in frontmatter:
+        return False, "Missing 'name' in frontmatter"
+    if "description" not in frontmatter:
+        return False, "Missing 'description' in frontmatter"
+
+    name = frontmatter.get("name", "")
+    if not isinstance(name, str):
+        return False, f"Name must be a string, got {type(name).__name__}"
+    name = name.strip()
+    if name:
+        if not re.match(r"^[a-z0-9-]+$", name):
+            return (
+                False,
+                f"Name '{name}' should be hyphen-case (lowercase letters, digits, and hyphens only)",
+            )
+        if name.startswith("-") or name.endswith("-") or "--" in name:
+            return (
+                False,
+                f"Name '{name}' cannot start/end with hyphen or contain consecutive hyphens",
+            )
+        if len(name) > MAX_SKILL_NAME_LENGTH:
+            return (
+                False,
+                f"Name is too long ({len(name)} characters). "
+                f"Maximum is {MAX_SKILL_NAME_LENGTH} characters.",
+            )
+
+    description = frontmatter.get("description", "")
+    if not isinstance(description, str):
+        return False, f"Description must be a string, got {type(description).__name__}"
+    description = description.strip()
+    if description:
+        if "<" in description or ">" in description:
+            return False, "Description cannot contain angle brackets (< or >)"
+        if len(description) > 1024:
+            return (
+                False,
+                f"Description is too long ({len(description)} characters). Maximum is 1024 characters.",
+            )
+
+    return True, "Skill is valid!"
+
+
+if __name__ == "__main__":
+    if len(sys.argv) != 2:
+        print("Usage: python quick_validate.py <skill_directory>")
+        sys.exit(1)
+
+    valid, message = validate_skill(sys.argv[1])
+    print(message)
+    sys.exit(0 if valid else 1)
@@ -0,0 +1,61 @@
+---
+name: storybook
+description: Use when writing or reviewing Storybook stories (`.stories.tsx`) for React components.
+---
+
+# Storybook Component Stories
+
+## Which Components Can Have Stories?
+
+Only create stories for components that **do not** do any of the following:
+
+- Depend on context.
+- Fetch data via any private API, including Langfuse’s own tRPC API.
+
+Stories should follow the `ComponentName.stories.tsx` filename pattern. Components covered by stories should have exactly one exported or public component.
+
+If this is not the case, suggest splitting up the file that includes the component to be covered first.
+
+Be mindful of the breadth a story covers. A story should show a component in isolation. Page-level compositions should be rare and intentional.
+
+## What to Do If a Component Violates the Criteria
+
+Suggest abstracting a presentational component that does not violate the criteria and receives relevant data via props.
+
+Make sure the props are well-defined using TypeScript.
+
+Keep the existing component, but update it to use the newly created component for rendering. These presentational components are easier to test and easier to reuse.
+
+## How Stories Should Be Written
+
+- Use "CSF Next" format by default.
+- Cover only the relevant component by default.
+- Avoid custom render functions by default.
+- Use `satisfies` and typed Storybook metadata so invalid args, decorators, and play functions are type-checked.
+- Use play functions to test user-relevant interactions after render, not to compensate for complex setup or hidden dependencies.
+- Name stories after the state they represent, not the implementation. Also do not include the component name in the story name.
+  - Prefer: `Default`, `Empty`, `WithLongName`, `Error`, `Disabled`, `Loading`
+  - Avoid: `Test1`, `CustomRenderExample`, `ButtonWithLongNameAndIcon`
+- Set callbacks up as Storybook Actions by default:
+
+  ```ts
+  import { fn } from "storybook/test";
+  ```
+
+- Avoid large fixtures.
+- Use the smallest meaningful data shape needed to render the state.
+- If fixtures are required and may be shared, check whether a reusable helper function exists. Otherwise, create one for defining the fixture.
+
+## Variant and Design Showcase Stories
+
+If a component has many variants, and the point of the story is to showcase the design of a component rather than its functionality, stories may render the component multiple times.
+
+For example, a `Button` with a `size: "sm" | "md" | "lg"` prop may have a story that shows three buttons side by side.
+
+If the button also has a `variant: "primary" | "secondary"` prop, consider using a matrix-like UI that showcases all possible combinations.
+
+These compositional stories **should not** contain Storybook play functions. They should also not allow the Storybook user to customize the predefined args, such as `size` and `variant`, via Storybook args. Having an arg for non-bound props, such as `text`, may be acceptable.
+
+## Additional Information
+
+- We do not use MSW and are not planning to add it.
@@ -94,13 +94,13 @@ cd packages/utils && bun add lodash

 ```bash
 # pnpm
-pnpm add jest --save-dev --filter=web --filter=@repo/ui
+pnpm add vitest --save-dev --filter=web --filter=@repo/ui

 # npm
-npm install jest --save-dev --workspace=web --workspace=@repo/ui
+npm install vitest --save-dev --workspace=web --workspace=@repo/ui

 # yarn (v2+)
-yarn workspaces foreach -R --from '{web,@repo/ui}' add jest --dev
+yarn workspaces foreach -R --from '{web,@repo/ui}' add vitest --dev
 ```

 ### Internal Packages
@@ -0,0 +1,319 @@
+---
+name: weekly-production-review
+description: |
+  Prepare Langfuse weekly production reviews that audit what broke, what was
+  fixed, what remains open, and where Datadog, incident.io, or Linear tracking
+  needs cleanup. Use when asked for a production review, "what broke last week",
+  fixed/open production bugs, Datadog alerted monitors/pages, Datadog error log
+  patterns, incident.io incidents, incident.io alert load, pager load by
+  engineer or time of day, or a source-table engineering review across
+  incident.io, Linear bugs, Datadog alerts, and Datadog logs.
+---
+
+# Weekly Production Review
+
+Use this skill to produce a source-grounded production review across Datadog,
+incident.io, and Linear. Keep the output factual, table-first, and easy to
+audit from the linked source rows.
+
+## Scope
+
+- Default "last week" to the previous Monday through Sunday in the user's
+  timezone. State both local and UTC query windows in one short scope line.
+- Cover all production environments unless the user narrows scope:
+  `prod-us`, `prod-eu`, `prod-hipaa`, and `prod-jp`.
+- Keep the first pass read-only. Do not create or update Linear issues,
+  comments, incident.io records, follow-ups, alerts, Datadog monitors, files,
+  Slack messages, or production systems unless the user explicitly asks after
+  reviewing the findings.
+- For chat-only reviews, avoid creating report artifacts or local analysis
+  workspaces unless a required tool workflow explicitly does so or the user asks
+  for a file. If incident.io analysis tooling requires a local playbook
+  workspace, mention it only when relevant and keep production systems
+  unchanged.
+- Write `No measurements found` when a requested signal cannot be queried or
+  measured.
+
+## Related Skills
+
+- Use [`datadog-query-recipes`](../datadog-query-recipes/SKILL.md) for
+  production Datadog query shapes and environment/site routing.
+- Use [`linear-bug-triage`](../linear-bug-triage/SKILL.md) only after a human
+  explicitly approves a Linear write-back.
+
+## Workflow
+
+1. Confirm the review window and timezone. If the user says "last week", use the
+   previous calendar week, not a rolling seven-day window.
+2. Gather public/customer-facing incidents from incident.io. Prefer incident.io
+   for accepted incidents, public incident visibility, follow-ups, and incident
+   status. Always produce the incident.io table below, even when no rows are
+   found.
+3. Gather incident.io alert load for the same review window. Prefer
+   incident.io alert or escalation stats for the primary Langfuse escalation
+   path/team. If the user provides an incident.io pager-load dashboard URL, use
+   its `escalation_path` parameter as a scope hint; use the dashboard date range
+   only when the user explicitly scopes the review to that range instead of the
+   default weekly window. Group by paged engineer and incident.io time-of-day
+   bucket so the table shows working-hours, evening, and night load. Always
+   produce the incident.io alert load table below.
+4. Gather Linear bugs from the `bug` label first. Include all `bug`-labeled
+   tickets created, updated, completed, or still open with production evidence
+   during the window. Inspect likely production bugs with issue details and
+   comments when status, owner, or evidence is unclear. Always produce the
+   Linear bug table below.
+5. Gather Datadog alert/page signals for the window. Use incident.io alerts or
+   escalations when they represent pages; use Datadog monitor/event data when
+   available. Build the exhaustive alert universe by paginating until no more
+   results remain for the window. Cover every production environment in scope.
+   Group repeated firings by monitor/page title or ID, environment, service/team,
+   and trigger reason.
+6. For every Datadog alert/page cluster, perform the deep dive before writing the
+   final row. Do not stop at the monitor title or count. Inspect matching APM
+   spans, representative traces, related logs, error records, exception details,
+   failed job logs, dependency spans, queue backlog/delay context, and monitor
+   time windows. Put the relevant trace/span evidence and relevant logs/errors
+   directly in the Datadog alerts table. Do not create a separate Datadog Issue
+   Deep Dives table.
+7. Gather Datadog error log patterns for the window. Use logs with
+   `status:error`, scope to production environments, and group by the clustered
+   `message` pattern plus service/env where available. Always produce the
+   Datadog logs table below. Preserve exact Datadog patterns, including wildcard
+   tokens, instead of paraphrasing them.
+8. Classify each incident, bug, alert, alert-load row, and log pattern. Separate
+   production breakage from self-hosted, internal-only, duplicate, canceled,
+   expected/test, staging/dev, monitor-noise, or unknown signals.
+9. Cross-reference source rows on a best-effort basis. Link Datadog rows to
+   matching incident.io incidents or Linear bugs when evidence supports the
+   relationship. Link incident.io and Linear rows back to Datadog evidence when
+   available. If a relationship is inferential, say so in the row.
+
+## Output Contract
+
+Return one short scope line followed by exactly these five source tables, in
+this order:
+
+1. incident.io
+2. incident.io Alert Load
+3. Linear Bugs
+4. Datadog Alerts
+5. Datadog Logs
+
+Do not add an executive summary, narrative summary, event-centric view, summary
+table, source synthesis table, or separate Datadog Issue Deep Dives table. Put
+counts and classifications inside the source tables.
+
+If a table has no rows, keep the table heading and write one row or sentence
+with `No rows found` or `No measurements found` plus the scoped source/query.
+If a row is unclear, classify it as `unclear` or `unknown/no measurements`
+instead of dropping it.
+
+## Cross-Source Linking
+
+Keep incident.io incidents, incident.io alert load, Linear, Datadog alerts, and
+Datadog logs as separate output tables. Use links inside each table to show
+relationships instead of synthesizing a separate cross-source table.
+
+Use Linear as the source of truth for deduplication across weeks and workflows.
+Before reporting a bug, security finding, cost concern, or alert as new, search
+Linear for matching issue keys, titles, source URLs, and comments. If an
+existing issue covers it, link to that issue and mark the row as already
+tracked instead of reporting it again as fresh work.
+
+Use short stable link labels:
+
+- `Datadog monitor: <monitor name>`
+- `Datadog logs: <env/service/symptom>`
+- `Datadog spans: <env/route/symptom>`
+- `Datadog trace: <trace id or route>`
+- `incident.io: <INC reference>`
+- `incident.io alert load: <escalation path or team>`
+- `Linear: <issue key>`
+
+Do not write links, create follow-ups, or update external systems unless the
+user explicitly asks for changes after reviewing the report.
+
+## incident.io Table
+
+Use this table for incident.io incidents with public or customer-facing impact
+in the review window. Query incident.io with `incident_list` scoped to the
+review window and relevant team when known. Include `summary`, `roles`,
+`custom_fields`, `timestamps`, `durations`, and `escalation_urgency` when the
+tool supports them. If the tool cannot filter by visibility server-side, list
+the scoped incidents and include only rows where `visibility` is `public` or
+the evidence supports customer-facing impact.
+
+| Incident | Severity / Status | Start / End / Duration | Impact | Linked Sources | Follow-ups / Notes |
+| --- | --- | --- | --- | --- | --- |
+
+Column rules:
+
+- `Incident`: incident.io reference linked to the incident.
+- `Severity / Status`: severity and current lifecycle status.
+- `Start / End / Duration`: reported, identified, resolved, and duration when
+  available.
+- `Impact`: short impact statement grounded in the incident summary.
+- `Linked Sources`: Datadog alerts/pages, Datadog logs/spans, Linear issues, or
+  `none found`.
+- `Follow-ups / Notes`: follow-up count/status or `none found`.
+
+## incident.io Alert Load Table
+
+Use this table for incident.io alert and pager load in the review window.
+Prefer incident.io alert or escalation stats filtered to the Langfuse escalation
+path or team. If the user provides a pager-load dashboard URL, parse and apply
+the `escalation_path[one_of]` filter when available. Count alerts when the
+source returns alert counts; otherwise count escalations/pages and label the
+count source in `Source / Notes`.
+
+incident.io time-of-day buckets are UTC:
+
+- `working_hours`: 09:00-18:00 Monday-Friday.
+- `late_evening`: 18:00-23:00 any day plus weekend daytime.
+- `overnight`: 23:00-09:00.
+
+Render one row per paged engineer, sorted by total descending, and include a
+final `All engineers` row when measurements exist. If user identity is missing,
+use `Unassigned / no responder`. Do not collapse this table into a narrative
+summary.
+
+| Engineer | Working Hours | Late Evening | Overnight | Total Alerts / Pages | Share | Source / Notes |
+| --- | ---: | ---: | ---: | ---: | ---: | --- |
+
+Column rules:
+
+- `Engineer`: paged user or escalation target. Use the engineer's display name
+  when available.
+- `Working Hours`: count in the `working_hours` bucket.
+- `Late Evening`: count in the `late_evening` bucket.
+- `Overnight`: count in the `overnight` bucket.
+- `Total Alerts / Pages`: row total. Label pages versus alerts in
+  `Source / Notes` when the source does not expose alert counts directly.
+- `Share`: row total divided by the measured alert/page total.
+- `Source / Notes`: source query, escalation path/team filter, dashboard link,
+  or `No measurements found`.
+
+## Linear Bugs Table
+
+Start from all Linear tickets with the `bug` label that were touched by the
+window. Do not rely only on text searches for `prod`, `incident`, or `Datadog`;
+those searches are useful for enrichment but are not the source universe. This
+table is the Linear source inventory. A Linear bug can be classified as
+non-production, duplicate, canceled, or no-action.
+
+| Linear | Title | Summary | Owner | Status | Touched Last Week Because | Production Evidence | Classification | Counted? |
+| --- | --- | --- | --- | --- | --- | --- | --- | --- |
+
+Column rules:
+
+- `Linear`: issue key linked to Linear, such as `LFE-123`.
+- `Title`: Linear issue title in a separate column.
+- `Summary`: one operational sentence based on the issue body, comments, and
+  evidence. Avoid fix guesses.
+- `Owner`: assignee if present; otherwise owning team if clear; otherwise
+  `Unassigned`.
+- `Status`: Linear status or state, plus completion timing when useful.
+- `Touched Last Week Because`: `created`, `updated`, `completed`, or
+  `open production bug`.
+- `Production Evidence`: prod env, customer impact, incident.io incident,
+  Datadog link, measured logs/spans/errors, or `No measurements found`.
+- `Classification`: use `production/customer-impacting`, `internal-only`,
+  `self-hosted`, `staging/dev`, `duplicate/canceled/no-action`, or `unclear`.
+- `Counted?`: `yes` only when the bug label and production/customer-impacting
+  evidence support including it in fixed/open production bug counts.
+
+## Datadog Alerts Table
+
+Use this table for every production Datadog alert/page cluster found in the full
+paginated alert/event pass, including clusters later classified as
+`expected/test`, `monitor noise`, or `unknown/no measurements`.
+
+| Monitor / Page Signal | Env / Service | Count / Window | Why It Alerted | Trace / Span Evidence | Relevant Logs / Errors | Verdict | incident.io / Linear Link |
+| --- | --- | ---: | --- | --- | --- | --- | --- |
+
+Column rules:
+
+- `Monitor / Page Signal`: monitor/page title or stable ID.
+- `Env / Service`: production env and service/team labels.
+- `Count / Window`: grouped firing/page count and relevant time window.
+- `Why It Alerted`: monitor threshold, trigger condition, route, queue, status
+  code, latency, or backlog signal.
+- `Trace / Span Evidence`: representative trace/span links, error counts,
+  latency, status codes, dependency spans, or `No measurements found`.
+- `Relevant Logs / Errors`: explicit exception class/message, exact or
+  normalized log message, failed job IDs when visible, DB/downstream errors,
+  retry exhaustion, validation failures, or `No measurements found`.
+- `Verdict`: use `customer incident`, `confirmed bug`, `infra/dependency`,
+  `expected/test`, `monitor noise`, or `unknown/no measurements`.
+- `incident.io / Linear Link`: matching incident.io reference, Linear issue key,
+  explicit disposition, or `none found`.
+
+For API route and queue consumer errors, start from APM spans matching:
+
+```text
+operation_name:(http.server OR bullmq.consumer) status:error env:<prod-env>
+```
+
+Then narrow by `service`, `resource_name`, route, queue, consumer, status code,
+error type, or monitor time window. For failed trace samples, explicitly query
+related Datadog logs and error records using the trace ID, span ID, service,
+resource name, environment, and same time window. If no related logs or error
+records are found, write `No measurements found`.
+
+Before finalizing, compare the final Datadog alerts table against the full
+paginated alert/event sweep. Confirm every production monitor title seen during
+the window appears in the table or is explicitly excluded as non-prod.
+
+## Datadog Logs Table
+
+Use this table for the most frequent production error-log patterns from the
+review window. This is a broad log-health pass and is separate from the alert
+cluster rows above, though rows should cross-link when possible.
+
+Start from a Datadog Logs query shaped like:
+
+```text
+status:error
+```
+
+Scope it to the review window and production environments in scope. Prefer the
+Datadog pattern/clustering view using the log `message` field as the clustering
+pattern field. Group or facet by `service`, `env`, and `status` when available,
+and sort by count descending. Review at least the top 10 patterns overall, plus
+any additional top pattern per production environment when the global top 10 is
+dominated by one env or service.
+
+| Exact Log Pattern | Env / Service | Count / Share | Representative Error | Related Signal / Link | Disposition |
+| --- | --- | ---: | --- | --- | --- |
+
+Column rules:
+
+- `Exact Log Pattern`: exact Datadog clustered pattern or exact raw log message.
+  Preserve Datadog wildcard syntax such as `[wildcard]...[/wildcard]`. Do not
+  paraphrase this column.
+- `Env / Service`: affected production envs and services.
+- `Count / Share`: count in the review window and share if available.
+- `Representative Error`: one exact short representative message, exception
+  class, or stack/log summary. Avoid long stack traces.
+- `Related Signal / Link`: related Datadog alert row, trace, log query,
+  incident.io incident, Linear issue, or `none found`.
+- `Disposition`: use `known incident`, `tracked bug`, `needs investigation`,
+  `expected/test`, `monitor noise`, or `unknown`.
+
+If a high-volume pattern maps to a failed API route or queue consumer, ensure
+the matching Datadog alert row includes the trace/log/error investigation. If a
+high-volume pattern has no alert/page row, keep it in this table anyway and
+mark it `needs investigation` or `unknown` based on evidence.
+
+## Output Format
+
+Return valid Markdown only.
+
+- Use valid Markdown syntax for headings, links, and tables.
+- Include a space after list markers such as `-`, `*`, and `1.`.
+- Close links and parentheses correctly.
+- Do not emit malformed tables, dangling backticks, or partially opened code
+  fences.
+- Escape table pipes inside log patterns or error messages when needed.
+- If a section would be fragile to format, prefer a plain paragraph over broken
+  Markdown.
@@ -0,0 +1,4 @@
+interface:
+  display_name: "Weekly Production Review"
+  short_description: "Production review plus alert load"
+  default_prompt: "Use $weekly-production-review to prepare a weekly production review with incident.io incidents and alert load by engineer/time bucket, Linear bugs, Datadog alerts, and exact Datadog log patterns."
@@ -1,13 +1,22 @@
 # Dev container Dockerfile
+FROM --platform=${BUILDPLATFORM} golang:1.24 AS migrate-builder
+ARG TARGETOS
+ARG TARGETARCH
+ENV CGO_ENABLED=0 \
+    GOBIN=/out \
+    GOOS=${TARGETOS} \
+    GOARCH=${TARGETARCH}
+# Build only the ClickHouse migrate CLI used in this repo.
+RUN /usr/local/go/bin/go install -trimpath -tags 'clickhouse' -ldflags='-s -w' \
+    github.com/golang-migrate/migrate/v4/cmd/migrate@v4.19.1
+
 FROM mcr.microsoft.com/devcontainers/universal:2

 # Install golang-migrate for database migrations
-RUN curl -L https://github.com/golang-migrate/migrate/releases/download/v4.19.1/migrate.linux-amd64.tar.gz | tar xvz && \
-    chmod +x migrate && \
-    mv migrate /usr/local/bin/migrate
+COPY --from=migrate-builder /out/migrate /usr/local/bin/migrate

 # Activate the repo's pinned pnpm via Corepack
-RUN corepack enable && corepack prepare pnpm@10.33.0 --activate
+RUN corepack enable && corepack prepare pnpm@11.4.0 --activate

 # Install Clickhouse
 RUN curl https://clickhouse.com/ | sh && \
@@ -77,6 +77,7 @@ LANGFUSE_AZURE_SKIP_CONTAINER_CHECK=false
 REDIS_HOST="127.0.0.1"
 REDIS_PORT=6379
 REDIS_AUTH="myredissecret"
+# LANGFUSE_BULLMQ_SKIP_REDIS_VERSION_CHECK="false"  # Set to true for Redis-compatible services that report a non-Redis version

 # openssl rand -hex 32 used only here
 ENCRYPTION_KEY=0000000000000000000000000000000000000000000000000000000000000000 
@@ -0,0 +1,196 @@
+#####################################################################
+# .env (template) — OCI Object Storage / S3-compatible configuration
+#
+# IMPORTANT SECURITY NOTES (Oracle best practice)
+# - Prefer OCI-native auth (Instance Principal / Workload Identity / Resource Principal)
+#   over static keys.
+# - If you must use static keys, store them in a secure secret manager
+#   (e.g., Kubernetes Secret / OCI Vault) and inject at runtime.
+# - Rotate/revoke any credentials that were previously shared or committed.
+#####################################################################
+
+
+#####################################################################
+# 1) Storage/Auth category (CHOOSE ONE)
+#
+# The app can read/write/download to/from an OCI object store for:
+# - Batch exports (exports/)
+# - Media uploads (media/)
+# - Event uploads (events/)
+#
+# Pick exactly ONE auth mechanism for OCI-native object storage by setting:
+#   LANGFUSE_USE_OCI_NATIVE_OBJECT_STORAGE=true
+#   LANGFUSE_OCI_AUTH_TYPE=<one of the values below>
+#
+# Supported values:
+#   workload_identity | instance_principal | resource_principal | oci_profile | session_token
+#####################################################################
+
+
+#####################################################################
+# Category A — OCI Object Storage with INSTANCE PRINCIPAL (recommended on OCI Compute)
+# Use when:
+# - Running on OCI Compute with IAM set up (dynamic group + policies)
+#
+# Set:
+# LANGFUSE_USE_OCI_NATIVE_OBJECT_STORAGE=true
+# LANGFUSE_OCI_AUTH_TYPE=instance_principal
+#
+# NOTE: Do NOT set *_ACCESS_KEY_ID / *_SECRET_ACCESS_KEY in this category.
+#####################################################################
+
+
+#####################################################################
+# Category B — OCI Object Storage with WORKLOAD IDENTITY (common on OKE)
+# Use when:
+# - Running on OKE with OCI Workload Identity configured
+#
+# Set:
+# LANGFUSE_USE_OCI_NATIVE_OBJECT_STORAGE=true
+# LANGFUSE_OCI_AUTH_TYPE=workload_identity
+#
+# Optional (only if your environment requires additional CA trust):
+# NODE_EXTRA_CA_CERTS=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
+#
+# NOTE: Do NOT set *_ACCESS_KEY_ID / *_SECRET_ACCESS_KEY in this category.
+#####################################################################
+
+
+#####################################################################
+# Category C — OCI Object Storage with RESOURCE PRINCIPAL (common for OCI services)
+# Use when:
+# - Running inside an OCI service/runtime that injects Resource Principal env vars
+#   (e.g., certain managed services / automation contexts)
+#
+# Set:
+# LANGFUSE_USE_OCI_NATIVE_OBJECT_STORAGE=true
+# LANGFUSE_OCI_AUTH_TYPE=resource_principal
+#
+# NOTE: Do NOT set *_ACCESS_KEY_ID / *_SECRET_ACCESS_KEY in this category.
+#####################################################################
+
+
+#####################################################################
+# Category D — OCI Object Storage with OCI CONFIG PROFILE (developer local)
+# Use when:
+# - You have an OCI config file locally or mounted in the runtime
+# - You want to use a named profile
+#
+# Set:
+# LANGFUSE_USE_OCI_NATIVE_OBJECT_STORAGE=true
+# LANGFUSE_OCI_AUTH_TYPE=oci_profile
+# OCI_CONFIG_FILE=/path/to/oci/config
+# OCI_CONFIG_PROFILE=DEFAULT
+#
+# NOTE: Avoid adding config files into images; mount/inject securely.
+#####################################################################
+
+
+#####################################################################
+# Category E — OCI Object Storage with SESSION TOKEN (short-lived user auth)
+# Use when:
+# - You use OCI CLI session authentication (short-lived token flow)
+# - USE oci session authenticate
+# - Appropriate for interactive/dev use; less common for long-running services
+#
+# Set:
+# LANGFUSE_USE_OCI_NATIVE_OBJECT_STORAGE=true
+# LANGFUSE_OCI_AUTH_TYPE=session_token
+# OCI_CONFIG_FILE=/path/to/oci/config
+# OCI_CONFIG_PROFILE=DEFAULT
+#####################################################################
+
+
+#####################################################################
+# Other possible setup — Non-OCI provider (AWS S3 / GCP / Azure / MinIO / etc.)
+# Use when:
+# - Your object storage is NOT OCI Object Storage
+#
+# Set:
+# LANGFUSE_USE_OCI_NATIVE_OBJECT_STORAGE=false
+#
+# Then configure endpoints/regions/credentials for your provider.
+#####################################################################
+
+
+
+#####################################################################
+# 2) Feature: S3 Batch Export
+#
+# Required (when enabled):
+# - *_BUCKET, *_REGION, *_ENDPOINT, *_PREFIX
+# Optional:
+# - *_EXTERNAL_ENDPOINT
+# - *_FORCE_PATH_STYLE=true (needed for many S3-compatible providers like MinIO)
+#
+# Credentials:
+# - Set *_ACCESS_KEY_ID/_SECRET_ACCESS_KEY ONLY for static-key auth
+#   (non-OCI S3-compatible providers)
+#####################################################################
+LANGFUSE_S3_BATCH_EXPORT_ENABLED=true
+LANGFUSE_S3_BATCH_EXPORT_BUCKET=langfuse-bucket
+LANGFUSE_S3_BATCH_EXPORT_PREFIX=exports/
+
+# OCI example region/endpoint:
+LANGFUSE_S3_BATCH_EXPORT_REGION=us-chicago-1
+LANGFUSE_S3_BATCH_EXPORT_ENDPOINT=https://objectstorage.us-chicago-1.oraclecloud.com
+LANGFUSE_S3_BATCH_EXPORT_EXTERNAL_ENDPOINT=https://objectstorage.us-chicago-1.oraclecloud.com
+
+# MinIO / S3-compat setting (safe to keep true for many S3-compatible endpoints)
+LANGFUSE_S3_BATCH_EXPORT_FORCE_PATH_STYLE=true
+
+# Static-key auth (non-OCI). Leave blank/commented for OCI-native auth types above.
+LANGFUSE_S3_BATCH_EXPORT_ACCESS_KEY_ID=__REPLACE_ME__
+LANGFUSE_S3_BATCH_EXPORT_SECRET_ACCESS_KEY=__REPLACE_ME__
+
+
+
+#####################################################################
+# 3) Feature: S3 Media Upload
+#####################################################################
+LANGFUSE_S3_MEDIA_UPLOAD_BUCKET=langfuse-bucket
+LANGFUSE_S3_MEDIA_UPLOAD_PREFIX=media/
+LANGFUSE_S3_MEDIA_UPLOAD_REGION=us-chicago-1
+LANGFUSE_S3_MEDIA_UPLOAD_ENDPOINT=https://objectstorage.us-chicago-1.oraclecloud.com
+LANGFUSE_S3_MEDIA_UPLOAD_FORCE_PATH_STYLE=true
+
+# Static-key auth (non-OCI). Leave blank/commented for OCI-native auth types above.
+LANGFUSE_S3_MEDIA_UPLOAD_ACCESS_KEY_ID=__REPLACE_ME__
+LANGFUSE_S3_MEDIA_UPLOAD_SECRET_ACCESS_KEY=__REPLACE_ME__
+
+
+
+#####################################################################
+# 4) Feature: S3 Event Upload (optional)
+#####################################################################
+LANGFUSE_S3_EVENT_UPLOAD_BUCKET=langfuse-bucket
+LANGFUSE_S3_EVENT_UPLOAD_PREFIX=events/
+LANGFUSE_S3_EVENT_UPLOAD_REGION=us-chicago-1
+LANGFUSE_S3_EVENT_UPLOAD_ENDPOINT=https://objectstorage.us-chicago-1.oraclecloud.com
+LANGFUSE_S3_EVENT_UPLOAD_FORCE_PATH_STYLE=true
+
+# Static-key auth (non-OCI). Leave blank/commented for OCI-native auth types above.
+LANGFUSE_S3_EVENT_UPLOAD_ACCESS_KEY_ID=__REPLACE_ME__
+LANGFUSE_S3_EVENT_UPLOAD_SECRET_ACCESS_KEY=__REPLACE_ME__
+
+
+
+#####################################################################
+# 5) OCI native auth configuration (used by oci_profile / session_token)
+#####################################################################
+# Only required when LANGFUSE_OCI_AUTH_TYPE is: oci_profile OR session_token
+OCI_CONFIG_FILE=__REPLACE_ME__/config
+OCI_CONFIG_PROFILE=DEFAULT
+
+
+
+#####################################################################
+# 6) Troubleshooting notes (comments only)
+#
+# - If you see TLS errors to the endpoint in Kubernetes/OKE, set NODE_EXTRA_CA_CERTS to the
+#   correct CA bundle path for your environment.
+# - If using MinIO or certain S3-compatible providers and you get bucket addressing errors,
+#   set *_FORCE_PATH_STYLE=true.
+# - If downloads work inside the cluster but not externally, configure
+#   LANGFUSE_S3_BATCH_EXPORT_EXTERNAL_ENDPOINT to a publicly reachable endpoint/DNS.
+#####################################################################
@@ -76,11 +76,13 @@ REDIS_AUTH="bitnami"
 # REDIS_KEY_PREFIX=""  # Optional: Prefix for Redis keys (useful for multi-tenant Redis instances)
                       # BullMQ queues will use this via BullMQ's native prefix option
                       # Cache operations will use this via ioredis keyPrefix
+# LANGFUSE_BULLMQ_SKIP_REDIS_VERSION_CHECK="false"  # Set to true for Redis-compatible services that report a non-Redis version
 REDIS_CLUSTER_ENABLED="true"
 REDIS_CLUSTER_NODES="127.0.0.1:6370,127.0.0.1:6371,127.0.0.1:6372,127.0.0.1:6373,127.0.0.1:6374,127.0.0.1:6375"
 LANGFUSE_INGESTION_QUEUE_SHARD_COUNT=8
 LANGFUSE_INGESTION_SECONDARY_QUEUE_SHARD_COUNT=8
 LANGFUSE_OTEL_INGESTION_QUEUE_SHARD_COUNT=4
+LANGFUSE_OTEL_INGESTION_SECONDARY_QUEUE_SHARD_COUNT=4
 LANGFUSE_EVAL_EXECUTION_QUEUE_SHARD_COUNT=4
 LANGFUSE_EVAL_EXECUTION_SECONDARY_QUEUE_SHARD_COUNT=4
 LANGFUSE_LLM_AS_JUDGE_EXECUTION_QUEUE_SHARD_COUNT=4
@@ -62,9 +62,18 @@ NEXTAUTH_SECRET="secret"

 # Langfuse Cloud Environment
 NEXT_PUBLIC_LANGFUSE_CLOUD_REGION="DEV"
+# Dev-only: override the legacy blob-export cutoff date for local testing.
+# Set to a past date (e.g. 2020-01-01T00:00:00.000Z) to make every project
+# post-cutoff, or a future date (e.g. 2099-01-01T00:00:00.000Z) to grandfather
+# all projects. Must be a valid ISO 8601 datetime string. Leave unset in prod.
+# NEXT_PUBLIC_LANGFUSE_BLOB_EXPORT_CUTOFF=
+# Same, for the integration-level cutoff applied to the blob storage
+# integration row's creation date instead of the project's.
+# NEXT_PUBLIC_LANGFUSE_BLOB_EXPORTER_CUTOFF=

 # Langfuse experimental features
 LANGFUSE_ENABLE_EXPERIMENTAL_FEATURES="false"
+LANGFUSE_ENABLE_WEB_CALLOUTS="true"

 # Salt for API key hashing
 SALT="salt"
@@ -116,6 +125,7 @@ REDIS_AUTH="myredissecret"
 # REDIS_KEY_PREFIX=""  # Optional: Prefix for Redis keys (useful for multi-tenant Redis instances)
                       # BullMQ queues will use this via BullMQ's native prefix option
                       # Cache operations will use this via ioredis keyPrefix
+# LANGFUSE_BULLMQ_SKIP_REDIS_VERSION_CHECK="false"  # Set to true for Redis-compatible services that report a non-Redis version
 # REDIS_SENTINEL_ENABLED="false"
 # REDIS_SENTINEL_NODES="sentinel1:26379,sentinel2:26379"
 # REDIS_SENTINEL_MASTER_NAME="mymaster"
@@ -134,6 +144,18 @@ NEXT_PUBLIC_LANGFUSE_RUN_NEXT_INIT="false"
 # LANGFUSE_EVAL_EXECUTION_WORKER_CONCURRENCY=5
 # LANGFUSE_LLM_AS_JUDGE_EXECUTION_WORKER_CONCURRENCY=5

+# Code-based eval dispatchers. Local dev defaults to the insecure in-process dispatcher.
+# To explicitly use the local dispatcher, set:
+# LANGFUSE_CODE_EVAL_DISPATCHER=insecure-local
+# To test the AWS Lambda dispatcher against Floci, set:
+# LANGFUSE_CODE_EVAL_DISPATCHER=aws-lambda
+# LANGFUSE_CODE_EVAL_AWS_LAMBDA_ENDPOINT=http://localhost:4566
+# Override Lambda function names if your deployment uses non-default names:
+# LANGFUSE_CODE_EVAL_AWS_LAMBDA_NODE_FUNCTION_NAME=code-based-eval-executor-node
+# LANGFUSE_CODE_EVAL_AWS_LAMBDA_PYTHON_FUNCTION_NAME=code-based-eval-executor-python
+# Override the local dispatcher execution timeout (default: 2000ms):
+# LANGFUSE_CODE_EVAL_LOCAL_TIMEOUT_MS=2000
+
 # Slack credentials for development
 SLACK_CLIENT_ID=your_slack_client_id
 SLACK_CLIENT_SECRET=your_slack_client_secret
@@ -149,17 +171,33 @@ LANGFUSE_AI_FEATURES_PROJECT_ID=7a88fb47-b4e2-43b8-a06c-a5ce950dc53a
 # LANGFUSE_LLM_CONNECTION_WHITELISTED_HOST=localhost
 # LANGFUSE_LLM_CONNECTION_WHITELISTED_IPS=127.0.0.1,::1
 # LANGFUSE_LLM_CONNECTION_WHITELISTED_IP_SEGMENTS=127.0.0.0/8
+# Self-hosted only: allow internal hosts/IPs for user-configured blob storage endpoints.
+# LANGFUSE_BLOB_STORAGE_ENDPOINT_WHITELISTED_HOST=localhost
+# LANGFUSE_BLOB_STORAGE_ENDPOINT_WHITELISTED_IPS=127.0.0.1,::1
+# LANGFUSE_BLOB_STORAGE_ENDPOINT_WHITELISTED_IP_SEGMENTS=127.0.0.0/8

 # Langfuse AI Bedrock credentials
 AWS_ACCESS_KEY_ID="A123456789"
 AWS_SECRET_ACCESS_KEY="SAK123456789"
+LANGFUSE_LLM_CONNECTION_BEDROCK_API_KEY="1234567890abcdef"
 LANGFUSE_AWS_BEDROCK_REGION="eu-west-1"
-LANGFUSE_AWS_BEDROCK_MODEL="eu.anthropic.claude-3-haiku-20240307-v1:0"
+LANGFUSE_AWS_BEDROCK_MODEL="eu.anthropic.claude-haiku-4-5-20251001-v1:0"
+LANGFUSE_IN_APP_AGENT_AWS_PROFILE="playground"

-# Events table migration
-LANGFUSE_ENABLE_EVENTS_TABLE_OBSERVATIONS=true
-LANGFUSE_ENABLE_EVENTS_TABLE_FLAGS=true
-LANGFUSE_ENABLE_EVENTS_TABLE_V2_APIS=true
-LANGFUSE_EXPERIMENT_INSERT_INTO_EVENTS_TABLE=true
+# V4 migration flags.
+# Write target: `legacy` | `dual` | `events_only`.
+LANGFUSE_MIGRATION_V4_WRITE_MODE=dual
+# Gate the V4 events_full read paths (UI, tRPC, v2 APIs, observations API).
+# Self-hosted `dual` deployments must set this to `true` to offer users the V4
+# preview toggle; with `dual` + `false` the preview stays unavailable to users.
+LANGFUSE_MIGRATION_V4_ALLOW_PREVIEW_OPT_IN=true
+# OTel ingestion behaviour: `dual_write` (SDK-version dispatch) or `direct`.
+LANGFUSE_MIGRATION_V4_NATIVE_OTEL_BEHAVIOUR=dual_write
+
+# Legacy tracing UI controls
+LANGFUSE_DISABLE_LEGACY_TRACING_IO_SEARCH=false

 CLICKHOUSE_USE_LIGHTWEIGHT_UPDATE="true"
+# `auto` detects affected ClickHouse versions on startup. Set `true` to force
+# the workaround or `false` to force-disable it.
+CLICKHOUSE_DISABLE_LAZY_MATERIALIZATION=auto
@@ -54,6 +54,7 @@ OTEL_SERVICE_NAME="langfuse"

 # Enable experimental features, optional
 # LANGFUSE_ENABLE_EXPERIMENTAL_FEATURES=false
+# LANGFUSE_ENABLE_WEB_CALLOUTS=false

 # Auth, optional configuration
 # AUTH_DOMAINS_WITH_SSO_ENFORCEMENT=domain1.com,domain2.com
@@ -69,6 +70,7 @@ OTEL_SERVICE_NAME="langfuse"
 # AUTH_GOOGLE_ALLOWED_DOMAINS=langfuse.com,google.com # optional allowlist of workspace domains that can sign in via Google
 # AUTH_GOOGLE_CLIENT_AUTH_METHOD=
 # AUTH_GOOGLE_CHECKS=
+# AUTH_GOOGLE_ID_TOKEN_SIGNED_RESPONSE_ALG=
 # AUTH_GITHUB_CLIENT_ID=
 # AUTH_GITHUB_CLIENT_SECRET=
 # AUTH_GITHUB_ALLOW_ACCOUNT_LINKING=false
@@ -86,6 +88,7 @@ OTEL_SERVICE_NAME="langfuse"
 # AUTH_GITLAB_ISSUER=
 # AUTH_GITLAB_CLIENT_AUTH_METHOD=
 # AUTH_GITLAB_CHECKS=
+# AUTH_GITLAB_ID_TOKEN_SIGNED_RESPONSE_ALG=
 # AUTH_GITLAB_URL=
 # AUTH_AZURE_AD_CLIENT_ID=
 # AUTH_AZURE_AD_CLIENT_SECRET=
@@ -93,30 +96,35 @@ OTEL_SERVICE_NAME="langfuse"
 # AUTH_AZURE_AD_ALLOW_ACCOUNT_LINKING=false
 # AUTH_AZURE_AD_CLIENT_AUTH_METHOD=
 # AUTH_AZURE_AD_CHECKS=
+# AUTH_AZURE_AD_ID_TOKEN_SIGNED_RESPONSE_ALG=
 # AUTH_OKTA_CLIENT_ID=
 # AUTH_OKTA_CLIENT_SECRET=
 # AUTH_OKTA_ISSUER=
 # AUTH_OKTA_ALLOW_ACCOUNT_LINKING=false
 # AUTH_OKTA_CLIENT_AUTH_METHOD=
 # AUTH_OKTA_CHECKS=
+# AUTH_OKTA_ID_TOKEN_SIGNED_RESPONSE_ALG=
 # AUTH_AUTH0_CLIENT_ID=
 # AUTH_AUTH0_CLIENT_SECRET=
 # AUTH_AUTH0_ISSUER=
 # AUTH_AUTH0_ALLOW_ACCOUNT_LINKING=false
 # AUTH_AUTH0_CLIENT_AUTH_METHOD=
 # AUTH_AUTH0_CHECKS=
+# AUTH_AUTH0_ID_TOKEN_SIGNED_RESPONSE_ALG=
 # AUTH_COGNITO_CLIENT_ID=
 # AUTH_COGNITO_CLIENT_SECRET=
 # AUTH_COGNITO_ISSUER=
 # AUTH_COGNITO_ALLOW_ACCOUNT_LINKING=false
 # AUTH_COGNITO_CLIENT_AUTH_METHOD=
 # AUTH_COGNITO_CHECKS=
+# AUTH_COGNITO_ID_TOKEN_SIGNED_RESPONSE_ALG=
 # AUTH_KEYCLOAK_CLIENT_ID=
 # AUTH_KEYCLOAK_CLIENT_SECRET=
 # AUTH_KEYCLOAK_ISSUER=
 # AUTH_KEYCLOAK_ALLOW_ACCOUNT_LINKING=false
 # AUTH_KEYCLOAK_CLIENT_AUTH_METHOD=
 # AUTH_KEYCLOAK_CHECKS=
+# AUTH_KEYCLOAK_ID_TOKEN_SIGNED_RESPONSE_ALG=
 # AUTH_KEYCLOAK_NAME=
 # AUTH_WORKOS_CLIENT_ID=
 # AUTH_WORKOS_CLIENT_SECRET=
@@ -133,12 +141,14 @@ OTEL_SERVICE_NAME="langfuse"
 # AUTH_CUSTOM_ID_TOKEN=false # optional, default is true
 # AUTH_CUSTOM_CLIENT_AUTH_METHOD=
 # AUTH_CUSTOM_CHECKS=
+# AUTH_CUSTOM_ID_TOKEN_SIGNED_RESPONSE_ALG=
 # AUTH_JUMPCLOUD_CLIENT_ID=
 # AUTH_JUMPCLOUD_CLIENT_SECRET=
 # AUTH_JUMPCLOUD_ISSUER=
 # AUTH_JUMPCLOUD_ALLOW_ACCOUNT_LINKING=
 # AUTH_JUMPCLOUD_CLIENT_AUTH_METHOD=
 # AUTH_JUMPCLOUD_CHECKS=
+# AUTH_JUMPCLOUD_ID_TOKEN_SIGNED_RESPONSE_ALG=
 # AUTH_JUMPCLOUD_SCOPE=

 # Transactional email, optional
@@ -191,6 +201,7 @@ LANGFUSE_ENABLE_BLOB_STORAGE_FILE_LOG=true
 # REDIS_KEY_PREFIX=  # Optional: Prefix for Redis keys (useful for multi-tenant Redis instances)
                     # BullMQ queues will use this via BullMQ's native prefix option
                     # Cache operations will use this via ioredis keyPrefix
+# LANGFUSE_BULLMQ_SKIP_REDIS_VERSION_CHECK=false  # Set to true for Redis-compatible services that report a non-Redis version

 # Redis Cluster configuration (optional)
 # REDIS_CLUSTER_ENABLED=false
@@ -216,6 +227,10 @@ LANGFUSE_ENABLE_BLOB_STORAGE_FILE_LOG=true
 # CLICKHOUSE_USER=
 # CLICKHOUSE_PASSWORD=
 # CLICKHOUSE_CLUSTER_ENABLED=true
+# LANGFUSE_CLICKHOUSE_DELETED_MASK_CLEANER_ENABLED=false
+# LANGFUSE_CLICKHOUSE_DELETED_MASK_CLEANER_INTERVAL_MS=3600000
+# LANGFUSE_CLICKHOUSE_DELETED_MASK_CLEANER_SUBMIT_TIMEOUT_MS=60000
+# LANGFUSE_CLICKHOUSE_DELETED_MASK_CLEANER_CLUSTER_MODE_ENABLED=false

 # Ingestion configuration
 # LANGFUSE_INGESTION_QUEUE_DELAY_MS=
@@ -239,6 +254,13 @@ LANGFUSE_ENABLE_BLOB_STORAGE_FILE_LOG=true
 # Valid values: core, io, scores, observations, metrics
 # LANGFUSE_API_TRACEBYID_DEFAULT_FIELDS=

+# Comma-separated additional hostnames/origins accepted by the MCP endpoint.
+# LANGFUSE_MCP_ALLOWED_HOSTS=internal-langfuse.example.com,https://api.example.com
+
+# Legacy tracing UI controls
+# Disable input/output full-text search in legacy V3 tracing tables (metadata search remains available)
+# LANGFUSE_DISABLE_LEGACY_TRACING_IO_SEARCH=false
+
 ### START Enterprise Edition Configuration

 # Allowlisted users that can create new organizations, by default all users can create organizations
@@ -291,7 +313,6 @@ LANGFUSE_ENABLE_BLOB_STORAGE_FILE_LOG=true
 # Plain Chat
 # NEXT_PUBLIC_PLAIN_APP_ID=
 # PLAIN_AUTHENTICATION_SECRET=
-# PLAIN_API_KEY=
 # PLAIN_CARDS_API_TOKEN=

 # Pylon Support
@@ -303,6 +324,10 @@ LANGFUSE_ENABLE_BLOB_STORAGE_FILE_LOG=true
 # LANGFUSE_LLM_CONNECTION_WHITELISTED_HOST=
 # LANGFUSE_LLM_CONNECTION_WHITELISTED_IPS=
 # LANGFUSE_LLM_CONNECTION_WHITELISTED_IP_SEGMENTS=
+# Self-hosted only: allow internal hosts/IPs for user-configured blob storage endpoints.
+# LANGFUSE_BLOB_STORAGE_ENDPOINT_WHITELISTED_HOST=
+# LANGFUSE_BLOB_STORAGE_ENDPOINT_WHITELISTED_IPS=
+# LANGFUSE_BLOB_STORAGE_ENDPOINT_WHITELISTED_IP_SEGMENTS=
 # LANGFUSE_CACHE_MODEL_MATCH_ENABLED=
 # LANGFUSE_CACHE_MODEL_MATCH_TTL_SECONDS=

@@ -1,2 +1,5 @@
 # Currently inactive
 # * @langfuse/maintainers
+
+# Require maintainer review for GitHub configuration changes
+.github/ @langfuse/maintainers
@@ -15,10 +15,10 @@ Fixes # (issue)
 <!-- Please delete bullets that are not relevant. -->

 - [ ] Bug fix (non-breaking change which fixes an issue)
- [ ] Chore (refactoring code, technical debt, workflow improvements)
+- [ ] Chore (tooling, dependencies, CI, workflows, repo upkeep, or other maintenance work)
 - [ ] New feature (non-breaking change which adds functionality)
 - [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
- [ ] Refactor (does not change functionality, e.g. code style improvements, linting)
+- [ ] Refactor (restructures existing code without changing behavior, e.g. simplify logic, split modules, reduce duplication)
 - [ ] This change requires a documentation update

 ## Mandatory Tasks
@@ -0,0 +1,30 @@
+name: Notify Slack Failure
+description: Send a CI failure notification to a Slack Workflow webhook.
+
+inputs:
+  title:
+    description: Slack notification header.
+    required: true
+  message:
+    description: Slack notification fallback text.
+    required: true
+  webhook-url:
+    description: Slack Workflow webhook URL.
+    required: true
+
+runs:
+  using: composite
+  steps:
+    - name: Notify Slack
+      uses: slackapi/slack-github-action@45a88b9581bfab2566dc881e2cd66d334e621e2c # v3.0.3
+      with:
+        webhook: ${{ inputs.webhook-url }}
+        webhook-type: webhook-trigger
+        payload: |
+          title: "${{ inputs.title }}"
+          message: "${{ inputs.message }}"
+          ref: "${{ github.ref_name }}"
+          actor: "${{ github.actor }}"
+          event: "${{ github.event_name }}"
+          commit: "${{ github.sha }}"
+          workflow_url: "${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}"
@@ -12,7 +12,7 @@ updates:
    schedule:
      interval: "daily"
    cooldown:
-      default-days: 5
+      default-days: 7
    versioning-strategy: "increase"
    commit-message:
      prefix: chore
@@ -54,6 +54,8 @@ updates:
    directory: "/"
    schedule:
      interval: "weekly"
+    cooldown:
+      default-days: 8
    commit-message:
      prefix: ci
      include: scope
@@ -10,10 +10,21 @@ on:
        type: string
        description: Name of the service to be deployed, e.g. web-ingestion, web, or worker.
        required: true
+    # Environment secrets don't auto-resolve in reusable workflows.
+    # See: https://github.com/actions/runner/issues/3206
+    secrets:
+      AWS_ACCESS_KEY_ID:
+        required: true
+      AWS_SECRET_ACCESS_KEY:
+        required: true
+      SENTRY_AUTH_TOKEN:
+        required: false
 jobs:
  ecs-deploy:
    runs-on: blacksmith-4vcpu-ubuntu-2404
    environment: ${{ inputs.environment }}
+    permissions:
+      contents: read
    steps:
      - name: Get app name
        uses: winterjung/split@a211a1c46e35fcdc4097d59dd6282d4a9859651b # v2
@@ -22,52 +33,68 @@ jobs:
          msg: ${{ inputs.service }}
          separator: "-"
      - name: Checkout code
-        uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
+        uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
+        with:
+          persist-credentials: false
      - name: Authenticate with AWS
        # GitHub/AWS recommend to use OIDC here: https://github.com/aws-actions/configure-aws-credentials?tab=readme-ov-file#oidc
        # Probably more painful to configure, but would remove all long-lived credentials.
-        uses: aws-actions/configure-aws-credentials@7474bc4690e29a8392af63c5b98e7449536d5c3a # v4
+        uses: aws-actions/configure-aws-credentials@e7f100cf4c008499ea8adda475de1042d6975c7b # v6.2.0
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: ${{ vars.AWS_REGION }}
      - name: Login to AWS ECR
        id: login-ecr
-        uses: aws-actions/amazon-ecr-login@183a1442edf41672e66566b7fc560e297a290896 # v2
+        uses: aws-actions/amazon-ecr-login@fa648b43de3d4d023bcb3f89ed6940096949c419 # v2.1.5
      - name: Build, tag, and push Docker image
        env:
          REGISTRY: ${{ steps.login-ecr.outputs.registry }}
          REPOSITORY: ${{ steps.split.outputs._0 }}
          IMAGE_TAG: ${{ github.sha }}
+          STEPS_SPLIT_OUTPUTS__0: ${{ steps.split.outputs._0 }}
+          VARS_NEXT_PUBLIC_LANGFUSE_CLOUD_REGION: ${{ vars.NEXT_PUBLIC_LANGFUSE_CLOUD_REGION }}
+          VARS_NEXT_LANGFUSE_TRACING_SAMPLE_RATE: ${{ vars.NEXT_LANGFUSE_TRACING_SAMPLE_RATE }}
+          VARS_NEXT_PUBLIC_SENTRY_ENVIRONMENT: ${{ vars.NEXT_PUBLIC_SENTRY_ENVIRONMENT }}
+          VARS_NEXT_PUBLIC_DEMO_ORG_ID: ${{ vars.NEXT_PUBLIC_DEMO_ORG_ID }}
+          VARS_NEXT_PUBLIC_DEMO_PROJECT_ID: ${{ vars.NEXT_PUBLIC_DEMO_PROJECT_ID }}
+          VARS_NEXT_PUBLIC_SENTRY_DSN: ${{ vars.NEXT_PUBLIC_SENTRY_DSN }}
+          VARS_NEXT_PUBLIC_POSTHOG_KEY: ${{ vars.NEXT_PUBLIC_POSTHOG_KEY }}
+          VARS_NEXT_PUBLIC_POSTHOG_HOST: ${{ vars.NEXT_PUBLIC_POSTHOG_HOST }}
+          VARS_NEXT_PUBLIC_PLAIN_APP_ID: ${{ vars.NEXT_PUBLIC_PLAIN_APP_ID }}
+          VARS_SENTRY_ORG: ${{ vars.SENTRY_ORG }}
+          VARS_SENTRY_PROJECT: ${{ vars.SENTRY_PROJECT }}
+          VARS_NEXT_PUBLIC_LANGFUSE_TRACING_SAMPLE_RATE: ${{ vars.NEXT_PUBLIC_LANGFUSE_TRACING_SAMPLE_RATE }}
+          SECRETS_SENTRY_AUTH_TOKEN: ${{ secrets.SENTRY_AUTH_TOKEN }}
        run: |
          docker build \
            -t $REGISTRY/$REPOSITORY:$IMAGE_TAG \
-            -f ./${{ steps.split.outputs._0 }}/Dockerfile \
-            --build-arg NEXT_PUBLIC_LANGFUSE_CLOUD_REGION=${{ vars.NEXT_PUBLIC_LANGFUSE_CLOUD_REGION }} \
-            --build-arg NEXT_LANGFUSE_TRACING_SAMPLE_RATE=${{ vars.NEXT_LANGFUSE_TRACING_SAMPLE_RATE }} \
-            --build-arg NEXT_PUBLIC_SENTRY_ENVIRONMENT=${{ vars.NEXT_PUBLIC_SENTRY_ENVIRONMENT }} \
-            --build-arg NEXT_PUBLIC_DEMO_ORG_ID=${{ vars.NEXT_PUBLIC_DEMO_ORG_ID }} \
-            --build-arg NEXT_PUBLIC_DEMO_PROJECT_ID=${{ vars.NEXT_PUBLIC_DEMO_PROJECT_ID }} \
-            --build-arg NEXT_PUBLIC_SENTRY_DSN=${{ vars.NEXT_PUBLIC_SENTRY_DSN }} \
-            --build-arg NEXT_PUBLIC_BUILD_ID=${{ github.sha }} \
-            --build-arg NEXT_PUBLIC_POSTHOG_KEY=${{ vars.NEXT_PUBLIC_POSTHOG_KEY }} \
-            --build-arg NEXT_PUBLIC_POSTHOG_HOST=${{ vars.NEXT_PUBLIC_POSTHOG_HOST }} \
-            --build-arg NEXT_PUBLIC_PLAIN_APP_ID=${{ vars.NEXT_PUBLIC_PLAIN_APP_ID }} \
-            --build-arg SENTRY_AUTH_TOKEN=${{ secrets.SENTRY_AUTH_TOKEN }} \
-            --build-arg SENTRY_ORG=${{ vars.SENTRY_ORG }} \
-            --build-arg SENTRY_PROJECT=${{ vars.SENTRY_PROJECT }} \
-            --build-arg NEXT_PUBLIC_LANGFUSE_TRACING_SAMPLE_RATE=${{ vars.NEXT_PUBLIC_LANGFUSE_TRACING_SAMPLE_RATE }} \
+            -f ./${STEPS_SPLIT_OUTPUTS__0}/Dockerfile \
+            --build-arg NEXT_PUBLIC_LANGFUSE_CLOUD_REGION=${VARS_NEXT_PUBLIC_LANGFUSE_CLOUD_REGION} \
+            --build-arg NEXT_LANGFUSE_TRACING_SAMPLE_RATE=${VARS_NEXT_LANGFUSE_TRACING_SAMPLE_RATE} \
+            --build-arg NEXT_PUBLIC_SENTRY_ENVIRONMENT=${VARS_NEXT_PUBLIC_SENTRY_ENVIRONMENT} \
+            --build-arg NEXT_PUBLIC_DEMO_ORG_ID=${VARS_NEXT_PUBLIC_DEMO_ORG_ID} \
+            --build-arg NEXT_PUBLIC_DEMO_PROJECT_ID=${VARS_NEXT_PUBLIC_DEMO_PROJECT_ID} \
+            --build-arg NEXT_PUBLIC_SENTRY_DSN=${VARS_NEXT_PUBLIC_SENTRY_DSN} \
+            --build-arg NEXT_PUBLIC_BUILD_ID=${IMAGE_TAG} \
+            --build-arg NEXT_PUBLIC_POSTHOG_KEY=${VARS_NEXT_PUBLIC_POSTHOG_KEY} \
+            --build-arg NEXT_PUBLIC_POSTHOG_HOST=${VARS_NEXT_PUBLIC_POSTHOG_HOST} \
+            --build-arg NEXT_PUBLIC_PLAIN_APP_ID=${VARS_NEXT_PUBLIC_PLAIN_APP_ID} \
+            --build-arg SENTRY_AUTH_TOKEN=${SECRETS_SENTRY_AUTH_TOKEN} \
+            --build-arg SENTRY_ORG=${VARS_SENTRY_ORG} \
+            --build-arg SENTRY_PROJECT=${VARS_SENTRY_PROJECT} \
+            --build-arg NEXT_PUBLIC_LANGFUSE_TRACING_SAMPLE_RATE=${VARS_NEXT_PUBLIC_LANGFUSE_TRACING_SAMPLE_RATE} \
            .
          docker push $REGISTRY/$REPOSITORY:$IMAGE_TAG
      - name: Render AWS ECS Task Definition
        id: render-task-definition
-        uses: aws-actions/amazon-ecs-render-task-definition@77954e213ba1f9f9cb016b86a1d4f6fcdea0d57e # v1
+        uses: aws-actions/amazon-ecs-render-task-definition@6853cfae8c3a7d978fbf68b5a55453395541dfbb # v1.8.5
        with:
          container-name: ${{ inputs.service }}
          image: ${{ steps.login-ecr.outputs.registry }}/${{ steps.split.outputs._0 }}:${{ github.sha }}
          task-definition-family: ${{ inputs.environment }}-${{ inputs.service }}
      - name: Update AWS ECS Service
-        uses: aws-actions/amazon-ecs-deploy-task-definition@fc8fc60f3a60ffd500fcb13b209c59d221ac8c8c # v2
+        uses: aws-actions/amazon-ecs-deploy-task-definition@a310a830f5c14e583e35d84e4e1ec7dd177c3c9c # v2.6.2
        with:
          task-definition: ${{ steps.render-task-definition.outputs.task-definition }}
          service: ${{ inputs.environment }}-${{ inputs.service }}
@@ -28,14 +28,14 @@ jobs:
        run: docker compose -f "docker-compose.yml" up -d --build

      - name: Setup pnpm
-        uses: pnpm/action-setup@a3252b78c470c02df07e9d59298aecedc3ccdd6d # v3
+        uses: pnpm/action-setup@739bfe42ca9233c5e6aca07c1a25a9d34aca49b0 # v6.0.7
        with:
-          version: 10.33.0
+          version: 11.4.0

-      - name: Setup Node 18
+      - name: Setup Node 24
        uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4
        with:
-          node-version: 18
+          node-version: 24

      - name: Get pnpm store directory
        id: pnpm-cache
@@ -9,6 +9,8 @@ on:
  issue_comment:
    types: [created]

+permissions: {}
+
 jobs:
  retrigger_cla:
    # Only run on PR comments (not issue comments) with the /check-cla command
@@ -0,0 +1,70 @@
+name: Security review
+
+on:
+  pull_request:
+    types:
+      - opened
+      - reopened
+      - synchronize
+      - ready_for_review
+
+permissions:
+  contents: read
+  pull-requests: write
+
+jobs:
+  security-review:
+    name: Security review
+    if: github.event.pull_request.draft == false && github.event.pull_request.head.repo.full_name == github.repository
+    runs-on: ubuntu-latest
+    timeout-minutes: 30
+
+    steps:
+      - name: Checkout
+        uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
+        with:
+          ref: ${{ github.event.pull_request.head.sha }}
+          fetch-depth: 2
+          persist-credentials: false
+
+      - name: Checkout Claude Code security review action
+        uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
+        with:
+          repository: anthropics/claude-code-security-review
+          ref: 0c6a49f1fa56a1d472575da86a94dbc1edb78eda
+          path: .github/actions/claude-code-security-review
+          fetch-depth: 1
+          persist-credentials: false
+
+      - name: Patch Claude API validation model
+        shell: bash
+        run: |
+          python <<'PY'
+          import os
+          from pathlib import Path
+
+          action_path = (
+              Path(os.environ["GITHUB_WORKSPACE"])
+              / ".github"
+              / "actions"
+              / "claude-code-security-review"
+              / "claudecode"
+              / "claude_api_client.py"
+          )
+          if not action_path.exists():
+              raise SystemExit(f"Expected Claude API client not found at {action_path}")
+          source = action_path.read_text()
+          old = 'model="claude-3-5-haiku-20241022",'
+          new = "model=self.model,"
+          if old not in source:
+              raise SystemExit(f"Expected validation model not found in {action_path}")
+          action_path.write_text(source.replace(old, new))
+          PY
+
+      - name: Run Claude Code security review
+        uses: ./.github/actions/claude-code-security-review
+        with:
+          claude-api-key: ${{ secrets.CLAUDE_API_KEY }}
+          claude-model: claude-opus-4-1-20250805
+          comment-pr: true
+          run-every-commit: true
@@ -1,14 +1,15 @@
 name: Claude Review on Maintainer PRs

 on:
-  pull_request_target:
+  pull_request:
    types:
      - opened
      - ready_for_review

 jobs:
  comment:
-    if: github.event.pull_request.draft == false
+    # Only run on PRs that are not drafts and are from the same repository (i.e., not from forks)
+    if: github.event.pull_request.draft == false && github.event.pull_request.head.repo.full_name == github.repository
    runs-on: ubuntu-latest
    permissions:
      issues: write
@@ -16,7 +17,7 @@ jobs:
    steps:
      - name: Check author permission and existing review request
        id: check
-        uses: actions/github-script@f28e40c7f34bde8b3046d885e986cb6290c5673b # v7
+        uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
        with:
          script: |
            const owner = context.repo.owner;
@@ -57,7 +58,7 @@ jobs:

      - name: Add Claude review comment
        if: steps.check.outputs.should_comment == 'true'
-        uses: actions/github-script@f28e40c7f34bde8b3046d885e986cb6290c5673b # v7
+        uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
        with:
          script: |
            await github.rest.issues.createComment({
@@ -55,11 +55,13 @@ jobs:
        # your codebase is analyzed, see https://docs.github.com/en/code-security/code-scanning/creating-an-advanced-setup-for-code-scanning/codeql-code-scanning-for-compiled-languages
    steps:
      - name: Checkout repository
-        uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
+        uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
+        with:
+          persist-credentials: false

      # Initializes the CodeQL tools for scanning.
      - name: Initialize CodeQL
-        uses: github/codeql-action/init@5c8a8a642e79153f5d047b10ec1cba1d1cc65699 # v3
+        uses: github/codeql-action/init@87557b9c84dde89fdd9b10e88954ac2f4248e463 # v4.36.1
        with:
          languages: ${{ matrix.language }}
          build-mode: ${{ matrix.build-mode }}
@@ -87,6 +89,6 @@ jobs:
          exit 1

      - name: Perform CodeQL Analysis
-        uses: github/codeql-action/analyze@5c8a8a642e79153f5d047b10ec1cba1d1cc65699 # v3
+        uses: github/codeql-action/analyze@87557b9c84dde89fdd9b10e88954ac2f4248e463 # v4.36.1
        with:
          category: "/language:${{matrix.language}}"
@@ -22,6 +22,8 @@ jobs:

    steps:
      - name: Checkout
-        uses: actions/checkout@f43a0e5ff2bd294095638e18286ca9a3d1956744 # v3
+        uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
+        with:
+          persist-credentials: false
      - name: Codespell
-        uses: codespell-project/actions-codespell@406322ec52dd7b488e48c1c4b82e2a8b3a1bf630 # v2
+        uses: codespell-project/actions-codespell@8f01853be192eb0f849a5c7d721450e7a467c579 # v2.2
@@ -6,6 +6,8 @@ on:
      - main
  workflow_dispatch:

+permissions: {}
+
 jobs:
  rebase-dependabot:
    runs-on: ubuntu-latest
@@ -27,6 +27,8 @@ on:
          - prod-jp
        required: true

+permissions: {}
+
 concurrency:
  # Support concurrent `push` and `workflow_dispatch`` actions
  group: deploy-${{ github.event_name }}-${{ github.ref }}
@@ -41,7 +43,7 @@ jobs:
    steps:
      - name: Get affected services
        id: affected-services
-        uses: actions/github-script@f28e40c7f34bde8b3046d885e986cb6290c5673b # v7
+        uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
        with:
          script: |
            if (context.eventName === "workflow_dispatch") {
@@ -56,7 +58,7 @@ jobs:
            return "[]"
          result-encoding: string
      - name: Print services to build
-        uses: actions/github-script@f28e40c7f34bde8b3046d885e986cb6290c5673b # v7
+        uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
        env:
          services: ${{ steps.affected-services.outputs.result }}
        with:
@@ -71,7 +73,7 @@ jobs:
    steps:
      - name: Get affected environments
        id: affected-environments
-        uses: actions/github-script@f28e40c7f34bde8b3046d885e986cb6290c5673b # v7
+        uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
        with:
          script: |
            if (context.eventName === "workflow_dispatch") {
@@ -88,7 +90,7 @@ jobs:
            return "[]"
          result-encoding: string
      - name: Print environments to build
-        uses: actions/github-script@f28e40c7f34bde8b3046d885e986cb6290c5673b # v7
+        uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
        env:
          environments: ${{ steps.affected-environments.outputs.result }}
        with:
@@ -99,7 +101,14 @@ jobs:
  ecs-deploy:
    uses: ./.github/workflows/_deploy_ecs_service.yml
    needs: [affected-services, affected-environments]
-    secrets: inherit
+    permissions:
+      contents: read
+    # Environment secrets must be passed explicitly to reusable workflows.
+    # See: https://github.com/actions/runner/issues/3206
+    secrets:
+      AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
+      AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
+      SENTRY_AUTH_TOKEN: ${{ secrets.SENTRY_AUTH_TOKEN }}
    strategy:
      matrix:
        service: ${{ fromJson(needs.affected-services.outputs.services) }}
@@ -107,3 +116,20 @@ jobs:
    with:
      service: ${{ matrix.service }}
      environment: ${{ matrix.environment }}
+
+  notify-slack-on-failure:
+    needs: [affected-services, affected-environments, ecs-deploy]
+    if: failure()
+    runs-on: blacksmith-4vcpu-ubuntu-2404
+    permissions:
+      contents: read
+    steps:
+      - uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
+        with:
+          persist-credentials: false
+      - name: Notify Slack
+        uses: ./.github/actions/notify-slack-failure
+        with:
+          title: "❌ Deploy Failed"
+          message: "❌ Deploy failed on ${{ github.ref_name }}"
+          webhook-url: ${{ secrets.SLACK_CI_FAILURE_WORKFLOW_WEBHOOK_URL }}
@@ -0,0 +1,80 @@
+name: Label PRs with conflicts
+
+on:
+  pull_request:
+    types: [opened, reopened, synchronize]
+  push:
+    branches: [main]
+
+permissions:
+  pull-requests: read
+  issues: write
+
+jobs:
+  label-prs-with-conflicts:
+    runs-on: ubuntu-latest
+    # Only run pull_request events for same-repo PRs to avoid permission issues with forks.
+    # Fork PRs are intentionally not labeled by this workflow.
+    if: github.event_name == 'push' || github.event.pull_request.head.repo.full_name == github.repository
+
+    steps:
+      - name: Check mergeability and update label
+        env:
+          GH_TOKEN: ${{ github.token }}
+          REPO: ${{ github.repository }}
+          EVENT_NAME: ${{ github.event_name }}
+          LABEL_NAME: has-conflicts
+          PR_NUMBER: ${{ github.event.pull_request.number }}
+        run: |
+          set -euo pipefail
+
+          update_label_if_needed() {
+            local pr_number="$1"
+            local remove_stale_label="${2:-false}"
+
+            mergeable=$(gh api "repos/${REPO}/pulls/${pr_number}" --jq '.mergeable')
+            for delay in 3 6 12 24; do
+              if [ "${mergeable}" != "null" ]; then
+                break
+              fi
+
+              sleep "${delay}"
+              mergeable=$(gh api "repos/${REPO}/pulls/${pr_number}" --jq '.mergeable')
+            done
+
+            echo "PR #${pr_number}: mergeable=${mergeable}"
+
+            if [ "${mergeable}" = "null" ]; then
+              echo "::warning::PR #${pr_number}: mergeability was not computed after retries"
+              return 1
+            fi
+
+            if [ "${mergeable}" = "false" ]; then
+              gh api \
+                --method POST \
+                "repos/${REPO}/issues/${pr_number}/labels" \
+                -f "labels[]=${LABEL_NAME}"
+            elif [ "${mergeable}" = "true" ] && [ "${remove_stale_label}" = "true" ]; then
+              gh api \
+                --method DELETE \
+                "repos/${REPO}/issues/${pr_number}/labels/${LABEL_NAME}" \
+                --silent || true
+            fi
+          }
+
+          if [ "${EVENT_NAME}" = "pull_request" ]; then
+            update_label_if_needed "${PR_NUMBER}" true
+          else
+            sleep 10
+            cutoff_date=$(date -u -d "14 days ago" +%F)
+            # On main pushes, only scan same-repo PRs that are not already labeled.
+            # Stale has-conflicts labels are intentionally removed only on PR updates.
+            gh api --paginate "search/issues?q=repo:${REPO}+is:pr+is:open+-label:${LABEL_NAME}+updated:>=${cutoff_date}&per_page=100" --jq '.items[].number' \
+              | while read -r pr; do
+                  head_repo=$(gh api "repos/${REPO}/pulls/${pr}" --jq '.head.repo.full_name') || continue
+                  if [ "${head_repo}" != "${REPO}" ]; then
+                    continue
+                  fi
+                  update_label_if_needed "${pr}" || true
+                done
+          fi
@@ -9,15 +9,21 @@ on:
    branches:
      - "main"

+permissions:
+  contents: read
+  checks: write # Needed to create a check run for the license compliance check results
+
 jobs:
  license_check:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
-        uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
+        uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
+        with:
+          persist-credentials: false

      - name: Setup node
-        uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4
+        uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
        with:
          node-version: 18

@@ -32,7 +38,7 @@ jobs:

      - name: Check license-checker CSV file without headers
        id: license_check_report
-        uses: pilosus/action-pip-license-checker@cc7a461bfa27b44ad187b8578c881ef5138c13fd # v2
+        uses: pilosus/action-pip-license-checker@e909b0226ff49d3235c99c4585bc617f49fff16a # v3.1.0
        with:
          external: "npm-license-checker.csv"
          external-format: "csv"
@@ -44,6 +50,8 @@ jobs:
          github-token: ${{ secrets.GITHUB_TOKEN }}
      - name: Echo error
        if: failure()
-        run: echo "::error::${{ steps.license_check_report.outputs.report }}"
+        run: echo "::error::${STEPS_LICENSE_CHECK_REPORT_OUTPUTS_REPORT}"
+        env:
+          STEPS_LICENSE_CHECK_REPORT_OUTPUTS_REPORT: ${{ steps.license_check_report.outputs.report }}
      - name: Delete license-checker CSV file
        run: rm npm-license-checker.csv
@@ -12,6 +12,8 @@ concurrency:
  group: promote-main-to-production
  cancel-in-progress: false

+permissions: {}
+
 jobs:
  promote:
    runs-on: ubuntu-latest
@@ -19,16 +21,19 @@ jobs:
    steps:
      - name: Validate confirmation
        run: |
-          if [ "${{ github.event.inputs.confirm }}" != "promote" ]; then
+          if [ "${INPUT_CONFIRM}" != "promote" ]; then
            echo "Input 'confirm' must be 'promote'."
            exit 1
          fi
+        env:
+          INPUT_CONFIRM: ${{ github.event.inputs.confirm }}

-      - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
+      - uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
        with:
          ref: main
          fetch-depth: 0
          token: ${{ secrets.GH_ACCESS_TOKEN }}
+          persist-credentials: false

      - name: Print commit refs
        run: |
@@ -41,4 +46,6 @@ jobs:
          fi

      - name: Force push main to production
-        run: git push origin +main:production
+        run: git push "https://x-access-token:${GH_ACCESS_TOKEN}@github.com/${GITHUB_REPOSITORY}.git" +main:production
+        env:
+          GH_ACCESS_TOKEN: ${{ secrets.GH_ACCESS_TOKEN }}
@@ -5,15 +5,37 @@ on:
    tags:
      - "v3.[0-9]+.[0-9]+" # Semantic version tags

+permissions: {}
+
 jobs:
  release:
-    runs-on: ubuntu-latest
+    runs-on: blacksmith-4vcpu-ubuntu-2404
    environment: "protected branches"
    steps:
-      - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
+      - uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
        with:
          ref: main # Always checkout main even for tagged releases
          fetch-depth: 0
          token: ${{ secrets.GH_ACCESS_TOKEN }}
+          persist-credentials: false
      - name: Push to production
-        run: git push origin +main:production
+        run: git push "https://x-access-token:${GH_ACCESS_TOKEN}@github.com/${GITHUB_REPOSITORY}.git" +main:production
+        env:
+          GH_ACCESS_TOKEN: ${{ secrets.GH_ACCESS_TOKEN }}
+
+  notify-slack-on-failure:
+    needs: release
+    if: failure()
+    runs-on: blacksmith-4vcpu-ubuntu-2404
+    permissions:
+      contents: read
+    steps:
+      - uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
+        with:
+          persist-credentials: false
+      - name: Notify Slack
+        uses: ./.github/actions/notify-slack-failure
+        with:
+          title: "❌ Release Failed"
+          message: "❌ Release failed on ${{ github.ref_name }}"
+          webhook-url: ${{ secrets.SLACK_CI_FAILURE_WORKFLOW_WEBHOOK_URL }}
@@ -12,23 +12,35 @@ concurrency:
  group: sdk-api-spec-${{ github.ref }}
  cancel-in-progress: true

+permissions:
+  contents: read
+
 jobs:
  generate-sdk-api-specs:
-    runs-on: ubuntu-latest
+    runs-on: blacksmith-4vcpu-ubuntu-2404
+    outputs:
+      python_has_changes: ${{ steps.prepare-python-sdk.outputs.has_changes }}
+      python_branch_name: ${{ steps.prepare-python-sdk.outputs.branch_name }}
+      typescript_has_changes: ${{ steps.prepare-typescript-sdk.outputs.has_changes }}
+      typescript_branch_name: ${{ steps.prepare-typescript-sdk.outputs.branch_name }}
    steps:
      - name: Checkout repository
-        uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
+        uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
+        with:
+          persist-credentials: false

      - name: Install pnpm
-        uses: pnpm/action-setup@eae0cfeb286e66ffb5155f1a79b90583a127a68b # v2
+        uses: pnpm/action-setup@0e279bb959325dab635dd2c09392533439d90093 # v6.0.8
        with:
-          version: 10.33.0
+          version: 11.4.0

+      # Keep this secret-bearing job cache-cold to avoid exposing Fern
+      # credentials to potentially poisoned shared package-manager state.
      - name: Setup Node.js
-        uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4
+        uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
        with:
          node-version: "24"
-          cache: "pnpm"
+          package-manager-cache: false

      - name: Install dependencies
        run: pnpm install
@@ -39,13 +51,12 @@ jobs:
        run: npx fern-api generate --api server --force

      - name: Install uv
-        uses: astral-sh/setup-uv@cec208311dfd045dd5311c1add060b2062131d57 # v8
+        uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0
        with:
          version: "0.11.2"

-      - name: Update Python SDK
-        env:
-          GH_ACCESS_TOKEN: ${{ secrets.GH_ACCESS_TOKEN }}
+      - name: Prepare Python SDK update
+        id: prepare-python-sdk
        run: |
          # Clone langfuse-python repository
          git clone https://github.com/langfuse/langfuse-python.git ../langfuse-python
@@ -53,8 +64,6 @@ jobs:

          git config user.name "langfuse-bot"
          git config user.email "langfuse-bot@langfuse.com"
-          echo "$GH_ACCESS_TOKEN" | gh auth login --with-token
-          gh auth setup-git

          # Delete existing API contents
          rm -rf langfuse/api/*
@@ -73,30 +82,31 @@ jobs:
          # Check for changes and show diff for debugging
          if git diff --quiet; then
            echo "No changes detected in Python SDK"
+            echo "has_changes=false" >> "$GITHUB_OUTPUT"
            exit 0
          fi

-          # Close existing api-spec-bot PRs
-          gh pr list --author langfuse-bot --state open --json number --jq '.[].number' | xargs -I {} gh pr close {}
-
-          # Get the GitHub username of the original commit author
-          cd ../langfuse
-          ORIGINAL_AUTHOR=$(gh api repos/langfuse/langfuse/commits/${GITHUB_SHA} --jq '.author.login')
-          cd ../langfuse-python
-
-          # Create new branch and push changes
+          # Create new branch and commit changes
          BRANCH_NAME="api-spec-bot-${GITHUB_SHA::7}"
          git checkout -b "$BRANCH_NAME"
          git add .
          git commit -m "feat(api): update API spec from langfuse/langfuse ${GITHUB_SHA::7}"
-          git push origin "$BRANCH_NAME"
+          git bundle create "$RUNNER_TEMP/langfuse-python-api-spec-bot.bundle" "$BRANCH_NAME"

-          # Create PR with original author as reviewer
-          gh pr create --title "feat(api): update API spec from langfuse/langfuse ${GITHUB_SHA::7}" --body "" --reviewer "$ORIGINAL_AUTHOR"
+          echo "has_changes=true" >> "$GITHUB_OUTPUT"
+          echo "branch_name=$BRANCH_NAME" >> "$GITHUB_OUTPUT"

-      - name: Update TypeScript SDK
-        env:
-          GH_ACCESS_TOKEN: ${{ secrets.GH_ACCESS_TOKEN }}
+      - name: Upload Python SDK update
+        if: steps.prepare-python-sdk.outputs.has_changes == 'true'
+        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
+        with:
+          name: langfuse-python-sdk-update
+          path: ${{ runner.temp }}/langfuse-python-api-spec-bot.bundle
+          if-no-files-found: error
+          retention-days: 1
+
+      - name: Prepare TypeScript SDK update
+        id: prepare-typescript-sdk
        run: |
          # Clone langfuse-js repository
          git clone https://github.com/langfuse/langfuse-js.git ../langfuse-js
@@ -105,8 +115,6 @@ jobs:
          # Configure git
          git config user.name "langfuse-bot"
          git config user.email "langfuse-bot@langfuse.com"
-          echo "$GH_ACCESS_TOKEN" | gh auth login --with-token
-          gh auth setup-git

          # Delete existing API contents
          rm -rf packages/core/src/api/*
@@ -125,30 +133,164 @@ jobs:

          # Install dependencies and format
          corepack enable
-          corepack prepare pnpm@10.33.0 --activate
+          corepack prepare pnpm@11.4.0 --activate
          pnpm install
          pnpm format

          # Check for changes and show diff for debugging
          if git diff --quiet; then
            echo "No changes detected in TypeScript SDK"
+            echo "has_changes=false" >> "$GITHUB_OUTPUT"
            exit 0
          fi

-          # Close existing api-spec-bot PRs
-          gh pr list --author langfuse-bot --state open --json number --jq '.[].number' | xargs -I {} gh pr close {}
-
-          # Get the GitHub username of the original commit author
-          cd ../langfuse
-          ORIGINAL_AUTHOR=$(gh api repos/langfuse/langfuse/commits/${GITHUB_SHA} --jq '.author.login')
-          cd ../langfuse-js
-
-          # Create new branch and push changes
+          # Create new branch and commit changes
          BRANCH_NAME="api-spec-bot-${GITHUB_SHA::7}"
          git checkout -b "$BRANCH_NAME"
          git add .
          git commit -m "feat(api): update API spec from langfuse/langfuse ${GITHUB_SHA::7}" --no-verify
-          git push origin "$BRANCH_NAME"
+          git bundle create "$RUNNER_TEMP/langfuse-js-api-spec-bot.bundle" "$BRANCH_NAME"

-          # Create PR with original author as reviewer
-          gh pr create --title "feat(api): update API spec from langfuse/langfuse ${GITHUB_SHA::7}" --body "" --reviewer "$ORIGINAL_AUTHOR"
+          echo "has_changes=true" >> "$GITHUB_OUTPUT"
+          echo "branch_name=$BRANCH_NAME" >> "$GITHUB_OUTPUT"
+
+      - name: Upload TypeScript SDK update
+        if: steps.prepare-typescript-sdk.outputs.has_changes == 'true'
+        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
+        with:
+          name: langfuse-js-sdk-update
+          path: ${{ runner.temp }}/langfuse-js-api-spec-bot.bundle
+          if-no-files-found: error
+          retention-days: 1
+
+  push-python-sdk:
+    needs: generate-sdk-api-specs
+    if: needs.generate-sdk-api-specs.outputs.python_has_changes == 'true'
+    runs-on: blacksmith-4vcpu-ubuntu-2404
+    steps:
+      - name: Download Python SDK update
+        uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
+        with:
+          name: langfuse-python-sdk-update
+          path: ${{ runner.temp }}/langfuse-python-sdk-update
+
+      - name: Push Python SDK update
+        env:
+          BRANCH_NAME: ${{ needs.generate-sdk-api-specs.outputs.python_branch_name }}
+          GH_TOKEN: ${{ secrets.GH_ACCESS_TOKEN }}
+        run: |
+          set -euo pipefail
+
+          GIT_BIN="$(command -v git)"
+          GH_BIN="$(command -v gh)"
+          BUNDLE_PATH="$RUNNER_TEMP/langfuse-python-sdk-update/langfuse-python-api-spec-bot.bundle"
+          PUSH_REPO="$RUNNER_TEMP/push-langfuse-python"
+
+          mkdir -p "$PUSH_REPO"
+          "$GIT_BIN" -C "$PUSH_REPO" init -q
+          "$GIT_BIN" -C "$PUSH_REPO" fetch --no-tags "$BUNDLE_PATH" "$BRANCH_NAME:refs/heads/$BRANCH_NAME"
+
+          # Close existing api-spec-bot PRs
+          "$GH_BIN" pr list --repo langfuse/langfuse-python --author langfuse-bot --state open --json number --jq '.[].number' | while read -r pr_number; do
+            if [ -n "$pr_number" ]; then
+              "$GH_BIN" pr close --repo langfuse/langfuse-python "$pr_number"
+            fi
+          done
+
+          # Resolve the reviewer before pushing so a transient GitHub API failure cannot orphan a branch.
+          ORIGINAL_AUTHOR="$("$GH_BIN" api "repos/langfuse/langfuse/commits/${GITHUB_SHA}" --jq '.author.login // empty')"
+          REVIEWER_ARGS=()
+          if [ -n "$ORIGINAL_AUTHOR" ]; then
+            REVIEWER_ARGS=(--reviewer "$ORIGINAL_AUTHOR")
+          fi
+
+          # Push new branch and create PR from a fresh repo that never ran SDK install scripts.
+          GIT_CONFIG_GLOBAL=/dev/null GIT_CONFIG_SYSTEM=/dev/null "$GIT_BIN" \
+            -C "$PUSH_REPO" \
+            -c credential.helper= \
+            -c credential.https://github.com.helper= \
+            -c core.hooksPath=/dev/null \
+            push "https://x-access-token:${GH_TOKEN}@github.com/langfuse/langfuse-python.git" "$BRANCH_NAME"
+
+          "$GH_BIN" pr create \
+            --repo langfuse/langfuse-python \
+            --head "$BRANCH_NAME" \
+            --title "feat(api): update API spec from langfuse/langfuse ${GITHUB_SHA::7}" \
+            --body "" \
+            "${REVIEWER_ARGS[@]}"
+
+  push-typescript-sdk:
+    needs: generate-sdk-api-specs
+    if: needs.generate-sdk-api-specs.outputs.typescript_has_changes == 'true'
+    runs-on: blacksmith-4vcpu-ubuntu-2404
+    steps:
+      - name: Download TypeScript SDK update
+        uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
+        with:
+          name: langfuse-js-sdk-update
+          path: ${{ runner.temp }}/langfuse-js-sdk-update
+
+      - name: Push TypeScript SDK update
+        env:
+          BRANCH_NAME: ${{ needs.generate-sdk-api-specs.outputs.typescript_branch_name }}
+          GH_TOKEN: ${{ secrets.GH_ACCESS_TOKEN }}
+        run: |
+          set -euo pipefail
+
+          GIT_BIN="$(command -v git)"
+          GH_BIN="$(command -v gh)"
+          BUNDLE_PATH="$RUNNER_TEMP/langfuse-js-sdk-update/langfuse-js-api-spec-bot.bundle"
+          PUSH_REPO="$RUNNER_TEMP/push-langfuse-js"
+
+          mkdir -p "$PUSH_REPO"
+          "$GIT_BIN" -C "$PUSH_REPO" init -q
+          "$GIT_BIN" -C "$PUSH_REPO" fetch --no-tags "$BUNDLE_PATH" "$BRANCH_NAME:refs/heads/$BRANCH_NAME"
+
+          # Close existing api-spec-bot PRs
+          "$GH_BIN" pr list --repo langfuse/langfuse-js --author langfuse-bot --state open --json number --jq '.[].number' | while read -r pr_number; do
+            if [ -n "$pr_number" ]; then
+              "$GH_BIN" pr close --repo langfuse/langfuse-js "$pr_number"
+            fi
+          done
+
+          # Resolve the reviewer before pushing so a transient GitHub API failure cannot orphan a branch.
+          ORIGINAL_AUTHOR="$("$GH_BIN" api "repos/langfuse/langfuse/commits/${GITHUB_SHA}" --jq '.author.login // empty')"
+          REVIEWER_ARGS=()
+          if [ -n "$ORIGINAL_AUTHOR" ]; then
+            REVIEWER_ARGS=(--reviewer "$ORIGINAL_AUTHOR")
+          fi
+
+          # Push new branch and create PR from a fresh repo that never ran SDK install scripts.
+          GIT_CONFIG_GLOBAL=/dev/null GIT_CONFIG_SYSTEM=/dev/null "$GIT_BIN" \
+            -C "$PUSH_REPO" \
+            -c credential.helper= \
+            -c credential.https://github.com.helper= \
+            -c core.hooksPath=/dev/null \
+            push "https://x-access-token:${GH_TOKEN}@github.com/langfuse/langfuse-js.git" "$BRANCH_NAME"
+
+          "$GH_BIN" pr create \
+            --repo langfuse/langfuse-js \
+            --head "$BRANCH_NAME" \
+            --title "feat(api): update API spec from langfuse/langfuse ${GITHUB_SHA::7}" \
+            --body "" \
+            "${REVIEWER_ARGS[@]}"
+
+  notify-slack-on-failure:
+    needs:
+      - generate-sdk-api-specs
+      - push-python-sdk
+      - push-typescript-sdk
+    if: failure()
+    runs-on: blacksmith-4vcpu-ubuntu-2404
+    permissions:
+      contents: read
+    steps:
+      - uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
+        with:
+          persist-credentials: false
+      - name: Notify Slack
+        uses: ./.github/actions/notify-slack-failure
+        with:
+          title: "❌ SDK API Spec Generation Failed"
+          message: "❌ SDK API spec generation failed on ${{ github.ref_name }}"
+          webhook-url: ${{ secrets.SLACK_CI_FAILURE_WORKFLOW_WEBHOOK_URL }}
@@ -0,0 +1,43 @@
+---
+name: Semgrep
+
+on:
+  pull_request:
+    branches:
+      - "**"
+    types:
+      - opened
+      - reopened
+      - synchronize
+      - ready_for_review
+
+permissions:
+  contents: read
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  semgrep:
+    name: Security scan
+    runs-on: ubuntu-latest
+    timeout-minutes: 30
+    container:
+      image: semgrep/semgrep:1.162.0@sha256:9349edbadf90c3f3c0c3f55867625354e89680e6fa10d9034042af52fdb0e0d0
+    steps:
+      - name: Checkout
+        uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
+        with:
+          fetch-depth: 0
+          persist-credentials: false
+
+      - name: Run Semgrep
+        env:
+          BASELINE_COMMIT: ${{ github.event.pull_request.base.sha }}
+          SEMGREP_SEND_METRICS: "off"
+        run: |
+          semgrep scan \
+            --config p/default \
+            --baseline-commit "$BASELINE_COMMIT" \
+            --error
@@ -3,46 +3,55 @@ on:
  push:
    branches: ["production", "main"]

+permissions:
+  contents: read
+  security-events: write
+
 jobs:
  snyk:
    runs-on: ubuntu-latest
+    environment: snyk
    steps:
-      - uses: actions/checkout@ee0669bd1cc54295c223e0bb666b733df41de1c5 # v2
+      - uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
+        with:
+          persist-credentials: false
+      - name: Set up Snyk CLI
+        uses: snyk/actions/setup@9adf32b1121593767fc3c057af55b55db032dc04 # master
+        with:
+          snyk-version: v1.1304.0
      - name: Run Snyk to check Docker image for vulnerabilities
        # Snyk can be used to break the build when it detects vulnerabilities.
        # In this case we want to upload the issues to GitHub Code Scanning
        continue-on-error: true
-        uses: snyk/actions/docker@9adf32b1121593767fc3c057af55b55db032dc04 # master
        env:
          # In order to use the Snyk Action you will need to have a Snyk API token.
          # See https://docs.snyk.io/integrations/ci-cd-integrations/github-actions-integration#getting-your-snyk-token
          # or you can sign up for free at https://snyk.io/login
          SNYK_TOKEN: ${{ secrets.SNYK_TOKEN }}
-        with:
-          image: langfuse/langfuse
-          args: --sarif-file-output=snyk.sarif
-      - name: Fix SARIF file
+        run: |
+          snyk container test langfuse/langfuse \
+            --file=web/Dockerfile \
+            --sarif-file-output=snyk.sarif
+      - name: Normalize SARIF file
        if: always()
        run: |
          if [ -f snyk.sarif ]; then
-            # Fix undefined security severity values in SARIF file
-            sed -i 's/"security-severity": "undefined"/"security-severity": "0"/g' snyk.sarif
+            # Snyk emits invalid security-severity values. upload-sarif requires a numeric string.
+            sed -i \
+              -e 's/"security-severity": "undefined"/"security-severity": "0"/g' \
+              -e 's/"security-severity": "null"/"security-severity": "0"/g' \
+              -e 's/"security-severity": null/"security-severity": "0"/g' \
+              snyk.sarif
            echo "SARIF file fixed"
          else
            echo "No SARIF file found"
          fi
-      - name: Echo SARIF file for debugging
-        if: always()
-        run: |
-          if [ -f snyk.sarif ]; then
-            echo "=== SARIF File Contents ==="
-            cat snyk.sarif
-            echo "=== End of SARIF File ==="
-          else
-            echo "No SARIF file found to display"
-          fi
      - name: Upload result to GitHub Code Scanning
-        uses: github/codeql-action/upload-sarif@c10b8064de6f491fea524254123dbe5e09572f13 # v4
+        uses: github/codeql-action/upload-sarif@87557b9c84dde89fdd9b10e88954ac2f4248e463 # v4.36.1
        if: always()
        with:
          sarif_file: snyk.sarif
+          category: snyk-container-web
+      - name: Echo SARIF file for debugging
+        if: failure() && hashFiles('snyk.sarif') != ''
+        run: cat snyk.sarif
@@ -3,46 +3,55 @@ on:
  push:
    branches: ["production", "main"]

+permissions:
+  contents: read
+  security-events: write
+
 jobs:
  snyk:
    runs-on: ubuntu-latest
+    environment: snyk
    steps:
-      - uses: actions/checkout@ee0669bd1cc54295c223e0bb666b733df41de1c5 # v2
+      - uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
+        with:
+          persist-credentials: false
+      - name: Set up Snyk CLI
+        uses: snyk/actions/setup@9adf32b1121593767fc3c057af55b55db032dc04 # master
+        with:
+          snyk-version: v1.1304.0
      - name: Run Snyk to check Docker image for vulnerabilities
        # Snyk can be used to break the build when it detects vulnerabilities.
        # In this case we want to upload the issues to GitHub Code Scanning
        continue-on-error: true
-        uses: snyk/actions/docker@9adf32b1121593767fc3c057af55b55db032dc04 # master
        env:
          # In order to use the Snyk Action you will need to have a Snyk API token.
          # See https://docs.snyk.io/integrations/ci-cd-integrations/github-actions-integration#getting-your-snyk-token
          # or you can sign up for free at https://snyk.io/login
          SNYK_TOKEN: ${{ secrets.SNYK_TOKEN }}
-        with:
-          image: langfuse/langfuse-worker
-          args: --sarif-file-output=snyk.sarif
-      - name: Fix SARIF file
+        run: |
+          snyk container test langfuse/langfuse-worker \
+            --file=worker/Dockerfile \
+            --sarif-file-output=snyk.sarif
+      - name: Normalize SARIF file
        if: always()
        run: |
          if [ -f snyk.sarif ]; then
-            # Fix undefined security severity values in SARIF file
-            sed -i 's/"security-severity": "undefined"/"security-severity": "0"/g' snyk.sarif
+            # Snyk emits invalid security-severity values. upload-sarif requires a numeric string.
+            sed -i \
+              -e 's/"security-severity": "undefined"/"security-severity": "0"/g' \
+              -e 's/"security-severity": "null"/"security-severity": "0"/g' \
+              -e 's/"security-severity": null/"security-severity": "0"/g' \
+              snyk.sarif
            echo "SARIF file fixed"
          else
            echo "No SARIF file found"
          fi
-      - name: Echo SARIF file for debugging
-        if: always()
-        run: |
-          if [ -f snyk.sarif ]; then
-            echo "=== SARIF File Contents ==="
-            cat snyk.sarif
-            echo "=== End of SARIF File ==="
-          else
-            echo "No SARIF file found to display"
-          fi
      - name: Upload result to GitHub Code Scanning
-        uses: github/codeql-action/upload-sarif@c10b8064de6f491fea524254123dbe5e09572f13 # v4
+        uses: github/codeql-action/upload-sarif@87557b9c84dde89fdd9b10e88954ac2f4248e463 # v4.36.1
        if: always()
        with:
          sarif_file: snyk.sarif
+          category: snyk-container-worker
+      - name: Echo SARIF file for debugging
+        if: failure() && hashFiles('snyk.sarif') != ''
+        run: cat snyk.sarif
--- a/Show More
+++ b/Show More