chore: release v3.174.0

ci: tighten GitHub Actions cache policy (#13613 )
* ci: tighten workflow cache policy * ci: disable fork pr cache restores * ci: fix workflow node version references
2026-05-13 13:30:53 +02:00 · 2026-05-13 10:03:27 +00:00 · 2026-05-13 11:46:32 +02:00 · 2026-05-13 09:07:27 +00:00 · 2026-05-12 15:58:12 +00:00 · 2026-05-12 12:32:01 +00:00
996 changed files with 51371 additions and 24534 deletions
@@ -11,7 +11,7 @@ evaluating, and debugging AI applications.

 - `AGENTS.md` is a living document.
 - Keep this file concise and router-like. Push narrow or conditional workflows
-  into package-local `AGENTS.md` files or shared skills under `skills/`.
+  into package-local `AGENTS.md` files or shared skills under `.agents/skills/`.
 - Update this file in the same PR when monorepo-level architecture, workflows,
  dependency boundaries, mandatory verification commands, or release/security
  processes materially change.
@@ -23,36 +23,56 @@ evaluating, and debugging AI applications.

 ## Start Here By Task

+- Architecture principles for high-scale observability and wide event data:
+  `.agents/ARCHITECTURE_PRINCIPLES.md`
+- Langfuse org navigation, cross-repo context, or when context/skills from
+  other Langfuse codebases may be required:
+  use the `langfuse-codebase-navigator` skill.
 - Repo-wide agent setup, `.agents/**`, provider shims, or MCP/bootstrap config:
-  [`README.md`](README.md),
-  [`skills/agent-setup-maintenance/SKILL.md`](skills/agent-setup-maintenance/SKILL.md)
+  `.agents/README.md`,
+  `.agents/skills/agent-setup-maintenance/SKILL.md`
+- Langfuse Cloud cost structure, infra spend, AWS/ClickHouse cost splits, or
+  Metabase cost marts:
+  `.agents/skills/analyze-cloud-costs/SKILL.md`
+- Production telemetry research, Datadog query recipes, tenant/public API usage
+  audits, or queue consumer telemetry:
+  `.agents/skills/datadog-query-recipes/SKILL.md`
 - Backend/API work in `web/src/server/**`, `web/src/pages/api/public/**`,
  `worker/src/**`, or `packages/shared/src/**`:
-  [`skills/backend-dev-guidelines/SKILL.md`](skills/backend-dev-guidelines/SKILL.md)
+  `.agents/skills/backend-dev-guidelines/SKILL.md`
 - Model pricing work in `worker/src/constants/default-model-prices.json`,
  `packages/shared/src/server/llm/types.ts`, or related pricing files:
-  [`skills/add-model-price/SKILL.md`](skills/add-model-price/SKILL.md)
+  `.agents/skills/add-model-price/SKILL.md`
 - Code review tasks:
-  [`skills/code-review/SKILL.md`](skills/code-review/SKILL.md)
+  `.agents/skills/code-review/SKILL.md`
+- Debugging a Linear issue, GitHub issue, or incident report using Datadog
+  (APM, logs, metrics) to establish a root cause:
+  `.agents/skills/debug-issue-with-datadog/SKILL.md`
+- Measured bug or regression evidence that needs Linear deduplication, evidence
+  comments, or Triage bug issue creation:
+  `.agents/skills/linear-bug-triage/SKILL.md`
+- Proactive production regression sweeps across Datadog errors, logs, spans, and
+  API latency with Linear handoff:
+  `.agents/skills/detect-prod-regressions/SKILL.md`
 - Changelog drafting for completed feature branches:
-  [`skills/changelog-writing/SKILL.md`](skills/changelog-writing/SKILL.md)
+  `.agents/skills/changelog-writing/SKILL.md`
 - ClickHouse schema/query review:
-  [`skills/clickhouse-best-practices/SKILL.md`](skills/clickhouse-best-practices/SKILL.md)
+  `.agents/skills/clickhouse-best-practices/SKILL.md`
 - Monorepo/Turbo task graph changes:
-  [`skills/turborepo/SKILL.md`](skills/turborepo/SKILL.md)
+  `.agents/skills/turborepo/SKILL.md`
 - pnpm dependency upgrades, package-version bumps, or `minimumReleaseAgeExclude`
  decisions in `pnpm-workspace.yaml`:
-  [`skills/pnpm-upgrade-package/SKILL.md`](skills/pnpm-upgrade-package/SKILL.md)
+  `.agents/skills/pnpm-upgrade-package/SKILL.md`
 - User-visible frontend changes, Playwright review, or browser signoff:
-  [`skills/frontend-browser-review/SKILL.md`](skills/frontend-browser-review/SKILL.md)
+  `.agents/skills/frontend-browser-review/SKILL.md`
 - Web UI and frontend entry points:
-  `../web/AGENTS.md`
+  `web/AGENTS.md`
 - Worker queues and processors:
-  `../worker/AGENTS.md`
+  `worker/AGENTS.md`
 - Shared contracts, exports, schema, and migrations:
-  `../packages/shared/AGENTS.md`
+  `packages/shared/AGENTS.md`
 - EE-only work:
-  `../ee/AGENTS.md`
+  `ee/AGENTS.md`

 Read the minimal set required for the task. More-specific package guides and
 shared skills take precedence over this root file for their scoped areas.
@@ -84,7 +104,7 @@ langfuse/
    `packages/shared/clickhouse/migrations/{clustered,unclustered}/*.sql`
 - Architecture handbook:
  [langfuse.com/handbook/product-engineering/architecture](https://langfuse.com/handbook/product-engineering/architecture)
-  with source markdown in
+  with source markdown in the sibling docs checkout at
  `../langfuse-docs/content/handbook/product-engineering/architecture.mdx`

 ## Core Commands
@@ -98,8 +118,8 @@ langfuse/
 - Build check: `pnpm run build:check`
 - Full build: `pnpm run build`
 - Full reset/bootstrap (destructive): `pnpm run dx`
- Codex environment bootstrap: `bash scripts/codex/setup.sh`
- Codex environment maintenance: `bash scripts/codex/maintenance.sh`
+- Environment/worktree bootstrap: `bash scripts/codex/setup.sh`
+- Environment/worktree maintenance: `bash scripts/codex/maintenance.sh`
 - Install Playwright Chromium for agent browser review: `pnpm run playwright:install`

 Minimum verification matrix:
@@ -124,13 +144,15 @@ Minimum verification matrix:
  - `*/dist/*`
  - `packages/shared/prisma/generated/*`
 - Public API contract changes must update Fern sources in `fern/apis/**` and
-  regenerated outputs; never hand-edit `generated/**`.
+  regenerated outputs. Never hand-edit `generated/**`.
+- Before adding constants, value lists, or display mappings, search for an
+  existing owner and reuse or extend that source of truth.
 - Keep tests independent and parallel-safe.
 - For bug fixes, write the failing test first, confirm it fails, then fix the
  bug.
 - For user-visible frontend changes in `web/**`, review the affected flow in a
  real browser with the Playwright MCP server before signoff. Use
-  `skills/frontend-browser-review/SKILL.md` and `../web/AGENTS.md` for the
+  `.agents/skills/frontend-browser-review/SKILL.md` and `web/AGENTS.md` for the
  browser-review loop.
 - Never commit secrets or credentials. Keep `.env*.example` files in sync with
  required env vars.
@@ -140,19 +162,19 @@ Minimum verification matrix:
 - `.agents/AGENTS.md` is the canonical root guide.
 - Root `AGENTS.md` is a symlink to `.agents/AGENTS.md`.
 - Root `CLAUDE.md` is a compatibility symlink to `AGENTS.md`.
- Shared agent/tool config lives in `config.json` and shared skills live in
-  `skills/`.
+- Shared agent/tool config lives in `.agents/config.json` and shared skills
+  live in `.agents/skills/`.
 - Project-scoped provider discovery files are generated local artifacts. Edit
  the canonical files under `.agents/` instead of editing generated tool
  directories by hand.
- If you change `.agents/config.json`, `skills/**`, or the shim-generation
-  workflow, run:
+- If you change `.agents/config.json`, `.agents/skills/**`, or the
+  shim-generation workflow, run:
  - `pnpm run agents:sync`
  - `pnpm run agents:check`
 - Do not commit generated provider config or shim outputs under `.claude/`,
  `.cursor/`, `.codex/`, `.vscode/`, or `.mcp.json`.
 - Durable cross-tool guidance belongs in root/package `AGENTS.md` files or
-  `skills/**`, not only in tool-specific config directories.
+  `.agents/skills/**`, not only in tool-specific config directories.

 ## Commit, PR, and Release Rules

@@ -0,0 +1,49 @@
+# Underlying Architecture Principles
+
+Langfuse architecture should optimize for high-scale, exploratory observability
+on wide, structured event data. These principles are grounded in current
+production scale and the reference material below.
+
+## Reference Posts
+
+- [Simplifying Langfuse for Scale](https://langfuse.com/blog/2026-03-10-simplify-langfuse-for-scale)
+- [Charity Majors on Observability 2.0](https://charity.wtf/tag/observability-2-0/)
+- [All you need is Wide Events, not "Metrics, Logs and Traces"](https://isburmistrov.substack.com/p/all-you-need-is-wide-events-not-metrics)
+
+## Principles
+
+- Model observations as the primary analytical unit. A trace is a correlation
+  handle that links related observations, not the only useful entry point.
+- Prefer wide, richly attributed events over fragmented metrics, logs, and trace
+  records that require later reconstruction.
+- Preserve high-cardinality context so users can slice, group, filter, and debug
+  unknown unknowns without predefining every future question.
+- Favor immutable or append-oriented event records for high-volume telemetry.
+  Updates that force read-time deduplication create hidden query costs at scale.
+- Denormalize carefully when it removes hot-path joins and makes common filters
+  into direct column predicates.
+- Design storage and query paths around columnar access patterns: narrow field
+  selection, time-bounded scans, useful ordering keys, and data pruning.
+- Keep list, dashboard, and aggregate views on compact query-optimized
+  representations. Fetch large raw payloads only for focused detail views.
+- Make API contracts scale-aware: require time windows where needed, expose field
+  selection, use token pagination, and avoid defaults that can scan all history.
+- Treat cost and operational simplicity as architectural constraints. Extra
+  databases, queues, materialized views, and migrations must earn their long-term
+  operational burden.
+- Preserve real-time or near-real-time debugging workflows. Batch processing can
+  help, but it should not make fresh production behavior invisible.
+
+## Practical Defaults For Agents
+
+- Before adding a metric, ask whether the same question is better answered from
+  wide event data.
+- Before adding a join, ask whether the attribute should be propagated or
+  denormalized onto the observation path.
+- Before reading large fields, ask whether the view needs them or can defer them
+  until a single-record fetch.
+- Before adding an update-heavy design, ask whether immutable events plus
+  derived representations would be simpler at production scale.
+- Before documenting public behavior, separate stable public contracts from
+  private production topology, account details, secret names, and incident
+  runbooks.
@@ -10,6 +10,8 @@ or `.vscode/`.
 ## Layout

 - `AGENTS.md`: canonical shared root instructions
+- `ARCHITECTURE_PRINCIPLES.md`: architecture principles for high-scale
+  observability
 - `config.json`: shared bootstrap and MCP configuration used to generate
  tool-specific shims
 - `skills/`: shared, tool-neutral implementation guidance for recurring
@@ -38,15 +40,42 @@ Current shape:
    "playwright": {
      "transport": "stdio",
      "command": "npx",
-      "args": ["-y", "@playwright/mcp@latest"]
+      "args": [
+        "-y",
+        "@playwright/mcp@latest",
+        "--isolated",
+        "--save-trace",
+        "--output-dir",
+        ".playwright-mcp",
+        "--test-id-attribute",
+        "data-testid"
+      ]
    },
-    "datadog": {
+    "langfuse-docs": {
      "transport": "http",
-      "url": "https://mcp.datadoghq.com/api/unstable/mcp-server/mcp"
+      "url": "https://langfuse.com/api/mcp"
+    },
+    "linear": {
+      "transport": "http",
+      "url": "https://mcp.linear.app/mcp"
    }
  },
  "claude": {
-    "settings": {}
+    "settings": {
+      "permissions": {
+        "allow": [
+          "Bash(find:*)",
+          "Bash(rg:*)",
+          "Bash(grep:*)",
+          "Bash(ls:*)",
+          "Bash(cat:*)",
+          "Bash(head:*)",
+          "Bash(tail:*)"
+        ],
+        "deny": []
+      },
+      "enableAllProjectMcpServers": true
+    }
  },
  "codex": {
    "environment": {
@@ -33,6 +33,28 @@ Use for:

 Open: [agent-setup-maintenance/SKILL.md](agent-setup-maintenance/SKILL.md)

+### analyze-cloud-costs
+
+Use for:
+- Langfuse Cloud infrastructure cost structure
+- AWS versus ClickHouse cost splits and cost drivers
+- Metabase infra cost dashboard and cost marts
+- daily cost per tracing event and cost regression analysis
+
+Open: [analyze-cloud-costs/SKILL.md](analyze-cloud-costs/SKILL.md)
+
+### datadog-query-recipes
+
+Use for:
+- production telemetry research across `prod-us`, `prod-eu`, `prod-hipaa`,
+  and `prod-jp`
+- Datadog query recipes for spans, logs, metrics, tenants, public API usage,
+  and queue consumers
+- ad hoc measured questions where the task is not yet an incident root-cause
+  analysis
+
+Open: [datadog-query-recipes/SKILL.md](datadog-query-recipes/SKILL.md)
+
 ### frontend-browser-review

 Use for:
@@ -72,6 +94,24 @@ Use for:

 Open: [code-review/SKILL.md](code-review/SKILL.md)

+### detect-prod-regressions
+
+Use for:
+- proactive Datadog sweeps across `prod-us`, `prod-eu`, `prod-hipaa`, and `prod-jp`
+- comparing recent production errors, logs, spans, and API latency to baselines
+- handing measured regressions to `linear-bug-triage` for Linear issues or comments
+
+Open: [detect-prod-regressions/SKILL.md](detect-prod-regressions/SKILL.md)
+
+### linear-bug-triage
+
+Use for:
+- deduplicating measured bug or regression evidence against Linear
+- creating new Linear bug issues in `Triage` with `bug` and related labels
+- adding concise evidence comments to related existing Linear issues
+
+Open: [linear-bug-triage/SKILL.md](linear-bug-triage/SKILL.md)
+
 ### changelog-writing

 Use for:
@@ -81,6 +121,24 @@ Use for:

 Open: [changelog-writing/SKILL.md](changelog-writing/SKILL.md)

+### clickhouse-best-practices
+
+Use for:
+- ClickHouse schema, query, or configuration review
+- ClickHouse migrations under `packages/shared/clickhouse/**`
+- applying the repo-specific ClickHouse rules layered on top of upstream best practices
+
+Open: [clickhouse-best-practices/SKILL.md](clickhouse-best-practices/SKILL.md)
+
+### debug-issue-with-datadog
+
+Use for:
+- root-causing Linear, GitHub, or incident reports with Datadog evidence
+- production debugging across APM spans, logs, metrics, and monitors
+- mapping observed error clusters back to Langfuse code paths
+
+Open: [debug-issue-with-datadog/SKILL.md](debug-issue-with-datadog/SKILL.md)
+
 ### pnpm-upgrade-package

 Use for:
@@ -92,6 +150,15 @@ Use for:

 Open: [pnpm-upgrade-package/SKILL.md](pnpm-upgrade-package/SKILL.md)

+### turborepo
+
+Use for:
+- `turbo.json` task graph, caching, filtering, or affected-run changes
+- root/package script organization for Turborepo workflows
+- monorepo package boundaries and shared-code layout decisions
+
+Open: [turborepo/SKILL.md](turborepo/SKILL.md)
+
 ## Adding a New Shared Skill

 1. Codex may create or refine shared skills under `.agents/skills/` when a
@@ -0,0 +1,62 @@
+---
+name: analyze-cloud-costs
+description: |
+  Analyze Langfuse Cloud infrastructure cost structure using Metabase cost
+  marts. Use when asked about cloud spend, AWS versus ClickHouse cost splits,
+  cost drivers by provider/service/usage type/account, daily cost per tracing
+  event, infra cost dashboards, or cost regressions visible in Metabase.
+---
+
+# Analyze Cloud Costs
+
+## Overview
+
+Use this skill for evidence-backed Langfuse Cloud cost analysis. The primary
+source is the Metabase infra cost dashboard and its production cost marts; the
+deliverable should name the time window, query grain, top drivers, and caveats.
+
+## Workflow
+
+1. Clarify the question and choose the grain:
+   - Headline daily totals: total, AWS, ClickHouse, tracing events, and cost per
+     100k events.
+   - Cost structure: provider, service, usage type, operation, account, and day.
+   - Driver or regression analysis: compare a recent complete-day window against
+     a prior baseline.
+2. Load [`references/cost-marts.md`](references/cost-marts.md) for table IDs,
+   field IDs, query examples, and caveats.
+3. Use the Metabase MCP. If the Metabase tools are not visible, discover them
+   with tool search before falling back to manual interpretation.
+4. Prefer complete UTC days. Avoid treating current-day AWS cost as final
+   because AWS CUR rows can arrive late.
+5. Start broad, then drill down:
+   - Provider split.
+   - Service split within the dominant provider.
+   - Usage type, operation, and account split for the top services.
+   - Daily trend when explaining change over time.
+6. Report only what the queried data supports. If a requested slice is absent,
+   say that no rows were found for that slice instead of inventing a driver.
+
+## Query Rules
+
+- Use `mcp__metabase__.query` for quick reads. Use
+  `construct_query` plus `execute_query` when you need to inspect or reuse the
+  opaque query.
+- Pass `filters`, `aggregations`, `group_by`, and `fields` as JSON arrays. Some
+  tool schemas may display these as strings; if that happens, serialize the same
+  arrays without changing their shape.
+- Keep limits explicit and small enough for analysis. Use pagination only when
+  the continuation token is needed.
+- Include the Metabase dashboard link or query result context in the final
+  answer when useful.
+
+## Output Expectations
+
+Summarize:
+
+- Time window and whether it uses complete UTC days.
+- Total cost and provider split when relevant.
+- Top cost drivers by service, usage type, operation, or account.
+- Trend or baseline comparison when the user asks "why did this change?"
+- Caveats, especially incomplete current-day AWS data and ClickHouse credit
+  labeling in the unified mart.
@@ -0,0 +1,4 @@
+interface:
+  display_name: "Analyze Cloud Costs"
+  short_description: "Analyze Langfuse Cloud cost structure"
+  default_prompt: "Use $analyze-cloud-costs to explain recent Langfuse Cloud cost drivers from Metabase."
@@ -0,0 +1,137 @@
+# Langfuse Cloud Cost Marts
+
+Use this reference when querying or explaining Langfuse Cloud cost structure.
+
+Dashboard:
+
+- https://langfuse.metabaseapp.com/dashboard/22-infra-cost?account=&date=past90days&tab=20-tab-1
+
+## Primary Tables
+
+| Purpose | Table | ID |
+| --- | --- | --- |
+| Unified AWS and ClickHouse cost rows by provider, service, usage type, account, and day | `langfuse_prod.mart_daily_cost_chart` | `739` |
+| Daily headline totals plus tracing event counts and cost per 100k events | `langfuse_prod.mart_daily_cost_with_events` | `784` |
+| Detailed AWS CUR summary by product, operation, account, and usage type | `langfuse_prod.mart_aws_cost_daily_by_service` | `610` |
+| Detailed ClickHouse costs by entity and metric | `langfuse_prod.mart_clickhouse_daily_cost` | `689` |
+
+Prefer table `739` for structural breakdowns. Prefer table `784` for daily
+headline totals.
+
+## Field IDs
+
+### `mart_daily_cost_chart` (`739`)
+
+| Field ID | Field |
+| --- | --- |
+| `t739-0` | `usage_date` |
+| `t739-1` | `service_provider` |
+| `t739-2` | `service_name` |
+| `t739-3` | `operation` |
+| `t739-4` | `usage_type` |
+| `t739-5` | `account_name` |
+| `t739-6` | `cost_usd` |
+
+### `mart_daily_cost_with_events` (`784`)
+
+| Field ID | Field |
+| --- | --- |
+| `t784-0` | `usage_date` |
+| `t784-1` | `total_cost_usd` |
+| `t784-2` | `clickhouse_cost_usd` |
+| `t784-3` | `aws_cost_usd` |
+| `t784-5` | `s3_api_operations_cost_usd` |
+| `t784-6` | `total_tracing_events` |
+| `t784-7` | `total_cost_per_100k_events` |
+
+## Metabase MCP Patterns
+
+The Metabase MCP supports `query` for direct reads and
+`construct_query` plus `execute_query` for reusable opaque queries. In practice,
+pass `filters`, `aggregations`, `group_by`, and `fields` as JSON arrays:
+
+```json
+{
+  "table_id": 739,
+  "filters": [
+    {
+      "field_id": "t739-0",
+      "operation": "greater-than-or-equal",
+      "value": "2026-05-09"
+    }
+  ],
+  "aggregations": [
+    {
+      "function": "sum",
+      "field_id": "t739-6"
+    }
+  ],
+  "group_by": [
+    { "field_id": "t739-1" },
+    { "field_id": "t739-2" }
+  ],
+  "limit": "200"
+}
+```
+
+If a tool surface insists on strings for those parameters, serialize the same
+arrays as JSON strings.
+
+## Common Breakdowns
+
+Provider split:
+
+- Table `739`
+- Aggregate `sum(t739-6)`
+- Group by `t739-1`
+
+Service split:
+
+- Table `739`
+- Aggregate `sum(t739-6)`
+- Group by `t739-1`, `t739-2`
+
+Usage type split:
+
+- Table `739`
+- Aggregate `sum(t739-6)`
+- Group by `t739-1`, `t739-4`
+
+Environment/account split:
+
+- Table `739`
+- Aggregate `sum(t739-6)`
+- Group by `t739-1`, `t739-5`
+
+Daily headline totals:
+
+- Table `784`
+- Filter `t784-0` by date.
+- Read `total_cost_usd`, `clickhouse_cost_usd`, `aws_cost_usd`,
+  `total_tracing_events`, and `total_cost_per_100k_events`.
+
+Daily trend by provider:
+
+- Table `739`
+- Aggregate `sum(t739-6)`
+- Group by `t739-0`, `t739-1`
+
+Drilldown sequence for a cost spike:
+
+1. Compare total daily cost in table `784`.
+2. Split the same days by provider in table `739`.
+3. Split the dominant provider by service.
+4. Split the dominant service by usage type, operation, and account.
+
+## Caveats
+
+- Current-day AWS cost can be incomplete because AWS CUR data may not have
+  landed yet.
+- For stable recent analysis, prefer the last complete UTC days rather than
+  including today.
+- ClickHouse cost rows are labeled `cost_usd` in the unified mart, but the
+  source metric is ClickHouse credits. Mention this when precision or billing
+  interpretation matters.
+- Field IDs can change if Metabase models are rebuilt. If a query fails, search
+  Metabase for the table name and inspect the returned metadata before
+  changing the analysis.
@@ -2,7 +2,12 @@

 ## Purpose

-Establish consistency and best practices across Langfuse's backend packages (web, worker, packages/shared) using Next.js 14, tRPC, BullMQ, and TypeScript patterns.
+Establish consistency and best practices across Langfuse's backend packages
+(`web`, `worker`, `packages/shared`) using Next.js, tRPC, BullMQ, and TypeScript
+patterns. Check package manifests such as `web/package.json` for current
+framework versions before version-sensitive work.
+Keep this file as an entrypoint; open reference files only when the task needs
+their details.

 ## When to Use This Skill

@@ -64,7 +69,7 @@ Use this guide when working on:
 ### Layered Architecture

 ```
-# Web Package (Next.js 14)
+# Web Package (Next.js)

 ┌─ tRPC API ──────────────────┐   ┌── Public REST API ──────────┐
 │                             │   │                             │
@@ -105,150 +110,6 @@ for complete details.

 ---

-## Directory Structure
-
-### Web Package (`/web/`)
-
-```
-web/src/
-├── features/                # Feature-organized code
-│   ├── [feature-name]/
-│   │   ├── server/          # Backend logic
-│   │   │   ├── *Router.ts   # tRPC router
-│   │   │   └── service.ts   # Business logic
-│   │   ├── components/      # React components
-│   │   └── types/           # Feature types
-├── server/
-│   ├── api/
-│   │   ├── routers/         # tRPC routers
-│   │   ├── trpc.ts          # tRPC setup & middleware
-│   │   └── root.ts          # Main router
-│   ├── auth.ts              # NextAuth.js config
-│   └── db.ts                # Database client
-├── pages/
-│   ├── api/
-│   │   ├── public/          # Public REST APIs
-│   │   └── trpc/            # tRPC endpoint
-│   └── [routes].tsx         # Next.js pages
-├── __tests__/               # Jest tests
-│   └── async/               # Integration tests
-├── instrumentation.ts       # OpenTelemetry (FIRST IMPORT)
-└── env.mjs                  # Environment config
-```
-
-### Worker Package (`/worker/`)
-
-```
-worker/src/
-├── queues/                  # BullMQ processors
-│   ├── evalQueue.ts
-│   ├── ingestionQueue.ts
-│   └── workerManager.ts
-├── features/                # Business logic
-│   └── [feature]/
-│       └── service.ts
-├── instrumentation.ts       # OpenTelemetry (FIRST IMPORT)
-├── app.ts                   # Express setup + queue registration
-├── env.ts                   # Environment config
-└── index.ts                 # Server start
-```
-
-### Shared Package (`/packages/shared/`)
-
-```
-shared/src/
-├── server/                  # Server utilities
-│   ├── auth/                # Authentication helpers
-│   ├── clickhouse/          # ClickHouse client & schema
-│   ├── instrumentation/     # OpenTelemetry helpers
-│   ├── llm/                 # LLM integration utilities
-│   ├── redis/               # Redis queues & cache
-│   ├── repositories/        # Data repositories
-│   ├── services/            # Shared services
-│   ├── utils/               # Server utilities
-│   ├── logger.ts
-│   └── queues.ts
-├── encryption/              # Encryption utilities
-├── features/                # Feature-specific code
-├── tableDefinitions/        # Table schemas
-├── utils/                   # Shared utilities
-├── constants.ts
-├── db.ts                    # Prisma client
-├── env.ts                   # Environment config
-└── index.ts                 # Main exports
-```
-
-**Import Paths (package.json exports):**
-
-The shared package exposes specific import paths for different use cases:
-
-| Import Path                                | Maps To                           | Use For                                                         |
-| ------------------------------------------ | --------------------------------- | --------------------------------------------------------------- |
-| `@langfuse/shared`                         | `dist/src/index.js`               | General types, schemas, utilities, constants                    |
-| `@langfuse/shared/src/db`                  | `dist/src/db.js`                  | Prisma client and database types                                |
-| `@langfuse/shared/src/server`              | `dist/src/server/index.js`        | Server-side utilities (queues, auth, services, instrumentation) |
-| `@langfuse/shared/src/server/auth/apiKeys` | `dist/src/server/auth/apiKeys.js` | API key management utilities                                    |
-| `@langfuse/shared/encryption`              | `dist/src/encryption/index.js`    | Encryption and signature utilities                              |
-
-**Usage Examples:**
-
-```typescript
-// General imports - types, schemas, constants, interfaces
-import {
-  CloudConfigSchema,
-  StringNoHTML,
-  AnnotationQueueObjectType,
-  type APIScoreV2,
-  type ColumnDefinition,
-  Role,
-} from "@langfuse/shared";
-
-// Database - Prisma client and types
-import { prisma, Prisma, JobExecutionStatus } from "@langfuse/shared/src/db";
-import { type DB as Database } from "@langfuse/shared";
-
-// Server utilities - queues, services, auth, instrumentation
-import {
-  logger,
-  instrumentAsync,
-  traceException,
-  redis,
-  getTracesTable,
-  StorageService,
-  sendMembershipInvitationEmail,
-  invalidateApiKeysForProject,
-  recordIncrement,
-  recordHistogram,
-} from "@langfuse/shared/src/server";
-
-// API key management (specific path)
-import { createAndAddApiKeysToDb } from "@langfuse/shared/src/server/auth/apiKeys";
-
-// Encryption utilities
-import { encrypt, decrypt, sign, verify } from "@langfuse/shared/encryption";
-```
-
-**What Goes Where:**
-
-The shared package provides types, utilities, and server code used by both web and worker packages. It has **5 export paths** that control frontend vs backend access:
-
-| Import Path                                | Usage                 | What's Included                                                                    |
-| ------------------------------------------ | --------------------- | ---------------------------------------------------------------------------------- |
-| `@langfuse/shared`                         | ✅ Frontend + Backend | Prisma types, Zod schemas, constants, table definitions, domain models, utilities  |
-| `@langfuse/shared/src/db`                  | 🔒 Backend only       | Prisma client instance                                                             |
-| `@langfuse/shared/src/server`              | 🔒 Backend only       | Services, repositories, queues, auth, ClickHouse, LLM integration, instrumentation |
-| `@langfuse/shared/src/server/auth/apiKeys` | 🔒 Backend only       | API key management (separated to avoid circular deps)                              |
-| `@langfuse/shared/encryption`              | 🔒 Backend only       | Database field encryption/decryption                                               |
-
-**Naming Conventions:**
-
- tRPC Routers: `camelCaseRouter.ts` - `datasetRouter.ts`
- Services: `service.ts` in feature directory
- Queue Processors: `camelCaseQueue.ts` - `evalQueue.ts`
- Public APIs: `kebab-case.ts` - `dataset-items.ts`
-
---
-
 ## Core Principles

 ### 1. tRPC Procedures Delegate to Services
@@ -355,51 +216,10 @@ const result = await instrumentAsync(

 ### 7. Comprehensive Testing Required

-Write tests for all new features and bug fixes. See [testing-guide.md](references/testing-guide.md) for detailed examples.
-
-**Test Types:**
-
-| Type        | Framework | Location                                | Purpose                      |
-| ----------- | --------- | --------------------------------------- | ---------------------------- |
-| Integration | Jest      | `web/src/__tests__/async/`              | Full API endpoint testing    |
-| tRPC        | Jest      | `web/src/__tests__/async/`              | tRPC procedures with auth    |
-| Service     | Jest      | `web/src/__tests__/async/repositories/` | Repository/service functions |
-| Worker      | Vitest    | `worker/src/__tests__/`                 | Queue processors & streams   |
-
-**Quick Examples:**
-
-```typescript
-// Integration Test (Public API)
-const res = await makeZodVerifiedAPICall(
-  PostDatasetsV1Response, "POST", "/api/public/datasets",
-  { name: "test-dataset" }, auth
-);
-expect(res.status).toBe(200);
-
-// tRPC Test
-const { caller } = await prepare(); // Creates session + caller
-const response = await caller.automations.getAutomations({ projectId });
-expect(response).toHaveLength(1);
-
-// Service Test
-const result = await getObservationsWithModelDataFromEventsTable({
-  projectId, filter: [...], limit: 1000, offset: 0
-});
-expect(result.length).toBeGreaterThan(0);
-
-// Worker Test (vitest)
-const stream = await getObservationStream({ projectId, filter: [] });
-const rows = [];
-for await (const chunk of stream) rows.push(chunk);
-expect(rows).toHaveLength(2);
-```
-
-**Key Principles:**
-
- Use unique IDs (`randomUUID()`) to avoid test interference
- Clean up test data or use unique project IDs
- Tests must be independent and runnable in any order
- Prefer scoped cleanup or unique project IDs over global reset helpers
+Add targeted tests for new backend behavior and bug fixes. Keep tests
+independent and parallel-safe. See
+[testing-guide.md](references/testing-guide.md) for tRPC, public API, service,
+repository, and worker examples.

 ### 8. Always Filter by projectId for Tenant Isolation

@@ -448,72 +268,26 @@ Trace:

 ---

-## Common Imports
+## Live Examples

-```typescript
-// tRPC (Web)
-import { z } from "zod/v4";
-import {
-  createTRPCRouter,
-  protectedProjectProcedure,
-} from "@/src/server/api/trpc";
-import { TRPCError } from "@trpc/server";
-
-// Database
-import { prisma } from "@langfuse/shared/src/db";
-import type { Prisma } from "@prisma/client";
-
-// ClickHouse
-import {
-  queryClickhouse,
-  queryClickhouseStream,
-  upsertClickhouse,
-} from "@langfuse/shared/src/server";
-
-// Observability - OpenTelemetry + DataDog (NOT Sentry for backend)
-import {
-  logger,          // Winston logger with OTEL/DataDog trace context
-  traceException,  // Record exceptions to OpenTelemetry spans
-  instrumentAsync, // Create instrumented spans for operations
-} from "@langfuse/shared/src/server";
-
-// Config
-import { env } from "@/src/env.mjs"; // web
-// or
-import { env } from "./env"; // worker
-
-// Public API (Web)
-import { withMiddlewares } from "@/src/features/public-api/server/withMiddlewares";
-import { createAuthedProjectAPIRoute } from "@/src/features/public-api/server/createAuthedProjectAPIRoute";
-
-// Queue Processing (Worker)
-import { Job } from "bullmq";
-import { QueueName, TQueueJobTypes } from "@langfuse/shared/src/server";
-```
+Reference existing Langfuse features for implementation patterns:
+- tRPC router with project auth and Zod input:
+  `web/src/features/events/server/eventsRouter.ts`
+- Public API route with middleware and typed request/response schemas:
+  `web/src/pages/api/public/datasets/index.ts`
+- Worker queue processor with typed jobs, logging, and retry behavior:
+  `worker/src/queues/evalQueue.ts`
+- Tenant filters for Prisma and ClickHouse:
+  `references/database-patterns.md`

 ---

-## Quick Reference
+## Naming Conventions

-### HTTP Status Codes
-
-| Code | Use Case     |
-| ---- | ------------ |
-| 200  | Success      |
-| 201  | Created      |
-| 400  | Bad Request  |
-| 401  | Unauthorized |
-| 403  | Forbidden    |
-| 404  | Not Found    |
-| 500  | Server Error |
-
-### Example Features to Reference
-
-Reference existing Langfuse features for implementation patterns:
- **Datasets** (`web/src/features/datasets/`) - Complete feature with tRPC router, public API, and service
- **Prompts** (`web/src/features/prompts/`) - Feature with versioning and templates
- **Evaluations** (`web/src/features/evals/`) - Complex feature with worker integration
- **Public API** (`web/src/features/public-api/`) - Middleware and route patterns
+- tRPC routers: `camelCaseRouter.ts`, e.g. `datasetRouter.ts`
+- Services: `service.ts` in the feature server directory
+- Queue processors: `camelCaseQueue.ts`, e.g. `evalQueue.ts`
+- Public API routes: kebab-case filenames, e.g. `dataset-items.ts`

 ---

@@ -530,6 +304,9 @@ Reference existing Langfuse features for implementation patterns:

 ## Navigation Guide

+Keep detailed backend guidance in these focused reference files and open only
+the one that matches the task.
+
 | Need to...                | Read this                                                    |
 | ------------------------- | ------------------------------------------------------------ |
 | Understand architecture   | [architecture-overview.md](references/architecture-overview.md)         |
@@ -541,37 +318,3 @@ Reference existing Langfuse features for implementation patterns:
 | Write tests               | [testing-guide.md](references/testing-guide.md)                         |

 ---
-
-## Reference Files
-
-### [architecture-overview.md](references/architecture-overview.md)
-
-Three-layer architecture (tRPC/Public API → Services → Data Access), request lifecycle for tRPC/Public API/Worker, Next.js 14 directory structure, dual database system (PostgreSQL + ClickHouse), separation of concerns, repository pattern for complex queries
-
-### [routing-and-controllers.md](references/routing-and-controllers.md)
-
-Next.js file-based routing, tRPC router patterns, Public REST API routes, layered architecture (Entry Points → Services → Repositories → Database), service layer organization, anti-patterns to avoid
-
-### [services-and-repositories.md](references/services-and-repositories.md)
-
-Service layer overview, dependency injection patterns, singleton patterns, repository pattern for data access, service design principles, caching strategies, testing services
-
-### [middleware-guide.md](references/middleware-guide.md)
-
-tRPC middleware (withErrorHandling, withOtelInstrumentation, enforceUserIsAuthed), seven tRPC procedure types (publicProcedure, authenticatedProcedure, protectedProjectProcedure, etc.), Public API middleware (withMiddlewares, createAuthedProjectAPIRoute), authentication patterns (NextAuth for tRPC, Basic Auth for Public API)
-
-### [database-patterns.md](references/database-patterns.md)
-
-Dual database architecture (PostgreSQL via Prisma + ClickHouse via direct client), PostgreSQL CRUD operations, ClickHouse query patterns (queryClickhouse, queryClickhouseStream, upsertClickhouse), repository pattern for complex queries, tenant isolation with projectId filtering, when to use which database
-
-### [configuration.md](references/configuration.md)
-
-Environment variable validation with Zod, package-specific configs (web/env.mjs with t3-oss/env-nextjs, worker/env.ts, shared/env.ts), NEXT_PUBLIC_LANGFUSE_CLOUD_REGION usage, LANGFUSE_EE_LICENSE_KEY for enterprise features, best practices for env management
-
-### [testing-guide.md](references/testing-guide.md)
-
-Integration tests (Public API with makeZodVerifiedAPICall), tRPC tests (createInnerTRPCContext, appRouter.createCaller), service-level tests (repository/service functions), worker tests (vitest with streams), test isolation principles, running tests (Jest for web, vitest for worker)
-
-**Skill Status**: COMPLETE ✅
-**Line Count**: ~540 lines
-**Progressive Disclosure**: 7 reference files ✅
@@ -1,6 +1,6 @@
 ---
 name: backend-dev-guidelines
-description: Shared backend guide for Langfuse's Next.js 14, tRPC, BullMQ, and TypeScript monorepo. Use when creating or reviewing tRPC routers, public REST endpoints, BullMQ queue processors, backend services, middleware, Prisma or ClickHouse data access, OpenTelemetry instrumentation, Zod validation, env configuration, or backend tests across web, worker, or packages/shared.
+description: Shared backend guide for Langfuse's Next.js, tRPC, BullMQ, and TypeScript monorepo. Use when creating or reviewing tRPC routers, public REST endpoints, BullMQ queue processors, backend services, middleware, Prisma or ClickHouse data access, OpenTelemetry instrumentation, Zod validation, env configuration, or backend tests across web, worker, or packages/shared.
 ---

 # Backend Development Guidelines
@@ -1,6 +1,6 @@
 # Architecture Overview - Langfuse Backend

-Complete guide to the layered architecture pattern used in Langfuse's Next.js 14/tRPC/Express monorepo.
+Complete guide to the layered architecture pattern used in Langfuse's Next.js/tRPC/Express monorepo. Check package manifests such as `web/package.json` for current framework versions before version-sensitive work.

 ## Table of Contents

@@ -20,7 +20,7 @@ Langfuse uses a **three-layer architecture** with two primary entry points (tRPC
 ### The Three Layers

 ```
-# Web Package (Next.js 14)
+# Web Package (Next.js)

 ┌─ tRPC API ──────────────────┐   ┌── Public REST API ──────────┐
 │                             │   │                             │
@@ -163,7 +163,7 @@ Two types of entry points:
 ```typescript
 1. HTTP POST /api/public/datasets
   ↓
-2. Next.js API route handler (pages/api/public/datasets.ts)
+2. Next.js API route handler (pages/api/public/datasets/index.ts)
   ↓
 3. withMiddlewares wrapper executes:
   - Basic auth verification
@@ -571,7 +571,7 @@ export async function createDataset(data: {
 **Public API (Alternative Entry Point):**

 ```typescript
-// web/src/pages/api/public/datasets.ts
+// web/src/pages/api/public/datasets/index.ts
 import { withMiddlewares } from "@/src/features/public-api/server/withMiddlewares";
 import { createAuthedProjectAPIRoute } from "@/src/features/public-api/server/createAuthedProjectAPIRoute";
 import { createDataset } from "@/src/features/datasets/server/service";
@@ -412,190 +412,19 @@ Repository: "Here's the Prisma query that does that"
 - ❌ Know about HTTP
 - ❌ Make decisions (that's service layer)

-### Repository Template
+### Langfuse Repository Examples

-```typescript
-// repositories/UserRepository.ts
-import { PrismaService } from "@project-lifecycle-portal/database";
-import type { User, Prisma } from "@project-lifecycle-portal/database";
+Use these current Langfuse files as repository templates:

-export class UserRepository {
-  /**
-   * Find user by ID with optimized query
-   */
-  async findById(userId: string): Promise<User | null> {
-    try {
-      return await PrismaService.main.user.findUnique({
-        where: { userID: userId },
-        select: {
-          userID: true,
-          email: true,
-          name: true,
-          isActive: true,
-          roles: true,
-          createdAt: true,
-          updatedAt: true,
-        },
-      });
-    } catch (error) {
-      console.error("[UserRepository] Error finding user by ID:", error);
-      throw new Error(`Failed to find user: ${userId}`);
-    }
-  }
+- PostgreSQL repository with project-scoped filters:
+  `packages/shared/src/server/repositories/comments.ts`
+- ClickHouse repository with project-scoped filters and query helpers:
+  `packages/shared/src/server/repositories/traces.ts`
+- Repository tests:
+  `web/src/__tests__/server/repositories/event-repository.servertest.ts`

-  /**
-   * Find all active users
-   */
-  async findActive(options?: {
-    orderBy?: Prisma.UserOrderByWithRelationInput;
-  }): Promise<User[]> {
-    try {
-      return await PrismaService.main.user.findMany({
-        where: { isActive: true },
-        orderBy: options?.orderBy || { name: "asc" },
-        select: {
-          userID: true,
-          email: true,
-          name: true,
-          roles: true,
-        },
-      });
-    } catch (error) {
-      console.error("[UserRepository] Error finding active users:", error);
-      throw new Error("Failed to find active users");
-    }
-  }
-
-  /**
-   * Find user by email
-   */
-  async findByEmail(email: string): Promise<User | null> {
-    try {
-      return await PrismaService.main.user.findUnique({
-        where: { email },
-      });
-    } catch (error) {
-      console.error("[UserRepository] Error finding user by email:", error);
-      throw new Error(`Failed to find user with email: ${email}`);
-    }
-  }
-
-  /**
-   * Create new user
-   */
-  async create(data: Prisma.UserCreateInput): Promise<User> {
-    try {
-      return await PrismaService.main.user.create({ data });
-    } catch (error) {
-      console.error("[UserRepository] Error creating user:", error);
-      throw new Error("Failed to create user");
-    }
-  }
-
-  /**
-   * Update user
-   */
-  async update(userId: string, data: Prisma.UserUpdateInput): Promise<User> {
-    try {
-      return await PrismaService.main.user.update({
-        where: { userID: userId },
-        data,
-      });
-    } catch (error) {
-      console.error("[UserRepository] Error updating user:", error);
-      throw new Error(`Failed to update user: ${userId}`);
-    }
-  }
-
-  /**
-   * Delete user (soft delete by setting isActive = false)
-   */
-  async delete(userId: string): Promise<User> {
-    try {
-      return await PrismaService.main.user.update({
-        where: { userID: userId },
-        data: { isActive: false },
-      });
-    } catch (error) {
-      console.error("[UserRepository] Error deleting user:", error);
-      throw new Error(`Failed to delete user: ${userId}`);
-    }
-  }
-
-  /**
-   * Check if email exists
-   */
-  async emailExists(email: string): Promise<boolean> {
-    try {
-      const count = await PrismaService.main.user.count({
-        where: { email },
-      });
-      return count > 0;
-    } catch (error) {
-      console.error("[UserRepository] Error checking email exists:", error);
-      throw new Error("Failed to check if email exists");
-    }
-  }
-}
-
-// Export singleton instance
-export const userRepository = new UserRepository();
-```
-
-**Using Repository in Service:**
-
-```typescript
-// services/userService.ts
-import { userRepository } from "../repositories/UserRepository";
-import { ConflictError, NotFoundError } from "../utils/errors";
-
-export class UserService {
-  /**
-   * Create new user with business rules
-   */
-  async createUser(data: {
-    email: string;
-    name: string;
-    roles: string[];
-  }): Promise<User> {
-    // Business rule: Check if email already exists
-    const emailExists = await userRepository.emailExists(data.email);
-    if (emailExists) {
-      throw new ConflictError("Email already exists");
-    }
-
-    // Business rule: Validate roles
-    const validRoles = ["admin", "operations", "user"];
-    const invalidRoles = data.roles.filter(
-      (role) => !validRoles.includes(role),
-    );
-    if (invalidRoles.length > 0) {
-      throw new ValidationError(`Invalid roles: ${invalidRoles.join(", ")}`);
-    }
-
-    // Create user via repository
-    return await userRepository.create({
-      email: data.email,
-      name: data.name,
-      roles: data.roles,
-      isActive: true,
-    });
-  }
-
-  /**
-   * Get user by ID
-   */
-  async getUser(userId: string): Promise<User> {
-    const user = await userRepository.findById(userId);
-
-    if (!user) {
-      throw new NotFoundError(`User not found: ${userId}`);
-    }
-
-    return user;
-  }
-}
-```
+Keep data-access concerns in repositories and business decisions in services.
+Project-scoped queries must include `projectId` or `project_id` filters.

 ---

@@ -803,69 +632,13 @@ class UserService {

 ## Testing Services

-### Unit Tests
+Use `testing-guide.md` for backend test patterns. Prefer current Langfuse tests
+over invented examples:

-```typescript
-// tests/userService.test.ts
-import { UserService } from "../services/userService";
-import { userRepository } from "../repositories/UserRepository";
-import { ConflictError } from "../utils/errors";
-
-// Mock repository
-jest.mock("../repositories/UserRepository");
-
-describe("UserService", () => {
-  let userService: UserService;
-
-  beforeEach(() => {
-    userService = new UserService();
-    jest.clearAllMocks();
-  });
-
-  describe("createUser", () => {
-    it("should create user when email does not exist", async () => {
-      // Arrange
-      const userData = {
-        email: "test@example.com",
-        name: "Test User",
-        roles: ["user"],
-      };
-
-      (userRepository.emailExists as jest.Mock).mockResolvedValue(false);
-      (userRepository.create as jest.Mock).mockResolvedValue({
-        userID: "123",
-        ...userData,
-      });
-
-      // Act
-      const user = await userService.createUser(userData);
-
-      // Assert
-      expect(user).toBeDefined();
-      expect(user.email).toBe(userData.email);
-      expect(userRepository.emailExists).toHaveBeenCalledWith(userData.email);
-      expect(userRepository.create).toHaveBeenCalled();
-    });
-
-    it("should throw ConflictError when email exists", async () => {
-      // Arrange
-      const userData = {
-        email: "existing@example.com",
-        name: "Test User",
-        roles: ["user"],
-      };
-
-      (userRepository.emailExists as jest.Mock).mockResolvedValue(true);
-
-      // Act & Assert
-      await expect(userService.createUser(userData)).rejects.toThrow(
-        ConflictError,
-      );
-      expect(userRepository.create).not.toHaveBeenCalled();
-    });
-  });
-});
-```
+- Repository tests:
+  `web/src/__tests__/server/repositories/event-repository.servertest.ts`
+- Pure service unit tests:
+  `web/src/__tests__/server/unit/`

 ---

@@ -4,25 +4,65 @@ Complete guide to testing Langfuse backend services across web, worker, and shar

 ## Table of Contents

+- [Key Testing Principles](#key-testing-principles)
 - [Test Types Overview](#test-types-overview)
 - [Integration Tests (Public API)](#integration-tests-public-api)
 - [Service-Level Tests (Repository/Service)](#service-level-tests-repositoryservice)
 - [tRPC Tests (Procedure Testing)](#trpc-tests-procedure-testing)
 - [Worker Tests (Queue Processing)](#worker-tests-queue-processing)
- [Key Testing Principles](#key-testing-principles)
 - [Running Tests](#running-tests)

 ---

+## Key Testing Principles
+
+### General Principles
+
+1. **Test Isolation**: Each test should be independent and runnable in any order
+2. **Unique IDs**: Use `randomUUID()` or unique project IDs to avoid test interference
+3. **Cleanup**: Always clean up test data in service tests (or use unique project IDs)
+4. **Avoid Global Resets**: Prefer scoped cleanup or unique project IDs over global reset helpers
+5. **Flags and Fallbacks**: When code branches on env flags, feature flags, or fallback data paths, test both branches and ensure fixtures are written to the same store the branch reads from
+
+### By Test Type
+
+| Test Type | Key Principles |
+|-----------|----------------|
+| **Integration** | Test HTTP endpoints, validate status codes and response shapes |
+| **tRPC** | Use `createInnerTRPCContext` and `appRouter.createCaller`, test auth/permissions |
+| **Service** | Test individual functions with isolated data, always cleanup |
+| **Worker** | Use vitest, test streams with async iteration, test filtering logic |
+
+### Test Data Management
+
+```typescript
+// ✅ GOOD: Use unique IDs
+const projectId = randomUUID();
+const traceId = randomUUID();
+
+// ✅ GOOD: Cleanup in service tests
+afterAll(async () => {
+  await prisma.model.delete({ where: { id: modelId } });
+});
+
+// ✅ GOOD: Use unique projects (no cleanup needed)
+const { projectId } = await createOrgProjectAndApiKey();
+
+// ❌ BAD: Shared test data between tests
+const projectId = "7a88fb47-b4e2-43b8-a06c-a5ce950dc53a";
+```
+
+---
+
 ## Test Types Overview

 Langfuse uses multiple testing strategies for different layers:

 | Test Type | Framework | Location | Purpose |
 |-----------|-----------|----------|---------|
-| Integration | Jest | `web/src/__tests__/async/` | Full API endpoint testing |
-| tRPC | Jest | `web/src/__tests__/async/` | tRPC procedure testing with auth |
-| Service | Jest | `web/src/__tests__/async/repositories/` | Repository/service function testing |
+| Integration | Vitest | `web/src/__tests__/server/` | Full API endpoint testing |
+| tRPC | Vitest | `web/src/__tests__/server/` | tRPC procedure testing with auth |
+| Service | Vitest | `web/src/__tests__/server/repositories/` | Repository/service function testing |
 | Worker | Vitest | `worker/src/__tests__/` | Queue processors and streams |

 ---
@@ -31,7 +71,7 @@ Langfuse uses multiple testing strategies for different layers:

 Test full REST API endpoints end-to-end using HTTP requests.

-**File location:** `web/src/__tests__/async/datasets-api.servertest.ts`
+**File location:** `web/src/__tests__/server/datasets-api.servertest.ts`

 ```typescript
 import { makeZodVerifiedAPICall } from "../helpers";
@@ -73,7 +113,7 @@ describe("Dataset API", () => {

 Test individual repository/service functions with isolated data.

-**File location:** `web/src/__tests__/async/repositories/event-repository.servertest.ts`
+**File location:** `web/src/__tests__/server/repositories/event-repository.servertest.ts`

 ```typescript
 import {
@@ -186,7 +226,7 @@ describe("Event Repository Tests", () => {

 Test tRPC procedures with caller pattern and auth context.

-**File location:** `web/src/__tests__/async/automations-trpc.servertest.ts`
+**File location:** `web/src/__tests__/server/automations-trpc.servertest.ts`

 ```typescript
 import { appRouter } from "@/src/server/api/root";
@@ -464,88 +504,15 @@ describe("batch export test suite", () => {

 ---

-## Key Testing Principles
-
-### General Principles
-
-1. **Test Isolation**: Each test should be independent and runnable in any order
-2. **Unique IDs**: Use `randomUUID()` or unique project IDs to avoid test interference
-3. **Cleanup**: Always clean up test data in service tests (or use unique project IDs)
-4. **Avoid Global Resets**: Prefer scoped cleanup or unique project IDs over global reset helpers
-
-### By Test Type
-
-| Test Type | Key Principles |
-|-----------|----------------|
-| **Integration** | Test HTTP endpoints, validate status codes and response shapes |
-| **tRPC** | Use `createInnerTRPCContext` and `appRouter.createCaller`, test auth/permissions |
-| **Service** | Test individual functions with isolated data, always cleanup |
-| **Worker** | Use vitest, test streams with async iteration, test filtering logic |
-
-### Test Data Management
-
-```typescript
-// ✅ GOOD: Use unique IDs
-const projectId = randomUUID();
-const traceId = randomUUID();
-
-// ✅ GOOD: Cleanup in service tests
-afterAll(async () => {
-  await prisma.model.delete({ where: { id: modelId } });
-});
-
-// ✅ GOOD: Use unique projects (no cleanup needed)
-const { projectId } = await createOrgProjectAndApiKey();
-
-// ❌ BAD: Shared test data between tests
-const projectId = "7a88fb47-b4e2-43b8-a06c-a5ce950dc53a";
-```
-
---
-
 ## Running Tests

-### Web Tests (Jest)
+Use the nearest package `AGENTS.md` as the source of truth for current test
+commands.

-```bash
-# Run all tests
-pnpm test
-
-# Run sync tests
-pnpm test-sync
-
-# Run async tests
-pnpm test -- --testPathPattern="async"
-
-# Run specific test file
-pnpm test -- --testPathPattern="datasets-api"
-
-# Run specific test
-pnpm test -- --testPathPattern="datasets-api" --testNamePattern="should create dataset"
-```
-
-### Worker Tests (Vitest)
-
-```bash
-# Run all worker tests
-pnpm run test --filter=worker
-
-# Run specific test file
-pnpm run test --filter=worker -- batchExport
-
-# Run specific test
-pnpm run test --filter=worker -- batchExport -t "should export observations"
-```
-
-### Coverage
-
-```bash
-# Web coverage
-pnpm test -- --coverage
-
-# Worker coverage
-pnpm run test --filter=worker -- --coverage
-```
+Common targeted forms:
+- Web server tests: `pnpm --filter web run test -- <pattern>`
+- Web client tests: `pnpm --filter web run test-client -- <pattern>`
+- Worker tests: `pnpm --filter worker run test <file-or-pattern>`

 ---

@@ -25,9 +25,20 @@ Comprehensive guidance for ClickHouse covering schema design, query optimization

 **Why rules take priority:** ClickHouse has specific behaviors (columnar storage, sparse indexes, merge tree mechanics) where general database intuition can be misleading. The rules encode validated, ClickHouse-specific guidance.

-### For Formal Reviews
+## Langfuse-Specific Rules

-When performing a formal review of schemas, queries, or data ingestion:
+- Use `packages/shared/src/server/queries/clickhouse-sql/event-query-builder.ts`
+  for queries against the `events` table. Do not hand-roll `events` SQL unless
+  you first confirm the query builder cannot express the query.
+- Never use `FINAL` on the `events` table; it is designed so `FINAL` is not
+  required and the keyword hurts performance.
+- Any migration in `packages/shared/clickhouse/migrations/clustered/**` with
+  more than one `ALTER` on the same table must end every metadata `ALTER`
+  (`ADD/DROP/MODIFY COLUMN`, `ADD/DROP INDEX`) with `SETTINGS alter_sync = 2`,
+  and every mutation-creating `ALTER` (`MATERIALIZE …`, `UPDATE`, `DELETE`)
+  with `SETTINGS mutations_sync = 2`. The matching `unclustered/` file runs
+  against plain `MergeTree` and does not need (and should not duplicate)
+  these settings.

 ---

@@ -53,6 +64,7 @@ When performing a formal review of schemas, queries, or data ingestion:
 - [ ] LowCardinality applied to appropriate string columns
 - [ ] Partition key cardinality bounded (100-1,000 values)
 - [ ] ReplacingMergeTree has version column if used
+- [ ] Clustered migration files with multiple ALTERs on the same table use `SETTINGS alter_sync = 2` (metadata) and `SETTINGS mutations_sync = 2` (`MATERIALIZE …`, `UPDATE`, `DELETE`); unclustered mirror has none

 ### For Query Reviews (SELECT, JOIN, aggregations)

@@ -227,8 +239,8 @@ Each rule file in `rules/` contains:

 ---

-## Full Compiled Document
+## Compatibility Entrypoint

-For the complete guide with all rules expanded inline: `AGENTS.md`
-
-Use `AGENTS.md` when you need to check multiple rules quickly without reading individual files.
+`AGENTS.md` is a short compatibility index for agents that open that file
+directly. The authoritative workflow lives in this `SKILL.md`, and detailed rule
+bodies live in `rules/`.
@@ -0,0 +1,73 @@
+---
+name: datadog-query-recipes
+description: |
+  Langfuse-specific Datadog query recipes for production telemetry research.
+  Use when asked to investigate tenant or project activity, public API endpoint
+  usage, queue consumer behavior, spans, logs, metrics, or ad hoc production
+  questions across prod-us, prod-eu, prod-hipaa, and prod-jp. This skill is for
+  reusable query shapes and measured research; pair it with
+  debug-issue-with-datadog when the task is an incident or root-cause analysis.
+---
+
+# Datadog Query Recipes
+
+Use this skill for Langfuse production telemetry research where the main work is
+finding the right Datadog data path. Keep findings evidence-based and include
+the exact Datadog links or query shapes that support the answer.
+
+## Required Scope
+
+Unless the user explicitly narrows the scope, cover every production
+environment:
+
+- `prod-us`
+- `prod-eu`
+- `prod-hipaa`
+- `prod-jp`
+
+Query both Datadog sites when needed. Default to the EU site for `prod-eu` and
+the US site for the other prod environments, but verify with a small count or
+facet query before concluding an environment has no data.
+
+Before querying live Datadog, load the relevant Datadog MCP guidance for the
+data domain you need: traces, logs, metrics, and visualizations.
+
+## Workflow
+
+1. Identify the entity and signal: tenant ID, org ID, project ID, route, queue,
+   service, error class, or metric.
+2. Read only the relevant reference:
+   - Prod environment/site routing:
+     [`references/environments.md`](references/environments.md)
+   - Public API tenant or legacy endpoint usage:
+     [`references/public-api-tenant-usage.md`](references/public-api-tenant-usage.md)
+   - Queue inventory, queue consumers, and queue metrics:
+     [`references/queue-consumers.md`](references/queue-consumers.md)
+3. Start with aggregate queries, grouped by environment, service, route,
+   queue, project, org, status, or error facets as appropriate.
+4. Fetch raw spans, logs, or traces only after aggregation identifies the
+   cluster or sample you need.
+5. For tenant-specific HTTP usage, prefer trace correlation over single-span
+   queries when tenant tags and route tags live on different spans.
+6. Report the windows, environments, sites, query links, and any sampling or
+   missing-data caveats.
+
+## When To Use Other Skills
+
+- Use [`debug-issue-with-datadog`](../debug-issue-with-datadog/SKILL.md) when a
+  Linear issue, GitHub issue, incident report, or monitor needs root-cause
+  analysis and patch recommendations.
+- Use [`detect-prod-regressions`](../detect-prod-regressions/SKILL.md) when the
+  user asks for a proactive production sweep or baseline comparison.
+- Use [`linear-bug-triage`](../linear-bug-triage/SKILL.md) only after a human
+  approves sharing measured findings in Linear.
+
+## Output Expectations
+
+Summarize what was checked, including:
+
+- Datadog site and `env` values covered.
+- Time windows.
+- Core filters or metrics used.
+- Count, rate, latency, queue depth, trace sample, or "No measurements found".
+- Datadog links or trace IDs that let the human rerun the query.
@@ -0,0 +1,4 @@
+interface:
+  display_name: "Datadog Query Recipes"
+  short_description: "Langfuse production telemetry queries"
+  default_prompt: "Use $datadog-query-recipes to investigate Langfuse production telemetry for a tenant, endpoint, queue, or regression."
@@ -0,0 +1,45 @@
+# Production Environments
+
+Langfuse production deploys cover these environments and services. The deploy
+matrix is defined in `.github/workflows/deploy.yml`.
+
+| Environment | Primary Datadog site | Common services |
+| --- | --- | --- |
+| `prod-us` | US, `datadoghq.com` | `web`, `web-ingestion`, `web-iso`, `worker`, `worker-cpu` |
+| `prod-eu` | EU, `datadoghq.eu` | `web`, `web-ingestion`, `web-iso`, `worker`, `worker-cpu` |
+| `prod-hipaa` | US, `datadoghq.com` | `web`, `web-ingestion`, `web-iso`, `worker`, `worker-cpu` |
+| `prod-jp` | US, `datadoghq.com` | `web`, `web-ingestion`, `web-iso`, `worker`, `worker-cpu` |
+
+The site mapping is a starting point, not proof. For cross-region research, run
+a small count or facet query on both Datadog sites before saying an environment
+has no data.
+
+## Starter Filters
+
+Use these as first-pass filters, then add the subsystem-specific route, queue,
+tenant, or error facets.
+
+```text
+env:prod-us
+env:prod-eu
+env:prod-hipaa
+env:prod-jp
+```
+
+```text
+env:<env> (service:web OR service:web-ingestion OR service:web-iso)
+env:<env> (service:worker OR service:worker-cpu)
+```
+
+For HTTP routes, start with `service:web` or `service:web-ingestion` depending
+on the endpoint. For queue consumers, start with `service:worker` and
+`service:worker-cpu`.
+
+## Cross-Site Rule
+
+When a query returns zero:
+
+1. Check the time window and spelling of `env`, `service`, and route/resource.
+2. Query facets on the same site for `env` and `service`.
+3. Repeat a small count query on the other Datadog site.
+4. Only then report "No measurements found".
@@ -0,0 +1,99 @@
+# Public API Tenant Usage
+
+Use this recipe when checking whether an org or project calls a public API
+route, including legacy endpoints such as:
+
+- `/api/public/traces`
+- `/api/public/traces/<traceId>` (`/api/public/traces/[traceId]` in Next.js)
+- `/api/public/observations`
+- `/api/public/observations/<observationId>`
+  (`/api/public/observations/[observationId]` in Next.js)
+- `/api/public/metrics`
+
+Migrated endpoints commonly include:
+
+- `/api/public/v2/observations`
+- `/api/public/v2/metrics`
+- `/api/public/otel/v1/traces`
+
+## Why Correlation Is Needed
+
+Public API auth attaches tenant attributes to the `api-auth-verify` child span:
+
+- `@langfuse.project.id`: project-scoped API key target.
+- `@langfuse.org.id`: owning org or org-scoped key target.
+- `@langfuse.org.plan`: plan when present.
+
+The HTTP route is on the request root span, usually in `http.path_group`,
+`http.route`, `http.target`, or `resource_name`.
+
+Because tenant tags and route tags often live on different spans in the same
+trace, do not expect this single-span query to work:
+
+```text
+@langfuse.project.id:<id> @http.path_group:/api/public/observations
+```
+
+## Per-Tenant Endpoint Recipe
+
+1. Aggregate auth spans for the tenant to confirm the identifier and active
+   projects.
+
+```text
+env:<env> @langfuse.org.id:<orgId> resource_name:api-auth-verify
+env:<env> @langfuse.project.id:<projectId> resource_name:api-auth-verify
+```
+
+Group by `service`, `@langfuse.project.id`, and optionally a daily interval.
+
+2. Fetch representative matching auth spans with `search_datadog_spans`.
+   Include custom attributes such as `langfuse.*`, then copy representative
+   `traceid` values.
+
+3. Open those traces with `get_datadog_trace`. Request service-entry spans and
+   include HTTP and Langfuse attributes:
+
+```text
+only_service_entry_spans: true
+extra_fields: ["http.*", "next.*", "langfuse.*"]
+```
+
+4. Read the request root span's `http.path_group`, `http.route`,
+   `http.target`, and `resource_name` to identify the endpoint.
+   ID lookup routes may appear with the concrete ID in `http.target` or with
+   the normalized Next.js route, such as
+   `/api/public/traces/[traceId]` or
+   `/api/public/observations/[observationId]`.
+
+5. Repeat across all relevant prod environments and both Datadog sites when the
+   user asks for a global answer.
+
+Treat per-tenant endpoint results as sampled unless you have an unsampled
+metric or log source with tenant and route on the same event.
+
+## Fleet-Level Endpoint Volume
+
+For endpoint volume without tenant scoping, aggregate request spans directly:
+
+```text
+env:<env> service:web resource_name:"GET /api/public/observations*"
+env:<env> service:web resource_name:"GET /api/public/observations/*"
+env:<env> service:web resource_name:"POST /api/public/traces*"
+env:<env> service:web resource_name:"GET /api/public/traces/*"
+env:<env> service:web resource_name:"POST /api/public/metrics*"
+```
+
+Use route facets where available:
+
+```text
+env:<env> service:web @http.path_group:/api/public/observations
+env:<env> service:web @http.path_group:/api/public/observations/*
+env:<env> service:web @http.path_group:/api/public/observations/[observationId]
+env:<env> service:web @http.route:/api/public/observations
+env:<env> service:web @http.route:/api/public/observations/[observationId]
+env:<env> service:web @http.path_group:/api/public/traces/[traceId]
+env:<env> service:web @http.route:/api/public/traces/[traceId]
+```
+
+If route facets and resource names disagree, fetch a few traces and inspect the
+root span before reporting the result.
@@ -0,0 +1,234 @@
+# Queue Consumers
+
+Use this reference when a task asks which Langfuse queues exist, whether a
+consumer is running, how much work a queue has, or how to query queue processor
+spans.
+
+## Source Of Truth
+
+- Queue names and job names:
+  `packages/shared/src/server/queues.ts` (`QueueName`, `QueueJobs`).
+- Queue producer classes and shard naming:
+  `packages/shared/src/server/redis/*.ts`.
+- Worker consumer registration and feature gates:
+  `worker/src/app.ts`.
+- Worker consumer env vars:
+  `worker/src/env.ts` (`QUEUE_CONSUMER_*_IS_ENABLED` plus feature-specific
+  gates).
+- Worker registration, request/error counters, wait/processing time, and
+  sampled old-style depth metrics:
+  `worker/src/queues/workerManager.ts`.
+- Queue depth background reporter:
+  `worker/src/features/queue-metrics-runner/index.ts`.
+- Metric name conversion:
+  `packages/shared/src/server/instrumentation/index.ts`
+  (`convertQueueNameToMetricName`).
+- Sharded queue registry:
+  `worker/src/queues/shardedQueueRegistry.ts`.
+- BullMQ tracing setup:
+  `worker/src/instrumentation.ts` (`BullMQInstrumentation`).
+
+## Queue Inventory
+
+Current `QueueName` values:
+
+| Queue | Notes |
+| --- | --- |
+| `trace-upsert` | Sharded. Registers all `TraceUpsertQueue` shards. |
+| `trace-delete` | Delete traces from storage. |
+| `project-delete` | Project deletion cleanup. |
+| `evaluation-execution-queue` | Sharded eval execution. |
+| `secondary-evaluation-execution-queue` | Sharded secondary eval execution. |
+| `llm-as-a-judge-execution-queue` | Sharded observation-based eval execution. |
+| `dataset-run-item-upsert-queue` | Dataset run item upserts. |
+| `batch-export-queue` | Batch exports. |
+| `otel-ingestion-queue` | Sharded OTel ingestion. |
+| `secondary-otel-ingestion-queue` | Sharded secondary OTel ingestion. |
+| `ingestion-queue` | Sharded single-event ingestion. |
+| `secondary-ingestion-queue` | Sharded secondary single-event ingestion. |
+| `cloud-usage-metering-queue` | Cloud-only, Stripe-gated. |
+| `cloud-spend-alert-queue` | Cloud-only, Stripe-gated. |
+| `cloud-free-tier-usage-threshold-queue` | Cloud-only, Stripe-gated. |
+| `experiment-create-queue` | Experiment creation. |
+| `posthog-integration-queue` | Schedules PostHog integration jobs. |
+| `posthog-integration-processing-queue` | Processes PostHog projects. |
+| `mixpanel-integration-queue` | Schedules Mixpanel integration jobs. |
+| `mixpanel-integration-processing-queue` | Processes Mixpanel projects. |
+| `blobstorage-integration-queue` | Schedules blob storage jobs. |
+| `blobstorage-integration-processing-queue` | Processes blob storage projects. |
+| `core-data-s3-export-queue` | Cloud export feature gate. |
+| `metering-data-postgres-export-queue` | Cloud export feature gate. |
+| `data-retention-queue` | Schedules data retention jobs. |
+| `data-retention-processing-queue` | Processes data retention projects. |
+| `batch-action-queue` | Batch actions. |
+| `create-eval-queue` | Eval job creation. |
+| `score-delete` | Score deletion cleanup. |
+| `dataset-delete-queue` | Dataset deletion cleanup. |
+| `dead-letter-retry-queue` | Dead letter retry worker. |
+| `webhook-queue` | Webhook delivery. |
+| `entity-change-queue` | Entity change propagation. |
+| `event-propagation-queue` | Experiment event propagation gate. |
+| `notification-queue` | Notifications. |
+
+Sharded queues use the base queue for shard 0 and append `-1`, `-2`, etc. for
+additional shards. The sharded base queues are:
+
+- `trace-upsert`
+- `evaluation-execution-queue`
+- `secondary-evaluation-execution-queue`
+- `llm-as-a-judge-execution-queue`
+- `otel-ingestion-queue`
+- `secondary-otel-ingestion-queue`
+- `ingestion-queue`
+- `secondary-ingestion-queue`
+
+## Consumer Gates
+
+Consumer registration is in `worker/src/app.ts`. Some gates register multiple
+queues or every shard for a sharded queue.
+
+| Gate | Queues registered |
+| --- | --- |
+| `QUEUE_CONSUMER_TRACE_UPSERT_QUEUE_IS_ENABLED` | `trace-upsert` shards |
+| `QUEUE_CONSUMER_CREATE_EVAL_QUEUE_IS_ENABLED` | `create-eval-queue` |
+| `LANGFUSE_S3_CORE_DATA_EXPORT_IS_ENABLED` | `core-data-s3-export-queue` |
+| `LANGFUSE_POSTGRES_METERING_DATA_EXPORT_IS_ENABLED` | `metering-data-postgres-export-queue` |
+| `QUEUE_CONSUMER_TRACE_DELETE_QUEUE_IS_ENABLED` | `trace-delete` |
+| `QUEUE_CONSUMER_SCORE_DELETE_QUEUE_IS_ENABLED` | `score-delete` |
+| `QUEUE_CONSUMER_DATASET_DELETE_QUEUE_IS_ENABLED` | `dataset-delete-queue` |
+| `QUEUE_CONSUMER_PROJECT_DELETE_QUEUE_IS_ENABLED` | `project-delete` |
+| `QUEUE_CONSUMER_DATASET_RUN_ITEM_UPSERT_QUEUE_IS_ENABLED` | `dataset-run-item-upsert-queue` |
+| `QUEUE_CONSUMER_EVAL_EXECUTION_QUEUE_IS_ENABLED` | `evaluation-execution-queue` shards, `llm-as-a-judge-execution-queue` shards |
+| `QUEUE_CONSUMER_EVAL_EXECUTION_SECONDARY_QUEUE_IS_ENABLED` | `secondary-evaluation-execution-queue` shards |
+| `QUEUE_CONSUMER_BATCH_EXPORT_QUEUE_IS_ENABLED` | `batch-export-queue` |
+| `QUEUE_CONSUMER_BATCH_ACTION_QUEUE_IS_ENABLED` | `batch-action-queue` |
+| `QUEUE_CONSUMER_OTEL_INGESTION_QUEUE_IS_ENABLED` | `otel-ingestion-queue` shards |
+| `QUEUE_CONSUMER_OTEL_INGESTION_SECONDARY_QUEUE_IS_ENABLED` | `secondary-otel-ingestion-queue` shards |
+| `QUEUE_CONSUMER_INGESTION_QUEUE_IS_ENABLED` | `ingestion-queue` shards |
+| `QUEUE_CONSUMER_INGESTION_SECONDARY_QUEUE_IS_ENABLED` | `secondary-ingestion-queue` shards |
+| `QUEUE_CONSUMER_CLOUD_USAGE_METERING_QUEUE_IS_ENABLED` plus `STRIPE_SECRET_KEY` | `cloud-usage-metering-queue` |
+| `QUEUE_CONSUMER_CLOUD_SPEND_ALERT_QUEUE_IS_ENABLED` plus `STRIPE_SECRET_KEY` | `cloud-spend-alert-queue` |
+| `QUEUE_CONSUMER_FREE_TIER_USAGE_THRESHOLD_QUEUE_IS_ENABLED` plus cloud region and Stripe gates | `cloud-free-tier-usage-threshold-queue` |
+| `QUEUE_CONSUMER_EXPERIMENT_CREATE_QUEUE_IS_ENABLED` | `experiment-create-queue` |
+| `QUEUE_CONSUMER_POSTHOG_INTEGRATION_QUEUE_IS_ENABLED` | `posthog-integration-queue`, `posthog-integration-processing-queue` |
+| `QUEUE_CONSUMER_MIXPANEL_INTEGRATION_QUEUE_IS_ENABLED` | `mixpanel-integration-queue`, `mixpanel-integration-processing-queue` |
+| `QUEUE_CONSUMER_BLOB_STORAGE_INTEGRATION_QUEUE_IS_ENABLED` | `blobstorage-integration-queue`, `blobstorage-integration-processing-queue` |
+| `QUEUE_CONSUMER_DATA_RETENTION_QUEUE_IS_ENABLED` | `data-retention-queue`, `data-retention-processing-queue` |
+| `QUEUE_CONSUMER_DEAD_LETTER_RETRY_QUEUE_IS_ENABLED` | `dead-letter-retry-queue` |
+| `QUEUE_CONSUMER_WEBHOOK_QUEUE_IS_ENABLED` | `webhook-queue` |
+| `QUEUE_CONSUMER_ENTITY_CHANGE_QUEUE_IS_ENABLED` | `entity-change-queue` |
+| `QUEUE_CONSUMER_EVENT_PROPAGATION_QUEUE_IS_ENABLED` plus events-table experiment gate | `event-propagation-queue` |
+| `QUEUE_CONSUMER_NOTIFICATION_QUEUE_IS_ENABLED` | `notification-queue` |
+
+## Query Consumer Spans
+
+Start with aggregate spans on worker services:
+
+```text
+env:<env> (service:worker OR service:worker-cpu) operation_name:bullmq.process
+```
+
+Then group by `resource_name`, queue facets such as `bullmq.queue` or
+`messaging.*`, and error fields. Facet names can differ between Datadog sites,
+so inspect one sample span before relying on a specific facet.
+
+Queue-specific starter query:
+
+```text
+env:<env> (service:worker OR service:worker-cpu) operation_name:bullmq.process (resource_name:"process otel-ingestion-queue" OR resource_name:"Worker.run otel-ingestion-queue" OR bullmq.queue:otel-ingestion-queue)
+```
+
+For sharded queues, query the base queue and shard suffixes:
+
+```text
+env:<env> (service:worker OR service:worker-cpu) operation_name:bullmq.process resource_name:"*otel-ingestion-queue*"
+```
+
+If a queue file wraps the handler with `instrumentAsync`, also search the
+domain-specific resource name. Examples:
+
+| Subsystem | Resource name |
+| --- | --- |
+| PostHog project processing | `process posthog-integration-project` |
+| Mixpanel project processing | `process mixpanel-integration-project` |
+| Blob storage project processing | `process blob-storage-project` |
+| Data retention project processing | `process data-retention-project` |
+| Event propagation | `process event-propagation` |
+| Cloud usage metering | `process cloud-usage-metering` |
+| Free-tier usage threshold | `process cloud-free-tier-usage-threshold` |
+
+Useful aggregations:
+
+- Count by `env`, `service`, and `resource_name`.
+- Count by queue facet and status.
+- Count by `error.type` and `error.message`.
+- p50, p95, and p99 duration by queue or shard.
+- Count by `messaging.bullmq.job.input.projectId` when the processor attaches
+  project IDs to the current span.
+
+## Query Queue Metrics
+
+Metric base names come from `convertQueueNameToMetricName(queueName)`:
+
+```text
+langfuse.queue.<queue-name-with-hyphens-replaced-by-underscores-and-trailing-_queue-removed>
+```
+
+Examples:
+
+| Queue | Metric base |
+| --- | --- |
+| `ingestion-queue` | `langfuse.queue.ingestion` |
+| `otel-ingestion-queue` | `langfuse.queue.otel_ingestion` |
+| `secondary-otel-ingestion-queue` | `langfuse.queue.secondary_otel_ingestion` |
+| `evaluation-execution-queue` | `langfuse.queue.evaluation_execution` |
+| `trace-upsert` | `langfuse.queue.trace_upsert` |
+| `batch-export-queue` | `langfuse.queue.batch_export` |
+
+Prefer the newer tagged metrics:
+
+```text
+<metric_base>.depth{env:<env>,type:waiting}
+<metric_base>.depth{env:<env>,type:failed}
+<metric_base>.depth{env:<env>,type:active}
+<metric_base>.rate{env:<env>,type:request}
+<metric_base>.rate{env:<env>,type:failed}
+<metric_base>.rate{env:<env>,type:error}
+<metric_base>.time{env:<env>,type:wait}
+<metric_base>.time{env:<env>,type:processing}
+```
+
+For sharded queues, use the `shard` tag when present. `shard:all` is emitted by
+the depth runner for aggregate depth across shards.
+
+Backward-compatible metrics may still appear:
+
+```text
+<metric_base>.length
+<metric_base>.dlq_length
+<metric_base>.active
+<metric_base>.request
+<metric_base>.failed
+<metric_base>.error
+<metric_base>.wait_time
+<metric_base>.processing_time
+```
+
+For non-BullMQ internal write buffering, `ClickhouseWriter` emits
+`langfuse.queue.clickhouse_writer.*` metrics, but it is not a `QueueName`
+consumer.
+
+## Consumer Running Checklist
+
+To establish whether a consumer is running in production:
+
+1. Check queue depth metrics for waiting, failed, and active counts.
+2. Check `rate{type:request}` or old `.request` metrics for recent processing.
+3. Search BullMQ processor spans on `worker` and `worker-cpu`.
+4. Search worker logs for the queue name or processor-specific log prefix.
+5. If all signals are empty, verify the relevant `QUEUE_CONSUMER_*_IS_ENABLED`
+   gate and any feature-specific gates in `worker/src/app.ts`.
+
+The queue metrics runner only polls queues with registered workers. Missing
+depth metrics can mean the consumer is not registered on that worker, queue
+metrics are disabled, or the data is on the other Datadog site.
@@ -0,0 +1,121 @@
+---
+name: debug-issue-with-datadog
+description: |
+  Debug a user-reported issue, Linear ticket, or incident report by combining
+  Datadog (APM, logs, metrics) with the Langfuse repo to establish a
+  root cause. Use when given a Linear issue URL/ID (e.g. LFE-XXXX), a GitHub
+  issue, or a pasted error/report and asked to investigate, root-cause, or
+  triage. Produces a structured analysis — error breakdown, hypothesis-by-class,
+  suggested patches with code references.
+---
+
+# Debug Issue with Datadog
+
+Use this skill whenever the task is **investigative** rather than
+implementational: a user, customer, or oncall has surfaced a problem and you
+need to figure out *what is actually happening in production* and *where in the
+code it lives*. The deliverable is an analysis, not a patch — though the
+analysis should make the right patch obvious.
+
+## When to Apply
+
+- A Linear issue (typically with an `LFE-XXXX` ID) describes a production
+  failure, error spike, or customer report.
+- A GitHub issue or pasted incident/error report needs triage.
+- A monitor alerted and you need to understand *why* before deciding what to
+  fix.
+- Existing tickets under the "Make monitoring useful again" project (parent
+  `LFE-8837`) and similar — these expect the structured analysis output below.
+
+If the task is "implement this fix" rather than "figure out what's broken",
+this is the wrong skill — go to `backend-dev-guidelines` or the relevant
+package guide.
+
+## Workflow
+
+Read the inputs first, then plan the Datadog sweep, then read the code, then
+write the analysis. Do not skip ahead to suggested patches before the data
+supports them.
+
+1. **Intake.** Pull every signal already available in the report. See
+   [`references/intake.md`](references/intake.md). For a Linear URL/ID, fetch
+   the issue *and* its comments via the Linear MCP — the description is often
+   updated inline as triage proceeds. For a GitHub issue, use `gh issue view`.
+   For pasted text, treat it as the description.
+
+2. **Scope the sweep.** From the intake, pick the affected subsystem and time
+   window. Use [`references/repo-debug-map.md`](references/repo-debug-map.md)
+   to translate "PostHog integration", "ingestion failures", "evals stuck",
+   etc. into the Datadog filters and source files you should be looking at.
+
+3. **Run the broad Datadog sweep.** Default to the full sweep in
+   [`references/datadog-playbook.md`](references/datadog-playbook.md): APM
+   spans, error logs, metrics, and monitors — split across `prod-eu`
+   and `prod-us` (and `prod-hipaa` / `prod-jp` when relevant). Always check
+   regional disparity first; it usually rules whole hypotheses in or out.
+   Use [`datadog-query-recipes`](../datadog-query-recipes/SKILL.md) for
+   reusable tenant, public API, queue consumer, and cross-environment query
+   shapes.
+
+4. **Cluster the errors.** Group by `(projectId, error.message)` or
+   `(error.type, error.message)`. Treat each distinct cluster as its own
+   hypothesis — Langfuse incidents commonly have *multiple* coexisting root
+   causes, not one.
+
+5. **Map clusters to code.** For each cluster, open the relevant handler file
+   from the repo-debug map and read enough of it to confirm or refute the
+   hypothesis. Cite specific files and line ranges in the output.
+
+6. **Write the analysis** using
+   [`references/output-template.md`](references/output-template.md).
+
+7. **Deliver.** Default: print the analysis in chat. If the user asked for it,
+   also save under the workflow they specified (file, Linear comment via, etc.).
+
+## Datadog MCP Usage Notes
+
+Two Datadog MCP servers are typically available — one bound to the EU site
+(`datadoghq.eu`) and one to the US site (`datadoghq.com`). Always run
+region-relevant queries against **both** unless intake clearly localizes the
+incident. The `prod-eu` / `prod-us` env tags live on each side respectively.
+
+- Span search filter pattern:
+  `service:worker resource_name:"process posthog-integration-project" status:error`
+- Log search filter pattern:
+  `service:worker env:prod-eu @langfuse.project.id:cm1r6u… status:error`
+- For high-volume queries, prefer `aggregate_spans` / `aggregate_events`
+  grouped by `(error.message, projectId)` over fetching individual traces.
+- Always link to the Datadog UI for the queries you ran (final section of the
+  output template).
+
+See [`references/datadog-playbook.md`](references/datadog-playbook.md) for the
+full set of starter queries and parameter shapes.
+
+## Output Expectations
+
+From the output template:
+
+- Header: data source, time window, region split (EU vs US table).
+- Hotspots: per-`projectId` (or per-cluster) error counts.
+- Root cause by error class: each cluster gets a short hypothesis with
+  reasoning, distinguishing primary causes from symptoms.
+- Suggested patches: P0/P1/P2 grouped, with concrete file paths and short code
+  sketches. Reference the actual handler in `worker/src/features/**` or
+  `web/src/**`.
+- Dashboards: paste the Datadog query URLs at the end.
+
+Findings come first, recommendations last. If the data is thin, say so
+explicitly and propose what would need to be true to confirm each hypothesis —
+do not invent root causes.
+
+## Cross-References
+
+- Production telemetry query recipes, tenant/public API usage, and queue
+  consumer measurements:
+  [`datadog-query-recipes`](../datadog-query-recipes/SKILL.md)
+- Backend layout, queue contracts, instrumentation patterns:
+  [`backend-dev-guidelines`](../backend-dev-guidelines/SKILL.md)
+- ClickHouse-related findings (memory ceilings, JOIN spills, slow queries):
+  [`clickhouse-best-practices`](../clickhouse-best-practices/SKILL.md)
+- Once a fix is identified and you switch to implementation, hand off to the
+  package `AGENTS.md` for the affected directory.
@@ -0,0 +1,141 @@
+# Datadog Query Playbook
+
+The default for this skill is the **broad sweep**: APM spans + logs + metrics
+ monitors + incidents, run against both EU and US sites unless intake clearly
+localizes. Cluster results before drilling in.
+
+Two MCP servers are typically connected — one for `datadoghq.eu`, one for
+`datadoghq.com`. Run the same query on both and compare. The contrast itself
+is often the most informative finding (e.g. LFE-9475's 23.5% EU vs 0.7% US
+error rate immediately ruled out PostHog Cloud as the global cause).
+
+## Tag Vocabulary
+
+- **Site:** `datadoghq.eu` for EU, `datadoghq.com` for US.
+- **Region tag (`env`):** `prod-eu`, `prod-us`, `prod-hipaa`, `prod-jp`.
+- **Service:** `worker` (default in `worker/src/env.ts`) or `web` (default in
+  `web/src/env.mjs`). Some deployments override to `langfuse`.
+- **Span resource names** for worker async jobs follow the pattern
+  `process <queue-name>` — see `repo-debug-map.md`.
+
+## 1. APM Span Sweep — find the failing handler
+
+Use `aggregate_spans` first; only fetch individual traces once a cluster is
+identified.
+
+Starter shape (rename the resource for the relevant subsystem):
+
+```text
+service:worker resource_name:"process posthog-integration-project" status:error
+```
+
+Aggregations to run, in order:
+
+1. Count by `env` — confirms region split.
+2. Count by `error.message` — primary error classes.
+3. Count by `(projectId, error.message)` — which tenants are affected, by
+   class. `projectId` lives on span tags as `@projectId` for log search and as
+   a tag for spans (depends on instrumentation site).
+4. p50 / p95 / p99 duration by region — fingerprints timeouts vs. crashes vs.
+   slow successes.
+
+If `aggregate_spans` returns no results, check:
+
+- the resource name is right (case-sensitive, see `repo-debug-map.md`);
+- the time window covers when the issue was actually firing;
+- the region tag matches reality (the EU MCP only sees EU traces).
+
+### Public API tenant / legacy-endpoint usage
+
+For tenant-specific public API route usage, use
+[`../../datadog-query-recipes/references/public-api-tenant-usage.md`](../../datadog-query-recipes/references/public-api-tenant-usage.md).
+The key gotcha is that tenant tags usually live on the `api-auth-verify` child
+span while the HTTP route lives on the request root span, so correlate by
+`traceid` rather than relying on one combined span filter.
+
+## 2. Log Sweep — read what the handler said
+
+Logs are the right tool for *messages* the handler emitted. Spans are the
+right tool for *which handler invocations failed*.
+
+Starter shapes:
+
+```text
+service:worker env:prod-eu @projectId:cm1r6u1iq00ccfvrkoy8vg3ms status:error
+service:worker env:prod-eu "[POSTHOG]" status:error
+```
+
+Useful log facets:
+
+- `@projectId` or `@langfuse.project.id` — Langfuse project (cuid).
+- `@error.kind` / `@error.message` / `@error.stack` — when the Winston logger
+  serialized an Error.
+- `@queue` / `@jobName` — when set by BullMQ instrumentation.
+
+For high-volume subsystems (`ingestion-queue`, `otel-ingestion-queue`),
+prefer `analyze_datadog_logs` with grouping over `search_datadog_logs` — the
+raw matches are too noisy.
+
+## 3. Metric Sweep — confirm the trend
+
+Pick 2–3 metrics that match the subsystem. Common ones:
+
+- `trace.bullmq.process.errors` and `trace.bullmq.process.duration` —
+  per-queue health from the BullMQ OTel instrumentation. Filter by
+  `resource_name:"process <queue-name>"`.
+- `trace.http_request.errors` and `trace.http_request.duration` for HTTP
+  handlers (`service:web`).
+- ClickHouse: cluster-level `clickhouse.query.duration`,
+  `clickhouse.memory_usage` — the worker doesn't emit these directly, they
+  come from the `clickhouse` integration in the infra repo.
+- Postgres: `aurora.databaseconnections`, `aurora.deadlocks` — relevant when
+  the symptom is `connection_limit` / `connection pool` errors.
+
+If the subsystem isn't already known, run `search_datadog_metrics` for the
+subsystem name and pick the obvious counter / gauge / histogram triplet.
+
+## 4. Monitors & Incidents
+
+- `search_datadog_monitors` for the subsystem name — tells you what alerts
+  *would* have fired and what their thresholds are. A muted monitor on the
+  affected subsystem is itself a finding (see LFE-9475: "EU alert muted for
+  a week").
+- `search_datadog_incidents` for the time window — links any pre-existing
+  incident the user may not have referenced.
+
+## 5. RUM / Frontend (only when the symptom is user-facing)
+
+Skip unless the issue is "page broken" / "slow load". Then:
+
+- `search_datadog_rum_events` filtered by `@view.url:` patterns matching the
+  affected route.
+- Cross-reference with `service:web` API errors at the same time.
+
+## 6. Trace Drill-Down
+
+Once a cluster is identified, fetch one or two representative traces with
+`get_datadog_trace` to read the actual stack and confirm where in the handler
+the throw originates. This is what lets you point at a specific file and
+line range in the analysis.
+
+## Anti-Patterns
+
+- Don't fetch individual logs/traces before aggregating. You'll burn context
+  on noise and miss the cluster pattern.
+- Don't trust a single-region query as global. Always compare EU and US.
+- Don't read an `error.message` literally if it goes through a custom error
+  wrapper — `validateWebhookURL` rejections, for example, are re-logged as
+  "DNS lookup failed" but are actually validator rejections.
+- Don't assume monitors are firing just because errors exist — check if the
+  monitor is muted.
+
+## Linking Out
+
+End the analysis with the actual Datadog UI URLs you queried, e.g.:
+
+```text
+https://app.datadoghq.eu/apm/traces?query=resource_name%3A%22process+posthog-integration-project%22+status%3Aerror
+https://app.datadoghq.com/apm/traces?query=resource_name%3A%22process+posthog-integration-project%22+status%3Aerror
+```
+
+so the human reader can re-run the same query.
@@ -0,0 +1,72 @@
+# Intake — What Are We Actually Investigating?
+
+Goal: extract every signal from the input *before* you touch Datadog. Cheap
+to do, expensive to skip — the wrong time window or wrong service tag can
+hide the real cause.
+
+## Source-by-Source Recipes
+
+### Linear issue (URL or `LFE-XXXX` ID)
+
+1. Fetch the issue via the Linear MCP:
+   `mcp__d39d26f2-…__get_issue` with `id: "LFE-XXXX"` and
+   `includeRelations: true`.
+2. Fetch comments separately:
+   `mcp__d39d26f2-…__list_comments` with the same `issueId`.
+3. Note: Langfuse triages **inside the issue description**. The description
+   often grows reaction sections (`projectId: <one-line note>`) as oncall
+   investigates. Treat both description and comments as authoritative state.
+4. Pull `attachments` for linked PRs/commits — these show what's already been
+   tried. Reading the diff of a half-merged fix often reveals the original
+   theory of the bug.
+5. Pull labels (`integration-posthog`, `feat-exports`, etc.) — they map
+   directly to the subsystem clusters in `repo-debug-map.md`.
+
+### GitHub issue
+
+1. `gh issue view <url-or-number> --json title,body,labels,comments,assignees`.
+2. If there's a stack trace in the issue body, copy the top frames into your
+   notes — those frames usually map straight to a handler file.
+
+### Pasted error / incident text
+
+1. Treat as the issue description. If it's a stack trace, identify:
+   - the throwing module (handler vs. SDK vs. infra),
+   - whether it looks like a *user-input* failure (HTTP 4xx, validation, auth)
+     or an *infra* failure (5xx, timeout, OOM, DNS, pool exhaustion),
+   - the error class (`Error`, `TypeError`, `PrismaClientKnownRequestError`,
+     etc.).
+2. If a `projectId` appears, that's gold — anchor every Datadog query on it.
+3. If a `traceId` (Datadog's, not Langfuse's) appears, jump straight to
+   `get_datadog_trace`.
+
+## Extract The Following Before Querying
+
+Build a small notes block. If a value is missing, mark it `?` rather than
+guessing — Datadog will tell you what's missing.
+
+- **Subsystem.** PostHog integration, blob-storage export, evaluation
+  execution, OTel ingestion, batch action, webhook delivery, etc. Map to
+  `repo-debug-map.md`.
+- **Region(s).** `prod-eu` / `prod-us` / `prod-hipaa` / `prod-jp`. If unknown,
+  query both EU and US.
+- **Service.** Most issues are `service:worker`; UI/API timeouts are
+  `service:web`. Check for both when unsure.
+- **Time window.** Default to 7 days back from the issue's `createdAt`. If
+  the issue references a specific incident or alert, use that window ±1 day.
+- **`projectId`(s).** Project IDs in Langfuse are `cuid`-shaped
+  (`cl…` / `cm…`, 25 chars). The reaction blocks in Linear descriptions
+  often *are* lists of affected project IDs.
+- **Error message fragments.** Exact substrings to grep for in DD logs:
+  `Header overflow`, `Timeout error.`, `HTTP 403`, `DNS lookup failed`,
+  `Cannot write to canceled buffer`, `connection pool`, etc.
+- **Already-attempted fixes.** Linked PRs/commits on the issue. Read their
+  diffs — your analysis must not re-recommend something that's already
+  shipped.
+
+## Output Of The Intake Step
+
+A short bullet list (not yet formatted as the final analysis) with each of
+the above filled in. The remaining steps key off this — `datadog-playbook.md`
+expects subsystem + region + window, `repo-debug-map.md` expects subsystem,
+and the output template wants the affected projects.
@@ -0,0 +1,127 @@
+# Output Template
+
+The analysis should be structured so it can be pasted directly as the first
+investigative comment on the Linear issue. The example to anchor on is the
+first comment on `LFE-9475` (PostHog Integration Processing Failures).
+
+Findings come first, recommendations last. If the data doesn't support a
+hypothesis, say so — do not invent root causes to fill the template.
+
+## Section Order
+
+1. **Header** — data source, time window, scope of sweep.
+2. **Volume & error-rate split** — table by region (always).
+3. **Hotspots** — table by `(projectId, dominant cause)` or by cluster.
+4. **Root cause by error class** — one numbered subsection per cluster.
+5. **Suggested patches** — P0 / P1 / P2, each with file paths and a short
+   code sketch.
+6. **Dashboards** — Datadog UI URLs for the queries you ran.
+
+## Skeleton (fill in with your findings)
+
+````markdown
+## Datadog APM + log analysis (<N>-day window, <YYYY-MM-DD> → <YYYY-MM-DD>)
+
+Source: APM spans with `resource_name:"process <queue-name>"` across EU and US.
+
+### Volume & error rate — <one-line summary of regional split>
+
+| Region | Total spans | Errors | Error rate |
+|---|---|---|---|
+| EU (`prod-eu`) | <n> | <n> | **<pct>%** |
+| US (`prod-us`) | <n> | <n> | **<pct>%** |
+
+<One sentence explaining where the noise actually lives.>
+
+### Hotspots — concentrated on ~<N> <region> projects
+
+<Region> errors break down by `(projectId, error.message)`:
+
+| ProjectId | Errors | Dominant cause |
+|---|---|---|
+| `<projectId>` | <n> | `<error message>` (<n>) + others |
+| ...           | ... | ...                              |
+
+<Optional: contrast with another subsystem if relevant — e.g.
+"Unlike blob storage, PostHog has multiple distinct root causes — not one
+hotspot pattern.">
+
+## Root cause by error class
+
+### 1. `<error message>` — <n> errors, <n> projects
+<2–4 sentences explaining what this error class actually is at the
+implementation level (which library, which call site). Then list candidate
+causes in order of likelihood. Mark which ones are confirmed by the data
+vs. speculative.>
+
+### 2. `<error message>` — <n> errors, mostly <n> projects
+<Same pattern.>
+
+### 3. <next class>
+<...>
+
+### <N>. <Symptom of upstream failure>
+<Use this slot when a class is a *symptom* of another class rather than an
+independent bug — call it out so suggested patches don't double-count.>
+
+## Suggested patches
+
+### P0 — <one-line summary, e.g. "Auto-disable integrations on persistent
+auth failures">
+<Why this is P0 — what noise it kills, what data it stops corrupting, what
+unblocks downstream work.>
+
+```ts
+// <relative path from repo root>
+// Short code sketch (5–20 lines). It does not need to compile —
+// it must communicate the shape of the change.
+```
+
+### P0 — <next P0>
+<...>
+
+### P1 — <smaller / less urgent fix>
+<Same shape.>
+
+### P2 — <separate-but-surfaced finding>
+<E.g. a Prisma pool sizing issue surfaced incidentally by this analysis but
+not the original bug. Call it out with its own section so it doesn't get
+lost.>
+
+### Regional split explanation (only if relevant)
+<One paragraph explaining why EU vs. US asymmetry exists — usually not an
+infra bug, just where the affected tenants happen to live.>
+
+Dashboards:
+- EU APM: <url>
+- US APM: <url>
+- (logs / metrics / monitor links as relevant)
+````
+
+## Style Rules
+
+- Lead with numbers, not adjectives. "23.5% error rate" beats "very noisy".
+- Distinguish **primary causes** from **symptoms** explicitly. Symptoms
+  shouldn't get their own P0 patch.
+- Always cite specific files when proposing a code change. A patch
+  recommendation without `worker/src/features/<…>/<file>.ts` is unfinished.
+- Code sketches are illustrative — clearly mark them as sketches if they
+  hand-wave types. The next agent / human will write the real diff.
+- If the analysis surfaces a finding *outside* the original ticket scope
+  (e.g. a Prisma pool issue while debugging PostHog), include it as a P2
+  with a sentence explaining it's separate.
+- If the data refuses to converge on a single root cause, say so. The
+  template handles N classes — use as many subsections as the data warrants.
+
+## When To Skip Sections
+
+- **Single-region deployments:** if the issue clearly affects only one
+  region, you can replace the "Volume & error rate" table with a single-row
+  variant, but still note that the other region was checked and clean.
+- **No code change recommended:** if the only finding is "the affected
+  tenants have misconfigured credentials and we should reach out", the
+  Suggested-patches section can be a single sentence — but still include
+  the dashboards.
+- **Aborted investigation:** if Datadog access fails or the data is
+  insufficient, write what you tried, what was missing, and what would let
+  the next investigator pick it up.
@@ -0,0 +1,100 @@
+# Repo Debug Map — Subsystem → Code → Datadog Filters
+
+For each subsystem we ship monitors and incidents on, this is the canonical
+map between the symptom, the Datadog query that surfaces it, and the source
+files where the bug almost certainly lives.
+
+When intake gives you a subsystem (PostHog, evals, exports, etc.), start
+here to pick the right Datadog filters and the right files to read.
+
+## Worker Async Jobs
+
+Worker handlers are wrapped by `instrumentAsync` in their queue file. The
+span resource name follows the pattern `process <queue-name>`. Queue and job
+name constants live in
+`packages/shared/src/server/queues.ts`
+(`QueueName` and `QueueJobs` enums).
+
+| Subsystem | Queue file | Handler dir | Span `resource_name` | Log prefix |
+| --- | --- | --- | --- | --- |
+| PostHog integration | `worker/src/queues/postHogIntegrationQueue.ts` | `worker/src/features/posthog/` | `process posthog-integration-project` | `[POSTHOG]` |
+| Mixpanel integration | `worker/src/queues/mixpanelIntegrationQueue.ts` | `worker/src/features/mixpanel/` | `process mixpanel-integration-project` | `[MIXPANEL]` |
+| Blob storage export | `worker/src/queues/blobStorageIntegrationQueue.ts` | `worker/src/features/blobstorage/` | `process blob-storage-project` | `[BLOBSTORAGE]` |
+| Data retention | `worker/src/queues/dataRetentionQueue.ts` | `worker/src/features/batch-data-retention-cleaner/` | `process data-retention-project` | n/a |
+| Event propagation | `worker/src/queues/eventPropagationQueue.ts` | `worker/src/features/eventPropagation/` | `process event-propagation` | n/a |
+| Cloud usage metering | `worker/src/queues/cloudUsageMeteringQueue.ts` | `worker/src/ee/` (cloud-only) | `process cloud-usage-metering` | n/a |
+| Free-tier usage threshold | `worker/src/queues/cloudFreeTierUsageThresholdQueue.ts` | `worker/src/ee/usageThresholds/` | `process cloud-free-tier-usage-threshold` | n/a |
+| Ingestion (single event) | `worker/src/queues/ingestionQueue.ts` | `worker/src/features/ingestion/` (and `IngestionService`) | BullMQ default span | n/a |
+| OTel ingestion | `worker/src/queues/otelIngestionQueue.ts` | `worker/src/features/otel/` | BullMQ default span | n/a |
+| Evaluation execution | `worker/src/queues/evalQueue.ts` | `worker/src/features/evaluation/` | BullMQ default span | n/a |
+| Batch export | `worker/src/queues/batchExportQueue.ts` | `worker/src/features/batchExport/` | BullMQ default span | n/a |
+| Webhook delivery | `worker/src/queues/webhooks.ts` | `worker/src/features/webhooks/` | BullMQ default span | n/a |
+| Trace / score / dataset / project delete | `worker/src/queues/{traceDelete,scoreDelete,datasetDelete,projectDelete}.ts` | `worker/src/features/traces/`, `…/scores/`, `…/datasets/` | BullMQ default span | n/a |
+
+For queues using BullMQ default spans (no `instrumentAsync` wrapper), search
+APM with `service:worker operation_name:bullmq.process` filtered by
+`bullmq.queue:<queue-name>`. For queue inventory, sharded queue naming, and
+queue metric recipes, use
+[`../../datadog-query-recipes/references/queue-consumers.md`](../../datadog-query-recipes/references/queue-consumers.md).
+
+## Web (Next.js / tRPC / public API)
+
+| Subsystem | Code | Span / log filter |
+| --- | --- | --- |
+| Public REST API | `web/src/pages/api/public/**` | Request span: `service:web resource_name:"GET /api/public/<path>"`; tenant span: `resource_name:api-auth-verify` with `@langfuse.project.id` / `@langfuse.org.id` |
+| tRPC procedures | `web/src/server/api/routers/**` | `service:web resource_name:"POST /api/trpc/<router>.<proc>"` |
+| Auth / API key verification | `web/src/features/public-api/server/apiAuth.ts` | look for `verifyAuthHeaderAndReturnScope` spans |
+| Stripe billing | `web/src/ee/features/billing/server/stripeBillingService.ts` | wrapped in `instrumentAsync`; spans named after the method |
+
+For tenant-specific public API usage questions, first query
+`resource_name:api-auth-verify` by `@langfuse.project.id` or
+`@langfuse.org.id`, then open representative trace IDs and inspect the request
+root span for `http.path_group`, `http.route`, and `http.target`. The tenant
+tags and endpoint path are usually on different spans, so a single-span query
+combining both may return no results even when the trace proves usage.
+For the full reusable recipe, use
+[`../../datadog-query-recipes/references/public-api-tenant-usage.md`](../../datadog-query-recipes/references/public-api-tenant-usage.md).
+
+## Shared Layers
+
+These are not subsystems on their own, but are *frequently the actual cause*
+behind a worker subsystem failure.
+
+| Layer | Location | Common failure modes |
+| --- | --- | --- |
+| ClickHouse access | `packages/shared/src/server/clickhouse/`, `packages/shared/src/server/repositories/` | OOM (`Code: 241`), buffer cancel (`Code: 734`), JOIN spills, slow queries on un-pre-filtered traces |
+| Prisma access | `packages/shared/src/db.ts` and per-feature repos | `connection pool timeout` (worker default `connection_limit=5`), N+1 queries |
+| Queue contracts | `packages/shared/src/server/queues.ts` | wrong queue name, missing schema validation |
+| Logger / instrumentation | `packages/shared/src/server/logger.ts`, `packages/shared/src/server/instrumentation.ts` | log silently dropped because `LANGFUSE_LOG_LEVEL` set wrong, or span missing because handler doesn't call `instrumentAsync` |
+| Webhook URL validation | `packages/shared/src/server/validateWebhookURL.ts` | rejects with messages that *look* like DNS errors but are SSRF guard rejections |
+| Encryption | `packages/shared/encryption` | bad keys → 403/auth-style failures masquerading as upstream errors |
+
+## Common Symptoms → First Files To Read
+
+- **"403 from upstream":** check the per-integration credentials table in
+  Postgres (`PostHogIntegration`, `BlobStorageIntegration`, `WebhookConfig`,
+  etc.) and the encryption layer.
+- **"Timeout":** check the SDK timeout default and the per-stream
+  flush/batch size in the handler. Worker async jobs default to long-running
+  but the upstream SDK does not.
+- **"DNS lookup failed":** distinguish actual DNS from `validateWebhookURL`
+  rejection. The error message wrapping is misleading on purpose.
+- **"Cannot write to canceled buffer" (CH):** ClickHouse stream wasn't
+  aborted when the downstream consumer threw. Look for an `AbortController`
+  threaded through the handler.
+- **"Connection pool timeout" (Prisma):** worker `connection_limit` is set
+  in the connection string; jobs doing per-row `findFirst()` exhaust it.
+  Check whether the integration row could be cached in closure scope.
+- **"memory limit exceeded" (CH):** look for unbounded JOINs without a
+  pre-filter CTE, especially in analytics integrations.
+- **"Header overflow":** Node HTTP parser's default 80 KB ceiling. Either
+  raise `--max-http-header-size` for the worker, or replace the SDK's HTTP
+  client.
+
+## Where to Look for Already-Shipped Fixes
+
+Before recommending a patch, confirm it isn't already merged or in flight:
+
+- `attachments` on the Linear issue (PRs and commits are auto-linked).
+- `git log --oneline --since=<recent-window> -- <handler-path>`.
+- Open PRs touching the file via `gh pr list --search "<filename>"`.
@@ -0,0 +1,171 @@
+---
+name: detect-prod-regressions
+description: |
+  Proactively detect production regressions in Langfuse by comparing recent
+  Datadog errors, error logs, error spans, and API route latency signals against
+  baseline benchmarks or traces across prod-us, prod-eu, prod-hipaa, and
+  prod-jp. Use when asked to sweep production for new bugs, catch regressions
+  early, catch low-occurrence coding bugs or edge cases, compare recent changes
+  to Datadog measurements, or prepare measured production evidence for human
+  review before any optional Linear handoff.
+---
+
+# Detect Prod Regressions
+
+Run this skill as an evidence-first production sweep. The deliverable is a set
+of measured candidate bugs summarized for a human reviewer in a compact table,
+plus a short summary of what was checked. Do not touch Linear automatically.
+
+## Required Scope
+
+Always review all production environments unless the user explicitly narrows the
+scope:
+
+- `prod-us`
+- `prod-eu`
+- `prod-hipaa`
+- `prod-jp`
+
+Query both Datadog sites when needed. Default to the EU site for `prod-eu` and
+the US site for the other production envs, but verify by querying facets or
+running a small count query rather than assuming where a tag lives.
+
+## Measurement Rules
+
+- Ground every bug claim in a measurable signal: counts, rates, p50/p95/p99
+  duration, trace samples, flamegraphs, monitor thresholds, or benchmark
+  comparisons.
+- If a requested measurement is missing or unavailable, write exactly
+  "No measurements found" for that signal.
+- Treat a bug as new only when a recent window shows a new or materially worse
+  error cluster or latency regression versus a baseline.
+- Do not rely only on top-volume clusters. Also inspect new low-occurrence
+  errors that look like coding bugs or edge cases, such as invariant failures,
+  unexpected null/undefined values, validation surprises, unhandled promise
+  rejections, serialization failures, impossible enum states, or one-off 500s
+  on uncommon routes.
+- For latency, error-rate, throughput, saturation, or performance findings,
+  focus on how the signal worsened over time: recent versus preceding window,
+  recent versus same window 7 days earlier, and post-change versus pre-change
+  when deployment markers are available.
+- Prefer high-level aggregations before individual events; drill into example
+  logs, spans, traces, or flamegraphs only after a cluster is identified.
+- Do not infer customer impact, root cause, or severity beyond what the
+  measurements support.
+
+## Baseline Window
+
+Use the user's stated window when provided. Otherwise:
+
+1. Compare the most recent 24 hours with the preceding 24 hours.
+2. Add a same-day or same-hour historical baseline, usually the matching window
+   7 days earlier, when Datadog data is available.
+3. If release or deployment markers, service versions, git SHAs, or change
+   timestamps are visible, compare post-change versus pre-change windows.
+
+Include the recent window and baseline window in any later handoff to
+`linear-bug-triage` after the human explicitly approves sharing findings in
+Linear.
+
+## Datadog Sweep
+
+Before querying Datadog, load the relevant Datadog MCP guidance for logs,
+traces, metrics, and visualizations. Use the existing
+[`datadog-query-recipes`](../datadog-query-recipes/SKILL.md) skill for query
+syntax, production environment coverage, tenant/public API usage, and queue
+consumer measurements. Use
+[`debug-issue-with-datadog`](../debug-issue-with-datadog/SKILL.md) when a
+candidate regression becomes an incident-style root-cause analysis.
+
+For each environment, check:
+
+1. **Datadog errors**
+   - Use Datadog Error Tracking to review error groups, new issues,
+     regressions, affected services, and occurrence timelines. Also check the
+     EU Datadog site equivalent when environment data is stored there.
+   - Aggregate error counts and rates by `env`, `service`, route/resource,
+     status code, `error.type`, and `error.message`.
+   - Compare recent counts/rates to the baseline.
+   - Include low-occurrence new error groups when the stack, message, route, or
+     affected code path suggests a coding bug or edge case, even if occurrence
+     counts are too small for top-N dashboards.
+
+2. **Datadog error logs**
+   - Aggregate `status:error` logs by service, route/resource, source,
+     `error.message`, tenant/project identifiers when present, and environment.
+   - Open representative logs only for clusters with measurable increase.
+
+3. **Datadog error spans**
+   - Aggregate error spans by `env`, `service`, `resource_name`, `http.route`,
+     `error.type`, and `error.message`.
+   - Fetch representative traces only after grouping identifies a candidate
+     regression.
+
+4. **API route latencies**
+   - Focus on `service:web` HTTP spans and metrics such as
+     `trace.http_request.duration` when available.
+   - Compare p50, p95, and p99 by route/resource and environment.
+   - Rank candidates by worsening over time, not only by absolute latency:
+     recent versus baseline deltas, slope, and newly introduced tail latency.
+   - Use trace samples or flamegraphs for routes whose latency materially
+     regressed.
+
+Keep Datadog links for every query, trace, dashboard, or flamegraph used as
+evidence.
+
+## Linear Handoff
+
+Before doing anything in Linear, show the human reviewer a findings table in
+chat and ask for permission to share the findings in Linear. The table should
+include one row per candidate with:
+
+- Candidate / cluster name.
+- Environments.
+- Service and route/resource.
+- Recent window measurement.
+- Baseline measurement.
+- Delta / regression summary.
+- Key evidence links.
+- Recommended Linear action (`comment existing`, `create new`, or `none`).
+
+Use a concrete markdown table shaped like this:
+
+```markdown
+| Candidate | Envs | Service / Route | Recent Window | Baseline | Delta | Evidence | Proposed Linear Action |
+| --- | --- | --- | --- | --- | --- | --- | --- |
+| Public scores latency regression | prod-us | `web` `GET /api/public/v2/scores/index` | p95 `7.44s`, p99 `12.77s`, count `71,448` | p95 `4.34s`, p99 `6.60s`, count `26,024` | p95 `+72%`, p99 `+94%`, volume `+174%` | [metrics](https://app.datadoghq.com/) [spans](https://app.datadoghq.com/) [trace](https://app.datadoghq.com/) | comment existing |
+| ClickHouse socket hang up cluster | prod-us | `web-iso` `POST /` | errors `14,088` | errors `6,026` | `+134%` errors | [spans](https://app.datadoghq.com/) [trace](https://app.datadoghq.com/) | comment existing |
+| Ingestion DLQ monitor noise | prod-eu | `worker-cpu` `langfuse.queue.ingestion.dlq_length` | max `150` | max `150` | flat | [metrics](https://app.datadoghq.eu/) | none |
+```
+
+If a requested signal is unavailable, write `No measurements found` in the
+relevant cell instead of leaving it blank.
+
+Only if the human explicitly approves, use
+[`linear-bug-triage`](../linear-bug-triage/SKILL.md) for Linear search,
+deduplication, evidence comments, Triage issue creation, labels, and ticket
+formatting. Treat that skill as the source of truth for Linear behavior once
+approval is granted.
+
+Hand off:
+
+- Recent window and baseline window.
+- Measured deltas or `No measurements found` for unavailable signals.
+- Affected environments, services, routes/resources, status codes, and top error
+  messages.
+- Datadog links for every query, trace, dashboard, metric graph, or flamegraph
+  used as evidence.
+
+## Final Response
+
+Summarize:
+
+- The windows compared and all prod environments checked.
+- The findings table shown to the human.
+- Whether the human approved sharing findings in Linear.
+- New Linear issues created, with links, only if approval was granted.
+- Existing Linear issues commented on, with links, only if approval was
+  granted.
+- Candidate signals skipped with "No measurements found".
+- A compact "no new bugs found" statement when no issue-worthy regressions were
+  measured.
@@ -0,0 +1,4 @@
+interface:
+  display_name: "Detect Prod Regressions"
+  short_description: "Datadog regression sweep with Linear handoff"
+  default_prompt: "Use $detect-prod-regressions to compare recent production Datadog signals against baselines across all prod environments and hand measured bugs to $linear-bug-triage."
@@ -0,0 +1,116 @@
+---
+name: linear-bug-triage
+description: |
+  Deduplicate measured bug or regression evidence against Linear, then either
+  add evidence comments to related existing issues or create concise Linear bug
+  issues in Triage. Use when Codex has confirmed evidence from Datadog,
+  benchmarks, traces, timings, flamegraphs, logs, or production measurements and
+  needs Linear search, comments, labels, Datadog links, or bug ticket creation
+  without fix suggestions, but only after a human has approved sharing the
+  findings in Linear.
+---
+
+# Linear Bug Triage
+
+Use this skill after a bug or regression candidate has measured evidence. This
+skill owns Linear search, deduplication, evidence comments, and ticket creation;
+the calling skill owns deciding whether the signal is issue-worthy.
+
+## Human Approval Gate
+
+Before doing anything in Linear, first show the findings to the human in a
+compact markdown table and ask for explicit permission to share them in Linear.
+The table should include one row per candidate with:
+
+- Candidate / cluster name.
+- Environments.
+- Service and route/resource.
+- Recent window measurement.
+- Baseline measurement.
+- Delta / regression summary.
+- Key Datadog evidence links.
+- Proposed Linear action (`comment existing`, `create new`, or `none`).
+
+If the human does not explicitly approve, stop after presenting the table. Do
+not search Linear, do not comment on issues, and do not create issues.
+
+If this skill was invoked by `detect-prod-regressions` and that calling skill
+already showed the findings table and obtained explicit human approval for a
+Linear handoff, skip this gate and proceed directly to deduplication.
+
+## Required Evidence
+
+For each candidate, gather:
+
+- Recent window and baseline window as absolute time ranges with timezone.
+- Measured signal: counts, rates, p50/p95/p99 latency, trace samples,
+  flamegraphs, monitor thresholds, or benchmark deltas.
+- Affected environments, services, routes/resources, status codes, and top error
+  messages.
+- Datadog links for logs, spans, traces, metrics, dashboards, or flamegraphs
+  used as evidence.
+- The exact text `No measurements found` for requested measurements that are
+  unavailable.
+
+Do not create or comment based on guesses, unsupported impact claims, or missing
+measurements alone.
+
+## Deduplication
+
+After the human explicitly approves, before creating a new issue:
+
+1. Search Linear for related open issues using exact error text, route/resource,
+   service, environment, monitor name, and Datadog link keywords.
+2. Search recently closed or canceled issues if the error is recurring or the
+   wording is distinctive.
+3. If a related issue exists, add a concise evidence comment instead of creating
+   a duplicate.
+4. If no related issue exists, create one Linear issue in the `Triage` state for
+   each distinct bug cluster.
+
+## Existing Issue Comments
+
+For related existing issues, add only:
+
+- Recent window and baseline window.
+- Measured delta or `No measurements found` for unavailable signals.
+- Affected environments, services, routes/resources, and top error messages.
+- Datadog links.
+
+Do not add fix suggestions, root-cause guesses, implementation notes, owner
+assignments, or next steps.
+
+## New Issue Format
+
+Create new issues with:
+
+- State/status `Triage`; pass the Linear state explicitly on creation and do not
+  rely on workspace defaults.
+- Label `bug`.
+- Additional existing labels that match the evidence, such as affected service,
+  environment, API, ingestion, latency, ClickHouse, Postgres, integrations, or
+  observability labels. Query labels first and use the repository/team's exact
+  label names.
+- Concise title: `bug: <service or route> <measured symptom> in <envs>`.
+- Concise body, evidence-only:
+
+```markdown
+Recent window: <absolute time range and timezone>
+Baseline: <absolute time range and timezone>
+
+Signal:
+- <count/rate/latency delta with env/service/route>
+- <"No measurements found" for missing requested measurements>
+
+Evidence:
+- Datadog logs: <url>
+- Datadog spans/traces: <url>
+- Datadog metrics, dashboard, or latency graph: <url>
+
+Related Linear search:
+- <brief search terms used and result>
+```
+
+Do not include fix suggestions, root-cause guesses, implementation notes, owner
+assignments, or next steps unless the user explicitly asks outside the Linear
+issue or comment.
@@ -0,0 +1,4 @@
+interface:
+  display_name: "Linear Bug Triage"
+  short_description: "Deduplicate and file Linear bug evidence"
+  default_prompt: "Use $linear-bug-triage to deduplicate measured bug evidence and create or comment Linear triage issues."
@@ -31,9 +31,6 @@ pnpm workspace.
     lock refresh / reinstall path before changing `package.json`.
   - If the current parent range does not cover the requested version, upgrade
     the direct parent dependency that pulls the package in.
-   - If a compatible transitive package still stays pinned after the normal
-     refresh path, you may suggest `pnpm dedupe` to the user as an optional
-     manual follow-up, but do not run it automatically and do not require it.
   - Do not add the transitive package directly unless the user explicitly asks.

 4. Ask before changing `minimumReleaseAgeExclude`.
@@ -53,6 +50,9 @@ pnpm workspace.
   - Use the nearest package `AGENTS.md` plus the root verification matrix.
   - Finish with `pnpm why -r <package>`.
   - If companions moved too, run `pnpm why -r <companion-package>` for them as well.
+   - After fixing or upgrading a package, strongly suggest that the user run
+     `pnpm dedupe` as an optional cleanup step, but do not run it automatically
+     and do not require it.

 ## Quick Commands

@@ -64,6 +64,8 @@ pnpm workspace.
  `npm view <parent>@<installedVersion> dependencies peerDependencies optionalDependencies --json`
 - Final graph verification:
  `pnpm why -r <package>`
+- Optional lockfile cleanup for the user to run after the fix:
+  `pnpm dedupe`
 - Bump in the root workspace:
  `pnpm -w up <package>@<version>`
 - Bump in one workspace:
@@ -28,9 +28,12 @@ Use this skill for interactive dependency bumps in Langfuse.
 - If the current parent range does not cover the requested transitive version,
  upgrade that parent dependency instead of adding the target package directly
  unless the user explicitly wants that.
- If a compatible transitive package still stays pinned after the normal
-  refresh path, you may suggest `pnpm dedupe` to the user as an optional manual
-  follow-up, but do not run it automatically and do not require it.
+- Never manually edit `pnpm-lock.yaml`; regenerate lockfile changes with
+  `pnpm` commands only. If a lockfile-only refresh causes unrelated churn,
+  adjust the pnpm command and rerun instead of patching the lockfile by hand.
+- After fixing or upgrading a package, strongly suggest that the user run
+  `pnpm dedupe` as an optional cleanup step, but do not run it automatically
+  and do not require it.
 - Resolve the registry latest version, but do not silently upgrade to latest
  unless the user asked for latest.
 - Compare the target version with the latest version installable under the
@@ -94,13 +94,13 @@ cd packages/utils && bun add lodash

 ```bash
 # pnpm
-pnpm add jest --save-dev --filter=web --filter=@repo/ui
+pnpm add vitest --save-dev --filter=web --filter=@repo/ui

 # npm
-npm install jest --save-dev --workspace=web --workspace=@repo/ui
+npm install vitest --save-dev --workspace=web --workspace=@repo/ui

 # yarn (v2+)
-yarn workspaces foreach -R --from '{web,@repo/ui}' add jest --dev
+yarn workspaces foreach -R --from '{web,@repo/ui}' add vitest --dev
 ```

 ### Internal Packages
@@ -0,0 +1,196 @@
+#####################################################################
+# .env (template) — OCI Object Storage / S3-compatible configuration
+#
+# IMPORTANT SECURITY NOTES (Oracle best practice)
+# - Prefer OCI-native auth (Instance Principal / Workload Identity / Resource Principal)
+#   over static keys.
+# - If you must use static keys, store them in a secure secret manager
+#   (e.g., Kubernetes Secret / OCI Vault) and inject at runtime.
+# - Rotate/revoke any credentials that were previously shared or committed.
+#####################################################################
+
+
+#####################################################################
+# 1) Storage/Auth category (CHOOSE ONE)
+#
+# The app can read/write/download to/from an OCI object store for:
+# - Batch exports (exports/)
+# - Media uploads (media/)
+# - Event uploads (events/)
+#
+# Pick exactly ONE auth mechanism for OCI-native object storage by setting:
+#   LANGFUSE_USE_OCI_NATIVE_OBJECT_STORAGE=true
+#   LANGFUSE_OCI_AUTH_TYPE=<one of the values below>
+#
+# Supported values:
+#   workload_identity | instance_principal | resource_principal | oci_profile | session_token
+#####################################################################
+
+
+#####################################################################
+# Category A — OCI Object Storage with INSTANCE PRINCIPAL (recommended on OCI Compute)
+# Use when:
+# - Running on OCI Compute with IAM set up (dynamic group + policies)
+#
+# Set:
+# LANGFUSE_USE_OCI_NATIVE_OBJECT_STORAGE=true
+# LANGFUSE_OCI_AUTH_TYPE=instance_principal
+#
+# NOTE: Do NOT set *_ACCESS_KEY_ID / *_SECRET_ACCESS_KEY in this category.
+#####################################################################
+
+
+#####################################################################
+# Category B — OCI Object Storage with WORKLOAD IDENTITY (common on OKE)
+# Use when:
+# - Running on OKE with OCI Workload Identity configured
+#
+# Set:
+# LANGFUSE_USE_OCI_NATIVE_OBJECT_STORAGE=true
+# LANGFUSE_OCI_AUTH_TYPE=workload_identity
+#
+# Optional (only if your environment requires additional CA trust):
+# NODE_EXTRA_CA_CERTS=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
+#
+# NOTE: Do NOT set *_ACCESS_KEY_ID / *_SECRET_ACCESS_KEY in this category.
+#####################################################################
+
+
+#####################################################################
+# Category C — OCI Object Storage with RESOURCE PRINCIPAL (common for OCI services)
+# Use when:
+# - Running inside an OCI service/runtime that injects Resource Principal env vars
+#   (e.g., certain managed services / automation contexts)
+#
+# Set:
+# LANGFUSE_USE_OCI_NATIVE_OBJECT_STORAGE=true
+# LANGFUSE_OCI_AUTH_TYPE=resource_principal
+#
+# NOTE: Do NOT set *_ACCESS_KEY_ID / *_SECRET_ACCESS_KEY in this category.
+#####################################################################
+
+
+#####################################################################
+# Category D — OCI Object Storage with OCI CONFIG PROFILE (developer local)
+# Use when:
+# - You have an OCI config file locally or mounted in the runtime
+# - You want to use a named profile
+#
+# Set:
+# LANGFUSE_USE_OCI_NATIVE_OBJECT_STORAGE=true
+# LANGFUSE_OCI_AUTH_TYPE=oci_profile
+# OCI_CONFIG_FILE=/path/to/oci/config
+# OCI_CONFIG_PROFILE=DEFAULT
+#
+# NOTE: Avoid adding config files into images; mount/inject securely.
+#####################################################################
+
+
+#####################################################################
+# Category E — OCI Object Storage with SESSION TOKEN (short-lived user auth)
+# Use when:
+# - You use OCI CLI session authentication (short-lived token flow)
+# - USE oci session authenticate
+# - Appropriate for interactive/dev use; less common for long-running services
+#
+# Set:
+# LANGFUSE_USE_OCI_NATIVE_OBJECT_STORAGE=true
+# LANGFUSE_OCI_AUTH_TYPE=session_token
+# OCI_CONFIG_FILE=/path/to/oci/config
+# OCI_CONFIG_PROFILE=DEFAULT
+#####################################################################
+
+
+#####################################################################
+# Other possible setup — Non-OCI provider (AWS S3 / GCP / Azure / MinIO / etc.)
+# Use when:
+# - Your object storage is NOT OCI Object Storage
+#
+# Set:
+# LANGFUSE_USE_OCI_NATIVE_OBJECT_STORAGE=false
+#
+# Then configure endpoints/regions/credentials for your provider.
+#####################################################################
+
+
+
+#####################################################################
+# 2) Feature: S3 Batch Export
+#
+# Required (when enabled):
+# - *_BUCKET, *_REGION, *_ENDPOINT, *_PREFIX
+# Optional:
+# - *_EXTERNAL_ENDPOINT
+# - *_FORCE_PATH_STYLE=true (needed for many S3-compatible providers like MinIO)
+#
+# Credentials:
+# - Set *_ACCESS_KEY_ID/_SECRET_ACCESS_KEY ONLY for static-key auth
+#   (non-OCI S3-compatible providers)
+#####################################################################
+LANGFUSE_S3_BATCH_EXPORT_ENABLED=true
+LANGFUSE_S3_BATCH_EXPORT_BUCKET=langfuse-bucket
+LANGFUSE_S3_BATCH_EXPORT_PREFIX=exports/
+
+# OCI example region/endpoint:
+LANGFUSE_S3_BATCH_EXPORT_REGION=us-chicago-1
+LANGFUSE_S3_BATCH_EXPORT_ENDPOINT=https://objectstorage.us-chicago-1.oraclecloud.com
+LANGFUSE_S3_BATCH_EXPORT_EXTERNAL_ENDPOINT=https://objectstorage.us-chicago-1.oraclecloud.com
+
+# MinIO / S3-compat setting (safe to keep true for many S3-compatible endpoints)
+LANGFUSE_S3_BATCH_EXPORT_FORCE_PATH_STYLE=true
+
+# Static-key auth (non-OCI). Leave blank/commented for OCI-native auth types above.
+LANGFUSE_S3_BATCH_EXPORT_ACCESS_KEY_ID=__REPLACE_ME__
+LANGFUSE_S3_BATCH_EXPORT_SECRET_ACCESS_KEY=__REPLACE_ME__
+
+
+
+#####################################################################
+# 3) Feature: S3 Media Upload
+#####################################################################
+LANGFUSE_S3_MEDIA_UPLOAD_BUCKET=langfuse-bucket
+LANGFUSE_S3_MEDIA_UPLOAD_PREFIX=media/
+LANGFUSE_S3_MEDIA_UPLOAD_REGION=us-chicago-1
+LANGFUSE_S3_MEDIA_UPLOAD_ENDPOINT=https://objectstorage.us-chicago-1.oraclecloud.com
+LANGFUSE_S3_MEDIA_UPLOAD_FORCE_PATH_STYLE=true
+
+# Static-key auth (non-OCI). Leave blank/commented for OCI-native auth types above.
+LANGFUSE_S3_MEDIA_UPLOAD_ACCESS_KEY_ID=__REPLACE_ME__
+LANGFUSE_S3_MEDIA_UPLOAD_SECRET_ACCESS_KEY=__REPLACE_ME__
+
+
+
+#####################################################################
+# 4) Feature: S3 Event Upload (optional)
+#####################################################################
+LANGFUSE_S3_EVENT_UPLOAD_BUCKET=langfuse-bucket
+LANGFUSE_S3_EVENT_UPLOAD_PREFIX=events/
+LANGFUSE_S3_EVENT_UPLOAD_REGION=us-chicago-1
+LANGFUSE_S3_EVENT_UPLOAD_ENDPOINT=https://objectstorage.us-chicago-1.oraclecloud.com
+LANGFUSE_S3_EVENT_UPLOAD_FORCE_PATH_STYLE=true
+
+# Static-key auth (non-OCI). Leave blank/commented for OCI-native auth types above.
+LANGFUSE_S3_EVENT_UPLOAD_ACCESS_KEY_ID=__REPLACE_ME__
+LANGFUSE_S3_EVENT_UPLOAD_SECRET_ACCESS_KEY=__REPLACE_ME__
+
+
+
+#####################################################################
+# 5) OCI native auth configuration (used by oci_profile / session_token)
+#####################################################################
+# Only required when LANGFUSE_OCI_AUTH_TYPE is: oci_profile OR session_token
+OCI_CONFIG_FILE=__REPLACE_ME__/config
+OCI_CONFIG_PROFILE=DEFAULT
+
+
+
+#####################################################################
+# 6) Troubleshooting notes (comments only)
+#
+# - If you see TLS errors to the endpoint in Kubernetes/OKE, set NODE_EXTRA_CA_CERTS to the
+#   correct CA bundle path for your environment.
+# - If using MinIO or certain S3-compatible providers and you get bucket addressing errors,
+#   set *_FORCE_PATH_STYLE=true.
+# - If downloads work inside the cluster but not externally, configure
+#   LANGFUSE_S3_BATCH_EXPORT_EXTERNAL_ENDPOINT to a publicly reachable endpoint/DNS.
+#####################################################################
@@ -81,6 +81,7 @@ REDIS_CLUSTER_NODES="127.0.0.1:6370,127.0.0.1:6371,127.0.0.1:6372,127.0.0.1:6373
 LANGFUSE_INGESTION_QUEUE_SHARD_COUNT=8
 LANGFUSE_INGESTION_SECONDARY_QUEUE_SHARD_COUNT=8
 LANGFUSE_OTEL_INGESTION_QUEUE_SHARD_COUNT=4
+LANGFUSE_OTEL_INGESTION_SECONDARY_QUEUE_SHARD_COUNT=4
 LANGFUSE_EVAL_EXECUTION_QUEUE_SHARD_COUNT=4
 LANGFUSE_EVAL_EXECUTION_SECONDARY_QUEUE_SHARD_COUNT=4
 LANGFUSE_LLM_AS_JUDGE_EXECUTION_QUEUE_SHARD_COUNT=4
@@ -153,11 +153,13 @@ LANGFUSE_AI_FEATURES_PROJECT_ID=7a88fb47-b4e2-43b8-a06c-a5ce950dc53a
 # Langfuse AI Bedrock credentials
 AWS_ACCESS_KEY_ID="A123456789"
 AWS_SECRET_ACCESS_KEY="SAK123456789"
+LANGFUSE_LLM_CONNECTION_BEDROCK_API_KEY="1234567890abcdef"
 LANGFUSE_AWS_BEDROCK_REGION="eu-west-1"
 LANGFUSE_AWS_BEDROCK_MODEL="eu.anthropic.claude-3-haiku-20240307-v1:0"

 # Events table migration
 LANGFUSE_ENABLE_EVENTS_TABLE_OBSERVATIONS=true
+LANGFUSE_ENABLE_EVENTS_TABLE_UI=true
 LANGFUSE_ENABLE_EVENTS_TABLE_FLAGS=true
 LANGFUSE_ENABLE_EVENTS_TABLE_V2_APIS=true
 LANGFUSE_EXPERIMENT_INSERT_INTO_EVENTS_TABLE=true
@@ -69,6 +69,7 @@ OTEL_SERVICE_NAME="langfuse"
 # AUTH_GOOGLE_ALLOWED_DOMAINS=langfuse.com,google.com # optional allowlist of workspace domains that can sign in via Google
 # AUTH_GOOGLE_CLIENT_AUTH_METHOD=
 # AUTH_GOOGLE_CHECKS=
+# AUTH_GOOGLE_ID_TOKEN_SIGNED_RESPONSE_ALG=
 # AUTH_GITHUB_CLIENT_ID=
 # AUTH_GITHUB_CLIENT_SECRET=
 # AUTH_GITHUB_ALLOW_ACCOUNT_LINKING=false
@@ -86,6 +87,7 @@ OTEL_SERVICE_NAME="langfuse"
 # AUTH_GITLAB_ISSUER=
 # AUTH_GITLAB_CLIENT_AUTH_METHOD=
 # AUTH_GITLAB_CHECKS=
+# AUTH_GITLAB_ID_TOKEN_SIGNED_RESPONSE_ALG=
 # AUTH_GITLAB_URL=
 # AUTH_AZURE_AD_CLIENT_ID=
 # AUTH_AZURE_AD_CLIENT_SECRET=
@@ -93,30 +95,35 @@ OTEL_SERVICE_NAME="langfuse"
 # AUTH_AZURE_AD_ALLOW_ACCOUNT_LINKING=false
 # AUTH_AZURE_AD_CLIENT_AUTH_METHOD=
 # AUTH_AZURE_AD_CHECKS=
+# AUTH_AZURE_AD_ID_TOKEN_SIGNED_RESPONSE_ALG=
 # AUTH_OKTA_CLIENT_ID=
 # AUTH_OKTA_CLIENT_SECRET=
 # AUTH_OKTA_ISSUER=
 # AUTH_OKTA_ALLOW_ACCOUNT_LINKING=false
 # AUTH_OKTA_CLIENT_AUTH_METHOD=
 # AUTH_OKTA_CHECKS=
+# AUTH_OKTA_ID_TOKEN_SIGNED_RESPONSE_ALG=
 # AUTH_AUTH0_CLIENT_ID=
 # AUTH_AUTH0_CLIENT_SECRET=
 # AUTH_AUTH0_ISSUER=
 # AUTH_AUTH0_ALLOW_ACCOUNT_LINKING=false
 # AUTH_AUTH0_CLIENT_AUTH_METHOD=
 # AUTH_AUTH0_CHECKS=
+# AUTH_AUTH0_ID_TOKEN_SIGNED_RESPONSE_ALG=
 # AUTH_COGNITO_CLIENT_ID=
 # AUTH_COGNITO_CLIENT_SECRET=
 # AUTH_COGNITO_ISSUER=
 # AUTH_COGNITO_ALLOW_ACCOUNT_LINKING=false
 # AUTH_COGNITO_CLIENT_AUTH_METHOD=
 # AUTH_COGNITO_CHECKS=
+# AUTH_COGNITO_ID_TOKEN_SIGNED_RESPONSE_ALG=
 # AUTH_KEYCLOAK_CLIENT_ID=
 # AUTH_KEYCLOAK_CLIENT_SECRET=
 # AUTH_KEYCLOAK_ISSUER=
 # AUTH_KEYCLOAK_ALLOW_ACCOUNT_LINKING=false
 # AUTH_KEYCLOAK_CLIENT_AUTH_METHOD=
 # AUTH_KEYCLOAK_CHECKS=
+# AUTH_KEYCLOAK_ID_TOKEN_SIGNED_RESPONSE_ALG=
 # AUTH_KEYCLOAK_NAME=
 # AUTH_WORKOS_CLIENT_ID=
 # AUTH_WORKOS_CLIENT_SECRET=
@@ -133,12 +140,14 @@ OTEL_SERVICE_NAME="langfuse"
 # AUTH_CUSTOM_ID_TOKEN=false # optional, default is true
 # AUTH_CUSTOM_CLIENT_AUTH_METHOD=
 # AUTH_CUSTOM_CHECKS=
+# AUTH_CUSTOM_ID_TOKEN_SIGNED_RESPONSE_ALG=
 # AUTH_JUMPCLOUD_CLIENT_ID=
 # AUTH_JUMPCLOUD_CLIENT_SECRET=
 # AUTH_JUMPCLOUD_ISSUER=
 # AUTH_JUMPCLOUD_ALLOW_ACCOUNT_LINKING=
 # AUTH_JUMPCLOUD_CLIENT_AUTH_METHOD=
 # AUTH_JUMPCLOUD_CHECKS=
+# AUTH_JUMPCLOUD_ID_TOKEN_SIGNED_RESPONSE_ALG=
 # AUTH_JUMPCLOUD_SCOPE=

 # Transactional email, optional
@@ -15,10 +15,10 @@ Fixes # (issue)
 <!-- Please delete bullets that are not relevant. -->

 - [ ] Bug fix (non-breaking change which fixes an issue)
- [ ] Chore (refactoring code, technical debt, workflow improvements)
+- [ ] Chore (tooling, dependencies, CI, workflows, repo upkeep, or other maintenance work)
 - [ ] New feature (non-breaking change which adds functionality)
 - [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
- [ ] Refactor (does not change functionality, e.g. code style improvements, linting)
+- [ ] Refactor (restructures existing code without changing behavior, e.g. simplify logic, split modules, reduce duplication)
 - [ ] This change requires a documentation update

 ## Mandatory Tasks
@@ -12,7 +12,7 @@ updates:
    schedule:
      interval: "daily"
    cooldown:
-      default-days: 5
+      default-days: 7
    versioning-strategy: "increase"
    commit-message:
      prefix: chore
@@ -54,6 +54,8 @@ updates:
    directory: "/"
    schedule:
      interval: "weekly"
+    cooldown:
+      default-days: 8
    commit-message:
      prefix: ci
      include: scope
@@ -10,10 +10,21 @@ on:
        type: string
        description: Name of the service to be deployed, e.g. web-ingestion, web, or worker.
        required: true
+    # Environment secrets don't auto-resolve in reusable workflows.
+    # See: https://github.com/actions/runner/issues/3206
+    secrets:
+      AWS_ACCESS_KEY_ID:
+        required: true
+      AWS_SECRET_ACCESS_KEY:
+        required: true
+      SENTRY_AUTH_TOKEN:
+        required: false
 jobs:
  ecs-deploy:
    runs-on: blacksmith-4vcpu-ubuntu-2404
    environment: ${{ inputs.environment }}
+    permissions:
+      contents: read
    steps:
      - name: Get app name
        uses: winterjung/split@a211a1c46e35fcdc4097d59dd6282d4a9859651b # v2
@@ -22,52 +33,68 @@ jobs:
          msg: ${{ inputs.service }}
          separator: "-"
      - name: Checkout code
-        uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
+        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        with:
+          persist-credentials: false
      - name: Authenticate with AWS
        # GitHub/AWS recommend to use OIDC here: https://github.com/aws-actions/configure-aws-credentials?tab=readme-ov-file#oidc
        # Probably more painful to configure, but would remove all long-lived credentials.
-        uses: aws-actions/configure-aws-credentials@7474bc4690e29a8392af63c5b98e7449536d5c3a # v4
+        uses: aws-actions/configure-aws-credentials@ec61189d14ec14c8efccab744f656cffd0e33f37 # v6.1.0
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: ${{ vars.AWS_REGION }}
      - name: Login to AWS ECR
        id: login-ecr
-        uses: aws-actions/amazon-ecr-login@183a1442edf41672e66566b7fc560e297a290896 # v2
+        uses: aws-actions/amazon-ecr-login@f2e9fc6c2b355c1890b65e6f6f0e2ac3e6e22f78 # v2.1.2
      - name: Build, tag, and push Docker image
        env:
          REGISTRY: ${{ steps.login-ecr.outputs.registry }}
          REPOSITORY: ${{ steps.split.outputs._0 }}
          IMAGE_TAG: ${{ github.sha }}
+          STEPS_SPLIT_OUTPUTS__0: ${{ steps.split.outputs._0 }}
+          VARS_NEXT_PUBLIC_LANGFUSE_CLOUD_REGION: ${{ vars.NEXT_PUBLIC_LANGFUSE_CLOUD_REGION }}
+          VARS_NEXT_LANGFUSE_TRACING_SAMPLE_RATE: ${{ vars.NEXT_LANGFUSE_TRACING_SAMPLE_RATE }}
+          VARS_NEXT_PUBLIC_SENTRY_ENVIRONMENT: ${{ vars.NEXT_PUBLIC_SENTRY_ENVIRONMENT }}
+          VARS_NEXT_PUBLIC_DEMO_ORG_ID: ${{ vars.NEXT_PUBLIC_DEMO_ORG_ID }}
+          VARS_NEXT_PUBLIC_DEMO_PROJECT_ID: ${{ vars.NEXT_PUBLIC_DEMO_PROJECT_ID }}
+          VARS_NEXT_PUBLIC_SENTRY_DSN: ${{ vars.NEXT_PUBLIC_SENTRY_DSN }}
+          VARS_NEXT_PUBLIC_POSTHOG_KEY: ${{ vars.NEXT_PUBLIC_POSTHOG_KEY }}
+          VARS_NEXT_PUBLIC_POSTHOG_HOST: ${{ vars.NEXT_PUBLIC_POSTHOG_HOST }}
+          VARS_NEXT_PUBLIC_PLAIN_APP_ID: ${{ vars.NEXT_PUBLIC_PLAIN_APP_ID }}
+          VARS_SENTRY_ORG: ${{ vars.SENTRY_ORG }}
+          VARS_SENTRY_PROJECT: ${{ vars.SENTRY_PROJECT }}
+          VARS_NEXT_PUBLIC_LANGFUSE_TRACING_SAMPLE_RATE: ${{ vars.NEXT_PUBLIC_LANGFUSE_TRACING_SAMPLE_RATE }}
+          SECRETS_SENTRY_AUTH_TOKEN: ${{ secrets.SENTRY_AUTH_TOKEN }}
        run: |
          docker build \
            -t $REGISTRY/$REPOSITORY:$IMAGE_TAG \
-            -f ./${{ steps.split.outputs._0 }}/Dockerfile \
-            --build-arg NEXT_PUBLIC_LANGFUSE_CLOUD_REGION=${{ vars.NEXT_PUBLIC_LANGFUSE_CLOUD_REGION }} \
-            --build-arg NEXT_LANGFUSE_TRACING_SAMPLE_RATE=${{ vars.NEXT_LANGFUSE_TRACING_SAMPLE_RATE }} \
-            --build-arg NEXT_PUBLIC_SENTRY_ENVIRONMENT=${{ vars.NEXT_PUBLIC_SENTRY_ENVIRONMENT }} \
-            --build-arg NEXT_PUBLIC_DEMO_ORG_ID=${{ vars.NEXT_PUBLIC_DEMO_ORG_ID }} \
-            --build-arg NEXT_PUBLIC_DEMO_PROJECT_ID=${{ vars.NEXT_PUBLIC_DEMO_PROJECT_ID }} \
-            --build-arg NEXT_PUBLIC_SENTRY_DSN=${{ vars.NEXT_PUBLIC_SENTRY_DSN }} \
-            --build-arg NEXT_PUBLIC_BUILD_ID=${{ github.sha }} \
-            --build-arg NEXT_PUBLIC_POSTHOG_KEY=${{ vars.NEXT_PUBLIC_POSTHOG_KEY }} \
-            --build-arg NEXT_PUBLIC_POSTHOG_HOST=${{ vars.NEXT_PUBLIC_POSTHOG_HOST }} \
-            --build-arg NEXT_PUBLIC_PLAIN_APP_ID=${{ vars.NEXT_PUBLIC_PLAIN_APP_ID }} \
-            --build-arg SENTRY_AUTH_TOKEN=${{ secrets.SENTRY_AUTH_TOKEN }} \
-            --build-arg SENTRY_ORG=${{ vars.SENTRY_ORG }} \
-            --build-arg SENTRY_PROJECT=${{ vars.SENTRY_PROJECT }} \
-            --build-arg NEXT_PUBLIC_LANGFUSE_TRACING_SAMPLE_RATE=${{ vars.NEXT_PUBLIC_LANGFUSE_TRACING_SAMPLE_RATE }} \
+            -f ./${STEPS_SPLIT_OUTPUTS__0}/Dockerfile \
+            --build-arg NEXT_PUBLIC_LANGFUSE_CLOUD_REGION=${VARS_NEXT_PUBLIC_LANGFUSE_CLOUD_REGION} \
+            --build-arg NEXT_LANGFUSE_TRACING_SAMPLE_RATE=${VARS_NEXT_LANGFUSE_TRACING_SAMPLE_RATE} \
+            --build-arg NEXT_PUBLIC_SENTRY_ENVIRONMENT=${VARS_NEXT_PUBLIC_SENTRY_ENVIRONMENT} \
+            --build-arg NEXT_PUBLIC_DEMO_ORG_ID=${VARS_NEXT_PUBLIC_DEMO_ORG_ID} \
+            --build-arg NEXT_PUBLIC_DEMO_PROJECT_ID=${VARS_NEXT_PUBLIC_DEMO_PROJECT_ID} \
+            --build-arg NEXT_PUBLIC_SENTRY_DSN=${VARS_NEXT_PUBLIC_SENTRY_DSN} \
+            --build-arg NEXT_PUBLIC_BUILD_ID=${IMAGE_TAG} \
+            --build-arg NEXT_PUBLIC_POSTHOG_KEY=${VARS_NEXT_PUBLIC_POSTHOG_KEY} \
+            --build-arg NEXT_PUBLIC_POSTHOG_HOST=${VARS_NEXT_PUBLIC_POSTHOG_HOST} \
+            --build-arg NEXT_PUBLIC_PLAIN_APP_ID=${VARS_NEXT_PUBLIC_PLAIN_APP_ID} \
+            --build-arg SENTRY_AUTH_TOKEN=${SECRETS_SENTRY_AUTH_TOKEN} \
+            --build-arg SENTRY_ORG=${VARS_SENTRY_ORG} \
+            --build-arg SENTRY_PROJECT=${VARS_SENTRY_PROJECT} \
+            --build-arg NEXT_PUBLIC_LANGFUSE_TRACING_SAMPLE_RATE=${VARS_NEXT_PUBLIC_LANGFUSE_TRACING_SAMPLE_RATE} \
            .
          docker push $REGISTRY/$REPOSITORY:$IMAGE_TAG
      - name: Render AWS ECS Task Definition
        id: render-task-definition
-        uses: aws-actions/amazon-ecs-render-task-definition@77954e213ba1f9f9cb016b86a1d4f6fcdea0d57e # v1
+        uses: aws-actions/amazon-ecs-render-task-definition@77954e213ba1f9f9cb016b86a1d4f6fcdea0d57e # v1.8.4
        with:
          container-name: ${{ inputs.service }}
          image: ${{ steps.login-ecr.outputs.registry }}/${{ steps.split.outputs._0 }}:${{ github.sha }}
          task-definition-family: ${{ inputs.environment }}-${{ inputs.service }}
      - name: Update AWS ECS Service
-        uses: aws-actions/amazon-ecs-deploy-task-definition@fc8fc60f3a60ffd500fcb13b209c59d221ac8c8c # v2
+        uses: aws-actions/amazon-ecs-deploy-task-definition@fc8fc60f3a60ffd500fcb13b209c59d221ac8c8c # v2.6.1
        with:
          task-definition: ${{ steps.render-task-definition.outputs.task-definition }}
          service: ${{ inputs.environment }}-${{ inputs.service }}
@@ -9,6 +9,8 @@ on:
  issue_comment:
    types: [created]

+permissions: {}
+
 jobs:
  retrigger_cla:
    # Only run on PR comments (not issue comments) with the /check-cla command
@@ -0,0 +1,70 @@
+name: Security review
+
+on:
+  pull_request:
+    types:
+      - opened
+      - reopened
+      - synchronize
+      - ready_for_review
+
+permissions:
+  contents: read
+  pull-requests: write
+
+jobs:
+  security-review:
+    name: Security review
+    if: github.event.pull_request.draft == false && github.event.pull_request.head.repo.full_name == github.repository
+    runs-on: ubuntu-latest
+    timeout-minutes: 30
+
+    steps:
+      - name: Checkout
+        uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
+        with:
+          ref: ${{ github.event.pull_request.head.sha }}
+          fetch-depth: 2
+          persist-credentials: false
+
+      - name: Checkout Claude Code security review action
+        uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
+        with:
+          repository: anthropics/claude-code-security-review
+          ref: 0c6a49f1fa56a1d472575da86a94dbc1edb78eda
+          path: .github/actions/claude-code-security-review
+          fetch-depth: 1
+          persist-credentials: false
+
+      - name: Patch Claude API validation model
+        shell: bash
+        run: |
+          python <<'PY'
+          import os
+          from pathlib import Path
+
+          action_path = (
+              Path(os.environ["GITHUB_WORKSPACE"])
+              / ".github"
+              / "actions"
+              / "claude-code-security-review"
+              / "claudecode"
+              / "claude_api_client.py"
+          )
+          if not action_path.exists():
+              raise SystemExit(f"Expected Claude API client not found at {action_path}")
+          source = action_path.read_text()
+          old = 'model="claude-3-5-haiku-20241022",'
+          new = "model=self.model,"
+          if old not in source:
+              raise SystemExit(f"Expected validation model not found in {action_path}")
+          action_path.write_text(source.replace(old, new))
+          PY
+
+      - name: Run Claude Code security review
+        uses: ./.github/actions/claude-code-security-review
+        with:
+          claude-api-key: ${{ secrets.CLAUDE_API_KEY }}
+          claude-model: claude-opus-4-1-20250805
+          comment-pr: true
+          run-every-commit: true
@@ -1,14 +1,15 @@
 name: Claude Review on Maintainer PRs

 on:
-  pull_request_target:
+  pull_request:
    types:
      - opened
      - ready_for_review

 jobs:
  comment:
-    if: github.event.pull_request.draft == false
+    # Only run on PRs that are not drafts and are from the same repository (i.e., not from forks)
+    if: github.event.pull_request.draft == false && github.event.pull_request.head.repo.full_name == github.repository
    runs-on: ubuntu-latest
    permissions:
      issues: write
@@ -16,7 +17,7 @@ jobs:
    steps:
      - name: Check author permission and existing review request
        id: check
-        uses: actions/github-script@f28e40c7f34bde8b3046d885e986cb6290c5673b # v7
+        uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
        with:
          script: |
            const owner = context.repo.owner;
@@ -57,7 +58,7 @@ jobs:

      - name: Add Claude review comment
        if: steps.check.outputs.should_comment == 'true'
-        uses: actions/github-script@f28e40c7f34bde8b3046d885e986cb6290c5673b # v7
+        uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
        with:
          script: |
            await github.rest.issues.createComment({
@@ -55,11 +55,13 @@ jobs:
        # your codebase is analyzed, see https://docs.github.com/en/code-security/code-scanning/creating-an-advanced-setup-for-code-scanning/codeql-code-scanning-for-compiled-languages
    steps:
      - name: Checkout repository
-        uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
+        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        with:
+          persist-credentials: false

      # Initializes the CodeQL tools for scanning.
      - name: Initialize CodeQL
-        uses: github/codeql-action/init@5c8a8a642e79153f5d047b10ec1cba1d1cc65699 # v3
+        uses: github/codeql-action/init@c10b8064de6f491fea524254123dbe5e09572f13 # v4.35.1
        with:
          languages: ${{ matrix.language }}
          build-mode: ${{ matrix.build-mode }}
@@ -87,6 +89,6 @@ jobs:
          exit 1

      - name: Perform CodeQL Analysis
-        uses: github/codeql-action/analyze@5c8a8a642e79153f5d047b10ec1cba1d1cc65699 # v3
+        uses: github/codeql-action/analyze@c10b8064de6f491fea524254123dbe5e09572f13 # v4.35.1
        with:
          category: "/language:${{matrix.language}}"
@@ -22,6 +22,8 @@ jobs:

    steps:
      - name: Checkout
-        uses: actions/checkout@f43a0e5ff2bd294095638e18286ca9a3d1956744 # v3
+        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        with:
+          persist-credentials: false
      - name: Codespell
-        uses: codespell-project/actions-codespell@406322ec52dd7b488e48c1c4b82e2a8b3a1bf630 # v2
+        uses: codespell-project/actions-codespell@8f01853be192eb0f849a5c7d721450e7a467c579 # v2.2
@@ -6,6 +6,8 @@ on:
      - main
  workflow_dispatch:

+permissions: {}
+
 jobs:
  rebase-dependabot:
    runs-on: ubuntu-latest
@@ -27,6 +27,8 @@ on:
          - prod-jp
        required: true

+permissions: {}
+
 concurrency:
  # Support concurrent `push` and `workflow_dispatch`` actions
  group: deploy-${{ github.event_name }}-${{ github.ref }}
@@ -41,7 +43,7 @@ jobs:
    steps:
      - name: Get affected services
        id: affected-services
-        uses: actions/github-script@f28e40c7f34bde8b3046d885e986cb6290c5673b # v7
+        uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
        with:
          script: |
            if (context.eventName === "workflow_dispatch") {
@@ -56,7 +58,7 @@ jobs:
            return "[]"
          result-encoding: string
      - name: Print services to build
-        uses: actions/github-script@f28e40c7f34bde8b3046d885e986cb6290c5673b # v7
+        uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
        env:
          services: ${{ steps.affected-services.outputs.result }}
        with:
@@ -71,7 +73,7 @@ jobs:
    steps:
      - name: Get affected environments
        id: affected-environments
-        uses: actions/github-script@f28e40c7f34bde8b3046d885e986cb6290c5673b # v7
+        uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
        with:
          script: |
            if (context.eventName === "workflow_dispatch") {
@@ -88,7 +90,7 @@ jobs:
            return "[]"
          result-encoding: string
      - name: Print environments to build
-        uses: actions/github-script@f28e40c7f34bde8b3046d885e986cb6290c5673b # v7
+        uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
        env:
          environments: ${{ steps.affected-environments.outputs.result }}
        with:
@@ -99,7 +101,14 @@ jobs:
  ecs-deploy:
    uses: ./.github/workflows/_deploy_ecs_service.yml
    needs: [affected-services, affected-environments]
-    secrets: inherit
+    permissions:
+      contents: read
+    # Environment secrets must be passed explicitly to reusable workflows.
+    # See: https://github.com/actions/runner/issues/3206
+    secrets:
+      AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
+      AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
+      SENTRY_AUTH_TOKEN: ${{ secrets.SENTRY_AUTH_TOKEN }}
    strategy:
      matrix:
        service: ${{ fromJson(needs.affected-services.outputs.services) }}
@@ -9,15 +9,21 @@ on:
    branches:
      - "main"

+permissions:
+  contents: read
+  checks: write # Needed to create a check run for the license compliance check results
+
 jobs:
  license_check:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
-        uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
+        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        with:
+          persist-credentials: false

      - name: Setup node
-        uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4
+        uses: actions/setup-node@53b83947a5a98c8d113130e565377fae1a50d02f # v6.3.0
        with:
          node-version: 18

@@ -32,7 +38,7 @@ jobs:

      - name: Check license-checker CSV file without headers
        id: license_check_report
-        uses: pilosus/action-pip-license-checker@cc7a461bfa27b44ad187b8578c881ef5138c13fd # v2
+        uses: pilosus/action-pip-license-checker@e909b0226ff49d3235c99c4585bc617f49fff16a # v3.1.0
        with:
          external: "npm-license-checker.csv"
          external-format: "csv"
@@ -44,6 +50,8 @@ jobs:
          github-token: ${{ secrets.GITHUB_TOKEN }}
      - name: Echo error
        if: failure()
-        run: echo "::error::${{ steps.license_check_report.outputs.report }}"
+        run: echo "::error::${STEPS_LICENSE_CHECK_REPORT_OUTPUTS_REPORT}"
+        env:
+          STEPS_LICENSE_CHECK_REPORT_OUTPUTS_REPORT: ${{ steps.license_check_report.outputs.report }}
      - name: Delete license-checker CSV file
        run: rm npm-license-checker.csv
@@ -12,10 +12,28 @@ on:
    branches:
      - "**"

+permissions:
+  contents: read
+
 concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: ${{ github.event_name == 'pull_request' }}

+env:
+  # Keep the test job node-version matrices in sync. GitHub does not allow the
+  # env context in jobs.<job_id>.name, so matrix jobs use matrix.node-version
+  # for their display names.
+  NODE_VERSION: 24
+
+  # Disable CI cache restores for fork PRs.
+  #
+  # Fork PRs can only read base-branch caches, and CI cache paths must not
+  # contain secrets. This is conservative defense-in-depth, not required for
+  # release cache integrity: PR-created caches are scoped to the PR merge ref,
+  # main does not restore them, and release image builds intentionally do not
+  # restore CI caches.
+  CI_CACHE_ALLOWED: ${{ github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository }}
+
 jobs:
  pre-job:
    runs-on: blacksmith-4vcpu-ubuntu-2404
@@ -24,19 +42,44 @@ jobs:
      llm_connections_changed: ${{ steps.filter.outputs.llm_connections }}
    timeout-minutes: 15
    permissions:
+      contents: read
      pull-requests: read
    steps:
+      # Replaces fkirc/skip-duplicate-actions — skips runs whose git tree
+      # was already tested in a prior successful run of this workflow.
      - id: skip_check
-        uses: fkirc/skip-duplicate-actions@f75f66ce1886f00957d99748a42c724f4330bdcf # v5
-        with:
-          do_not_skip: '["workflow_dispatch"]'
-      - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
+        env:
+          EVENT_NAME: ${{ github.event_name }}
+          GH_TOKEN: ${{ github.token }}
+          REPOSITORY: ${{ github.repository }}
+          COMMIT_SHA: ${{ github.sha }}
+          RUN_ID: ${{ github.run_id }}
+        run: |
+          if [[ "$EVENT_NAME" == "workflow_dispatch" ]]; then
+            echo "should_skip=false" >> "$GITHUB_OUTPUT"
+            exit 0
+          fi
+
+          CURRENT_TREE=$(gh api "repos/$REPOSITORY/git/commits/$COMMIT_SHA" --jq '.tree.sha')
+
+          MATCH=$(gh api "repos/$REPOSITORY/actions/workflows/pipeline.yml/runs?status=success&per_page=20" \
+            | jq -r --argjson run_id "$RUN_ID" --arg current_tree "$CURRENT_TREE" \
+              '[.workflow_runs[] | select(.id != $run_id and .head_commit.tree_id == $current_tree)] | first | .head_sha // empty')
+
+          if [[ -n "$MATCH" ]]; then
+            echo "::notice::Tree $CURRENT_TREE already tested in a prior successful run (commit $MATCH) — skipping"
+            echo "should_skip=true" >> "$GITHUB_OUTPUT"
+          else
+            echo "should_skip=false" >> "$GITHUB_OUTPUT"
+          fi
+      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
        with:
          # Recommended for paths-filter action
          # may save additional git fetch roundtrip if
          # merge-base is found within latest N commits
          fetch-depth: 20
-      - uses: dorny/paths-filter@de90cc6fb38fc0963ad72b210f1f284cd68cea36 # v3
+          persist-credentials: false
+      - uses: dorny/paths-filter@fbd0ab8f3e69293af611ebaee6363fc25e6d187d # v4.0.1
        id: filter
        with:
          filters: |
@@ -50,17 +93,28 @@ jobs:
      - pre-job
    if: needs.pre-job.outputs.should_skip != 'true'
    steps:
-      - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
-      - uses: pnpm/action-setup@a3252b78c470c02df07e9d59298aecedc3ccdd6d # v3
+      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        with:
+          persist-credentials: false
+      - uses: pnpm/action-setup@fc06bc1257f339d1d5d8b3a19a8cae5388b55320 # v5.0.0
        with:
          version: 10.33.0
-      - uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4
+      - name: Use Node.js ${{ env.NODE_VERSION }} without cache
+        if: env.CI_CACHE_ALLOWED != 'true'
+        uses: actions/setup-node@53b83947a5a98c8d113130e565377fae1a50d02f # v6.3.0
        with:
-          node-version: 24
+          node-version: ${{ env.NODE_VERSION }}
+          package-manager-cache: false
+      - name: Use Node.js ${{ env.NODE_VERSION }} with pnpm cache
+        if: env.CI_CACHE_ALLOWED == 'true'
+        uses: actions/setup-node@53b83947a5a98c8d113130e565377fae1a50d02f # v6.3.0 # zizmor: ignore[cache-poisoning] lint job dependency cache only; no released artifacts are built or published from this cached state
+        with:
+          node-version: ${{ env.NODE_VERSION }}
          cache: "pnpm"
          cache-dependency-path: "pnpm-lock.yaml"
      - name: Setup Turbo cache
-        uses: actions/cache@0057852bfaa89a56745cba8c7296529d2fc39830 # v4
+        if: env.CI_CACHE_ALLOWED == 'true'
+        uses: actions/cache@668228422ae6a00e4ad889ee87cd7109ec5666a7 # v5.0.4 # zizmor: ignore[cache-poisoning] lint cache only; publish jobs rebuild artifacts and do not restore this cache
        with:
          path: .turbo
          key: ${{ runner.os }}-turbo-lint-${{ github.sha }}
@@ -75,6 +129,8 @@ jobs:
          cp .env.dev.example .env
      - name: lint web
        run: pnpm run lint
+      - name: typecheck
+        run: pnpm run typecheck

  prettier-check:
    runs-on: blacksmith-4vcpu-ubuntu-2404
@@ -82,15 +138,24 @@ jobs:
      - pre-job
    if: needs.pre-job.outputs.should_skip != 'true'
    steps:
-      - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
+      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
        with:
          fetch-depth: 0
-      - uses: pnpm/action-setup@a3252b78c470c02df07e9d59298aecedc3ccdd6d # v3
+          persist-credentials: false
+      - uses: pnpm/action-setup@fc06bc1257f339d1d5d8b3a19a8cae5388b55320 # v5.0.0
        with:
          version: 10.33.0
-      - uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4
+      - name: Use Node.js ${{ env.NODE_VERSION }} without cache
+        if: env.CI_CACHE_ALLOWED != 'true'
+        uses: actions/setup-node@53b83947a5a98c8d113130e565377fae1a50d02f # v6.3.0
        with:
-          node-version: 24
+          node-version: ${{ env.NODE_VERSION }}
+          package-manager-cache: false
+      - name: Use Node.js ${{ env.NODE_VERSION }} with pnpm cache
+        if: env.CI_CACHE_ALLOWED == 'true'
+        uses: actions/setup-node@53b83947a5a98c8d113130e565377fae1a50d02f # v6.3.0 # zizmor: ignore[cache-poisoning] prettier check dependency cache only; no released artifacts are built or published from this cached state
+        with:
+          node-version: ${{ env.NODE_VERSION }}
          cache: "pnpm"
          cache-dependency-path: "pnpm-lock.yaml"
      - name: install dependencies
@@ -100,25 +165,56 @@ jobs:
        run: |
          cp .env.dev.example .env
      - name: Check formatting on changed files
+        env:
+          EVENT_NAME: ${{ github.event_name }}
+          PR_BASE_SHA: ${{ github.event.pull_request.base.sha }}
        run: |
-          if [ "${{ github.event_name }}" = "pull_request" ]; then
-            BASE_SHA=${{ github.event.pull_request.base.sha }}
+          if [ "$EVENT_NAME" = "pull_request" ]; then
+            BASE_SHA="$PR_BASE_SHA"
          else
            BASE_SHA=$(git merge-base origin/main HEAD)
          fi

          echo "Checking files changed from $BASE_SHA to HEAD"

-          # Get changed files
-          CHANGED_FILES=$(git diff --name-only $BASE_SHA HEAD -- '*.js' '*.jsx' '*.ts' '*.tsx' '*.css' | tr '\n' ' ')
-
-          if [ -n "$CHANGED_FILES" ] && [ "$CHANGED_FILES" != " " ]; then
-            echo "Files to check: $CHANGED_FILES"
-            pnpm prettier --check --experimental-cli $CHANGED_FILES
-          else
+          if git diff --quiet --exit-code --diff-filter=d "$BASE_SHA" HEAD -- '*.js' '*.jsx' '*.ts' '*.tsx' '*.css'; then
            echo "No JS/TS/CSS files changed - skipping prettier check"
+          else
+            git diff --name-only -z --diff-filter=d "$BASE_SHA" HEAD -- '*.js' '*.jsx' '*.ts' '*.tsx' '*.css' \
+              | xargs -0 --no-run-if-empty pnpm prettier --check --experimental-cli --
          fi

+  tests-eslint-plugin:
+    runs-on: blacksmith-4vcpu-ubuntu-2404
+    needs:
+      - pre-job
+    if: needs.pre-job.outputs.should_skip != 'true'
+    steps:
+      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        with:
+          persist-credentials: false
+      - uses: pnpm/action-setup@fc06bc1257f339d1d5d8b3a19a8cae5388b55320 # v5.0.0
+        with:
+          version: 10.33.0
+      - name: Use Node.js ${{ env.NODE_VERSION }} without cache
+        if: env.CI_CACHE_ALLOWED != 'true'
+        uses: actions/setup-node@53b83947a5a98c8d113130e565377fae1a50d02f # v6.3.0
+        with:
+          node-version: ${{ env.NODE_VERSION }}
+          package-manager-cache: false
+      - name: Use Node.js ${{ env.NODE_VERSION }} with pnpm cache
+        if: env.CI_CACHE_ALLOWED == 'true'
+        uses: actions/setup-node@53b83947a5a98c8d113130e565377fae1a50d02f # v6.3.0 # zizmor: ignore[cache-poisoning] eslint-plugin test dependency cache only; no published artifacts are built from this cached state
+        with:
+          node-version: ${{ env.NODE_VERSION }}
+          cache: "pnpm"
+          cache-dependency-path: "pnpm-lock.yaml"
+      - name: install dependencies
+        run: |
+          pnpm install
+      - name: run eslint-plugin tests
+        run: pnpm --filter @repo/eslint-plugin run test
+
  test-docker-build:
    timeout-minutes: 20
    runs-on: blacksmith-4vcpu-ubuntu-2404
@@ -129,10 +225,12 @@ jobs:
      DOCKER_COMPOSE_FILE: docker-compose.build.yml
    steps:
      - name: Checkout
-        uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
+        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        with:
+          persist-credentials: false
      - name: Login to Docker Hub
        if: github.repository == 'langfuse/langfuse' && (github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository) && (github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository)
-        uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
+        uses: docker/login-action@4907a6ddec9925e35a0a9e82d7399ccc52663121 # v4.1.0
        with:
          username: ${{ secrets.DOCKERHUB_USERNAME_READ }}
          password: ${{ secrets.DOCKERHUB_TOKEN_READ }}
@@ -178,7 +276,7 @@ jobs:
          done
      - name: Upload docker diagnostics
        if: failure()
-        uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
+        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
        with:
          name: test-docker-build-diagnostics-${{ github.run_id }}-${{ github.run_attempt }}
          path: /tmp/docker-diagnostics
@@ -189,31 +287,42 @@ jobs:
    needs:
      - pre-job
    if: needs.pre-job.outputs.should_skip != 'true'
-    name: tests-web (node${{ matrix.node-version }}, pg${{ matrix.postgres-version }}, mode${{ matrix.deploy-mode }}, shard${{ matrix.shard }}/3)
+    name: tests-web (node${{ matrix.node-version }}, pg${{ matrix.postgres-version }}, mode${{ matrix.deploy-mode }})
    strategy:
      matrix:
        node-version: [24]
        postgres-version: [12, 15]
        deploy-mode: ["", "-azure", "-redis-cluster"]
-        shard: [1, 2, 3]
    steps:
-      - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
+      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        with:
+          persist-credentials: false
      - name: Install golang-migrate for Clickhouse migrations
        run: |
-          curl -L https://github.com/golang-migrate/migrate/releases/download/v4.19.1/migrate.linux-amd64.tar.gz | tar xvz
+          curl --fail --location --retry 5 --retry-delay 2 --retry-all-errors \
+            --output migrate.linux-amd64.tar.gz \
+            https://github.com/golang-migrate/migrate/releases/download/v4.19.1/migrate.linux-amd64.tar.gz
+          tar xzf migrate.linux-amd64.tar.gz
          sudo mv migrate /usr/bin/migrate
          which migrate
-      - uses: pnpm/action-setup@a3252b78c470c02df07e9d59298aecedc3ccdd6d # v3
+      - uses: pnpm/action-setup@fc06bc1257f339d1d5d8b3a19a8cae5388b55320 # v5.0.0
        with:
          version: 10.33.0
      - name: Login to Docker Hub
        if: github.repository == 'langfuse/langfuse' && (github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository)
-        uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
+        uses: docker/login-action@4907a6ddec9925e35a0a9e82d7399ccc52663121 # v4.1.0
        with:
          username: ${{ secrets.DOCKERHUB_USERNAME_READ }}
          password: ${{ secrets.DOCKERHUB_TOKEN_READ }}
-      - name: Use Node.js ${{ matrix.node-version }}
-        uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4
+      - name: Use Node.js ${{ matrix.node-version }} without cache
+        if: env.CI_CACHE_ALLOWED != 'true'
+        uses: actions/setup-node@53b83947a5a98c8d113130e565377fae1a50d02f # v6.3.0
+        with:
+          node-version: ${{ matrix.node-version }}
+          package-manager-cache: false
+      - name: Use Node.js ${{ matrix.node-version }} with pnpm cache
+        if: env.CI_CACHE_ALLOWED == 'true'
+        uses: actions/setup-node@53b83947a5a98c8d113130e565377fae1a50d02f # v6.3.0 # zizmor: ignore[cache-poisoning] test job dependency cache only; no released artifacts are built or published from this cached state
        with:
          node-version: ${{ matrix.node-version }}
          cache: "pnpm"
@@ -228,18 +337,22 @@ jobs:
          echo "LANGFUSE_INGESTION_QUEUE_DELAY_MS=1" >> .env
          echo "LANGFUSE_CACHE_PROMPT_ENABLED=false" >> .env
          echo "LANGFUSE_INGESTION_CLICKHOUSE_WRITE_INTERVAL_MS=1" >> .env
+          echo "LANGFUSE_TRACE_DELETE_DELAY_MS=1" >> .env
          echo "LANGFUSE_TRACE_DELETE_CONCURRENCY=100" >> .env
          echo "ADMIN_API_KEY=admin-api-key" >> .env
          echo "LANGFUSE_EE_LICENSE_KEY=langfuse_ee_test" >> .env
+          echo "LANGFUSE_SKIP_EVALUATOR_MODEL_CALL_VALIDATION=true" >> .env
      - name: Setup Turbo cache
-        uses: actions/cache@0057852bfaa89a56745cba8c7296529d2fc39830 # v4
+        if: env.CI_CACHE_ALLOWED == 'true'
+        uses: actions/cache@668228422ae6a00e4ad889ee87cd7109ec5666a7 # v5.0.4 # zizmor: ignore[cache-poisoning] test job cache only; publish jobs rebuild artifacts and do not restore this cache
        with:
          path: .turbo
          key: ${{ runner.os }}-turbo-${{ github.sha }}
          restore-keys: |
            ${{ runner.os }}-turbo-
      - name: Cache Next.js builds
-        uses: actions/cache@0057852bfaa89a56745cba8c7296529d2fc39830 # v4
+        if: env.CI_CACHE_ALLOWED == 'true'
+        uses: actions/cache@668228422ae6a00e4ad889ee87cd7109ec5666a7 # v5.0.4 # zizmor: ignore[cache-poisoning] test-only Next.js cache; not consumed by artifact publishing or release jobs
        with:
          path: |
            ~/.npm
@@ -276,6 +389,9 @@ jobs:
        run: pnpm run build
        env:
          NODE_OPTIONS: --max_old_space_size=8192
+          # TypeScript is checked explicitly in the lint job; skip duplicate
+          # Next.js type checks in test builds to reduce CI runtime.
+          NEXT_IGNORE_BUILD_ERRORS: "true"
      - name: Start Langfuse
        run: (pnpm run start&)
        env:
@@ -291,11 +407,11 @@ jobs:
          LANGFUSE_INIT_USER_PASSWORD: "password"
      - name: run tests
        working-directory: web
-        run: npx dotenv -e ../.env.test -e ../.env -- npx jest --verbose --runInBand --detectOpenHandles --selectProjects server --shard=${{ matrix.shard }}/3
+        run: npx dotenv -e ../.env.test -e ../.env -- vitest run --project server
      - name: run test-client
-        if: matrix.deploy-mode == '' && matrix.shard == 1
+        if: matrix.deploy-mode == ''
        working-directory: web
-        run: npx dotenv -e ../.env.test -e ../.env -- npx jest --verbose --runInBand --detectOpenHandles --selectProjects client
+        run: npx dotenv -e ../.env.test -e ../.env -- vitest run --project client

  tests-worker:
    timeout-minutes: 20
@@ -310,18 +426,27 @@ jobs:
        postgres-version: [12, 15]
        deploy-mode: ["", "-azure", "-redis-cluster"]
    steps:
-      - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
-      - uses: pnpm/action-setup@a3252b78c470c02df07e9d59298aecedc3ccdd6d # v3
+      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        with:
+          persist-credentials: false
+      - uses: pnpm/action-setup@fc06bc1257f339d1d5d8b3a19a8cae5388b55320 # v5.0.0
        with:
          version: 10.33.0
      - name: Login to Docker Hub
        if: github.repository == 'langfuse/langfuse' && (github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository)
-        uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
+        uses: docker/login-action@4907a6ddec9925e35a0a9e82d7399ccc52663121 # v4.1.0
        with:
          username: ${{ secrets.DOCKERHUB_USERNAME_READ }}
          password: ${{ secrets.DOCKERHUB_TOKEN_READ }}
-      - name: Use Node.js ${{ matrix.node-version }}
-        uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4
+      - name: Use Node.js ${{ matrix.node-version }} without cache
+        if: env.CI_CACHE_ALLOWED != 'true'
+        uses: actions/setup-node@53b83947a5a98c8d113130e565377fae1a50d02f # v6.3.0
+        with:
+          node-version: ${{ matrix.node-version }}
+          package-manager-cache: false
+      - name: Use Node.js ${{ matrix.node-version }} with pnpm cache
+        if: env.CI_CACHE_ALLOWED == 'true'
+        uses: actions/setup-node@53b83947a5a98c8d113130e565377fae1a50d02f # v6.3.0 # zizmor: ignore[cache-poisoning] worker test dependency cache only; no release artifacts are produced from this cache
        with:
          node-version: ${{ matrix.node-version }}
          cache: "pnpm"
@@ -331,7 +456,10 @@ jobs:
          pnpm install
      - name: Install golang-migrate for Clickhouse migrations
        run: |
-          curl -L https://github.com/golang-migrate/migrate/releases/download/v4.19.1/migrate.linux-amd64.tar.gz | tar xvz
+          curl --fail --location --retry 5 --retry-delay 2 --retry-all-errors \
+            --output migrate.linux-amd64.tar.gz \
+            https://github.com/golang-migrate/migrate/releases/download/v4.19.1/migrate.linux-amd64.tar.gz
+          tar xzf migrate.linux-amd64.tar.gz
          sudo mv migrate /usr/bin/migrate
          which migrate
      - name: Load default env
@@ -381,28 +509,34 @@ jobs:
    if: startsWith(github.ref, 'refs/tags/') || (needs.pre-job.outputs.should_skip != 'true' && (needs.pre-job.outputs.llm_connections_changed == 'true' || github.event_name == 'workflow_dispatch'))
    name: test-worker-llm-connections (node24, pg15)
    steps:
-      - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
-      - uses: pnpm/action-setup@a3252b78c470c02df07e9d59298aecedc3ccdd6d # v3
+      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        with:
+          persist-credentials: false
+      - uses: pnpm/action-setup@fc06bc1257f339d1d5d8b3a19a8cae5388b55320 # v5.0.0
        with:
          version: 10.33.0
      - name: Login to Docker Hub
        if: github.repository == 'langfuse/langfuse' && (github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository)
-        uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
+        uses: docker/login-action@4907a6ddec9925e35a0a9e82d7399ccc52663121 # v4.1.0
        with:
          username: ${{ secrets.DOCKERHUB_USERNAME_READ }}
          password: ${{ secrets.DOCKERHUB_TOKEN_READ }}
-      - name: Use Node.js 24
-        uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4
+      # Keep this secret-bearing job cache-cold to avoid exposing real provider
+      # credentials to potentially poisoned shared package-manager state.
+      - name: Use Node.js ${{ env.NODE_VERSION }}
+        uses: actions/setup-node@53b83947a5a98c8d113130e565377fae1a50d02f # v6.3.0
        with:
-          node-version: 24
-          cache: "pnpm"
-          cache-dependency-path: "pnpm-lock.yaml"
+          node-version: ${{ env.NODE_VERSION }}
+          package-manager-cache: false
      - name: install dependencies
        run: |
          pnpm install
      - name: Install golang-migrate for Clickhouse migrations
        run: |
-          curl -L https://github.com/golang-migrate/migrate/releases/download/v4.19.1/migrate.linux-amd64.tar.gz | tar xvz
+          curl --fail --location --retry 5 --retry-delay 2 --retry-all-errors \
+            --output migrate.linux-amd64.tar.gz \
+            https://github.com/golang-migrate/migrate/releases/download/v4.19.1/migrate.linux-amd64.tar.gz
+          tar xzf migrate.linux-amd64.tar.gz
          sudo mv migrate /usr/bin/migrate
          which migrate
      - name: Load default env
@@ -442,6 +576,7 @@ jobs:
          LANGFUSE_LLM_CONNECTION_BEDROCK_ACCESS_KEY_ID: ${{ secrets.LANGFUSE_LLM_CONNECTION_BEDROCK_ACCESS_KEY_ID }}
          LANGFUSE_LLM_CONNECTION_BEDROCK_SECRET_ACCESS_KEY: ${{ secrets.LANGFUSE_LLM_CONNECTION_BEDROCK_SECRET_ACCESS_KEY }}
          LANGFUSE_LLM_CONNECTION_BEDROCK_REGION: ${{ secrets.LANGFUSE_LLM_CONNECTION_BEDROCK_REGION }}
+          LANGFUSE_LLM_CONNECTION_BEDROCK_API_KEY: ${{ secrets.LANGFUSE_LLM_CONNECTION_BEDROCK_API_KEY }}
          LANGFUSE_LLM_CONNECTION_VERTEXAI_KEY: ${{ secrets.LANGFUSE_LLM_CONNECTION_VERTEXAI_KEY }}
          LANGFUSE_LLM_CONNECTION_GOOGLEAISTUDIO_KEY: ${{ secrets.LANGFUSE_LLM_CONNECTION_GOOGLEAISTUDIO_KEY }}

@@ -451,23 +586,34 @@ jobs:
      - pre-job
    if: needs.pre-job.outputs.should_skip != 'true'
    steps:
-      - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
-      - uses: pnpm/action-setup@a3252b78c470c02df07e9d59298aecedc3ccdd6d # v3
+      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        with:
+          persist-credentials: false
+      - uses: pnpm/action-setup@fc06bc1257f339d1d5d8b3a19a8cae5388b55320 # v5.0.0
        with:
          version: 10.33.0
-      - uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4
+      - name: Use Node.js ${{ env.NODE_VERSION }} without cache
+        if: env.CI_CACHE_ALLOWED != 'true'
+        uses: actions/setup-node@53b83947a5a98c8d113130e565377fae1a50d02f # v6.3.0
        with:
-          node-version: 24
+          node-version: ${{ env.NODE_VERSION }}
+          package-manager-cache: false
+      - name: Use Node.js ${{ env.NODE_VERSION }} with pnpm cache
+        if: env.CI_CACHE_ALLOWED == 'true'
+        uses: actions/setup-node@53b83947a5a98c8d113130e565377fae1a50d02f # v6.3.0 # zizmor: ignore[cache-poisoning] e2e dependency cache only; release images are rebuilt later without restoring this cache
+        with:
+          node-version: ${{ env.NODE_VERSION }}
          cache: "pnpm"
          cache-dependency-path: "pnpm-lock.yaml"
      - name: Login to Docker Hub
        if: github.repository == 'langfuse/langfuse' && (github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository)
-        uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
+        uses: docker/login-action@4907a6ddec9925e35a0a9e82d7399ccc52663121 # v4.1.0
        with:
          username: ${{ secrets.DOCKERHUB_USERNAME_READ }}
          password: ${{ secrets.DOCKERHUB_TOKEN_READ }}
      - name: Setup Turbo cache
-        uses: actions/cache@0057852bfaa89a56745cba8c7296529d2fc39830 # v4
+        if: env.CI_CACHE_ALLOWED == 'true'
+        uses: actions/cache@668228422ae6a00e4ad889ee87cd7109ec5666a7 # v5.0.4 # zizmor: ignore[cache-poisoning] e2e cache only; no published artifact path restores this cache
        with:
          path: .turbo
          key: ${{ runner.os }}-turbo-e2e-${{ github.sha }}
@@ -475,7 +621,8 @@ jobs:
            ${{ runner.os }}-turbo-e2e-
            ${{ runner.os }}-turbo-
      - name: Cache Next.js builds
-        uses: actions/cache@0057852bfaa89a56745cba8c7296529d2fc39830 # v4
+        if: env.CI_CACHE_ALLOWED == 'true'
+        uses: actions/cache@668228422ae6a00e4ad889ee87cd7109ec5666a7 # v5.0.4 # zizmor: ignore[cache-poisoning] e2e-only Next.js cache; not part of any artifact build or publishing flow
        with:
          path: |
            ~/.npm
@@ -493,7 +640,10 @@ jobs:
          cp .env.dev.example web/.env
      - name: Install golang-migrate for Clickhouse migrations
        run: |
-          curl -L https://github.com/golang-migrate/migrate/releases/download/v4.19.1/migrate.linux-amd64.tar.gz | tar xvz
+          curl --fail --location --retry 5 --retry-delay 2 --retry-all-errors \
+            --output migrate.linux-amd64.tar.gz \
+            https://github.com/golang-migrate/migrate/releases/download/v4.19.1/migrate.linux-amd64.tar.gz
+          tar xzf migrate.linux-amd64.tar.gz
          sudo mv migrate /usr/bin/migrate
          which migrate
      - name: Run + migrate
@@ -517,8 +667,11 @@ jobs:
        run: pnpm run build
        env:
          NODE_OPTIONS: --max_old_space_size=8192
+          # TypeScript is checked explicitly in the lint job; skip duplicate
+          # Next.js type checks in test builds to reduce CI runtime.
+          NEXT_IGNORE_BUILD_ERRORS: "true"
      - name: Install playwright
-        run: pnpm --filter=web exec playwright install --with-deps
+        run: pnpm --filter=web exec playwright install --with-deps --only-shell chromium
      - name: Run e2e tests
        run: pnpm --filter=web run test:e2e

@@ -528,19 +681,29 @@ jobs:
      - pre-job
    if: needs.pre-job.outputs.should_skip != 'true'
    steps:
-      - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
+      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        with:
+          persist-credentials: false
      - name: Login to Docker Hub
        if: github.repository == 'langfuse/langfuse' && (github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository)
-        uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
+        uses: docker/login-action@4907a6ddec9925e35a0a9e82d7399ccc52663121 # v4.1.0
        with:
          username: ${{ secrets.DOCKERHUB_USERNAME_READ }}
          password: ${{ secrets.DOCKERHUB_TOKEN_READ }}
-      - uses: pnpm/action-setup@a3252b78c470c02df07e9d59298aecedc3ccdd6d # v3
+      - uses: pnpm/action-setup@fc06bc1257f339d1d5d8b3a19a8cae5388b55320 # v5.0.0
        with:
          version: 10.33.0
-      - uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4
+      - name: Use Node.js ${{ env.NODE_VERSION }} without cache
+        if: env.CI_CACHE_ALLOWED != 'true'
+        uses: actions/setup-node@53b83947a5a98c8d113130e565377fae1a50d02f # v6.3.0
        with:
-          node-version: 24
+          node-version: ${{ env.NODE_VERSION }}
+          package-manager-cache: false
+      - name: Use Node.js ${{ env.NODE_VERSION }} with pnpm cache
+        if: env.CI_CACHE_ALLOWED == 'true'
+        uses: actions/setup-node@53b83947a5a98c8d113130e565377fae1a50d02f # v6.3.0 # zizmor: ignore[cache-poisoning] server e2e dependency cache only; release/publish steps rebuild separately
+        with:
+          node-version: ${{ env.NODE_VERSION }}
          cache: "pnpm"
          cache-dependency-path: "pnpm-lock.yaml"
      - name: install dependencies
@@ -548,7 +711,10 @@ jobs:
          pnpm install
      - name: Install golang-migrate for Clickhouse migrations
        run: |
-          curl -L https://github.com/golang-migrate/migrate/releases/download/v4.19.1/migrate.linux-amd64.tar.gz | tar xvz
+          curl --fail --location --retry 5 --retry-delay 2 --retry-all-errors \
+            --output migrate.linux-amd64.tar.gz \
+            https://github.com/golang-migrate/migrate/releases/download/v4.19.1/migrate.linux-amd64.tar.gz
+          tar xzf migrate.linux-amd64.tar.gz
          sudo mv migrate /usr/bin/migrate
          which migrate
      - name: Load default env
@@ -575,6 +741,9 @@ jobs:
        run: pnpm run build
        env:
          NODE_OPTIONS: --max_old_space_size=8192
+          # TypeScript is checked explicitly in the lint job; skip duplicate
+          # Next.js type checks in test builds to reduce CI runtime.
+          NEXT_IGNORE_BUILD_ERRORS: "true"
      - name: Run server
        run: (pnpm run start&)
      - name: Check worker health
@@ -589,9 +758,11 @@ jobs:
  all-ci-passed:
    # This allows us to have a branch protection rule for tests and deploys with matrix
    runs-on: blacksmith-4vcpu-ubuntu-2404
-    needs: [
+    needs:
+      [
        lint,
        prettier-check,
+        tests-eslint-plugin,
        tests-web,
        tests-worker,
        test-worker-llm-connections,
@@ -620,16 +791,58 @@ jobs:
        run: exit 1
        working-directory: .
      - name: Notify Slack
-        uses: ravsamhq/notify-slack-action@be814b201e233b2dc673608aa46e5447c8ab13f2 # v2
-        if: always() && github.event_name == 'push' && (github.ref == 'refs/heads/main' || startsWith(github.ref, 'refs/tags/'))
+        if: failure() && github.event_name == 'push' && (github.ref == 'refs/heads/main' || startsWith(github.ref, 'refs/tags/'))
+        uses: slackapi/slack-github-action@af78098f536edbc4de71162a307590698245be95 # v3.0.1
        with:
-          status: ${{ job.status }}
-          notify_when: "failure"
-        env:
-          SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }}
+          webhook: ${{ secrets.SLACK_WEBHOOK_URL }}
+          webhook-type: incoming-webhook
+          payload: |
+            {
+              "text": "❌ CI failed on ${{ github.ref_name }}",
+              "blocks": [
+                {
+                  "type": "header",
+                  "text": {
+                    "type": "plain_text",
+                    "text": "❌ CI Failed",
+                    "emoji": true
+                  }
+                },
+                {
+                  "type": "section",
+                  "fields": [
+                    {
+                      "type": "mrkdwn",
+                      "text": "*Branch/Tag:*\n`${{ github.ref_name }}`"
+                    },
+                    {
+                      "type": "mrkdwn",
+                      "text": "*Triggered by:*\n${{ github.actor }}"
+                    }
+                  ]
+                },
+                {
+                  "type": "actions",
+                  "elements": [
+                    {
+                      "type": "button",
+                      "text": {
+                        "type": "plain_text",
+                        "text": "View Workflow Logs",
+                        "emoji": true
+                      },
+                      "url": "${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}",
+                      "style": "danger"
+                    }
+                  ]
+                }
+              ]
+            }
      - name: Output job results
        run: |
-          echo "Job results: ${{ steps.set-success-output.outputs.success }}"
+          echo "Job results: ${STEPS_SET_SUCCESS_OUTPUT_OUTPUTS_SUCCESS}"
+        env:
+          STEPS_SET_SUCCESS_OUTPUT_OUTPUTS_SUCCESS: ${{ steps.set-success-output.outputs.success }}

  build-docker-image-release:
    needs: all-ci-passed
@@ -671,25 +884,27 @@ jobs:

    steps:
      - name: Checkout
-        uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
+        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        with:
+          persist-credentials: false
      - name: Set NEXT_PUBLIC_BUILD_ID
        run: echo "NEXT_PUBLIC_BUILD_ID=$(git rev-parse --short HEAD)" >> $GITHUB_ENV
      - name: Log in to the GitHub Container registry
-        uses: docker/login-action@465a07811f14bebb1938fbed4728c6a1ff8901fc # v2
+        uses: docker/login-action@4907a6ddec9925e35a0a9e82d7399ccc52663121 # v4.1.0
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}
      - name: Log in to Docker Hub
-        uses: docker/login-action@465a07811f14bebb1938fbed4728c6a1ff8901fc # v2
+        uses: docker/login-action@4907a6ddec9925e35a0a9e82d7399ccc52663121 # v4.1.0
        with:
          username: ${{ secrets.DOCKERHUB_USERNAME }}
          password: ${{ secrets.DOCKERHUB_TOKEN }}
      - name: Setup Blacksmith Builder
-        uses: useblacksmith/setup-docker-builder@5241b2e9423e8b1fa37ed6050ecb62d0fb9a4e38 # v1
+        uses: useblacksmith/setup-docker-builder@5241b2e9423e8b1fa37ed6050ecb62d0fb9a4e38 # v1.6.0
      - name: Extract metadata (labels) for Docker
        id: meta
-        uses: docker/metadata-action@818d4b7b91585d195f67373fd9cb0332e31a7175 # v4
+        uses: docker/metadata-action@030e881283bb7a6894de51c315a6bfe6a94e05cf # v6.0.0
        with:
          images: |
            ghcr.io/langfuse/${{ matrix.image_name }}
@@ -706,7 +921,7 @@ jobs:
            type=raw,value=latest,enable=${{ startsWith(github.ref, 'refs/tags/v3') && !contains(github.ref, '-rc') }}
      - name: Build and push image by digest to GitHub Container Registry (${{ matrix.component }}, ${{ matrix.platform_tag }})
        id: build-ghcr
-        uses: useblacksmith/build-push-action@cbd1f60d194a98cb3be5523b15134501eaf0fbf3 # v2
+        uses: useblacksmith/build-push-action@cbd1f60d194a98cb3be5523b15134501eaf0fbf3 # v2.1.0
        with:
          context: .
          file: ${{ matrix.dockerfile }}
@@ -717,7 +932,7 @@ jobs:
          sbom: false
      - name: Build and push image by digest to Docker Hub (${{ matrix.component }}, ${{ matrix.platform_tag }})
        id: build-dockerhub
-        uses: useblacksmith/build-push-action@cbd1f60d194a98cb3be5523b15134501eaf0fbf3 # v2
+        uses: useblacksmith/build-push-action@cbd1f60d194a98cb3be5523b15134501eaf0fbf3 # v2.1.0
        with:
          context: .
          file: ${{ matrix.dockerfile }}
@@ -727,17 +942,21 @@ jobs:
          provenance: false
          sbom: false
      - name: Record pushed digests
+        env:
+          DIGEST_GHCR: ${{ steps.build-ghcr.outputs.digest }}
+          DIGEST_DOCKERHUB: ${{ steps.build-dockerhub.outputs.digest }}
+          PLATFORM_TAG: ${{ matrix.platform_tag }}
        run: |
-          if [ -z '${{ steps.build-ghcr.outputs.digest }}' ] || [ -z '${{ steps.build-dockerhub.outputs.digest }}' ]; then
+          if [ -z "$DIGEST_GHCR" ] || [ -z "$DIGEST_DOCKERHUB" ]; then
            echo "Missing registry digest output"
            exit 1
          fi

          mkdir -p "$RUNNER_TEMP/digests/ghcr" "$RUNNER_TEMP/digests/dockerhub"
-          printf '%s\n' '${{ steps.build-ghcr.outputs.digest }}' > "$RUNNER_TEMP/digests/ghcr/${{ matrix.platform_tag }}.txt"
-          printf '%s\n' '${{ steps.build-dockerhub.outputs.digest }}' > "$RUNNER_TEMP/digests/dockerhub/${{ matrix.platform_tag }}.txt"
+          printf '%s\n' "$DIGEST_GHCR" > "$RUNNER_TEMP/digests/ghcr/${PLATFORM_TAG}.txt"
+          printf '%s\n' "$DIGEST_DOCKERHUB" > "$RUNNER_TEMP/digests/dockerhub/${PLATFORM_TAG}.txt"
      - name: Upload release digests
-        uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
+        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
        with:
          name: release-digests-${{ matrix.component }}-${{ matrix.platform_tag }}
          path: |
@@ -765,27 +984,27 @@ jobs:

    steps:
      - name: Log in to the GitHub Container registry
-        uses: docker/login-action@465a07811f14bebb1938fbed4728c6a1ff8901fc # v2
+        uses: docker/login-action@4907a6ddec9925e35a0a9e82d7399ccc52663121 # v4.1.0
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}
      - name: Log in to Docker Hub
-        uses: docker/login-action@465a07811f14bebb1938fbed4728c6a1ff8901fc # v2
+        uses: docker/login-action@4907a6ddec9925e35a0a9e82d7399ccc52663121 # v4.1.0
        with:
          username: ${{ secrets.DOCKERHUB_USERNAME }}
          password: ${{ secrets.DOCKERHUB_TOKEN }}
      - name: Setup Blacksmith Builder
-        uses: useblacksmith/setup-docker-builder@5241b2e9423e8b1fa37ed6050ecb62d0fb9a4e38 # v1
+        uses: useblacksmith/setup-docker-builder@5241b2e9423e8b1fa37ed6050ecb62d0fb9a4e38 # v1.6.0
      - name: Download release digests
-        uses: actions/download-artifact@fa0a91b85d4f404e444e00e005971372dc801d16 # v4
+        uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
        with:
          pattern: release-digests-${{ matrix.component }}-*
          merge-multiple: true
          path: ${{ runner.temp }}/digests
      - name: Extract metadata (tags) for GitHub Container Registry
        id: meta-ghcr
-        uses: docker/metadata-action@818d4b7b91585d195f67373fd9cb0332e31a7175 # v4
+        uses: docker/metadata-action@030e881283bb7a6894de51c315a6bfe6a94e05cf # v6.0.0
        with:
          images: ghcr.io/langfuse/${{ matrix.image_name }}
          flavor: |
@@ -800,7 +1019,7 @@ jobs:
            type=raw,value=latest,enable=${{ startsWith(github.ref, 'refs/tags/v3') && !contains(github.ref, '-rc') }}
      - name: Extract metadata (tags) for Docker Hub
        id: meta-dockerhub
-        uses: docker/metadata-action@818d4b7b91585d195f67373fd9cb0332e31a7175 # v4
+        uses: docker/metadata-action@030e881283bb7a6894de51c315a6bfe6a94e05cf # v6.0.0
        with:
          images: langfuse/${{ matrix.image_name }}
          flavor: |
@@ -814,6 +1033,10 @@ jobs:
            type=semver,pattern={{major}},enable=${{ !contains(github.ref, '-rc') }}
            type=raw,value=latest,enable=${{ startsWith(github.ref, 'refs/tags/v3') && !contains(github.ref, '-rc') }}
      - name: Publish multi-platform manifest to GitHub Container Registry
+        env:
+          STEPS_META_GHCR_OUTPUTS_TAGS: ${{ steps.meta-ghcr.outputs.tags }}
+          IMAGE_NAME: ${{ matrix.image_name }}
+          COMPONENT: ${{ matrix.component }}
        run: |
          ghcr_tags=()
          ghcr_sources=()
@@ -821,23 +1044,27 @@ jobs:
          while IFS= read -r tag; do
            [ -n "$tag" ] || continue
            ghcr_tags+=("-t" "$tag")
-          done <<'EOF'
-          ${{ steps.meta-ghcr.outputs.tags }}
+          done <<EOF
+          ${STEPS_META_GHCR_OUTPUTS_TAGS}
          EOF

          shopt -s nullglob
          for digest_file in "$RUNNER_TEMP"/digests/ghcr/*.txt; do
            digest="$(cat "$digest_file")"
-            ghcr_sources+=("ghcr.io/langfuse/${{ matrix.image_name }}@$digest")
+            ghcr_sources+=("ghcr.io/langfuse/${IMAGE_NAME}@$digest")
          done

          if [ "${#ghcr_sources[@]}" -lt 2 ]; then
-            echo "Expected amd64 and arm64 GHCR digests for ${{ matrix.component }}"
+            echo "Expected amd64 and arm64 GHCR digests for $COMPONENT"
            exit 1
          fi

          docker buildx imagetools create "${ghcr_tags[@]}" "${ghcr_sources[@]}"
      - name: Publish multi-platform manifest to Docker Hub
+        env:
+          STEPS_META_DOCKERHUB_OUTPUTS_TAGS: ${{ steps.meta-dockerhub.outputs.tags }}
+          IMAGE_NAME: ${{ matrix.image_name }}
+          COMPONENT: ${{ matrix.component }}
        run: |
          dockerhub_tags=()
          dockerhub_sources=()
@@ -845,29 +1072,32 @@ jobs:
          while IFS= read -r tag; do
            [ -n "$tag" ] || continue
            dockerhub_tags+=("-t" "$tag")
-          done <<'EOF'
-          ${{ steps.meta-dockerhub.outputs.tags }}
+          done <<EOF
+          ${STEPS_META_DOCKERHUB_OUTPUTS_TAGS}
          EOF

          shopt -s nullglob
          for digest_file in "$RUNNER_TEMP"/digests/dockerhub/*.txt; do
            digest="$(cat "$digest_file")"
-            dockerhub_sources+=("langfuse/${{ matrix.image_name }}@$digest")
+            dockerhub_sources+=("langfuse/${IMAGE_NAME}@$digest")
          done

          if [ "${#dockerhub_sources[@]}" -lt 2 ]; then
-            echo "Expected amd64 and arm64 Docker Hub digests for ${{ matrix.component }}"
+            echo "Expected amd64 and arm64 Docker Hub digests for $COMPONENT"
            exit 1
          fi

          docker buildx imagetools create "${dockerhub_tags[@]}" "${dockerhub_sources[@]}"
      - name: Inspect published manifests
        run: |
-          ghcr_first_tag="$(printf '%s\n' "${{ steps.meta-ghcr.outputs.tags }}" | sed -n '1p')"
-          dockerhub_first_tag="$(printf '%s\n' "${{ steps.meta-dockerhub.outputs.tags }}" | sed -n '1p')"
+          ghcr_first_tag="$(printf '%s\n' "${STEPS_META_GHCR_OUTPUTS_TAGS}" | sed -n '1p')"
+          dockerhub_first_tag="$(printf '%s\n' "${STEPS_META_DOCKERHUB_OUTPUTS_TAGS}" | sed -n '1p')"

          docker buildx imagetools inspect "$ghcr_first_tag"
          docker buildx imagetools inspect "$dockerhub_first_tag"
+        env:
+          STEPS_META_GHCR_OUTPUTS_TAGS: ${{ steps.meta-ghcr.outputs.tags }}
+          STEPS_META_DOCKERHUB_OUTPUTS_TAGS: ${{ steps.meta-dockerhub.outputs.tags }}

  notify-docker-image-release:
    needs:
@@ -880,10 +1110,50 @@ jobs:
        if: contains(needs.*.result, 'failure') || contains(needs.*.result, 'cancelled')
        run: exit 1
      - name: Notify Slack
-        uses: ravsamhq/notify-slack-action@be814b201e233b2dc673608aa46e5447c8ab13f2 # v2
-        if: always()
+        if: failure()
+        uses: slackapi/slack-github-action@af78098f536edbc4de71162a307590698245be95 # v3.0.1
        with:
-          status: ${{ job.status }}
-          notify_when: "failure"
-        env:
-          SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }}
+          webhook: ${{ secrets.SLACK_WEBHOOK_URL }}
+          webhook-type: incoming-webhook
+          payload: |
+            {
+              "text": "❌ Docker release failed on ${{ github.ref_name }}",
+              "blocks": [
+                {
+                  "type": "header",
+                  "text": {
+                    "type": "plain_text",
+                    "text": "❌ Docker Release Failed",
+                    "emoji": true
+                  }
+                },
+                {
+                  "type": "section",
+                  "fields": [
+                    {
+                      "type": "mrkdwn",
+                      "text": "*Tag:*\n`${{ github.ref_name }}`"
+                    },
+                    {
+                      "type": "mrkdwn",
+                      "text": "*Triggered by:*\n${{ github.actor }}"
+                    }
+                  ]
+                },
+                {
+                  "type": "actions",
+                  "elements": [
+                    {
+                      "type": "button",
+                      "text": {
+                        "type": "plain_text",
+                        "text": "View Workflow Logs",
+                        "emoji": true
+                      },
+                      "url": "${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}",
+                      "style": "danger"
+                    }
+                  ]
+                }
+              ]
+            }
@@ -12,6 +12,8 @@ concurrency:
  group: promote-main-to-production
  cancel-in-progress: false

+permissions: {}
+
 jobs:
  promote:
    runs-on: ubuntu-latest
@@ -19,16 +21,19 @@ jobs:
    steps:
      - name: Validate confirmation
        run: |
-          if [ "${{ github.event.inputs.confirm }}" != "promote" ]; then
+          if [ "${INPUT_CONFIRM}" != "promote" ]; then
            echo "Input 'confirm' must be 'promote'."
            exit 1
          fi
+        env:
+          INPUT_CONFIRM: ${{ github.event.inputs.confirm }}

-      - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
+      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
        with:
          ref: main
          fetch-depth: 0
          token: ${{ secrets.GH_ACCESS_TOKEN }}
+          persist-credentials: false

      - name: Print commit refs
        run: |
@@ -41,4 +46,6 @@ jobs:
          fi

      - name: Force push main to production
-        run: git push origin +main:production
+        run: git push "https://x-access-token:${GH_ACCESS_TOKEN}@github.com/${GITHUB_REPOSITORY}.git" +main:production
+        env:
+          GH_ACCESS_TOKEN: ${{ secrets.GH_ACCESS_TOKEN }}
@@ -5,15 +5,20 @@ on:
    tags:
      - "v3.[0-9]+.[0-9]+" # Semantic version tags

+permissions: {}
+
 jobs:
  release:
    runs-on: ubuntu-latest
    environment: "protected branches"
    steps:
-      - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
+      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
        with:
          ref: main # Always checkout main even for tagged releases
          fetch-depth: 0
          token: ${{ secrets.GH_ACCESS_TOKEN }}
+          persist-credentials: false
      - name: Push to production
-        run: git push origin +main:production
+        run: git push "https://x-access-token:${GH_ACCESS_TOKEN}@github.com/${GITHUB_REPOSITORY}.git" +main:production
+        env:
+          GH_ACCESS_TOKEN: ${{ secrets.GH_ACCESS_TOKEN }}
@@ -12,23 +12,31 @@ concurrency:
  group: sdk-api-spec-${{ github.ref }}
  cancel-in-progress: true

+permissions:
+  contents: read
+
 jobs:
  generate-sdk-api-specs:
    runs-on: ubuntu-latest
+    environment: "protected branches"
    steps:
      - name: Checkout repository
-        uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
+        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        with:
+          persist-credentials: false

      - name: Install pnpm
-        uses: pnpm/action-setup@eae0cfeb286e66ffb5155f1a79b90583a127a68b # v2
+        uses: pnpm/action-setup@fc06bc1257f339d1d5d8b3a19a8cae5388b55320 # v5.0.0
        with:
          version: 10.33.0

+      # Keep this secret-bearing job cache-cold to avoid exposing Fern and
+      # GitHub write credentials to potentially poisoned shared package-manager state.
      - name: Setup Node.js
-        uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4
+        uses: actions/setup-node@53b83947a5a98c8d113130e565377fae1a50d02f # v6.3.0
        with:
          node-version: "24"
-          cache: "pnpm"
+          package-manager-cache: false

      - name: Install dependencies
        run: pnpm install
@@ -39,7 +47,7 @@ jobs:
        run: npx fern-api generate --api server --force

      - name: Install uv
-        uses: astral-sh/setup-uv@cec208311dfd045dd5311c1add060b2062131d57 # v8
+        uses: astral-sh/setup-uv@cec208311dfd045dd5311c1add060b2062131d57 # v8.0.0
        with:
          version: "0.11.2"

@@ -0,0 +1,43 @@
+---
+name: Semgrep
+
+on:
+  pull_request:
+    branches:
+      - "**"
+    types:
+      - opened
+      - reopened
+      - synchronize
+      - ready_for_review
+
+permissions:
+  contents: read
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  semgrep:
+    name: Security scan
+    runs-on: ubuntu-latest
+    timeout-minutes: 30
+    container:
+      image: semgrep/semgrep:1.162.0@sha256:9349edbadf90c3f3c0c3f55867625354e89680e6fa10d9034042af52fdb0e0d0
+    steps:
+      - name: Checkout
+        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        with:
+          fetch-depth: 0
+          persist-credentials: false
+
+      - name: Run Semgrep
+        env:
+          BASELINE_COMMIT: ${{ github.event.pull_request.base.sha }}
+          SEMGREP_SEND_METRICS: "off"
+        run: |
+          semgrep scan \
+            --config p/default \
+            --baseline-commit "$BASELINE_COMMIT" \
+            --error
@@ -3,46 +3,55 @@ on:
  push:
    branches: ["production", "main"]

+permissions:
+  contents: read
+  security-events: write
+
 jobs:
  snyk:
    runs-on: ubuntu-latest
+    environment: snyk
    steps:
-      - uses: actions/checkout@ee0669bd1cc54295c223e0bb666b733df41de1c5 # v2
+      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        with:
+          persist-credentials: false
+      - name: Set up Snyk CLI
+        uses: snyk/actions/setup@9adf32b1121593767fc3c057af55b55db032dc04 # master
+        with:
+          snyk-version: v1.1304.0
      - name: Run Snyk to check Docker image for vulnerabilities
        # Snyk can be used to break the build when it detects vulnerabilities.
        # In this case we want to upload the issues to GitHub Code Scanning
        continue-on-error: true
-        uses: snyk/actions/docker@9adf32b1121593767fc3c057af55b55db032dc04 # master
        env:
          # In order to use the Snyk Action you will need to have a Snyk API token.
          # See https://docs.snyk.io/integrations/ci-cd-integrations/github-actions-integration#getting-your-snyk-token
          # or you can sign up for free at https://snyk.io/login
          SNYK_TOKEN: ${{ secrets.SNYK_TOKEN }}
-        with:
-          image: langfuse/langfuse
-          args: --sarif-file-output=snyk.sarif
-      - name: Fix SARIF file
+        run: |
+          snyk container test langfuse/langfuse \
+            --file=web/Dockerfile \
+            --sarif-file-output=snyk.sarif
+      - name: Normalize SARIF file
        if: always()
        run: |
          if [ -f snyk.sarif ]; then
-            # Fix undefined security severity values in SARIF file
-            sed -i 's/"security-severity": "undefined"/"security-severity": "0"/g' snyk.sarif
+            # Snyk emits invalid security-severity values. upload-sarif requires a numeric string.
+            sed -i \
+              -e 's/"security-severity": "undefined"/"security-severity": "0"/g' \
+              -e 's/"security-severity": "null"/"security-severity": "0"/g' \
+              -e 's/"security-severity": null/"security-severity": "0"/g' \
+              snyk.sarif
            echo "SARIF file fixed"
          else
            echo "No SARIF file found"
          fi
-      - name: Echo SARIF file for debugging
-        if: always()
-        run: |
-          if [ -f snyk.sarif ]; then
-            echo "=== SARIF File Contents ==="
-            cat snyk.sarif
-            echo "=== End of SARIF File ==="
-          else
-            echo "No SARIF file found to display"
-          fi
      - name: Upload result to GitHub Code Scanning
-        uses: github/codeql-action/upload-sarif@c10b8064de6f491fea524254123dbe5e09572f13 # v4
+        uses: github/codeql-action/upload-sarif@c10b8064de6f491fea524254123dbe5e09572f13 # v4.35.1
        if: always()
        with:
          sarif_file: snyk.sarif
+          category: snyk-container-web
+      - name: Echo SARIF file for debugging
+        if: failure() && hashFiles('snyk.sarif') != ''
+        run: cat snyk.sarif
@@ -3,46 +3,55 @@ on:
  push:
    branches: ["production", "main"]

+permissions:
+  contents: read
+  security-events: write
+
 jobs:
  snyk:
    runs-on: ubuntu-latest
+    environment: snyk
    steps:
-      - uses: actions/checkout@ee0669bd1cc54295c223e0bb666b733df41de1c5 # v2
+      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        with:
+          persist-credentials: false
+      - name: Set up Snyk CLI
+        uses: snyk/actions/setup@9adf32b1121593767fc3c057af55b55db032dc04 # master
+        with:
+          snyk-version: v1.1304.0
      - name: Run Snyk to check Docker image for vulnerabilities
        # Snyk can be used to break the build when it detects vulnerabilities.
        # In this case we want to upload the issues to GitHub Code Scanning
        continue-on-error: true
-        uses: snyk/actions/docker@9adf32b1121593767fc3c057af55b55db032dc04 # master
        env:
          # In order to use the Snyk Action you will need to have a Snyk API token.
          # See https://docs.snyk.io/integrations/ci-cd-integrations/github-actions-integration#getting-your-snyk-token
          # or you can sign up for free at https://snyk.io/login
          SNYK_TOKEN: ${{ secrets.SNYK_TOKEN }}
-        with:
-          image: langfuse/langfuse-worker
-          args: --sarif-file-output=snyk.sarif
-      - name: Fix SARIF file
+        run: |
+          snyk container test langfuse/langfuse-worker \
+            --file=worker/Dockerfile \
+            --sarif-file-output=snyk.sarif
+      - name: Normalize SARIF file
        if: always()
        run: |
          if [ -f snyk.sarif ]; then
-            # Fix undefined security severity values in SARIF file
-            sed -i 's/"security-severity": "undefined"/"security-severity": "0"/g' snyk.sarif
+            # Snyk emits invalid security-severity values. upload-sarif requires a numeric string.
+            sed -i \
+              -e 's/"security-severity": "undefined"/"security-severity": "0"/g' \
+              -e 's/"security-severity": "null"/"security-severity": "0"/g' \
+              -e 's/"security-severity": null/"security-severity": "0"/g' \
+              snyk.sarif
            echo "SARIF file fixed"
          else
            echo "No SARIF file found"
          fi
-      - name: Echo SARIF file for debugging
-        if: always()
-        run: |
-          if [ -f snyk.sarif ]; then
-            echo "=== SARIF File Contents ==="
-            cat snyk.sarif
-            echo "=== End of SARIF File ==="
-          else
-            echo "No SARIF file found to display"
-          fi
      - name: Upload result to GitHub Code Scanning
-        uses: github/codeql-action/upload-sarif@c10b8064de6f491fea524254123dbe5e09572f13 # v4
+        uses: github/codeql-action/upload-sarif@c10b8064de6f491fea524254123dbe5e09572f13 # v4.35.1
        if: always()
        with:
          sarif_file: snyk.sarif
+          category: snyk-container-worker
+      - name: Echo SARIF file for debugging
+        if: failure() && hashFiles('snyk.sarif') != ''
+        run: cat snyk.sarif
@@ -10,7 +10,7 @@ jobs:
      issues: write
      pull-requests: write
    steps:
-      - uses: actions/stale@5bef64f19d7facfb25b37b414482c7164d639639 # v9
+      - uses: actions/stale@b5d41d4e1d5dceea10e7104786b73624c18a190f # v10.2.0
        with:
          days-before-issue-stale: 30
          days-before-issue-close: 14
@@ -19,7 +19,7 @@ jobs:
      pull-requests: read
    steps:
      - name: Validate PR title follows conventional commits
-        uses: amannn/action-semantic-pull-request@48f256284bd46cdaab1048c3721360e808335d50 # v6
+        uses: amannn/action-semantic-pull-request@48f256284bd46cdaab1048c3721360e808335d50 # v6.1.1
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        with:
@@ -0,0 +1,43 @@
+---
+name: Check GitHub Actions
+
+on:
+  workflow_dispatch:
+  push:
+    branches:
+      - "main"
+  merge_group:
+  pull_request:
+    branches:
+      - "main"
+
+permissions: {}
+
+jobs:
+  zizmor:
+    name: Check GitHub Actions security
+    runs-on: ubuntu-latest
+    permissions:
+      security-events: write
+      contents: read
+    steps:
+      - name: Checkout
+        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        with:
+          persist-credentials: false
+
+      - name: Run zizmor (fork PR - fail on findings)
+        # Fork PRs don't get security-events: write on GITHUB_TOKEN, so SARIF
+        # upload isn't possible. Fail the job on findings instead so contributors
+        # see the error directly in the PR check.
+        if: github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name != github.repository
+        uses: zizmorcore/zizmor-action@b1d7e1fb5de872772f31590499237e7cce841e8e # v0.5.3
+
+      - name: Run zizmor (trusted - upload SARIF)
+        if: github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository
+        uses: zizmorcore/zizmor-action@b1d7e1fb5de872772f31590499237e7cce841e8e # v0.5.3
+        with:
+          # Means that the action will only report issues, but not fail the workflow.
+          # Blocking merges are handled by rulesets:
+          # https://docs.github.com/en/code-security/concepts/code-scanning/about-code-scanning-alerts#pull-request-check-failures-for-code-scanning-alerts
+          advanced-security: true
@@ -0,0 +1,21 @@
+rules:
+  secrets-outside-env:
+    config:
+      allow:
+        # Shared read-only CI credentials used to avoid Docker Hub rate limits in test jobs.
+        - DOCKERHUB_USERNAME_READ
+        - DOCKERHUB_TOKEN_READ
+        # Shared integration-test credentials intentionally kept together for the multi-provider LLM connection test job.
+        - LANGFUSE_LLM_CONNECTION_OPENAI_KEY
+        - LANGFUSE_LLM_CONNECTION_ANTHROPIC_KEY
+        - LANGFUSE_LLM_CONNECTION_AZURE_KEY
+        - LANGFUSE_LLM_CONNECTION_AZURE_BASE_URL
+        - LANGFUSE_LLM_CONNECTION_AZURE_MODEL
+        - LANGFUSE_LLM_CONNECTION_BEDROCK_ACCESS_KEY_ID
+        - LANGFUSE_LLM_CONNECTION_BEDROCK_SECRET_ACCESS_KEY
+        - LANGFUSE_LLM_CONNECTION_BEDROCK_API_KEY
+        - LANGFUSE_LLM_CONNECTION_BEDROCK_REGION
+        - LANGFUSE_LLM_CONNECTION_VERTEXAI_KEY
+        - LANGFUSE_LLM_CONNECTION_GOOGLEAISTUDIO_KEY
+        # Fern token is generation-only; GitHub write actions are handled by GH_ACCESS_TOKEN in a protected environment.
+        - FERN_TOKEN
@@ -47,6 +47,7 @@ yarn-error.log*
 !.env.dev-redis-cluster.example
 !.env.prod.example
 !.env.test.example
+!.env.dev-oci.example

 # vercel
 .vercel
@@ -92,3 +93,7 @@ web/test-results/*

 # Refactoring planning files (local only)
 **/.refactor/
+**/.pi-lens/
+
+# Serena MCP server config (local only)
+.serena/
@@ -316,18 +316,16 @@ Tests automatically create the PostgreSQL test database if it doesn't exist and

 ### Tests in the `web` package (public API)

-We're using Jest with in the `web` package. Therefore, if you want to provide an argument to the test runner, do it directly without an intermittent `--`.
-
-There are two types of unit tests:
+We're using Vitest in the `web` package. There are two types of unit tests:

 - `test` (server tests)
 - `test-client`

-To run a specific test, for example the test: `"should handle special characters in prompt names"` in `prompts.v2.servertest.ts`, run:
+To run a specific test by name within a file, run:

 ```sh
 cd web  # or with --filter=web
-pnpm test --testPathPatterns="prompts\.v2\.servertest" --testNamePattern="should handle special characters in prompt names"
+pnpm test -- prompts.v2.servertest -t "should handle special characters in prompt names"
 ```

 To run all tests:
@@ -344,7 +342,7 @@ pnpm run test:watch

 ### Tests in the `worker` package

-For the `worker` package, we're using `vitest` to run unit tests.
+For the `worker` package, we're also using Vitest to run unit tests.

 ```sh
 pnpm run test --filter=worker -- FILE_YOU_WANT_TO_TEST.ts -t "test name"
@@ -357,7 +355,7 @@ We use GitHub Actions for CI/CD, the configuration is in [`.github/workflows/pip
 CI on `main` and `pull_request`

 - Check Linting
- E2E test of API using Jest
+- E2E test of API using Vitest
 - E2E tests of UI using Playwright

 CD on `main`
@@ -31,6 +31,8 @@ services:
      CLICKHOUSE_PASSWORD: ${CLICKHOUSE_PASSWORD:-clickhouse} # CHANGEME
      CLICKHOUSE_CLUSTER_ENABLED: ${CLICKHOUSE_CLUSTER_ENABLED:-false}
      LANGFUSE_USE_AZURE_BLOB: ${LANGFUSE_USE_AZURE_BLOB:-false}
+      LANGFUSE_USE_OCI_NATIVE_OBJECT_STORAGE: ${LANGFUSE_USE_OCI_NATIVE_OBJECT_STORAGE:-false}
+      LANGFUSE_OCI_AUTH_TYPE: ${LANGFUSE_OCI_AUTH_TYPE:-workload_identity}
      LANGFUSE_S3_EVENT_UPLOAD_BUCKET: ${LANGFUSE_S3_EVENT_UPLOAD_BUCKET:-langfuse}
      LANGFUSE_S3_EVENT_UPLOAD_REGION: ${LANGFUSE_S3_EVENT_UPLOAD_REGION:-auto}
      LANGFUSE_S3_EVENT_UPLOAD_ACCESS_KEY_ID: ${LANGFUSE_S3_EVENT_UPLOAD_ACCESS_KEY_ID:-minio}
@@ -29,7 +29,7 @@
    "@langfuse/shared": "workspace:*",
    "@opentelemetry/api": ">=1.0.0 <1.10.0",
    "https-proxy-agent": "^7.0.6",
-    "next": "16.2.3",
+    "next": "16.2.6",
    "next-auth": "^4.24.13",
    "zod": "^4.3.6"
  },
@@ -39,7 +39,7 @@
    "@types/node": "^24.3.0",
    "@typescript/native-preview": "7.0.0-dev.20260122.3",
    "eslint": "^9.39.2",
-    "prettier": "^3.8.1",
+    "prettier": "^3.8.3",
    "ts-node": "^10.9.2",
    "tsc-watch": "^6.2.0",
    "typescript": "^5.7.2"
@@ -32,6 +32,9 @@ types:
      configId:
        type: optional<string>
        docs: Reference a score config on a score. When set, the score name must equal the config name and scores must comply with the config's range and data type. For categorical scores, the value must map to a config category. Numeric scores might be constrained by the score config's max and min values
+      source:
+        type: optional<CreateScoreSource>
+        docs: The source of the score. Defaults to API. Set to ANNOTATION to prefill scores (e.g. from an LLM) for a human reviewer to verify in an annotation queue. When source is ANNOTATION, a configId is required unless dataType is CORRECTION. EVAL is reserved for internal evaluator outputs and is not accepted on this endpoint.
    examples:
      - value:
          name: "novelty"
@@ -77,6 +80,22 @@ types:
          name: "hallucination"
          value: 0
          datasetRunId: "7891-5678-90ab-hijk"
+      - value:
+          name: "accuracy"
+          value: 0.9
+          dataType: "NUMERIC"
+          configId: "9203-4567-89ab-cdef"
+          traceId: "cdef-1234-5678-90ab"
+          source: "ANNOTATION"
+
+  CreateScoreSource:
+    docs: |
+      Source values accepted when creating a score via the public API.
+      EVAL is reserved for internal evaluator outputs and is intentionally not
+      exposed here.
+    enum:
+      - API
+      - ANNOTATION

  ScoreDataType:
    enum:
@@ -56,10 +56,39 @@ types:

  BlobStorageExportFrequency:
    enum:
+      - every_20_minutes
      - hourly
      - daily
      - weekly

+  BlobStorageExportSource:
+    docs: |
+      What data the integration exports.
+      - `TRACES_OBSERVATIONS`: legacy traces + observations + scores tables with a fixed column set. The `exportFieldGroups` field is not applicable.
+      - `EVENTS`: enriched observations_v2 events; columns are controlled by `exportFieldGroups`.
+      - `TRACES_OBSERVATIONS_EVENTS`: both sets. For the `EVENTS` portion, columns are controlled by `exportFieldGroups`.
+
+      **Note:** `EVENTS` and the events portion of `TRACES_OBSERVATIONS_EVENTS` rely on the observations_v2 events table (Langfuse Fast Preview / v4), which is currently available on Langfuse Cloud only. See https://langfuse.com/docs/v4.
+    enum:
+      - EVENTS
+      - TRACES_OBSERVATIONS
+      - TRACES_OBSERVATIONS_EVENTS
+
+  BlobStorageExportFieldGroup:
+    docs: Field group for the EVENTS export.
+    enum:
+      - core
+      - basic
+      - time
+      - io
+      - metadata
+      - model
+      - usage
+      - prompt
+      - metrics
+      - tools
+      - trace_context
+
  CreateBlobStorageIntegrationRequest:
    properties:
      projectId:
@@ -99,6 +128,21 @@ types:
      compressed:
        type: optional<boolean>
        docs: Enable gzip compression for exported files (.csv.gz, .json.gz, .jsonl.gz). Defaults to true.
+      exportSource:
+        type: optional<BlobStorageExportSource>
+        docs: |
+          Data to export. When omitted on update, the existing value is preserved; when omitted on create, the column default (`TRACES_OBSERVATIONS`) applies. Required when `exportFieldGroups` is provided.
+
+      exportFieldGroups:
+        type: optional<list<BlobStorageExportFieldGroup>>
+        docs: |
+          Field groups to include in each exported row.
+
+          For exportSource `EVENTS` or `TRACES_OBSERVATIONS_EVENTS`: must include `core` if provided. When omitted on create, the column default (all groups) applies. When omitted on update, the existing value is preserved.
+
+          For exportSource `TRACES_OBSERVATIONS`: this field must be omitted or null. Sending an array (including an empty array) returns 400, because that source uses a fixed column set and does not honor field groups.
+
+          `exportFieldGroups` requires `exportSource` to be provided in the same request.

  BlobStorageIntegrationResponse:
    properties:
@@ -117,6 +161,11 @@ types:
      exportMode: BlobStorageExportMode
      exportStartDate: nullable<datetime>
      compressed: boolean
+      exportSource: BlobStorageExportSource
+      exportFieldGroups:
+        type: nullable<list<BlobStorageExportFieldGroup>>
+        docs: |
+          Field groups included in each exported row for `EVENTS` / `TRACES_OBSERVATIONS_EVENTS` sources. Always `null` when exportSource is `TRACES_OBSERVATIONS` (the field does not apply to that source; any legacy DB value is hidden from the public surface).
      nextSyncAt: nullable<datetime>
      lastSyncAt: nullable<datetime>
      lastError: nullable<string>
@@ -138,8 +187,8 @@ types:
      - `up_to_date` — all available data has been exported; next export is scheduled for the future

      **ETL usage**: poll this endpoint and check for `up_to_date` status. Compare `lastSyncAt` against your
-      ETL bookmark to determine if new data is available. Note that exports run with a 30-minute lag buffer,
-      so `lastSyncAt` will always be at least 30 minutes behind real-time.
+      ETL bookmark to determine if new data is available. Note that exports run with a 20-minute lag buffer,
+      so `lastSyncAt` will always be at least 20 minutes behind real-time.
    enum:
      - idle
      - queued
@@ -46,6 +46,9 @@ types:
      configId:
        type: optional<string>
        docs: Reference a score config on a score. The unique langfuse identifier of a score config. When passing this field, the dataType and stringValue fields are automatically populated.
+      source:
+        type: optional<CreateScoreSource>
+        docs: The source of the score. Defaults to API. Set to ANNOTATION to prefill scores (e.g. from an LLM) for a human reviewer to verify in an annotation queue. When source is ANNOTATION, a configId is required unless dataType is CORRECTION. EVAL is reserved for internal evaluator outputs and is not accepted on this endpoint.
    examples:
      - value:
          name: "novelty"
@@ -90,6 +93,22 @@ types:
          value: "Great explanation of the concept"
          dataType: "TEXT"
          traceId: "cdef-1234-5678-90ab"
+      - value:
+          name: "accuracy"
+          value: 0.9
+          dataType: "NUMERIC"
+          configId: "9203-4567-89ab-cdef"
+          traceId: "cdef-1234-5678-90ab"
+          source: "ANNOTATION"
+          queueId: "aq-1234-5678-90ab-cdef"
+  CreateScoreSource:
+    docs: |
+      Source values accepted when creating a score via the public REST API.
+      EVAL is reserved for internal evaluator outputs and is intentionally not
+      exposed here — use commons.ScoreSource when reading scores.
+    enum:
+      - API
+      - ANNOTATION
  CreateScoreResponse:
    properties:
      id:
@@ -26,6 +26,13 @@ service:
      path: /llm-connections
      request: UpsertLlmConnectionRequest
      response: LlmConnection
+    delete:
+      method: DELETE
+      docs: Delete an LLM connection by id. Evaluators that depend on the deleted connection are automatically paused.
+      path: /llm-connections/{id}
+      path-parameters:
+        id: string
+      response: DeleteLlmConnectionResponse

 types:
  LlmConnection:
@@ -96,6 +103,10 @@ types:
          - **VertexAI**: Optional. If provided, must be `{"location": "<gcp-location>"}` (e.g., `{"location":"us-central1"}`)
          - **Other adapters**: Not supported. Omit this field or set to null.

+  DeleteLlmConnectionResponse:
+    properties:
+      message: string
+
  LlmAdapter:
    enum:
      - value: anthropic
@@ -54,7 +54,9 @@ types:
      meta: pagination.MetaResponse
  CreateScoreConfigRequest:
    properties:
-      name: string
+      name:
+        type: string
+        docs: "Name of the score config. Max 35 characters. Only letters, numbers, underscores, spaces, periods, parentheses, and hyphens are allowed."
      dataType: commons.ScoreConfigDataType
      categories:
        type: optional<list<commons.ConfigCategory>>
@@ -75,7 +77,7 @@ types:
        docs: The status of the score config showing if it is archived or not
      name:
        type: optional<string>
-        docs: The name of the score config
+        docs: "Name of the score config. Max 35 characters. Only letters, numbers, underscores, spaces, periods, parentheses, and hyphens are allowed."
      categories:
        type: optional<list<commons.ConfigCategory>>
        docs: Configure custom categories for categorical scores. Pass a list of objects with `label` and `value` properties. Categories are autogenerated for boolean configs and cannot be passed
@@ -18,7 +18,7 @@ service:
            docs: Page number, starts at 1.
          limit:
            type: optional<integer>
-            docs: Limit of items per page. If you encounter api issues due to too large page sizes, try to reduce the limit.
+            docs: Limit of items per page. Maximum 100. Defaults to 50. Requests with a limit greater than 100 return HTTP 400. If you encounter api issues due to too large page sizes, try to reduce the limit.
          userId:
            type: optional<string>
            docs: Retrieve only scores with this userId associated to the trace.
@@ -0,0 +1,564 @@
+# yaml-language-server: $schema=https://raw.githubusercontent.com/fern-api/fern/main/fern.schema.json
+types:
+  EvaluatorType:
+    docs: |
+      The evaluator engine type.
+
+      The unstable public API currently supports only LLM-as-a-judge evaluators.
+    enum:
+      - llm_as_judge
+
+  EvaluatorScope:
+    docs: |
+      Where an evaluator comes from.
+
+      - `project`: created in your project
+      - `managed`: provided by Langfuse
+    enum:
+      - project
+      - managed
+
+  EvaluationRuleTarget:
+    docs: |
+      The ingestion object type that should trigger evaluation runs.
+
+      Choose the target first, because it changes both the valid filter columns and the valid variable-mapping sources:
+      - `observation` evaluates live-ingested observations such as generations, spans, and events.
+        It supports mapping from `input`, `output`, and `metadata`.
+      - `experiment` evaluates live experiment executions and can additionally map `expected_output`.
+        It currently supports filtering by `datasetId`.
+        Discover valid dataset IDs with `GET /api/public/v2/datasets`, then use the returned dataset `id` values in your filter.
+    enum:
+      - observation
+      - experiment
+
+  EvaluationRuleStatus:
+    docs: |
+      Effective runtime status of the evaluation rule.
+
+      - `active`: enabled and currently runnable.
+      - `inactive`: disabled by configuration.
+      - `paused`: enabled, but Langfuse has blocked execution until the underlying issue is resolved.
+    enum:
+      - active
+      - inactive
+      - paused
+
+  EvaluationRuleMappingSource:
+    docs: |
+      Source field used to populate a prompt variable.
+
+      Use these values when mapping evaluator prompt variables to live data.
+
+      Target-specific rules:
+      - `target=observation` supports `input`, `output`, and `metadata`
+      - `target=experiment` supports `input`, `output`, `metadata`, and `expected_output`
+
+      Source semantics:
+      - `input`: the observation or experiment input payload
+      - `output`: the observation or experiment output payload
+      - `metadata`: the metadata object for the target. Combine with `jsonPath` when you need one nested field instead of the whole object.
+      - `expected_output`: the experiment item's expected output. Only valid for `target=experiment`.
+    enum:
+      - input
+      - output
+      - metadata
+      - expected_output
+
+  EvaluatorModelConfig:
+    docs: |
+      Optional explicit model configuration for an evaluator.
+
+      If omitted, Langfuse uses the project's default evaluation model.
+      If provided, the model must be available to the project when the evaluator or evaluation rule is enabled.
+
+      To discover valid configured `provider` values for a project, call `GET /api/public/llm-connections` and read the `provider` field from the returned connections.
+      Use a `provider` value that matches one of the connections already configured in the same project.
+
+      Recovery guidance:
+      - If evaluator creation returns `422` with `code=evaluator_preflight_failed`, either provide a valid explicit `modelConfig` here or configure the project's default evaluation model, then retry the same request.
+    properties:
+      provider:
+        type: string
+        docs: |
+          Provider identifier to use for this evaluator, for example `openai` or `anthropic`.
+
+          To discover valid values for the current project, call `GET /api/public/llm-connections` and use one of the returned `provider` values.
+      model:
+        type: string
+        docs: Model identifier exposed by the provider, for example `gpt-4.1-mini`.
+    examples:
+      - value:
+          provider: openai
+          model: gpt-4.1-mini
+
+  EvaluatorOutputDataType:
+    docs: |
+      Structured score type returned by an evaluator.
+
+      This controls the type of score value Langfuse stores for evaluation results:
+      - `NUMERIC`: a numeric score such as `0.82`
+      - `BOOLEAN`: a boolean score such as `true`
+      - `CATEGORICAL`: one or more category labels from a fixed list
+    enum:
+      - NUMERIC
+      - BOOLEAN
+      - CATEGORICAL
+
+  EvaluatorOutputFieldDefinition:
+    properties:
+      description:
+        type: string
+        docs: Human-readable instructions for what the evaluator should return in this field.
+
+  EvaluatorOutputDefinition:
+    docs: |
+      Structured output definition to send when creating an evaluator.
+
+      Agent guidance:
+      - `dataType` is required.
+      - Do not send `version`; that is an internal storage detail and is not part of the public request contract.
+      - For `NUMERIC` and `BOOLEAN`, provide `reasoning.description` and `score.description`.
+      - For `CATEGORICAL`, also provide `score.categories` and `score.shouldAllowMultipleMatches`.
+    discriminant: dataType
+    union:
+      NUMERIC:
+        type: PublicNumericEvaluatorOutputDefinition
+      BOOLEAN:
+        type: PublicBooleanEvaluatorOutputDefinition
+      CATEGORICAL:
+        type: PublicCategoricalEvaluatorOutputDefinition
+    examples:
+      - name: Numeric
+        value:
+          dataType: NUMERIC
+          reasoning:
+            description: Explain why the answer is correct or incorrect.
+          score:
+            description: Return a score between 0 and 1.
+      - name: Boolean
+        value:
+          dataType: BOOLEAN
+          reasoning:
+            description: Explain why the output satisfies the requirement.
+          score:
+            description: Return true if the output satisfies the requirement, otherwise false.
+      - name: Categorical
+        value:
+          dataType: CATEGORICAL
+          reasoning:
+            description: Explain which category best fits the output.
+          score:
+            description: Choose the best category.
+            categories:
+              - correct
+              - partially_correct
+              - incorrect
+            shouldAllowMultipleMatches: false
+
+  PublicNumericEvaluatorOutputDefinition:
+    properties:
+      dataType:
+        type: EvaluatorOutputDataType
+        docs: Always `NUMERIC`.
+      reasoning:
+        type: EvaluatorOutputFieldDefinition
+      score:
+        type: EvaluatorOutputFieldDefinition
+
+  PublicBooleanEvaluatorOutputDefinition:
+    properties:
+      dataType:
+        type: EvaluatorOutputDataType
+        docs: Always `BOOLEAN`.
+      reasoning:
+        type: EvaluatorOutputFieldDefinition
+      score:
+        type: EvaluatorOutputFieldDefinition
+
+  PublicCategoricalEvaluatorOutputScoreDefinition:
+    properties:
+      description:
+        type: string
+      categories:
+        type: list<string>
+      shouldAllowMultipleMatches:
+        type: boolean
+
+  PublicCategoricalEvaluatorOutputDefinition:
+    properties:
+      dataType:
+        type: EvaluatorOutputDataType
+        docs: Always `CATEGORICAL`.
+      reasoning:
+        type: EvaluatorOutputFieldDefinition
+      score:
+        type: PublicCategoricalEvaluatorOutputScoreDefinition
+
+  PublicEvaluatorOutputDefinition:
+    docs: |
+      Evaluator output definition returned by the public API.
+
+      This response always includes `dataType` and never includes an internal output-definition `version`.
+      Legacy stored evaluator definitions are normalized into this shape before they are returned.
+
+      Use this response shape when deciding how to interpret future evaluation scores:
+      - `NUMERIC`: expect numeric score values
+      - `BOOLEAN`: expect `true` / `false`
+      - `CATEGORICAL`: expect one or more values from `score.categories`
+    discriminant: dataType
+    union:
+      NUMERIC:
+        type: PublicNumericEvaluatorOutputDefinition
+      BOOLEAN:
+        type: PublicBooleanEvaluatorOutputDefinition
+      CATEGORICAL:
+        type: PublicCategoricalEvaluatorOutputDefinition
+    examples:
+      - name: PublicNumeric
+        value:
+          dataType: NUMERIC
+          reasoning:
+            description: Explain why the answer is correct or incorrect.
+          score:
+            description: Return a score between 0 and 1.
+      - name: PublicCategorical
+        value:
+          dataType: CATEGORICAL
+          reasoning:
+            description: Explain which label best fits the output.
+          score:
+            description: Choose the best label.
+            categories:
+              - correct
+              - partially_correct
+              - incorrect
+            shouldAllowMultipleMatches: false
+
+  EvaluationRuleStringFilterOperator:
+    enum:
+      - name: Equals
+        value: "="
+      - name: Contains
+        value: contains
+      - name: DoesNotContain
+        value: does not contain
+      - name: StartsWith
+        value: starts with
+      - name: EndsWith
+        value: ends with
+
+  EvaluationRuleNumberFilterOperator:
+    enum:
+      - name: Equals
+        value: "="
+      - name: GreaterThan
+        value: ">"
+      - name: LessThan
+        value: "<"
+      - name: GreaterThanOrEqual
+        value: ">="
+      - name: LessThanOrEqual
+        value: "<="
+
+  EvaluationRuleOptionsFilterOperator:
+    enum:
+      - name: AnyOf
+        value: any of
+      - name: NoneOf
+        value: none of
+
+  EvaluationRuleArrayOptionsFilterOperator:
+    enum:
+      - name: AnyOf
+        value: any of
+      - name: NoneOf
+        value: none of
+      - name: AllOf
+        value: all of
+
+  EvaluationRuleBooleanFilterOperator:
+    enum:
+      - name: Equals
+        value: "="
+      - name: NotEquals
+        value: "<>"
+
+  EvaluationRuleNullFilterOperator:
+    enum:
+      - name: IsNull
+        value: is null
+      - name: IsNotNull
+        value: is not null
+
+  DateTimeEvaluationRuleFilter:
+    properties:
+      column:
+        type: string
+        docs: Column to filter on.
+      operator:
+        type: EvaluationRuleNumberFilterOperator
+        docs: Comparison operator for datetime values.
+      value:
+        type: datetime
+        docs: Datetime value to compare against.
+
+  StringEvaluationRuleFilter:
+    properties:
+      column:
+        type: string
+        docs: Column to filter on.
+      operator:
+        type: EvaluationRuleStringFilterOperator
+      value:
+        type: string
+
+  NumberEvaluationRuleFilter:
+    properties:
+      column:
+        type: string
+        docs: Column to filter on.
+      operator:
+        type: EvaluationRuleNumberFilterOperator
+      value:
+        type: double
+
+  StringOptionsEvaluationRuleFilter:
+    properties:
+      column:
+        type: string
+        docs: Column to filter on.
+      operator:
+        type: EvaluationRuleOptionsFilterOperator
+      value:
+        type: list<string>
+        docs: One or more allowed string values.
+
+  ArrayOptionsEvaluationRuleFilter:
+    properties:
+      column:
+        type: string
+        docs: Column to filter on.
+      operator:
+        type: EvaluationRuleArrayOptionsFilterOperator
+      value:
+        type: list<string>
+        docs: One or more array elements to match.
+
+  StringObjectEvaluationRuleFilter:
+    properties:
+      column:
+        type: string
+        docs: Object-valued column to filter on. In the unstable public API this is currently `metadata`.
+      key:
+        type: string
+        docs: Top-level key inside the object-valued column to filter on.
+      operator:
+        type: EvaluationRuleStringFilterOperator
+      value:
+        type: string
+
+  NumberObjectEvaluationRuleFilter:
+    properties:
+      column:
+        type: string
+        docs: Object-valued column to filter on.
+      key:
+        type: string
+        docs: Key inside the object-valued column to filter on.
+      operator:
+        type: EvaluationRuleNumberFilterOperator
+      value:
+        type: double
+
+  CategoryOptionsEvaluationRuleFilter:
+    properties:
+      column:
+        type: string
+        docs: Object-valued column to filter on.
+      key:
+        type: string
+        docs: Key inside the object-valued column to filter on.
+      operator:
+        type: EvaluationRuleOptionsFilterOperator
+      value:
+        type: list<string>
+
+  BooleanEvaluationRuleFilter:
+    properties:
+      column:
+        type: string
+        docs: Column to filter on.
+      operator:
+        type: EvaluationRuleBooleanFilterOperator
+      value:
+        type: boolean
+
+  NullEvaluationRuleFilter:
+    properties:
+      column:
+        type: string
+        docs: Column to filter on. In the unstable public API this is currently `parentObservationId`.
+      operator:
+        type: EvaluationRuleNullFilterOperator
+      value:
+        type: optional<string>
+        docs: Ignored placeholder value. Clients may omit it or send an empty string.
+
+  EvaluationRuleMapping:
+    docs: |
+      Maps one evaluator prompt variable to one source field from the target object.
+
+      How to build a valid mapping list:
+      1. Create the evaluator or fetch it with `GET /evaluators/{id}`.
+      2. Read the evaluator `variables` array.
+      3. Add exactly one mapping object for each variable in that array.
+      4. Use the variable name exactly as returned, without braces such as `{{` or `}}`.
+      5. Choose a `source` that is valid for the selected `target`.
+
+      `jsonPath` is optional. Use it only when the selected source is a JSON object and you want to extract one nested field before inserting it into the evaluator prompt.
+
+      Recovery guidance:
+      - `invalid_variable_mapping`: the variable name is unknown for this evaluator, or the selected `source` is not valid for the chosen `target`
+      - `missing_variable_mapping`: one or more evaluator variables are not mapped yet
+      - `duplicate_variable_mapping`: the same evaluator variable appears more than once
+      - `invalid_json_path`: the JSONPath expression is malformed. Remove it or correct it.
+    properties:
+      variable:
+        type: string
+        docs: |
+          Prompt variable name without braces.
+
+          Example: for the prompt `Judge {{input}} against {{output}}`, use `input` and `output`.
+      source:
+        type: EvaluationRuleMappingSource
+        docs: |
+          Source field that should populate the prompt variable.
+
+          Quick reference:
+          - `target=observation`: `input`, `output`, `metadata`
+          - `target=experiment`: `input`, `output`, `metadata`, `expected_output`
+      jsonPath:
+        type: optional<string>
+        docs: |
+          Optional JSONPath selector applied to the selected source before it is passed to the evaluator prompt.
+
+          Requirements:
+          - Must start with `$`
+          - Must be a syntactically valid JSONPath expression
+          - Most useful with `source=metadata`
+    examples:
+      - name: BasicObservationMapping
+        value:
+          variable: input
+          source: input
+      - name: MetadataProjectionMapping
+        value:
+          variable: customer_tier
+          source: metadata
+          jsonPath: "$.customer.tier"
+      - name: ExperimentExpectedOutputMapping
+        value:
+          variable: expected_output
+          source: expected_output
+
+  EvaluationRuleFilter:
+    docs: |
+      One filter condition used to decide whether a live-ingested target should be evaluated.
+
+      An evaluation rule can include zero or more filter objects. All filters must be satisfied for the target to run.
+
+      How to build a valid filter object:
+      - Pick the `target` first, because it changes the supported columns.
+      - Pick the filter `type`. That determines which fields are required.
+      - Use `key` only for object filters such as `metadata`.
+      - Use the correct `value` shape for the chosen filter `type`.
+
+      Operator quick reference by filter `type`:
+      - `string`: `"="`, `contains`, `does not contain`, `starts with`, `ends with`
+      - `number`: `"="`, `">"`, `"<"`, `">="`, `"<="`
+      - `datetime`: `"="`, `">"`, `"<"`, `">="`, `"<="`
+      - `stringOptions`: `any of`, `none of`
+      - `arrayOptions`: `any of`, `none of`, `all of`
+      - `stringObject`: same operators as `string`
+      - `null`: `is null`, `is not null`
+
+      Supported columns by target:
+      - `target=observation`
+        - `type`: `stringOptions`, operators `any of` / `none of`, values `GENERATION`, `SPAN`, `EVENT`
+        - `name`: `stringOptions`, operators `any of` / `none of`
+        - `environment`: `stringOptions`, operators `any of` / `none of`
+        - `level`: `stringOptions`, operators `any of` / `none of`, values `DEBUG`, `DEFAULT`, `WARNING`, `ERROR`
+        - `version`: `string`
+        - `traceName`: `stringOptions`, operators `any of` / `none of`
+        - `userId`: `string`
+        - `sessionId`: `string`
+        - `tags`: `arrayOptions`, operators `any of` / `none of` / `all of`
+        - `metadata`: `stringObject` with `key`
+        - `parentObservationId`: `null`, operators `is null` / `is not null`
+        - `calledToolNames`: `arrayOptions`, operators `any of` / `none of` / `all of`
+        - `toolCalls`: `number`
+      - `target=experiment`
+        - `datasetId`: `stringOptions`, operators `any of` / `none of`
+          Use dataset `id` values from `GET /api/public/v2/datasets`, not dataset names.
+
+      Recovery guidance:
+      - `invalid_filter_value` with `details.column` but no `invalidValues`: the selected `column` is not supported for the chosen `target`
+      - `invalid_filter_value` with `details.invalidValues`: the selected values are not allowed for that column. Replace them with one of `details.allowedValues` when provided.
+      - `invalid_filter_value` for `column=datasetId`: call `GET /api/public/v2/datasets`, then retry with dataset `id` values from that response.
+    discriminant: type
+    union:
+      "datetime":
+        type: DateTimeEvaluationRuleFilter
+      "string":
+        type: StringEvaluationRuleFilter
+      "number":
+        type: NumberEvaluationRuleFilter
+      "stringOptions":
+        type: StringOptionsEvaluationRuleFilter
+      "categoryOptions":
+        type: CategoryOptionsEvaluationRuleFilter
+      "arrayOptions":
+        type: ArrayOptionsEvaluationRuleFilter
+      "stringObject":
+        type: StringObjectEvaluationRuleFilter
+      "numberObject":
+        type: NumberObjectEvaluationRuleFilter
+      "boolean":
+        type: BooleanEvaluationRuleFilter
+      "null":
+        type: NullEvaluationRuleFilter
+    examples:
+      - name: ObservationTypeFilter
+        value:
+          type: stringOptions
+          column: type
+          operator: any of
+          value:
+            - GENERATION
+      - name: ObservationMetadataFilter
+        value:
+          type: stringObject
+          column: metadata
+          key: customerTier
+          operator: "="
+          value: enterprise
+      - name: ObservationRootOnlyFilter
+        value:
+          type: "null"
+          column: parentObservationId
+          operator: is null
+      - name: ObservationTagsFilter
+        value:
+          type: arrayOptions
+          column: tags
+          operator: any of
+          value:
+            - production
+      - name: ExperimentDatasetFilter
+        value:
+          type: stringOptions
+          column: datasetId
+          operator: any of
+          value:
+            - "550e8400-e29b-41d4-a716-446655440000"
@@ -0,0 +1,269 @@
+# yaml-language-server: $schema=https://raw.githubusercontent.com/fern-api/fern/main/fern.schema.json
+types:
+  PublicApiErrorCode:
+    docs: |
+      Machine-readable error code returned by the unstable evaluators API.
+
+      SDKs, CLIs, and agents should branch on `code` rather than parsing the human-readable `message`.
+      The HTTP status still indicates the broad error class, while `code` gives the specific failure reason.
+    enum:
+      - authentication_failed
+      - access_denied
+      - invalid_request
+      - invalid_query
+      - invalid_body
+      - invalid_filter_value
+      - invalid_json_path
+      - invalid_variable_mapping
+      - missing_variable_mapping
+      - duplicate_variable_mapping
+      - resource_not_found
+      - name_conflict
+      - evaluator_preflight_failed
+      - conflict
+      - unprocessable_content
+      - rate_limited
+      - method_not_allowed
+      - internal_error
+
+  PublicApiValidationIssue:
+    docs: |
+      One validation issue returned for malformed request bodies or query parameters.
+
+      This mirrors the most important parts of a Zod issue: a machine-readable `code`,
+      a human-readable `message`, and a structured `path`.
+    properties:
+      code:
+        type: string
+        docs: Machine-readable validation issue code emitted by the server validator.
+      message:
+        type: string
+        docs: Human-readable explanation of the validation failure.
+      path:
+        type: list<unknown>
+        docs: Path to the invalid field, for example `["mapping", 0, "jsonPath"]`.
+
+  PublicApiErrorDetails:
+    docs: |
+      Optional structured context attached to an unstable-evals error.
+
+      The populated fields depend on the error `code`:
+      - request parsing failures populate `issues`
+      - filter validation failures populate `field`, `column`, `invalidValues`, and `allowedValues`
+      - variable mapping failures populate `field`, `variable`, or `variables`
+      - JSONPath validation failures populate `field`, `variable`, and `value`
+      - evaluator preflight failures populate `evaluatorName`, `provider`, and `model`
+      - rate limiting populates `retryAfterSeconds`, `limit`, `remaining`, and `resetAt`
+    properties:
+      issues:
+        type: optional<list<PublicApiValidationIssue>>
+        docs: Validation issues for malformed request bodies or query parameters.
+      field:
+        type: optional<string>
+        docs: Path-like reference to the failing field, for example `mapping[1].jsonPath`.
+      column:
+        type: optional<string>
+        docs: Filter column that failed validation.
+      invalidValues:
+        type: optional<list<string>>
+        docs: Unsupported values supplied by the caller.
+      allowedValues:
+        type: optional<list<string>>
+        docs: Allowed values for the failing filter column.
+      variable:
+        type: optional<string>
+        docs: Evaluator variable involved in the failure.
+      variables:
+        type: optional<list<string>>
+        docs: Multiple evaluator variables involved in the failure, for example missing mappings.
+      value:
+        type: optional<string>
+        docs: Raw invalid value supplied by the caller.
+      evaluatorName:
+        type: optional<string>
+        docs: Evaluator name used during preflight validation.
+      provider:
+        type: optional<nullable<string>>
+        docs: Provider resolved during evaluator preflight, if any.
+      model:
+        type: optional<nullable<string>>
+        docs: Model resolved during evaluator preflight, if any.
+      retryAfterSeconds:
+        type: optional<integer>
+        docs: Suggested retry delay for rate-limited requests.
+      limit:
+        type: optional<integer>
+        docs: Numeric limit associated with the failure, for example the active evaluation-rule cap or the current rate-limit window.
+      remaining:
+        type: optional<integer>
+        docs: Remaining requests in the current rate-limit window.
+      resetAt:
+        type: optional<string>
+        docs: ISO-8601 timestamp when the current rate-limit window resets.
+
+  PublicApiError:
+    docs: |
+      Standard error envelope for the unstable evaluators API.
+
+      Response handling guidance:
+      - Use the HTTP status code for the broad class of failure.
+      - Use `code` for precise branching in SDKs, CLIs, or agents.
+      - Inspect `details` for field-level validation context such as invalid filter values, malformed JSONPath expressions, or missing variable mappings.
+      - Retry only after fixing the specific issue described by `code` and `details`.
+    properties:
+      message:
+        type: string
+        docs: Human-readable description of the failure.
+      code:
+        type: PublicApiErrorCode
+        docs: Stable machine-readable error code.
+      details:
+        type: optional<PublicApiErrorDetails>
+        docs: Optional structured error context. Inspect the populated fields based on `code`.
+    examples:
+      - name: InvalidFilterValue
+        value:
+          message: 'Filter column "type" contains unsupported value(s): INVALID'
+          code: invalid_filter_value
+          details:
+            field: filter[0].value
+            column: type
+            invalidValues:
+              - INVALID
+            allowedValues:
+              - GENERATION
+              - SPAN
+              - EVENT
+      - name: InvalidExperimentDatasetFilterValue
+        value:
+          message: 'Filter column "datasetId" contains dataset id(s) that do not exist in this project: dataset-prod'
+          code: invalid_filter_value
+          details:
+            field: filter[0].value
+            column: datasetId
+            invalidValues:
+              - dataset-prod
+      - name: EvaluatorPreflightFailed
+        value:
+          message: 'No valid LLM model found for evaluator "answer-correctness". No default model or custom model configured for project project_123'
+          code: evaluator_preflight_failed
+          details:
+            evaluatorName: answer-correctness
+            provider: null
+            model: null
+      - name: MissingVariableMapping
+        value:
+          message: "Missing mappings for evaluator variable(s): output"
+          code: missing_variable_mapping
+          details:
+            field: mapping
+            variables:
+              - output
+      - name: InvalidJsonPath
+        value:
+          message: 'Mapping for variable "customer_tier" has an invalid jsonPath "$[". JSONPath expressions must use balanced quotes, brackets, and parentheses.'
+          code: invalid_json_path
+          details:
+            field: mapping[0].jsonPath
+            variable: customer_tier
+            value: "$["
+      - name: NameConflict
+        value:
+          message: 'An evaluation rule named "answer-quality-live" already exists in this project. Use PATCH /api/public/unstable/evaluation-rules/erule_123 to update it instead of creating a duplicate.'
+          code: name_conflict
+          details:
+            field: name
+      - name: ActiveEvaluationRuleLimit
+        value:
+          message: This project already has the maximum number of active evaluation rules (50). Disable an existing active evaluation rule before enabling another one.
+          code: conflict
+          details:
+            limit: 50
+      - name: RateLimited
+        value:
+          message: Rate limit exceeded
+          code: rate_limited
+          details:
+            retryAfterSeconds: 60
+            limit: 1000
+            remaining: 0
+            resetAt: "2026-03-30T10:00:00.000Z"
+
+errors:
+  BadRequestError:
+    docs: |
+      Request parsing or validation failed.
+
+      Typical unstable-eval examples:
+      - malformed request body or query parameters
+      - unsupported filter value
+      - invalid JSONPath selector
+      - missing or duplicate variable mappings
+
+      Recovery guidance:
+      - read `details.issues` for malformed bodies or queries
+      - read `details.column`, `details.invalidValues`, and `details.allowedValues` for filter problems
+      - for `details.column=datasetId`, call `GET /api/public/v2/datasets` and retry with dataset `id` values from that response
+      - read `details.variable` or `details.variables` for mapping problems
+    status-code: 400
+    type: PublicApiError
+  UnauthorizedError:
+    docs: |
+      Authentication failed.
+
+      Typical unstable-eval examples:
+      - the API key or secret key is invalid
+      - the request uses the wrong host or stale credentials
+    status-code: 401
+    type: PublicApiError
+  AccessDeniedError:
+    docs: |
+      Authenticated, but the caller is not allowed to access the requested resource.
+
+      Typical unstable-eval examples:
+      - using an organization-scoped key against project-scoped evaluator endpoints
+      - using a valid key that lacks the required access level for this resource
+    status-code: 403
+    type: PublicApiError
+  NotFoundError:
+    docs: The requested evaluator or evaluation rule does not exist within the authorized project.
+    status-code: 404
+    type: PublicApiError
+  MethodNotAllowedError:
+    docs: The HTTP method is not supported for this endpoint.
+    status-code: 405
+    type: PublicApiError
+  ConflictError:
+    docs: |
+      The request conflicts with existing state.
+
+      Typical unstable-eval examples:
+      - evaluator version changed during concurrent creation
+      - an evaluation rule name already exists in the project, so the caller should PATCH the existing resource instead
+      - enabling another evaluation rule would exceed the project limit of 50 active evaluation rules
+      - the underlying state changed between read and write operations, so the client should retry
+
+      Recovery guidance:
+      - for `name_conflict`, PATCH the existing evaluation rule instead of creating another one
+      - for the active-limit conflict, disable another active evaluation rule before enabling a new one
+    status-code: 409
+    type: PublicApiError
+  UnprocessableContentError:
+    docs: |
+      The request is syntactically valid, but Langfuse cannot accept it in its current form.
+
+      The most important unstable-eval case is `code=evaluator_preflight_failed`, which means the evaluator cannot currently run with the resolved model configuration.
+
+      Recovery guidance:
+      - configure the project's default evaluation model, or send a valid explicit evaluator `modelConfig`
+      - once the model configuration is valid, retry the same request
+    status-code: 422
+    type: PublicApiError
+  TooManyRequestsError:
+    docs: The project is rate limited. Retry after the interval described in the response headers and error details.
+    status-code: 429
+    type: PublicApiError
+  InternalServerError:
+    docs: An unexpected server-side error occurred.
+    status-code: 500
+    type: PublicApiError
@@ -0,0 +1,494 @@
+# yaml-language-server: $schema=https://raw.githubusercontent.com/fern-api/fern/main/fern.schema.json
+imports:
+  commons: ./commons.yml
+  errors: ./errors.yml
+  pagination: ../utils/pagination.yml
+
+service:
+  auth: true
+  base-path: /api/public/unstable
+  endpoints:
+    create:
+      docs: |
+        Create an evaluation rule.
+
+        An evaluation rule defines **what** incoming data should be evaluated and **how prompt variables should be populated** from that data.
+
+        Use this resource after choosing an evaluator from the evaluator endpoints.
+
+        Key rules:
+        - `name` must be unique within the project for public evaluation rules
+        - `target` must be `observation` or `experiment`
+        - `evaluator.name` + `evaluator.scope` must identify an existing evaluator family returned by the evaluator endpoints
+        - Langfuse resolves that family to its latest version before saving the evaluation rule
+        - for `target=experiment`, use dataset `id` values from `GET /api/public/v2/datasets` when filtering by `datasetId`
+        - every evaluator prompt variable must be mapped exactly once
+        - `expected_output` mappings are only valid for `target=experiment`
+        - if `enabled=true`, Langfuse validates that the referenced evaluator can currently run
+        - at most 50 evaluation rules can be effectively active in one project at the same time
+
+        If an evaluation rule with the same `name` already exists in the project, the API returns `409`.
+        In that case, update the existing resource with `PATCH /api/public/unstable/evaluation-rules/{evaluationRuleId}` instead of creating a second one.
+
+        If enabling this resource would exceed the 50-active limit, the API also returns `409`.
+        In that case, disable or pause another active evaluation rule before enabling a new one.
+
+        Current scope:
+        - evaluation rules are live-ingestion rules only
+        - they do not trigger historical backfills
+
+        Recovery guidance:
+        - `400 invalid_filter_value`: fix the filter `column` or `value` using `details.column`, `details.invalidValues`, and `details.allowedValues`
+        - `400 invalid_filter_value` with `details.column=datasetId`: call `GET /api/public/v2/datasets`, then retry with dataset `id` values from that response
+        - `400 missing_variable_mapping`: fetch the evaluator again and make sure every variable in `variables` appears exactly once in `mapping`
+        - `400 duplicate_variable_mapping`: remove repeated mappings for the same variable
+        - `400 invalid_variable_mapping`: switch to a valid `source` for the selected `target`, or fix the variable name
+        - `400 invalid_json_path`: remove or correct the `jsonPath`
+        - `422 evaluator_preflight_failed`: the selected evaluator cannot run with the resolved model configuration. Fix the evaluator/default model setup, then retry the create request.
+      method: POST
+      path: /evaluation-rules
+      request: CreateEvaluationRuleRequest
+      response: EvaluationRule
+      errors:
+        - errors.BadRequestError
+        - errors.UnauthorizedError
+        - errors.AccessDeniedError
+        - errors.NotFoundError
+        - errors.ConflictError
+        - errors.MethodNotAllowedError
+        - errors.UnprocessableContentError
+        - errors.TooManyRequestsError
+        - errors.InternalServerError
+      examples:
+        - name: CreateObservationEvaluationRule
+          docs: Deploy an evaluator to score live generations only.
+          request:
+            name: answer-correctness-live
+            evaluator:
+              name: answer-correctness
+              scope: project
+            target: observation
+            enabled: true
+            sampling: 1
+            filter:
+              - type: stringOptions
+                column: type
+                operator: any of
+                value:
+                  - GENERATION
+            mapping:
+              - variable: input
+                source: input
+              - variable: output
+                source: output
+          response:
+            body:
+              id: erule_123
+              name: answer-correctness-live
+              evaluator:
+                id: evaltmpl_123
+                name: answer-correctness
+                scope: project
+              target: observation
+              enabled: true
+              status: active
+              pausedReason: null
+              pausedMessage: null
+              sampling: 1
+              filter:
+                - type: stringOptions
+                  column: type
+                  operator: any of
+                  value:
+                    - GENERATION
+              mapping:
+                - variable: input
+                  source: input
+                - variable: output
+                  source: output
+              createdAt: "2026-03-30T09:20:00.000Z"
+              updatedAt: "2026-03-30T09:20:00.000Z"
+        - name: CreateExperimentEvaluationRule
+          docs: Deploy an evaluator to compare experiment outputs against expected outputs. Discover valid dataset IDs with `GET /api/public/v2/datasets` first.
+          request:
+            name: experiment-expected-output-match
+            evaluator:
+              name: expected-output-match
+              scope: project
+            target: experiment
+            enabled: true
+            sampling: 0.5
+            filter:
+              - type: stringOptions
+                column: datasetId
+                operator: any of
+                value:
+                  - "550e8400-e29b-41d4-a716-446655440000"
+            mapping:
+              - variable: output
+                source: output
+              - variable: expected_output
+                source: expected_output
+          response:
+            body:
+              id: erule_456
+              name: experiment-expected-output-match
+              evaluator:
+                id: evaltmpl_456
+                name: expected-output-match
+                scope: project
+              target: experiment
+              enabled: true
+              status: active
+              pausedReason: null
+              pausedMessage: null
+              sampling: 0.5
+              filter:
+                - type: stringOptions
+                  column: datasetId
+                  operator: any of
+                  value:
+                    - "550e8400-e29b-41d4-a716-446655440000"
+              mapping:
+                - variable: output
+                  source: output
+                - variable: expected_output
+                  source: expected_output
+              createdAt: "2026-03-30T09:30:00.000Z"
+              updatedAt: "2026-03-30T09:30:00.000Z"
+
+    list:
+      docs: |
+        List evaluation rules in the authenticated project.
+
+        Each item describes one live evaluation rule and its effective runtime status.
+      method: GET
+      path: /evaluation-rules
+      request:
+        name: ListEvaluationRulesRequest
+        query-parameters:
+          page:
+            type: optional<integer>
+            docs: 1-based page number. Defaults to `1`.
+          limit:
+            type: optional<integer>
+            docs: Maximum number of items per page. Defaults to `50`.
+      response: EvaluationRules
+      errors:
+        - errors.BadRequestError
+        - errors.UnauthorizedError
+        - errors.AccessDeniedError
+        - errors.MethodNotAllowedError
+        - errors.TooManyRequestsError
+        - errors.InternalServerError
+
+    get:
+      docs: |
+        Get one evaluation rule by its identifier.
+
+        Use this endpoint to inspect the current evaluator, target, mapping, filters, and effective runtime status.
+      method: GET
+      path: /evaluation-rules/{evaluationRuleId}
+      path-parameters:
+        evaluationRuleId:
+          type: string
+          docs: Evaluation rule identifier returned by the evaluation rule endpoints.
+      response: EvaluationRule
+      errors:
+        - errors.BadRequestError
+        - errors.UnauthorizedError
+        - errors.AccessDeniedError
+        - errors.NotFoundError
+        - errors.MethodNotAllowedError
+        - errors.TooManyRequestsError
+        - errors.InternalServerError
+
+    update:
+      docs: |
+        Update an evaluation rule.
+
+        Typical uses:
+        - enable or disable live execution
+        - switch to another evaluator
+        - adjust sampling
+        - change filters
+        - update variable mappings
+
+        Important behavior:
+        - provide only the fields you want to change
+        - if you provide `evaluator`, Langfuse resolves that evaluator family to its latest version before saving
+        - changing `target`, `filter`, or `mapping` must still produce a valid target-specific configuration
+        - if you change `target`, also send a compatible `filter` and `mapping` in the same request unless the existing ones are still valid for the new target
+        - if the resulting config is enabled, Langfuse re-validates that the selected evaluator can run
+        - if the update would move a non-active evaluation rule into the active state and the project already has 50 active evaluation rules, the API returns `409`
+
+        Recovery guidance:
+        - if the update fails with `missing_variable_mapping` or `invalid_variable_mapping` after changing `evaluator` or `target`, resend the request with a complete new `mapping`
+        - if the update fails with `invalid_filter_value` after changing `target`, resend the request with a target-compatible `filter`
+      method: PATCH
+      path: /evaluation-rules/{evaluationRuleId}
+      path-parameters:
+        evaluationRuleId:
+          type: string
+          docs: Evaluation rule identifier.
+      request: UpdateEvaluationRuleRequest
+      response: EvaluationRule
+      errors:
+        - errors.BadRequestError
+        - errors.UnauthorizedError
+        - errors.AccessDeniedError
+        - errors.NotFoundError
+        - errors.MethodNotAllowedError
+        - errors.UnprocessableContentError
+        - errors.TooManyRequestsError
+        - errors.InternalServerError
+
+    delete:
+      docs: |
+        Delete an evaluation rule.
+
+        This removes the live-ingestion rule only. It does not delete the referenced evaluator.
+      method: DELETE
+      path: /evaluation-rules/{evaluationRuleId}
+      path-parameters:
+        evaluationRuleId:
+          type: string
+          docs: Evaluation rule identifier.
+      response: DeleteEvaluationRuleResponse
+      errors:
+        - errors.BadRequestError
+        - errors.UnauthorizedError
+        - errors.AccessDeniedError
+        - errors.NotFoundError
+        - errors.MethodNotAllowedError
+        - errors.TooManyRequestsError
+        - errors.InternalServerError
+
+types:
+  EvaluationRule:
+    docs: |
+      Live evaluation rule for incoming data.
+
+      An evaluation rule answers:
+      - which evaluator should be used
+      - which target objects should trigger scoring
+      - how often scoring should run
+      - which target fields should populate each evaluator variable
+      - whether the deployment is active, inactive, or paused
+
+      Important status semantics:
+      - `enabled` is the desired on/off setting from the client
+      - `status` is the effective runtime state after Langfuse applies validation and blocking rules
+      - `enabled=true` with `status=paused` means the rule should run, but Langfuse has paused it until the underlying problem is fixed
+    properties:
+      id:
+        type: string
+        docs: Stable evaluation rule identifier.
+      name:
+        type: string
+        docs: Human-readable deployment name. This is independent from the evaluator name.
+      evaluator:
+        type: EvaluationRuleEvaluator
+        docs: |
+          Evaluator currently used by this rule.
+
+          `name` and `scope` identify the evaluator family conceptually.
+          `id` is the currently active evaluator version in that family.
+          If you create a newer project version with the same evaluator name later, existing evaluation rules are moved to it automatically.
+      target:
+        type: commons.EvaluationRuleTarget
+        docs: Target object type that should trigger scoring.
+      enabled:
+        type: boolean
+        docs: Desired enabled state configured by the client.
+      status:
+        type: commons.EvaluationRuleStatus
+        docs: Effective runtime status after Langfuse applies validation and blocking rules.
+      pausedReason:
+        type: nullable<string>
+        docs: Machine-readable reason when `status=paused`, otherwise `null`.
+      pausedMessage:
+        type: nullable<string>
+        docs: Human-readable explanation when `status=paused`, otherwise `null`.
+      sampling:
+        type: double
+        docs: |
+          Fraction of matching target objects that should be evaluated.
+
+          Must be greater than `0` and less than or equal to `1`.
+          - `1` means evaluate every matching target.
+          - `0.25` means evaluate approximately 25% of matching targets.
+      filter:
+        type: list<commons.EvaluationRuleFilter>
+        docs: List of filter conditions used to decide whether a target should be evaluated.
+      mapping:
+        type: list<commons.EvaluationRuleMapping>
+        docs: Variable mappings used to populate the evaluator prompt from the live target object.
+      createdAt:
+        type: datetime
+        docs: Timestamp when the evaluation rule was created.
+      updatedAt:
+        type: datetime
+        docs: Timestamp when the evaluation rule was last updated.
+    examples:
+      - value:
+          id: erule_123
+          name: answer-correctness-live
+          evaluator:
+            id: evaltmpl_123
+            name: answer-correctness
+            scope: project
+          target: observation
+          enabled: true
+          status: active
+          pausedReason: null
+          pausedMessage: null
+          sampling: 1
+          filter:
+            - type: stringOptions
+              column: type
+              operator: any of
+              value:
+                - GENERATION
+          mapping:
+            - variable: input
+              source: input
+            - variable: output
+              source: output
+          createdAt: "2026-03-30T09:20:00.000Z"
+          updatedAt: "2026-03-30T09:20:00.000Z"
+
+  EvaluationRules:
+    docs: Paginated list of evaluation rules.
+    properties:
+      data:
+        type: list<EvaluationRule>
+        docs: Evaluation rules in the current page.
+      meta:
+        type: pagination.MetaResponse
+        docs: Standard pagination metadata.
+
+  CreateEvaluationRuleRequest:
+    docs: |
+      Request body for creating an evaluation rule.
+
+      Checklist for agents and SDK clients:
+      - reference an existing evaluator family by `evaluator.name` and `evaluator.scope`
+      - choose `target=observation` or `target=experiment`
+      - if `target=experiment` and you want a dataset filter, call `GET /api/public/v2/datasets` first and use dataset `id` values in `filter[].value`
+      - fetch or inspect the evaluator first, then provide a complete variable mapping for every evaluator variable listed in `variables`
+      - optionally narrow execution with `filter`
+      - set `enabled=true` only when you want live execution immediately
+    properties:
+      name:
+        type: string
+        docs: Human-readable deployment name.
+      evaluator:
+        type: EvaluationRuleEvaluatorReference
+        docs: |
+          Evaluator family to use.
+
+          Use `name` and `scope` from the evaluator endpoints.
+          Langfuse resolves that family to its latest version before saving the rule.
+      target:
+        type: commons.EvaluationRuleTarget
+        docs: Target object type to evaluate.
+      enabled:
+        type: boolean
+        docs: Whether the deployment should be active immediately after creation.
+      sampling:
+        type: optional<double>
+        docs: Optional sampling fraction. Defaults to `1`.
+      filter:
+        type: optional<list<commons.EvaluationRuleFilter>>
+        docs: |
+          Optional filter list.
+
+          Omit or pass an empty list to evaluate all matching targets for the selected `target`.
+          Each filter object must use a column that is valid for that `target`.
+          For `target=experiment`, `column=datasetId` expects dataset `id` values from `GET /api/public/v2/datasets`, not dataset names.
+      mapping:
+        type: list<commons.EvaluationRuleMapping>
+        docs: |
+          Required variable mappings.
+
+          Every evaluator variable must appear exactly once.
+          Build this list from the evaluator `variables` array returned by the evaluator endpoints.
+
+  UpdateEvaluationRuleRequest:
+    docs: |
+      Partial update body for an evaluation rule.
+
+      Provide only the fields you want to change.
+      An empty body is rejected.
+
+      Practical guidance:
+      - If you only want to rename the rule or change sampling, send just those fields.
+      - If you change `evaluator`, send a fresh `mapping` unless you are certain the existing mapping still matches the evaluator variables.
+      - If you change `target`, usually send both `filter` and `mapping` in the same request.
+      - If you change an experiment `datasetId` filter, call `GET /api/public/v2/datasets` and use dataset `id` values from that response.
+    properties:
+      name:
+        type: optional<string>
+        docs: Updated deployment name.
+      evaluator:
+        type: optional<EvaluationRuleEvaluatorReference>
+        docs: |
+          Updated evaluator family.
+
+          Langfuse resolves the provided evaluator family to its latest version before saving the rule.
+      target:
+        type: optional<commons.EvaluationRuleTarget>
+        docs: Updated target object type.
+      enabled:
+        type: optional<boolean>
+        docs: Updated desired enabled state.
+      sampling:
+        type: optional<double>
+        docs: Updated sampling fraction.
+      filter:
+        type: optional<list<commons.EvaluationRuleFilter>>
+        docs: |
+          Updated filter list.
+
+          For `target=experiment`, `column=datasetId` expects dataset `id` values from `GET /api/public/v2/datasets`, not dataset names.
+      mapping:
+        type: optional<list<commons.EvaluationRuleMapping>>
+        docs: Updated variable mappings.
+
+  DeleteEvaluationRuleResponse:
+    docs: Confirmation response returned after successful deletion.
+    properties:
+      message:
+        type: string
+        docs: Always `Evaluation rule successfully deleted`.
+
+  EvaluationRuleEvaluatorReference:
+    docs: |
+      Evaluator family reference used when creating or updating an evaluation rule.
+
+      `name` and `scope` are enough to identify the evaluator family in the authenticated project context.
+    properties:
+      name:
+        type: string
+        docs: Evaluator family name.
+      scope:
+        type: commons.EvaluatorScope
+        docs: Whether the evaluator family is project-owned or Langfuse-managed.
+
+  EvaluationRuleEvaluator:
+    docs: |
+      Resolved evaluator currently used by the evaluation rule.
+
+      `id` is the exact active evaluator version.
+      `name` and `scope` identify the evaluator family conceptually.
+    properties:
+      id:
+        type: string
+        docs: Identifier of the exact evaluator version currently used by the rule.
+      name:
+        type: string
+        docs: Evaluator family name.
+      scope:
+        type: commons.EvaluatorScope
+        docs: Whether the evaluator family is project-owned or Langfuse-managed.
@@ -0,0 +1,250 @@
+# yaml-language-server: $schema=https://raw.githubusercontent.com/fern-api/fern/main/fern.schema.json
+imports:
+  commons: ./commons.yml
+  errors: ./errors.yml
+  pagination: ../utils/pagination.yml
+
+service:
+  auth: true
+  base-path: /api/public/unstable
+  endpoints:
+    create:
+      docs: |
+        Create an evaluator in the authenticated project.
+
+        Use evaluators to define **how** Langfuse should score data: the prompt, the expected structured output, and the optional model configuration.
+
+        Naming behavior:
+        - If this is a new evaluator name in your project, Langfuse creates version `1`.
+        - If the name already exists in your project, Langfuse creates the next version and returns it.
+        - When a new project version is created, existing evaluation rules in that project automatically move to the newest version for that evaluator name.
+
+        Recommended workflow:
+        1. Create the evaluator.
+        2. Read the returned `variables` array.
+        3. Read the returned `outputDefinition.dataType` so the client knows whether future scores will be numeric, boolean, or categorical.
+        4. Create one or more evaluation rules that reference the returned evaluator family using `name` and `scope`.
+
+        Recovery guidance:
+        - `422` with `code=evaluator_preflight_failed`: the evaluator cannot run with the resolved model configuration. Add a valid explicit `modelConfig`, or configure the project's default evaluation model, then retry the same request.
+        - `400` with `code=invalid_body`: the request shape is malformed. Use the structured `details.issues` array to fix the specific fields and retry.
+        - `400` with `code=invalid_body` on `outputDefinition`: send `dataType`, `reasoning.description`, and `score.description`. Do not send `version`; it is not part of the public request shape.
+
+        Unstable API note:
+        - This surface may evolve while the underlying evaluation data model is being redesigned.
+      method: POST
+      path: /evaluators
+      request: CreateEvaluatorRequest
+      response: Evaluator
+      errors:
+        - errors.BadRequestError
+        - errors.UnauthorizedError
+        - errors.AccessDeniedError
+        - errors.MethodNotAllowedError
+        - errors.ConflictError
+        - errors.UnprocessableContentError
+        - errors.TooManyRequestsError
+        - errors.InternalServerError
+      examples:
+        - name: CreateEvaluatorVersion
+          docs: Create a new version of an evaluator named `answer-correctness`.
+          request:
+            name: answer-correctness
+            prompt: |
+              You are grading an answer.
+
+              Input:
+              {{input}}
+
+              Output:
+              {{output}}
+
+              Return a score between 0 and 1.
+            outputDefinition:
+              dataType: NUMERIC
+              reasoning:
+                description: Explain why the score was assigned.
+              score:
+                description: Correctness score between 0 and 1.
+            modelConfig:
+              provider: openai
+              model: gpt-4.1-mini
+          response:
+            body:
+              id: evaltmpl_123
+              name: answer-correctness
+              version: 2
+              scope: project
+              type: llm_as_judge
+              prompt: |
+                You are grading an answer.
+
+                Input:
+                {{input}}
+
+                Output:
+                {{output}}
+
+                Return a score between 0 and 1.
+              variables:
+                - input
+                - output
+              outputDefinition:
+                dataType: NUMERIC
+                reasoning:
+                  description: Explain why the score was assigned.
+                score:
+                  description: Correctness score between 0 and 1.
+              modelConfig:
+                provider: openai
+                model: gpt-4.1-mini
+              evaluationRuleCount: 0
+              createdAt: "2026-03-30T09:00:00.000Z"
+              updatedAt: "2026-03-30T09:00:00.000Z"
+
+    list:
+      docs: |
+        List the evaluators available to the authenticated project.
+
+        Important behavior:
+        - This endpoint returns the latest version of each available evaluator.
+        - Results can include evaluators from your project and Langfuse-managed evaluators.
+        - If the same evaluator name exists in both places, both are returned as separate items with different `scope` values.
+      method: GET
+      path: /evaluators
+      request:
+        name: ListEvaluatorsRequest
+        query-parameters:
+          page:
+            type: optional<integer>
+            docs: 1-based page number. Defaults to `1`.
+          limit:
+            type: optional<integer>
+            docs: Maximum number of items per page. Defaults to `50`.
+      response: Evaluators
+      errors:
+        - errors.BadRequestError
+        - errors.UnauthorizedError
+        - errors.AccessDeniedError
+        - errors.MethodNotAllowedError
+        - errors.TooManyRequestsError
+        - errors.InternalServerError
+
+    get:
+      docs: |
+        Get one evaluator by `id`.
+
+        Use this endpoint when you want the prompt, output definition, model configuration, and derived variables for the evaluator you plan to use in an evaluation rule.
+      method: GET
+      path: /evaluators/{evaluatorId}
+      path-parameters:
+        evaluatorId:
+          type: string
+          docs: Evaluator identifier returned by the evaluator endpoints.
+      response: Evaluator
+      errors:
+        - errors.BadRequestError
+        - errors.UnauthorizedError
+        - errors.AccessDeniedError
+        - errors.NotFoundError
+        - errors.MethodNotAllowedError
+        - errors.TooManyRequestsError
+        - errors.InternalServerError
+
+types:
+  Evaluator:
+    docs: |
+      One evaluator that can be used for scoring.
+
+      An evaluator describes **how** to score data:
+      - prompt
+      - extracted prompt variables
+      - output schema
+      - optional explicit model configuration
+
+      It does not define **which** live objects are evaluated. That is the job of `evaluation-rules`.
+
+      For agent clients, the most important fields are:
+      - `variables`: use these exact names when building the evaluation-rule `mapping` array
+      - `outputDefinition`: tells you the expected score type and the evaluator's response instructions
+      - `modelConfig`: tells you whether the evaluator uses the project default model (`null`) or an explicit provider/model
+
+      Versioning behavior:
+      - `GET /evaluators` returns the latest version of each available evaluator.
+      - `GET /evaluators/{id}` can return an older version.
+      - Evaluation rules always run against the latest version for the selected evaluator name within the same source (`project` or `managed`).
+    properties:
+      id:
+        type: string
+        docs: Identifier of this evaluator.
+      name:
+        type: string
+        docs: Evaluator name.
+      version:
+        type: integer
+        docs: Version number of this evaluator.
+      scope:
+        type: commons.EvaluatorScope
+        docs: "Where this evaluator comes from: your project or Langfuse-managed defaults."
+      type:
+        type: commons.EvaluatorType
+        docs: Evaluator engine type. Currently always `llm_as_judge`.
+      prompt:
+        type: string
+        docs: Prompt template used during evaluation.
+      variables:
+        type: list<string>
+        docs: |
+          Variables extracted from the evaluator prompt.
+
+          Every variable in this list must be mapped exactly once when creating an evaluation rule.
+      outputDefinition:
+        type: commons.PublicEvaluatorOutputDefinition
+        docs: |
+          Structured output schema returned by this evaluator.
+
+          Responses always include `dataType` and omit the internal output-definition `version`.
+          Use `dataType` to decide how future scores should be interpreted.
+      modelConfig:
+        type: nullable<commons.EvaluatorModelConfig>
+        docs: Explicit model configuration, or `null` when the project default evaluation model is used.
+      evaluationRuleCount:
+        type: integer
+        docs: Number of evaluation rules in the project that currently use this evaluator version.
+      createdAt:
+        type: datetime
+        docs: Timestamp when this evaluator was created.
+      updatedAt:
+        type: datetime
+        docs: Timestamp when this evaluator was last updated.
+
+  Evaluators:
+    properties:
+      data:
+        type: list<Evaluator>
+      meta:
+        type: pagination.MetaResponse
+
+  CreateEvaluatorRequest:
+    docs: |
+      Request body for creating an evaluator.
+
+      If the same `name` already exists in your project, Langfuse creates the next version and returns it.
+      Existing evaluation rules in the same project are then moved to that new latest version automatically.
+    properties:
+      name:
+        type: string
+        docs: Evaluator name within the authenticated project.
+      prompt:
+        type: string
+        docs: Prompt template used by the evaluator.
+      outputDefinition:
+        type: commons.EvaluatorOutputDefinition
+        docs: |
+          Structured output schema the evaluator must return.
+
+          Always send `dataType`.
+          Do not send `version`; it is an internal storage detail and not part of the public request contract.
+      modelConfig:
+        type: optional<nullable<commons.EvaluatorModelConfig>>
+        docs: Optional explicit model configuration. Omit or set to `null` to use the project default evaluation model.
@@ -1,6 +1,6 @@
 {
  "name": "langfuse",
-  "version": "3.167.4",
+  "version": "3.174.0",
  "author": "engineering@langfuse.com",
  "license": "MIT",
  "private": true,
@@ -46,9 +46,9 @@
    "braces": "3.0.3",
    "dotenv-cli": "^7.4.4",
    "husky": "^9.1.7",
-    "prettier": "^3.8.1",
+    "prettier": "^3.8.3",
    "release-it": "^19.2.4",
-    "turbo": "2.9.5"
+    "turbo": "2.9.12"
  },
  "release-it": {
    "git": {
@@ -107,9 +107,11 @@
      "rollup@^4.0.0": "^4.22.4",
      "@types/node-fetch": "^2.6.13",
      "@types/react-dom": "19.2.3",
+      "fast-xml-builder": "1.1.7",
      "glob": "^10.5.0",
      "qs": "6.14.1",
-      "path-to-regexp@0.1.12": "0.1.13"
+      "path-to-regexp@0.1.12": "0.1.13",
+      "ip-address@10.1.0": "10.2.0"
    },
    "patchedDependencies": {
      "next-auth@4.24.13": "patches/next-auth@4.24.13.patch"
@@ -3,6 +3,7 @@ import globals from "globals";
 import tseslint from "typescript-eslint";
 import turboConfig from "eslint-config-turbo/flat";
 import eslintPluginPrettierRecommended from "eslint-plugin-prettier/recommended";
+import langfusePlugin from "@repo/eslint-plugin";
 import "eslint-plugin-only-warn";

 export default tseslint.config(
@@ -59,6 +60,9 @@ export default tseslint.config(
    },
    languageOptions: {
      parser: tseslint.parser,
+      parserOptions: {
+        projectService: true,
+      },
    },
    rules: {
      "no-undef": "off", // TypeScript handles this
@@ -82,6 +86,18 @@ export default tseslint.config(
          ignoreRestSiblings: true,
        },
      ],
+      "@typescript-eslint/no-deprecated": "warn",
+    },
+  },
+
+  // Vitest in-source testing should only be used while developing, not in committed code.
+  {
+    name: "langfuse/no-in-source-vitest",
+    plugins: {
+      "@repo": langfusePlugin,
+    },
+    rules: {
+      "@repo/no-in-source-vitest": "warn",
    },
  },
 );
@@ -2,6 +2,7 @@ import nextCoreWebVitals from "eslint-config-next/core-web-vitals";
 import eslintPluginPrettierRecommended from "eslint-plugin-prettier/recommended";
 import turboConfig from "eslint-config-turbo/flat";
 import "eslint-plugin-only-warn";
+import langfusePlugin from "@repo/eslint-plugin";

 export default [
  // Global ignores - include config files
@@ -62,6 +63,9 @@ export default [
    name: "langfuse/next/typescript",
    files: ["**/*.ts", "**/*.tsx"],
    languageOptions: {
+      parserOptions: {
+        projectService: true,
+      },
      globals: {
        React: "readonly",
        JSX: "readonly",
@@ -74,8 +78,12 @@ export default [
        },
      },
    },
+    plugins: {
+      "@repo": langfusePlugin,
+    },
    rules: {
      "no-unused-vars": "off", // Use @typescript-eslint/no-unused-vars instead
+      "@repo/no-tailwind-overflow-scroll": "warn",
      // Custom rules from old config
      "@typescript-eslint/consistent-type-imports": [
        "warn",
@@ -94,7 +102,20 @@ export default [
          ignoreRestSiblings: true,
        },
      ],
+      "@typescript-eslint/no-deprecated": "warn",
      "react/jsx-key": ["error", { warnOnDuplicates: true }],
+      "react/no-unused-prop-types": "warn",
+    },
+  },
+
+  // Vitest in-source testing should only be used while developing, not in committed code.
+  {
+    name: "langfuse/no-in-source-vitest",
+    plugins: {
+      "@repo": langfusePlugin,
+    },
+    rules: {
+      "@repo/no-in-source-vitest": "warn",
    },
  },
 ];
@@ -16,9 +16,10 @@
  ],
  "dependencies": {
    "@eslint/js": "^9.39.2",
-    "eslint-config-next": "16.2.3",
+    "@repo/eslint-plugin": "workspace:*",
+    "eslint-config-next": "16.2.6",
    "eslint-config-prettier": "^10.1.8",
-    "eslint-config-turbo": "2.9.5",
+    "eslint-config-turbo": "2.9.12",
    "eslint-plugin-only-warn": "^1.1.0",
    "eslint-plugin-prettier": "^5.5.4",
    "globals": "^16.0.0",
@@ -27,7 +27,6 @@
    "isolatedModules": true,
    "jsx": "preserve",
    "types": [
-      "jest",
      "node"
    ],
  },
@@ -0,0 +1,21 @@
+{
+  "name": "@repo/eslint-plugin",
+  "version": "0.0.0",
+  "license": "MIT",
+  "private": true,
+  "type": "module",
+  "main": "dist/index.js",
+  "scripts": {
+    "test": "vitest run",
+    "build": "tsc"
+  },
+  "devDependencies": {
+    "@repo/typescript-config": "workspace:*",
+    "@types/node": "^24.3.0",
+    "@typescript-eslint/parser": "^8.59.0",
+    "@typescript-eslint/rule-tester": "^8.59.0",
+    "@typescript-eslint/utils": "^8.59.0",
+    "typescript": "^5.9.2",
+    "vitest": "^4.1.4"
+  }
+}
@@ -0,0 +1,11 @@
+import { default as noTailwindOverflowScroll } from "./rules/no-tailwind-overflow-scroll.js";
+import { default as noInSourceVitest } from "./rules/no-in-source-vitest.js";
+
+export const plugin = {
+  rules: {
+    "no-in-source-vitest": noInSourceVitest,
+    "no-tailwind-overflow-scroll": noTailwindOverflowScroll,
+  },
+};
+
+export default plugin;
@@ -0,0 +1,36 @@
+import { RuleTester } from "@typescript-eslint/rule-tester";
+import * as typescriptEslintParser from "@typescript-eslint/parser";
+import rule from "./no-in-source-vitest.js";
+
+const ruleTester = new RuleTester({
+  languageOptions: {
+    parser: typescriptEslintParser,
+    parserOptions: {
+      ecmaVersion: 2022,
+      sourceType: "module",
+      ecmaFeatures: {
+        jsx: true,
+      },
+    },
+  },
+});
+
+ruleTester.run("no-in-source-vitest", rule, {
+  valid: [
+    {
+      code: `export const value = 42;`,
+      filename: "/repo/web/src/features/example/value.ts",
+    },
+    {
+      code: `if (import.meta.env.DEV) console.log("dev");`,
+      filename: "/repo/web/src/features/example/value.ts",
+    },
+  ],
+  invalid: [
+    {
+      code: `if (import.meta.vitest) {}`,
+      filename: "/repo/web/src/features/example/value.ts",
+      errors: [{ messageId: "unexpected" }],
+    },
+  ],
+});
@@ -0,0 +1,29 @@
+import { createRule } from "../util.js";
+
+const rule = createRule({
+  name: "no-in-source-vitest",
+  meta: {
+    type: "suggestion",
+    docs: {
+      description:
+        "Vitest in-source testing should only be used while developing, not in committed code.",
+    },
+    messages: {
+      unexpected:
+        "Vitest in-source testing should only be used while developing, not in committed code.",
+    },
+    schema: [],
+  },
+  defaultOptions: [],
+  create(context) {
+    return {
+      'MemberExpression[computed=false][property.name="vitest"][object.type="MetaProperty"][object.meta.name="import"][object.property.name="meta"]'(
+        node,
+      ) {
+        context.report({ node, messageId: "unexpected" });
+      },
+    };
+  },
+});
+
+export default rule;
@@ -0,0 +1,93 @@
+import { RuleTester } from "@typescript-eslint/rule-tester";
+import * as typescriptEslintParser from "@typescript-eslint/parser";
+import { describe, expect, it, vi } from "vitest";
+import rule from "./no-tailwind-overflow-scroll.js";
+
+const ruleTester = new RuleTester({
+  languageOptions: {
+    parser: typescriptEslintParser,
+    parserOptions: {
+      ecmaFeatures: {
+        jsx: true,
+      },
+    },
+  },
+});
+
+ruleTester.run("no-tailwind-overflow-scroll", rule, {
+  valid: [
+    `<div className="button" />`,
+    `<div className="overflow-auto" />`,
+    `<div className="overflow-x-auto" />`,
+    `<div className="overflow-y-auto" />`,
+    `<div className="overflow-scrollbar" />`,
+    `const value = 42;`,
+  ],
+  invalid: [
+    {
+      code: `<div className="overflow-scroll" />`,
+      errors: [{ messageId: "unexpected" }],
+    },
+    {
+      code: `<div className="overflow-y-scroll" />`,
+      errors: [{ messageId: "unexpected" }],
+    },
+    {
+      code: `<div className="overflow-x-scroll" />`,
+      errors: [{ messageId: "unexpected" }],
+    },
+    {
+      code: `<div className="flex h-full overflow-x-scroll" />`,
+      errors: [{ messageId: "unexpected" }],
+    },
+    {
+      code: `const className = "flex h-full overflow-y-scroll";`,
+      errors: [{ messageId: "unexpected" }],
+    },
+    {
+      code: `const className = cn("text-muted-foreground flex h-full overflow-x-scroll", className);`,
+      errors: [{ messageId: "unexpected" }],
+    },
+    {
+      code: `<div className="md:overflow-scroll" />`,
+      errors: [{ messageId: "unexpected" }],
+    },
+    {
+      code: `<div className={\`flex overflow-x-scroll\`} />`,
+      errors: [{ messageId: "unexpected" }],
+    },
+    {
+      code: `<div className="!overflow-scroll" />`,
+      errors: [{ messageId: "unexpected" }],
+    },
+    {
+      code: `<div className="overflow-scroll!" />`,
+      errors: [{ messageId: "unexpected" }],
+    },
+    {
+      code: `<div className="md:!overflow-scroll" />`,
+      errors: [{ messageId: "unexpected" }],
+    },
+    {
+      code: `<div className="md:overflow-scroll!" />`,
+      errors: [{ messageId: "unexpected" }],
+    },
+  ],
+});
+
+describe("no-tailwind-overflow-scroll", () => {
+  it("ignores template elements with non-string raw values", () => {
+    const report = vi.fn();
+    const listeners = rule.create({
+      report,
+    } as never);
+
+    listeners.TemplateElement?.({
+      value: {
+        raw: null,
+      },
+    } as never);
+
+    expect(report).not.toHaveBeenCalled();
+  });
+});
@@ -0,0 +1,75 @@
+import { createRule } from "../util.js";
+
+const FORBIDDEN_TAILWIND_UTILITIES = new Set([
+  "overflow-scroll",
+  "overflow-x-scroll",
+  "overflow-y-scroll",
+]);
+
+// Tailwind important modifiers can wrap a utility token in either position,
+// for example `!overflow-scroll` or `overflow-scroll!`.
+function normalizeTailwindToken(token: string): string {
+  return token.replace(/^!|!$/g, "");
+}
+
+// Yield normalized utility tokens, for example `md:overflow-scroll!` ->
+// `overflow-scroll`.
+function* extractTailwindUtilityTokens(value: string): Generator<string> {
+  for (const match of value.matchAll(/\S+/g)) {
+    const normalizedToken = normalizeTailwindToken(match[0]);
+    const variantSeparatorIndex = normalizedToken.lastIndexOf(":");
+    yield variantSeparatorIndex === -1
+      ? normalizedToken
+      : normalizeTailwindToken(
+          normalizedToken.slice(variantSeparatorIndex + 1),
+        );
+  }
+}
+
+const rule = createRule({
+  name: "no-tailwind-overflow-scroll",
+  meta: {
+    type: "suggestion",
+    docs: {
+      description:
+        "Avoid Tailwind's forced scrollbars (`overflow-scroll`, `overflow-x-scroll`, `overflow-y-scroll`). Use `overflow-auto`, `overflow-x-auto`, `overflow-y-auto` instead to show scrollbars only when necessary.",
+    },
+    schema: [],
+    messages: {
+      unexpected:
+        "Avoid Tailwind's forced scrollbars (`overflow-scroll`, `overflow-x-scroll`, `overflow-y-scroll`). Use `overflow-auto`, `overflow-x-auto`, `overflow-y-auto` instead to show scrollbars only when necessary.",
+    },
+  },
+  defaultOptions: [],
+  create(context) {
+    return {
+      // Define AST listeners here
+      Literal(node) {
+        if (typeof node.value === "string") {
+          for (const candidateToken of extractTailwindUtilityTokens(
+            node.value,
+          )) {
+            if (FORBIDDEN_TAILWIND_UTILITIES.has(candidateToken)) {
+              context.report({ node, messageId: "unexpected" });
+              break;
+            }
+          }
+        }
+      },
+      TemplateElement(node) {
+        if (typeof node.value.raw === "string") {
+          for (const candidateToken of extractTailwindUtilityTokens(
+            node.value.raw,
+          )) {
+            if (FORBIDDEN_TAILWIND_UTILITIES.has(candidateToken)) {
+              context.report({ node, messageId: "unexpected" });
+              break;
+            }
+          }
+        }
+      },
+    };
+  },
+});
+
+export default rule;
@@ -0,0 +1,4 @@
+import { ESLintUtils } from "@typescript-eslint/utils";
+
+// There is no hosted documentation for our internal rules
+export const createRule = ESLintUtils.RuleCreator((name) => `#${name}`);
@@ -0,0 +1,14 @@
+{
+  "extends": "@repo/typescript-config/base",
+  "compilerOptions": {
+    "declaration": false,
+    "declarationMap": false,
+    "rootDir": "src",
+    "outDir": "dist",
+    "target": "ES2022",
+    "strictNullChecks": true,
+    "types": ["node", "vitest/globals"]
+  },
+  "exclude": ["node_modules"],
+  "include": ["src"]
+}
@@ -0,0 +1,19 @@
+import { defineConfig } from "vitest/config";
+
+export default defineConfig({
+  test: {
+    setupFiles: ["./vitest.setup.ts"],
+    coverage: {
+      enabled: true,
+      provider: "v8",
+      include: ["src/**/*.ts"],
+      exclude: ["src/**/*.test.ts", "src/index.ts"],
+      thresholds: {
+        statements: 100,
+        branches: 100,
+        functions: 100,
+        lines: 100,
+      },
+    },
+  },
+});
@@ -0,0 +1,8 @@
+import { RuleTester } from "@typescript-eslint/rule-tester";
+import { afterAll, describe, it } from "vitest";
+
+// Required for RuleTester to work with Vitest
+RuleTester.afterAll = afterAll;
+RuleTester.it = it;
+RuleTester.itOnly = it.only;
+RuleTester.describe = describe;
@@ -25,11 +25,15 @@ Use root [AGENTS.md](../../AGENTS.md) for monorepo-level rules.
 - Main exports: `src/index.ts`
 - DB clients and types: `src/db.ts`
 - Server exports: `src/server/index.ts`
+- Server cache utilities: `src/server/cache/*`
 - Domain model types: `src/domain/*`
 - Repository layer: `src/server/repositories/*`
 - Queue payload schemas: `src/server/queues.ts`
 - Queue helpers: `src/server/redis/*`
 - Postgres schema: `prisma/schema.prisma`
+- For unstable public eval APIs, the public `evaluatorId` is currently the
+  exact `EvalTemplate.id`. Latest-version family grouping is derived from
+  `(projectId, name)` rather than stored on extra evaluator identity fields.
 - Prisma migrations: `prisma/migrations/*`
 - ClickHouse migrations: `clickhouse/migrations/{clustered,unclustered}/*`
 - Seeder and support scripts: `scripts/seeder/*`, `clickhouse/scripts/*`
@@ -1,2 +1,2 @@
-ALTER TABLE traces ON CLUSTER default ADD INDEX IF NOT EXISTS idx_session_id session_id TYPE bloom_filter() GRANULARITY 1;
-ALTER TABLE traces ON CLUSTER default MATERIALIZE INDEX IF EXISTS idx_session_id;
+ALTER TABLE traces ON CLUSTER default ADD INDEX IF NOT EXISTS idx_session_id session_id TYPE bloom_filter() GRANULARITY 1 SETTINGS alter_sync = 2;
+ALTER TABLE traces ON CLUSTER default MATERIALIZE INDEX IF EXISTS idx_session_id SETTINGS mutations_sync = 2;
@@ -1,2 +1,2 @@
-ALTER TABLE traces ON CLUSTER default ADD INDEX IF NOT EXISTS idx_user_id user_id TYPE bloom_filter() GRANULARITY 1;
-ALTER TABLE traces ON CLUSTER default MATERIALIZE INDEX IF EXISTS idx_user_id;
+ALTER TABLE traces ON CLUSTER default ADD INDEX IF NOT EXISTS idx_user_id user_id TYPE bloom_filter() GRANULARITY 1 SETTINGS alter_sync = 2;
+ALTER TABLE traces ON CLUSTER default MATERIALIZE INDEX IF EXISTS idx_user_id SETTINGS mutations_sync = 2;
@@ -1,3 +1,3 @@
-ALTER TABLE traces ON CLUSTER default DROP COLUMN IF EXISTS environment;
-ALTER TABLE observations ON CLUSTER default DROP COLUMN IF EXISTS environment;
-ALTER TABLE scores ON CLUSTER default DROP COLUMN IF EXISTS environment;
+ALTER TABLE traces ON CLUSTER default DROP COLUMN IF EXISTS environment SETTINGS alter_sync = 2;
+ALTER TABLE observations ON CLUSTER default DROP COLUMN IF EXISTS environment SETTINGS alter_sync = 2;
+ALTER TABLE scores ON CLUSTER default DROP COLUMN IF EXISTS environment SETTINGS alter_sync = 2;
@@ -1,3 +1,3 @@
-ALTER TABLE traces ON CLUSTER default ADD COLUMN environment LowCardinality(String) DEFAULT 'default' AFTER project_id;
-ALTER TABLE observations ON CLUSTER default ADD COLUMN environment LowCardinality(String) DEFAULT 'default' AFTER project_id;
-ALTER TABLE scores ON CLUSTER default ADD COLUMN environment LowCardinality(String) DEFAULT 'default' AFTER project_id;
+ALTER TABLE traces ON CLUSTER default ADD COLUMN environment LowCardinality(String) DEFAULT 'default' AFTER project_id SETTINGS alter_sync = 2;
+ALTER TABLE observations ON CLUSTER default ADD COLUMN environment LowCardinality(String) DEFAULT 'default' AFTER project_id SETTINGS alter_sync = 2;
+ALTER TABLE scores ON CLUSTER default ADD COLUMN environment LowCardinality(String) DEFAULT 'default' AFTER project_id SETTINGS alter_sync = 2;
@@ -1,2 +1,2 @@
-ALTER TABLE scores ON CLUSTER default ADD INDEX IF NOT EXISTS idx_project_trace_observation (project_id, trace_id, observation_id) TYPE bloom_filter(0.001) GRANULARITY 1 SETTINGS mutations_sync = 2;
+ALTER TABLE scores ON CLUSTER default ADD INDEX IF NOT EXISTS idx_project_trace_observation (project_id, trace_id, observation_id) TYPE bloom_filter(0.001) GRANULARITY 1 SETTINGS alter_sync = 2;
 ALTER TABLE scores ON CLUSTER default MATERIALIZE INDEX IF EXISTS idx_project_trace_observation SETTINGS mutations_sync = 2;
@@ -1,2 +1,2 @@
-ALTER TABLE scores ON CLUSTER default ADD INDEX IF NOT EXISTS idx_project_trace_observation (project_id, trace_id, observation_id) TYPE bloom_filter(0.001) GRANULARITY 1 SETTINGS mutations_sync = 2;
+ALTER TABLE scores ON CLUSTER default ADD INDEX IF NOT EXISTS idx_project_trace_observation (project_id, trace_id, observation_id) TYPE bloom_filter(0.001) GRANULARITY 1 SETTINGS alter_sync = 2;
 ALTER TABLE scores ON CLUSTER default MATERIALIZE INDEX IF EXISTS idx_project_trace_observation SETTINGS mutations_sync = 2;
@@ -1,2 +1,2 @@
-ALTER TABLE scores ON CLUSTER default ADD INDEX IF NOT EXISTS idx_project_session (project_id, session_id) TYPE bloom_filter(0.001) GRANULARITY 1 SETTINGS mutations_sync = 2;
+ALTER TABLE scores ON CLUSTER default ADD INDEX IF NOT EXISTS idx_project_session (project_id, session_id) TYPE bloom_filter(0.001) GRANULARITY 1 SETTINGS alter_sync = 2;
 ALTER TABLE scores ON CLUSTER default MATERIALIZE INDEX IF EXISTS idx_project_session SETTINGS mutations_sync = 2;
@@ -1,2 +1,2 @@
-ALTER TABLE scores ON CLUSTER default ADD INDEX IF NOT EXISTS idx_project_dataset_run (project_id, dataset_run_id) TYPE bloom_filter(0.001) GRANULARITY 1 SETTINGS mutations_sync = 2;
+ALTER TABLE scores ON CLUSTER default ADD INDEX IF NOT EXISTS idx_project_dataset_run (project_id, dataset_run_id) TYPE bloom_filter(0.001) GRANULARITY 1 SETTINGS alter_sync = 2;
 ALTER TABLE scores ON CLUSTER default MATERIALIZE INDEX IF EXISTS idx_project_dataset_run SETTINGS mutations_sync = 2;
@@ -1,2 +1,2 @@
-ALTER TABLE observations ON CLUSTER default DROP INDEX IF EXISTS idx_res_metadata_value;
-ALTER TABLE observations ON CLUSTER default DROP INDEX IF EXISTS idx_res_metadata_key;
+ALTER TABLE observations ON CLUSTER default DROP INDEX IF EXISTS idx_res_metadata_value SETTINGS alter_sync = 2;
+ALTER TABLE observations ON CLUSTER default DROP INDEX IF EXISTS idx_res_metadata_key SETTINGS alter_sync = 2;
@@ -1,4 +1,4 @@
-ALTER TABLE observations ON CLUSTER default ADD INDEX IF NOT EXISTS idx_res_metadata_key mapKeys(metadata) TYPE bloom_filter(0.01) GRANULARITY 1;
-ALTER TABLE observations ON CLUSTER default ADD INDEX IF NOT EXISTS idx_res_metadata_value mapValues(metadata) TYPE bloom_filter(0.01) GRANULARITY 1;
-ALTER TABLE observations ON CLUSTER default MATERIALIZE INDEX IF EXISTS idx_res_metadata_key;
-ALTER TABLE observations ON CLUSTER default MATERIALIZE INDEX IF EXISTS idx_res_metadata_value;
+ALTER TABLE observations ON CLUSTER default ADD INDEX IF NOT EXISTS idx_res_metadata_key mapKeys(metadata) TYPE bloom_filter(0.01) GRANULARITY 1 SETTINGS alter_sync = 2;
+ALTER TABLE observations ON CLUSTER default ADD INDEX IF NOT EXISTS idx_res_metadata_value mapValues(metadata) TYPE bloom_filter(0.01) GRANULARITY 1 SETTINGS alter_sync = 2;
+ALTER TABLE observations ON CLUSTER default MATERIALIZE INDEX IF EXISTS idx_res_metadata_key SETTINGS mutations_sync = 2;
+ALTER TABLE observations ON CLUSTER default MATERIALIZE INDEX IF EXISTS idx_res_metadata_value SETTINGS mutations_sync = 2;
@@ -1,2 +1,2 @@
-ALTER TABLE dataset_run_items_rmt ON CLUSTER default ADD INDEX IF NOT EXISTS idx_trace_id trace_id TYPE bloom_filter(0.001) GRANULARITY 1;
-ALTER TABLE dataset_run_items_rmt ON CLUSTER default MATERIALIZE INDEX IF EXISTS idx_trace_id;
+ALTER TABLE dataset_run_items_rmt ON CLUSTER default ADD INDEX IF NOT EXISTS idx_trace_id trace_id TYPE bloom_filter(0.001) GRANULARITY 1 SETTINGS alter_sync = 2;
+ALTER TABLE dataset_run_items_rmt ON CLUSTER default MATERIALIZE INDEX IF EXISTS idx_trace_id SETTINGS mutations_sync = 2;
@@ -1,2 +1,2 @@
-ALTER TABLE observations ON CLUSTER default DROP COLUMN IF EXISTS usage_pricing_tier_name;
-ALTER TABLE observations ON CLUSTER default DROP COLUMN IF EXISTS usage_pricing_tier_id;
+ALTER TABLE observations ON CLUSTER default DROP COLUMN IF EXISTS usage_pricing_tier_name SETTINGS alter_sync = 2;
+ALTER TABLE observations ON CLUSTER default DROP COLUMN IF EXISTS usage_pricing_tier_id SETTINGS alter_sync = 2;
@@ -1,2 +1,2 @@
-ALTER TABLE observations ON CLUSTER default ADD COLUMN usage_pricing_tier_id Nullable(String);
-ALTER TABLE observations ON CLUSTER default ADD COLUMN usage_pricing_tier_name Nullable(String);
+ALTER TABLE observations ON CLUSTER default ADD COLUMN usage_pricing_tier_id Nullable(String) SETTINGS alter_sync = 2;
+ALTER TABLE observations ON CLUSTER default ADD COLUMN usage_pricing_tier_name Nullable(String) SETTINGS alter_sync = 2;
--- a/Show More
+++ b/Show More