ci(sdk): use repo token for sdk spec workflow
Remove the protected branches environment so the SDK API spec job uses the repository GH_ACCESS_TOKEN instead of the environment-scoped token.
* fix(llm): secure google model fetches
Route Google AI Studio and Vertex AI LangChain calls through validated secure fetch clients.
* fix(llm): secure openai and anthropic fetches
Route OpenAI, Azure OpenAI, and Anthropic LangChain calls through validated secure fetch.
* fix(llm): preserve google ai studio base url prefixes
* fix(llm): simplify secure google fetch clients
* fix(llm): preserve proxies and bump langchain core
Keep explicit undici dispatchers on secure LLM fetches so HTTPS_PROXY is honored, including Google clients. Bump @langchain/core to 1.1.48 to pick up the CJS uuid export fix.
* fix(llm): avoid dispatcher type conflicts
Keep proxy dispatchers opaque across secure fetch wrappers to avoid mixing undici and undici-types Dispatcher identities during typecheck.
* refactor(llm): clarify dispatcher handoff in secure outbound fetch
Document that any caller-provided dispatcher takes ownership of
connection-time safety, share the dispatcher-aware RequestInit type, and
harden the Google secure API client tests against NEXT_PUBLIC_LANGFUSE_CLOUD_REGION
env contamination.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(llm): drop redundant proxy fetchOptions and tighten secure fetch tests
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(llm): handle ChatGoogle mixed content blocks and thought parts
The unified @langchain/google SDK emits tool-calling responses as a mixed
array of `{type: "text"}` and `{type: "functionCall"}` blocks, and marks
reasoning text with `thought: true` instead of `type: "reasoning"`.
Accept a per-element content union and detect thought blocks so VertexAI
thinking + tool calling parses again.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore(deps): drop @langchain/core release-age exception
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(llm): consume standard contentBlocks for reasoning and tool calls
Route every chat model through @langchain/core's documented
AIMessage#contentBlocks accessor instead of inspecting raw provider
content. The registered translators normalize Bedrock reasoning_content,
Gemini thought parts, and Anthropic thinking/tool_use blocks into the
standard { type: "reasoning" | "text" | "tool_call" | ... } shape, so
splitAIMessage no longer needs per-adapter block-type sets or
undocumented field checks.
Tightens streaming to handle AIMessageChunk explicitly and replaces the
ad-hoc Anthropic/Google content unions in ToolCallResponseSchema with a
single standard ContentBlock shape.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test(llm): use real bedrock message instances
* fix(llm): ignore caller dispatchers in secure fetch
* fix(llm): pass google thinking options flat
* fix(llm): mark secureLlmFetch validation errors as non-retryable
Synchronous errors from validateLlmConnectionBaseURL and the
fetchWithSecureRedirects error classes carry no HTTP status, so the
catch block defaulted them to 500 + retryable and re-enqueued
permanently broken configs against the 24h eval-retry budget.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(llm): walk error cause chain for non-retryable pattern check
Anthropic/OpenAI/Azure SDKs wrap synchronous custom-fetch errors as
APIConnectionError { message: "Connection error.", cause: original },
so the secureLlmFetch validation patterns added in the previous commit
never matched for those three adapters. Walking the .cause chain (with
cycle guard) makes the non-retryable classification fire end-to-end.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(llm): surface secure fetch validation messages
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(dashboards): drop scoreName filter on traces and observations views
The dashboard exposes a global "Score Name" filter that gets routed to
every widget via dashboardUiTableToViewMapping. The traces and
observations entries mapped it to a scoreName column, but neither
traceView nor observationsView declares a scoreName dimension in the
query data model. queryBuilder.resolveDimension then hit the generic
*Name -> name fallback and silently rewrote the filter to traces.name
or observations.name -- so picking a score label appeared to match
trace names instead.
Removing the mappings lets the filter partition as unsupported on those
views (still applied correctly on scores-numeric / scores-categorical),
which avoids the silent miscarriage without changing the score-side UX.
Fixes LFE-9773.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(dashboards): hide scoreName and observationName filters on traces
Follow-up to the previous commit. The widget builder's filter-column
dropdown is fed by web/src/features/widgets/components/widgetFilterColumns.ts,
not by dashboardUiTableToViewMapping. "Observation Name" and "Score Name"
were in the unconditional base list, so they kept appearing for every
selectedView -- including traces, where neither is a real dimension.
Removing them from the mapping silenced the wrong-query bug but the UI
still misleadingly offered the options.
Gate both columns per-view:
- Observation Name: only on observations / scores-numeric / scores-categorical.
- Score Name: only on scores-numeric / scores-categorical.
Also drops observationName from the traces entry in
dashboardUiTableToViewMapping (same 1:n problem as scoreName: traceView
has no observationName dimension, so the *Name->name fallback would
silently rewrite to traces.name).
Refs LFE-9773.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(evals): handle mac format shortcut by physical key
* fix(evals): support python evaluator formatting
Enable the code evaluator format action for Python by reusing Ruff WASM and preserving the generated editor prelude.
* fix(evals): align code evaluator editor and runtime globals
Keep code evaluator language drafts separate in the editor and allow the same async helpers/globals in validation and local execution.
* fix(evals): cover code eval promise and byte helpers
Add Promise combinators and Uint8Array helpers to the synthetic validator declarations so editor validation matches the local runtime.
* fix(evals): expand code eval validation globals
Round out URL and array declarations and avoid helper type collisions in the synthetic TypeScript validator environment.
* feat(ui): add notification for agent skills launch; stack notifications
* test(ui): dismiss LW notifications in sidebar test
Stacked notifications only render the front card's content, so the GitHub
stars badge stays hidden while a higher-ranked Launch Week notification
is within its TTL. Pre-seed the dismissed list with the LW IDs so
github-star surfaces and the badge alt-text assertion remains stable.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat: introducing matches operator that is guaranteed to use FTS index.
* chore: refactor and consolidate a bit
* chore: addressig review and fixing CI
* chore: addressing review comments
* chore: clarifying API docs
* attach query id to logs
* ensure causes are correctly reported
* re-add error stack
* fix(blob-export): surface root cause and query_id in blob storage error logs
Before this change, blob export job failures reported only "Failed to upload
file to S3 (buffered)" — masking ClickHouse OOM errors and making triage
require manual trace correlation.
- Append [query_id: <id>] to errors thrown from queryClickhouseStream so
the query id survives the full error chain to the BullMQ job failure log
- Add formatErrorChain helper that walks .cause and joins messages with
"caused by", used in logger.error and the rethrown job error so both the
Datadog log and BullMQ failure entry show the full root cause inline
- Pass { stack } (not the Error) to logger.error to capture the stack
without triggering Winston's message-concatenation behaviour
- Copy the original stack onto the rethrown error so the queue processor
sees the real failure site, not the rethrow line
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* fix(mcp): inject root type: "object" for intersection/union inputSchema
Zod's JSON Schema converter emits intersections as bare `allOf` and unions
as `anyOf`, omitting the root `type` keyword. The MCP TypeScript SDK
validates `Tool.inputSchema.type` as `z.literal("object")`, so SDK-based
clients (e.g. Claude Code) reject these tools and silently drop them
from `tools/list`.
Normalize the generated JSON Schema so the root always declares
`type: "object"`. Draft-7 permits `type` alongside `allOf`/`oneOf`/`anyOf`
- all constraints must hold - so this is semantically a no-op for
already-conformant schemas.
This restores compatibility for `createScore`, `createScoreConfig`, and
`updateScoreConfig` introduced in #13781.
Fixes#13804
* refactor: Format code
* fix: Make `defineTool` type injection more robust
---------
Co-authored-by: Nimar <l.nimar.b@gmail.com>
Co-authored-by: Ben Bachem <10088265+bezbac@users.noreply.github.com>
Manual batch evaluation runs now bypass the LIVE/INACTIVE toggle so
users can re-evaluate historic data with a paused evaluator. Blocked
configs (auth/model issues) are still skipped.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(email): support AWS SES transport via default credential chain
Detect SES from a `ses://<region>` scheme in SMTP_CONNECTION_URL and build a
nodemailer transport on top of @aws-sdk/client-sesv2 using the default AWS
credential chain. SMTP and SES-over-SMTP (smtps://) paths are unchanged.
A new shared helper (createMailTransport / buildMailServerConfig) replaces the
duplicated createTransport(parseConnectionUrl(...)) call sites across the email
services and is wired through NextAuth's EmailProvider in web/src/server/auth.ts.
Refs langfuse/langfuse#13669
* docs(email): document ses:// URL form on SMTP_CONNECTION_URL
Refs langfuse/langfuse#13669
* fix(email): guard SES SentMessageInfo lacking rejected/pending fields
nodemailer's SES transport returns `{ envelope, messageId, response, raw }`
with no `rejected`/`pending` arrays, so `.concat()` on those undefined fields
threw on every successful SES send through the NextAuth password-reset path.
Refs langfuse/langfuse#13669
* test(email): fix expected SES transport name
nodemailer assigns `this.name = 'SESTransport'` (not "SES") at
ses-transport/index.js:23, so the assertion was wrong from the start.
Refs langfuse/langfuse#13669
* test(email): test parseSesRegion directly instead of probing SESv2Client
Inspecting `sesClient.config.region` returned the SDK's async region provider
(`AsyncFunction`) rather than the string we passed in, so the assertion
diverged from runtime behavior. Test the region extraction via the exposed
`__testing.parseSesRegion` helper instead and keep the transport-shape checks
limited to the dispatch boundary (transporter name + options-object shape).
No AWS credential resolution; full suite runs in ~15 ms.
Refs langfuse/langfuse#13669
---------
Co-authored-by: Steffen Schmitz <steffen@langfuse.com>
* feat(mcp): Make scores available via MCP
* Continue to accept empty string ids in the public scores API
* better error messages for agent
---------
Co-authored-by: Nimar <l.nimar.b@gmail.com>
* fix(tool-calls): parse available tools correctly from AI sdk
* add mapping test
* full otel mapping
* fix parsing
* add test
* only parse metadata if tools are found
* fix parse
* simplify
* comment todod
* fix available tools
* perf
* parse tools also from IO
* playground parse
* fix build
* fix
* adapters
* also make it work for new other adapters
* tighten remapping
* fix langgraph
* ordering
* working on the widget import/Export feature. Done for now but i am working on making it future-proof
* fixed minor edgecase in regards to malformed json
* minor fixes/changes
* working version
* lint fix
* safe commit
* safe commit
* changed error message to pass test
* sign off
* cleanup of classes and added filter-config. also added several code snippets to shared
* multi -> single upload
* undoing shared modules
* added claude preview changes
* more claude changes
* more claude changes
* claude review fix
* added claude review fix
* safe commit
* claude review fix
* cleanup
* more cleanup
* merge conflict hopefully resolved
* added claude correction
* merge fix
* fix(widgets): normalize traces imports and drop get parsing
* fix(widgets): narrow exported widget metric aggs
* fix(widgets): surface dropped import filters
---------
Co-authored-by: Nimar <l.nimar.b@gmail.com>
* feat(monitors): seeded the monitors package with schema and data validators
* feat(monitors): validate handlebars message templates against MonitorMessageContext
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test(monitors): align test imports + fixtures with refactored schema
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(monitors): address PR review feedback and codespell findings
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(monitors): coerce BigInt wire fields and add top-level barrel export
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(monitors): address remaining PR review feedback
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(monitors): added the monitor service
* fix(monitors): remove orphan features/monitor leftovers from MonitorService move
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* improvement(monitors): cleaned up types
* improvement(monitors): refactor into a feature partition
* refactor(monitors): drop handlebars template validator and message field
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(monitors): include `key` in sortFiltersCanonically canonical order
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(monitors): fix js docs typo nit
* fix(monitors): enforce nonnegative schedulerBatchId on the queue wire schema
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(monitors): use Object.hasOwn for query validation + correct threshold-order message
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(monitors): canonicalize set-semantics value arrays in sortFiltersCanonically
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(monitors): narrow input DTO status + orderBy column; reject histogram
- Add MonitorWriteStatusSchema (active|paused) and use it on
CreateMonitorInputSchema / UpdateMonitorInputSchema. `error-bad-query` is
scheduler-owned; callers can no longer forge a broken state or clear a
legitimately broken monitor without scheduler revalidation.
- Narrow MonitorListInputSchema.orderBy.column to the columns the admin
table actually sorts on (name/status/severity/createdAt). Without this,
an unknown column reached Prisma and raised a 500-class
PrismaClientValidationError instead of a clean 400.
- Reject `histogram` aggregation in isValidQuery — it returns a
bucket-array at the ClickHouse layer, but monitor thresholds are scalar.
Catch at the input boundary rather than failing in the worker.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(monitors): allow updatedAt as a sortable column on MonitorListInputSchema
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(monitors): expand orderBy columns + sort NULLS LAST on list
- Add severityChangedAt, alertedAt to the orderBy.column allowlist.
- Apply NULLS LAST unconditionally on list ordering so nullable columns
(alertedAt, severityChangedAt) sort intuitively in both directions; no-op
on non-nullable columns.
- Cover all 7 allowed columns in the input-schema test, plus a real-Postgres
integration test asserting NULLS LAST holds under ASC and DESC for both
nullable columns.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(monitors): reorder severity + status enums; drop unused message column
Reorder MonitorSeverity to UNKNOWN, NO_DATA, OK, WARNING, ALERT so DESC
sorts in attention-priority order (ALERT first). Reorder MonitorStatus to
PAUSED, ACTIVE, ERROR_BAD_QUERY so DESC reads as ERROR_BAD_QUERY → ACTIVE
→ PAUSED. Drop the unused `message` column (templating was removed
earlier; the column was kept then under Option A and is now retired).
Migration uses the canonical Postgres enum-swap pattern (CREATE _new, cast
column via text, rename _old, drop _old, rename _new). Default values
preserved. Zod enum order + service mapper cases reordered to match the
new canonical sequence.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(monitors): gate orderBy nulls last on the nullable column subset
Prisma's validator only accepts the { sort, nulls } object form on nullable
columns — unconditional nulls last raised PrismaClientValidationError for
the 5 non-nullable columns (name/status/severity/createdAt/updatedAt). Add
nullableOrderColumns typed against MonitorListOrderBy so the set stays in
sync with the sortable allowlist, attach nulls only on its members. Add
5 integration cases proving each non-nullable column list call goes
through.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs(monitors): trim stale JSDoc on MonitorListInputSchema
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test(monitors): cover schedulerBatchId invariance to property + value array order
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(monitors): reject non-stringObject metadata filters; lock scheduler batch id invariance under property order
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test(monitors): drop stale message field from prismaRow fixture
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(monitors): reject set-semantics filters with duplicate values; relocate JSDoc
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(monitors): reject scalar filters on array-typed dimensions; add positive set-semantics coverage
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(blob-export): gate pricing fields on model group, not usage
Previously input_price / output_price / total_price were enriched whenever
the `usage` field group was selected, and model_export columns (model_id,
provided_model_name, model_parameters) were fetched from ClickHouse for
both `model` and `usage` requests — even when only `usage` was asked for,
so the pricing lookup had the model_id it needed.
This split the semantics: `usage` gave you cost data, but enriched prices
came from asking for `usage` too. The model_export columns were then
silently dropped unless `model` was also selected.
After this change:
- `model` gates everything model-related: identification columns AND prices.
- `usage` covers only the cost/usage maps (usage_details, cost_details,
total_cost, usage_pricing_tier_name) — no pricing lookup, no model_export
fetch in ClickHouse.
- Selecting `usage` without `model` is cheaper (skips the model_export SQL
field set) and produces no price columns in the output.
Changes:
- worker handler: `includePricing` gate collapsed into `includeModelId`
- shared events.ts: `needsModelFields` drops the `|| usage` branch
- analytics-integrations labels: prices moved to model description, removed
from usage description
- unit tests: flip expectations to match new semantics
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* refactor(blob-export): inline model_export selection into the field group loop
Now that needsModelFields is just fieldGroups.includes("model"), the
two-step pattern (skip "model" in the loop, select model_export below)
is redundant. Collapse into a single conditional branch inside the loop.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* refactor(blob-export): remove dead model_export scrub in enrichObservationStream
The else-branch that deleted model_id/provided_model_name/model_parameters
was needed when model_export was fetched for usage-without-model requests
(so the pricing lookup had a model_id, then the columns were scrubbed before
output). That code path no longer exists after gating model_export on the
model group only — the columns are never present in the row when model is
absent, so the deletes were a no-op.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* fix(blob-export): add usage_pricing_tier_id to usage group description
The usage FieldSet projects usage_pricing_tier_id but the UI label omitted
it, causing the column to appear undocumented in exports. Pre-existing gap
surfaced by the PR review.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* fix(blob-export): remove FieldSetName cast that was not imported
The refactor introduced `group as FieldSetName` but FieldSetName is not
imported in events.ts. Revert to the plain `group` call that main used,
which satisfies the TypeScript overload without an explicit cast.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* fix comment
---------
Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* feat(dashboarding): support experiment filters, dimensions in observations view via preset
* feat(widget-form): render experiment as view rather than preset
* Revert "feat(widget-form): render experiment as view rather than preset"
This reverts commit 841d8437c85b27897e57ccd97b3d12118856713f.
* refactor: handle metadata filter appropriately
* chore: push
* chore: push
* feat(widget-form): keep only view agnostic filters on view change
* feat(tests): add v2-only experiment filters for observations in dashboard widget tests
* fix(dataModel): enable highCardinality for experimentName and experimentDatasetId fields
* revert: rm experiment_metadata as dimension
* fix: add highCardinality flag for experimentId field
* chore: push
* fix: correct import path for views type in widgetFilterPresets
The import path was using a non-existent local path @/src/features/query/types
instead of the correct shared package path @langfuse/shared/query. This was
causing TypeScript build failures.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
* feat(evals): add experiment item metadata mapping for experiment evaluators
* chore: push
* feat(evals): add experiment_item_metadata mapping and validation
- Updated PUBLIC_MAPPING_SOURCE_TO_INTERNAL_COLUMN to include experiment_item_metadata.
- Enhanced SUPPORTED_MAPPING_SOURCES_BY_TARGET to support experiment_item_metadata for the experiment target.
- Modified ExperimentEvaluationRuleMappingSource to include experiment_item_metadata in the schema.
* fix: protect against correct mapping in front-end
* fix: protect against correct mapping in back-end
* Revert "fix: protect against correct mapping in back-end"
This reverts commit 6f7a1f4cc11b01a8233e055063610608d3dca2ef.
* fix(evals): include experiment metadata in batch eval stream
* fix(evals): update fern mapping source contract
## Summary
The v2 observations endpoint always returns `modelId`, `inputPrice`, `outputPrice`, and `totalPrice` on every response row. These fields are populated when the `model` field group is requested; otherwise they are `null`. They were flowing through the Zod `.loose()` passthrough undeclared, making them invisible in the API contract.
- **`fern/apis/server/definition/commons.yml`**: Add `modelId`, `inputPrice`, `outputPrice`, `totalPrice` to `ObservationV2` as `nullable<T>` (always present in the response, not optional). Includes gating docs explaining the `model` field group condition.
- **`web/src/features/public-api/types/observations.ts`**: Declare the three price fields in the `APIObservationV2` Zod schema alongside the existing `modelId`.
- **`web/src/pages/api/public/v2/observations/index.ts`**: Normalize `Decimal` price values to `number` in the route handler for wire-format parity with v1 (mirrors `transformDbToApiObservation`).
- **`packages/shared/src/domain/observation-field-groups.ts`**: Expand docstring with enrichment field gating notes and code cross-references.
* fix(blob-storage): harden endpoint connection validation
Block blob storage DNS rebinding for S3-compatible and Azure endpoints while keeping self-hosted validation opt-in via allowlist env vars.
* fix(blob-storage): address endpoint validation review feedback
* fix(blob-storage): keep secure storage agents alive
* test(blob-storage): run worker integration suite as self-hosted
* test(blob-storage): run integration suite as self-hosted
* feat(monitors): add Monitor Prisma schema and migration
* feat(monitors): add UNKNOWN severity as default for cold-start monitors
* feat(monitors): decouple Monitor.view from DashboardWidgetViews
* fix(monitors): align Monitor.id with cuid convention and wire createdBy/updatedBy FKs
## Summary
Upstream `@playwright/mcp` renamed `--save-trace` to `--save-session`. The current `latest` build (v0.0.75) rejects the old flag with `error: unknown option '--save-trace'` and the server exits immediately, so Claude Code (and any other MCP client launching the server via `.mcp.json`) reports `Failed to reconnect to playwright`. This blocks the `frontend-browser-review` skill end-to-end.
Swapping to `--save-session` is upstream's straight rename and keeps the same intent: Playwright MCP writes its session artifacts (including traces) under `--output-dir .playwright-mcp`, which is what the skill (`.agents/skills/frontend-browser-review/SKILL.md`) tells reviewers to inspect on failure.
## Impacted packages
- `.agents/config.json` — canonical MCP server config (source of truth)
- `.agents/README.md` — illustrative snippet kept in sync to avoid re-introducing the stale flag via copy-paste
Generated provider configs (`.mcp.json`, `.claude/`, `.codex/`, `.cursor/`, `.vscode/`) are regenerated by `pnpm run agents:sync` and remain gitignored per the agent-setup contract — no changes need committing there.
## Verification
- `pnpm run agents:sync` — regenerates all provider shims with the new flag
- `pnpm run agents:check` — clean
- Manual launch with the new args (`npx -y @playwright/mcp@latest --isolated --save-session --output-dir .playwright-mcp --test-id-attribute data-testid`) — process stays alive past handshake (old `--save-trace` exited with code 1)
- Live confirmation: with the fix applied locally, the Playwright MCP tools loaded successfully in my Claude Code session, whereas `/mcp` had previously reported `Failed to reconnect to playwright`
<!-- greptile_comment -->
<details open><summary><h3>Greptile Summary</h3></summary>
This PR corrects the Playwright MCP flag in the agent configuration from `--save-trace` to `--save-session`, which is the correct flag for automatic session recording per the Playwright MCP documentation.
- Both `.agents/config.json` (the canonical source) and `.agents/README.md` (which embeds the current JSON shape inline) are updated consistently, keeping documentation and config in sync.
- Generated shim files (`.claude/settings.json`, `.cursor/mcp.json`, etc.) are not committed to the repo and are regenerated from `.agents/config.json` via `pnpm run agents:sync`, so no further changes are needed in this PR.
</details>
<details><summary><h3>Confidence Score: 5/5</h3></summary>
Safe to merge — both changed files are updated consistently and the replacement flag is documented as correct by Playwright MCP.
The change swaps a single CLI flag in two files that are intentionally kept in sync (the canonical config and its embedded README snapshot). The flag --save-session is confirmed in the Playwright MCP documentation as the correct option for automatic session recording. Generated shim files are not committed and will pick up the corrected flag on the next pnpm install or agents:sync run.
No files require special attention.
</details>
<details><summary><h3>Flowchart</h3></summary>
```mermaid
%%{init: {'theme': 'neutral'}}%%
flowchart TD
A[".agents/config.json\n(canonical source)"] -->|pnpm run agents:sync| B["Generated Shim Files\n(.claude/settings.json\n.cursor/mcp.json\n.vscode/mcp.json\n.mcp.json etc.)"]
A -->|inline snapshot| C[".agents/README.md\n(documentation)"]
B --> D["Playwright MCP Server\nnpx @playwright/mcp@latest\n--isolated\n--save-session\n--output-dir .playwright-mcp\n--test-id-attribute data-testid"]
style D fill:#d4edda,stroke:#28a745
```
</details>
<sub>Reviews (1): Last reviewed commit: ["fix(agents): use --save-session for Play..."](https://github.com/langfuse/langfuse/commit/789bceb39b392e4a677e455c9d819ae864a17e8a) | [Re-trigger Greptile](https://app.greptile.com/api/retrigger?id=32588333)</sub>
<!-- /greptile_comment -->
## Summary
Follow-up to [LFE-9688](https://linear.app/langfuse/issue/LFE-9688) /
[#13627](https://app.graphite.com/github/pr/langfuse/langfuse/13627).
Tracked as [LFE-9830](https://linear.app/langfuse/issue/LFE-9830).
- After PR #13627 merged, post-cutoff Cloud projects saw a one-option Export Source dropdown. Team feedback: hide the field entirely.
- Wrap the FormField in `{showExportSourceField && (...)}` where `showExportSourceField = isBetaEnabled && !isPostCutoffCloud`. Composes with the existing `isPostCutoffCloud` derivation — `isLegacyBlobExportAllowed` is called exactly once per render.
- Form value stays pinned to `EVENTS` via the existing `defaultValues` / `reset` logic so submission is valid even with the field hidden (`react-hook-form` retains values of unmounted fields by default).
- Drops dead `availableExportSourceOptions` filter, the unreachable `isPostCutoffCloud` arm of `FormDescription`, and the unused `LEGACY_BLOB_EXPORT_SOURCES` import.
### Why no unit test
The rendering condition is a single-line AND of two existing booleans. The cutoff-bracket logic lives in `isLegacyBlobExportAllowed` (covered by its own tests in the shared package). Browser review of the affected page is the intended safety net — see the Test plan below.
### CI fix (fixup commit)
This PR also removes `--experimental-cli` from the `prettier-check` CI job (`pipeline.yml`). The flag's glob resolver treats bracket characters in Next.js dynamic-route paths (e.g. `[projectId]`) as glob character classes, causing exit 123 ("No files matching the given patterns were found") for any PR that touches a file under such a directory. `blobstorage.tsx` lives under `[projectId]`, which is what triggered the failure here. Standard prettier resolves explicit file paths correctly; the parallelism and ephemeral cache that `--experimental-cli` adds provide no practical benefit when checking a handful of changed files per PR.
### Impacted packages
- `web` — single page component, net 10+/26−.
- `.github/workflows/pipeline.yml` — CI prettier-check fix.
## Test plan
- [x] `pnpm --filter web run typecheck`
- [x] `pnpm --filter web exec vitest run --project=client src/__tests__/blob-storage-form-field-groups.clienttest.ts` — 5/5 passed (regression check on related form schema)
- [x] Reproduced prettier-check failure locally with the old command; confirmed fix passes with the new command.
- [ ] Browser review on Cloud with a seeded pre-cutoff and a seeded post-cutoff project (assert Export Source field hidden in the post-cutoff case, visible in the pre-cutoff case)
- [ ] Self-hosted parity check (`LANGFUSE_CLOUD_REGION` unset → field visible)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
## Summary
Follow-up to [LFE-9688](https://linear.app/langfuse/issue/LFE-9688) /
[#13627](https://app.graphite.com/github/pr/langfuse/langfuse/13627),
which intentionally scoped the cutoff gate to blob-storage and listed
PostHog / Mixpanel under "Out of scope". Tracked as
[LFE-9838](https://linear.app/langfuse/issue/LFE-9838); planning doc:
`ideabox/Implementations/Proposed/2026-05-18 extend-cutoff-gate-to-posthog-mixpanel.md`.
- Post-cutoff Cloud projects (`createdAt >= 2026-05-20`) now see the Export Source field hidden in PostHog and Mixpanel settings pages (form value pinned to `EVENTS` via `defaultValues`).
- The matching tRPC `update` mutations reject any legacy `exportSource` (`TRACES_OBSERVATIONS`, `TRACES_OBSERVATIONS_EVENTS`) for post-cutoff Cloud projects with `BAD_REQUEST`.
- Pre-cutoff Cloud projects and self-hosted deployments keep full choice — no behavior change.
- Pure parity work: reuses the shared `isLegacyBlobExportAllowed` predicate and `assertLegacyBlobExportSourceAllowed` guard. No new constants, no shared-package edits, no public REST surface to gate (neither integration has one).
- The `LEGACY_BLOB_EXPORT_*` / `assertLegacyBlobExportSourceAllowed` names retain their "Blob" prefix; renaming is deferred to a separate cleanup PR (see planning doc, Decision #2).
### Impacted packages
- `web` — two settings pages, two routers, one extended servertest, one new servertest. No other package touched.
## Test plan
- [x] `pnpm --filter web run typecheck`
- [x] `pnpm --filter web exec vitest run --project=server src/__tests__/server/posthog-integration.servertest.ts src/__tests__/server/mixpanel-integration.servertest.ts` — 11/11 passed (PostHog 6, Mixpanel 5)
- [x] Browser review on dev (Playwright MCP): post-cutoff cloud projects (dev `.env` overrides cutoff to `2020-01-01`) — Export Source hidden on both PostHog and Mixpanel settings pages, Enabled switch + other form fields still render correctly
- [ ] Browser review with pre-cutoff Cloud (toggle `NEXT_PUBLIC_LANGFUSE_BLOB_EXPORT_CUTOFF` to a future date, restart dev server)
- [ ] Self-hosted parity check (`LANGFUSE_CLOUD_REGION` unset → field visible for any `createdAt`)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- greptile_comment -->
<details open><summary><h3>Greptile Summary</h3></summary>
This PR extends the legacy export source cutoff gate (originally applied to blob-storage in LFE-9688) to the PostHog and Mixpanel analytics integrations. Post-cutoff Cloud projects (`createdAt >= 2026-05-20`) can no longer save a legacy `exportSource` value; the field is hidden in the UI and pinned to `EVENTS`, while the tRPC `update` mutations enforce the same rule server-side.
- **Routers**: Both `posthogIntegrationRouter` and `mixpanelIntegrationRouter` gain the same `assertLegacyBlobExportSourceAllowed` guard that already protects the blob-storage router; the gate is correctly placed before the audit log and DB write.
- **UI pages**: Both settings pages derive `isPostCutoffCloud` from `useQueryProject` + `isLegacyBlobExportAllowed`, hide the Export Source `FormField` when that flag is true, and pin the form default to `EVENTS` — exactly mirroring the blob-storage page pattern.
- **Tests**: New `servertest` files (and an extended PostHog file) cover all five gate scenarios (pre-cutoff Cloud allow, two legacy-source rejections, `EVENTS` allow, self-hosted bypass) using a shared `buildSession` helper refactored from the existing SSRF test.
</details>
<details><summary><h3>Confidence Score: 5/5</h3></summary>
Safe to merge — the server-side gate is correctly placed and always reachable, the UI correctly hides and pins the field, and tests cover all five gate scenarios for both integrations.
The change is tightly scoped: two routers gain the same guard already proven in blob-storage, two settings pages hide a single field for post-cutoff Cloud projects, and tests exercise every code branch. No new public API surface is added, and self-hosted / pre-cutoff behaviour is unchanged.
No files require special attention. The only observation is a duplicated buildSession helper in the two new test files, which is a maintenance concern rather than a functional one.
</details>
<details><summary><h3>Flowchart</h3></summary>
```mermaid
%%{init: {'theme': 'neutral'}}%%
flowchart TD
A[User submits PostHog / Mixpanel settings form] --> B{exportSource provided?}
B -- "always truthy after Zod .default()" --> C[Fetch project.createdAt from DB]
C --> D{Is legacy export source?}
D -- No: EVENTS --> E[Allow — skip gate]
D -- Yes: TRACES_OBSERVATIONS / TRACES_OBSERVATIONS_EVENTS --> F{isCloud AND project.createdAt >= cutoff?}
F -- No: self-hosted OR pre-cutoff --> G[Allow]
F -- Yes: post-cutoff Cloud --> H[Throw InvalidRequestError → BAD_REQUEST]
E --> I[Audit log + DB upsert]
G --> I
```
</details>
<details><summary>Prompt To Fix All With AI</summary>
`````markdown
Fix the following 1 code review issue. Work through them one at a time, proposing concise fixes.
---
### Issue 1 of 1
web/src/__tests__/server/mixpanel-integration.servertest.ts:13-44
**Duplicated `buildSession` helper across test files**
The `buildSession` function in this file is byte-for-byte identical to the one added to `posthog-integration.servertest.ts`. If the session shape ever changes (e.g., a new required project field), both copies need updating in sync. Consider extracting it to a shared test utility (e.g., `web/src/__tests__/server/fixtures/session.ts`) so there is a single source of truth.
`````
</details>
<sub>Reviews (1): Last reviewed commit: ["feat(analytics-integrations): extend leg..."](https://github.com/langfuse/langfuse/commit/6418f93afafa9dede9899f65747256c8e42fd309) | [Re-trigger Greptile](https://app.greptile.com/api/retrigger?id=32589990)</sub>
<!-- /greptile_comment -->
* refactor(shared): promote query feature to @langfuse/shared (LFE-9806)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(shared): import query server deps from source modules
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore(shared): drop no-barrel comment; qualify mapDashboards path
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(shared): inject rootEventCondition threshold into QueryBuilder
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(web): move dashboardUiTableToViewMapping back to dashboard/lib
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(shared): flatten features/query — drop server/ subdir
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor: prefer direct subpath imports for query module
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* revert(shared): drop QueryBuilder rootEventCondition override; self-import env
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(shared): read rootEventCondition threshold from process.env
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(shared): address review feedback on query module
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* revert(shared): rolled back diff to bare minumum
* fix(shared): added back an explicit query export following AGENTS.md guildelines
* fix(shared): added a formal threshold hours overried to side step the dual package hazard
* fix(web): missing query imports in execute query stream
* fix(shared): split query server-only files under @langfuse/shared/query/server
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(web): route QueryBuilder/executeQuery imports in 3 tests via /query/server
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Summary
- Cloud projects created on or after **2026-05-20T00:00:00Z** can no longer use `LEGACY_TRACES_OBSERVATIONS` or `LEGACY_TRACES_AND_ENRICHED_OBSERVATIONS`. Attempts via tRPC or the public REST API return `BAD_REQUEST` / HTTP 400.
- Self-hosted deployments and projects created before the cutoff are fully unaffected.
- Single shared helper (`assertLegacyBlobExportSourceAllowed`) enforces the rule identically on both write surfaces.
- Settings UI hides legacy options and defaults to `OBSERVATIONS_V2` for post-cutoff Cloud projects, with an inline message explaining the restriction.
- `project.createdAt` added to the NextAuth session so the UI can derive the gate without an extra DB round-trip.
## What does this PR do?
Fixes a wire-format leak in the V2 observations endpoint where `traceName`, `tags`, `release`, `userId`, `sessionId`, `bookmarked`, and `public` appeared as `null` keys in responses even when the client did not request the `trace_context` field group.
**Root cause:** `convertEventsObservation` in `observations_converters.ts` used unconditional `record.x ?? null` assignments for those seven fields regardless of the `complete` flag. When ClickHouse omits a column from the projection (because the field group was not requested), the value is `undefined`, and `undefined ?? null === null` — so the key was always emitted. The peer converter `convertObservationPartial` already uses conditional spreads (`...(record.x !== undefined && { x: record.x })`) to enforce this discipline; `convertEventsObservation` deviated from that pattern.
**Fix:**
- Split the `complete`/partial branches in `convertEventsObservation`. The `complete: true` (V1) branch keeps the unconditional defaults since V1 always returns all fields. The `complete: false` (V2) branch gates each extra field on presence, matching `convertObservationPartial`.
- Tighten the contract test assertion in `observations-api-v2.servertest.ts`: removes the `?? undefined` softening that allowed `null` values to pass `toBeUndefined()`.
- Add a unit test for `convertEventsObservation` directly — no such test existed before.
## Type of change
- [x] Bug fix (non-breaking change which fixes an issue)
## Impacted packages
- `packages/shared` — `observations_converters.ts`
- `web` — contract test + new unit test
## Verification
- `pnpm --filter @langfuse/shared run lint` ✓
- `pnpm --filter @langfuse/shared run typecheck` ✓
- CI green (tests-web, tests-worker, e2e)
<!-- greptile_comment -->
<details open><summary><h3>Greptile Summary</h3></summary>
This PR fixes a wire-format leak in the V2 observations endpoint where seven `trace_context` fields (`traceName`, `tags`, `release`, `userId`, `sessionId`, `bookmarked`, `public`) appeared as explicit `null` keys even when the client did not request that field group. The root cause was `undefined ?? null === null` in the single shared code path.
- **Core fix** (`observations_converters.ts`): Splits `convertEventsObservation` into separate `complete: true` (V1) and `complete: false` (V2) branches. The V1 path keeps unconditional `?? null` defaults; the V2 path gates each field on `!== undefined` using conditional spreads, matching the pattern already used in `convertObservationPartial`.
- **Contract test** (`observations-api-v2.servertest.ts`): Tightens the assertion from `obs[field] ?? undefined` (which let `null` pass `toBeUndefined()`) to a direct `obs[field]` check, so the test now correctly fails when a field leaks as `null`.
- **New unit test** (`observations-converters.servertest.ts`): Adds direct coverage for `convertEventsObservation` that was previously missing, testing both branches with absent, null, and non-null field values.
</details>
<details><summary><h3>Confidence Score: 4/5</h3></summary>
The production change to `observations_converters.ts` is safe: the fix is minimal, well-scoped, and the conditional-spread pattern it introduces for the V2 path already exists in the peer converter.
The converter change and the contract-test tightening are both correct. The new unit test has two TypeScript type incompatibilities (`tags: null` where only `string[] | undefined` is valid, and `undefined` passed for a required `RenderingProps` parameter). Both are caught only at type-check time — the PR description reports running typecheck only for `@langfuse/shared`, not for the `web` package where the new test lives. The issues don't affect runtime or production behaviour, but they leave the test file in a state that fails strict type-checking.
web/src/__tests__/server/unit/observations-converters.servertest.ts — two call-site type errors; all other files are clean.
</details>
<details><summary>Prompt To Fix All With AI</summary>
`````markdown
Fix the following 2 code review issues. Work through them one at a time, proposing concise fixes.
---
### Issue 1 of 2
web/src/__tests__/server/unit/observations-converters.servertest.ts:137-143
The `makeRecord` override passes `tags: null`, but `tags` in `EventsObservationRecordReadType` is typed as `z.array(z.string()).optional()` — i.e. `string[] | undefined` — which is not nullable. Passing `null` here is a TypeScript type error that would surface under `tsc --noEmit` on the `web` package. The intent of the test (verifying `?? null` defaulting) is better expressed by omitting `tags` entirely (letting it be `undefined`) or by asserting that the complete path emits `null` when the field is absent rather than null in the row.
```suggestion
const record = makeRecord({
user_id: null,
session_id: null,
trace_name: null,
release: null,
// tags is omitted → undefined in the record; the converter defaults it to null
});
```
### Issue 2 of 2
web/src/__tests__/server/unit/observations-converters.servertest.ts:70-72
The overload signatures for `convertEventsObservation` declare `renderingProps` as a required `RenderingProps` parameter (not `RenderingProps | undefined`). Passing `undefined` here works at runtime (the implementation has a default value), but TypeScript checks call sites against the overloads and will flag this as a type error. The same pattern appears at the other `convertEventsObservation` call sites in this file. Passing the exported `DEFAULT_RENDERING_PROPS` is the idiomatic fix and makes the intent explicit — note that the import for `DEFAULT_RENDERING_PROPS` from `@langfuse/shared/src/server` would also need to be added.
```suggestion
const result = convertEventsObservation(record, DEFAULT_RENDERING_PROPS, false);
for (const field of TRACE_CONTEXT_FIELDS) {
```
`````
</details>
<sub>Reviews (1): Last reviewed commit: ["fix(test): import convertEventsObservati..."](https://github.com/langfuse/langfuse/commit/5074fe72723a7812e8ff7f3b6ff0f9d0820ed316) | [Re-trigger Greptile](https://app.greptile.com/api/retrigger?id=32323301)</sub>
<!-- /greptile_comment -->
The self-service SSO form exposes `idToken` but not `scope`. When an
admin sets `idToken: false` (for IdPs that release email only via the
userinfo endpoint), the stored config inherits the runtime default
`openid email profile` scope. NextAuth then chooses
`client.oauthCallback()` (idToken === false branch), which openid-client
refuses with
"id_token detected in the response, you must use client.callback()
instead of client.oauthCallback()"
because the IdP still returns an id_token whenever `openid` is in scope.
Normalize the stored scope at write time in `ssoConfig.save`: when the
saved provider is `custom` and `idToken === false`, drop the `openid`
token from the scope (falling back to `email profile` if stripping
leaves it empty). This also handles the merge case where the existing
config (often written via the legacy admin endpoint) supplied a scope
that the new save needs to bring into line.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The events.all read path returned 500 when an observation's
model_parameters contained a non-JSON sentinel string (e.g. Python
SDK v4's "<not serializable object of type: dict>"). Replace the bare
JSON.parse in convertObservationPartial with parseJsonPrioritised so
unparseable values fall through to the raw string instead of throwing
for the entire row. This converter feeds both convertObservation and
convertEventsObservation.
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
## Summary
Stacked on top of [#13617](https://app.graphite.com/github/pr/langfuse/langfuse/13617) (the `OBSERVATION_FIELD_GROUPS` / `BLOB_EXPORT_FIELD_GROUPS` split). Exposes `trace_context` on the public `/api/public/v2/observations` API after that split, restoring the group to the v2 contract with a documented surface (it had been incidentally available via the un-narrowed constant before #13617).
- Adds `trace_context` back to `OBSERVATION_FIELD_GROUPS` — the narrowed list of public v2 groups, distinct from `BLOB_EXPORT_FIELD_GROUPS`. `tools` stays only on the blob-export side as before.
- Fern source documents `trace_context` in both the field-selection list and the `Available groups` line on the `fields` query parameter. Regenerated `openapi.yml` propagates to the SDK clients.
- Columns exposed: `tags`, `release`, `traceName` (denormalized trace metadata). `usagePricingTierName` was moved to the `usage` group by #13617 and is documented there.
### How does this differ from pre-#13617 behavior?
Before #13617, `trace_context` was already accepted at runtime via the Zod filter against `OBSERVATION_FIELD_GROUPS` — but the Fern docstring listed only 9 of 11 groups, so it was undocumented. #13617 narrowed the constant for safety (separating v2 API from blob-export selection). This PR restores `trace_context` on the v2 side intentionally, with explicit Fern documentation, while leaving `tools` blob-export-only.
### Test coverage
Extends the parametrized `field group contract` loop in `observations-api-v2.servertest.ts` with a `trace_context` row that asserts the three denormalized fields flow through when `fields=trace_context` is requested. Uses fixture values for `tags`, `release`, `traceName` so a null regression is caught.
### Impacted packages
- `@langfuse/shared` — re-adds `trace_context` to `OBSERVATION_FIELD_GROUPS`; updates the docblock to reflect that only `tools` is now in the broader blob-export set
- `web` — extends servertest coverage; regenerated `openapi.yml`
- `fern` — observations endpoint docstring
## Test plan
- [x] `pnpm --filter web run typecheck`
- [x] `dotenv -e .env -- pnpm --filter web exec vitest run observations-api-v2` — 27/27 passed
- [ ] Manual: verify regenerated SDKs (Python, TypeScript) include `trace_context` as a valid `fields` value
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- greptile_comment -->
<details open><summary><h3>Greptile Summary</h3></summary>
This PR re-adds `trace_context` to `OBSERVATION_FIELD_GROUPS` so that `tags`, `release`, and `traceName` are exposed on the public `/api/public/v2/observations` endpoint, and documents the group in both the Fern definition and the generated OpenAPI spec.
- **`packages/shared`**: `trace_context` is appended to `OBSERVATION_FIELD_GROUPS`; docblock updated to reflect `tools` is now the only blob-export-only group.
- **`fern` / `openapi.yml`**: `trace_context` and its three columns are added to the field-selection list and the `Available groups` doc line.
- **Server test**: parametrized contract test extended with a `trace_context` row and fixture values for `tags`, `release`, `traceName`, but `ALL_NON_CORE_FIELDS` (the sentinel list used for absence checks) is not updated, leaving a gap in isolation coverage.
</details>
<details><summary><h3>Confidence Score: 4/5</h3></summary>
The implementation change itself is straightforward and isolated; the test gap means field-isolation regressions won't be caught by the existing suite.
Adding three fields to `ALL_NON_CORE_FIELDS` is the only change needed to complete the test contract — without it, the parametrized absence loop never fires for `traceName`, `tags`, or `release` when a different group is requested, so a future leak of `trace_context` data into unrelated group responses would go undetected in CI.
web/src/__tests__/server/observations-api-v2.servertest.ts — the `ALL_NON_CORE_FIELDS` constant needs to include `traceName`, `tags`, and `release`.
</details>
<details><summary><h3>Sequence Diagram</h3></summary>
```mermaid
sequenceDiagram
participant Client
participant PublicAPI as /api/public/v2/observations
participant QueryBuilder as buildObservationsQueryComponents
participant CH as ClickHouse (events table)
participant Traces as traces CTE
Client->>PublicAPI: "GET ?fields=trace_context&traceId=..."
PublicAPI->>QueryBuilder: "fields=["trace_context"]"
QueryBuilder->>QueryBuilder: Validates group in OBSERVATION_FIELD_GROUPS
QueryBuilder->>Traces: JOIN traces CTE (tags, release, traceName)
QueryBuilder->>CH: SELECT core + trace_context columns
CH-->>PublicAPI: rows with tags, release, traceName
PublicAPI-->>Client: "{ data: [{ id, traceId, ..., tags, release, traceName }] }"
```
</details>
<details><summary>Prompt To Fix All With AI</summary>
`````markdown
Fix the following 1 code review issue. Work through them one at a time, proposing concise fixes.
---
### Issue 1 of 1
web/src/__tests__/server/observations-api-v2.servertest.ts:538-542
The `ALL_NON_CORE_FIELDS` list is not updated with the three fields from the new `trace_context` group (`traceName`, `tags`, `release`). The absence-check loop only iterates over members of this list, so when any other group (e.g. `basic`, `io`, `usage`) is tested in isolation, the test never asserts that `traceName`, `tags`, and `release` are absent from the response. A regression that leaks `trace_context` fields into unrelated group responses would pass undetected.
```suggestion
// prompt
"promptId",
"promptName",
"promptVersion",
// trace_context
"traceName",
"tags",
"release",
] as const;
```
`````
</details>
<sub>Reviews (1): Last reviewed commit: ["feat(observations-v2): expose trace\_cont..."](https://github.com/langfuse/langfuse/commit/ae0d5f26f2afb8019cb6a7aff75452d0184b08cc) | [Re-trigger Greptile](https://app.greptile.com/api/retrigger?id=31989455)</sub>
<!-- /greptile_comment -->
* refactor(blob-export): split OBSERVATION_FIELD_GROUPS from BLOB_EXPORT_FIELD_GROUPS
OBSERVATION_FIELD_GROUPS was driving both the public /v2/observations API
contract and the blob exporter's column selection. The blob exporter needs
a broader set (tools, trace_context) that shouldn't silently leak into the
public observations API.
Decouple the two:
- OBSERVATION_FIELD_GROUPS stays narrow (9 groups) — drives /v2/observations
- BLOB_EXPORT_FIELD_GROUPS owns the broader 11 groups — drives blob exporter
- Worker handler now imports BLOB_EXPORT_FIELD_GROUPS from the shared
analytics-integrations module, not the repository symbol
Also relocate usagePricingTierName from trace_context to usage. It's a
pricing-tier attribute on the observation, not part of the trace context.
The /v2/observations API now exposes it under the usage group; the Fern
docstring and openapi.yml are updated to match. Blob export's
trace_context shrinks accordingly.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* feat(observations-v2): expose usagePricingTierName field in API response
Adds usagePricingTierName to the v2 observations API response contract:
- Fern commons.yml: optional<nullable<string>> on Observations type
- Zod APIObservationV2: nullable + optional
- Regenerated openapi.yml propagates to SDKs
usagePricingTierName was already runtime-selectable via the `usage`
field group (FIELD_SETS.usage in event-query-builder.ts:296), but the
public response contract didn't declare it — so the value was being
stripped on the way out. This adds the field to the surface that
matches the selection.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* refactor(observations): own field-group vocabulary in domain layer
Per review feedback: the events repository depended on a feature-flavored
type (`BLOB_EXPORT_FIELD_GROUPS` / `BlobExportFieldGroup`) defined in
`features/analytics-integrations`. A foundational component shouldn't
depend on a feature-named type.
Move both group lists to a new client-safe domain module
(`packages/shared/src/domain/observation-field-groups.ts`):
- `OBSERVATION_FIELD_GROUPS_PUBLIC_API` (was `OBSERVATION_FIELD_GROUPS`):
the v2 public API contract — 9 groups exposed by the v2 observations
endpoint.
- `OBSERVATION_FIELD_GROUPS_FULL` (was `BLOB_EXPORT_FIELD_GROUPS`): the
complete set of column groups the events repository can project —
adds `tools` and `trace_context` on top of the API surface. Mirrors
the existing `events_full` / `events_core` ClickHouse naming.
`events.ts` (repository) and `analytics-integrations/index.ts` (feature)
both consume from the domain module instead of from each other. Frontend
forms, Zod enums, and worker jobs reach the values via the existing
`@langfuse/shared` barrel. No behavior change.
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* test(otel): add e2e tenant isolation test for OTEL ingestion
Adds a server-side e2e test (LFE-9771) proving that a span POSTed via
project A's API key lands exclusively in project A's observations table
and is absent from project B's. Guards the auth-scope invariant
(projectId resolved from API key, never from the OTLP payload) across
the web route, BullMQ job payload, and ClickHouse write layers.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* style: apply prettier formatting to otel tenant isolation test
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* test(e2e): fix fragile Redis key count assertion in ingest trace test
The `ingest a trace` test asserted exactly 1 `api-key:*` key in Redis,
which breaks when another e2e test file runs concurrently and caches its
own API key. Replace the count check with a lookup by projectId so the
assertion is robust against parallel test execution.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* add assertion
* test(otel): clean up created orgs in tenant isolation test afterAll
Claude CI review on PR #13622 flagged that the two test orgs/projects created
via createOrgProjectAndApiKey() were never deleted. Add an afterAll that
deletes them by org ID — Prisma cascades to project and apiKey rows.
Track org IDs in module scope and push immediately after each creation so
the cleanup fires even if a later assertion fails or the test times out.
Mirrors the pattern used by blob-storage-integration-trpc.servertest.ts.
ClickHouse observation rows from the test span aren't cleaned up here —
they're partitioned by the new project_id which won't be reused, so they're
inert. The Postgres org/project rows are the actual leak.
* test(otel): dedup observations count to tolerate ReplacingMergeTree retries
Claude CI review on PR #13622 flagged that the bare `SELECT count() FROM
observations` returns physical (pre-merge) rows on the ReplacingMergeTree.
If the OtelIngestionQueue retries the ingestion job during the 40s
waitForExpect window (attempts: 6 per otelIngestionQueue.ts), a second
insert for the same span lands and `toBe(1)` fails for a retry reason,
not a tenant-isolation reason.
Wrap the count in the repo's standard dedup pattern:
`ORDER BY event_ts DESC + LIMIT 1 BY id, project_id` (same shape used
throughout observations.ts). The count of the deduped subquery is 0 or 1
regardless of how many physical inserts occurred.
---------
Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* refactor(blob-export): replace internal exportSource enum with public LEGACY/ENRICHED/LEGACY_AND_ENRICHED
The REST API was exposing AnalyticsIntegrationExportSource (a Prisma enum
intended as an internal identifier) directly to public consumers. Its values
— TRACES_OBSERVATIONS, EVENTS, TRACES_OBSERVATIONS_EVENTS — don't match the
UI labels users see ("Enriched observations" etc.) and bake legacy internal
naming into the public contract.
Introduce a distinct public enum and a mapping layer:
- Public values: LEGACY, ENRICHED, LEGACY_AND_ENRICHED
- toInternalExportSource / toPublicExportSource bidirectional helpers
- PUT handler maps public → internal before Prisma; GET responses map
internal → public before serializing
- Fern docstring references /api/public/v2/observations for ENRICHED so
consumers have a concrete anchor for the data model
This is a hard break of the enum values exposed in PR #13598 (merged ~5h
ago); no SDKs are believed to have been published or integrated against
those values. The internal AnalyticsIntegrationExportSource enum (Prisma,
tRPC, UI) is unchanged.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* test(blob-export): align test labels with public enum and add LEGACY_AND_ENRICHED coverage
Two CI review nits on the test file:
1. Renames — describe/it labels, section comments, and the
`tracesObsIntegration` / `eventsIntegration` variable names now use the
public enum values (LEGACY, ENRICHED, LEGACY_AND_ENRICHED) instead of the
legacy internal names (TRACES_OBSERVATIONS, EVENTS,
TRACES_OBSERVATIONS_EVENTS). The Prisma-direct seeds and DB-read
assertions inside the test bodies still use internal values, since those
reach the Postgres enum directly.
2. New regression test — adds `GET response maps internal
TRACES_OBSERVATIONS_EVENTS to public LEGACY_AND_ENRICHED`. The existing
multi-project test only covered the first two public values; the third
was relying on compile-time exhaustiveness via `satisfies Record<…>` in
INTERNAL_TO_PUBLIC_EXPORT_SOURCE, which won't catch a copy-paste error.
A runtime assertion through the public REST surface closes that gap.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* refactor(blob-export): rename public exportSource values to be self-descriptive
Per colleague feedback on PR #13619, swap the public REST enum tag-style
names for more self-describing identifiers:
- LEGACY → LEGACY_TRACES_OBSERVATIONS
- ENRICHED → OBSERVATIONS_V2
- LEGACY_AND_ENRICHED → LEGACY_TRACES_AND_ENRICHED_OBSERVATIONS
Updates Fern source, regenerated openapi.yml, the public-API Zod schema's
mapping helpers, and the server-test fixtures. Internal Prisma enum values
(TRACES_OBSERVATIONS / EVENTS / TRACES_OBSERVATIONS_EVENTS) are unchanged —
this is purely a public-surface rename, isolated by the toPublic /
toInternal mapping helpers introduced in this PR.
* updated API docs
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
fix(otel): recognize OpenInference prompt_details.cache_read/cache_write
Adds prompt_details.cache_read and prompt_details.cache_write to the
cache token resolver in extractGenericGenAiUsageDetails so cache tokens
emitted by openinference-instrumentation-* (openai, anthropic, agno) are
normalized into Langfuse's canonical input_cached_tokens /
input_cache_creation usage_details keys instead of being passed through
as opaque raw keys.
Without this, Langfuse ingests the values (they show up in usageDetails
as prompt_details.cache_read etc.) but the cost engine prices input at
full base rate and silently skips the cache keys, leading to ~30%
under-reported cost on cache-heavy observations. The values also are not
subtracted from input, which risks double-counting depending on what the
instrumentor populates.
llm.token_count.prompt_details.cache_read and cache_write are the
canonical OpenInference semantic-convention names (defined in
openinference-semantic-conventions/src/openinference/semconv/trace/__init__.py),
emitted by every OpenInference instrumentor that supports prompt caching.
The right place to fix is here, not upstream.
Same shape of fix as #12248 (pydantic-ai cache token names).
Fixes#13571 (partial — addresses the OpenInference half of #12635).
Co-authored-by: gragragrab <12702336+gragragrab@users.noreply.github.com>
Co-authored-by: Hassieb Pakzad <68423100+hassiebp@users.noreply.github.com>
## Summary
[LFE-9456](https://linear.app/langfuse/issue/LFE-9456) — final part of the stack. Exposes `exportSource` and `exportFieldGroups` on the public REST API for blob storage integrations.
- **Request**: `CreateBlobStorageIntegrationRequest` now accepts `exportSource` (optional, defaults to `TRACES_OBSERVATIONS`) and `exportFieldGroups` (`nullable<list<ExportFieldGroup>>`, optional).
- **Response**: `BlobStorageIntegrationResponse` now returns `exportSource` (non-null) and `exportFieldGroups` (`nullable<list<…>>`).
- **Strict validation**: REST contract is intentionally stricter than tRPC:
- `TRACES_OBSERVATIONS` + non-null `exportFieldGroups` → 400 ("not applicable"). Covers `[]`, partial arrays, full arrays alike.
- `EVENTS` / `TRACES_OBSERVATIONS_EVENTS` + provided `exportFieldGroups` without `core` → 400 (delegates to existing shared `validateExportFieldGroups`).
- **Source-conditional handler**: `TRACES_OBSERVATIONS` writes `undefined` (Prisma preserves the column / applies default for new rows) and reads return `null` to hide any inert legacy value. `EVENTS` family defaults to all 11 groups when omitted/null.
- **Fern source updated** with new `ExportSource` / `ExportFieldGroup` enums, request and response field additions, and rule docstrings. OpenAPI spec regenerated.
### Why the REST/tRPC divergence is intentional
The UI's tRPC path still submits all 11 groups for `TRACES_OBSERVATIONS` because the form always carries them; the shared `validateExportFieldGroups` doesn't enforce `core` for that source. The worker ignores the column entirely for `TRACES_OBSERVATIONS` (uses fixed-column exports). The REST contract should not expose a knob that's inert at export time — so it rejects on write and hides on read.
### Drive-by
Includes `openapi.yml` regen sweep for #13126 (the `every_20_minutes` enum value was added to Fern source but never regenerated). Generated artifact only; no behavior change.
### Impacted packages
- `web` — Zod request/response types, GET list + PUT handlers, server tests, regenerated OpenAPI spec
- `fern` — Fern source definition
## Test plan
- [x] `pnpm --filter web run lint`
- [x] `pnpm --filter web run typecheck`
- [x] `dotenv -e .env -- pnpm --filter web exec vitest run blob-storage-integration-api blob-storage-integration-trpc` — 50/50 passed (38 new REST + 12 tRPC regression)
- [x] `npx fern-api generate --api server` — Python SDK, TypeScript SDK, OpenAPI spec all regenerated cleanly
- [ ] Manual API client smoke test against staging once merged (verify SDK round-trip on EVENTS payload)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- greptile_comment -->
<details open><summary><h3>Greptile Summary</h3></summary>
This PR exposes `exportSource` and `exportFieldGroups` on the public REST API for blob storage integrations, with strict validation rules (TRACES_OBSERVATIONS rejects non-null field groups; EVENTS/TRACES_OBSERVATIONS_EVENTS requires `core` when groups are provided) and source-conditional response masking.
- **PUT handler**: correctly defaults a null/omitted `exportFieldGroups` to all 11 groups for `EVENTS`/`TRACES_OBSERVATIONS_EVENTS`, and passes `undefined` for `TRACES_OBSERVATIONS` so Prisma preserves the existing DB column without overwriting it.
- **GET handler and PUT response block**: both cast the raw DB `exportFieldGroups` value without applying the same null-→-all-groups default that the write path uses, meaning legacy `EVENTS` rows with a null DB column return `null` in the response rather than the full group list.
- **Test schema**: local `BlobStorageIntegrationResponseSchema` marks `exportSource` as `.optional()`, which is weaker than the production contract; tests would not fail if the field were accidentally dropped from the response.
</details>
<details><summary><h3>Confidence Score: 3/5</h3></summary>
The write path is correct and well-tested, but the read path has an inconsistency: GET returns raw DB null for EVENTS integrations with an unset exportFieldGroups column, while PUT always writes the full default list. Any legacy EVENTS row would expose this gap to API consumers.
The write-side logic (defaulting, masking, validation) is solid and tests cover it well. The inconsistency lives in the GET handler and the PUT response builder, both of which skip the null-to-all-groups normalization that the write path applies. For organizations with legacy EVENTS integrations (created before exportFieldGroups was populated), the GET response would return null for a field that should carry the full 11-group list, which could mislead consumers and break SDK round-trips.
web/src/pages/api/public/integrations/blob-storage/index.ts — both response-building blocks (GET and PUT) need the same null-defaulting logic that the write path already has for EVENTS/TRACES_OBSERVATIONS_EVENTS sources.
</details>
<details><summary><h3>Flowchart</h3></summary>
```mermaid
%%{init: {'theme': 'neutral'}}%%
flowchart TD
A[PUT /integrations/blob-storage] --> B{Zod validation}
B -->|exportSource = TRACES_OBSERVATIONS\nexportFieldGroups != null| C[400 not applicable]
B -->|exportSource = EVENTS/TOE\nexportFieldGroups provided without 'core'| D[400 core required]
B -->|valid| E{exportSource?}
E -->|TRACES_OBSERVATIONS| F[pass exportFieldGroups: undefined\nPrisma skips column on update]
E -->|EVENTS / TRACES_OBSERVATIONS_EVENTS| G[pass exportFieldGroups ?? all 11 groups]
F --> H[upsertBlobStorageIntegration]
G --> H
H --> I{Build response}
I -->|TRACES_OBSERVATIONS| J[exportFieldGroups: null]
I -->|EVENTS/TOE| K[exportFieldGroups: DB value as-is\nno null → all-groups default]
L[GET /integrations/blob-storage] --> M[fetch all org integrations]
M --> N{for each integration}
N -->|TRACES_OBSERVATIONS| O[exportFieldGroups: null]
N -->|EVENTS/TOE| P[exportFieldGroups: DB value as-is\nno null → all-groups default]
O --> Q[200 response array]
P --> Q
```
</details>
<details><summary>Prompt To Fix All With AI</summary>
`````markdown
Fix the following 2 code review issues. Work through them one at a time, proposing concise fixes.
---
### Issue 1 of 2
web/src/pages/api/public/integrations/blob-storage/index.ts:94-98
**GET doesn't normalize null `exportFieldGroups` for EVENTS sources**
The PUT handler defaults a null/omitted `exportFieldGroups` to all 11 groups for `EVENTS` and `TRACES_OBSERVATIONS_EVENTS` sources (`validatedData.exportFieldGroups ?? [...BLOB_EXPORT_FIELD_GROUPS]`). The GET handler here simply passes the raw DB value through, so a legacy `EVENTS` row where the column was never populated returns `null` instead of the full group list. A consumer reading the GET response on such a row sees `null`—indistinguishable from `TRACES_OBSERVATIONS`'s "not applicable" null—while a subsequent PUT would return the full list. The same cast-without-defaulting pattern repeats on the PUT response block at lines 213–217.
### Issue 2 of 2
web/src/__tests__/server/blob-storage-integration-api.servertest.ts:31-38
**Test schema marks `exportSource` optional — weaker than production contract**
The production `BlobStorageIntegrationResponse` schema requires `exportSource` as non-optional (it's always present in the response). Marking it `.optional()` in the test schema means the tests would pass even if the field were accidentally dropped from the API response, making the test suite weaker than intended for this new field.
`````
</details>
<sub>Reviews (1): Last reviewed commit: ["feat(blob-export): expose exportSource a..."](https://github.com/langfuse/langfuse/commit/8601e56bd7e996def93d14761a4419b607b77923) | [Re-trigger Greptile](https://app.greptile.com/api/retrigger?id=31769478)</sub>
> Greptile also left **2 inline comments** on this PR.
<!-- /greptile_comment -->
Emit langfuse.queue.clickhouse_writer.rows_dropped with entity_type tag
when rows are discarded after max flush attempts, alongside existing
error increments and logs.
Co-authored-by: Cursor <cursoragent@cursor.com>
Some Entra tenants emit an external or personal address in the `email`
claim while the tenant UPN sits in `preferred_username` / `upn`. When the
email domain doesn't match the configured SSO domain, fall back to either
of those if they're valid emails on the configured domain so the
reverse-domain check in `auth.ts` doesn't reject the user.
## Summary
[LFE-9456](https://linear.app/langfuse/issue/LFE-9456) — stacked on top of the DB migration PR.
Adds configurable field groups for the events blob export, covering the query layer, worker wiring, and UI.
**Query layer (`packages/shared`)**
- Rewrites `getEventsForBlobStorageExport` to select only the requested field groups instead of hardcoding all fields
- Adds two new field sets to the query builder: `trace_context` (tags, release, traceName, usagePricingTierName) and `model_export` (providedModelName, modelId, modelParameters — uses `model_id` alias for blob consumers)
- Extends `OBSERVATION_FIELD_GROUPS` from 9 → 11 groups (adds `tools`, `trace_context`)
**Worker (`worker`)**
- Wires `exportFieldGroups` from the Prisma record through `processBlobStorageExport` into the query function and `enrichObservationStream`
- Gates pricing enrichment (input/output/total price) on the `usage` field group — skips model lookup entirely when `usage` is not selected
- Drops `provided_model_name` and `model_parameters` from enrichment output when `model` group is not selected
- Default = all 11 groups, so output is identical to the previous hardcoded path
**UI (`web`)**
- Adds a multi-checkbox field group selector to the blob storage settings form, visible when `exportSource` is `EVENTS` or `TRACES_OBSERVATIONS_EVENTS`
- Resets `exportFieldGroups` to all groups on source switch to prevent silent validation failures
- Surfaces tRPC mutation errors via toast
**Validation**
- `core` group is required and non-deselectable in the UI
- Schema enforces `core` must be present; `exportFieldGroups` must be non-empty when `exportSource` is `EVENTS`
- Extracts `validateExportFieldGroups` as a reusable validator shared between tRPC and schema
The tools popover capped visible cards at ~4 with no working scrollbar
because `<ScrollArea max-h-[...]>` lands the height constraint on the
Radix Root. The Viewport's `h-full` doesn't resolve against a parent
with only `max-height` (CSS resolves `height: 100%` against parent
`height`, not `max-height`), so the Viewport sized itself to content,
never reported overflow, and Radix never activated its scrollbar.
Meanwhile the Root's `overflow: hidden` clipped the rest.
Apply the height constraint to the Viewport via a Tailwind arbitrary
child variant so Radix sees the overflow and renders its scrollbar.
Closes#13433
Co-authored-by: Nimar <l.nimar.b@gmail.com>
Harden secure outbound fetches against DNS rebinding by validating
connection-time lookup results with the existing outbound URL blocklist and
whitelist policy.
Co-authored-by: Codex Opus 4.6 (1M context) <noreply@anthropic.com>
Part 2 of [LFE-9456](https://linear.app/langfuse/issue/LFE-9456) — stacked on top of the test-coverage PR.
- Adds `export_field_groups TEXT[] NOT NULL DEFAULT ARRAY[...]` column to `blob_storage_integrations`
- Adds `BLOB_EXPORT_FIELD_GROUPS` client-safe constant to `@langfuse/shared` (analytics-integrations module)
- Adds `exportFieldGroups` to the Zod form schema (`types.ts`) with `min(1)` validation and all-groups default
- Wires `exportFieldGroups` through `upsertBlobStorageIntegration` service and tRPC `update` mutation
- No behaviour change: default = all 11 groups = today's output
The 11-group default includes `tools` and `trace_context` which the query builder doesn't handle yet (PR 3). These values are stored but ignored by the worker until PR 3 is merged.
Part 1 of 4 for [LFE-9456](https://linear.app/langfuse/issue/LFE-9456) (export field group selection for blob storage integrations). No implementation changes — establishes regression baselines before the refactor.
* refactor(trace): rename folder from `trace2` to `trace`
* docs: rm `trace2` from code comments
* fixup: delete trace and observation preview files
* refactor(trace): update badge rendering logic in Observation and Trace detail views to conditionally display based on annotation mode
* chore: push
* fix: ensure observation id is selected
* fix(scim): block removing last organization owner
SCIM DELETE, PUT(active:false), and PATCH(active:false) deprovisioning paths
unconditionally removed the target user's organization membership, allowing
the last OWNER to be deleted and orphaning the organization.
Mirror the tRPC `deleteMembership` invariant: count remaining OWNERs and
reject with 403 when the request would remove the final OWNER. The error
body uses the SCIM error schema with the same message as the tRPC path.
Tests cover all three deprovisioning verbs against a sole owner plus a
positive control where a second OWNER exists.
Resolves INT-1223.
* fix(scim): wrap last-owner check + delete in serializable txn
Closes a TOCTOU race where two concurrent SCIM deprovision requests with
exactly two OWNERs could both pass the owner-count guard and both delete,
leaving the org with zero owners. The check and delete now run inside a
single Prisma transaction with Serializable isolation; on a serialization
failure (P2034) the endpoint returns 409 so the SCIM client retries.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
ADMIN previously held organization:CRUD_apiKeys, allowing any ADMIN
to mint organization-scoped API keys. Combined with SCIM PUT not
enforcing role hierarchy, this enabled ADMIN -> OWNER privilege
escalation (INT-1222). Removing the scope from ADMIN closes the
escalation primitive at the key-creation boundary; OWNER remains
the only role that can create, list, update, or delete org API keys.
Adds RateLimitService check (after auth + admin-api entitlement gates)
to the three org-scoped admin handlers so that compromised
organization-scoped API keys can no longer issue unbounded writes or
probe global user existence at full request rate.
Resolves INT-1270.
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
fix(scim): normalize userName casing in user POST flow (INT-1320)
The SCIM POST flow checked for an existing user with the case-preserved
userName but performed the upsert with a lowercased email. With case-sensitive
uniqueness on User.email, a case-variant userName slipped past the duplicate
check and either linked to an unrelated existing user or hit a unique
constraint instead of returning 409.
Lowercase userName once and reuse it for both the existing-user lookup and
the upsert. Adds a server test covering case-variant duplicate detection.
The signIn callback consulted only the cloud-only multi-tenant SSO
provider check, which is a no-op on self-hosted instances. This left
the password-reset OTP path as an alternate authentication channel
for users on domains that AUTH_DOMAINS_WITH_SSO_ENFORCEMENT was
meant to lock down. Block the email provider for enforced domains
the same way the credentials authorize() and signup handler do.
* feat(clickhouse): add analytics_events_core view for project-level analytics (LFE-8734)
Adds a ClickHouse VIEW on events_core with per-project, per-hour aggregations
including type/source/scope/SDK counts via sumMap, unique counts via uniqIf
and uniqArray, and has_* boolean flags for feature detection.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat(auth): add email verification on signup gated behind AUTH_EMAIL_VERIFICATION_REQUIRED (LFE-8709)
Add an optional OTP email verification step before user creation during
email/password signup. When AUTH_EMAIL_VERIFICATION_REQUIRED=true (and
SMTP is configured), the signup flow becomes: enter email+name → receive
OTP → verify code → set password. Self-hosters without SMTP or without
the flag keep the current direct signup behavior. SSO/social logins are
unaffected.
Key changes:
- New env var AUTH_EMAIL_VERIFICATION_REQUIRED
- New POST /api/auth/signup-verify endpoint (creates passwordless user)
- New /auth/setup-password page for initial password setup
- Merged set/reset password into one ResetPasswordPage component
- Context-aware email template (welcome vs reset wording)
- hasPassword added to session for mode detection
- Direct /api/auth/signup blocked when verification is required
- Parameterized email verification cutoff (default 10 minutes)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* fix(dashboards): Use correct units for charts
* Fix view version logic
* Support value formatter in `BigNumber` and `HistogramChart`
* Fix value formatter usage
* Resolve PR comments
* Improve formatting single digit millisecond values
* Introduce `formatMetric`
* Fix chart label in `LatencyChart`
* Remove unit form latency label in `score-analytics-utils.ts`
* fix rounding to m for 1000k
* more compact tests
* compact
---------
Co-authored-by: Nimar <l.nimar.b@gmail.com>
* fix(shared): reject DNS-failing hostnames in outbound URL validation
The LLM base URL validator silently bypassed the IP blocklist whenever
DNS resolution failed (NXDOMAIN/SERVFAIL/timeouts/split-horizon), which
enabled DNS-rebinding SSRF against cloud metadata and internal services.
Treat DNS failure as a hard error and drop the per-caller opt-out flag;
self-hosters must use LANGFUSE_LLM_CONNECTION_WHITELISTED_HOST for
gateways the validator cannot resolve. Closes INT-1226.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test(web): use resolvable placeholders in llm-api-key servertest
The new strict DNS validation rejects custom.openai.com / new-custom.openai.com
/ new-endpoint.example.com because they NXDOMAIN. Swap to IANA-reserved
example.com / example.org / example.net which always resolve to public IPs
and pass the IP blocklist.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(sso): add DNS-based verified domains
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(sso): add ssoConfig tRPC router
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(sso): require https:// on user-supplied OIDC issuer urls
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(sso): show zone-relative host and query fresh DNS for domain verification
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(sso): add per-domain self-service SSO config UI
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(sso): pre-flight OIDC discovery on ssoConfig.save and the legacy support endpoint
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(sso): enforce verified-domain invariant on SsoConfig lifecycle
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(sso): include name field on custom OIDC provider form and payload
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(sso): translate P2002 race on verifiedDomain.create to CONFLICT
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(sso): allow pending verified-domain claims to coexist across orgs
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(sso): require non-empty Azure AD tenantId on save
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(sso): restore URL grammar validation on OIDC issuer and GH Enterprise baseUrl
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(sso): refuse redirects on OIDC discovery fetch (SSRF defense)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(sso): surface schema errors for github-enterprise baseUrl in the form
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(sso): gate verifiedDomain mutations on the SSO entitlement
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(sso): preserve advanced authConfig fields when re-saving the same provider
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(sso): skip orphan-prevention check on pending verified-domain deletes
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(sso): accept Azure AD multi-tenant {tenantid} placeholder in OIDC discovery
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(sso): require non-empty name on custom OIDC provider
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(sso): clarify verified-domain delete dialog copy when SSO config exists
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(sso): satisfy codespell and clean up validation messages
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(scim): write audit log on user creation via SCIM POST
POST /api/public/scim/Users now emits an auditLog entry for the
created organizationMembership, matching the tRPC members.create
behavior. Without this, an ADMIN using SCIM to add users with
arbitrary roles left no audit trail (INT-1250).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(projects): persist parsed metadata on project create/update
handleCreateProject and handleUpdateProject parsed string metadata
via JSON.parse for validation but discarded the parsed value and
wrote the raw string into Prisma. As a result, metadata sent as a
JSON string was stored as a string-typed JSON value instead of the
intended object (INT-1338).
Capture the parsed value and persist it.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(projects): reject non-object project metadata
After parsing the metadata JSON string the value can still be null,
a primitive, or an array. Persisting those in the Json? column
violated the API contract, and JS null in particular wrote SQL NULL
to Prisma — silently wiping any existing metadata on update.
Add a shape check after JSON.parse on both create and update paths
to reject non-object metadata with 400. Adds regression tests for
"null", arrays, numbers, strings, and a direct JS null.
Addresses review feedback on PR #13497.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test(projects): explicit coverage for omitted metadata
Document the contract that omitting metadata is valid:
- create without metadata returns {} and stores NULL
- update without metadata preserves the existing object
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three legacy Next.js handlers authenticate directly via ApiAuthService
without going through createAuthedProjectAPIRoute and were not invoking
RateLimitService:
- projects/[projectId]/apiKeys (GET/POST) — backdoor-credential minting
via caller-chosen publicKey/secretKey was unbounded with a leaked
org-scoped key (INT-1271)
- projects/[projectId]/apiKeys/[apiKeyId] (DELETE) — unbounded enumeration
/ churn of project API keys (INT-1265)
- prompts POST — unlimited prompt-version writes; GET already used the
"prompts" rate-limit bucket but POST silently bypassed it (INT-1260)
Add RateLimitService.rateLimitRequest with isRateLimited() short-circuit
after auth/entitlement checks. Uses "public-api" for the apiKeys admin
endpoints and "prompts" for the prompt POST, matching the bucket the GET
branch already consumes.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
POST /api/public/scim/Users now emits an auditLog entry for the
created organizationMembership, matching the tRPC members.create
behavior. Without this, an ADMIN using SCIM to add users with
arbitrary roles left no audit trail (INT-1250).
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(worker): add secondary otel ingestion queue
Mirrors the existing secondary ingestion queue pattern for the OTel pipeline
so high-throughput projects can be redirected to a dedicated processing pool
via LANGFUSE_SECONDARY_OTEL_INGESTION_QUEUE_ENABLED_PROJECT_IDS.
Refs LFE-6579.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore: remove unused values
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The node:24-alpine base image pre-populates /root/.cache/node/corepack/
when corepack is enabled during the build. Removing the module directory
alone left this cache intact in the final image, giving Snyk something
to scan. Add it to the rm -rf in both web and worker runtime-base stages.
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
The "Number of Observations (7d)" column on the Prompts table was passing
toTimestamp = end of yesterday, which excluded every observation with a
start_time during the current day. New prompt calls therefore never
appeared to increment the counter, and prompts only used today showed 0.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: Nimar <l.nimar.b@gmail.com>
* feat(widgets): use latency formatter for millisecond measure units
Adds getMeasureUnit helper to dataModel and wires latencyFormatter into
both DashboardWidget and WidgetForm preview when the selected measure
unit is "millisecond".
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* feat(widgets): add day unit to latency formatter
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(widgets): apply latency formatter to pivot table values
Threads the existing valueFormatter prop from Chart through to PivotTable
so latency metrics render in auto-scaled units (ms/s/min/hr/day) instead
of raw milliseconds, matching the other chart widget types.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(widgets): memoize valueFormatter with unit switch
Replaces the inline measureUnit ternary with a memoized switch keyed on
measureUnit, making it trivial to add more unit-to-formatter mappings
(usd, tokens, etc.) as they arrive.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(widgets): apply latency formatter to histogram bin labels
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(widgets): apply value formatter to vertical bar y-axis ticks
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(widgets): apply value formatter to pie center label and pad tooltip rows
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(widgets): trim latency formatter
Drop unused latencyFormatterParts export and stop padding latency labels
with trailing zeros (1.00s -> 1s).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(widgets): pick formatter from agg-aware result unit and add USD
Add getResultUnit alongside getMeasureUnit so count/uniq aggregations
resolve to "integer" instead of inheriting the source measure's unit
(e.g. count(latency) no longer renders as milliseconds). Both
DashboardWidget and WidgetForm now derive the formatter from
getResultUnit and switch on the result, with a new USD branch routing
to usdFormatter alongside the existing millisecond -> latencyFormatter.
WidgetForm's inline ternary becomes a useMemo to mirror DashboardWidget.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(widgets): per-column pivot formatting via units overlay
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(widgets): drive chart formatting from chartConfig.unit
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* improvement(widgets): cleaned up some dead code
* improvement(formatting): reduced allocations of time duration formatters
* perf(widgets): memoize chart valueFormatter
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(widgets): route sub-millesimal values through compactSmallNumberFormatter
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(widgets): merge sanitized defaultSort back into pivot chartConfig
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(widgets): scale negative latencies in latencyFormatter
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(widgets): preserve precision for sub-unit magnitudes
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
uuid v7+ ships bundled TypeScript declarations ("types": "./dist/index.d.ts"),
so @types/uuid is redundant after the v9→v14 upgrade and risks the stale
v9-era DefinitelyTyped types shadowing the bundled ones.
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
chore(deps): upgrade uuid from v9 to v14 to resolve Snyk alert
Snyk flagged uuid@9 for improper index validation. The package is only
used for v4() random ID generation with no untrusted input, so not
exploitable, but upgrading clears the alert cleanly.
uuid v14 requires Node 20+ (we run Node 24) and keeps the same
import API (import { v4 } from "uuid").
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
* ci: disable web test sharding
* test: isolate comment fixtures
* test: make score update fixture deterministic
* test: report slowest vitest tests
* ci: install only chromium for e2e tests
* test: speed up slow server tests
* test: show slowest vitest tests only in ci
* ci: re-enable web test sharding
* test: reduce ingestion and rate limit test latency
* test: report top 50 slowest vitest tests
* test: parallelize prompt name validation cases
* ci: limit vitest server workers
* test: stabilize score comparison sampling test
* test: show slowest worker tests only in ci
* ci: disable web test sharding
* test: reduce slow trace and annotation fixtures
* test: report slowest vitest files
* test: reduce slow trace and score fixtures
* test: reuse score and prompt list fixtures
* test: streamline slow server fixtures
* test: use valid dataset list limit
* test: stabilize and speed up worker tests
* test: isolate dataset item backfill assertions
* test: speed up API key fixture creation
* test: keep legacy api auth coverage explicit
* ci: skip duplicate next typecheck in test builds
* ci: build dependencies before typecheck
* ci: install playwright headless shell
* test: retry flaky vitest tests in ci
* ci: run prisma generate without turbo cache
* test: isolate dataset schema fixtures for retries
* refactor(web): Simplify `TablePeekView` props
* fix(traces): Show trace id in trace peek view title
* fix(web): Align observation peek title with trace detail view title
* fix(web): Align trace peek title with trace detail view title
* fix(web): Apply same fix as 2f235d831 to trace peek detail view
---------
Co-authored-by: Nimar <l.nimar.b@gmail.com>
* fix(clickhouse): set alter_sync/mutations_sync on multi-ALTER clustered migrations
Back-to-back ALTERs on the same table in a single migration file race the
replicated metadata-version update on ReplicatedMergeTree / SharedMergeTree
(ClickHouse Cloud), where alter_sync defaults to 0 and the second statement
can land on a replica whose metadata version still lags Keeper, producing
CANNOT_ASSIGN_ALTER (517) during initial bootstrap.
Add the per-statement settings to every clustered migration that issues
multiple ALTERs on the same table:
- alter_sync = 2 on metadata ALTERs (ADD/DROP/MODIFY COLUMN, ADD/DROP INDEX)
- mutations_sync = 2 on mutation-creating ALTERs (MATERIALIZE INDEX) so the
index is fully built on all replicas before the migration returns
Covers 0005, 0006, 0008, 0025, 0026, 0031. The unclustered/ mirror runs on
plain MergeTree where these settings are no-ops, so it is left untouched.
Document the rule and the metadata-vs-mutation distinction in the
clickhouse-best-practices skill so future migrations follow the convention.
Validated end-to-end by bootstrapping migrations 1-34 against a fresh
ClickHouse Cloud instance with no 517 errors.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(clickhouse): use alter_sync on ADD INDEX in 0013/0015/0016/0018
These four pre-existing migrations applied SETTINGS mutations_sync = 2 to
both ALTERs, but mutations_sync is a no-op on metadata ALTERs like
ADD INDEX, so the metadata-version race that this PR is fixing in
0005/0006/0026 still applied here. Switch the ADD INDEX line in each to
SETTINGS alter_sync = 2 (the MATERIALIZE INDEX line stays on
mutations_sync = 2). Caught by review on PR #13398.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* cleanup
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(events): use release field for trace release in events table adapter
eventsToTraceAdapter was using earliest.version for both version and
release fields when synthesizing trace data from the events table
(SDK v5+ direct-write path). This caused langfuse.release span
attribute values to be overwritten with the langfuse.version value.
Also adds 'release' to base/baseWithoutTools/byIdBase field sets in
EventsQueryBuilder so it is actually selected from ClickHouse, and
adds release to EventsObservationSchema so the TypeScript type includes it.
Fixes#13273
* test(events): verify release field is returned from events table queries
Adds two tests to event-repository.servertest.ts:
- getObservationsWithModelDataFromEventsTable returns release separately from version
- getObservationByIdFromEventsTable returns release separately from version
These confirm the fix in event-query-builder.ts (adding 'release' to
base/byIdBase field sets) and EventsObservationSchema (adding the release
field) are wired up end-to-end.
* fix(events): include release field in EventsObservationRecordReadType and converter
Add `release` to `eventsObservationRecordReadSchema` so the field is
typed and preserved through ClickHouse deserialization, and map it in
`convertEventsObservation` so it reaches the domain object.
Previously the field was selected by the query builder but silently
dropped because neither the Zod schema nor the converter passed it
through.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(events): add release field to APIObservation schema
The release field is now returned in GET /api/public/observations
responses (via EventsObservation ...rest spread), but APIObservation
was .strict() and did not declare release, causing makeZodVerifiedAPICall
to fail with "Unrecognized key: release" in all useEventsTable=true tests.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Steffen Schmitz <steffen@langfuse.com>
Adds a shared agent skill for triaging Linear/GitHub issues and incident
reports against Datadog APM, logs, and metrics, with a repo-debug map and
output template that produces a structured root-cause analysis.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore(env): add LANGFUSE_ENABLE_EVENTS_TABLE_UI flag for UI events table support
* refactor: rm env variable from wrong usage
* tests: remove env flag
The ClickHouse index analyzer cannot extract has(names, k) semantics
from
values[indexOf(names, k)] OP v (cross-array arrayElement form), so a
bloom_filter skipping index on metadata_names cannot prune granules for
this shape. Wrap every operator branch in StringObjectFilter.apply() for
events_core / events_full / events_proto with an explicit
has(names, k) AND (...) conjunct.
Also corrects a latent semantic bug: today arr[indexOf(names, missing)]
resolves to the empty string (Array(String) default), making
"does not contain V" match rows where the key was never written, and
"OP empty-value" match similarly. The has(...) conjunct fixes this for
all five operators uniformly.
Add bloom_filter on events_core.metadata_names. events_core is the
table that takes filters in the V2 split-query pattern; events_full
reads by ID after the base CTE narrows things down, so the symmetric
index is intentionally omitted for now.
Pre-filters the trace JOIN in a CTE so the trace timestamp window prunes
partitions directly instead of living alongside the LEFT JOIN where the
planner cannot push it down. Applied to the score and generation analytics
queries that drive the PostHog and Mixpanel exports.
Switches `grace_hash` from unconditional to retry-gated: first attempt uses
ClickHouse's `auto` algorithm, retries fall back to `grace_hash` so an OOM
recovers without manual intervention while healthy syncs stay fast.
Note: the generation analytics query previously used `LEFT JOIN ... WHERE
t.project_id = {projectId}` which silently dropped generations whose trace
was missing or outside the 7-day window. With the CTE-based LEFT JOIN those
generations now ship with NULL trace fields instead.
Refs: LFE-9475
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The observations_agg CTE in getTracesForAnalyticsIntegrations only had a
lower bound on o.start_time, so ClickHouse scanned observations from
minTimestamp - 1h to the end of the table on every run. For long-lived
projects or repeated retries this is an unbounded scan.
Cap the CTE at maxTimestamp + OBSERVATIONS_TO_TRACE_INTERVAL (2 days),
matching the existing observation-to-trace join convention elsewhere in
the file. Traces outside [minTimestamp, maxTimestamp) are already
filtered in the outer SELECT, and in steady state the 30-min now-buffer
ensures a trace's observations have settled well before the window
boundary advances, so the cap does not truncate data that would
otherwise be emitted.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(worker): cap PostHog export window at next UTC day boundary (LFE-9475)
When a PostHog integration failed repeatedly, lastSyncAt never advanced
and each hourly retry re-scanned an ever-growing window against
ClickHouse. Cap maxTimestamp at the next UTC day boundary after
lastSyncAt so per-run work is bounded and aligned with the toDate(...)
partition/ordering keys. Healthy integrations are unaffected because
now - 30min wins whenever the sync is within a day of present. Initial
backfills (no lastSyncAt) skip the cap to avoid pathological
day-by-day stepping from 2000-01-01.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(worker): use project createdAt as PostHog sync floor on first run
Replace the 2000-01-01 fallback for minTimestamp with the project's
createdAt. No trace data can precede it, so this is a tight lower bound
that lets the day-cap also apply to initial backfills. Old projects
still catch up incrementally (one UTC day per hourly run) instead of
re-scanning all of history in one shot.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore: update comments
* fix(shared): use half-open upper bound for analytics integration queries
Switch the primary event-timestamp filter from `<=` to `<` in the four
getXForAnalyticsIntegrations queries (traces, generations, scores,
events). Since the PostHog and Mixpanel schedulers advance
`lastSyncAt` to the previous run's `maxTimestamp`, a row whose
timestamp falls exactly on a window boundary was previously emitted
once per run on both sides. Half-open semantics ensure each row is
emitted exactly once across consecutive runs. Secondary trace-join
upper bounds (7-day lookback for metadata) stay `<=` because they are
range optimizations, not emission bounds.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(scores-api): allow source=ANNOTATION on POST /api/public/scores
Expose the `source` field on the create-score request so callers can
post scores as `ANNOTATION` (or `EVAL`). This unblocks LLM-prefilled
scores that show up in an annotation queue for a human reviewer.
When `source` is `ANNOTATION`, `configId` is required unless
`dataType` is `CORRECTION` (matches the existing tRPC annotation
contract).
Ref: LFE-9330
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(scores-api): narrow source to API/ANNOTATION and enforce rule on ingestion path
Addresses review feedback:
- Drop EVAL from the public create-score surface. EVAL remains a valid
stored/query value but is reserved for internal evaluator outputs; external
callers should use API (default) or ANNOTATION. Narrowed via a new
`CreateScoreSource` enum in Fern and inline zod enum at `PostScoresBody`.
- Move the "ANNOTATION requires configId unless CORRECTION" constraint into
`validateAndInflateScore` so it applies to both the REST `POST /scores` path
and the ingestion/SDK path (`POST /ingestion` → score-create event), closing
the gap flagged on the PR. The zod refine on `PostScoresBody` is kept so REST
callers still get a synchronous 400 instead of an async drop.
- Fix the stale path in the `PostScoresBody` comment that pointed to a
non-existent file.
- Add a servertest covering the ingestion path: HTTP returns 207, worker drops
the ANNOTATION-without-configId event, sentinel score confirms the batch was
processed.
Ref: LFE-9330
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(scores-api): replace magic strings with ScoreSourceEnum and a named subset schema
Follow-up to review feedback on stringly-typed source checks.
- Add PublicApiCreateScoreSourceArray / Domain / Type in domain/scores.ts
with `satisfies readonly ScoreSourceType[]` so the public-API subset stays
provably ⊂ ScoreSourceArray at compile time. Dropping a value from
ScoreSourceArray would now break this declaration first.
- Use PublicApiCreateScoreSourceDomain on PostScoresBody instead of the
inline z.enum(["API", "ANNOTATION"]).
- Reference ScoreSourceEnum.ANNOTATION / ScoreSourceEnum.API and
ScoreDataTypeEnum.CORRECTION instead of raw string literals in the
PostScoresBody refine and in validateAndInflateScore.
No behavior change; all source-field tests continue to pass.
Ref: LFE-9330
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs(scores-api): clarify which entry points trigger the annotation/configId check
The "REST create-score path vs ingestion/SDK path" phrasing was misleading:
both are REST, just different HTTP endpoints (/scores vs /ingestion). Spell
that out and note that both funnel through this function in the worker.
Ref: LFE-9330
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(scores-api): consolidate annotation/configId rule into a shared predicate
- Drop unused PublicApiCreateScoreSourceArray/Type exports; inline the
subset tuple into the Domain declaration.
- Add `isAnnotationScoreMissingConfigId` + ANNOTATION_SCORE_REQUIRES_CONFIG_ID_MESSAGE
in domain/scores.ts. Both the zod refine on PostScoresBody and the throw in
validateAndInflateScore now call the same predicate with the same message,
removing the copy-pasted rule and its comments.
- Trim redundant prose comments that duplicated the predicate's intent.
Net -18 lines. No behavior change; all 6 source-field tests still pass.
Ref: LFE-9330
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test(scores-api): unit-test validateAndInflateScore and expose source on client Fern
Addresses PR review feedback:
- Add unit tests for validateAndInflateScore covering the configId tenancy
check (the reviewer's specific ask) plus the ANNOTATION/configId rule.
Direct function calls — no HTTP, no queue, no sentinel waiting; each test
runs in <400ms.
- Drop the flaky ingestion-endpoint sentinel test that asserted the same
ANNOTATION drop behavior; coverage is now more precise at the function
level where the rule actually lives.
- Add `source` + `CreateScoreSource` to the client Fern definition so the
public client spec matches the server Fern surface. Regenerated the
corresponding OpenAPI.
Ref: LFE-9330
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Wrap each day of the usage aggregation loop in its own span with
langfuse.free_tier.dayStart/dayEnd/dayDate/daysAgo attributes so daily
work is visible individually in traces. Extend the ClickHouse
request_timeout to 120s on the three per-day count queries, and retry
each query up to two additional times on failure — a single re-run is
cheap compared to a full job retry.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(web): prevent crash on invalid JSONPath in dataset mapping editor
CustomMappingEditor crashed the dialog when a user entered a malformed
JSONPath (e.g. `$..[?(@.x=)]`), because MappingPreviewPanel called
applyFieldMappingConfig in a useMemo and jsonpath-plus threw during
render. applyFieldMappingConfig now wraps evaluateJsonPath in a safe
helper, reports failures via a new onJsonPathError callback, and returns
undefined for that entry/field instead of throwing. applyFullMapping
propagates the callback so json_path_error entries carry the actual
source field and mapping key.
On the frontend, MappingPreviewPanel surfaces the error as a destructive
banner and invalidates validation when a schema is present. Notice boxes
are deduplicated into a small IssueList/IssueItem helper.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(web): surface invalid JSON path in evaluator prompt preview
renderPromptPreviewFromObservation used to discard the error returned by
extractValueFromObject, so an invalid jsonSelector silently rendered the
raw column value. Surface it inline as `<invalid JSON path "X": …>` so
the user sees which mapping is broken instead of getting a misleading
preview (or a generic "Unexpected Error" toast via useExtractVariables).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(web): show real error message when evaluator variable extraction fails
useExtractVariables was forwarding plain Errors (e.g. invalid JSON path
thrown by jsonpath-plus) to trpcErrorToast, whose non-TRPC fallback
renders a generic "Unexpected Error" toast and drops the actual message.
Use showErrorToast directly so the user sees "Invalid JSON path: …"
instead of a useless generic error.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(web): label evaluator extraction error as JSON path issue
"Failed to extract variable" read like a semantic failure. The only
throw path in extractValueFromObject is the jsonpath-plus call, so any
error surfaced here is a JSON path syntax problem — reflect that in the
toast title so users know where to look.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(web): include JSON path in evaluator extraction error toast
Match the phrasing of the dataset mapping banner and preview inline
message so all three surfaces show both the offending path and the
underlying parser message.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore(web,shared): standardize on "JSONPath" in user-facing strings
"JSON path" / "JSON Path" / "json path" were used interchangeably in
dataset mapping, evaluator preview, and their shared helpers. Use the
canonical "JSONPath" everywhere these strings surface to the user so the
three error surfaces read consistently.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Revert "fix(web): surface invalid JSON path in evaluator prompt preview"
This reverts commit f943fd87f. Scope creep relative to LFE-9336 — the
inline <invalid JSONPath …> marker inside a rendered prompt is noisy,
and the batch-actions preview is a separate concern that deserves its
own pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(web): distinguish JSONPath syntax errors from misses in final preview
FinalPreviewStep collapsed both json_path_error and json_path_miss into
a single amber "did not match" banner, so a syntax error reaching this
step (possible when the field has no schema or for metadata mappings)
was rendered as a soft warning with misleading wording. Split the
errors by type and render syntax errors with destructive styling and
"invalid syntax" wording, while keeping misses as amber warnings. When
a card has both, prefer the destructive treatment and combine the two
counts into one line. Extract the banner chrome into a small IssueBanner
helper to share variant styling between the top banner and per-card
footer.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(shared): return errors/misses from applyFieldMappingConfig
Replace the onJsonPathMiss/onJsonPathError callback parameters on
applyFieldMappingConfig with a direct FieldMappingResult return shape
of { value, misses, errors }. Callers no longer mutate external arrays
via closures — applyFullMapping consumes result.misses / result.errors
directly and MappingPreviewPanel destructures them out of the return.
Also tightens per-field fault isolation semantics in applyFullMapping
and cleans up a couple of low-value comments.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(web): extract shared IssueBanner from mapping preview panels
MappingPreviewPanel and FinalPreviewStep were each carrying their own
variant lookups, notice-box markup, and icon pickers for the error /
warning JSONPath chrome. Consolidate into a single
AddObservationsToDatasetDialog/components/IssueBanner module that exports:
- IssueBanner, IssueList, IssueItem components
- issueChromeVariants (border + bg + text for banner / list / card footer)
- issueCardVariants (outer card border with an explicit "none" variant)
- issueTextVariants (text color for children that override CSS inheritance)
- issueIcons (variant -> lucide icon)
Both callers now compose cva output with their layout classes via cn().
No visual change; colors and spacing are identical.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(web): address PR review nits on JSONPath extraction + preview copy
useExtractVariables was labelling every error that reached the toast
effect as "Invalid JSONPath in variable mapping", including unrelated
runtime errors that hit the outer Promise.all().catch() branch. Replace
the plain Error state with a discriminated ExtractionError so only
errors surfaced by extractValueFromObject use the JSONPath title;
unexpected failures fall back to a generic "Failed to extract variable".
FinalPreviewStep's per-card footer rendered "1 path have invalid
syntax" for a single error; move the verb into the ternary so the
singular reads "1 path has invalid syntax".
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(experiments): add enabled toggle for remote dataset run trigger (#13221)
* feat(experiments): add enabled toggle for remote dataset run trigger
Allows users to temporarily disable the remote trigger without losing
the configured URL and default payload.
- Add `remoteExperimentEnabled` boolean column (default true) to the
`datasets` table
- Add an "Enable trigger" switch to the upsert form
- Hide the "Run" button in the experiment dialog when disabled
- Server-side early return in `triggerRemoteExperiment` when disabled
(returns `{ success: true, skipped: true }`)
Motivation: once a URL is configured, every Run click fetches the URL
and shows a "Failed to trigger remote experiment" error toast if the
endpoint is unreachable. The only way to silence it today is to delete
the URL, which is lossy. This adds a simple toggle to pause the trigger
while keeping the config.
Backward compatible: default is `true` at both column and form levels,
so existing datasets behave identically.
* fix(public-api): include remoteExperimentEnabled in Dataset v2 response schema
Without this, POST/GET/PUT /api/public/v2/datasets responses fail strict
Zod validation with "Unrecognized key: remoteExperimentEnabled" since the
tRPC/API layer now returns this field for the new toggle.
* fix(experiments): address review feedback on enabled toggle
- deleteRemoteExperiment: reset remoteExperimentEnabled back to true so
a later upsert without the optional enabled flag does not silently
inherit the previously disabled state
- RemoteExperimentTriggerModal.onSuccess: distinguish the
{ success: true, skipped: true } response and show a "Remote trigger
is disabled" toast instead of the misleading success toast
- allDatasets: add remoteExperimentEnabled to the Omit exclusion list
so the $queryRaw return type matches the actual SQL SELECT (the
field is not fetched, so the type annotation would otherwise lie)
* fix(public-api): select remoteExperimentEnabled in list dataset handlers
GET /api/public/datasets (v1) and GET /api/public/v2/datasets (v2) both
use an explicit Prisma select that narrows the return type. The APIDataset
response schema now requires remoteExperimentEnabled, so the select needs
to include it or the build fails with "Property 'remoteExperimentEnabled'
is missing".
The single-dataset GETs (v1 /datasets/[name] and v2 /datasets/[datasetName])
use the default all-fields findFirst, so they're already fine.
---------
Co-authored-by: Nimar <l.nimar.b@gmail.com>
* chore: fix migration order
* refactor: remove remoteExperimentEnabled field from API dataset types and endpoints
* refactor: wording
* test: ensure remote experiment fields are not exposed in public API responses
* chore: wording nits
* fix: invalidate remote experiment cache on dataset removal in RemoteExperimentUpsertForm
---------
Co-authored-by: Yuto Toya <97585904+toyayuto@users.noreply.github.com>
Co-authored-by: Nimar <l.nimar.b@gmail.com>
* chore(observability): upgrade opentelemetry and datadog SDKs
* fix(ci): disable tracing in tests-web startup
* fix(ci): remove local codex files from pr
* test(worker): retry bedrock llm connection smoke tests
* chore; revert pipeline changes
* revert(test): remove bedrock retry logic from llm connection tests
Reverts the retry/backoff additions from cdaacaf45 — these should be
handled in a separate PR since they are unrelated to the OTel/DD upgrade.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* chore; disable DD tracing for web tests
* chore: downgrade dd-trace to 5.82.0
* chore: downgrade prisma instrumentation
* chore: bump dd-trace-js
* chore: release v3.170.0-0
* revert release push
---------
Co-authored-by: steffen911 <steffen@langfuse.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Steffen Schmitz <steffenschmitz@hotmail.de>
* feat: add 5-minute and 20-minute blob storage export frequency options
Add sub-hourly export frequency options (every 5 minutes, every 20 minutes)
to the blob storage integration, in addition to the existing hourly/daily/weekly.
The scheduler cron is updated from hourly to every 5 minutes to support the
new intervals.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: remove 5-minute export option, reduce lag buffer from 30 to 20 minutes
Per reviewer feedback, 5-minute export frequency is too granular given
internal data paths that may take longer in the worst case. Keeps only
the 20-minute option as the new minimum frequency, adjusts the lag
buffer to 20 minutes, and updates the queue schedule accordingly.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: address PR review feedback for blob storage export frequency changes
- Add removeRepeatable for old hourly cron pattern to prevent duplicate schedules
- Extract lag buffer duration into BLOB_STORAGE_LAG_BUFFER_MS constant
- Add every_20_minutes to Fern BlobStorageExportFrequency enum
- Update lag buffer docs from 30 to 20 minutes
- Fix test comment referencing old 30-min lag buffer
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix: update test timing assertion from 30-min to 20-min lag buffer
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix: export BLOB_STORAGE_LAG_BUFFER_MS and use it in tests
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix: prettier formatting for exportFrequency enum in types.ts
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* feat(ui): decode unicode escapes in PrettyJsonView for trace detail
Apply decodeUnicodeEscapesOnly() recursively to parsed JSON in the trace
detail PrettyJsonView so that \uXXXX sequences (produced by Python SDK's
json.dumps with ensure_ascii=True) are decoded to their original
characters when viewing traces in the UI.
This is a follow-up to PR #12882 which applied the same decoder to the
batch export pipeline, and to the earlier IOTableCell change (PR #9686).
Together these ensure non-ASCII content (e.g. Japanese, Chinese, Korean)
renders correctly everywhere a user encounters it in Langfuse.
Uses greedy mode to cover both \uXXXX (single-escaped) and \\uXXXX
(double-escaped) ingest paths. Only \uXXXX escapes are decoded; other
escapes (\n, \t, \", etc.) are left untouched.
Refs #10972
* test(ui): add unit tests for decodeUnicodeInJson in PrettyJsonView
Cover primitive passthrough, string decoding (including double-escaped
greedy mode), recursive decoding of arrays / nested objects, surrogate
pairs, and mixed already-decoded input. Same style as unicode.clienttest
from PR #12882.
* refactor(ui): make decodeUnicodeInJson iterative and bounded
Address review feedback from @nimarb on PR #13223: very large or deeply
nested JSON payloads could blow the call stack or freeze the browser tab.
- Replace recursion with an explicit-stack iterative walk, so stack depth
is no longer bounded by JS engine limits.
- Add two guards:
- DECODE_UNICODE_MAX_NODES (50,000): stop decoding once the total number
of visited entries exceeds the budget; remaining values are kept as-is.
- DECODE_UNICODE_MAX_DEPTH (200): do not descend into subtrees past this
depth; the subtree is returned undecoded.
- Export the caps so tests can assert behavior at the boundary.
Tests: two new cases in PrettyJsonView.clienttest.ts -- one for chains
~10x deeper than MAX_DEPTH (must not throw), one for arrays larger than
MAX_NODES (first entries decoded, tail preserved verbatim).
* fix(ui): clone parsedJson for JSONView and decode escaped object keys
Address follow-up review feedback on PR #13223.
1. JSONView was receiving `parsedJson` directly, but JSONView internally
calls `deepParseJson` which mutates nested string fields in place. Since
baseTableData[].rawChildData holds references back into `parsedJson`,
sharing the same reference corrupted the table's lazy-loaded children
(parsed sub-objects replaced the original maxDepth:2 strings). Pass a
`structuredClone` of `parsedJson` to JSONView via a `useMemo` so the two
views stay independent without cloning on every render.
2. `decodeUnicodeInJson` was decoding values but leaving object keys as-is.
Payloads like `{"\\u4f60\\u597d": "value"}` ended up with escaped keys
alongside decoded values. Apply `decodeUnicodeEscapesOnly` to keys too.
Two new tests cover key decoding (flat + nested); existing tests cover the
value path.
---------
Co-authored-by: Nimar <l.nimar.b@gmail.com>
Track event size distributions across projects via a MergeTree table
fed by two materialized views (one from traces, one from observations).
Every insert including updates gets its own row for full ingestion
visibility.
* fix(evals): remove broken score value filter from evaluator runs page
The score value filter on the evaluator runs page never worked because
it LEFT JOINed the PostgreSQL scores table, but scores live in
ClickHouse. Remove the filter from the UI and strip it in the backend
for backward compatibility with bookmarked URLs.
Closes LFE-9279
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(evals): remove session ID filter and column from evaluator runs page
The session ID filter/column depended on the Postgres `traces` table via
LEFT JOIN, but traces now live in ClickHouse so the join never resolved.
Drop the UI filter facet, table column, and the unused traces JOIN.
Also generalize the bookmarked-URL stripping into a DEPRECATED_FILTER_COLUMNS
constant (currently scoreValue + sessionId).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(prompts): Add time window filtering to prompt metrics
* Normalize time window for prompt metrics to start of day / end of day
* Add explanatory tooltip to "last used" and "first used" columns
* refactor(web): migrate test framework from Jest to Vitest
Replace Jest with Vitest in the web package for faster test execution
and native ESM/TypeScript support without requiring next/jest SWC transforms.
- Replace jest/jest-environment-jsdom/@types/jest with vitest/@vitejs/plugin-react/vite-tsconfig-paths
- Add vitest.config.mts with 3 projects (client/server/e2e-server) matching the original Jest config
- Convert all jest.* API calls to vi.* equivalents across ~28 test files
- Convert jest.requireActual() to async vi.importActual() with async factory functions
- Remove @jest-environment docblock pragmas (environment set in vitest config)
- Add resolveWorkspaceDeps vite plugin for pnpm strict mode compatibility
- Set testTimeout to 30s to match previous Jest/next-jest default
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(web): fix type error in jumptoplayground test mock
Cast vi.importActual("zod") to any to satisfy both the TypeScript
compiler and the consistent-type-imports lint rule.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: remove jest from shared tsconfig types
The nextjs.json shared tsconfig included "jest" in the types array,
which required @types/jest to be installed. Since we migrated to vitest,
vitest globals are provided via web/vitest-env.d.ts instead.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: update CI pipeline to use vitest instead of jest
Replace jest CLI calls with vitest equivalents in the GitHub Actions
pipeline. Also regenerate pnpm-lock.yaml to remove stale jest entries.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(web): include client project in test:watch
The old Jest test:watch ran all projects. The Vitest migration narrowed
it to only --project server, silently excluding *.clienttest.{ts,tsx}
files from watch mode.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* refactor(web): simplify vitest shared aliases and add web to root workspace
- Auto-generate @langfuse/shared resolve aliases from its package.json
exports instead of hardcoding each subpath
- Add web to root vitest.workspace.ts alongside worker
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(web): address PR review feedback
- Update AGENTS.md quick commands to use vitest positional patterns
instead of Jest --testPathPatterns flag
- Move admin-access-webhook.servertest.ts from src/__tests__/async/ to
src/__tests__/server/async/ so it matches the server project include
pattern (was silently orphaned under both Jest and Vitest)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(web): fix admin-access-webhook test for 5-minute dedupe window
The dedupe window was increased from 60s to 5 minutes in bbd209194 but
the test was never updated because it was orphaned (not picked up by any
test project). Now that it runs, fix the test to advance the clock past
the actual 5-minute window.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* refactor(web): simplify vitest config using deps.inline
Replace the custom resolve aliases, resolveWorkspaceDeps plugin, and
esbuild.tsconfigRaw workaround with a single deps.inline config — the
same approach used by the worker package. This tells Vitest to process
@langfuse/* packages through its transform pipeline using normal Node
resolution (following pnpm symlinks) instead of Vite's strict exports
resolver.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* chore(web): remove unnecessary vitest-env.d.ts
Test files are excluded from the build tsconfig, and vitest injects
global types at runtime when globals: true is set. The declaration
file is not needed.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(web): add vitest/globals to tsconfig types and update docs
- Add "vitest/globals" to web/tsconfig.json types so Next.js build
can resolve describe/it/expect in test files (fixes CI build error)
- Move @testing-library/jest-dom/vitest to client project setupFiles
instead of per-file imports
- Update CONTRIBUTING.md, AGENTS.md, and skill reference docs to
replace Jest references with Vitest
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(web): add @testing-library/jest-dom to tsconfig types
Without this, IDE shows type errors on jest-dom matchers like
toBeInTheDocument() in client test files. The setupFiles config
registers the matchers at runtime but doesn't provide the types.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: correct file location paths in testing-guide.md
Update three embedded file location annotations from async/ to server/
to match the actual directory structure.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(web): enable parallel server tests, clean up CONTRIBUTING.md diff, revert turborepo skill
- Remove fileParallelism: false from server/e2e-server projects to
enable parallel test execution (faster CI)
- Rewrite CONTRIBUTING.md changes preserving original CRLF line endings
to minimize diff noise
- Revert turborepo dependencies.md change (generic skill, not repo-specific)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(web): restore sequential server tests, clean up docs
- Add maxWorkers: 1 to server/e2e-server projects to match Jest's
--runInBand (shared DB requires sequential execution)
- Clean CONTRIBUTING.md diff (preserve CRLF line endings)
- Revert turborepo skill change (generic skill, not repo-specific)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: align CONTRIBUTING.md test command description with example
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(web): convert remaining jest.* calls in controller.clienttest.ts
New test code merged from main still used jest.mock/jest.fn/jest.mocked
instead of vi.* equivalents.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Nimar <l.nimar.b@gmail.com>
Tightens `# v1`/`# v2`/`# v6` floating-major comments on SHA-pinned
actions to their exact patch-level tags (`v1.8.4`, `v2.1.0`, `v2.6.1`,
`v6.1.1`). Same SHAs, no behavior change — just removes ambiguity about
which release the pin corresponds to, and keeps the comment truthful if
the upstream major ever moves to a new SHA (which is exactly what bit
us on actions/setup-node in langfuse-js).
Left alone:
- winterjung/split — only publishes a `v2` floating tag; no patch-level
tag exists to tighten to.
- orange-buffalo/dependabot-auto-rebase — pins a `v1` branch head (not
a tag). Switching off a mutable ref is a separate decision.
- .github/workflows/ci.yml.template — not an active workflow.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(api): add DELETE endpoint for LLM connections
Adds `DELETE /api/public/llm-connections/{id}` so API clients (e.g. the
Terraform provider) can fully manage the lifecycle of LLM connections.
Mirrors the tRPC delete behavior by pausing dependent evaluator configs
when the connection is removed.
Refs: langfuse/terraform-provider-langfuse#19,
langfuse/terraform-provider-langfuse#20
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore: remove admin api key auth
* chore: revert
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(web): Add region selector to user menu
* Only show region switcher in cloud region
* Create `isProductionRegion` function
* Use same regions for auth and user navigation
* feat: detect SDK version from langfuse events table
* chore: push
* feat(events): set preferred Clickhouse service for SDK metadata retrieval
* refactor: rename SDK metadata functions for clarity and update documentation
* refactor(events): update metadata selection to use selectMetadataExpanded for full values
* tests: ai sdk test case
* refactor: enhance SDK metadata extraction to include telemetrySdkName for improved identification
---------
Co-authored-by: Nimar <l.nimar.b@gmail.com>
AWS SDK v3 >= 3.729 sends a composite CRC32 header on
CompleteMultipartUpload by default, which GCS's S3-compat layer rejects
with a 412 PreconditionFailed. Set requestChecksumCalculation and
responseChecksumValidation to WHEN_REQUIRED so buffered multipart and
lib-storage Upload both work against GCS HMAC endpoints.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- `updateOrgMembership` now passes the updated record as `after` so org-level
role changes log both before and after in the audit trail.
- `updateProjectRole` passes `updatedProjectMembership` as `after`, uses
`action: "create"` when the upsert creates a new row, and standardises the
`resourceId` to `projectId--userId` across create/update/delete so one
membership can be tracked end-to-end.
Fixes LFE-9056.
fix(batch-actions): allow dialog dismissal on status step and fix Go to Dataset 404
Previously the add-observations-to-dataset dialog blocked ESC / outside-click
on the status step, and the "Go to Dataset" link 404'd for datasets whose ids
contain slashes (e.g. seed datasets named `folder/simple-dataset`). Allow
ambient dismissal except while the batch action is in flight, and navigate
via `router.push` with an encoded dataset id so Next.js doesn't decode `%2F`
back to `/` on render.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replace the scores-numeric segment's "does not contain CATEGORICAL"
exclusion with a positive allow-list of data_type IN ('NUMERIC', 'BOOLEAN').
Previously, free-text (TEXT) and correction scores leaked into the
scores-numeric view because only CATEGORICAL was excluded; their null
numeric values also distorted avg/min/max aggregations.
The positive allow-list is safer by default — any future data_type value
added to the enum is excluded unless explicitly opted in.
Refs https://linear.app/langfuse/issue/LFE-9399
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(web): Stale search highlights
* refactor(web): Do not compute search match ranges twice
* Fix PR review issue about active match not being restored
* Resolve PR comment
* Split `useSyncMessageSearchMessages` useEffects
Emit the S3 fileKey on every OTEL ingestion failure path so operators can
scan their log platform for dropped or errored batches and feed the list
into worker/src/scripts/replayIngestionEventsV2. Covers: masking
fail-closed drops, observation parse failures, per-event record
creation/eval/write failures, and the job-level ForbiddenError/catch
fallthrough. The masking drop log additionally carries orgId and the
propagated callback headers to support zero-trust audit trails.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Avoids rate limit conflicts when VertexAI and GoogleAIStudio tests
run concurrently against the same model.
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(datasets): downgrade schema example error to warn to avoid Sentry noise
json-schema-faker can crash on certain user-provided schemas (e.g. arrays
without items). The function already gracefully returns "" and the UI hides the
example section, but console.error was captured by Sentry's capture_console
integration. Downgrade to console.warn so this expected edge case no longer
triggers alerts.
Fixes LANGFUSE-4S5
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(datasets): upgrade json-schema-faker to 0.6.1 and migrate to async API
- Upgrade json-schema-faker from 0.5.9 to 0.6.1 (ESM-only, zero deps,
TypeScript rewrite) which fixes the array-without-items crash
- Migrate from deprecated synchronous default-export API to the new named
async `generateJson` export with per-call options
- Update callers (DatasetSchemaHoverCard, NewDatasetItemForm) to handle
async generation with proper cancellation cleanup
- Add Jest moduleNameMapper + transformIgnorePatterns for ESM-only package
- Refactor jest.config.mjs to pre-resolve configs and avoid duplicate calls
Fixes LANGFUSE-4S5
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* refactor(datasets): revert jest.config.mjs changes, use virtual mock in test
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* chore(datasets): remove generateSchemaExample test file
ESM-only json-schema-faker@0.6.1 can't be resolved by Jest's CJS
resolver without jest.config changes. The wrapper is trivial — drop the
test rather than adding workarounds.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* test(datasets): add generateSchemaExample tests with Jest ESM resolution
Add moduleNameMapper for json-schema-faker (ESM-only package) so Jest
can resolve it, and restore the client tests.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(datasets): add type assertion for JsonSchema compatibility
Prisma.JsonValue narrows to JsonObject | JsonArray which isn't
assignable to json-schema-faker's JsonSchema type. Cast explicitly
since we already guard for non-objects.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* refactor(datasets): remove unnecessary cancellation in schema example effects
Schema generation is near-instant — cancellation cleanup adds
complexity for no practical benefit.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(datasets): merge moduleNameMapper with Next.js defaults in jest config
Spread sharedOverrides was overwriting Next.js's built-in
moduleNameMapper (CSS, assets, server-only). Use a helper that merges
our ESM mapper with the resolved config's existing mappings instead.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(datasets): add type annotation to withEsmMapper parameter
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(datasets): add cancellation guard to NewDatasetItemForm schema effect
Rapidly switching datasets could let a stale promise overwrite the
correct placeholder. Add cancelled flag for the multi-dependency effect.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(datasets): prevent cascading re-runs and user input overwrite in schema effect
- Remove inputValue/expectedOutputValue from the effect dependency array
to prevent form.setValue triggering cascading effect re-runs
- Use form.getValues() at both dispatch and resolution time to avoid
overwriting content the user typed while generation was in-flight
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(datasets): add cancellation guard to DatasetSchemaHoverCard effect
For consistency with NewDatasetItemForm's async pattern.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* refactor(web): Make `useSidebarFilterState` state location more explicit
* Use discriminated union for state location & further cleanup
* Remove incorrect if condition
* Update peek readme
* Remove unneccessary usePeekTableState hooks
The SHA 5241b2e9 resolves to v1.6.0, not the floating v1 tag. Fixes
zizmor ref-version-mismatch alerts #500 and #501.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Fork PRs don't receive security-events: write on GITHUB_TOKEN, so the
SARIF upload path is unavailable. Previously the whole job was skipped
on fork PRs, meaning contributor changes to workflows bypassed zizmor.
Add a fork-PR step without advanced-security that fails the job on
findings so contributors see the error directly; keep the SARIF upload
step for trusted events so the code-scanning ruleset still blocks merges.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(web): warn about unencoded special characters in DATABASE_URL on migration failure
When Prisma migrations fail, the error message now detects whether the
DATABASE_URL credentials contain special characters that need percent-encoding
and prints an actionable hint with an example and documentation link.
Closes#3923
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(web): add ClickHouse password special character detection on migration failure
Extends the migration failure diagnostics to also check CLICKHOUSE_PASSWORD
for characters (&, =, #, ?, %, +, @) that would break the query-string
interpolation in the ClickHouse migration script.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* chore: feedback
* chore: patch
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Prefix every SCIM API log with [SCIM] so they can be filtered out of
aggregate logs. Add user-id-based confirmation logs after PUT/PATCH
provisioning and deprovisioning and after DELETE, mirroring the
existing POST assignment log, so operations can be traced without
emitting userName/email. Drop the email from the 409 already-exists
log in POST /Users and log the existing membership userId instead.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(security): use constant-time comparison for admin API key auth
Replace direct string comparison (!==) with crypto.timingSafeEqual for
ADMIN_API_KEY verification to prevent timing side-channel attacks
(CWE-208). This aligns with the secure pattern already used for project
API key authentication in createAuthedProjectAPIRoute.ts.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(security): handle timingSafeEqual byte-length mismatch in admin auth
Align AdminApiAuthService with the project-scoped admin-key check in
createAuthedProjectAPIRoute.ts: drop the JS-string-length pre-check (which
misreads UTF-8 byte length and can crash on multibyte tokens) and wrap
timingSafeEqual in try/catch. Remove the dead !env.ADMIN_API_KEY guard
already handled by the early-return. Replace the duplicated inline admin
key comparison in createNewSsoConfigHandler with AdminApiAuthService so
both admin paths share one timing-safe implementation, and add
cross-reference comments between the two remaining call sites.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(security): sanitize score config names to prevent CSS injection
Score config names were interpolated unsanitized into a <style
dangerouslySetInnerHTML> block in ChartStyle, allowing stored CSS
injection via crafted config names (e.g. `x}*{background:red}/*`).
- Sanitize ChartConfig keys at the render site in chart.tsx (defense in
depth for existing data)
- Add ScoreConfigNameSchema in shared domain with regex validation
(^[\w\s.()-]+$) for use at input boundaries
- Apply name validation to tRPC create/update and public API POST/PUT
score-config endpoints
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* chore: score config docs
* chore: udnerscores
* chore: adjust chart.tsx schema
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The zizmor ref-version-mismatch rule flags hash-pinned actions whose
version comment references a moving major tag (e.g. `# v2`) instead of
the exact release tag the hash belongs to. Update the three flagged
actions:
- aws-actions/amazon-ecr-login: `# v2` → `# v2.1.2`
- github/codeql-action/upload-sarif (snyk-worker): `# v4` → `# v4.35.1`
- github/codeql-action/upload-sarif (snyk-web): `# v4` → `# v4.35.1`
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(evals): return full JSONPath slice result and deduplicate eval JSONPath logic
JSONPath slice expressions (e.g. $[1:]) returned only the first matched
element due to an unconditional result[0] in parseJsonDefault. Now
multi-match results return the full array while single-match results
remain unwrapped for backward compatibility.
Also consolidates three separate JSONPath evaluation paths (UI preview,
trace eval, observation eval) into the shared extractValueFromObject,
removing duplicated logic from the worker. The snakeToCamel column ID
fallback and parseUnknownToString remain in the worker since they are
database-specific concerns.
Closes LFE-8416
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(evals): address PR review — scope parseMultiEncodedJson, fix error logging and test expectations
- Only call parseMultiEncodedJson when a jsonSelector is present to avoid
mutating formatting on the no-selector passthrough path.
- Preserve the raw original value (not the parsed one) in the error
fallback.
- Add error logging in extractObservationVariables (was already done in
parseDatabaseRowToString but missed here).
- Update three pre-existing test expectations to match the new unwrap
semantics: single-match results are unwrapped, non-matching paths
return empty string instead of "[]".
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(evals): update evalService test expectations for single-match unwrap
Four more test assertions in evalService.test.ts still expected
array-wrapped JSONPath results (e.g. '["Hello world"]'). Updated to
match the new unwrap semantics for single-match queries.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* test(evals): remove duplicate slice/single-element tests from extractObservationVariables
These cases are already covered by extractValueFromObject.test.ts.
The pre-existing tests ($.prompt, $.response, non-matching path) remain
as integration tests for the observation eval path.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* style(evals): use @langfuse/shared alias instead of relative path in test import
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* chore(ci): replace fkirc/skip-duplicate-actions with inline gh script
Remove third-party action dependency and replicate the tree-hash
deduplication logic using gh api. Compares the current commit's git
tree SHA against recent successful workflow runs to skip redundant CI.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* chore(ci): replace ravsamhq/notify-slack-action with slackapi/slack-github-action
Switch to the official Slack GitHub Action (v3.0.1) for failure
notifications. Uses Block Kit payload for richer messages with
branch/tag, actor, and a direct link to the workflow run.
Also removes the now-unnecessary SLACK_WEBHOOK_URL entry from the
zizmor secrets-outside-env allowlist since the webhook is now passed
via action input rather than env var.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Crafted OTel attribute keys like `gen_ai.prompt.__proto__.POLLUTED` could
pollute Object.prototype via the nested-object construction in
convertKeyPathToNestedObject. Guard against dangerous keys (__proto__,
constructor, prototype) and use Object.create(null) for result objects.
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(worker): sync managed evaluator vars on template updates
* chore: timestamp
* chore: filter based on project id
* Revert "chore: filter based on project id"
This reverts commit 132ac226ad68c3dec25194433e99f95261974294.
* fix: update log message for managed evaluators upsert completion
* perf(dual-write): clamp min start time to past day and optimize trace sorting
* chore: exclude project_id 'cmbktgdyf0059ad07yexqm2gp' from dual write
* chore: introducing LANGFUSE_EVENT_PROPAGATION_EXCLUDE_PROJECT_IDS
---------
Co-authored-by: Valery Meleshkin <valeriy@langfuse.com>
fix(ci): handle null/undefined security-severity in Snyk SARIF output
Snyk emits invalid security-severity values (null, "undefined", "null")
that cause codeql-action/upload-sarif to reject the file. Replace the
sed-based fix with jq to handle all non-numeric values.
See: https://github.com/github/codeql-action/issues/2187
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Slack natively timestamps every message. Our custom footer showed the
worker's server timezone which confused users in different timezones.
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(slack): show change author in Slack prompt notification
Display the user who made the change in the Slack notification message
for prompt version events. Falls back to email when name is unavailable,
and shows "API User" for API key-initiated changes.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(slack): escape mrkdwn in change author and use || for empty string fallback
Escape &, <, > in user name/email to prevent Slack mrkdwn injection
(e.g. <!channel> triggering mass notifications). Use || instead of ??
so empty string names fall back to email correctly.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(slack): escape all user-controlled mrkdwn fields in prompt notification
Apply escapeSlackMrkdwn to prompt.name, prompt.tags, and
prompt.commitMessage to prevent injection via those fields too.
Labels are safe (validated by PROMPT_LABEL_REGEX).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Add zizmor workflow and skip forked Claude review PRs
* ci: test if workflow breaks
* ci: add dependabot cooldown
* ci: zizmor auto-fixes
* ci: harden GitHub Actions workflows
* ci: refine zizmor workflow configuration
* ci: scope sdk and snyk secrets to environments
* ci: address zizmor workflow review feedback
* ci: fix license check
* ci: align sdk workflow secret handling
* ci: enable snyk checks on pull requests
* ci: remove temporary snyk pull request trigger
* ci: bump back to the zizmor minimum of 7 days
* style: move to nicer config syntax for secrets-outside-of-env
* ci: fix template-injection warnings in pipeline digest step
Move step outputs and matrix values from ${{ }} interpolation in run
blocks to env variables, preventing potential shell code injection.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(ci): fix broken heredoc expansion in Docker publish steps
The publish-manifest steps used <<'EOF' (single-quoted heredoc) which
suppresses bash variable expansion, and the env vars were never defined
in those steps. This meant every tag-triggered release would fail with
literal ${VAR} strings passed as Docker tags.
- Change <<'EOF' to <<EOF to enable variable expansion
- Add env: blocks defining STEPS_META_*_OUTPUTS_TAGS from step outputs
- Move remaining ${{ matrix.* }} interpolations to env vars
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* ci: rename reserved GITHUB_ env var prefix to INPUT_
Rename GITHUB_EVENT_INPUTS_CONFIRM to INPUT_CONFIRM. GitHub reserves
the GITHUB_ prefix for built-in runner variables.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* ci: use environment secret instead of inherit
* ci: switch to latest action
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(shared): treat end-of-life model errors as non-retryable
Extract non-retryable error patterns into a shared constant and add
"reached the end of its life" to the list so that Bedrock end-of-life
model errors surface immediately instead of being retried.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* refactor(shared): keep isNonRetryableLLMErrorMessage private
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(shared): extract status code from AWS SDK $metadata.httpStatusCode
The AWS SDK puts the HTTP status on `$metadata.httpStatusCode`, not on
`.status` or `.response.status`. Without this, the fallback defaulted to
500, making 4xx errors appear retryable.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* test: use real ResourceNotFoundException from AWS SDK
Instead of hardcoding the error shape, resolve and instantiate the real
`ResourceNotFoundException` from `@aws-sdk/client-bedrock-runtime` via
`@langchain/aws`'s dependency tree.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* OCI Object Storage Native SDK client Integration for StorageService with identity access Management options.
Updated tests accordingly.
Updated docker file with env variables and build options to build from code.
Updated package json file with OCI libraries used for Object Storage.
Added a sample .env file with instructions on how to use workload_identity' | 'instance_principal' | 'resource_principal' | 'oci_profile' | 'session_token.
Added !.env.dev-oci.example to gitignore to commit the file.
* Update StorageService.ts
Fixed Lint errors
* Added the pnpm lock file
* Add uploadFileBuffered per upstream PR requirements; rename env vars to langfuse_ prefix
Implemented uploadFileBuffered to satisfy requirements introduced by an upstream pull request.
Updated environment variable names to use the langfuse_ prefix for consistency/alignment
* Remove prisma-extension-kysely dependency
* Remove prisma-extension-kysely from pnpm-lock.yaml
Removed prisma-extension-kysely dependency and related entries.
---------
Co-authored-by: Steffen Schmitz <steffen@langfuse.com>
* feat(experiments): direct-write prompt experiment root events
* feat(tracing): centralize internal direct event writes
* push
* chore: move asRecord function to utils and update references in experiment service
* chore: move type coercion functions to utils for better organization and reuse
* chore(tracing): refactor internal tracing to use new events writer interface
* fix(experimentService): fix dataset item version conversion
* fix: remove invalid dependency
* fixup(tracing): ensure ordering key parity with experiment backfill
* fix: do not write to events table for self-hosters
* test: fix
* test: fix
* fix: do not write to events-table for self-hosters
* fix: rebase
* chore: type
* fix: skip remapping of IDs for self-hosters
* fix: test
* chore: push
* chore: move away from de-duplication approach
* chore: push
---------
Co-authored-by: Marlies Mayerhofer <74332854+marliessophie@users.noreply.github.com>
* fix(web): Create new `TableCellWithCopyButton` for `ApiKeyList`
* Fix react re-rendering issue after copying text
* Handle rejections when copying to clipboard
* Simplify useCopyToClipboard tests
* fix(web): Prevent toast error when toggling v4 with selected saved view
* Prevent table being rendered until flag is initialized
* Add mistakenly removed "as const" to StringParam
* chore(experiments): rewrite metrics aggregation for total cost and latency to skip trace-level aggregation
* fix(experiments): handle null values in latency and total cost cells in ExperimentsTable
* fix: typo
* feat(web): add support for AWS Bedrock API Keys (Bearer Tokens)
Add Bedrock API key authentication as an alternative to AWS access keys
(SigV4) for Amazon Bedrock LLM connections. Users can now choose between
AWS access keys and Bedrock API keys via a tab-based selector in the UI.
- Add BedrockApiKeySchema and BedrockAccessKeysSchema as a discriminated
union in shared credential schemas
- Add resolveBedrockAuth() to route between bearer token and SigV4 auth
- Add server-side validation of Bedrock credentials on create and update
- Derive and expose a safe authMethod enum (api-key, access-keys,
default-credentials) in the tRPC list response without leaking secrets
- Add auth method tab selector to the create/update LLM API key form
- Add Bedrock credential validation to the public API PUT endpoint
- Add comprehensive unit, integration, and e2e tests
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* ci: add secret
* fix(web): fix Bedrock DefaultCredentials test for cloud environment
The test assumed updating to BEDROCK_USE_DEFAULT_CREDENTIALS would
succeed, but the test env sets NEXT_PUBLIC_LANGFUSE_CLOUD_REGION="DEV"
which makes the server reject default credentials. Updated the test to
assert the expected rejection on cloud deployments.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(web): add self-hosted happy path test for DefaultCredentials update
The previous fix only asserted cloud rejection. Add back the original
happy-path test that temporarily sets NEXT_PUBLIC_LANGFUSE_CLOUD_REGION
to undefined (simulating self-hosted) so the update-to-DefaultCredentials
path is actually exercised.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(web): add .min(1) to BedrockAccessKeysSchema, guard default creds in public API
- Add .min(1) to accessKeyId and secretAccessKey in BedrockAccessKeysSchema
to reject empty-string credentials at validation time
- Add cloud guard in PUT /api/public/llm-connections rejecting the
BEDROCK_USE_DEFAULT_CREDENTIALS sentinel on Langfuse Cloud
- Fix DefaultCredentials tests: use per-test env override with try/finally
to simulate self-hosted deployments
- Add public API tests for sentinel rejection and invalid credential JSON
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The backfill cursor was only advanced inside the chunk-processing loop,
which is never reached when the query returns zero dataset run items.
This caused last_run_delay_seconds to grow indefinitely in environments
with no recent experiment activity.
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-10 15:41:29 +00:00
1418 changed files with 96652 additions and 32319 deletions
Establish consistency and best practices across Langfuse's backend packages (web, worker, packages/shared) using Next.js 14, tRPC, BullMQ, and TypeScript patterns.
Establish consistency and best practices across Langfuse's backend packages
(`web`, `worker`, `packages/shared`) using Next.js, tRPC, BullMQ, and TypeScript
patterns. Check package manifests such as `web/package.json` for current
framework versions before version-sensitive work.
Keep this file as an entrypoint; open reference files only when the task needs
their details.
## When to Use This Skill
@@ -64,7 +69,7 @@ Use this guide when working on:
### Layered Architecture
```
# Web Package (Next.js 14)
# Web Package (Next.js)
┌─ tRPC API ──────────────────┐ ┌── Public REST API ──────────┐
The shared package provides types, utilities, and server code used by both web and worker packages. It has **5 export paths** that control frontend vs backend access:
Service layer overview, dependency injection patterns, singleton patterns, repository pattern for data access, service design principles, caching strategies, testing services
Dual database architecture (PostgreSQL via Prisma + ClickHouse via direct client), PostgreSQL CRUD operations, ClickHouse query patterns (queryClickhouse, queryClickhouseStream, upsertClickhouse), repository pattern for complex queries, tenant isolation with projectId filtering, when to use which database
Environment variable validation with Zod, package-specific configs (web/env.mjs with t3-oss/env-nextjs, worker/env.ts, shared/env.ts), NEXT_PUBLIC_LANGFUSE_CLOUD_REGION usage, LANGFUSE_EE_LICENSE_KEY for enterprise features, best practices for env management
Integration tests (Public API with makeZodVerifiedAPICall), tRPC tests (createInnerTRPCContext, appRouter.createCaller), service-level tests (repository/service functions), worker tests (vitest with streams), test isolation principles, running tests (Jest for web, vitest for worker)
description:Shared backend guide for Langfuse's Next.js 14, tRPC, BullMQ, and TypeScript monorepo. Use when creating or reviewing tRPC routers, public REST endpoints, BullMQ queue processors, backend services, middleware, Prisma or ClickHouse data access, OpenTelemetry instrumentation, Zod validation, env configuration, or backend tests across web, worker, or packages/shared.
description:Shared backend guide for Langfuse's Next.js, tRPC, BullMQ, and TypeScript monorepo. Use when creating or reviewing tRPC routers, public REST endpoints, BullMQ queue processors, backend services, middleware, Prisma or ClickHouse data access, OpenTelemetry instrumentation, Zod validation, env configuration, or backend tests across web, worker, or packages/shared.
Complete guide to the layered architecture pattern used in Langfuse's Next.js 14/tRPC/Express monorepo.
Complete guide to the layered architecture pattern used in Langfuse's Next.js/tRPC/Express monorepo. Check package manifests such as `web/package.json` for current framework versions before version-sensitive work.
## Table of Contents
@@ -20,7 +20,7 @@ Langfuse uses a **three-layer architecture** with two primary entry points (tRPC
### The Three Layers
```
# Web Package (Next.js 14)
# Web Package (Next.js)
┌─ tRPC API ──────────────────┐ ┌── Public REST API ──────────┐
1.**Test Isolation**: Each test should be independent and runnable in any order
2.**Unique IDs**: Use `randomUUID()` or unique project IDs to avoid test interference
3.**Cleanup**: Always clean up test data in service tests (or use unique project IDs)
4.**Avoid Global Resets**: Prefer scoped cleanup or unique project IDs over global reset helpers
5.**Flags and Fallbacks**: When code branches on env flags, feature flags, or fallback data paths, test both branches and ensure fixtures are written to the same store the branch reads from
### By Test Type
| Test Type | Key Principles |
|-----------|----------------|
| **Integration** | Test HTTP endpoints, validate status codes and response shapes |
| **tRPC** | Use `createInnerTRPCContext` and `appRouter.createCaller`, test auth/permissions |
| **Service** | Test individual functions with isolated data, always cleanup |
| **Worker** | Use vitest, test streams with async iteration, test filtering logic |
### Test Data Management
```typescript
// ✅ GOOD: Use unique IDs
constprojectId=randomUUID();
consttraceId=randomUUID();
// ✅ GOOD: Cleanup in service tests
afterAll(async()=>{
awaitprisma.model.delete({where:{id: modelId}});
});
// ✅ GOOD: Use unique projects (no cleanup needed)
**Why rules take priority:** ClickHouse has specific behaviors (columnar storage, sparse indexes, merge tree mechanics) where general database intuition can be misleading. The rules encode validated, ClickHouse-specific guidance.
### For Formal Reviews
## Langfuse-Specific Rules
When performing a formal review of schemas, queries, or data ingestion:
- Use `packages/shared/src/server/queries/clickhouse-sql/event-query-builder.ts`
for queries against the `events` table. Do not hand-roll `events` SQL unless
you first confirm the query builder cannot express the query.
- Never use `FINAL` on the `events` table; it is designed so `FINAL` is not
required and the keyword hurts performance.
- Any migration in `packages/shared/clickhouse/migrations/clustered/**` with
more than one `ALTER` on the same table must end every metadata `ALTER`
(`ADD/DROP/MODIFY COLUMN`, `ADD/DROP INDEX`) with `SETTINGS alter_sync = 2`,
and every mutation-creating `ALTER` (`MATERIALIZE …`, `UPDATE`, `DELETE`)
with `SETTINGS mutations_sync = 2`. The matching `unclustered/` file runs
against plain `MergeTree` and does not need (and should not duplicate)
these settings.
---
@@ -53,6 +64,7 @@ When performing a formal review of schemas, queries, or data ingestion:
- [ ] LowCardinality applied to appropriate string columns
- [ ] ReplacingMergeTree has version column if used
- [ ] Clustered migration files with multiple ALTERs on the same table use `SETTINGS alter_sync = 2` (metadata) and `SETTINGS mutations_sync = 2` (`MATERIALIZE …`, `UPDATE`, `DELETE`); unclustered mirror has none
### For Query Reviews (SELECT, JOIN, aggregations)
@@ -227,8 +239,8 @@ Each rule file in `rules/` contains:
---
## Full Compiled Document
## Compatibility Entrypoint
For the complete guide with all rules expanded inline: `AGENTS.md`
Use `AGENTS.md` when you need to check multiple rules quickly without reading individual files.
`AGENTS.md` is a short compatibility index for agents that open that file
directly. The authoritative workflow lives in this `SKILL.md`, and detailed rule
env:<env> (service:worker OR service:worker-cpu) operation_name:bullmq.process
```
Then group by `resource_name`, queue facets such as `bullmq.queue` or
`messaging.*`, and error fields. Facet names can differ between Datadog sites,
so inspect one sample span before relying on a specific facet.
Queue-specific starter query:
```text
env:<env> (service:worker OR service:worker-cpu) operation_name:bullmq.process (resource_name:"process otel-ingestion-queue" OR resource_name:"Worker.run otel-ingestion-queue" OR bullmq.queue:otel-ingestion-queue)
```
For sharded queues, query the base queue and shard suffixes:
```text
env:<env> (service:worker OR service:worker-cpu) operation_name:bullmq.process resource_name:"*otel-ingestion-queue*"
```
If a queue file wraps the handler with `instrumentAsync`, also search the
| Logger / instrumentation | `packages/shared/src/server/logger.ts`, `packages/shared/src/server/instrumentation.ts` | log silently dropped because `LANGFUSE_LOG_LEVEL` set wrong, or span missing because handler doesn't call `instrumentAsync` |
| Webhook URL validation | `packages/shared/src/server/validateWebhookURL.ts` | rejects with messages that *look* like DNS errors but are SSRF guard rejections |
| Encryption | `packages/shared/encryption` | bad keys → 403/auth-style failures masquerading as upstream errors |
## Common Symptoms → First Files To Read
- **"403 from upstream":** check the per-integration credentials table in
description:Use when upgrading a dependency in this pnpm workspace, including requests to bump a package to a specific version, compare the registry latest version with the latest version installable under the current minimum-release-age window, or decide whether minimumReleaseAgeExclude in pnpm-workspace.yaml must change. Ask the user for the package name or target version when either is missing.
description:>-
Upgrade pnpm workspace dependencies to target/latest versions:
description:Guide for creating effective skills. This skill should be used when users want to create a new skill (or update an existing skill) that extends Codex's capabilities with specialized knowledge, workflows, or tool integrations.
metadata:
short-description:Create or update Codex skills
---
# Skill Creator
This skill provides guidance for creating effective skills.
## About Skills
Skills are modular, self-contained folders that extend Codex's capabilities by providing
specialized knowledge, workflows, and tools. Think of them as "onboarding guides" for specific
domains or tasks—they transform Codex from a general-purpose agent into a specialized agent
equipped with procedural knowledge that no model can fully possess.
### What Skills Provide
1. Specialized workflows - Multi-step procedures for specific domains
2. Tool integrations - Instructions for working with specific file formats or APIs
3. Domain expertise - Company-specific knowledge, schemas, business logic
4. Bundled resources - Scripts, references, and assets for complex and repetitive tasks
## Core Principles
### Concise is Key
The context window is a public good. Skills share the context window with everything else Codex needs: system prompt, conversation history, other Skills' metadata, and the actual user request.
**Default assumption: Codex is already very smart.** Only add context Codex doesn't already have. Challenge each piece of information: "Does Codex really need this explanation?" and "Does this paragraph justify its token cost?"
Prefer concise examples over verbose explanations.
### Set Appropriate Degrees of Freedom
Match the level of specificity to the task's fragility and variability:
**High freedom (text-based instructions)**: Use when multiple approaches are valid, decisions depend on context, or heuristics guide the approach.
**Medium freedom (pseudocode or scripts with parameters)**: Use when a preferred pattern exists, some variation is acceptable, or configuration affects behavior.
**Low freedom (specific scripts, few parameters)**: Use when operations are fragile and error-prone, consistency is critical, or a specific sequence must be followed.
Think of Codex as exploring a path: a narrow bridge with cliffs needs specific guardrails (low freedom), while an open field allows many routes (high freedom).
### Require Human Review for Ticket Writes
For skills that create Linear tickets, update Linear tickets, or add evidence to
existing tickets, require human review before any write. The skill must present
all findings in a table, ask the human which findings to create or update in
Linear, and wait for an explicit selection before making changes.
Use this table structure unless the domain needs additional columns:
| ID | Finding | Evidence | Impact / Scope | Existing Ticket Match | Proposed Linear Action | Confidence | Human Decision |
| --- | --- | --- | --- | --- | --- | --- | --- |
| F1 | Concise symptom or bug claim | Measured counts, deltas, links, traces, logs, or "No measurements found" | Affected env, service, route, customer segment, or blast radius supported by evidence | Existing issue key/link, duplicate candidate, or "None found" | Create new ticket, add evidence comment, update status/labels, or no action | High/medium/low plus one short reason | Leave blank for the human to choose |
In the skill instructions, state that Codex must not create tickets, comment on
tickets, edit ticket fields, or add evidence until the human chooses one or more
row IDs and actions. If the human asks for an automated sweep, still pause at
this review table before writing to Linear.
### Protect Validation Integrity
You may use subagents during iteration to validate whether a skill works on realistic tasks or whether a suspected problem is real. This is most useful when you want an independent pass on the skill's behavior, outputs, or failure modes after a revision. Only do this when it is possible to start new subagents.
When using subagents for validation, treat that as an evaluation surface. The goal is to learn whether the skill generalizes, not whether another agent can reconstruct the answer from leaked context.
Prefer raw artifacts such as example prompts, outputs, diffs, logs, or traces. Give the minimum task-local context needed to perform the validation. Avoid passing the intended answer, suspected bug, intended fix, or your prior conclusions unless the validation explicitly requires them.
### Anatomy of a Skill
Every skill consists of a required SKILL.md file and optional bundled resources:
```
skill-name/
├── SKILL.md (required)
│ ├── YAML frontmatter metadata (required)
│ │ ├── name: (required)
│ │ └── description: (required)
│ └── Markdown instructions (required)
├── agents/ (recommended)
│ └── openai.yaml - UI metadata for skill lists and chips
└── Bundled Resources (optional)
├── scripts/ - Executable code (Python/Bash/etc.)
├── references/ - Documentation intended to be loaded into context as needed
└── assets/ - Files used in output (templates, icons, fonts, etc.)
```
#### SKILL.md (required)
Every SKILL.md consists of:
- **Frontmatter** (YAML): Contains `name` and `description` fields. These are the only fields that Codex reads to determine when the skill gets used, thus it is very important to be clear and comprehensive in describing what the skill is, and when it should be used.
- **Body** (Markdown): Instructions and guidance for using the skill. Only loaded AFTER the skill triggers (if at all).
#### Agents metadata (recommended)
- UI-facing metadata for skill lists and chips
- Read references/openai_yaml.md before generating values and follow its descriptions and constraints
- Create: human-facing `display_name`, `short_description`, and `default_prompt` by reading the skill
- Generate deterministically by passing the values as `--interface key=value` to `scripts/generate_openai_yaml.py` or `scripts/init_skill.py`
- On updates: validate `agents/openai.yaml` still matches SKILL.md; regenerate if stale
- Only include other optional interface fields (icons, brand color) if explicitly provided
- See references/openai_yaml.md for field definitions and examples
#### Bundled Resources (optional)
##### Scripts (`scripts/`)
Executable code (Python/Bash/etc.) for tasks that require deterministic reliability or are repeatedly rewritten.
- **When to include**: When the same code is being rewritten repeatedly or deterministic reliability is needed
- **Example**: `scripts/rotate_pdf.py` for PDF rotation tasks
- **Benefits**: Token efficient, deterministic, may be executed without loading into context
- **Note**: Scripts may still need to be read by Codex for patching or environment-specific adjustments
##### References (`references/`)
Documentation and reference material intended to be loaded as needed into context to inform Codex's process and thinking.
- **When to include**: For documentation that Codex should reference while working
- **Examples**: `references/finance.md` for financial schemas, `references/mnda.md` for company NDA template, `references/policies.md` for company policies, `references/api_docs.md` for API specifications
- **Use cases**: Database schemas, API documentation, domain knowledge, company policies, detailed workflow guides
- **Benefits**: Keeps SKILL.md lean, loaded only when Codex determines it's needed
- **Best practice**: If files are large (>10k words), include grep search patterns in SKILL.md
- **Avoid duplication**: Information should live in either SKILL.md or references files, not both. Prefer references files for detailed information unless it's truly core to the skill—this keeps SKILL.md lean while making information discoverable without hogging the context window. Keep only essential procedural instructions and workflow guidance in SKILL.md; move detailed reference material, schemas, and examples to references files.
##### Assets (`assets/`)
Files not intended to be loaded into context, but rather used within the output Codex produces.
- **When to include**: When the skill needs files that will be used in the final output
- **Examples**: `assets/logo.png` for brand assets, `assets/slides.pptx` for PowerPoint templates, `assets/frontend-template/` for HTML/React boilerplate, `assets/font.ttf` for typography
- **Use cases**: Templates, images, icons, boilerplate code, fonts, sample documents that get copied or modified
- **Benefits**: Separates output resources from documentation, enables Codex to use files without loading them into context
#### What to Not Include in a Skill
A skill should only contain essential files that directly support its functionality. Do NOT create extraneous documentation or auxiliary files, including:
- README.md
- INSTALLATION_GUIDE.md
- QUICK_REFERENCE.md
- CHANGELOG.md
- etc.
The skill should only contain the information needed for an AI agent to do the job at hand. It should not contain auxiliary context about the process that went into creating it, setup and testing procedures, user-facing documentation, etc. Creating additional documentation files just adds clutter and confusion.
### Progressive Disclosure Design Principle
Skills use a three-level loading system to manage context efficiently:
1.**Metadata (name + description)** - Always in context (~100 words)
2.**SKILL.md body** - When skill triggers (<5k words)
3.**Bundled resources** - As needed by Codex (Unlimited because scripts can be executed without reading into context window)
#### Progressive Disclosure Patterns
Keep SKILL.md body to the essentials and under 500 lines to minimize context bloat. Split content into separate files when approaching this limit. When splitting out content into other files, it is very important to reference them from SKILL.md and describe clearly when to read them, to ensure the reader of the skill knows they exist and when to use them.
**Key principle:** When a skill supports multiple variations, frameworks, or options, keep only the core workflow and selection guidance in SKILL.md. Move variant-specific details (patterns, examples, configuration) into separate reference files.
**Pattern 1: High-level guide with references**
```markdown
# PDF Processing
## Quick start
Extract text with pdfplumber:
[code example]
## Advanced features
- **Form filling**: See [FORMS.md](FORMS.md) for complete guide
- **API reference**: See [REFERENCE.md](REFERENCE.md) for all methods
- **Examples**: See [EXAMPLES.md](EXAMPLES.md) for common patterns
```
Codex loads FORMS.md, REFERENCE.md, or EXAMPLES.md only when needed.
**Pattern 2: Domain-specific organization**
For Skills with multiple domains, organize content by domain to avoid loading irrelevant context:
```
bigquery-skill/
├── SKILL.md (overview and navigation)
└── reference/
├── finance.md (revenue, billing metrics)
├── sales.md (opportunities, pipeline)
├── product.md (API usage, features)
└── marketing.md (campaigns, attribution)
```
When a user asks about sales metrics, Codex only reads sales.md.
Similarly, for skills supporting multiple frameworks or variants, organize by variant:
```
cloud-deploy/
├── SKILL.md (workflow + provider selection)
└── references/
├── aws.md (AWS deployment patterns)
├── gcp.md (GCP deployment patterns)
└── azure.md (Azure deployment patterns)
```
When the user chooses AWS, Codex only reads aws.md.
**Pattern 3: Conditional details**
Show basic content, link to advanced content:
```markdown
# DOCX Processing
## Creating documents
Use docx-js for new documents. See [DOCX-JS.md](DOCX-JS.md).
## Editing documents
For simple edits, modify the XML directly.
**For tracked changes**: See [REDLINING.md](REDLINING.md)
**For OOXML details**: See [OOXML.md](OOXML.md)
```
Codex reads REDLINING.md or OOXML.md only when the user needs those features.
**Important guidelines:**
- **Avoid deeply nested references** - Keep references one level deep from SKILL.md. All reference files should link directly from SKILL.md.
- **Structure longer reference files** - For files longer than 100 lines, include a table of contents at the top so Codex can see the full scope when previewing.
## Skill Creation Process
Skill creation involves these steps:
1. Understand the skill with concrete examples
2. Plan reusable skill contents (scripts, references, assets)
3. Initialize the skill (run init_skill.py)
4. Edit the skill (implement resources and write SKILL.md)
5. Validate the skill (run quick_validate.py)
6. Iterate based on real usage and forward-test complex skills.
Follow these steps in order, skipping only if there is a clear reason why they are not applicable.
### Skill Naming
- Use lowercase letters, digits, and hyphens only; normalize user-provided titles to hyphen-case (e.g., "Plan Mode" -> `plan-mode`).
- When generating names, generate a name under 64 characters (letters, digits, hyphens).
- Prefer short, verb-led phrases that describe the action.
- Namespace by tool when it improves clarity or triggering (e.g., `gh-address-comments`, `linear-address-issue`).
- Name the skill folder exactly after the skill name.
### Step 1: Understanding the Skill with Concrete Examples
Skip this step only when the skill's usage patterns are already clearly understood. It remains valuable even when working with an existing skill.
To create an effective skill, clearly understand concrete examples of how the skill will be used. This understanding can come from either direct user examples or generated examples that are validated with user feedback.
For example, when building an image-editor skill, relevant questions include:
- "What functionality should the image-editor skill support? Editing, rotating, anything else?"
- "Can you give some examples of how this skill would be used?"
- "I can imagine users asking for things like 'Remove the red-eye from this image' or 'Rotate this image'. Are there other ways you imagine this skill being used?"
- "What would a user say that should trigger this skill?"
- "Where should I create this skill? If you do not have a preference, I will place it in `$CODEX_HOME/skills` (or `~/.codex/skills` when `CODEX_HOME` is unset) so Codex can discover it automatically."
To avoid overwhelming users, avoid asking too many questions in a single message. Start with the most important questions and follow up as needed for better effectiveness.
Conclude this step when there is a clear sense of the functionality the skill should support.
### Step 2: Planning the Reusable Skill Contents
To turn concrete examples into an effective skill, analyze each example by:
1. Considering how to execute on the example from scratch
2. Identifying what scripts, references, and assets would be helpful when executing these workflows repeatedly
Example: When building a `pdf-editor` skill to handle queries like "Help me rotate this PDF," the analysis shows:
1. Rotating a PDF requires re-writing the same code each time
2. A `scripts/rotate_pdf.py` script would be helpful to store in the skill
Example: When designing a `frontend-webapp-builder` skill for queries like "Build me a todo app" or "Build me a dashboard to track my steps," the analysis shows:
1. Writing a frontend webapp requires the same boilerplate HTML/React each time
2. An `assets/hello-world/` template containing the boilerplate HTML/React project files would be helpful to store in the skill
Example: When building a `big-query` skill to handle queries like "How many users have logged in today?" the analysis shows:
1. Querying BigQuery requires re-discovering the table schemas and relationships each time
2. A `references/schema.md` file documenting the table schemas would be helpful to store in the skill
To establish the skill's contents, analyze each concrete example to create a list of the reusable resources to include: scripts, references, and assets.
### Step 3: Initializing the Skill
At this point, it is time to actually create the skill.
Skip this step only if the skill being developed already exists. In this case, continue to the next step.
Before running `init_skill.py`, ask where the user wants the skill created. If they do not specify a location, default to `$CODEX_HOME/skills`; when `CODEX_HOME` is unset, fall back to `~/.codex/skills` so the skill is auto-discovered.
When creating a new skill from scratch, always run the `init_skill.py` script. The script conveniently generates a new template skill directory that automatically includes everything a skill requires, making the skill creation process much more efficient and reliable.
- Creates the skill directory at the specified path
- Generates a SKILL.md template with proper frontmatter and TODO placeholders
- Creates `agents/openai.yaml` using agent-generated `display_name`, `short_description`, and `default_prompt` passed via `--interface key=value`
- Optionally creates resource directories based on `--resources`
- Optionally adds example files when `--examples` is set
After initialization, customize the SKILL.md and add resources as needed. If you used `--examples`, replace or delete placeholder files.
Generate `display_name`, `short_description`, and `default_prompt` by reading the skill, then pass them as `--interface key=value` to `init_skill.py` or regenerate with:
Only include other optional interface fields when the user explicitly provides them. For full field descriptions and examples, see references/openai_yaml.md.
### Step 4: Edit the Skill
When editing the (newly-generated or existing) skill, remember that the skill is being created for another instance of Codex to use. Include information that would be beneficial and non-obvious to Codex. Consider what procedural knowledge, domain-specific details, or reusable assets would help another Codex instance execute these tasks more effectively.
After substantial revisions, or if the skill is particularly tricky, you should use subagents to forward-test the skill on realistic tasks or artifacts. When doing so, pass the artifact under validation rather than your diagnosis of what is wrong, and keep the prompt generic enough that success depends on transferable reasoning rather than hidden ground truth.
#### Start with Reusable Skill Contents
To begin implementation, start with the reusable resources identified above: `scripts/`, `references/`, and `assets/` files. Note that this step may require user input. For example, when implementing a `brand-guidelines` skill, the user may need to provide brand assets or templates to store in `assets/`, or documentation to store in `references/`.
Added scripts must be tested by actually running them to ensure there are no bugs and that the output matches what is expected. If there are many similar scripts, only a representative sample needs to be tested to ensure confidence that they all work while balancing time to completion.
If you used `--examples`, delete any placeholder files that are not needed for the skill. Only create resource directories that are actually required.
#### Update SKILL.md
**Writing Guidelines:** Always use imperative/infinitive form.
##### Frontmatter
Write the YAML frontmatter with `name` and `description`:
-`name`: The skill name
-`description`: This is the primary triggering mechanism for your skill, and helps Codex understand when to use the skill.
- Include both what the Skill does and specific triggers/contexts for when to use it.
- Include all "when to use" information here - Not in the body. The body is only loaded after triggering, so "When to Use This Skill" sections in the body are not helpful to Codex.
- Example description for a `docx` skill: "Comprehensive document creation, editing, and analysis with support for tracked changes, comments, formatting preservation, and text extraction. Use when Codex needs to work with professional documents (.docx files) for: (1) Creating new documents, (2) Modifying or editing content, (3) Working with tracked changes, (4) Adding comments, or any other document tasks"
Do not include any other fields in YAML frontmatter.
##### Body
Write instructions for using the skill and its bundled resources.
### Step 5: Validate the Skill
Once development of the skill is complete, validate the skill folder to catch basic issues early:
```bash
scripts/quick_validate.py <path/to/skill-folder>
```
The validation script checks YAML frontmatter format, required fields, and naming rules. If validation fails, fix the reported issues and run the command again.
### Step 6: Iterate
After testing the skill, you may detect the skill is complex enough that it requires forward-testing; or users may request improvements.
User testing often this happens right after using the skill, with fresh context of how the skill performed.
**Forward-testing and iteration workflow:**
1. Use the skill on real tasks
2. Notice struggles or inefficiencies
3. Identify how SKILL.md or bundled resources should be updated
4. Implement changes and test again
5. Forward-test if it is reasonable and appropriate
## Forward-testing
To forward-test, launch subagents as a way to stress test the skill with minimal context.
Subagents should *not* know that they are being asked to test the skill. They should be treated as
an agent asked to perform a task by the user. Prompts to subagents should look like:
`Use $skill-x at /path/to/skill-x to solve problem y`
Not:
`Review the skill at /path/to/skill-x; pretend a user asks you to...`
Decision rule for forward-testing:
- Err on the side of forward-testing
- Ask for approval if you think there's a risk that forward-testing would:
* take a long time,
* require additional approvals from the user, or
* modify live production systems
In these cases, show the user your proposed prompt and request (1) a yes/no decision, and
(2) any suggested modifictions.
Considerations when forward-testing:
- use fresh threads for independent passes
- pass the skill, and a request in a similar way the user would.
- pass raw artifacts, not your conclusions
- avoid showing expected answers or intended fixes
- rebuild context from source artifacts after each iteration
- review the subagent's output and reasoning and emitted artifacts
- avoid leaving artifacts the agent can find on disk between iterations;
clean up subagents' artifacts to avoid additional contamination.
If forward-testing only succeeds when subagents see leaked context, tighten the skill or the
# openai.yaml fields (full example + descriptions)
`agents/openai.yaml` is an extended, product-specific config intended for the machine/harness to read, not the agent. Other product-specific config can also live in the `agents/` folder.
default_prompt:"Optional surrounding prompt to use the skill with"
dependencies:
tools:
- type:"mcp"
value:"github"
description:"GitHub MCP server"
transport:"streamable_http"
url:"https://api.githubcopilot.com/mcp/"
policy:
allow_implicit_invocation:true
```
## Field descriptions and constraints
Top-level constraints:
- Quote all string values.
- Keep keys unquoted.
- For `interface.default_prompt`: generate a helpful, short (typically 1 sentence) example starting prompt based on the skill. It must explicitly mention the skill as `$skill-name` (e.g., "Use $skill-name-here to draft a concise weekly status update.").
-`interface.display_name`: Human-facing title shown in UI skill lists and chips.
-`interface.short_description`: Human-facing short UI blurb (25–64 chars) for quick scanning.
-`interface.icon_small`: Path to a small icon asset (relative to skill dir). Default to `./assets/` and place icons in the skill's `assets/` folder.
-`interface.icon_large`: Path to a larger logo asset (relative to skill dir). Default to `./assets/` and place icons in the skill's `assets/` folder.
-`interface.brand_color`: Hex color used for UI accents (e.g., badges).
-`interface.default_prompt`: Default prompt snippet inserted when invoking the skill.
-`dependencies.tools[].type`: Dependency category. Only `mcp` is supported for now.
-`dependencies.tools[].value`: Identifier of the tool or dependency.
-`dependencies.tools[].description`: Human-readable explanation of the dependency.
-`dependencies.tools[].transport`: Connection type when `type` is `mcp`.
-`dependencies.tools[].url`: MCP server URL when `type` is `mcp`.
-`policy.allow_implicit_invocation`: When false, the skill is not injected into
the model context by default, but can still be invoked explicitly via `$skill`.
description: [TODO: Complete and informative explanation of what the skill does and when to use it. Include WHEN to use this skill - specific scenarios, file types, or tasks that trigger it.]
---
# {skill_title}
## Overview
[TODO: 1-2 sentences explaining what this skill enables]
## Structuring This Skill
[TODO: Choose the structure that best fits this skill's purpose. Common patterns:
**1. Workflow-Based** (best for sequential processes)
- Works well when there are clear step-by-step procedures
- BigQuery: API reference documentation and query examples
- Finance: Schema documentation, company policies
**Appropriate for:** In-depth documentation, API references, database schemas, comprehensive guides, or any detailed information that Codex should reference while working.
### assets/
Files not intended to be loaded into context, but rather used within the output Codex produces.
**Examples from other skills:**
- Brand styling: PowerPoint template files (.pptx), logo files
**Appropriate for:** Templates, boilerplate code, document templates, images, icons, fonts, or any files meant to be copied or used in the final output.
---
**Not every skill requires all three types of resources.**
"""
EXAMPLE_SCRIPT='''#!/usr/bin/env python3
"""
Example helper script for {skill_name}
This is a placeholder script that can be executed directly.
Replace with actual implementation or delete if not needed.
Example real scripts from other skills:
- pdf/scripts/fill_fillable_fields.py - Fills PDF form fields
- pdf/scripts/convert_pdf_to_images.py - Converts PDF pages to images
"""
def main():
print("This is an example script for {skill_name}")
# TODO: Add actual script logic here
# This could be data processing, file conversion, API calls, etc.
if __name__ == "__main__":
main()
'''
EXAMPLE_REFERENCE="""# Reference Documentation for {skill_title}
This is a placeholder for detailed reference documentation.
Replace with actual reference content or delete if not needed.
Example real reference docs from other skills:
- product-management/references/communication.md - Comprehensive guide for status updates
- product-management/references/context_building.md - Deep-dive on gathering context
- bigquery/references/ - API references and query examples
## When Reference Docs Are Useful
Reference docs are ideal for:
- Comprehensive API documentation
- Detailed workflow guides
- Complex multi-step processes
- Information too lengthy for main SKILL.md
- Content that's only needed for specific use cases
## Structure Suggestions
### API Reference Example
- Overview
- Authentication
- Endpoints with examples
- Error codes
- Rate limits
### Workflow Guide Example
- Prerequisites
- Step-by-step instructions
- Common patterns
- Troubleshooting
- Best practices
"""
EXAMPLE_ASSET="""# Example Asset File
This placeholder represents where asset files would be stored.
Replace with actual asset files (templates, images, fonts, etc.) or delete if not needed.
Asset files are NOT intended to be loaded into context, but rather used within
| Linear production bug | Datadog monitor/query/trace/log links, incident.io incident if any, status incident if any | Linear bug evidence plus event row sources |
| Alert disposition | monitor ID/title, env, reason, verdict, owner/team if visible | Datadog table row with `Linked Event` set to disposition |
For a healthy review, each real production event should satisfy one of:
```text
Canonical event = incident.io incident
OR canonical event = Linear production bug
OR canonical event = explicit alert disposition
```
### Proposed Link Titles
When proposing or later creating links, use short stable titles:
-`Datadog monitor: <monitor name>`
-`Datadog logs: <env/service/symptom>`
-`Datadog spans: <env/route/symptom>`
-`Datadog trace: <trace id or route>`
-`Status incident: <status title>`
-`incident.io: <INC reference>`
-`Linear follow-up: <issue key>`
Do not write any of these links unless the user explicitly asks for changes
after reviewing the report.
## Linear Bug Table
Start from all Linear tickets with the `bug` label that were touched by the
window. Do not rely only on text searches for `prod`, `incident`, or `Datadog`;
those searches are useful for enrichment but are not the source universe.
Use this table for the bug section:
| Linear | Title | Summary | Owner | Status | Touched Last Week Because | Production Evidence | Classification | Counted? |
short_description:"Summarize bugs, pages, and incidents"
default_prompt:"Use $weekly-production-review to prepare an event-centric weekly production review from Linear bugs, Datadog alert/page signals, and status-page or incident data."
# your codebase is analyzed, see https://docs.github.com/en/code-security/code-scanning/creating-an-advanced-setup-for-code-scanning/codeql-code-scanning-for-compiled-languages
@@ -111,87 +111,87 @@ We built a monorepo using [pnpm](https://pnpm.io/motivation) and [turbo](https:/
Requirements
- Node.js 24 as specified in the [.nvmrc](.nvmrc)
- Pnpm v.10.33.0
- Pnpm v.11.1.3
- Docker to run the database locally
- Clickhouse client
**Note:** You can also simply run Langfuse in a **GitHub Codespace** via the provided devcontainer. To do this, click on the green "Code" button in the top right corner of the repository and select "Open with Codespaces".
### Codex Cloud Setup
You can also attach this repository to an OpenAI Codex cloud environment. The
cloud environment itself is configured in the Codex UI, while the repo-owned
bootstrap is versioned in:
-`scripts/codex/setup.sh`
-`scripts/codex/maintenance.sh`
Recommended Codex UI configuration:
1. Create a new cloud environment for this repository in the Codex UI.
2. Choose a base environment with Node.js 24 support.
3. Set the setup script to:
```bash
bash scripts/codex/setup.sh
```
4. Set the maintenance script to:
```bash
bash scripts/codex/maintenance.sh
```
5. Keep internet access disabled by default, or only allow the minimum domains
needed for your task.
6. Add secrets and environment variables in the Codex UI instead of committing
them to the repository.
Notes:
- This Codex setup is intended for repository tasks such as code changes,
linting, typechecking, and targeted tests.
- It does **not** start the full Langfuse stack. Local development still uses
Docker and `pnpm run dx` / `pnpm run dev`.
- Running the full application inside Codex requires external services for
PostgreSQL, Redis, ClickHouse, and object storage, plus matching environment
variables in the Codex UI.
### Shared Agent Setup
This repository keeps the shared agent setup in source control so developers
using different tools can work against the same instructions, bootstrap, and
- Tool-specific MCP configs generated locally from that catalog and not committed:
- `.mcp.json`
- `.cursor/mcp.json`
- `.vscode/mcp.json`
- `.codex/config.toml`
- Tool-specific runtime shims generated locally from the shared config and not committed:
- `.claude/settings.json`
- `.codex/environments/environment.toml`
- `.cursor/environment.json`
- Tool-specific skill projections generated locally and not committed:
- `.claude/skills/*`
- Shared bootstrap for agent environments: `bash scripts/codex/setup.sh`
When you change the shared MCP setup:
1. Edit `.agents/config.json`
2. Run `pnpm run agents:sync`
3. Run `pnpm run agents:check`
4. Do not commit the generated MCP config files or runtime shims
**Steps**
**Note:** You can also simply run Langfuse in a **GitHub Codespace** via the provided devcontainer. To do this, click on the green "Code" button in the top right corner of the repository and select "Open with Codespaces".
### Codex Cloud Setup
You can also attach this repository to an OpenAI Codex cloud environment. The
cloud environment itself is configured in the Codex UI, while the repo-owned
bootstrap is versioned in:
-`scripts/codex/setup.sh`
-`scripts/codex/maintenance.sh`
Recommended Codex UI configuration:
1. Create a new cloud environment for this repository in the Codex UI.
2. Choose a base environment with Node.js 24 support.
3. Set the setup script to:
```bash
bash scripts/codex/setup.sh
```
4. Set the maintenance script to:
```bash
bash scripts/codex/maintenance.sh
```
5. Keep internet access disabled by default, or only allow the minimum domains
needed for your task.
6. Add secrets and environment variables in the Codex UI instead of committing
them to the repository.
Notes:
- This Codex setup is intended for repository tasks such as code changes,
linting, typechecking, and targeted tests.
- It does **not** start the full Langfuse stack. Local development still uses
Docker and `pnpm run dx` / `pnpm run dev`.
- Running the full application inside Codex requires external services for
PostgreSQL, Redis, ClickHouse, and object storage, plus matching environment
variables in the Codex UI.
### Shared Agent Setup
This repository keeps the shared agent setup in source control so developers
using different tools can work against the same instructions, bootstrap, and
- Tool-specific MCP configs generated locally from that catalog and not committed:
- `.mcp.json`
- `.cursor/mcp.json`
- `.vscode/mcp.json`
- `.codex/config.toml`
- Tool-specific runtime shims generated locally from the shared config and not committed:
- `.claude/settings.json`
- `.codex/environments/environment.toml`
- `.cursor/environment.json`
- Tool-specific skill projections generated locally and not committed:
- `.claude/skills/*`
- Shared bootstrap for agent environments: `bash scripts/codex/setup.sh`
When you change the shared MCP setup:
1. Edit `.agents/config.json`
2. Run `pnpm run agents:sync`
3. Run `pnpm run agents:check`
4. Do not commit the generated MCP config files or runtime shims
**Steps**
1. Install development dependencies:
- [golang-migrate](https://github.com/golang-migrate/migrate/tree/master/cmd/migrate#migrate-cli) as CLI
@@ -316,18 +316,16 @@ Tests automatically create the PostgreSQL test database if it doesn't exist and
### Tests in the `web` package (public API)
We're using Jest with in the `web` package. Therefore, if you want to provide an argument to the test runner, do it directly without an intermittent `--`.
There are two types of unit tests:
We're using Vitest in the `web` package. There are two types of unit tests:
- `test` (server tests)
- `test-client`
To run a specific test, for example the test: `"should handle special characters in prompt names"` in `prompts.v2.servertest.ts`, run:
To run a specific test by name within a file, run:
```sh
cd web # or with --filter=web
pnpm test --testPathPatterns="prompts\.v2\.servertest" --testNamePattern="should handle special characters in prompt names"
cd web # or use `pnpm --filterweb test ...` from the repository root
pnpm test prompts.v2.servertest -t "should handle special characters in prompt names"
```
To run all tests:
@@ -344,10 +342,10 @@ pnpm run test:watch
### Tests in the `worker` package
For the `worker` package, we're using `vitest` to run unit tests.
For the `worker` package, we're also using Vitest to run unit tests.
```sh
pnpm run test --filter=worker -- FILE_YOU_WANT_TO_TEST.ts -t "test name"
pnpm --filterworker run test FILE_YOU_WANT_TO_TEST.ts -t "test name"
```
## CI/CD
@@ -357,7 +355,7 @@ We use GitHub Actions for CI/CD, the configuration is in [`.github/workflows/pip
CI on `main` and `pull_request`
- Check Linting
- E2E test of API using Jest
- E2E test of API using Vitest
- E2E tests of UI using Playwright
CD on `main`
@@ -513,5 +511,5 @@ npx fern-api generate --api organizations # for the organizations API
Langfuse is MIT licensed, except for `ee/` folder. See [LICENSE](LICENSE) and [docs](https://langfuse.com/docs/open-source) for more details.
When contributing to the Langfuse codebase, you need to agree to the [Contributor License Agreement](https://cla-assistant.io/langfuse/langfuse). You only need to do this once and the CLA bot will remind you if you haven't signed it yet.
If the CLA check gets stuck after signing (a [known cla-assistant bug](https://github.com/cla-assistant/cla-assistant/issues/520)), comment `/check-cla` on your PR to retrigger it.
If the CLA check gets stuck after signing (a [known cla-assistant bug](https://github.com/cla-assistant/cla-assistant/issues/520)), comment `/check-cla` on your PR to retrigger it.
@@ -90,7 +90,7 @@ Langfuse is an **open source LLM engineering** platform. It helps teams collabor
- [Prompt Management](https://langfuse.com/docs/prompt-management/get-started) helps you centrally manage, version control, and collaboratively iterate on your prompts. Thanks to strong caching on server and client side, you can iterate on prompts without adding latency to your application.
- [Evaluations](https://langfuse.com/docs/evaluation/overview) are key to the LLM application development workflow, and Langfuse adapts to your needs. It supports LLM-as-a-judge, user feedback collection, manual labeling, and custom evaluation pipelines via APIs/SDKs.
- [Evaluations](https://langfuse.com/docs/evaluation/overview) are key to the LLM application development workflow, and Langfuse adapts to your needs. It supports LLM-as-a-judge, Code evaluators, user feedback collection, manual labeling, and custom evaluation pipelines via APIs/SDKs.
- [Datasets](https://langfuse.com/docs/evaluation/dataset-runs/datasets) enable test sets and benchmarks for evaluating your LLM application. They support continuous improvement, pre-deployment testing, structured experiments, flexible evaluation, and seamless integration with frameworks like LangChain and LlamaIndex.
@@ -120,7 +120,7 @@ Run Langfuse on your own infrastructure:
docs:Reference a score config on a score. When set, the score name must equal the config name and scores must comply with the config's range and data type. For categorical scores, the value must map to a config category. Numeric scores might be constrained by the score config's max and min values
source:
type:optional<CreateScoreSource>
docs:The source of the score. Defaults to API. Set to ANNOTATION to prefill scores (e.g. from an LLM) for a human reviewer to verify in an annotation queue. When source is ANNOTATION, a configId is required unless dataType is CORRECTION. EVAL is reserved for internal evaluator outputs and is not accepted on this endpoint.
examples:
- value:
name:"novelty"
@@ -77,6 +80,22 @@ types:
name:"hallucination"
value:0
datasetRunId:"7891-5678-90ab-hijk"
- value:
name:"accuracy"
value:0.9
dataType:"NUMERIC"
configId:"9203-4567-89ab-cdef"
traceId:"cdef-1234-5678-90ab"
source:"ANNOTATION"
CreateScoreSource:
docs:|
Source values accepted when creating a score via the public API.
EVAL is reserved for internal evaluator outputs and is intentionally not
- `LEGACY_TRACES_OBSERVATIONS`: traces, observations, and scores tables with a fixed column set. The `exportFieldGroups` field is not applicable.
- `OBSERVATIONS_V2`: same data model as the `/api/public/v2/observations` endpoint, plus scores. Columns are controlled by `exportFieldGroups`.
- `LEGACY_TRACES_AND_ENRICHED_OBSERVATIONS`: both sets. For the `OBSERVATIONS_V2` portion, columns are controlled by `exportFieldGroups`.
**Note:** `OBSERVATIONS_V2` and the enriched-observations portion of `LEGACY_TRACES_AND_ENRICHED_OBSERVATIONS` rely on the enriched observations table (Langfuse Fast Preview / v4), which is currently available on Langfuse Cloud only. See https://langfuse.com/docs/v4.
enum:
- LEGACY_TRACES_OBSERVATIONS
- OBSERVATIONS_V2
- LEGACY_TRACES_AND_ENRICHED_OBSERVATIONS
BlobStorageExportFieldGroup:
docs:Field group for the OBSERVATIONS_V2 and LEGACY_TRACES_AND_ENRICHED_OBSERVATIONS export.
enum:
- core
- basic
- time
- io
- metadata
- model
- usage
- prompt
- metrics
- tools
- trace_context
CreateBlobStorageIntegrationRequest:
properties:
projectId:
@@ -99,6 +128,23 @@ types:
compressed:
type:optional<boolean>
docs:Enable gzip compression for exported files (.csv.gz, .json.gz, .jsonl.gz). Defaults to true.
exportSource:
type:optional<BlobStorageExportSource>
docs:|
Data to export. When omitted on update, the existing value is preserved. When omitted on create: pre-cutoff Cloud projects and self-hosted deployments fall back to `LEGACY_TRACES_OBSERVATIONS`; post-cutoff Cloud projects (created on or after 2026-05-20) auto-default to `OBSERVATIONS_V2`. Required when `exportFieldGroups` is provided.
**Cloud-only deprecation gate (effective 2026-05-20):** For projects created on or after 2026-05-20 on Langfuse Cloud, `LEGACY_TRACES_OBSERVATIONS` and `LEGACY_TRACES_AND_ENRICHED_OBSERVATIONS` are rejected with HTTP 400. Omitting `exportSource` on these projects silently defaults to `OBSERVATIONS_V2` rather than the schema column default. Use `OBSERVATIONS_V2` for all new integrations. Projects created before 2026-05-20 and self-hosted deployments are unaffected.
exportFieldGroups:
type:optional<list<BlobStorageExportFieldGroup>>
docs:|
Field groups to include in each exported row.
For exportSource `OBSERVATIONS_V2` or `LEGACY_TRACES_AND_ENRICHED_OBSERVATIONS`: must include `core` if provided. When omitted on create, the column default (all groups) applies. When omitted on update, the existing value is preserved.
For exportSource `LEGACY_TRACES_OBSERVATIONS`: this field must be omitted or null. Sending an array (including an empty array) returns 400, because that source uses a fixed column set and does not honor field groups.
`exportFieldGroups` requires `exportSource` to be provided in the same request.
BlobStorageIntegrationResponse:
properties:
@@ -117,6 +163,11 @@ types:
exportMode:BlobStorageExportMode
exportStartDate:nullable<datetime>
compressed:boolean
exportSource:BlobStorageExportSource
exportFieldGroups:
type:nullable<list<BlobStorageExportFieldGroup>>
docs:|
Field groups included in each exported row for `OBSERVATIONS_V2` / `LEGACY_TRACES_AND_ENRICHED_OBSERVATIONS` sources. Always `null` when exportSource is `LEGACY_TRACES_OBSERVATIONS` (the field does not apply to that source; any legacy DB value is hidden from the public surface).
nextSyncAt:nullable<datetime>
lastSyncAt:nullable<datetime>
lastError:nullable<string>
@@ -138,8 +189,8 @@ types:
- `up_to_date` — all available data has been exported; next export is scheduled for the future
**ETL usage**: poll this endpoint and check for `up_to_date` status. Compare `lastSyncAt` against your
ETL bookmark to determine if new data is available. Note that exports run with a 30-minute lag buffer,
so `lastSyncAt` will always be at least 30 minutes behind real-time.
ETL bookmark to determine if new data is available. Note that exports run with a 20-minute lag buffer,
so `lastSyncAt` will always be at least 20 minutes behind real-time.
docs:The name of the pricing tier applied to this observation's usage costs
# Prompt fields (field group: prompt)
promptId:
@@ -318,10 +321,30 @@ types:
type:optional<nullable<double>>
docs:The time to first token in seconds
# Enrichment fields (added by server-side enrichment)
# Enrichment fields (always present on v2 responses, null when `model` field group not requested)
modelId:
type:nullable<string>
docs:The matched model ID. Null when the `model` field group is not requested.
inputPrice:
type:nullable<string>
docs:The input token price (USD per unit) from the matched model, serialized as a decimal string (e.g. "0.0001"). Null when the `model` field group is not requested.
outputPrice:
type:nullable<string>
docs:The output token price (USD per unit) from the matched model, serialized as a decimal string (e.g. "0.0001"). Null when the `model` field group is not requested.
totalPrice:
type:nullable<string>
docs:The total token price (USD per unit) from the matched model, serialized as a decimal string (e.g. "0.0001"). Null when the `model` field group is not requested.
docs:Reference a score config on a score. The unique langfuse identifier of a score config. When passing this field, the dataType and stringValue fields are automatically populated.
source:
type:optional<CreateScoreSource>
docs:The source of the score. Defaults to API. Set to ANNOTATION to prefill scores (e.g. from an LLM) for a human reviewer to verify in an annotation queue. When source is ANNOTATION, a configId is required unless dataType is CORRECTION. EVAL is reserved for internal evaluator outputs and is not accepted on this endpoint.
examples:
- value:
name:"novelty"
@@ -90,6 +93,22 @@ types:
value:"Great explanation of the concept"
dataType:"TEXT"
traceId:"cdef-1234-5678-90ab"
- value:
name:"accuracy"
value:0.9
dataType:"NUMERIC"
configId:"9203-4567-89ab-cdef"
traceId:"cdef-1234-5678-90ab"
source:"ANNOTATION"
queueId:"aq-1234-5678-90ab-cdef"
CreateScoreSource:
docs:|
Source values accepted when creating a score via the public REST API.
EVAL is reserved for internal evaluator outputs and is intentionally not
exposed here — use commons.ScoreSource when reading scores.
- `promptVersion` (number) - Associated prompt version
### Structured Data
- `input` (string) - Observation input. Supports accelerated token search with the `matches` operator.
- `output` (string) - Observation output. Supports accelerated token search with the `matches` operator.
- `metadata` (stringObject/numberObject/categoryOptions) - Metadata key-value pairs. Use `key` parameter to filter on specific metadata keys.
The `matches` operator is only supported for `input`, `output`, and stringObject `metadata` filters. It performs token-based full-text search using the events table text indexes. Case sensitivity differs by target: `input` and `output` matches are case-insensitive, while metadata value matches are case-sensitive. Use `contains` for substring semantics. Any v2 `input` or `output` filter must be accompanied by at least one `=` or `matches` filter on `input` or `output`; standalone `contains`, `starts with`, `ends with`, and `does not contain` filters on these columns are rejected.
docs:"Name of the score config. Max 35 characters. Only letters, numbers, underscores, spaces, periods, parentheses, and hyphens are allowed."
dataType:commons.ScoreConfigDataType
categories:
type:optional<list<commons.ConfigCategory>>
@@ -75,7 +77,7 @@ types:
docs:The status of the score config showing if it is archived or not
name:
type:optional<string>
docs:The name of the score config
docs:"Name of the score config. Max 35 characters. Only letters, numbers, underscores, spaces, periods, parentheses, and hyphens are allowed."
categories:
type:optional<list<commons.ConfigCategory>>
docs:Configure custom categories for categorical scores. Pass a list of objects with `label` and `value` properties. Categories are autogenerated for boolean configs and cannot be passed
docs:Limit of items per page. If you encounter api issues due to too large page sizes, try to reduce the limit.
docs:Limit of items per page. Maximum 100. Defaults to 50. Requests with a limit greater than 100 return HTTP 400. If you encounter api issues due to too large page sizes, try to reduce the limit.
userId:
type:optional<string>
docs:Retrieve only scores with this userId associated to the trace.
The unstable public API currently supports only LLM-as-a-judge evaluators.
enum:
- llm_as_judge
EvaluatorScope:
docs:|
Where an evaluator comes from.
- `project`:created in your project
- `managed`:provided by Langfuse
enum:
- project
- managed
EvaluationRuleTarget:
docs:|
The ingestion object type that should trigger evaluation runs.
Choose the target first, because it changes both the valid filter columns and the valid variable-mapping sources:
- `observation` evaluates live-ingested observations such as generations, spans, and events.
It supports mapping from `input`, `output`, and `metadata`.
- `experiment` evaluates live experiment executions and can additionally map `expected_output` and `experiment_item_metadata`.
It currently supports filtering by `datasetId`.
Discover valid dataset IDs with `GET /api/public/v2/datasets`, then use the returned dataset `id` values in your filter.
enum:
- observation
- experiment
EvaluationRuleStatus:
docs:|
Effective runtime status of the evaluation rule.
- `active`:enabled and currently runnable.
- `inactive`:disabled by configuration.
- `paused`:enabled, but Langfuse has blocked execution until the underlying issue is resolved.
enum:
- active
- inactive
- paused
EvaluationRuleMappingSource:
docs:|
Source field used to populate a prompt variable.
Use these values when mapping evaluator prompt variables to live data.
Target-specific rules:
- `target=observation` supports `input`, `output`, and `metadata`
- `target=experiment` supports `input`, `output`, `metadata`, `expected_output`, and `experiment_item_metadata`
Source semantics:
- `input`:the observation or experiment input payload
- `output`:the observation or experiment output payload
- `metadata`:the metadata object for the target. Combine with `jsonPath` when you need one nested field instead of the whole object.
- `expected_output`:the experiment item's expected output. Only valid for `target=experiment`.
- `experiment_item_metadata`:the experiment item's metadata object. Only valid for `target=experiment`.
enum:
- input
- output
- metadata
- expected_output
- experiment_item_metadata
EvaluatorModelConfig:
docs:|
Optional explicit model configuration for an evaluator.
If omitted, Langfuse uses the project's default evaluation model.
If provided, the model must be available to the project when the evaluator or evaluation rule is enabled.
To discover valid configured `provider` values for a project, call `GET /api/public/llm-connections` and read the `provider` field from the returned connections.
Use a `provider` value that matches one of the connections already configured in the same project.
Recovery guidance:
- If evaluator creation returns `422` with `code=evaluator_preflight_failed`, either provide a valid explicit `modelConfig` here or configure the project's default evaluation model, then retry the same request.
properties:
provider:
type:string
docs:|
Provider identifier to use for this evaluator, for example `openai` or `anthropic`.
To discover valid values for the current project, call `GET /api/public/llm-connections` and use one of the returned `provider` values.
model:
type:string
docs:Model identifier exposed by the provider, for example `gpt-4.1-mini`.
examples:
- value:
provider:openai
model:gpt-4.1-mini
EvaluatorOutputDataType:
docs:|
Structured score type returned by an evaluator.
This controls the type of score value Langfuse stores for evaluation results:
- `NUMERIC`:a numeric score such as `0.82`
- `BOOLEAN`:a boolean score such as `true`
- `CATEGORICAL`:one or more category labels from a fixed list
enum:
- NUMERIC
- BOOLEAN
- CATEGORICAL
EvaluatorOutputFieldDefinition:
properties:
description:
type:string
docs:Human-readable instructions for what the evaluator should return in this field.
EvaluatorOutputDefinition:
docs:|
Structured output definition to send when creating an evaluator.
Agent guidance:
- `dataType` is required.
- Do not send `version`; that is an internal storage detail and is not part of the public request contract.
- For `NUMERIC` and `BOOLEAN`, provide `reasoning.description` and `score.description`.
- For `CATEGORICAL`, also provide `score.categories` and `score.shouldAllowMultipleMatches`.
discriminant:dataType
union:
NUMERIC:
type:PublicNumericEvaluatorOutputDefinition
BOOLEAN:
type:PublicBooleanEvaluatorOutputDefinition
CATEGORICAL:
type:PublicCategoricalEvaluatorOutputDefinition
examples:
- name:Numeric
value:
dataType:NUMERIC
reasoning:
description:Explain why the answer is correct or incorrect.
score:
description:Return a score between 0 and 1.
- name:Boolean
value:
dataType:BOOLEAN
reasoning:
description:Explain why the output satisfies the requirement.
score:
description:Return true if the output satisfies the requirement, otherwise false.
- name:Categorical
value:
dataType:CATEGORICAL
reasoning:
description:Explain which category best fits the output.
Evaluator output definition returned by the public API.
This response always includes `dataType` and never includes an internal output-definition `version`.
Legacy stored evaluator definitions are normalized into this shape before they are returned.
Use this response shape when deciding how to interpret future evaluation scores:
- `NUMERIC`:expect numeric score values
- `BOOLEAN`:expect `true` / `false`
- `CATEGORICAL`:expect one or more values from `score.categories`
discriminant:dataType
union:
NUMERIC:
type:PublicNumericEvaluatorOutputDefinition
BOOLEAN:
type:PublicBooleanEvaluatorOutputDefinition
CATEGORICAL:
type:PublicCategoricalEvaluatorOutputDefinition
examples:
- name:PublicNumeric
value:
dataType:NUMERIC
reasoning:
description:Explain why the answer is correct or incorrect.
score:
description:Return a score between 0 and 1.
- name:PublicCategorical
value:
dataType:CATEGORICAL
reasoning:
description:Explain which label best fits the output.
score:
description:Choose the best label.
categories:
- correct
- partially_correct
- incorrect
shouldAllowMultipleMatches:false
EvaluationRuleStringFilterOperator:
enum:
- name:Equals
value:"="
- name:Contains
value:contains
- name:DoesNotContain
value:does not contain
- name:StartsWith
value:starts with
- name:EndsWith
value:ends with
EvaluationRuleNumberFilterOperator:
enum:
- name:Equals
value:"="
- name:GreaterThan
value:">"
- name:LessThan
value:"<"
- name:GreaterThanOrEqual
value:">="
- name:LessThanOrEqual
value:"<="
EvaluationRuleOptionsFilterOperator:
enum:
- name:AnyOf
value:any of
- name:NoneOf
value:none of
EvaluationRuleArrayOptionsFilterOperator:
enum:
- name:AnyOf
value:any of
- name:NoneOf
value:none of
- name:AllOf
value:all of
EvaluationRuleBooleanFilterOperator:
enum:
- name:Equals
value:"="
- name:NotEquals
value:"<>"
EvaluationRuleNullFilterOperator:
enum:
- name:IsNull
value:is null
- name:IsNotNull
value:is not null
DateTimeEvaluationRuleFilter:
properties:
column:
type:string
docs:Column to filter on.
operator:
type:EvaluationRuleNumberFilterOperator
docs:Comparison operator for datetime values.
value:
type:datetime
docs:Datetime value to compare against.
StringEvaluationRuleFilter:
properties:
column:
type:string
docs:Column to filter on.
operator:
type:EvaluationRuleStringFilterOperator
value:
type:string
NumberEvaluationRuleFilter:
properties:
column:
type:string
docs:Column to filter on.
operator:
type:EvaluationRuleNumberFilterOperator
value:
type:double
StringOptionsEvaluationRuleFilter:
properties:
column:
type:string
docs:Column to filter on.
operator:
type:EvaluationRuleOptionsFilterOperator
value:
type:list<string>
docs:One or more allowed string values.
ArrayOptionsEvaluationRuleFilter:
properties:
column:
type:string
docs:Column to filter on.
operator:
type:EvaluationRuleArrayOptionsFilterOperator
value:
type:list<string>
docs:One or more array elements to match.
StringObjectEvaluationRuleFilter:
properties:
column:
type:string
docs:Object-valued column to filter on. In the unstable public API this is currently `metadata`.
key:
type:string
docs:Top-level key inside the object-valued column to filter on.
operator:
type:EvaluationRuleStringFilterOperator
value:
type:string
NumberObjectEvaluationRuleFilter:
properties:
column:
type:string
docs:Object-valued column to filter on.
key:
type:string
docs:Key inside the object-valued column to filter on.
operator:
type:EvaluationRuleNumberFilterOperator
value:
type:double
CategoryOptionsEvaluationRuleFilter:
properties:
column:
type:string
docs:Object-valued column to filter on.
key:
type:string
docs:Key inside the object-valued column to filter on.
operator:
type:EvaluationRuleOptionsFilterOperator
value:
type:list<string>
BooleanEvaluationRuleFilter:
properties:
column:
type:string
docs:Column to filter on.
operator:
type:EvaluationRuleBooleanFilterOperator
value:
type:boolean
NullEvaluationRuleFilter:
properties:
column:
type:string
docs:Column to filter on. In the unstable public API this is currently `parentObservationId`.
operator:
type:EvaluationRuleNullFilterOperator
value:
type:optional<string>
docs:Ignored placeholder value. Clients may omit it or send an empty string.
EvaluationRuleMapping:
docs:|
Maps one evaluator prompt variable to one source field from the target object.
How to build a valid mapping list:
1. Create the evaluator or fetch it with `GET /evaluators/{id}`.
2. Read the evaluator `variables` array.
3. Add exactly one mapping object for each variable in that array.
4. Use the variable name exactly as returned, without braces such as `{{` or `}}`.
5. Choose a `source` that is valid for the selected `target`.
`jsonPath` is optional. Use it only when the selected source is a JSON object and you want to extract one nested field before inserting it into the evaluator prompt.
Recovery guidance:
- `invalid_variable_mapping`:the variable name is unknown for this evaluator, or the selected `source` is not valid for the chosen `target`
- `missing_variable_mapping`:one or more evaluator variables are not mapped yet
- `duplicate_variable_mapping`:the same evaluator variable appears more than once
- `invalid_json_path`:the JSONPath expression is malformed. Remove it or correct it.
properties:
variable:
type:string
docs:|
Prompt variable name without braces.
Example:for the prompt `Judge {{input}} against {{output}}`, use `input` and `output`.
source:
type:EvaluationRuleMappingSource
docs:|
Source field that should populate the prompt variable.
Use dataset `id` values from `GET /api/public/v2/datasets`, not dataset names.
Recovery guidance:
- `invalid_filter_value` with `details.column` but no `invalidValues`:the selected `column` is not supported for the chosen `target`
- `invalid_filter_value` with `details.invalidValues`:the selected values are not allowed for that column. Replace them with one of `details.allowedValues` when provided.
- `invalid_filter_value` for `column=datasetId`:call `GET /api/public/v2/datasets`, then retry with dataset `id` values from that response.
message:'Filter column "datasetId" contains dataset id(s) that do not exist in this project: dataset-prod'
code:invalid_filter_value
details:
field:filter[0].value
column:datasetId
invalidValues:
- dataset-prod
- name:EvaluatorPreflightFailed
value:
message:'No valid LLM model found for evaluator "answer-correctness". No default model or custom model configured for project project_123'
code:evaluator_preflight_failed
details:
evaluatorName:answer-correctness
provider:null
model:null
- name:MissingVariableMapping
value:
message:"Missing mappings for evaluator variable(s): output"
code:missing_variable_mapping
details:
field:mapping
variables:
- output
- name:InvalidJsonPath
value:
message:'Mapping for variable "customer_tier" has an invalid jsonPath "$[". JSONPath expressions must use balanced quotes, brackets, and parentheses.'
code:invalid_json_path
details:
field:mapping[0].jsonPath
variable:customer_tier
value:"$["
- name:NameConflict
value:
message:'An evaluation rule named "answer-quality-live" already exists in this project. Use PATCH /api/public/unstable/evaluation-rules/erule_123 to update it instead of creating a duplicate.'
code:name_conflict
details:
field:name
- name:ActiveEvaluationRuleLimit
value:
message:This project already has the maximum number of active evaluation rules (50). Disable an existing active evaluation rule before enabling another one.
code:conflict
details:
limit:50
- name:RateLimited
value:
message:Rate limit exceeded
code:rate_limited
details:
retryAfterSeconds:60
limit:1000
remaining:0
resetAt:"2026-03-30T10:00:00.000Z"
errors:
BadRequestError:
docs:|
Request parsing or validation failed.
Typical unstable-eval examples:
- malformed request body or query parameters
- unsupported filter value
- invalid JSONPath selector
- missing or duplicate variable mappings
Recovery guidance:
- read `details.issues` for malformed bodies or queries
- read `details.column`, `details.invalidValues`, and `details.allowedValues` for filter problems
- for `details.column=datasetId`, call `GET /api/public/v2/datasets` and retry with dataset `id` values from that response
- read `details.variable` or `details.variables` for mapping problems
status-code:400
type:PublicApiError
UnauthorizedError:
docs:|
Authentication failed.
Typical unstable-eval examples:
- the API key or secret key is invalid
- the request uses the wrong host or stale credentials
status-code:401
type:PublicApiError
AccessDeniedError:
docs:|
Authenticated, but the caller is not allowed to access the requested resource.
Typical unstable-eval examples:
- using an organization-scoped key against project-scoped evaluator endpoints
- using a valid key that lacks the required access level for this resource
status-code:403
type:PublicApiError
NotFoundError:
docs:The requested evaluator or evaluation rule does not exist within the authorized project.
status-code:404
type:PublicApiError
MethodNotAllowedError:
docs:The HTTP method is not supported for this endpoint.
status-code:405
type:PublicApiError
ConflictError:
docs:|
The request conflicts with existing state.
Typical unstable-eval examples:
- evaluator version changed during concurrent creation
- an evaluation rule name already exists in the project, so the caller should PATCH the existing resource instead
- enabling another evaluation rule would exceed the project limit of 50 active evaluation rules
- the underlying state changed between read and write operations, so the client should retry
Recovery guidance:
- for `name_conflict`, PATCH the existing evaluation rule instead of creating another one
- for the active-limit conflict, disable another active evaluation rule before enabling a new one
status-code:409
type:PublicApiError
UnprocessableContentError:
docs:|
The request is syntactically valid, but Langfuse cannot accept it in its current form.
The most important unstable-eval case is `code=evaluator_preflight_failed`, which means the evaluator cannot currently run with the resolved model configuration.
Recovery guidance:
- configure the project's default evaluation model, or send a valid explicit evaluator `modelConfig`
- once the model configuration is valid, retry the same request
status-code:422
type:PublicApiError
TooManyRequestsError:
docs:The project is rate limited. Retry after the interval described in the response headers and error details.
An evaluation rule defines **what** incoming data should be evaluated and **how prompt variables should be populated** from that data.
Use this resource after choosing an evaluator from the evaluator endpoints.
Key rules:
- `name` must be unique within the project for public evaluation rules
- `target` must be `observation` or `experiment`
- `evaluator.name` + `evaluator.scope` must identify an existing evaluator family returned by the evaluator endpoints
- Langfuse resolves that family to its latest version before saving the evaluation rule
- for `target=experiment`, use dataset `id` values from `GET /api/public/v2/datasets` when filtering by `datasetId`
- every evaluator prompt variable must be mapped exactly once
- `expected_output` and `experiment_item_metadata` mappings are only valid for `target=experiment`
- if `enabled=true`, Langfuse validates that the referenced evaluator can currently run
- at most 50 evaluation rules can be effectively active in one project at the same time
If an evaluation rule with the same `name` already exists in the project, the API returns `409`.
In that case, update the existing resource with `PATCH /api/public/unstable/evaluation-rules/{evaluationRuleId}` instead of creating a second one.
If enabling this resource would exceed the 50-active limit, the API also returns `409`.
In that case, disable or pause another active evaluation rule before enabling a new one.
Current scope:
- evaluation rules are live-ingestion rules only
- they do not trigger historical backfills
Recovery guidance:
- `400 invalid_filter_value`:fix the filter `column` or `value` using `details.column`, `details.invalidValues`, and `details.allowedValues`
- `400 invalid_filter_value` with `details.column=datasetId`:call `GET /api/public/v2/datasets`, then retry with dataset `id` values from that response
- `400 missing_variable_mapping`:fetch the evaluator again and make sure every variable in `variables` appears exactly once in `mapping`
- `400 duplicate_variable_mapping`:remove repeated mappings for the same variable
- `400 invalid_variable_mapping`:switch to a valid `source` for the selected `target`, or fix the variable name
- `400 invalid_json_path`:remove or correct the `jsonPath`
- `422 evaluator_preflight_failed`:the selected evaluator cannot run with the resolved model configuration. Fix the evaluator/default model setup, then retry the create request.
method:POST
path:/evaluation-rules
request:CreateEvaluationRuleRequest
response:EvaluationRule
errors:
- errors.BadRequestError
- errors.UnauthorizedError
- errors.AccessDeniedError
- errors.NotFoundError
- errors.ConflictError
- errors.MethodNotAllowedError
- errors.UnprocessableContentError
- errors.TooManyRequestsError
- errors.InternalServerError
examples:
- name:CreateObservationEvaluationRule
docs:Deploy an evaluator to score live generations only.
request:
name:answer-correctness-live
evaluator:
name:answer-correctness
scope:project
target:observation
enabled:true
sampling:1
filter:
- type:stringOptions
column:type
operator:any of
value:
- GENERATION
mapping:
- variable:input
source:input
- variable:output
source:output
response:
body:
id:erule_123
name:answer-correctness-live
evaluator:
id:evaltmpl_123
name:answer-correctness
scope:project
target:observation
enabled:true
status:active
pausedReason:null
pausedMessage:null
sampling:1
filter:
- type:stringOptions
column:type
operator:any of
value:
- GENERATION
mapping:
- variable:input
source:input
- variable:output
source:output
createdAt:"2026-03-30T09:20:00.000Z"
updatedAt:"2026-03-30T09:20:00.000Z"
- name:CreateExperimentEvaluationRule
docs:Deploy an evaluator to compare experiment outputs against expected outputs. Discover valid dataset IDs with `GET /api/public/v2/datasets` first.
request:
name:experiment-expected-output-match
evaluator:
name:expected-output-match
scope:project
target:experiment
enabled:true
sampling:0.5
filter:
- type:stringOptions
column:datasetId
operator:any of
value:
- "550e8400-e29b-41d4-a716-446655440000"
mapping:
- variable:output
source:output
- variable:expected_output
source:expected_output
response:
body:
id:erule_456
name:experiment-expected-output-match
evaluator:
id:evaltmpl_456
name:expected-output-match
scope:project
target:experiment
enabled:true
status:active
pausedReason:null
pausedMessage:null
sampling:0.5
filter:
- type:stringOptions
column:datasetId
operator:any of
value:
- "550e8400-e29b-41d4-a716-446655440000"
mapping:
- variable:output
source:output
- variable:expected_output
source:expected_output
createdAt:"2026-03-30T09:30:00.000Z"
updatedAt:"2026-03-30T09:30:00.000Z"
list:
docs:|
List evaluation rules in the authenticated project.
Each item describes one live evaluation rule and its effective runtime status.
method:GET
path:/evaluation-rules
request:
name:ListEvaluationRulesRequest
query-parameters:
page:
type:optional<integer>
docs:1-based page number. Defaults to `1`.
limit:
type:optional<integer>
docs:Maximum number of items per page. Defaults to `50`.
response:EvaluationRules
errors:
- errors.BadRequestError
- errors.UnauthorizedError
- errors.AccessDeniedError
- errors.MethodNotAllowedError
- errors.TooManyRequestsError
- errors.InternalServerError
get:
docs:|
Get one evaluation rule by its identifier.
Use this endpoint to inspect the current evaluator, target, mapping, filters, and effective runtime status.
method:GET
path:/evaluation-rules/{evaluationRuleId}
path-parameters:
evaluationRuleId:
type:string
docs:Evaluation rule identifier returned by the evaluation rule endpoints.
response:EvaluationRule
errors:
- errors.BadRequestError
- errors.UnauthorizedError
- errors.AccessDeniedError
- errors.NotFoundError
- errors.MethodNotAllowedError
- errors.TooManyRequestsError
- errors.InternalServerError
update:
docs:|
Update an evaluation rule.
Typical uses:
- enable or disable live execution
- switch to another evaluator
- adjust sampling
- change filters
- update variable mappings
Important behavior:
- provide only the fields you want to change
- if you provide `evaluator`, Langfuse resolves that evaluator family to its latest version before saving
- changing `target`, `filter`, or `mapping` must still produce a valid target-specific configuration
- if you change `target`, also send a compatible `filter` and `mapping` in the same request unless the existing ones are still valid for the new target
- if the resulting config is enabled, Langfuse re-validates that the selected evaluator can run
- if the update would move a non-active evaluation rule into the active state and the project already has 50 active evaluation rules, the API returns `409`
Recovery guidance:
- if the update fails with `missing_variable_mapping` or `invalid_variable_mapping` after changing `evaluator` or `target`, resend the request with a complete new `mapping`
- if the update fails with `invalid_filter_value` after changing `target`, resend the request with a target-compatible `filter`
method:PATCH
path:/evaluation-rules/{evaluationRuleId}
path-parameters:
evaluationRuleId:
type:string
docs:Evaluation rule identifier.
request:UpdateEvaluationRuleRequest
response:EvaluationRule
errors:
- errors.BadRequestError
- errors.UnauthorizedError
- errors.AccessDeniedError
- errors.NotFoundError
- errors.MethodNotAllowedError
- errors.UnprocessableContentError
- errors.TooManyRequestsError
- errors.InternalServerError
delete:
docs:|
Delete an evaluation rule.
This removes the live-ingestion rule only. It does not delete the referenced evaluator.
method:DELETE
path:/evaluation-rules/{evaluationRuleId}
path-parameters:
evaluationRuleId:
type:string
docs:Evaluation rule identifier.
response:DeleteEvaluationRuleResponse
errors:
- errors.BadRequestError
- errors.UnauthorizedError
- errors.AccessDeniedError
- errors.NotFoundError
- errors.MethodNotAllowedError
- errors.TooManyRequestsError
- errors.InternalServerError
types:
EvaluationRule:
docs:|
Live evaluation rule for incoming data.
An evaluation rule answers:
- which evaluator should be used
- which target objects should trigger scoring
- how often scoring should run
- which target fields should populate each evaluator variable
- whether the deployment is active, inactive, or paused
Important status semantics:
- `enabled` is the desired on/off setting from the client
- `status` is the effective runtime state after Langfuse applies validation and blocking rules
- `enabled=true` with `status=paused` means the rule should run, but Langfuse has paused it until the underlying problem is fixed
properties:
id:
type:string
docs:Stable evaluation rule identifier.
name:
type:string
docs:Human-readable deployment name. This is independent from the evaluator name.
evaluator:
type:EvaluationRuleEvaluator
docs:|
Evaluator currently used by this rule.
`name` and `scope` identify the evaluator family conceptually.
`id` is the currently active evaluator version in that family.
If you create a newer project version with the same evaluator name later, existing evaluation rules are moved to it automatically.
target:
type:commons.EvaluationRuleTarget
docs:Target object type that should trigger scoring.
enabled:
type:boolean
docs:Desired enabled state configured by the client.
status:
type:commons.EvaluationRuleStatus
docs:Effective runtime status after Langfuse applies validation and blocking rules.
pausedReason:
type:nullable<string>
docs:Machine-readable reason when `status=paused`, otherwise `null`.
pausedMessage:
type:nullable<string>
docs:Human-readable explanation when `status=paused`, otherwise `null`.
sampling:
type:double
docs:|
Fraction of matching target objects that should be evaluated.
Must be greater than `0` and less than or equal to `1`.
- `1` means evaluate every matching target.
- `0.25` means evaluate approximately 25% of matching targets.
filter:
type:list<commons.EvaluationRuleFilter>
docs:List of filter conditions used to decide whether a target should be evaluated.
mapping:
type:list<commons.EvaluationRuleMapping>
docs:Variable mappings used to populate the evaluator prompt from the live target object.
createdAt:
type:datetime
docs:Timestamp when the evaluation rule was created.
updatedAt:
type:datetime
docs:Timestamp when the evaluation rule was last updated.
examples:
- value:
id:erule_123
name:answer-correctness-live
evaluator:
id:evaltmpl_123
name:answer-correctness
scope:project
target:observation
enabled:true
status:active
pausedReason:null
pausedMessage:null
sampling:1
filter:
- type:stringOptions
column:type
operator:any of
value:
- GENERATION
mapping:
- variable:input
source:input
- variable:output
source:output
createdAt:"2026-03-30T09:20:00.000Z"
updatedAt:"2026-03-30T09:20:00.000Z"
EvaluationRules:
docs:Paginated list of evaluation rules.
properties:
data:
type:list<EvaluationRule>
docs:Evaluation rules in the current page.
meta:
type:pagination.MetaResponse
docs:Standard pagination metadata.
CreateEvaluationRuleRequest:
docs:|
Request body for creating an evaluation rule.
Checklist for agents and SDK clients:
- reference an existing evaluator family by `evaluator.name` and `evaluator.scope`
- choose `target=observation` or `target=experiment`
- if `target=experiment` and you want a dataset filter, call `GET /api/public/v2/datasets` first and use dataset `id` values in `filter[].value`
- fetch or inspect the evaluator first, then provide a complete variable mapping for every evaluator variable listed in `variables`
- optionally narrow execution with `filter`
- set `enabled=true` only when you want live execution immediately
properties:
name:
type:string
docs:Human-readable deployment name.
evaluator:
type:EvaluationRuleEvaluatorReference
docs:|
Evaluator family to use.
Use `name` and `scope` from the evaluator endpoints.
Langfuse resolves that family to its latest version before saving the rule.
target:
type:commons.EvaluationRuleTarget
docs:Target object type to evaluate.
enabled:
type:boolean
docs:Whether the deployment should be active immediately after creation.
sampling:
type:optional<double>
docs:Optional sampling fraction. Defaults to `1`.
filter:
type:optional<list<commons.EvaluationRuleFilter>>
docs:|
Optional filter list.
Omit or pass an empty list to evaluate all matching targets for the selected `target`.
Each filter object must use a column that is valid for that `target`.
For `target=experiment`, `column=datasetId` expects dataset `id` values from `GET /api/public/v2/datasets`, not dataset names.
mapping:
type:list<commons.EvaluationRuleMapping>
docs:|
Required variable mappings.
Every evaluator variable must appear exactly once.
Build this list from the evaluator `variables` array returned by the evaluator endpoints.
UpdateEvaluationRuleRequest:
docs:|
Partial update body for an evaluation rule.
Provide only the fields you want to change.
An empty body is rejected.
Practical guidance:
- If you only want to rename the rule or change sampling, send just those fields.
- If you change `evaluator`, send a fresh `mapping` unless you are certain the existing mapping still matches the evaluator variables.
- If you change `target`, usually send both `filter` and `mapping` in the same request.
- If you change an experiment `datasetId` filter, call `GET /api/public/v2/datasets` and use dataset `id` values from that response.
properties:
name:
type:optional<string>
docs:Updated deployment name.
evaluator:
type:optional<EvaluationRuleEvaluatorReference>
docs:|
Updated evaluator family.
Langfuse resolves the provided evaluator family to its latest version before saving the rule.
target:
type:optional<commons.EvaluationRuleTarget>
docs:Updated target object type.
enabled:
type:optional<boolean>
docs:Updated desired enabled state.
sampling:
type:optional<double>
docs:Updated sampling fraction.
filter:
type:optional<list<commons.EvaluationRuleFilter>>
docs:|
Updated filter list.
For `target=experiment`, `column=datasetId` expects dataset `id` values from `GET /api/public/v2/datasets`, not dataset names.
Use evaluators to define **how** Langfuse should score data:the prompt, the expected structured output, and the optional model configuration.
Naming behavior:
- If this is a new evaluator name in your project, Langfuse creates version `1`.
- If the name already exists in your project, Langfuse creates the next version and returns it.
- When a new project version is created, existing evaluation rules in that project automatically move to the newest version for that evaluator name.
Recommended workflow:
1. Create the evaluator.
2. Read the returned `variables` array.
3. Read the returned `outputDefinition.dataType` so the client knows whether future scores will be numeric, boolean, or categorical.
4. Create one or more evaluation rules that reference the returned evaluator family using `name` and `scope`.
Recovery guidance:
- `422` with `code=evaluator_preflight_failed`:the evaluator cannot run with the resolved model configuration. Add a valid explicit `modelConfig`, or configure the project's default evaluation model, then retry the same request.
- `400` with `code=invalid_body`:the request shape is malformed. Use the structured `details.issues` array to fix the specific fields and retry.
- `400` with `code=invalid_body` on `outputDefinition`:send `dataType`, `reasoning.description`, and `score.description`. Do not send `version`; it is not part of the public request shape.
Unstable API note:
- This surface may evolve while the underlying evaluation data model is being redesigned.
method:POST
path:/evaluators
request:CreateEvaluatorRequest
response:Evaluator
errors:
- errors.BadRequestError
- errors.UnauthorizedError
- errors.AccessDeniedError
- errors.MethodNotAllowedError
- errors.ConflictError
- errors.UnprocessableContentError
- errors.TooManyRequestsError
- errors.InternalServerError
examples:
- name:CreateEvaluatorVersion
docs:Create a new version of an evaluator named `answer-correctness`.
request:
name:answer-correctness
prompt:|
You are grading an answer.
Input:
{{input}}
Output:
{{output}}
Return a score between 0 and 1.
outputDefinition:
dataType:NUMERIC
reasoning:
description:Explain why the score was assigned.
score:
description:Correctness score between 0 and 1.
modelConfig:
provider:openai
model:gpt-4.1-mini
response:
body:
id:evaltmpl_123
name:answer-correctness
version:2
scope:project
type:llm_as_judge
prompt:|
You are grading an answer.
Input:
{{input}}
Output:
{{output}}
Return a score between 0 and 1.
variables:
- input
- output
outputDefinition:
dataType:NUMERIC
reasoning:
description:Explain why the score was assigned.
score:
description:Correctness score between 0 and 1.
modelConfig:
provider:openai
model:gpt-4.1-mini
evaluationRuleCount:0
createdAt:"2026-03-30T09:00:00.000Z"
updatedAt:"2026-03-30T09:00:00.000Z"
list:
docs:|
List the evaluators available to the authenticated project.
Important behavior:
- This endpoint returns the latest version of each available evaluator.
- Results can include evaluators from your project and Langfuse-managed evaluators.
- If the same evaluator name exists in both places, both are returned as separate items with different `scope` values.
method:GET
path:/evaluators
request:
name:ListEvaluatorsRequest
query-parameters:
page:
type:optional<integer>
docs:1-based page number. Defaults to `1`.
limit:
type:optional<integer>
docs:Maximum number of items per page. Defaults to `50`.
response:Evaluators
errors:
- errors.BadRequestError
- errors.UnauthorizedError
- errors.AccessDeniedError
- errors.MethodNotAllowedError
- errors.TooManyRequestsError
- errors.InternalServerError
get:
docs:|
Get one evaluator by `id`.
Use this endpoint when you want the prompt, output definition, model configuration, and derived variables for the evaluator you plan to use in an evaluation rule.
method:GET
path:/evaluators/{evaluatorId}
path-parameters:
evaluatorId:
type:string
docs:Evaluator identifier returned by the evaluator endpoints.
response:Evaluator
errors:
- errors.BadRequestError
- errors.UnauthorizedError
- errors.AccessDeniedError
- errors.NotFoundError
- errors.MethodNotAllowedError
- errors.TooManyRequestsError
- errors.InternalServerError
types:
Evaluator:
docs:|
One evaluator that can be used for scoring.
An evaluator describes **how** to score data:
- prompt
- extracted prompt variables
- output schema
- optional explicit model configuration
It does not define **which** live objects are evaluated. That is the job of `evaluation-rules`.
For agent clients, the most important fields are:
- `variables`:use these exact names when building the evaluation-rule `mapping` array
- `outputDefinition`:tells you the expected score type and the evaluator's response instructions
- `modelConfig`:tells you whether the evaluator uses the project default model (`null`) or an explicit provider/model
Versioning behavior:
- `GET /evaluators` returns the latest version of each available evaluator.
- `GET /evaluators/{id}` can return an older version.
- Evaluation rules always run against the latest version for the selected evaluator name within the same source (`project` or `managed`).
properties:
id:
type:string
docs:Identifier of this evaluator.
name:
type:string
docs:Evaluator name.
version:
type:integer
docs:Version number of this evaluator.
scope:
type:commons.EvaluatorScope
docs:"Where this evaluator comes from: your project or Langfuse-managed defaults."
type:
type:commons.EvaluatorType
docs:Evaluator engine type. Currently always `llm_as_judge`.
prompt:
type:string
docs:Prompt template used during evaluation.
variables:
type:list<string>
docs:|
Variables extracted from the evaluator prompt.
Every variable in this list must be mapped exactly once when creating an evaluation rule.
outputDefinition:
type:commons.PublicEvaluatorOutputDefinition
docs:|
Structured output schema returned by this evaluator.
Responses always include `dataType` and omit the internal output-definition `version`.
Use `dataType` to decide how future scores should be interpreted.
modelConfig:
type:nullable<commons.EvaluatorModelConfig>
docs:Explicit model configuration, or `null` when the project default evaluation model is used.
evaluationRuleCount:
type:integer
docs:Number of evaluation rules in the project that currently use this evaluator version.
createdAt:
type:datetime
docs:Timestamp when this evaluator was created.
updatedAt:
type:datetime
docs:Timestamp when this evaluator was last updated.
Evaluators:
properties:
data:
type:list<Evaluator>
meta:
type:pagination.MetaResponse
CreateEvaluatorRequest:
docs:|
Request body for creating an evaluator.
If the same `name` already exists in your project, Langfuse creates the next version and returns it.
Existing evaluation rules in the same project are then moved to that new latest version automatically.
properties:
name:
type:string
docs:Evaluator name within the authenticated project.
prompt:
type:string
docs:Prompt template used by the evaluator.
outputDefinition:
type:commons.EvaluatorOutputDefinition
docs:|
Structured output schema the evaluator must return.
Always send `dataType`.
Do not send `version`; it is an internal storage detail and not part of the public request contract.
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
