* feat(monitors): add no-data resolution modes
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(monitors): refine no-data modes and dropdown UI
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor(monitors): reference schema enums instead of string literals
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(monitors): add triggerIds to base monitor service test input
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor(monitors): merge resolveNoDataSeverity into computeSeverity
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* style(monitors): restyle no-data SEVERITY badge to slate
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(monitors): drop unreachable fallthrough in matches() to keep exhaustiveness guard
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* ci: cut pipeline critical path in test and docker build jobs
The pipeline wall clock is gated by the slowest tests-web matrix job,
with test-docker-build second. All changes are measured on PR #14138.
tests-web (slowest matrix job 285s -> ~215s):
- vitest --maxWorkers=8: the server tests are I/O-bound, so extra
workers overlap DB waits (run tests 76-144s -> 64-86s)
- start dev containers and the golang-migrate download in the
background, overlapped by the build
- shim the clickhouse client to the binary already inside the dev
container instead of downloading the ~300MB package (~9s)
- drop the Next.js build cache: ~30s/job restore+save for ~40s of
build savings, while ~1GB churn per commit evicted caches that pay
- scope turbo cache keys by deploy mode (turbo hashes .env, which
differs per mode, so only the save-race winner ever replayed) and
prune entries older than three days to stop the archive snowballing
test-docker-build (~230s -> ~150s):
- healthcheck start_interval: containers are ready in seconds but the
first probe fired only after the 30s interval (compose wait 45s->12s)
- start dependency containers during the image build
- skip the Next.js type check inside the smoke-test image build (the
lint job type-checks already); release builds are unaffected
* test(web): run state-clean server tests without worker isolation
The web server suite spends more CI time importing the @langfuse/shared
module graph per test file than running tests (371s cumulative import
vs 264s tests). Split the server vitest project:
- "server" runs with isolate: false so each worker imports the module
graph once; its per-file teardown no longer disconnects the shared
redis/ClickHouse singletons (workers are reaped at run end)
- "server-isolated" keeps per-file isolation for files that touch
process-global state: module mocks, spies, fake timers, process.env
writes, and connection lifecycle calls (redis.disconnect,
disconnectQueues). Classification is content-based at config load so
new test files sort themselves.
Raise the teardown hook timeout to 30s: it imports the heavy shared
module graph, which can exceed the 10s default on a loaded runner.
CI effect on the gating tests-web jobs: run tests 64-86s -> 41-56s.
* ci: stop gating jobs on the duplicate-run check for non-push events
Every job waited for pre-job (runner pickup + check + dispatch), which
cost 45-100s of wall clock on every run. The duplicate-tree skip only
fires on push events in practice - 8 of the last 12 main-push runs
skipped (the merge queue had just tested the identical tree), while
pull_request and merge_group trees are almost always new.
pre-job therefore only runs for push events now; everywhere else it
resolves as skipped at run creation and jobs start immediately. The
llm-connections paths filter moves to its own non-gating job since the
filter is still needed for pull requests.
Behavior changes: merge_group runs no longer skip duplicate trees
(rare, ~1 in 10), and a pre-job API failure now fails open instead of
silently skipping all jobs.
* test(web): prebundle shared deps for the shared-context server project
vitest's ssr optimizer collapses the @langfuse/shared module graph into
prebundled chunks: cumulative import time drops 29.3s -> 0.8s locally.
Only enabled for the "server" project, whose files by definition do not
mock modules (module mocking is what prebundling can break).
* Revert "test(web): prebundle shared deps for the shared-context server project"
This reverts commit b58fefaa70c40a051949428c6c518995928a8211.
* ci: overlap setup downloads and container startup in e2e jobs
Apply the same treatment the tests-web jobs got: the golang-migrate
download, dev container startup, and (for browser e2e) the playwright
chromium install have no dependency on the build, so they run in the
background overlapped by it. The e2e jobs are co-critical with
tests-web (~170-210s) since jobs stopped waiting for pre-job.
* ci: address review findings on background-start observability and env classifier
- catch `delete process.env.X` and bracket-notation env assignments in
the global-state classifier (future-proofing; no current file affected)
- surface the docker build smoke test's background dependency startup
log and exit code instead of discarding them
* ci: fix background-start failure capture and turbo key prefix collision
Review findings from the Claude review:
- background subshells inherited the runner shell's errexit, so a
failing command aborted the subshell before the exit-code marker was
written and the waiting step burned its full timeout; set +e inside
the capture subshells
- the default deploy-mode's turbo restore prefix (mode-) also matched
the -azure and -redis-cluster keys; give the default mode an explicit
"-default" segment
* ci: fail wait steps explicitly when background markers never appear
Review follow-up: the || true + exit "$(cat marker)" shape produced a
cryptic "numeric argument required" when the outer poll itself timed
out. Emit a real error with the captured log instead, and size the
budgets for what they actually wait on (image pulls are not covered by
compose --wait-timeout; the playwright install has no inner timeout).
* test(web): catch remaining state-mutating vitest APIs in the classifier
Review follow-up: vi.resetModules, vi.setSystemTime, and the unstub
helpers also mutate worker-global state; no current file in scope is
affected.
* test(web): extend dataset run deletion propagation window
The deletion travels through the worker queue and a ClickHouse
mutation; 4x30s polls all timed out on a loaded azure-mode runner.
* test(web): isolate files that mutate the t3-env singleton
Review finding: `(env as any).X = value` mutates the env object from
env.mjs, which does not proxy process.env and so bypassed the
classifier. Three in-scope files used it while running in the shared
context; they now route to server-isolated.
* fix(fern/scores): move v3 scores definition back to scores-v3.yml
Promoting v3 to the canonical scores.yml (#14126) renamed the SDK's
scores group, which would force a major SDK release. Move the v3
definition back to its own file so it generates as a separate group.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(fern/scores): restore scores v2 as canonical scores.yml
Moves the v2 GET endpoints out of legacy/ back to the default scores
group so the generated SDK keeps its existing scores.* methods and type
names. Import paths adjusted for the new location. The deprecation
notes pointing to GET /api/public/v3/scores are kept.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(fern/scores): restore V3-suffixed identifiers in scores-v3.yml
Reverts the identifier renames from #14126 so generated names match the
pre-rename SDK surface: getMany → getManyV3, GetScoresRequest →
GetScoresV3Request, GetScoresResponse → GetScoresV3Response,
GetScoresMeta → GetScoresV3Meta, and all ScoreSubject* types →
ScoreSubject*V3. This also resolves the name collisions with the
restored v2 scores.yml, which declares GetScoresRequest and
GetScoresResponse.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* chore(generated): regenerate OpenAPI spec after scores file split
Schema names, operationIds, and SDK groups now match the pre-#14126
state (scores = v2, scoresV3 = v3); only documentation text differs.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* fix(scim): stop PUT role downgrade and audit membership mutations (INT-1249)
SCIM `active: true` (PUT and PATCH) is an idempotent "ensure provisioned"
signal, not a role assignment. The PUT handler previously upserted the
membership with `update: { role }`, where `role` defaulted to NONE whenever the
request omitted a parseable `roles` array. A periodic IdP full-sync (e.g. Okta)
that re-sends every user with `active: true` and no roles therefore overwrote
existing members' roles with NONE, silently stripping their permissions.
Changes:
- PUT/PATCH `active: true` now only create a membership (with the default NONE
role) when one does not already exist, and make no change when it does. Any
`roles` in the payload are ignored on PUT; roles are set via the create (POST)
flow or the in-app UI. Concurrent provisioning races are treated as
idempotent no-ops (P2002).
- Add audit_logs entries for SCIM membership mutations that actually change
state: a `create` entry when provisioning adds a membership, and a `delete`
entry when deprovisioning (PUT/PATCH active:false, DELETE) removes one. No-ops
do not write audit entries. This closes INT-1249, where PUT/PATCH/DELETE
membership changes bypassed the audit log entirely.
Adds regression tests asserting an existing member's role is preserved on
`active: true`, and that create/delete audit entries are written only on real
changes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(scim): honor explicit PUT roles, preserve role when roles omitted
Refine the PUT provisioning behaviour per review: an explicit `roles` value is
honoured again (the membership is created or updated to that role), and only a
request that omits `roles` leaves an existing membership untouched (creating a
default NONE membership when none exists). This keeps a roleless IdP full-sync
from resetting a member's role to NONE while still letting SCIM set roles
explicitly.
- Replace `provisionMembershipIfMissing` with `provisionMembership`, which takes
an optional role: create-or-update to the role when provided, else create NONE
if missing / no-op if present.
- Audit a `create` on provisioning and an `update` (with before/after) when an
explicit role actually changes an existing membership; no audit on no-ops.
- PATCH still passes no role (PatchOp values only carry `active`).
Updates the regression tests accordingly: explicit role honoured on create and
on update (with update audit), and no role downgrade when `roles` is omitted.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: test cleanup
* fix(scim): include ProjectMemberships in deprovision audit before payload
The SCIM deprovision path (PUT/PATCH active:false, DELETE) deletes the
OrganizationMembership, which cascade-deletes its ProjectMembership rows
(ProjectMembership.organizationMembership is onDelete: Cascade) without emitting
any separate audit entry. The delete audit's `before` is the only place those
rows can be preserved, but `deprovisionOrReject` fetched the membership without
the relation, so a compliance reviewer could not reconstruct which projects a
deprovisioned user had access to.
Add `include: { ProjectMemberships: true }` to the findUnique inside the
deprovision transaction, matching the tRPC deleteMembership audit entry
(membersRouter.ts). The role-update path is intentionally left as-is: it mirrors
the tRPC updateOrgRole path and does not delete project memberships, so they
remain queryable and need not be captured.
Adds a regression test asserting the cascaded project membership is gone from
the DB after a SCIM DELETE but recoverable from the audit before payload.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(scim): guard last-OWNER demotion on PUT role update
Honoring an explicit role on PUT (commit acf4bf2) reintroduced a path that could
demote the sole OWNER of an organization: applyToExisting updated the role with
no owner-count check, so `PUT /Users/{ownerId}` with `roles:['MEMBER']` against
the last OWNER left the org with zero owners and unmanageable from the UI. Every
sibling path guards this (deprovisionOrReject, tRPC updateOrgMembership) — the
PUT role-update path did not.
Wrap the role update in a Serializable transaction that, when demoting an OWNER
(existing OWNER → non-OWNER), counts remaining OWNERs and rejects with 403 and
the same detail string as the deprovision guard when none would remain. Two
concurrent demotions cannot both pass (serialization failures surface as 409 for
client retry). provisionMembership now returns null when it has already written
a rejection response, and the PUT/PATCH handlers short-circuit on it.
Adds last-OWNER tests for the PUT role-update vector: rejects demoting the only
OWNER (403), and allows demotion when another OWNER remains.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(scim): read membership inside the role-update transaction
The last-OWNER guard in provisionMembership decided "is this a demotion" from a
membership row read OUTSIDE the Serializable transaction. Postgres SSI only
protects reads made within the transaction, so a stale read could skip the
guard: with X read as MEMBER, a concurrent promotion of X to OWNER plus a
demotion of the previous OWNER could interleave such that the unguarded update
left the org with zero owners. The same stale read meant a concurrent
deprovision between read and update surfaced as an unhandled P2025 → 500.
Re-read the membership inside the transaction and base the no-op check, the
OWNER guard, and the update on that snapshot. When the row vanished before the
transaction (concurrent deprovision), fall through to the create path instead
of erroring, preserving the "ensure provisioned" semantics. The P2002
concurrent-create fallback now reuses the same in-transaction path.
No behavioral change for sequential requests; audit writes stay post-commit per
the established pattern.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: typo
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Document which response fields each v3 scores field group adds (core
fields always returned; details, subject, annotation additive). The
"core" token remains accepted by the schema for compatibility but is
not documented, since passing it has no effect
- Replace the incorrect "requires Langfuse v4" note on the v3 endpoint
with removal notices on the deprecated v2 GET endpoints, including the
id-filter migration path for get-by-id
- Regenerate OpenAPI spec (docs-only; Python/TS SDK outputs unchanged)
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* fix(monitors): correct Slack alert formatting
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* test(monitors): update slack-processor tests for attachment fallback
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* refactor(monitors): extract buildMonitorAlertSlackMessage into shared
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(worker): correct 'scaleable' typo in webhooks comment
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* feat(mcp): expose unstable evaluator and evaluation-rule tools
Add an `evals` MCP feature with eight tools wrapping the unstable public
evaluator and evaluation-rule APIs: list/get/create evaluators and
list/get/create/update/delete evaluation rules.
Tool runtime validation reuses the public-API contract schemas directly;
the discovery-facing base schemas flatten the contract's union schemas
(evaluator type, rule target, output definition, filters) where the MCP
layer forbids union JSON schemas. Reads are exposed to in-app agent keys;
mutating tools are marked destructive.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(mcp): cover code evaluators; clarify mapping source description
Address PR review: broaden RuleMappingBaseSchema description to surface
experiment-only sources (expected_output, experiment_item_metadata), and
add code-evaluator coverage — a code evaluator + managed-mapping rule plus
a check that supplying a mapping on a code rule is rejected.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor(mcp): share advanced-filter base schema
scores, metrics, and evaluation-rule tools each defined an identical loose
{column, operator, value, type, key?} filter base for MCP discovery. Extract
it to core/filter-schema.ts (McpAdvancedFilterBaseSchema) and reuse it.
listObservations keeps its narrower variant (type inferred from the column).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor(evals): centralize public eval audit logging
* refactor(mcp): rename createEvaluator to upsertEvaluator; consolidate eval tests
- Rename the createEvaluator MCP tool to upsertEvaluator: creating with an
existing name versions the evaluator and migrates its rules, so it is not a
plain create (mirrors the upsertDataset convention).
- Move eval tool coverage into the shared mcp-tools-{read,write} servertests and
mock the evaluator preflight instead of provisioning a default eval model.
- Keep one happy path per union; drop cross-project and redundant negative tests.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(fern/scores): move v2 GET endpoints to legacy/scores-v2.yml
Copies the scores v2 service definition into the legacy/ directory to
reflect that it is superseded by v3. Import paths adjusted for the new
location (../utils/pagination.yml, ../commons.yml).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* feat(fern/scores): mark legacy scores v2 endpoints as deprecated
Points consumers to GET /api/public/v3/scores in the endpoint docs.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* feat(fern/scores): promote scores-v3.yml to canonical scores.yml
Replaces scores.yml (v2 GET endpoints, now in legacy/) with the v3
definition. Deletes the superseded scores-v3.yml.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* feat(fern/scores): rename V3-suffixed identifiers in canonical scores.yml
Now that scores.yml is the primary definition, drop the V3 suffix from
the endpoint (getManyV3 → getMany), the request/response wrappers
(GetScoresV3Request → GetScoresRequest, GetScoresV3Response →
GetScoresResponse, GetScoresV3Meta → GetScoresMeta), and all ScoreSubject
types (ScoreSubjectV3 → ScoreSubject, etc.).
The concrete score model types (ScoreV3, NumericScoreV3, etc.) keep their
V3 suffix because commons.yml already declares Score, NumericScore, etc.
for the v1/v2 API surface — Fern uses a flat global namespace so those
names are taken.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* chore(generated): regenerate OpenAPI spec after scores section rename
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
* feat(automations): narrow getAutomations by eventSource and matches
* feat(monitors): list page
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(monitors): create and edit monitors
* feat(monitors): prefill create-automation form from monitor draft
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(monitors): polish create/edit form UX
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(monitors): toggle automations from the monitor form
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(automations): drop unused matches arg from getAutomations
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(monitors): gate page entry on feature flag and rbac scope
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(monitors): highlight automations matching an untagged monitor
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(monitors): apply style guide to existing components
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(automations): gate Monitor event source on monitors flag
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(monitors): point breadcrumb at the monitors list, not the project root
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor: tighten jsdoc and inline comments per style guide
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(monitors): extract updateSeverityForStatus and lock in transition tests
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(monitors): move FilterState mapping out of the service layer
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(monitors): introduce getValidMonitorAggregationsForMeasure and structural isValidThresholdOrder
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(query): drop formatAggregation, reuse widget form startCase mapping
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(tag): make TagManager a controlled component and rename alignPopover
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(monitors): make none-of automation rows inert in the panel
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(monitors): ignore stale form status on save so pause/resume sticks
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(automations): UTF-8-safe prefill encoder so non-ASCII tags don't throw
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(query): drop aggs.agg pin from getValidAggregationsForMeasure, keep it in monitors wrapper
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(monitors): typed ListMonitorFilter as a singleFilter[] subset
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* refactor(monitors): fold projectId into toPrismaWhere and let updateSeverityForStatus take an optional current
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* fix(mcp): use the new getValidAggregationsForMeasure
* refactor(query): restore getValidAggregationsForMeasureType(measureType)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* refactor(widgets): restore WidgetForm.tsx from main
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* refactor(tag): trim TagManager diff to additive triggerButton/alignPopover and reintroduce mutate-on-close with a downward initialTags sync
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* feat(monitors): gate monitors to Langfuse Cloud deployments only
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* feat(monitors): add MonitorScheduler with deterministic next_run_at and run_at queue payload
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test(monitors): scaffold MonitorScheduler integration test with table-driven runner
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(monitors): treat NULL last_completed_run_at as never-completed in runIsPending
The runIsPending negation introduced a three-valued-logic bug: when
last_published_run_at is set but last_completed_run_at is NULL (worker has
the first job but hasn't finished), last_completed < last_published is NULL,
the AND chain short-circuits to NULL, and the CASE stamps a duplicate run.
Treating NULL as "never completed" closes the gap.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test(monitors): cover advance/publish branches for MonitorScheduler
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test(monitors): cover next_run_at determinism and cadence-aligned boundary math
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(monitors): lowerCamelCase scheduler test constants, group by role
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(monitors): typos in scheduler SQL comments
* fix(monitors): clear lifecycle stamps when MonitorService.update changes schedulerBatchId
When the query shape changes (filters/view/window), the previous publish's
lifecycle stamps refer to the OLD batch. Leaving them would suppress the next
publish via runIsPending for up to monitorProcessorTtl. Cosmetic edits
(name/tags/threshold) preserve the stamps so an in-flight worker on the same
batch still dedups.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(monitors): add monitors.last_claimed_run_at lifecycle column
MonitorProcessor.claim writes this when a worker takes ownership of a
published run. The CAS in claim uses last_published_run_at as the TTL
anchor; this column is the per-run claim discriminator so duplicate
BullMQ deliveries become no-ops.
* feat(monitors): add MonitorProcessor.claim with TTL CAS
Replaces the Redis lock at step 1 of the RFC §Monitor Queue Processor.
The claim CAS uses three guards: last_published_run_at = event.runAt
rejects stale republishes, last_completed_run_at < last_published_run_at
rejects BullMQ replays of completed runs, and a TTL branch on
last_claimed_run_at allows reclaim after a dead worker. TTL constant is
shared with the scheduler's runIsPending gate.
Servertest covers the three success branches (fresh, prior-run, TTL),
five denial branches (in-flight within TTL, TTL boundary exact,
completed, stale event, never-published), two multi-monitor batches,
empty/missing inputs, and projectId mismatch.
* feat(monitors): add computeSeverity threshold check
Pure mapping of (metric value, operator, alert/warning thresholds) to
NO_DATA / OK / WARNING / ALERT. Alert is checked before warning so the
more-severe outcome wins when both match (relevant for NEQ overlaps).
Table-driven test covers null values, every operator branch, both
boundary cases (strict GT/LT do not match at threshold; GTE/LTE do),
and missing-warning-band fallthrough.
* feat(monitors): add applyStateMachine for severity transitions
Encodes the RFC §Severity State Machine table: given a transition from
prev to computed severity plus noData/renotify config, returns the emit
flag and the next lifecycle stamps (severity, severityChangedAt,
alertedAt).
Notable interpretations:
- The RFC text writes the cooldown gate as `alertedAt + interval >
scheduledAt`, which inverts the natural "wait long enough" intent;
implemented here as `scheduledAt - prevAlertedAt >= interval`.
- NULL prevAlertedAt on a noData escalation fires immediately (no prior
emit to cool down from); on a renotify self-loop it stays silent
(renotify needs a baseline to re-emit from).
28-case table-driven test covers every row of the RFC table plus the
cooldown / NULL edges and the OK-self-loop renotify exemption.
* feat(monitors): add MonitorProcessor.complete bulk lifecycle UPDATE
Lands the post-evaluation lifecycle stamps for an entire batch in one
statement via a VALUES-join: last_completed_run_at, severity,
severity_changed_at, alerted_at. The caller passes pre-computed values
from applyStateMachine; complete just writes them.
Servertest covers seven row-shape cases (no-change, severity-only,
emit-only, both, multi-monitor mixed, empty no-op, missing-id ignored)
and a project-scoping case proving a completion against a row in
another project leaves it untouched.
* feat(monitors): MonitorProcessor.process shell — claim, CH query, state machine, complete
Wires the four pure units (claim, computeSeverity, applyStateMachine,
complete) to real I/O on the project's ClickHouse and Postgres. Adds a
MonitorPublisher seam to the constructor as a placeholder; the
trigger-filter and publisher-emit branches land in the follow-up commit.
Per RFC step 9, the publish-before-commit ordering will go between the
per-row state machine loop and the complete() call.
Servertest covers four orchestration shapes:
- empty claim (event with no monitors) → no CH query, no row write
- OK → OK steady state (CH count 50, threshold 100) → only last_completed_run_at advances
- cold-start UNKNOWN → ALERT (CH count 142) → state machine emits, alertedAt advances
- partial claim (1 of 2 already completed) → only the claimable row updates
NOTE: NO_DATA semantics replaced with cold-start ALERT in case E because
count() returns 0 (not NULL) on empty ClickHouse results. NO_DATA returns
as a follow-up once we wire a NULL-returning aggregation.
* feat(monitors): MonitorProcessor.process emits alerts via publisher
Wires the trigger-filter and publisher-emit branches of process().
Builds a MonitorAlert per state-machine emit, projects (severity, tags,
monitorId, monitorName) into the trigger-filter shape, calls publisher
once per surviving monitor (RFC step 9: not once per matching trigger),
then commits.
Alert message body distinguishes the kind of transition:
- NO_DATA → "<agg>(<view>.<measure>) has no data over the last <window>"
- NO_DATA → OK → "<agg>(<view>.<measure>) has data again"
- WARN/ALERT/NO_DATA → OK → "<agg>(<view>.<measure>) is back within threshold"
- WARN/ALERT (any other transition or self-loop renotify)
→ "<agg>(<view>.<measure>) is <above/below/...> <threshold>"
Servertest covers four publisher-side cases:
- ALERT + matching severity trigger → publish called once with the expected payload shape
- ALERT but trigger filter matches WARNING → publish NOT called; row still written per state machine
- tags-based trigger filter matches → publish called once
- multiple triggers, one matches → publish called ONCE (per-monitor, not per-trigger)
* test(monitors): rewrite MonitorProcessor.process servertest as branch table; inject CH+trigger seams
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* refactor(monitors): rename lifecycle columns to wallclock semantics
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(monitors): scheduler rescue branch + wallclock publish stamps
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(monitors): add publishedAt wallclock to MonitorQueueEvent
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(monitors): processor claim CAS keyed on event.publishedAt; complete CAS owner key
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(monitors): wire MonitorScheduler + MonitorProcessor into BullMQ
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test(monitors): add publishedAt to MonitorQueueEvent type fixtures
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(monitors): PAUSED guard in state machine + accept path-only permalink
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(monitors): replace BullMQ scheduler queue with PeriodicExclusiveRunner shards
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(monitors): rename MonitorProcessorQueue→MonitorQueue, MonitorSchedulerRunner→MonitorRunner; hoist scheduler into constructor; fail-fast on missing queue
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(monitors): replace stalledInterval:0 with maxStalledCount:0 on MonitorQueue worker
BullMQ 5 rejects stalledInterval:0; use maxStalledCount:0 to achieve the same intent (no stalled-job redelivery) without violating the constraint.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(monitors): emit wire shape from scheduler to keep BullMQ JSON-safe
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(automations): allow monitor in AvailableWebhookApiSchema
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(monitors): MonitorWebhookQueueEventSchema gains id+timestamp; rename version→apiVersion
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(webhooks): WebhookOutboundEnvelopeSchema discriminates on type
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(automations): expose automations[] on TriggerDomainWithActions
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(monitors): processor fans alerts out to WebhookQueue envelopes
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore(worker): add slackify-markdown dependency
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(slack): SlackMessageParams accepts optional attachments
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(slack): buildMonitorMessage with severity emoji + color attachment
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(webhooks): dispatch monitor-alert envelopes with redis-backed auto-disable
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(monitors): publish MonitorAlerts to WebhookQueue
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test(monitors): e2e scheduler → dispatcher → webhook URL
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(monitors): add triggerIds column
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* feat(monitors): add triggerIds to MonitorSchema
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(monitors): backfill triggerIds in test fixture after schema change
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(automations): inject synthetic triggerIds clause
* feat(monitors): persist triggerIds on create and update
* refactor(automations): drop filter UI from monitor-source trigger form
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(automations): align monitor trigger eventActions with plan (created/updated/deleted)
* refactor(monitors): switch automations panel from tag-matching to triggerIds
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* feat(monitors): wire triggerIds through MonitorForm; move TagManager under name
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* revert(test-utils): unrelated stash-pop content accidentally bundled with monitor work
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* refactor(monitors): restore automations-panel card UX (drop tag pills) and trim TagManager chrome
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* refactor(monitors): polish trigger UI - info card, no event actions, hide execution history
To achieve this we:
- Replaced the monitor-source eventAction picker with an Alert info card linking to the create-monitors page; the picker (with created/updated/deleted options) and its setValue side-effects are gone.
- Hid the Execution History tab for monitor-source automations on the detail page; the form renders directly with no TabsBar.
- Dropped the actionType parameter from the automations create deep-link and collapsed the type-specific dropdown items into a single "New automation" link.
- Skipped name auto-fill on create and added autoFocus to the name input so the cursor lands there when the form opens.
- Bumped per-row padding on the automations panel from px-2 py-1 to p-2.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* revert(monitors): restore per-action-type entries in automations dropdown
Restores the dropdown menu with Webhook / Slack / GitHub Dispatch entries on top of the generic "New automation" link, and re-adds the actionType arg to the create deep-link prefill. Walks back the over-broad collapse from ca7060610 - the dropdown UX was the right shape; only the tag-related prefill needed stripping.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(monitors): normalize filter columns in resolver so view validation passes
The InlineFilterBuilder writes UI-table display labels (e.g. "Environment") into form state, while CreateMonitorSchema.validateMonitorQuery validates against view-space dimension keys (e.g. "environment"). With mode: "onChange" the validator was rejecting valid filters before the submit handler could map them.
To achieve this we:
- Wrapped zodResolver so it maps filters via mapWidgetUiTableFilterToView before delegating to the schema; form state itself stays in UI-table-space so the builder still renders display labels.
- Added the inline "Send Alerts to Slack, Webhooks, and GitHub Actions." explainer under the Automations heading.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(entitlements): add monitor-count limit at 10 across all plans
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* test(monitors): cover monitor-count limit enforcement and count query
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* feat(monitors): gate create at monitor-count limit and expose org-wide count
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* feat(monitors): show at-limit state on the New monitor action button
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* feat(monitors): add onboarding splash for empty monitors list
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* feat(automations): support redirectUrl in create-form deep links
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* fix(automations): fire pending redirect when secret dialog is dismissed via X or Escape
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* refactor(automations): consolidate post-flow redirect into a single finishFlow helper
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* feat(monitors): shift CH evaluation window back by 30s for ingestion lag
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(monitors): narrow isValidQuery to a disallow-list
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(automations): scope triggerIds clause to monitors; preserve triggerIds on toolbar pause
- matchesTriggerFilter only appends the synthetic `triggerIds any-of [id]`
clause for monitor-source triggers. Prompt-source events don't carry a
triggerIds field, so the unconditional clause broke every existing prompt
automation in the worker's promptVersionProcessor.
- EditMonitorPage's toolbar Pause/Resume payload was missing `triggerIds`;
zod's `.default([])` filled in `[]` server-side and the service wrote it,
silently wiping linked automations on every toggle.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(automations): gate synthetic triggerIds clause on data presence, not source enum
Drops the eventSource conditional in matchesTriggerFilter. The clause now
appends when the event data carries a triggerIds array, which is the actual
opt-in signal — monitor events publish the field, prompt-version events do
not. New sources opt in by including the field, no matcher change required.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(monitors): 0-indexed list pagination, redirect URL on add-automation, EQ threshold rendering, empty-state flash
- MonitorService.list was 1-indexed (`(page - 1) * limit`) while every other
router and the MonitorsTable call site send a 0-indexed page. Aligns the
service with the codebase convention and updates the servertest.
- MonitorAutomationsPanel's "+ Automation" dropdown now threads
`router.asPath` into `automationCreateHref` so saving an automation returns
the user to the monitor form they were editing instead of dumping them on
the automations list.
- ThresholdOverlay adds an EQ arm (thin band centered on the value, mirroring
NEQ) and floors `bandEpsilon` at 1 so a `threshold.value === 0` while data
loads doesn't collapse the band to zero height.
- MonitorAutomationsPanel gates the empty splash on
`automations.isSuccess && data.length === 0` so loading and error states no
longer flash "Set up automations" over a monitor that already has them.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(monitors): hold list-page render until hasAny resolves to drop empty-state flicker
When the project has no monitors, the page was rendering the table branch
(loading skeleton) while monitors.hasAny was in flight, then swapping to the
onboarding splash. Now gates the splash-vs-table choice on hasAny.isSuccess
and renders a minimal page chrome during the loading window so the table
never mounts on the empty path.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(monitors): MonitorsTable pageIndex+1 for 1-indexed service, extract toPrismaOrderBy mapper
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(monitors): display ListMonitorsPage if hasAny fails
* fix(monitors): monitors test start paging at 1
* fix(monitors): gate onboarding splash CTA on monitors:CUD
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* improvements(monitors): better help description
* refactor(table-controls): template filter empty-state copy with tableName
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(monitors): pass hasCUDAccess in MonitorsOnboarding client tests
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(monitors): release entitlement for projects pending deletion
* fix(monitors): status severity consistant across create and update operations
* fix(monitors): status severity consistant across create and update operations
* ci(codespell): ignore false-positive usera
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(monitors): rename userA to sessionUser to satisfy codespell
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(monitors): processor filters by triggerIds
* fix(monitors): update not found error
* improvement(monitors): take placeholder name as the default name
* fix(monitors): prevent monitor edits from toggeling status
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(monitors): removed dead isPinnedRight field
* feat(monitors): show preview query errors inline in live preview
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* style(monitors): fill preview height with flex and shrink error overlay
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor(monitors): return claimed monitors directly from claim
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(monitors): rename to monitors-e2e and link seeded trigger via triggerIds
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor(monitors): clean up processor and scheduler
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(monitors): monitor processor receives inputs instead of parsed domain model
* fix(monitors): correct stale fixture field names and typo to unblock CI
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(monitors): null lastClaimedAt on query-shape edit to void stale-completion CAS
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(monitors): NO_DATA self-loop honors noData SILENT instead of renotify
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(monitors): omit permalink at producer when NEXTAUTH_URL unset
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(monitors): parse monitor-alert payload in executeSlackAction to recover Dates
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(monitors): guard complete CAS on status=ACTIVE to not overwrite a post-claim PAUSED row
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(monitors): scope MonitorService.update pre-read by projectId
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(monitors): clear monitor-alert failure counter on auto-disable
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(monitors): guard complete CAS on publish identity to void scheduler-rescued stale completion
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(monitors): isolate success-path failure-counter reset so a Redis blip can't cascade into the failure branch
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(monitors): use alert title as Slack text fallback for monitor-alert push previews
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(monitors): gate claimMonitors on status=ACTIVE to close pause-then-resume PAUSED-stick race
* fix(monitors): cap Slack monitor-alert header at 150 chars and drop severity emoji
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(monitors): drive no-AutomationExecution test through real dispatch path
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(monitors): hoist Slack config read out of failure-counting try
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(monitors): update stale Slack header emoji assertion to title-only
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(monitors): empty payload.filters in oversize GitHub-dispatch monitor-alert fallback
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(monitors): MIN-rollup scheduler run_at and reset stamps on resume
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(monitors): escape Slack text fallback for monitor-alert dispatches
* fix(monitors): MAX-rollup scheduler run_at so resumed siblings pull batch forward
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(monitors): NOTIFY no-data monitor alerts on sustained NO_DATA from cold start
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(monitors): reset alertedAt on PAUSED->ACTIVE resume
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(monitors): wrap failure-side resetAutomationFailures in try/catch
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(monitors): correct MonitorNoDataSchema SILENT/NOTIFY JSDoc
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(monitors): NO_DATA persistence emits when prior-stretch alert predates current stretch
* fix(monitors): gate NO_DATA on appended count_count so empty-window count/sum/min/max monitors report NO_DATA
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(monitors): list row actions, polling, and monitors/<id> route
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(monitors): minor ui refinements
* fix(monitors): theme-aware foreground ramp for row action icons
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(monitors): make monitor queue concurrency configurable via env
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* improvement(monitors): removed ComputedSeverity enum
* test(monitors): null vs 0 coverage for scalar queries
* style(monitors): MonitorRunner log name clarity
* test(monitors): skip scalar query test when events_core is absent
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(monitors): flip monitor to ERROR_BAD_QUERY on permanent query-shape failure
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(monitors): scope ERROR_BAD_QUERY to query-shape failures only
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* improvement(monitors): better error handling
* test(monitors): executeQuery error flips ERROR_BAD_QUERY, no throw
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(monitors): per-metric ErrorBadQuery validation, unify into complete
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor(monitors): rename toActve typo, inline ACTIVE transition
* fix(monitors): gate MonitorAutomationsPanel row toggles on hasAccess
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(automations): seed webhook apiVersion from trigger eventSource
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(monitors): gate AddAutomationDropdown trigger on hasAccess
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(monitors): log batch context when queryMetrics executeQuery rejects
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(nav): drop DEV region gate from cloudAdmin check
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(monitors): require at least one automation on create/update
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(monitors): make form validation errors visible on submit
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(monitors): seed valid triggerIds in monitors servertest fixture
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(monitors): silent label on no-data is correct now
* fix(monitors): forward eventSource when switching action type to WEBHOOK
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(monitors): gate nav entry on isLangfuseCloud to match page gate
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(api): expose code evaluators in public endpoints
* fix(api): correct code-evaluator error mapping and patch type handling
Address review findings on the public unstable evals API:
- Translate raw TRPCErrors from the shared code-eval preflight into the
documented structured 4xx errors instead of a 500 (dispatcher/template/
language failures previously fell through to internal_error).
- Restore BAD_REQUEST for invalid code-evaluator targets in the tRPC path.
- Stop the PATCH evaluator reference from defaulting `type` to llm_as_judge;
inherit the rule's current evaluator type when omitted so code rules are
not silently retargeted.
Update Fern docs + regenerate OpenAPI, add regression tests.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* test(evals): update stale unstable evals assertions for evaluator type
- Drop the removed LLM_AS_JUDGE filter from the active-rule count expectation
(the query now spans code evaluators too).
- Expect the evaluator `type` field now returned on evaluation-rule responses.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(api): disallow changing an evaluation rule's evaluator type on patch
A PATCH that switched a code rule to an LLM evaluator inherited the
synthesized canonical code mapping and validated it against the LLM
evaluator, producing a 400 referencing fields the caller never sent.
Drop `type` from the patch evaluator reference so the rule always inherits
its current evaluator type; the family lookup stays scoped to that type, so
cross-type retargeting can no longer happen. Update Fern docs + regenerate
OpenAPI, and assert the stripped `type` in the contract test.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* refactor(evals): use typed domain errors for code-eval validation
Replace ad-hoc TRPCErrors in the code-eval test run with a typed
CodeEvalTestRunSetupError discriminated by `code`, and consolidate the
job-config preflight into a single CodeEvalJobConfigError carrying a
typed `code` (drops the redundant invalid-target class). Transport
mapping (tRPC + unstable public API) now lives at each boundary with
exhaustive switches. Also simplifies the evaluator-type contract to
literal<"code"> and reuses PublicEvaluatorTypeType.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(evals): align code-eval job-config tRPC error codes and fixtures
Map CodeEvalJobConfigError to granular tRPC codes (resource_not_found →
NOT_FOUND, invalid_request → BAD_REQUEST, preflight_failed →
PRECONDITION_FAILED) via a dedicated helper, restoring the pre-existing
404/400 responses on the create/update job-config paths and keeping the
tRPC boundary consistent with the public API.
Also fixes test fixtures left structurally invalid by the widened
StoredPublicEvaluatorTemplate / narrowed evalTemplate Pick types: add
type/sourceCode/sourceCodeLanguage to the LLM fixtures, add type and
drop excess vars/prompt from the rule-config fixture, pass evaluator
type on the query fixtures, and narrow the discriminated evaluator
record before asserting outputDefinition.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* docs(api): note code-eval source is validated at rule-link time
Clarify in the create-evaluator endpoint docs that a code evaluator's
sourceCode is not executed at creation; it is first preflight-tested
when linked to an evaluation rule, so runtime errors surface there.
Regenerate the OpenAPI output accordingly.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* refactor(web): create support threads only in Pylon
The support form previously created threads in both Pylon and Plain. As
Plain is being deprecated, this removes all Plain thread-creation code so
new support threads are created only in Pylon.
- Replace plainRouter with a Pylon-only supportRouter; drop the Plain-only
prepareAttachmentUploads procedure and all Plain calls (customer upsert,
tenant/tier sync, thread + thread-event creation, presigned S3 uploads).
Org/project validation and plan derivation are kept for Pylon
priority/severity/tier mapping. A missing PYLON_API_KEY now surfaces as
pylonIssueFailed instead of a silent no-op.
- Update SupportFormSection to upload attachments only to Pylon and call
api.supportRouter.createSupportThread.
- Delete the support-chat/plain directory and add pylonConstants.ts for the
attachment size limit (10MB, matching the Pylon upload endpoint).
- Remove the now-unused PLAIN_API_KEY env var from env.mjs and
.env.prod.example.
PLAIN_AUTHENTICATION_SECRET / createSupportEmailHash (Plain dashboard login
via auth) are left untouched as they are out of scope.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(web): preserve support form state when Pylon submission fails
Now that Pylon is the sole destination for support threads,
pylonIssueFailed=true means no ticket was created anywhere. The previous
onSuccess handler reset the form (message, topic, severity, attachments)
before checking pylonIssueFailed, so a failed submission wiped everything
the user typed and left them no way to retry.
Move the reset/clear into the success-only path and early-return after
showing the error toast on failure, keeping the form state intact for retry.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(web): prevent silent attachment drops on support form
Two regressions from removing Plain as a support destination:
1. The per-file cap was raised to 10MB but maxCombinedBytes stayed at 50MB.
Attachments are POSTed to /api/support/upload-attachments as base64 JSON
(~33% larger), and that endpoint caps the body at 50MB, so a valid <50MB
raw selection could exceed the body limit and be rejected. Lower
maxCombinedBytes to 35MB (~47MB once base64-encoded) for headroom.
2. uploadFilesToPylon is now the sole attachment path but was still wrapped
in a .catch that returned [], silently dropping the user's files while the
thread was still created with a success toast. Remove the swallow so upload
errors propagate to the outer try/catch (form.setError), letting the user
retry instead of losing attachments.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* refactor(web): use shared PYLON_MAX_FILE_SIZE_BYTES in upload endpoint
The upload-attachments endpoint hardcoded its own 10MB per-file limit while
pylonConstants.ts claimed to be the single source of truth. Import the shared
constant so the endpoint and the form stay in sync and a future bump only
needs to change one place.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(support): allow Team/Enterprise to flag urgent (Sev-1) requests
Add an "Urgent (Sev-1)" checkbox next to the severity dropdown in the
support form, shown only to Team/Enterprise plans. When checked, the
Pylon issue is created with case_severity "Sev-1" and priority "urgent".
The flag is enforced server-side: non-high-tier plans cannot force Sev-1
even if the input is supplied. An info tooltip explains intended use.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* fix(Storage): use GOOGLE_CLOUD_UNIVERSE_DOMAIN
* feat(Storage): Set default value for GOOGLE_CLOUD_UNIVERSE_DOMAIN
* feat(Storage): Add universeDomain to others Storage instantiation
---------
Co-authored-by: Steffen Schmitz <steffen@langfuse.com>
Handing the nodemailer SES transport-options object (containing a live
SESv2Client) to NextAuth's EmailProvider `server` field crashed every
NextAuth-adjacent endpoint, including /api/auth/session, with
`RangeError: Maximum call stack size exceeded`. NextAuth's
`parseProviders` deep-merges provider options on every request, and the
recursion follows circular references inside the AWS SDK client.
Pass the connection URL string to EmailProvider instead and let the
custom `sendResetPasswordVerificationRequest` build the transport via
the existing `createMailTransport` helper. The now-unused
`buildMailServerConfig` helper and `MailServerConfig` type are removed.
Adds a regression test that replays NextAuth's deep-merge algorithm
against each supported SMTP_CONNECTION_URL scheme (smtp://, smtps://,
ses://) and asserts none of them blow the stack.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: Steffen Schmitz <steffen@langfuse.com>
Promotes the v3 scores list endpoint from preview to general availability.
- Move Fern source from `_drafts/scores-v3.yml` to `definition/scores-v3.yml`
so the endpoint appears in the published OpenAPI spec and generated SDKs.
- Remove the `LANGFUSE_ENABLE_SCORES_V3_API` feature flag from `env.mjs`
and both `.env.*.example` files. The handler no longer 404s when the
flag is unset — the endpoint is now reachable on every deployment.
- Regenerate `web/public/generated/api/openapi.yml` to include
`/api/public/v3/scores`.
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* feat(scores): flatten v3 response shape to match proposal
Hoist `comment`, `configId`, `metadata`, `authorUserId`, `queueId` from
the `details` / `annotation` wrappers to top-level fields on the v3
score response. Matches the canonical JSON in the Scores v3 API proposal.
`subject` stays nested (discriminated union; flattening would lose the
`kind`/`traceId` type relationship and collide with the top-level `id`).
`fields=` syntax and gating semantics unchanged — group names still
control which fields appear, only the wrapping changes. The v3 endpoint
is flag-gated (LANGFUSE_ENABLE_SCORES_V3_API=false by default), so no
production callers are affected.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* docs(scores): describe field + group dependency in v3 schema
Pair each hoisted field's docs with a "Present when <group> is included"
clause so SDK readers see the field-group gating alongside the
description. Replace ScoreSubjectV3's gating-only type-level docs with a
description of the discriminated union itself.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* feat(scores): version v3 cursor with v: 1 for forward migration
* remove comment
* fix(scores): encode cursor version from cursor.v instead of hardcoded literal
encodeCursorV3 hardcoded v: 1 in the serialized payload, ignoring cursor.v.
This undermined the discriminated-union forward-compat contract: a future
v: N arm sharing lastTimestamp/lastId would silently serialize as v: 1.
Use cursor.v so the encoded payload always reflects the cursor's version.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
External security reports keep landing on the same shape — a user-supplied URL
or endpoint that the server fetches without going through outbound-URL
validation. Add a shared `.agents/skills/security-review` skill so agents are
prompted to catch the pattern during PR review and during plan/design mode,
not after the fact.
The skill is intentionally extensible: SKILL.md is a slim entrypoint,
`references/checklist.md` is the mental sweep, and each recurring finding
class becomes its own `references/<topic>.md`. SSRF/outbound URL validation
ships today, anchored in the existing canonical helpers
(`validateLlmConnectionBaseURL`, `validateWebhookURL`,
`validateBlobStorageEndpoint`, `validateOutboundUrlHost`,
`fetchWithSecureRedirects`) so reviewers point authors at known-good call
sites rather than re-deriving fixes.
Wire it into the existing routers so it loads at the right moments:
- root `.agents/AGENTS.md` Start Here entry
- `.agents/skills/README.md` registration
- `code-review/SKILL.md` + `code-review/references/review-checklist.md`
- `backend-dev-guidelines/SKILL.md` + `backend-dev-guidelines/AGENTS.md`
anti-pattern bullet for raw `fetch(<userUrl>)`
Verified with `pnpm run agents:sync` and `pnpm run agents:check`; the new
skill is linked into `.claude/skills/security-review` and surfaces in the
available-skills listing.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(api): return 503 for public auth db failures
* fix(api): trace public auth db failures
---------
Co-authored-by: Steffen Schmitz <steffen@langfuse.com>
* fix(worker): bump decommissioned gemini-2.0-flash to gemini-2.5-flash in VertexAI tests
* fix(worker): use gemini-2.5-flash-lite for VertexAI tests to avoid thinking-token starvation
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor(web): create support threads only in Pylon
The support form previously created threads in both Pylon and Plain. As
Plain is being deprecated, this removes all Plain thread-creation code so
new support threads are created only in Pylon.
- Replace plainRouter with a Pylon-only supportRouter; drop the Plain-only
prepareAttachmentUploads procedure and all Plain calls (customer upsert,
tenant/tier sync, thread + thread-event creation, presigned S3 uploads).
Org/project validation and plan derivation are kept for Pylon
priority/severity/tier mapping. A missing PYLON_API_KEY now surfaces as
pylonIssueFailed instead of a silent no-op.
- Update SupportFormSection to upload attachments only to Pylon and call
api.supportRouter.createSupportThread.
- Delete the support-chat/plain directory and add pylonConstants.ts for the
attachment size limit (10MB, matching the Pylon upload endpoint).
- Remove the now-unused PLAIN_API_KEY env var from env.mjs and
.env.prod.example.
PLAIN_AUTHENTICATION_SECRET / createSupportEmailHash (Plain dashboard login
via auth) are left untouched as they are out of scope.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(web): preserve support form state when Pylon submission fails
Now that Pylon is the sole destination for support threads,
pylonIssueFailed=true means no ticket was created anywhere. The previous
onSuccess handler reset the form (message, topic, severity, attachments)
before checking pylonIssueFailed, so a failed submission wiped everything
the user typed and left them no way to retry.
Move the reset/clear into the success-only path and early-return after
showing the error toast on failure, keeping the form state intact for retry.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(web): prevent silent attachment drops on support form
Two regressions from removing Plain as a support destination:
1. The per-file cap was raised to 10MB but maxCombinedBytes stayed at 50MB.
Attachments are POSTed to /api/support/upload-attachments as base64 JSON
(~33% larger), and that endpoint caps the body at 50MB, so a valid <50MB
raw selection could exceed the body limit and be rejected. Lower
maxCombinedBytes to 35MB (~47MB once base64-encoded) for headroom.
2. uploadFilesToPylon is now the sole attachment path but was still wrapped
in a .catch that returned [], silently dropping the user's files while the
thread was still created with a success toast. Remove the swallow so upload
errors propagate to the outer try/catch (form.setError), letting the user
retry instead of losing attachments.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* refactor(web): use shared PYLON_MAX_FILE_SIZE_BYTES in upload endpoint
The upload-attachments endpoint hardcoded its own 10MB per-file limit while
pylonConstants.ts claimed to be the single source of truth. Import the shared
constant so the endpoint and the form stay in sync and a future bump only
needs to change one place.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
## What does this PR do?
> PR title must follow Conventional Commits, for example `feat(web): add trace filters` or `fix: handle empty dataset names`.
<!-- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change. -->
Fixes # (issue)
<!-- Please provide a loom video for visual changes to speed up reviews
Loom Video: https://www.loom.com/
-->
## Type of change
<!-- Please delete bullets that are not relevant. -->
- [ ] Bug fix (non-breaking change which fixes an issue)
- [ ] Chore (tooling, dependencies, CI, workflows, repo upkeep, or other maintenance work)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
- [ ] Refactor (restructures existing code without changing behavior, e.g. simplify logic, split modules, reduce duplication)
- [ ] This change requires a documentation update
## Mandatory Tasks
- [ ] Make sure you have self-reviewed the code. A decent size PR without self-review might be rejected.
## Checklist
<!-- Remove bullet points below that don't apply to you -->
- I haven't read the [contributing guide](https://github.com/langfuse/langfuse/blob/main/CONTRIBUTING.md)
- My code doesn't follow the style guidelines of this project (`pnpm run format`)
- I haven't commented my code, particularly in hard-to-understand areas
- I haven't checked if my PR needs changes to the documentation
- I haven't checked if my changes generate no new warnings (`npm run lint`)
- I haven't added tests that prove my fix is effective or that my feature works
- I haven't checked if new and existing unit tests pass locally with my changes
<!-- greptile_comment -->
<details open><summary><h3>Greptile Summary</h3></summary>
This PR adds a new "Versioned API Type Location" subsection to the backend dev guidelines, documenting the convention for placing versioned public API types under `packages/shared/src/features/<domain>/interfaces/api/v{N}/`. The described directory structure and canonical example (the `scores` feature) accurately match the current repository layout.
- Adds directory-tree illustration and three recommended files per version (`schemas.ts`, `endpoints.ts`, `validation.ts`), matching what already exists for `scores` v1–v3.
- Documents two anti-patterns to avoid: flat versioned files and placing types in `web/src/features/public-api/types/`.
- One minor cross-reference error: the text says "example below" but the cited code block appears above the new section.
</details>
<details><summary><h3>Confidence Score: 5/5</h3></summary>
Documentation-only change that accurately describes an existing pattern already used across the scores feature. Safe to merge.
The added section correctly reflects the actual directory structure in the repo (verified against packages/shared/src/features/scores/interfaces/api/). The only issue is a minor directional error in a cross-reference ('below' vs 'above') that does not affect the correctness of the convention being documented.
The single changed file .agents/skills/backend-dev-guidelines/references/routing-and-controllers.md contains a misdirected cross-reference on line 384 worth a quick fix before merge.
</details>
<details><summary><h3>Flowchart</h3></summary>
```mermaid
%%{init: {'theme': 'neutral'}}%%
flowchart TD
A[New versioned API feature] --> B{Where to place types?}
B -->|Correct| C["packages/shared/src/features/domain/interfaces/api/vN/"]
C --> D["schemas.ts — Zod request/response shapes"]
C --> E["endpoints.ts — Composed types"]
C --> F["validation.ts — Cross-field helpers"]
B -->|Anti-pattern 1| G["❌ Flat file: domain-api-v2.ts"]
B -->|Anti-pattern 2| H["❌ web/src/features/public-api/types/"]
C --> I["Export via @langfuse/shared"]
I --> J["Consumed by web/src/pages/api/public/..."]
```
</details>
<details><summary>Prompt To Fix All With AI</summary>
`````markdown
Fix the following 1 code review issue. Work through them one at a time, proposing concise fixes.
---
### Issue 1 of 1
.agents/skills/backend-dev-guidelines/references/routing-and-controllers.md:383-385
**Stale cross-reference direction**
The text says "in the example below", but the code block that uses `GetScoresQueryV1` / `GetScoresResponseV1` (imported from `@langfuse/shared`) lives in the **REST API Pattern** section immediately *above* this new section, not below it. Readers following the pointer downward will find nothing.
Change "example below" → "example above".
```suggestion
The scores feature (`packages/shared/src/features/scores/interfaces/api/`) is
the canonical example. The `GetScoresQueryV1` / `GetScoresResponseV1` symbols
imported from `@langfuse/shared` in the example above originate there.
```
`````
</details>
<sub>Reviews (1): Last reviewed commit: ["docs(agents): document versioned API typ..."](https://github.com/langfuse/langfuse/commit/f0ff59bd10074b5d86f5c84106a1972c53f99d87) | [Re-trigger Greptile](https://app.greptile.com/api/retrigger?id=36165686)</sub>
> Greptile also left **1 inline comment** on this PR.
<!-- /greptile_comment -->
* fix(security): enforce data-retention entitlement server-side in setRetention
The data-retention entitlement was only enforced in the UI
(ConfigureRetention.tsx disables the input client-side). The backing
projects.setRetention tRPC mutation validated project membership and the
project:update RBAC scope but not the plan entitlement, so a project
OWNER/ADMIN on a plan without data-retention could call the mutation
directly and set a destructive retentionDays, triggering the worker
deletion scheduler despite the feature not being available for the plan.
Add a server-side throwIfNoEntitlement check scoped to the project, after
the existing RBAC check, matching the pattern used in other routers. Keep
the UI check as presentation only. Add a server-side regression test
covering both the non-entitled (FORBIDDEN, not persisted) and entitled
(succeeds) plans.
Fixes LFE-10054
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: feedback
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## What does this PR do?
Phase 3 of [LFE-9539](https://linear.app/langfuse/issue/LFE-9539) — adds optional `fields=` query parameter to `GET /api/public/v3/scores`. Default is `core` when omitted; callers opt in to additional groups:
- **`details`** — `comment`, `configId`, `metadata` (always an object, never null)
- **`subject`** — `kind` (trace/observation/session/experiment), entity-specific `id`, and `traceId` (observation-kind only; absent, not null, when missing)
- **`annotation`** — `authorUserId`, `queueId`
Unknown group names → 400. `fields=trace` is reserved → 400.
### Design notes
- `deriveSubject` throws `InternalServerError` when a score reaches the trace branch with a null `traceId` — surfaces data integrity issues instead of silently returning a wrong ID; the row-drop handler in `listScoresV3ForPublicApi` catches it so callers never see a 500
- `ScoreDetailsV3.metadata` is non-nullable (implementation always coalesces to `{}`); OpenAPI and Zod schema aligned accordingly
- `nullable: true` is not emitted for `traceId`, `details`, `subject`, or `annotation` in the Fern draft — these fields are either present or absent, never null; optionality is expressed via absence from the `required` array. The generated `openapi.yml` is **not** updated in this PR because the endpoint lives in `fern/apis/server/_drafts/` and is excluded from Fern's discovery path; it will be regenerated when the draft is promoted to `definition/`
Stacked on #13996 (Phase 2).
### Impacted packages
- `@langfuse/shared` — `SCORE_FIELD_GROUPS_V3`, `fieldsParam` Zod transform, `ScoreDetailsV3` / `ScoreSubjectV3` / `ScoreAnnotationV3` schemas
- `web` — `deriveSubject` helper, `domainToV3` accepts `fields`, handler passes `fields` through, tests (`createSessionScore` / `createDatasetRunScore` imported for kind coverage)
- `fern/` — `fields` param and group types added to draft spec (`_drafts/scores-v3.yml`); generated `openapi.yml` unchanged (draft not yet promoted)
## Type of change
- [x] New feature (non-breaking change which adds functionality)
## Mandatory Tasks
- [x] Make sure you have self-reviewed the code. A decent size PR without self-review might be rejected.
## Checklist
- [x] My code follows the style guidelines of this project (`pnpm run format`)
- [x] I have commented my code where needed (particularly `deriveSubject` priority order and the draft-vs-definition OAS note)
- [x] My changes generate no new warnings (`pnpm run lint`)
- [x] I have added tests that prove my feature works (all four `subject.kind` values, field group combinations, reserved/unknown group rejection)
- [x] New and existing tests pass locally with my changes
## Summary
Phase 2 of [LFE-9539](https://linear.app/langfuse/issue/LFE-9539) — adds cursor-based pagination to \`GET /api/public/v3/scores\`.
Stacked on #13995 (Phase 1).
- **Cursor tuple**: \`(timestamp, id)\` — base64url-encoded JSON, matches the leading sort columns
- **\`meta.cursor\`** is present when more pages exist; absent on the final page — no COUNT query needed
- **Invalid cursor → 400** — decoded via Zod transform; bad base64 or wrong-shape JSON surfaces as \`InvalidRequestError("Invalid cursor format")\`
- **Fetch \`limit + 1\`** trick: service fetches one extra row; if it arrives, trim it and emit cursor from the last kept row
- **Cursor seek**: \`WHERE (s.timestamp, s.id) < (...)\` — matches the \`ORDER BY timestamp DESC, id DESC\` prefix exactly
- **\`s.event_ts\` removed from SELECT** — no longer consumed after cursor simplification; ORDER BY on \`event_ts\` still works without it in SELECT
### Why \`(timestamp, id)\` and not \`(timestamp, event_ts, id)\`?
The \`ORDER BY\` is \`timestamp DESC, id DESC, event_ts DESC\`. \`id\` is the stable second sort key across scores — putting it second in the cursor means the seek predicate \`(timestamp, id) < (...)\` exactly matches the sort prefix. \`event_ts\` is only a tiebreaker for \`LIMIT 1 BY\` deduplication of multi-version rows; after deduplication each \`(id, project_id)\` pair appears at most once, so \`event_ts\` never determines inter-score ordering. Using it in the cursor would cause page boundary instability when concurrent writes produce scores with the same timestamp but inverse \`event_ts\`/\`id\` orderings.
### Impacted packages
- \`web\` — handler, service, new \`types/scores.ts\` cursor encode/decode, tests
- \`@langfuse/shared\` — \`GetScoresV3\` gains \`cursor?: string\`; \`GetScoresResponseV3\` meta gains \`cursor?: string\`
- \`fern/\` + \`web/public/generated/api/openapi.yml\` — cursor added to request params and response meta
## Test plan
- [x] \`pnpm --filter web run lint\` — passed
- [x] \`pnpm --filter web run typecheck\` — passed
- [x] Fern generation — all checks passed
- [ ] Integration tests with \`LANGFUSE_ENABLE_SCORES_V3_API=true\` — require live ClickHouse
- cursor present when more pages exist, absent on final page
- full pagination without duplicates or skips
- invalid cursor → 400
- stale cursor → empty page, no cursor
🤖 Generated with [Claude Code](https://claude.com/claude-code)
* feat(scores): add v3 scores API phase 1 — polymorphic value, flag-gated
Introduces GET /api/public/v3/scores and GET /api/public/v3/scores/{scoreId}
behind LANGFUSE_ENABLE_SCORES_V3_API feature flag. Phase 1 delivers the
polymorphic `value` field (number/boolean/string/null by dataType), a hard
limit of 100, and the core response shape — no cursor or filters yet.
New files:
- packages/shared/src/features/scores/interfaces/api/scores-v3.ts: GetScoresV3,
GetScoreV3, GetScoresResponseV3, GetScoreResponseV3, APIScoreSchemaV3 Zod
- web/src/features/public-api/server/scores-api-v3.ts: standalone v3 service
(listScoresV3ForPublicApi, getScoreV3ForPublicApi, polymorphicValue). Does not
import v1/v2 service helpers.
- web/src/__tests__/server/scores-api-v3.servertest.ts: polymorphicValue unit
tests always run; integration tests gate on LANGFUSE_ENABLE_SCORES_V3_API=true
Fern update deferred to follow-on task (endpoint is flag-off by default).
Part of LFE-9539 / LFE-9926.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* feat(scores): add Fern v3 scores service definition and regenerate OpenAPI
Phase 1 of LFE-9539: adds fern/apis/server/definition/scores-v3.yml with
GET /v3/scores (limit-only) and GET /v3/scores/{scoreId} endpoints, using
a discriminated union on dataType for the polymorphic value field.
Regenerated openapi.yml via `npx fern generate --api server --group local`.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(scores): use z.discriminatedUnion for v3 response schema and align CORRECTION value
- Replace flat APIScoreSchemaV3 z.object with z.discriminatedUnion keyed on
dataType, matching the Fern/OpenAPI contract — invalid pairs like
{ dataType: 'NUMERIC', value: 'oops' } now fail Zod validation
- Remove unused ScoreDataTypeDomain import
- Fix CORRECTION polymorphicValue: || null → ?? null so empty string is
preserved rather than coerced to null
- Add symmetric "longStringValue" in score guard to match existing stringValue
guard in domainToV3
- Cast construction result as APIScoreV3 since ScoreDomain is flat and
TypeScript cannot verify the dataType/value pair statically
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* test(scores): add CORRECTION round-trip integration test
Covers the full GET /v3/scores path for a CORRECTION score with a
non-empty longStringValue, closing the coverage gap alongside the
existing NUMERIC, BOOLEAN, CATEGORICAL, and TEXT integration tests.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* chore: remove self-evident comment from env var declaration
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* test(scores): use it.skipIf for feature-flag-off test
Replaces the early-return guard with it.skipIf so Vitest reports a
real skip rather than a zero-assertion pass when the flag is enabled
in CI.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* fix(scores): drop tautological in-guards in domainToV3
ScoreDomain always has both stringValue and longStringValue as own keys,
so the 'key in score' checks and their null fallbacks were unreachable.
Replace with direct property access.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* fix(scores): route v3 single-score GET to ReadOnly ClickHouse cluster
getScoreV3ForPublicApi was calling the getScoreById wrapper which does
not forward preferredClickhouseService, causing single-score reads to
hit the primary cluster. Switch to _handleGetScoreById directly with
scoreScope="all" and preferredClickhouseService="ReadOnly", mirroring
ScoresApiService.getScoreById in v1/v2.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* fix(scores): use nullable<string> for CATEGORICAL/TEXT/CORRECTION value in Fern
optional<string> means the field may be absent; nullable<string> means
the field is always present but can be null — which matches the wire
format and the Zod schema (z.string().nullable()).
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* fix(scores): make v3 string value non-nullable for CATEGORICAL/TEXT/CORRECTION
The discriminated value is always a string for these data types: the
domain model guarantees stringValue is set for CATEGORICAL/TEXT, and
CORRECTION reads longStringValue which defaults to "". value can never
be null, so the contract should be string, not nullable<string>.
Switch Fern types to string and Zod to z.string(), and update the
now-inaccurate "or null" docs.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* fix(scores): throw on impossible null value in polymorphicValue
value is non-nullable for every v3 dataType, so coalescing a missing
string to null produced an invalid object that would only fail later
during response validation with an opaque error. Throw
InternalServerError at the source instead — for a missing stringValue
(CATEGORICAL/TEXT), a missing longStringValue (CORRECTION), or an
unknown dataType — and drop null from the return type.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* chore(scores): regenerate OpenAPI for non-nullable v3 string value
Regenerated web/public/generated/api/openapi.yml via
`fern generate --api server --group local` to reflect the
non-nullable CATEGORICAL/TEXT/CORRECTION value.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* fix(scores): filter is_deleted=0 in v3 list query and add dev env flag
The v3ListQuery was missing the `is_deleted = 0` filter, causing soft-deleted
scores to be returned in the GET /v3/scores response. Also selects
`is_deleted` to satisfy the ScoreRecordReadType schema. Adds the env var to
.env.dev.example alongside the other events-table feature flags.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* remove(scores): drop GET /v3/scores/{scoreId} endpoint
Scores can be retrieved by ID via GET /v3/scores?id={id}, making a
dedicated single-resource route redundant. Removes the route handler,
GetScoreV3/GetScoreResponseV3 schemas, getScoreV3ForPublicApi service
function, getByIdV3 Fern endpoint, and all associated tests. Regenerates
the OpenAPI spec. Also adds .playwright-mcp/ to .gitignore.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* fix(scores): remove dormant is_deleted filter from v3 list query
Per backend-dev-guidelines database-patterns doc: is_deleted on scores
is dormant — no write path sets it to 1 and all deletes use lightweight
DELETE. The filter was dead weight and an odd-one-out vs every other
score read in the codebase.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* refactor(scores): reorder ORDER BY to timestamp, id, event_ts in v3 list query
Puts id before event_ts so the 2-tuple cursor (timestamp, id) matches the
sort prefix exactly. event_ts remains as the final tiebreaker for LIMIT 1 BY
deduplication of multi-version rows. Eliminates same-millisecond page
boundary instability.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* chore(scores): move v3 Fern spec to _drafts so it stays out of public OpenAPI
Relocates fern/apis/server/definition/scores-v3.yml to
fern/apis/server/_drafts/scores-v3.yml. Fern auto-discovers every yml under
definition/; moving the file out keeps the v3 endpoint and schemas out of the
generated OpenAPI spec and downstream SDKs while the runtime endpoint behind
LANGFUSE_ENABLE_SCORES_V3_API is unchanged. Reinstate later with a git mv back
to definition/ and revert the import path one line.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* docs(scores): describe v3 availability without naming the env flag
Replaces the LANGFUSE_ENABLE_SCORES_V3_API reference in the v3 endpoint
docs with a consumer-facing description of preview availability. Customer-
facing Fern docs blocks should not name internal env vars or feature flags;
they describe behavior from the consumer's perspective.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* refactor(scores): split v3 schemas into folder matching v1/v2 layout
Replaces the flat packages/shared/src/features/scores/interfaces/api/scores-v3.ts
with a v3/ folder containing schemas.ts and endpoints.ts, matching the v1/ and
v2/ layout already in this directory. Renames GetScoresV3 to GetScoresQueryV3
for naming parity with GetScoresQueryV1 and GetScoresQueryV2. The private
foundation schema becomes ScoreFoundationSchemaV3 (was ScoreBaseV3) for the same
reason.
No behavior change; consumers using the @langfuse/shared barrel continue to
resolve the public symbols (APIScoreV3, APIScoreSchemaV3, GetScoresResponseV3)
unchanged.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* feat(scores): add v3 row-level response validation matching v2 pattern
Adds filterAndValidateV3GetScoreList in v3/validation.ts and wires it into
listScoresV3ForPublicApi so the v3 list endpoint mirrors the v1/v2 contract:
malformed rows are logged and dropped per row rather than crashing the whole
response with a 500. createAuthedProjectAPIRoute only validates the response
schema in development, so v3 had no production-side row validation before this
change.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* refactor(scores): tighten polymorphicValue dataType + add exhaustive check
Replaces the loose dataType: string parameter with the discriminated union
ScoreDataTypeType. With the tightened input, the default branch's
const _exhaustiveCheck: never = score.dataType assignment now does
compile-time work: if a new score dataType is added to the union, this line
fails to compile and points the developer at the missing case.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* refactor(scores): prune unused is_deleted and wire v3 row-drop observability
Removes `s.is_deleted` from the v3 list query SELECT — nothing in the read
path consumes it, and ClickHouse is columnar so the column read was wasted
I/O. Also routes filterAndValidateV3GetScoreList parse errors through
@langfuse/shared logger so silent row drops become observable in Datadog
instead of getting buried in stdout via console.error.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* test(scores): tighten v3 input tests and tidy off-path coverage
Genericizes the v3-disabled 404 message to "Not Found" so the disabled state
no longer discloses the feature flag's existence to unauthenticated probes.
Adds limit=0, limit=-1, limit=abc → 400 tests to lock the Zod query schema
contract. Drops the CORRECTION-with-null-longStringValue unit test because
the production read path coerces longStringValue to "" via ScoreDomain's
schema default — the live code can never reach the guard.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* chore(env): document LANGFUSE_ENABLE_SCORES_V3_API in prod example
Adds a commented optional entry in .env.prod.example matching .env.dev.example's
flag. Defaults to unset on self-hosted; self-hosters can enable explicitly once
the v3 API graduates from preview.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* refactor(scores): contain per-row errors in v3 list so one bad row drops alone
Wraps the records.map(domainToV3, ...) call in a try/catch loop so a throw
inside polymorphicValue (e.g. a CATEGORICAL/TEXT row with stringValue null
or a CORRECTION row with longStringValue null) log-drops the single row
instead of crashing the entire listing with 500. Mirrors the row-level
graceful-drop semantics that filterAndValidateV3GetScoreList provides for
Zod failures one layer down.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* chore: remove stray .playwright-mcp/ from .gitignore
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* ci: enable LANGFUSE_ENABLE_SCORES_V3_API in server test environment
Without this flag set, all v3 scores integration tests were silently
skipped in CI via the `maybe` = describe.skip pattern.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* test(scores): refactor v3 servertest — drop feature-flag guard, extract polymorphicValue unit tests, compress tabular cases with it.each
- Remove the `maybe` / `it.skipIf` / `env` pattern; the flag is now set
unconditionally in CI so tests always run.
- Extract `polymorphicValue` unit tests out of the servertest (they need
no DB/ClickHouse) into `server/unit/polymorphicValue.servertest.ts`,
which runs under the lighter server-unit Vitest project.
- Compress the four limit-validation cases and five data-type value cases
into `it.each` tables.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* remove duplicate error line
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
* feat(worker): add V4 self-hosted historic backfill chain
Ships the events_full / events_core ClickHouse tables and a 5-step
background-migration chain (M1-M5) that backfills them from existing
traces, observations, and dataset run items. All steps are dormant
behind env gates so v3 self-hosters upgrade without data movement
until they opt in.
Refs: LFE-8833
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(clickhouse): promote observations_batch_staging to production migration
Adds CH migration 0037 (clustered + unclustered) that ships the
staging table the dual-write pipeline relies on. With this in place,
a self-hoster can flip LANGFUSE_EXPERIMENT_INSERT_INTO_EVENTS_TABLE=true
after the V4 historic backfill chain completes and the existing
event-propagation job will populate events_full from the staging
table — no longer dev-only.
TTL is set to 48h (vs. 12h used internally) so self-hosters get a
multi-day recovery window if the propagation job stops catching up.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore: remove unnecessary env var
* chore: drop unused env flag
* chore: drop unused env flag
* chore: update env variable setup
* chore(clickhouse): split events_core MV into its own migration and add IF NOT EXISTS guards
Move the events_core_mv definition out of 0036 into a dedicated 0037 so the
materialized view can evolve independently of the table. Renumber the
observations_batch_staging migration from 0037 to 0038 to keep ordering
sequential. Add IF NOT EXISTS to every new CREATE TABLE / CREATE MATERIALIZED
VIEW so the migrations are idempotent on environments where the objects were
provisioned out-of-band (e.g. Langfuse Cloud).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore(clickhouse): use DROP VIEW for events_core_mv down migration
DROP TABLE works on materialized views in ClickHouse, but DROP VIEW is the
canonical form and validates that the object is actually a view, per the
ClickHouse docs: "DROP VIEW checks that [db.]name is a view."
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore(clickhouse): align unclustered events_full and observations_batch_staging comments with clustered
Mirror the trailing `index_granularity_bytes` rationale onto the unclustered
events_full migration and drop the LFE-7122 reference from the unclustered
observations_batch_staging migration so the two variants only differ by
ON CLUSTER + engine replication, as intended.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore: drop comment
* chore(prisma): rebase V4 background-migration timestamps from 20260509 to 20260521
Rename the five Prisma migration directories that register the V4 backfill
rows and update the matching name prefix used in each row plus the
self-registering upsert in the worker. The migrations were originally
authored on 2026-05-09; bumping to 2026-05-21 keeps them as the
last-applied migrations on rollout.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(worker): prevent dormant background migrations from head-of-line blocking
The envGate check ran after lock acquisition and aborted the run loop on
the first dormant row, so any later un-gated migration (or one whose gate
was on) never got a chance. Push the gate check into the findFirst
predicate via Prisma JSON path equality so dormant rows are invisible to
the manager and unrelated successors run normally.
Also rename gate env vars to a discoverable LANGFUSE_BACKGROUND_MIGRATION_
prefix, register them in the worker EnvSchema (default "false"), and
document the envGate mechanism in worker/src/backgroundMigrations/README.md.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore: update background migration sqls
* chore: cleanup backfillBase
* chore: refactor partition discovery
* chore: clean up observations pid tid rewrite
* chore: cleanup drop pid tid tables validation
* chore: use parts based observations migration
* chore: confirm column match-ups
* chore: cleanup
* chore: validation changes
* chore: default the event prop queue to on
* chore: add legacy write path skipping behaviour
* chore: remove useEventsTable fallbacks
* chore: move FTS indexes to dev-tables.sh
* chore(clickhouse): move v4 events tables back to dev-tables.sh
Self-hosters on ClickHouse < 24.5 cannot use `enable_block_number_column`
/ `enable_block_offset_column`, and `text` indexes need >= 25.x. Keeping
the v4 events_full / events_core / events_core_mv / observations_batch_staging
DDLs as migrations would block those upgrades, so move them back into the
dev-only script until v4 becomes mainline and we can require 25.12+.
FTS indexes and enable_full_text_index are inlined into the table
definitions instead of being applied via separate ALTER statements.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore: wording and formatting
* chore: extend eval tests
* chore: patch partition id
* chore: only stop merges on observations_pid_tid_sorting if no errors
* chore: corrections
* chore: address dependency chain in background migrations
* chore: add database filters
* chore: upgrade events_only opt-in message
* chore: patch ingestion api log message
* chore: only push bookmarked on root spans
* chore: stop backfill on data integrity issues
* chore: fallback to events table for getTraceById lookups in events_only mode
* chore: handle descendant limit exceptions with flushing
* chore: remove concurrency
* chore: patch tests and add observations by id wrapper
* chore: select an empty map as metadata
* chore: make test conditional
* chore: feedback
* chore(worker): extract V4 background migration chain into stacked branch
Moves the V4 historic backfill background migrations into a dedicated
stacked branch (feat/v4-historic-backfill-migrations) so the data-movement
logic can be reviewed independently from the events_only read-path changes.
Extracted (now lives in the stacked branch):
- Prisma migrations registering the V4 backfill chain
- createRootSpansFromTraces / rewriteObservationsToPidTidSorting /
backfillEventsFullFromObservations / backfillEventsFullFromDatasetRunItems /
dropPidTidSortingTables and utils/backfillBase
- BackgroundMigrationManager env-gate logic + README docs
- LANGFUSE_BACKGROUND_MIGRATION_* env gates
Retained here: removals of the old backfill scripts and all
LANGFUSE_MIGRATION_V4_WRITE_MODE / preview-opt-in read-path changes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: update doc strings and change session mode
* chore: toggle default for lazy materialization disablement
* chore: revert default for lazy materialization
* chore: feedback
* chore: adjust comment reference checks and health check
* chore: configure opt-in flags for fast (preview)
* chore: build shared for lint
* chore: lint
* chore: lint
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(dashboarding): support experiment filters, dimensions in observations view via preset
* feat(widget-form): render experiment as view rather than preset
* Revert "feat(widget-form): render experiment as view rather than preset"
This reverts commit 841d8437c85b27897e57ccd97b3d12118856713f.
* refactor: handle metadata filter appropriately
* chore: push
* chore: push
* feat(tests): add v2-only experiment filters for observations in dashboard widget tests
* feat: support experiment_name as entity dimension
* feat: enhance experiment widget configuration with new score types and filters
* feat: add hook and API for fetching experiment item filter options
* feat: implement experiment run score filtering and chart grid component
* refactor: move datasetRunId to scoresV2 dimension
* fix: add query parameter to include entity dimension relation tables in QueryBuilder
* refactor: remove TODO comments related to queryId and validation in InlineWidget component
* feat: enhance experiment score widgets with timestamp metrics for ordering
* fix: ensure toTimestamp is correctly set in ExperimentsTable component
* fix: enforce required schedulerId for query scheduling in InlineWidget component
* fix: add score name filter and improve experiment charts UX
- Add missing score name filter to createScoreWidgetConfig for proper
score filtering (fixes empty observation-level score charts)
- Extract chart constants, types, and utils to separate modular files
- Add horizontal scroll layout for charts grid on smaller screens
- Show "No data available" state for stale metric selections
- Display metric label in dropdown even when not in available options
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* test(experiments): add tests for getExperimentItemsFilterOptions
Add tests covering:
- Empty experiment IDs returns empty arrays
- Non-existent experiments returns empty arrays
- Trace-level numeric and categorical scores
- Observation-level numeric and categorical scores
- Multiple experiments aggregation
- Categorical value aggregation across experiments
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* feat(experiments): enhance score filter options with source and column definitions
- Added 'source' field to score filter options for both observation-level and trace-level scores.
- Updated the `buildScoreFilterOptionsQuery` to include 'source' in the selection and grouping.
- Enhanced `processScoreFilterOptionsResults` to track unique score columns by name, source, and data type.
- Modified `useExperimentItemsFilterOptions` to return full score column definitions for better integration with the ExperimentItemsTable.
- Refactored `ExperimentItemsTable` to utilize the new score column definitions for improved column visibility and filtering.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix(experiments): optimize score filter option query
* refactor(experiments): remove experimentMetadata from data model and related components
* feat(query): add highCardinality property to experiment score dimensions for improved data handling
* chore: push
* chore: push
* chore: push
* chore: push
* chore: push
* feat(query): implement high cardinality entity dimension validation in query processing
- Introduced a set of allowed high cardinality entity dimensions to enhance validation logic.
- Updated `getHighCardinalityDimensions` to include checks for entity dimensions.
- Added tests to ensure proper validation behavior for both allowlisted and non-allowlisted entity dimensions in various scenarios.
* feat(query): extend high cardinality entity dimensions and update related components
* chore: push
* fix(query): Remove unsupported aggregation types for datetime measure
* fix(query): expose datetime measures
* fixup: revert changes to queryBuilder
* feat(query): Implement bucket dimension handling in QueryBuilder
* feat(query): Enhance query validation for entity dimensions and high cardinality checks
* refactor(experiments): pass experiment id and name filters to comply with entity dimension requirements
* test(query): Update validation tests for high cardinality entity dimensions and refine query structure
* chore: push
* chore: push
* feat(experiments): add entity dimension label mapping to ExperimentChartSlot and InlineWidget for improved data presentation
* fix(charts): change order direction of min_timestamp in BASE_SCORE_CHART_CONFIG from descending to ascending
* fix(charts): update order direction of min_startTime and min_timestamp in chart configurations from ascending to descending
* fix(query): enforce entityDimension support only for v2 queries and update validation tests
* refactor(query): rename AppliedBucketDimension to AppliedBucketingDimension for consistency and clarity
* refactor(query): remove timestamp aggregation from scores and events observations views, update related tests and configurations
* fix(experiments): handle metric availability and enablement in ExperimentChartSlot component
* refactor(tests): remove max aggregation tests for scores and observations from queryBuilder tests
* feat(widget): enhance InlineWidget to order x-axis based on entity dimension label mapping for improved chart alignment
---------
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
OTLP/JSON encodes int64 fields (e.g. attribute intValues) as decimal
strings per the spec, since JSON numbers cannot represent the full
int64 range. The attribute converter only handled the Long object
produced by the protobuf decoder and a plain number, so a JSON-encoded
intValue like "7" fell through to `high * 2^32 + low` with `high`
undefined and produced NaN. That NaN propagated into resourceAttributes
/ metadata and was rejected by ingestion validation
(invalid_union on body.metadata), so spans from OTLP/JSON exporters such
as the OpenTelemetry PHP SDK were dropped.
Centralize int parsing in convertOtelIntValue, which handles the Long
object, plain number, and decimal-string representations and returns
undefined (falling back to JSON.stringify) instead of NaN for
unparseable values. Also coerce string-encoded doubleValues, keeping
non-finite specials ("NaN"/"Infinity") as JSON-valid strings.
Adds otelMapping coverage for string-encoded int/double attributes and a
regression test for the reported process.pid resource attribute.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replace the per-user `isBetaEnabled` gate on the export source picker
and export field groups with a deployment-level `eventsExportAvailable`
check (`isEnrichedBlobExportAvailable(isLangfuseCloud)`), so all Cloud
users on pre-cutoff projects see consistent form behaviour regardless of
their personal beta flag.
Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Adds a new "MCP & CLI" project settings page that introduces users to the
Langfuse Agent Skill, MCP server, and CLI. The page is content-only (no
functionality) with a short description, copyable install/usage snippets, and
links to the docs for each tool.
Also adds a dismissible informational banner on the organization overview page
highlighting that Langfuse works well with AI coding agents (Claude Code,
Codex, etc.) via the Agent Skill, MCP server, and CLI. The banner reuses the
existing Callout primitive (localStorage-backed dismissal with TTL).
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* fix(mixpanel): use empty distinct_id for events without a user
Events from automated evaluators have no user context — the trace they
belong to genuinely has no user_id, not a missing one. Using the
per-event insert_id as distinct_id was counting each evaluation as a new
unique Mixpanel user, inflating MTU billing.
Mixpanel's documented approach for events not attributable to any user is
an empty string distinct_id, which it distributes across shards without
creating user profiles or incurring MTU cost. This avoids both the
billing spike and the hot-shard risk that a shared sentinel like
"langfuse_unknown_user" would carry at high event volume.
The langfuse_user_id property (distinct from distinct_id) is set to
"langfuse_unknown_user" to match the documented property value.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* refactor(mixpanel): drop langfuse_user_id property override
The property already flows through via ...otherProps unchanged — adding an
explicit override risked breaking customers who depend on the current null
value. Only distinct_id needed fixing.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* feat(llm): support OpenAI Responses API connections
Add OpenAI LLM connection config for routing compatible endpoints through LangChain's Responses API.
* fix(llm): preserve VertexAI config parsing
Keep VertexAI config ahead of OpenAI config so empty VertexAI config is not defaulted as OpenAI Responses API config.
* feat(web): derive code eval support from dispatcher
Remove the public code eval UI flag and gate self-hosted code evaluator support from the configured dispatcher.
* fix(web): stabilize eval template validation dependencies
* fix(web): validate code eval table actions
* fix(web): repair code eval CI failures
* fix(web): stabilize code eval test run env
* fix(web): stabilize detail page list context
* test(worker): stabilize unrelated ingestion flake
Fix an unrelated flaky worker ingestion integration test that timed out in CI while validating code eval changes.
* avoid creating noise for intellij to pick up
* chore: standardize playwright-mcp output dir to /tmp/playwright-mcp
Update all references from the old `.playwright-mcp/` repo-local path to
`/tmp/playwright-mcp`, and remove the now-obsolete .gitignore entry.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* fix(security): enforce auditLogs:read and audit-logs entitlement for audit_logs batch exports
A project member with batchExports:create could request a batch export
of the audit_logs table, bypassing the stricter auditLogs:read RBAC scope
and audit-logs entitlement checks used by the normal audit log route.
The worker would then stream raw audit log rows to blob storage.
Add table-level authorization in batchExport.create: when tableName is
audit_logs, require both the audit-logs entitlement and auditLogs:read
project scope — mirroring the checks in auditLogs.allByProject.
Fixes LFE-10025.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* chore: remove comment from audit log batch export guard
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
* chore: update pnpm-lock and package.json for @codemirror/lang-javascript and adjust theme colors
* fix(evals): increase debounce time in useCodeEvalSourceValidation and simplify isValid calculation
* chore: push
* test(search): add failing tests for non-English full-text search (issue #11538)
Reproduces GitHub issue #11538: full-text search over trace/observation input/output
returns nothing for non-ASCII text because the OpenTelemetry / Python-SDK ingestion path
stores I/O as JSON with ensure_ascii=True (so 你好 is persisted as the literal 你好),
while clickhouseSearchCondition only matches the raw query string.
- web/src/__tests__/server/multilingual-fulltext-search.servertest.ts: integration tests
(via traces.all / generations.all tRPC + the ingestion API -> worker -> ClickHouse path)
with one case per writing system used by >=1M people (separate Simplified vs. Traditional
Chinese, separate Hiragana vs. Katakana), plus edge cases (input-only / output-only /
mixed Latin+CJK queries, astral-plane / surrogate-pair characters, observations, the
issue's exact {en,ar,zh} scenario) and a few unit assertions on the SQL builder.
- web/src/__e2e__/multilingual-search.spec.ts: Playwright e2e tests driving the Traces
table UI with non-ASCII full-text queries.
All 47 tests fail today (honest assertion failures, not missing-module errors); they pass
once clickhouseSearchCondition also matches the JSON-\uXXXX-escaped form of the query.
* fix(search): match \uXXXX-escaped content in full-text search (issue #11538)
Trace/observation input and output ingested through the OpenTelemetry / Python-SDK path
are persisted in ClickHouse verbatim as JSON serialised with ensure_ascii=True, so a value
like 你好 is stored as the literal 你好. Full-text search built input ILIKE '%你好%',
which never matched, so non-English content was unsearchable while ASCII worked.
clickhouseSearchCondition now also matches the JSON-\uXXXX-escaped form of the query
(astral code points -> UTF-16 surrogate pair) on the input/output columns. ASCII-only
queries are unchanged: the escaped form is identical, so no extra parameter or ILIKE clause
is emitted and the existing query plan is preserved. Plain-string columns (id/user_id/name)
are untouched.
* fixed test comments so they're not stale anymore
* test(search): remove redundant multilingual e2e spec
* perf(eval): skip FINAL and unused aggregations in checkTraceExistsAndGetTimestamp
The function is only used by evalService to decide whether a trace needs
evaluation. Drop the latency, usage_details, and cost_details aggregations
since they are not consumed, and remove FINAL from the traces and
observations reads. Updates are additive enough that a transient
non-matching state is acceptable in exchange for the performance gain.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore: remove outdated tests
* chore: remove outdated tests
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Summary
Fixes three production OOM incidents ([LFE-10031](https://linear.app/langfuse/issue/LFE-10031)) traced to `MEMORY_LIMIT_EXCEEDED` on `FROM observations FINAL` in the v3 blob storage export.
`FINAL` forces a k-way merge-sort across all parts before the time-range `WHERE` can be applied — measured: **75 GiB read for 1.08M rows in ~9 minutes** on a single 1-hour export window. The replacement `ORDER BY event_ts DESC / LIMIT 1 BY` subquery reads the same window in **0.5 s, reading 1.3 MB**.
* feat(ui): notification for code evals launch
* feat(ui): notification for mcp v2
* test(ui): make sidebar notifications test resilient to new entries
Derive the dismissed-notification list from the exported notifications
array instead of hardcoding launch-week IDs, so the GitHub star badge
test no longer needs an update each time a new notification is added.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Stabilize score config pagination order
* fix(score-configs): stabilize tRPC pagination order
* docs: prefer WSL and preflight local env
* chore: drop unrelated docs from score-config PR
---------
Co-authored-by: Nimar <l.nimar.b@gmail.com>
Co-authored-by: Ben Bachem <10088265+bezbac@users.noreply.github.com>
* feat(ui): notification for code evals launch
* test(ui): make sidebar notifications test resilient to new entries
Derive the dismissed-notification list from the exported notifications
array instead of hardcoding launch-week IDs, so the GitHub star badge
test no longer needs an update each time a new notification is added.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
ci(sdk): use repo token for sdk spec workflow
Remove the protected branches environment so the SDK API spec job uses the repository GH_ACCESS_TOKEN instead of the environment-scoped token.
* fix(llm): secure google model fetches
Route Google AI Studio and Vertex AI LangChain calls through validated secure fetch clients.
* fix(llm): secure openai and anthropic fetches
Route OpenAI, Azure OpenAI, and Anthropic LangChain calls through validated secure fetch.
* fix(llm): preserve google ai studio base url prefixes
* fix(llm): simplify secure google fetch clients
* fix(llm): preserve proxies and bump langchain core
Keep explicit undici dispatchers on secure LLM fetches so HTTPS_PROXY is honored, including Google clients. Bump @langchain/core to 1.1.48 to pick up the CJS uuid export fix.
* fix(llm): avoid dispatcher type conflicts
Keep proxy dispatchers opaque across secure fetch wrappers to avoid mixing undici and undici-types Dispatcher identities during typecheck.
* refactor(llm): clarify dispatcher handoff in secure outbound fetch
Document that any caller-provided dispatcher takes ownership of
connection-time safety, share the dispatcher-aware RequestInit type, and
harden the Google secure API client tests against NEXT_PUBLIC_LANGFUSE_CLOUD_REGION
env contamination.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(llm): drop redundant proxy fetchOptions and tighten secure fetch tests
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(llm): handle ChatGoogle mixed content blocks and thought parts
The unified @langchain/google SDK emits tool-calling responses as a mixed
array of `{type: "text"}` and `{type: "functionCall"}` blocks, and marks
reasoning text with `thought: true` instead of `type: "reasoning"`.
Accept a per-element content union and detect thought blocks so VertexAI
thinking + tool calling parses again.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore(deps): drop @langchain/core release-age exception
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(llm): consume standard contentBlocks for reasoning and tool calls
Route every chat model through @langchain/core's documented
AIMessage#contentBlocks accessor instead of inspecting raw provider
content. The registered translators normalize Bedrock reasoning_content,
Gemini thought parts, and Anthropic thinking/tool_use blocks into the
standard { type: "reasoning" | "text" | "tool_call" | ... } shape, so
splitAIMessage no longer needs per-adapter block-type sets or
undocumented field checks.
Tightens streaming to handle AIMessageChunk explicitly and replaces the
ad-hoc Anthropic/Google content unions in ToolCallResponseSchema with a
single standard ContentBlock shape.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test(llm): use real bedrock message instances
* fix(llm): ignore caller dispatchers in secure fetch
* fix(llm): pass google thinking options flat
* fix(llm): mark secureLlmFetch validation errors as non-retryable
Synchronous errors from validateLlmConnectionBaseURL and the
fetchWithSecureRedirects error classes carry no HTTP status, so the
catch block defaulted them to 500 + retryable and re-enqueued
permanently broken configs against the 24h eval-retry budget.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(llm): walk error cause chain for non-retryable pattern check
Anthropic/OpenAI/Azure SDKs wrap synchronous custom-fetch errors as
APIConnectionError { message: "Connection error.", cause: original },
so the secureLlmFetch validation patterns added in the previous commit
never matched for those three adapters. Walking the .cause chain (with
cycle guard) makes the non-retryable classification fire end-to-end.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(llm): surface secure fetch validation messages
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(dashboards): drop scoreName filter on traces and observations views
The dashboard exposes a global "Score Name" filter that gets routed to
every widget via dashboardUiTableToViewMapping. The traces and
observations entries mapped it to a scoreName column, but neither
traceView nor observationsView declares a scoreName dimension in the
query data model. queryBuilder.resolveDimension then hit the generic
*Name -> name fallback and silently rewrote the filter to traces.name
or observations.name -- so picking a score label appeared to match
trace names instead.
Removing the mappings lets the filter partition as unsupported on those
views (still applied correctly on scores-numeric / scores-categorical),
which avoids the silent miscarriage without changing the score-side UX.
Fixes LFE-9773.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(dashboards): hide scoreName and observationName filters on traces
Follow-up to the previous commit. The widget builder's filter-column
dropdown is fed by web/src/features/widgets/components/widgetFilterColumns.ts,
not by dashboardUiTableToViewMapping. "Observation Name" and "Score Name"
were in the unconditional base list, so they kept appearing for every
selectedView -- including traces, where neither is a real dimension.
Removing them from the mapping silenced the wrong-query bug but the UI
still misleadingly offered the options.
Gate both columns per-view:
- Observation Name: only on observations / scores-numeric / scores-categorical.
- Score Name: only on scores-numeric / scores-categorical.
Also drops observationName from the traces entry in
dashboardUiTableToViewMapping (same 1:n problem as scoreName: traceView
has no observationName dimension, so the *Name->name fallback would
silently rewrite to traces.name).
Refs LFE-9773.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(evals): handle mac format shortcut by physical key
* fix(evals): support python evaluator formatting
Enable the code evaluator format action for Python by reusing Ruff WASM and preserving the generated editor prelude.
* fix(evals): align code evaluator editor and runtime globals
Keep code evaluator language drafts separate in the editor and allow the same async helpers/globals in validation and local execution.
* fix(evals): cover code eval promise and byte helpers
Add Promise combinators and Uint8Array helpers to the synthetic validator declarations so editor validation matches the local runtime.
* fix(evals): expand code eval validation globals
Round out URL and array declarations and avoid helper type collisions in the synthetic TypeScript validator environment.
* feat(ui): add notification for agent skills launch; stack notifications
* test(ui): dismiss LW notifications in sidebar test
Stacked notifications only render the front card's content, so the GitHub
stars badge stays hidden while a higher-ranked Launch Week notification
is within its TTL. Pre-seed the dismissed list with the LW IDs so
github-star surfaces and the badge alt-text assertion remains stable.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat: introducing matches operator that is guaranteed to use FTS index.
* chore: refactor and consolidate a bit
* chore: addressig review and fixing CI
* chore: addressing review comments
* chore: clarifying API docs
* attach query id to logs
* ensure causes are correctly reported
* re-add error stack
* fix(blob-export): surface root cause and query_id in blob storage error logs
Before this change, blob export job failures reported only "Failed to upload
file to S3 (buffered)" — masking ClickHouse OOM errors and making triage
require manual trace correlation.
- Append [query_id: <id>] to errors thrown from queryClickhouseStream so
the query id survives the full error chain to the BullMQ job failure log
- Add formatErrorChain helper that walks .cause and joins messages with
"caused by", used in logger.error and the rethrown job error so both the
Datadog log and BullMQ failure entry show the full root cause inline
- Pass { stack } (not the Error) to logger.error to capture the stack
without triggering Winston's message-concatenation behaviour
- Copy the original stack onto the rethrown error so the queue processor
sees the real failure site, not the rethrow line
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* fix(mcp): inject root type: "object" for intersection/union inputSchema
Zod's JSON Schema converter emits intersections as bare `allOf` and unions
as `anyOf`, omitting the root `type` keyword. The MCP TypeScript SDK
validates `Tool.inputSchema.type` as `z.literal("object")`, so SDK-based
clients (e.g. Claude Code) reject these tools and silently drop them
from `tools/list`.
Normalize the generated JSON Schema so the root always declares
`type: "object"`. Draft-7 permits `type` alongside `allOf`/`oneOf`/`anyOf`
- all constraints must hold - so this is semantically a no-op for
already-conformant schemas.
This restores compatibility for `createScore`, `createScoreConfig`, and
`updateScoreConfig` introduced in #13781.
Fixes#13804
* refactor: Format code
* fix: Make `defineTool` type injection more robust
---------
Co-authored-by: Nimar <l.nimar.b@gmail.com>
Co-authored-by: Ben Bachem <10088265+bezbac@users.noreply.github.com>
Manual batch evaluation runs now bypass the LIVE/INACTIVE toggle so
users can re-evaluate historic data with a paused evaluator. Blocked
configs (auth/model issues) are still skipped.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(email): support AWS SES transport via default credential chain
Detect SES from a `ses://<region>` scheme in SMTP_CONNECTION_URL and build a
nodemailer transport on top of @aws-sdk/client-sesv2 using the default AWS
credential chain. SMTP and SES-over-SMTP (smtps://) paths are unchanged.
A new shared helper (createMailTransport / buildMailServerConfig) replaces the
duplicated createTransport(parseConnectionUrl(...)) call sites across the email
services and is wired through NextAuth's EmailProvider in web/src/server/auth.ts.
Refs langfuse/langfuse#13669
* docs(email): document ses:// URL form on SMTP_CONNECTION_URL
Refs langfuse/langfuse#13669
* fix(email): guard SES SentMessageInfo lacking rejected/pending fields
nodemailer's SES transport returns `{ envelope, messageId, response, raw }`
with no `rejected`/`pending` arrays, so `.concat()` on those undefined fields
threw on every successful SES send through the NextAuth password-reset path.
Refs langfuse/langfuse#13669
* test(email): fix expected SES transport name
nodemailer assigns `this.name = 'SESTransport'` (not "SES") at
ses-transport/index.js:23, so the assertion was wrong from the start.
Refs langfuse/langfuse#13669
* test(email): test parseSesRegion directly instead of probing SESv2Client
Inspecting `sesClient.config.region` returned the SDK's async region provider
(`AsyncFunction`) rather than the string we passed in, so the assertion
diverged from runtime behavior. Test the region extraction via the exposed
`__testing.parseSesRegion` helper instead and keep the transport-shape checks
limited to the dispatch boundary (transporter name + options-object shape).
No AWS credential resolution; full suite runs in ~15 ms.
Refs langfuse/langfuse#13669
---------
Co-authored-by: Steffen Schmitz <steffen@langfuse.com>
* feat(mcp): Make scores available via MCP
* Continue to accept empty string ids in the public scores API
* better error messages for agent
---------
Co-authored-by: Nimar <l.nimar.b@gmail.com>
* fix(tool-calls): parse available tools correctly from AI sdk
* add mapping test
* full otel mapping
* fix parsing
* add test
* only parse metadata if tools are found
* fix parse
* simplify
* comment todod
* fix available tools
* perf
* parse tools also from IO
* playground parse
* fix build
* fix
* adapters
* also make it work for new other adapters
* tighten remapping
* fix langgraph
* ordering
* working on the widget import/Export feature. Done for now but i am working on making it future-proof
* fixed minor edgecase in regards to malformed json
* minor fixes/changes
* working version
* lint fix
* safe commit
* safe commit
* changed error message to pass test
* sign off
* cleanup of classes and added filter-config. also added several code snippets to shared
* multi -> single upload
* undoing shared modules
* added claude preview changes
* more claude changes
* more claude changes
* claude review fix
* added claude review fix
* safe commit
* claude review fix
* cleanup
* more cleanup
* merge conflict hopefully resolved
* added claude correction
* merge fix
* fix(widgets): normalize traces imports and drop get parsing
* fix(widgets): narrow exported widget metric aggs
* fix(widgets): surface dropped import filters
---------
Co-authored-by: Nimar <l.nimar.b@gmail.com>
* feat(monitors): seeded the monitors package with schema and data validators
* feat(monitors): validate handlebars message templates against MonitorMessageContext
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test(monitors): align test imports + fixtures with refactored schema
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(monitors): address PR review feedback and codespell findings
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(monitors): coerce BigInt wire fields and add top-level barrel export
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(monitors): address remaining PR review feedback
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(monitors): added the monitor service
* fix(monitors): remove orphan features/monitor leftovers from MonitorService move
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* improvement(monitors): cleaned up types
* improvement(monitors): refactor into a feature partition
* refactor(monitors): drop handlebars template validator and message field
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(monitors): include `key` in sortFiltersCanonically canonical order
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(monitors): fix js docs typo nit
* fix(monitors): enforce nonnegative schedulerBatchId on the queue wire schema
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(monitors): use Object.hasOwn for query validation + correct threshold-order message
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(monitors): canonicalize set-semantics value arrays in sortFiltersCanonically
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(monitors): narrow input DTO status + orderBy column; reject histogram
- Add MonitorWriteStatusSchema (active|paused) and use it on
CreateMonitorInputSchema / UpdateMonitorInputSchema. `error-bad-query` is
scheduler-owned; callers can no longer forge a broken state or clear a
legitimately broken monitor without scheduler revalidation.
- Narrow MonitorListInputSchema.orderBy.column to the columns the admin
table actually sorts on (name/status/severity/createdAt). Without this,
an unknown column reached Prisma and raised a 500-class
PrismaClientValidationError instead of a clean 400.
- Reject `histogram` aggregation in isValidQuery — it returns a
bucket-array at the ClickHouse layer, but monitor thresholds are scalar.
Catch at the input boundary rather than failing in the worker.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(monitors): allow updatedAt as a sortable column on MonitorListInputSchema
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(monitors): expand orderBy columns + sort NULLS LAST on list
- Add severityChangedAt, alertedAt to the orderBy.column allowlist.
- Apply NULLS LAST unconditionally on list ordering so nullable columns
(alertedAt, severityChangedAt) sort intuitively in both directions; no-op
on non-nullable columns.
- Cover all 7 allowed columns in the input-schema test, plus a real-Postgres
integration test asserting NULLS LAST holds under ASC and DESC for both
nullable columns.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(monitors): reorder severity + status enums; drop unused message column
Reorder MonitorSeverity to UNKNOWN, NO_DATA, OK, WARNING, ALERT so DESC
sorts in attention-priority order (ALERT first). Reorder MonitorStatus to
PAUSED, ACTIVE, ERROR_BAD_QUERY so DESC reads as ERROR_BAD_QUERY → ACTIVE
→ PAUSED. Drop the unused `message` column (templating was removed
earlier; the column was kept then under Option A and is now retired).
Migration uses the canonical Postgres enum-swap pattern (CREATE _new, cast
column via text, rename _old, drop _old, rename _new). Default values
preserved. Zod enum order + service mapper cases reordered to match the
new canonical sequence.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(monitors): gate orderBy nulls last on the nullable column subset
Prisma's validator only accepts the { sort, nulls } object form on nullable
columns — unconditional nulls last raised PrismaClientValidationError for
the 5 non-nullable columns (name/status/severity/createdAt/updatedAt). Add
nullableOrderColumns typed against MonitorListOrderBy so the set stays in
sync with the sortable allowlist, attach nulls only on its members. Add
5 integration cases proving each non-nullable column list call goes
through.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs(monitors): trim stale JSDoc on MonitorListInputSchema
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test(monitors): cover schedulerBatchId invariance to property + value array order
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(monitors): reject non-stringObject metadata filters; lock scheduler batch id invariance under property order
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test(monitors): drop stale message field from prismaRow fixture
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(monitors): reject set-semantics filters with duplicate values; relocate JSDoc
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(monitors): reject scalar filters on array-typed dimensions; add positive set-semantics coverage
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(blob-export): gate pricing fields on model group, not usage
Previously input_price / output_price / total_price were enriched whenever
the `usage` field group was selected, and model_export columns (model_id,
provided_model_name, model_parameters) were fetched from ClickHouse for
both `model` and `usage` requests — even when only `usage` was asked for,
so the pricing lookup had the model_id it needed.
This split the semantics: `usage` gave you cost data, but enriched prices
came from asking for `usage` too. The model_export columns were then
silently dropped unless `model` was also selected.
After this change:
- `model` gates everything model-related: identification columns AND prices.
- `usage` covers only the cost/usage maps (usage_details, cost_details,
total_cost, usage_pricing_tier_name) — no pricing lookup, no model_export
fetch in ClickHouse.
- Selecting `usage` without `model` is cheaper (skips the model_export SQL
field set) and produces no price columns in the output.
Changes:
- worker handler: `includePricing` gate collapsed into `includeModelId`
- shared events.ts: `needsModelFields` drops the `|| usage` branch
- analytics-integrations labels: prices moved to model description, removed
from usage description
- unit tests: flip expectations to match new semantics
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* refactor(blob-export): inline model_export selection into the field group loop
Now that needsModelFields is just fieldGroups.includes("model"), the
two-step pattern (skip "model" in the loop, select model_export below)
is redundant. Collapse into a single conditional branch inside the loop.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* refactor(blob-export): remove dead model_export scrub in enrichObservationStream
The else-branch that deleted model_id/provided_model_name/model_parameters
was needed when model_export was fetched for usage-without-model requests
(so the pricing lookup had a model_id, then the columns were scrubbed before
output). That code path no longer exists after gating model_export on the
model group only — the columns are never present in the row when model is
absent, so the deletes were a no-op.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* fix(blob-export): add usage_pricing_tier_id to usage group description
The usage FieldSet projects usage_pricing_tier_id but the UI label omitted
it, causing the column to appear undocumented in exports. Pre-existing gap
surfaced by the PR review.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* fix(blob-export): remove FieldSetName cast that was not imported
The refactor introduced `group as FieldSetName` but FieldSetName is not
imported in events.ts. Revert to the plain `group` call that main used,
which satisfies the TypeScript overload without an explicit cast.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* fix comment
---------
Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* feat(dashboarding): support experiment filters, dimensions in observations view via preset
* feat(widget-form): render experiment as view rather than preset
* Revert "feat(widget-form): render experiment as view rather than preset"
This reverts commit 841d8437c85b27897e57ccd97b3d12118856713f.
* refactor: handle metadata filter appropriately
* chore: push
* chore: push
* feat(widget-form): keep only view agnostic filters on view change
* feat(tests): add v2-only experiment filters for observations in dashboard widget tests
* fix(dataModel): enable highCardinality for experimentName and experimentDatasetId fields
* revert: rm experiment_metadata as dimension
* fix: add highCardinality flag for experimentId field
* chore: push
* fix: correct import path for views type in widgetFilterPresets
The import path was using a non-existent local path @/src/features/query/types
instead of the correct shared package path @langfuse/shared/query. This was
causing TypeScript build failures.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
* feat(evals): add experiment item metadata mapping for experiment evaluators
* chore: push
* feat(evals): add experiment_item_metadata mapping and validation
- Updated PUBLIC_MAPPING_SOURCE_TO_INTERNAL_COLUMN to include experiment_item_metadata.
- Enhanced SUPPORTED_MAPPING_SOURCES_BY_TARGET to support experiment_item_metadata for the experiment target.
- Modified ExperimentEvaluationRuleMappingSource to include experiment_item_metadata in the schema.
* fix: protect against correct mapping in front-end
* fix: protect against correct mapping in back-end
* Revert "fix: protect against correct mapping in back-end"
This reverts commit 6f7a1f4cc11b01a8233e055063610608d3dca2ef.
* fix(evals): include experiment metadata in batch eval stream
* fix(evals): update fern mapping source contract
## Summary
The v2 observations endpoint always returns `modelId`, `inputPrice`, `outputPrice`, and `totalPrice` on every response row. These fields are populated when the `model` field group is requested; otherwise they are `null`. They were flowing through the Zod `.loose()` passthrough undeclared, making them invisible in the API contract.
- **`fern/apis/server/definition/commons.yml`**: Add `modelId`, `inputPrice`, `outputPrice`, `totalPrice` to `ObservationV2` as `nullable<T>` (always present in the response, not optional). Includes gating docs explaining the `model` field group condition.
- **`web/src/features/public-api/types/observations.ts`**: Declare the three price fields in the `APIObservationV2` Zod schema alongside the existing `modelId`.
- **`web/src/pages/api/public/v2/observations/index.ts`**: Normalize `Decimal` price values to `number` in the route handler for wire-format parity with v1 (mirrors `transformDbToApiObservation`).
- **`packages/shared/src/domain/observation-field-groups.ts`**: Expand docstring with enrichment field gating notes and code cross-references.
* fix(blob-storage): harden endpoint connection validation
Block blob storage DNS rebinding for S3-compatible and Azure endpoints while keeping self-hosted validation opt-in via allowlist env vars.
* fix(blob-storage): address endpoint validation review feedback
* fix(blob-storage): keep secure storage agents alive
* test(blob-storage): run worker integration suite as self-hosted
* test(blob-storage): run integration suite as self-hosted
* feat(monitors): add Monitor Prisma schema and migration
* feat(monitors): add UNKNOWN severity as default for cold-start monitors
* feat(monitors): decouple Monitor.view from DashboardWidgetViews
* fix(monitors): align Monitor.id with cuid convention and wire createdBy/updatedBy FKs
## Summary
Upstream `@playwright/mcp` renamed `--save-trace` to `--save-session`. The current `latest` build (v0.0.75) rejects the old flag with `error: unknown option '--save-trace'` and the server exits immediately, so Claude Code (and any other MCP client launching the server via `.mcp.json`) reports `Failed to reconnect to playwright`. This blocks the `frontend-browser-review` skill end-to-end.
Swapping to `--save-session` is upstream's straight rename and keeps the same intent: Playwright MCP writes its session artifacts (including traces) under `--output-dir .playwright-mcp`, which is what the skill (`.agents/skills/frontend-browser-review/SKILL.md`) tells reviewers to inspect on failure.
## Impacted packages
- `.agents/config.json` — canonical MCP server config (source of truth)
- `.agents/README.md` — illustrative snippet kept in sync to avoid re-introducing the stale flag via copy-paste
Generated provider configs (`.mcp.json`, `.claude/`, `.codex/`, `.cursor/`, `.vscode/`) are regenerated by `pnpm run agents:sync` and remain gitignored per the agent-setup contract — no changes need committing there.
## Verification
- `pnpm run agents:sync` — regenerates all provider shims with the new flag
- `pnpm run agents:check` — clean
- Manual launch with the new args (`npx -y @playwright/mcp@latest --isolated --save-session --output-dir .playwright-mcp --test-id-attribute data-testid`) — process stays alive past handshake (old `--save-trace` exited with code 1)
- Live confirmation: with the fix applied locally, the Playwright MCP tools loaded successfully in my Claude Code session, whereas `/mcp` had previously reported `Failed to reconnect to playwright`
<!-- greptile_comment -->
<details open><summary><h3>Greptile Summary</h3></summary>
This PR corrects the Playwright MCP flag in the agent configuration from `--save-trace` to `--save-session`, which is the correct flag for automatic session recording per the Playwright MCP documentation.
- Both `.agents/config.json` (the canonical source) and `.agents/README.md` (which embeds the current JSON shape inline) are updated consistently, keeping documentation and config in sync.
- Generated shim files (`.claude/settings.json`, `.cursor/mcp.json`, etc.) are not committed to the repo and are regenerated from `.agents/config.json` via `pnpm run agents:sync`, so no further changes are needed in this PR.
</details>
<details><summary><h3>Confidence Score: 5/5</h3></summary>
Safe to merge — both changed files are updated consistently and the replacement flag is documented as correct by Playwright MCP.
The change swaps a single CLI flag in two files that are intentionally kept in sync (the canonical config and its embedded README snapshot). The flag --save-session is confirmed in the Playwright MCP documentation as the correct option for automatic session recording. Generated shim files are not committed and will pick up the corrected flag on the next pnpm install or agents:sync run.
No files require special attention.
</details>
<details><summary><h3>Flowchart</h3></summary>
```mermaid
%%{init: {'theme': 'neutral'}}%%
flowchart TD
A[".agents/config.json\n(canonical source)"] -->|pnpm run agents:sync| B["Generated Shim Files\n(.claude/settings.json\n.cursor/mcp.json\n.vscode/mcp.json\n.mcp.json etc.)"]
A -->|inline snapshot| C[".agents/README.md\n(documentation)"]
B --> D["Playwright MCP Server\nnpx @playwright/mcp@latest\n--isolated\n--save-session\n--output-dir .playwright-mcp\n--test-id-attribute data-testid"]
style D fill:#d4edda,stroke:#28a745
```
</details>
<sub>Reviews (1): Last reviewed commit: ["fix(agents): use --save-session for Play..."](https://github.com/langfuse/langfuse/commit/789bceb39b392e4a677e455c9d819ae864a17e8a) | [Re-trigger Greptile](https://app.greptile.com/api/retrigger?id=32588333)</sub>
<!-- /greptile_comment -->
## Summary
Follow-up to [LFE-9688](https://linear.app/langfuse/issue/LFE-9688) /
[#13627](https://app.graphite.com/github/pr/langfuse/langfuse/13627).
Tracked as [LFE-9830](https://linear.app/langfuse/issue/LFE-9830).
- After PR #13627 merged, post-cutoff Cloud projects saw a one-option Export Source dropdown. Team feedback: hide the field entirely.
- Wrap the FormField in `{showExportSourceField && (...)}` where `showExportSourceField = isBetaEnabled && !isPostCutoffCloud`. Composes with the existing `isPostCutoffCloud` derivation — `isLegacyBlobExportAllowed` is called exactly once per render.
- Form value stays pinned to `EVENTS` via the existing `defaultValues` / `reset` logic so submission is valid even with the field hidden (`react-hook-form` retains values of unmounted fields by default).
- Drops dead `availableExportSourceOptions` filter, the unreachable `isPostCutoffCloud` arm of `FormDescription`, and the unused `LEGACY_BLOB_EXPORT_SOURCES` import.
### Why no unit test
The rendering condition is a single-line AND of two existing booleans. The cutoff-bracket logic lives in `isLegacyBlobExportAllowed` (covered by its own tests in the shared package). Browser review of the affected page is the intended safety net — see the Test plan below.
### CI fix (fixup commit)
This PR also removes `--experimental-cli` from the `prettier-check` CI job (`pipeline.yml`). The flag's glob resolver treats bracket characters in Next.js dynamic-route paths (e.g. `[projectId]`) as glob character classes, causing exit 123 ("No files matching the given patterns were found") for any PR that touches a file under such a directory. `blobstorage.tsx` lives under `[projectId]`, which is what triggered the failure here. Standard prettier resolves explicit file paths correctly; the parallelism and ephemeral cache that `--experimental-cli` adds provide no practical benefit when checking a handful of changed files per PR.
### Impacted packages
- `web` — single page component, net 10+/26−.
- `.github/workflows/pipeline.yml` — CI prettier-check fix.
## Test plan
- [x] `pnpm --filter web run typecheck`
- [x] `pnpm --filter web exec vitest run --project=client src/__tests__/blob-storage-form-field-groups.clienttest.ts` — 5/5 passed (regression check on related form schema)
- [x] Reproduced prettier-check failure locally with the old command; confirmed fix passes with the new command.
- [ ] Browser review on Cloud with a seeded pre-cutoff and a seeded post-cutoff project (assert Export Source field hidden in the post-cutoff case, visible in the pre-cutoff case)
- [ ] Self-hosted parity check (`LANGFUSE_CLOUD_REGION` unset → field visible)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
## Summary
Follow-up to [LFE-9688](https://linear.app/langfuse/issue/LFE-9688) /
[#13627](https://app.graphite.com/github/pr/langfuse/langfuse/13627),
which intentionally scoped the cutoff gate to blob-storage and listed
PostHog / Mixpanel under "Out of scope". Tracked as
[LFE-9838](https://linear.app/langfuse/issue/LFE-9838); planning doc:
`ideabox/Implementations/Proposed/2026-05-18 extend-cutoff-gate-to-posthog-mixpanel.md`.
- Post-cutoff Cloud projects (`createdAt >= 2026-05-20`) now see the Export Source field hidden in PostHog and Mixpanel settings pages (form value pinned to `EVENTS` via `defaultValues`).
- The matching tRPC `update` mutations reject any legacy `exportSource` (`TRACES_OBSERVATIONS`, `TRACES_OBSERVATIONS_EVENTS`) for post-cutoff Cloud projects with `BAD_REQUEST`.
- Pre-cutoff Cloud projects and self-hosted deployments keep full choice — no behavior change.
- Pure parity work: reuses the shared `isLegacyBlobExportAllowed` predicate and `assertLegacyBlobExportSourceAllowed` guard. No new constants, no shared-package edits, no public REST surface to gate (neither integration has one).
- The `LEGACY_BLOB_EXPORT_*` / `assertLegacyBlobExportSourceAllowed` names retain their "Blob" prefix; renaming is deferred to a separate cleanup PR (see planning doc, Decision #2).
### Impacted packages
- `web` — two settings pages, two routers, one extended servertest, one new servertest. No other package touched.
## Test plan
- [x] `pnpm --filter web run typecheck`
- [x] `pnpm --filter web exec vitest run --project=server src/__tests__/server/posthog-integration.servertest.ts src/__tests__/server/mixpanel-integration.servertest.ts` — 11/11 passed (PostHog 6, Mixpanel 5)
- [x] Browser review on dev (Playwright MCP): post-cutoff cloud projects (dev `.env` overrides cutoff to `2020-01-01`) — Export Source hidden on both PostHog and Mixpanel settings pages, Enabled switch + other form fields still render correctly
- [ ] Browser review with pre-cutoff Cloud (toggle `NEXT_PUBLIC_LANGFUSE_BLOB_EXPORT_CUTOFF` to a future date, restart dev server)
- [ ] Self-hosted parity check (`LANGFUSE_CLOUD_REGION` unset → field visible for any `createdAt`)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- greptile_comment -->
<details open><summary><h3>Greptile Summary</h3></summary>
This PR extends the legacy export source cutoff gate (originally applied to blob-storage in LFE-9688) to the PostHog and Mixpanel analytics integrations. Post-cutoff Cloud projects (`createdAt >= 2026-05-20`) can no longer save a legacy `exportSource` value; the field is hidden in the UI and pinned to `EVENTS`, while the tRPC `update` mutations enforce the same rule server-side.
- **Routers**: Both `posthogIntegrationRouter` and `mixpanelIntegrationRouter` gain the same `assertLegacyBlobExportSourceAllowed` guard that already protects the blob-storage router; the gate is correctly placed before the audit log and DB write.
- **UI pages**: Both settings pages derive `isPostCutoffCloud` from `useQueryProject` + `isLegacyBlobExportAllowed`, hide the Export Source `FormField` when that flag is true, and pin the form default to `EVENTS` — exactly mirroring the blob-storage page pattern.
- **Tests**: New `servertest` files (and an extended PostHog file) cover all five gate scenarios (pre-cutoff Cloud allow, two legacy-source rejections, `EVENTS` allow, self-hosted bypass) using a shared `buildSession` helper refactored from the existing SSRF test.
</details>
<details><summary><h3>Confidence Score: 5/5</h3></summary>
Safe to merge — the server-side gate is correctly placed and always reachable, the UI correctly hides and pins the field, and tests cover all five gate scenarios for both integrations.
The change is tightly scoped: two routers gain the same guard already proven in blob-storage, two settings pages hide a single field for post-cutoff Cloud projects, and tests exercise every code branch. No new public API surface is added, and self-hosted / pre-cutoff behaviour is unchanged.
No files require special attention. The only observation is a duplicated buildSession helper in the two new test files, which is a maintenance concern rather than a functional one.
</details>
<details><summary><h3>Flowchart</h3></summary>
```mermaid
%%{init: {'theme': 'neutral'}}%%
flowchart TD
A[User submits PostHog / Mixpanel settings form] --> B{exportSource provided?}
B -- "always truthy after Zod .default()" --> C[Fetch project.createdAt from DB]
C --> D{Is legacy export source?}
D -- No: EVENTS --> E[Allow — skip gate]
D -- Yes: TRACES_OBSERVATIONS / TRACES_OBSERVATIONS_EVENTS --> F{isCloud AND project.createdAt >= cutoff?}
F -- No: self-hosted OR pre-cutoff --> G[Allow]
F -- Yes: post-cutoff Cloud --> H[Throw InvalidRequestError → BAD_REQUEST]
E --> I[Audit log + DB upsert]
G --> I
```
</details>
<details><summary>Prompt To Fix All With AI</summary>
`````markdown
Fix the following 1 code review issue. Work through them one at a time, proposing concise fixes.
---
### Issue 1 of 1
web/src/__tests__/server/mixpanel-integration.servertest.ts:13-44
**Duplicated `buildSession` helper across test files**
The `buildSession` function in this file is byte-for-byte identical to the one added to `posthog-integration.servertest.ts`. If the session shape ever changes (e.g., a new required project field), both copies need updating in sync. Consider extracting it to a shared test utility (e.g., `web/src/__tests__/server/fixtures/session.ts`) so there is a single source of truth.
`````
</details>
<sub>Reviews (1): Last reviewed commit: ["feat(analytics-integrations): extend leg..."](https://github.com/langfuse/langfuse/commit/6418f93afafa9dede9899f65747256c8e42fd309) | [Re-trigger Greptile](https://app.greptile.com/api/retrigger?id=32589990)</sub>
<!-- /greptile_comment -->
* refactor(shared): promote query feature to @langfuse/shared (LFE-9806)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(shared): import query server deps from source modules
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore(shared): drop no-barrel comment; qualify mapDashboards path
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(shared): inject rootEventCondition threshold into QueryBuilder
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(web): move dashboardUiTableToViewMapping back to dashboard/lib
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(shared): flatten features/query — drop server/ subdir
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor: prefer direct subpath imports for query module
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* revert(shared): drop QueryBuilder rootEventCondition override; self-import env
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(shared): read rootEventCondition threshold from process.env
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(shared): address review feedback on query module
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* revert(shared): rolled back diff to bare minumum
* fix(shared): added back an explicit query export following AGENTS.md guildelines
* fix(shared): added a formal threshold hours overried to side step the dual package hazard
* fix(web): missing query imports in execute query stream
* fix(shared): split query server-only files under @langfuse/shared/query/server
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(web): route QueryBuilder/executeQuery imports in 3 tests via /query/server
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Summary
- Cloud projects created on or after **2026-05-20T00:00:00Z** can no longer use `LEGACY_TRACES_OBSERVATIONS` or `LEGACY_TRACES_AND_ENRICHED_OBSERVATIONS`. Attempts via tRPC or the public REST API return `BAD_REQUEST` / HTTP 400.
- Self-hosted deployments and projects created before the cutoff are fully unaffected.
- Single shared helper (`assertLegacyBlobExportSourceAllowed`) enforces the rule identically on both write surfaces.
- Settings UI hides legacy options and defaults to `OBSERVATIONS_V2` for post-cutoff Cloud projects, with an inline message explaining the restriction.
- `project.createdAt` added to the NextAuth session so the UI can derive the gate without an extra DB round-trip.
## What does this PR do?
Fixes a wire-format leak in the V2 observations endpoint where `traceName`, `tags`, `release`, `userId`, `sessionId`, `bookmarked`, and `public` appeared as `null` keys in responses even when the client did not request the `trace_context` field group.
**Root cause:** `convertEventsObservation` in `observations_converters.ts` used unconditional `record.x ?? null` assignments for those seven fields regardless of the `complete` flag. When ClickHouse omits a column from the projection (because the field group was not requested), the value is `undefined`, and `undefined ?? null === null` — so the key was always emitted. The peer converter `convertObservationPartial` already uses conditional spreads (`...(record.x !== undefined && { x: record.x })`) to enforce this discipline; `convertEventsObservation` deviated from that pattern.
**Fix:**
- Split the `complete`/partial branches in `convertEventsObservation`. The `complete: true` (V1) branch keeps the unconditional defaults since V1 always returns all fields. The `complete: false` (V2) branch gates each extra field on presence, matching `convertObservationPartial`.
- Tighten the contract test assertion in `observations-api-v2.servertest.ts`: removes the `?? undefined` softening that allowed `null` values to pass `toBeUndefined()`.
- Add a unit test for `convertEventsObservation` directly — no such test existed before.
## Type of change
- [x] Bug fix (non-breaking change which fixes an issue)
## Impacted packages
- `packages/shared` — `observations_converters.ts`
- `web` — contract test + new unit test
## Verification
- `pnpm --filter @langfuse/shared run lint` ✓
- `pnpm --filter @langfuse/shared run typecheck` ✓
- CI green (tests-web, tests-worker, e2e)
<!-- greptile_comment -->
<details open><summary><h3>Greptile Summary</h3></summary>
This PR fixes a wire-format leak in the V2 observations endpoint where seven `trace_context` fields (`traceName`, `tags`, `release`, `userId`, `sessionId`, `bookmarked`, `public`) appeared as explicit `null` keys even when the client did not request that field group. The root cause was `undefined ?? null === null` in the single shared code path.
- **Core fix** (`observations_converters.ts`): Splits `convertEventsObservation` into separate `complete: true` (V1) and `complete: false` (V2) branches. The V1 path keeps unconditional `?? null` defaults; the V2 path gates each field on `!== undefined` using conditional spreads, matching the pattern already used in `convertObservationPartial`.
- **Contract test** (`observations-api-v2.servertest.ts`): Tightens the assertion from `obs[field] ?? undefined` (which let `null` pass `toBeUndefined()`) to a direct `obs[field]` check, so the test now correctly fails when a field leaks as `null`.
- **New unit test** (`observations-converters.servertest.ts`): Adds direct coverage for `convertEventsObservation` that was previously missing, testing both branches with absent, null, and non-null field values.
</details>
<details><summary><h3>Confidence Score: 4/5</h3></summary>
The production change to `observations_converters.ts` is safe: the fix is minimal, well-scoped, and the conditional-spread pattern it introduces for the V2 path already exists in the peer converter.
The converter change and the contract-test tightening are both correct. The new unit test has two TypeScript type incompatibilities (`tags: null` where only `string[] | undefined` is valid, and `undefined` passed for a required `RenderingProps` parameter). Both are caught only at type-check time — the PR description reports running typecheck only for `@langfuse/shared`, not for the `web` package where the new test lives. The issues don't affect runtime or production behaviour, but they leave the test file in a state that fails strict type-checking.
web/src/__tests__/server/unit/observations-converters.servertest.ts — two call-site type errors; all other files are clean.
</details>
<details><summary>Prompt To Fix All With AI</summary>
`````markdown
Fix the following 2 code review issues. Work through them one at a time, proposing concise fixes.
---
### Issue 1 of 2
web/src/__tests__/server/unit/observations-converters.servertest.ts:137-143
The `makeRecord` override passes `tags: null`, but `tags` in `EventsObservationRecordReadType` is typed as `z.array(z.string()).optional()` — i.e. `string[] | undefined` — which is not nullable. Passing `null` here is a TypeScript type error that would surface under `tsc --noEmit` on the `web` package. The intent of the test (verifying `?? null` defaulting) is better expressed by omitting `tags` entirely (letting it be `undefined`) or by asserting that the complete path emits `null` when the field is absent rather than null in the row.
```suggestion
const record = makeRecord({
user_id: null,
session_id: null,
trace_name: null,
release: null,
// tags is omitted → undefined in the record; the converter defaults it to null
});
```
### Issue 2 of 2
web/src/__tests__/server/unit/observations-converters.servertest.ts:70-72
The overload signatures for `convertEventsObservation` declare `renderingProps` as a required `RenderingProps` parameter (not `RenderingProps | undefined`). Passing `undefined` here works at runtime (the implementation has a default value), but TypeScript checks call sites against the overloads and will flag this as a type error. The same pattern appears at the other `convertEventsObservation` call sites in this file. Passing the exported `DEFAULT_RENDERING_PROPS` is the idiomatic fix and makes the intent explicit — note that the import for `DEFAULT_RENDERING_PROPS` from `@langfuse/shared/src/server` would also need to be added.
```suggestion
const result = convertEventsObservation(record, DEFAULT_RENDERING_PROPS, false);
for (const field of TRACE_CONTEXT_FIELDS) {
```
`````
</details>
<sub>Reviews (1): Last reviewed commit: ["fix(test): import convertEventsObservati..."](https://github.com/langfuse/langfuse/commit/5074fe72723a7812e8ff7f3b6ff0f9d0820ed316) | [Re-trigger Greptile](https://app.greptile.com/api/retrigger?id=32323301)</sub>
<!-- /greptile_comment -->
The self-service SSO form exposes `idToken` but not `scope`. When an
admin sets `idToken: false` (for IdPs that release email only via the
userinfo endpoint), the stored config inherits the runtime default
`openid email profile` scope. NextAuth then chooses
`client.oauthCallback()` (idToken === false branch), which openid-client
refuses with
"id_token detected in the response, you must use client.callback()
instead of client.oauthCallback()"
because the IdP still returns an id_token whenever `openid` is in scope.
Normalize the stored scope at write time in `ssoConfig.save`: when the
saved provider is `custom` and `idToken === false`, drop the `openid`
token from the scope (falling back to `email profile` if stripping
leaves it empty). This also handles the merge case where the existing
config (often written via the legacy admin endpoint) supplied a scope
that the new save needs to bring into line.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The events.all read path returned 500 when an observation's
model_parameters contained a non-JSON sentinel string (e.g. Python
SDK v4's "<not serializable object of type: dict>"). Replace the bare
JSON.parse in convertObservationPartial with parseJsonPrioritised so
unparseable values fall through to the raw string instead of throwing
for the entire row. This converter feeds both convertObservation and
convertEventsObservation.
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
## Summary
Stacked on top of [#13617](https://app.graphite.com/github/pr/langfuse/langfuse/13617) (the `OBSERVATION_FIELD_GROUPS` / `BLOB_EXPORT_FIELD_GROUPS` split). Exposes `trace_context` on the public `/api/public/v2/observations` API after that split, restoring the group to the v2 contract with a documented surface (it had been incidentally available via the un-narrowed constant before #13617).
- Adds `trace_context` back to `OBSERVATION_FIELD_GROUPS` — the narrowed list of public v2 groups, distinct from `BLOB_EXPORT_FIELD_GROUPS`. `tools` stays only on the blob-export side as before.
- Fern source documents `trace_context` in both the field-selection list and the `Available groups` line on the `fields` query parameter. Regenerated `openapi.yml` propagates to the SDK clients.
- Columns exposed: `tags`, `release`, `traceName` (denormalized trace metadata). `usagePricingTierName` was moved to the `usage` group by #13617 and is documented there.
### How does this differ from pre-#13617 behavior?
Before #13617, `trace_context` was already accepted at runtime via the Zod filter against `OBSERVATION_FIELD_GROUPS` — but the Fern docstring listed only 9 of 11 groups, so it was undocumented. #13617 narrowed the constant for safety (separating v2 API from blob-export selection). This PR restores `trace_context` on the v2 side intentionally, with explicit Fern documentation, while leaving `tools` blob-export-only.
### Test coverage
Extends the parametrized `field group contract` loop in `observations-api-v2.servertest.ts` with a `trace_context` row that asserts the three denormalized fields flow through when `fields=trace_context` is requested. Uses fixture values for `tags`, `release`, `traceName` so a null regression is caught.
### Impacted packages
- `@langfuse/shared` — re-adds `trace_context` to `OBSERVATION_FIELD_GROUPS`; updates the docblock to reflect that only `tools` is now in the broader blob-export set
- `web` — extends servertest coverage; regenerated `openapi.yml`
- `fern` — observations endpoint docstring
## Test plan
- [x] `pnpm --filter web run typecheck`
- [x] `dotenv -e .env -- pnpm --filter web exec vitest run observations-api-v2` — 27/27 passed
- [ ] Manual: verify regenerated SDKs (Python, TypeScript) include `trace_context` as a valid `fields` value
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- greptile_comment -->
<details open><summary><h3>Greptile Summary</h3></summary>
This PR re-adds `trace_context` to `OBSERVATION_FIELD_GROUPS` so that `tags`, `release`, and `traceName` are exposed on the public `/api/public/v2/observations` endpoint, and documents the group in both the Fern definition and the generated OpenAPI spec.
- **`packages/shared`**: `trace_context` is appended to `OBSERVATION_FIELD_GROUPS`; docblock updated to reflect `tools` is now the only blob-export-only group.
- **`fern` / `openapi.yml`**: `trace_context` and its three columns are added to the field-selection list and the `Available groups` doc line.
- **Server test**: parametrized contract test extended with a `trace_context` row and fixture values for `tags`, `release`, `traceName`, but `ALL_NON_CORE_FIELDS` (the sentinel list used for absence checks) is not updated, leaving a gap in isolation coverage.
</details>
<details><summary><h3>Confidence Score: 4/5</h3></summary>
The implementation change itself is straightforward and isolated; the test gap means field-isolation regressions won't be caught by the existing suite.
Adding three fields to `ALL_NON_CORE_FIELDS` is the only change needed to complete the test contract — without it, the parametrized absence loop never fires for `traceName`, `tags`, or `release` when a different group is requested, so a future leak of `trace_context` data into unrelated group responses would go undetected in CI.
web/src/__tests__/server/observations-api-v2.servertest.ts — the `ALL_NON_CORE_FIELDS` constant needs to include `traceName`, `tags`, and `release`.
</details>
<details><summary><h3>Sequence Diagram</h3></summary>
```mermaid
sequenceDiagram
participant Client
participant PublicAPI as /api/public/v2/observations
participant QueryBuilder as buildObservationsQueryComponents
participant CH as ClickHouse (events table)
participant Traces as traces CTE
Client->>PublicAPI: "GET ?fields=trace_context&traceId=..."
PublicAPI->>QueryBuilder: "fields=["trace_context"]"
QueryBuilder->>QueryBuilder: Validates group in OBSERVATION_FIELD_GROUPS
QueryBuilder->>Traces: JOIN traces CTE (tags, release, traceName)
QueryBuilder->>CH: SELECT core + trace_context columns
CH-->>PublicAPI: rows with tags, release, traceName
PublicAPI-->>Client: "{ data: [{ id, traceId, ..., tags, release, traceName }] }"
```
</details>
<details><summary>Prompt To Fix All With AI</summary>
`````markdown
Fix the following 1 code review issue. Work through them one at a time, proposing concise fixes.
---
### Issue 1 of 1
web/src/__tests__/server/observations-api-v2.servertest.ts:538-542
The `ALL_NON_CORE_FIELDS` list is not updated with the three fields from the new `trace_context` group (`traceName`, `tags`, `release`). The absence-check loop only iterates over members of this list, so when any other group (e.g. `basic`, `io`, `usage`) is tested in isolation, the test never asserts that `traceName`, `tags`, and `release` are absent from the response. A regression that leaks `trace_context` fields into unrelated group responses would pass undetected.
```suggestion
// prompt
"promptId",
"promptName",
"promptVersion",
// trace_context
"traceName",
"tags",
"release",
] as const;
```
`````
</details>
<sub>Reviews (1): Last reviewed commit: ["feat(observations-v2): expose trace\_cont..."](https://github.com/langfuse/langfuse/commit/ae0d5f26f2afb8019cb6a7aff75452d0184b08cc) | [Re-trigger Greptile](https://app.greptile.com/api/retrigger?id=31989455)</sub>
<!-- /greptile_comment -->
* refactor(blob-export): split OBSERVATION_FIELD_GROUPS from BLOB_EXPORT_FIELD_GROUPS
OBSERVATION_FIELD_GROUPS was driving both the public /v2/observations API
contract and the blob exporter's column selection. The blob exporter needs
a broader set (tools, trace_context) that shouldn't silently leak into the
public observations API.
Decouple the two:
- OBSERVATION_FIELD_GROUPS stays narrow (9 groups) — drives /v2/observations
- BLOB_EXPORT_FIELD_GROUPS owns the broader 11 groups — drives blob exporter
- Worker handler now imports BLOB_EXPORT_FIELD_GROUPS from the shared
analytics-integrations module, not the repository symbol
Also relocate usagePricingTierName from trace_context to usage. It's a
pricing-tier attribute on the observation, not part of the trace context.
The /v2/observations API now exposes it under the usage group; the Fern
docstring and openapi.yml are updated to match. Blob export's
trace_context shrinks accordingly.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* feat(observations-v2): expose usagePricingTierName field in API response
Adds usagePricingTierName to the v2 observations API response contract:
- Fern commons.yml: optional<nullable<string>> on Observations type
- Zod APIObservationV2: nullable + optional
- Regenerated openapi.yml propagates to SDKs
usagePricingTierName was already runtime-selectable via the `usage`
field group (FIELD_SETS.usage in event-query-builder.ts:296), but the
public response contract didn't declare it — so the value was being
stripped on the way out. This adds the field to the surface that
matches the selection.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* refactor(observations): own field-group vocabulary in domain layer
Per review feedback: the events repository depended on a feature-flavored
type (`BLOB_EXPORT_FIELD_GROUPS` / `BlobExportFieldGroup`) defined in
`features/analytics-integrations`. A foundational component shouldn't
depend on a feature-named type.
Move both group lists to a new client-safe domain module
(`packages/shared/src/domain/observation-field-groups.ts`):
- `OBSERVATION_FIELD_GROUPS_PUBLIC_API` (was `OBSERVATION_FIELD_GROUPS`):
the v2 public API contract — 9 groups exposed by the v2 observations
endpoint.
- `OBSERVATION_FIELD_GROUPS_FULL` (was `BLOB_EXPORT_FIELD_GROUPS`): the
complete set of column groups the events repository can project —
adds `tools` and `trace_context` on top of the API surface. Mirrors
the existing `events_full` / `events_core` ClickHouse naming.
`events.ts` (repository) and `analytics-integrations/index.ts` (feature)
both consume from the domain module instead of from each other. Frontend
forms, Zod enums, and worker jobs reach the values via the existing
`@langfuse/shared` barrel. No behavior change.
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* test(otel): add e2e tenant isolation test for OTEL ingestion
Adds a server-side e2e test (LFE-9771) proving that a span POSTed via
project A's API key lands exclusively in project A's observations table
and is absent from project B's. Guards the auth-scope invariant
(projectId resolved from API key, never from the OTLP payload) across
the web route, BullMQ job payload, and ClickHouse write layers.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* style: apply prettier formatting to otel tenant isolation test
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* test(e2e): fix fragile Redis key count assertion in ingest trace test
The `ingest a trace` test asserted exactly 1 `api-key:*` key in Redis,
which breaks when another e2e test file runs concurrently and caches its
own API key. Replace the count check with a lookup by projectId so the
assertion is robust against parallel test execution.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* add assertion
* test(otel): clean up created orgs in tenant isolation test afterAll
Claude CI review on PR #13622 flagged that the two test orgs/projects created
via createOrgProjectAndApiKey() were never deleted. Add an afterAll that
deletes them by org ID — Prisma cascades to project and apiKey rows.
Track org IDs in module scope and push immediately after each creation so
the cleanup fires even if a later assertion fails or the test times out.
Mirrors the pattern used by blob-storage-integration-trpc.servertest.ts.
ClickHouse observation rows from the test span aren't cleaned up here —
they're partitioned by the new project_id which won't be reused, so they're
inert. The Postgres org/project rows are the actual leak.
* test(otel): dedup observations count to tolerate ReplacingMergeTree retries
Claude CI review on PR #13622 flagged that the bare `SELECT count() FROM
observations` returns physical (pre-merge) rows on the ReplacingMergeTree.
If the OtelIngestionQueue retries the ingestion job during the 40s
waitForExpect window (attempts: 6 per otelIngestionQueue.ts), a second
insert for the same span lands and `toBe(1)` fails for a retry reason,
not a tenant-isolation reason.
Wrap the count in the repo's standard dedup pattern:
`ORDER BY event_ts DESC + LIMIT 1 BY id, project_id` (same shape used
throughout observations.ts). The count of the deduped subquery is 0 or 1
regardless of how many physical inserts occurred.
---------
Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* refactor(blob-export): replace internal exportSource enum with public LEGACY/ENRICHED/LEGACY_AND_ENRICHED
The REST API was exposing AnalyticsIntegrationExportSource (a Prisma enum
intended as an internal identifier) directly to public consumers. Its values
— TRACES_OBSERVATIONS, EVENTS, TRACES_OBSERVATIONS_EVENTS — don't match the
UI labels users see ("Enriched observations" etc.) and bake legacy internal
naming into the public contract.
Introduce a distinct public enum and a mapping layer:
- Public values: LEGACY, ENRICHED, LEGACY_AND_ENRICHED
- toInternalExportSource / toPublicExportSource bidirectional helpers
- PUT handler maps public → internal before Prisma; GET responses map
internal → public before serializing
- Fern docstring references /api/public/v2/observations for ENRICHED so
consumers have a concrete anchor for the data model
This is a hard break of the enum values exposed in PR #13598 (merged ~5h
ago); no SDKs are believed to have been published or integrated against
those values. The internal AnalyticsIntegrationExportSource enum (Prisma,
tRPC, UI) is unchanged.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* test(blob-export): align test labels with public enum and add LEGACY_AND_ENRICHED coverage
Two CI review nits on the test file:
1. Renames — describe/it labels, section comments, and the
`tracesObsIntegration` / `eventsIntegration` variable names now use the
public enum values (LEGACY, ENRICHED, LEGACY_AND_ENRICHED) instead of the
legacy internal names (TRACES_OBSERVATIONS, EVENTS,
TRACES_OBSERVATIONS_EVENTS). The Prisma-direct seeds and DB-read
assertions inside the test bodies still use internal values, since those
reach the Postgres enum directly.
2. New regression test — adds `GET response maps internal
TRACES_OBSERVATIONS_EVENTS to public LEGACY_AND_ENRICHED`. The existing
multi-project test only covered the first two public values; the third
was relying on compile-time exhaustiveness via `satisfies Record<…>` in
INTERNAL_TO_PUBLIC_EXPORT_SOURCE, which won't catch a copy-paste error.
A runtime assertion through the public REST surface closes that gap.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* refactor(blob-export): rename public exportSource values to be self-descriptive
Per colleague feedback on PR #13619, swap the public REST enum tag-style
names for more self-describing identifiers:
- LEGACY → LEGACY_TRACES_OBSERVATIONS
- ENRICHED → OBSERVATIONS_V2
- LEGACY_AND_ENRICHED → LEGACY_TRACES_AND_ENRICHED_OBSERVATIONS
Updates Fern source, regenerated openapi.yml, the public-API Zod schema's
mapping helpers, and the server-test fixtures. Internal Prisma enum values
(TRACES_OBSERVATIONS / EVENTS / TRACES_OBSERVATIONS_EVENTS) are unchanged —
this is purely a public-surface rename, isolated by the toPublic /
toInternal mapping helpers introduced in this PR.
* updated API docs
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
fix(otel): recognize OpenInference prompt_details.cache_read/cache_write
Adds prompt_details.cache_read and prompt_details.cache_write to the
cache token resolver in extractGenericGenAiUsageDetails so cache tokens
emitted by openinference-instrumentation-* (openai, anthropic, agno) are
normalized into Langfuse's canonical input_cached_tokens /
input_cache_creation usage_details keys instead of being passed through
as opaque raw keys.
Without this, Langfuse ingests the values (they show up in usageDetails
as prompt_details.cache_read etc.) but the cost engine prices input at
full base rate and silently skips the cache keys, leading to ~30%
under-reported cost on cache-heavy observations. The values also are not
subtracted from input, which risks double-counting depending on what the
instrumentor populates.
llm.token_count.prompt_details.cache_read and cache_write are the
canonical OpenInference semantic-convention names (defined in
openinference-semantic-conventions/src/openinference/semconv/trace/__init__.py),
emitted by every OpenInference instrumentor that supports prompt caching.
The right place to fix is here, not upstream.
Same shape of fix as #12248 (pydantic-ai cache token names).
Fixes#13571 (partial — addresses the OpenInference half of #12635).
Co-authored-by: gragragrab <12702336+gragragrab@users.noreply.github.com>
Co-authored-by: Hassieb Pakzad <68423100+hassiebp@users.noreply.github.com>
## Summary
[LFE-9456](https://linear.app/langfuse/issue/LFE-9456) — final part of the stack. Exposes `exportSource` and `exportFieldGroups` on the public REST API for blob storage integrations.
- **Request**: `CreateBlobStorageIntegrationRequest` now accepts `exportSource` (optional, defaults to `TRACES_OBSERVATIONS`) and `exportFieldGroups` (`nullable<list<ExportFieldGroup>>`, optional).
- **Response**: `BlobStorageIntegrationResponse` now returns `exportSource` (non-null) and `exportFieldGroups` (`nullable<list<…>>`).
- **Strict validation**: REST contract is intentionally stricter than tRPC:
- `TRACES_OBSERVATIONS` + non-null `exportFieldGroups` → 400 ("not applicable"). Covers `[]`, partial arrays, full arrays alike.
- `EVENTS` / `TRACES_OBSERVATIONS_EVENTS` + provided `exportFieldGroups` without `core` → 400 (delegates to existing shared `validateExportFieldGroups`).
- **Source-conditional handler**: `TRACES_OBSERVATIONS` writes `undefined` (Prisma preserves the column / applies default for new rows) and reads return `null` to hide any inert legacy value. `EVENTS` family defaults to all 11 groups when omitted/null.
- **Fern source updated** with new `ExportSource` / `ExportFieldGroup` enums, request and response field additions, and rule docstrings. OpenAPI spec regenerated.
### Why the REST/tRPC divergence is intentional
The UI's tRPC path still submits all 11 groups for `TRACES_OBSERVATIONS` because the form always carries them; the shared `validateExportFieldGroups` doesn't enforce `core` for that source. The worker ignores the column entirely for `TRACES_OBSERVATIONS` (uses fixed-column exports). The REST contract should not expose a knob that's inert at export time — so it rejects on write and hides on read.
### Drive-by
Includes `openapi.yml` regen sweep for #13126 (the `every_20_minutes` enum value was added to Fern source but never regenerated). Generated artifact only; no behavior change.
### Impacted packages
- `web` — Zod request/response types, GET list + PUT handlers, server tests, regenerated OpenAPI spec
- `fern` — Fern source definition
## Test plan
- [x] `pnpm --filter web run lint`
- [x] `pnpm --filter web run typecheck`
- [x] `dotenv -e .env -- pnpm --filter web exec vitest run blob-storage-integration-api blob-storage-integration-trpc` — 50/50 passed (38 new REST + 12 tRPC regression)
- [x] `npx fern-api generate --api server` — Python SDK, TypeScript SDK, OpenAPI spec all regenerated cleanly
- [ ] Manual API client smoke test against staging once merged (verify SDK round-trip on EVENTS payload)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- greptile_comment -->
<details open><summary><h3>Greptile Summary</h3></summary>
This PR exposes `exportSource` and `exportFieldGroups` on the public REST API for blob storage integrations, with strict validation rules (TRACES_OBSERVATIONS rejects non-null field groups; EVENTS/TRACES_OBSERVATIONS_EVENTS requires `core` when groups are provided) and source-conditional response masking.
- **PUT handler**: correctly defaults a null/omitted `exportFieldGroups` to all 11 groups for `EVENTS`/`TRACES_OBSERVATIONS_EVENTS`, and passes `undefined` for `TRACES_OBSERVATIONS` so Prisma preserves the existing DB column without overwriting it.
- **GET handler and PUT response block**: both cast the raw DB `exportFieldGroups` value without applying the same null-→-all-groups default that the write path uses, meaning legacy `EVENTS` rows with a null DB column return `null` in the response rather than the full group list.
- **Test schema**: local `BlobStorageIntegrationResponseSchema` marks `exportSource` as `.optional()`, which is weaker than the production contract; tests would not fail if the field were accidentally dropped from the response.
</details>
<details><summary><h3>Confidence Score: 3/5</h3></summary>
The write path is correct and well-tested, but the read path has an inconsistency: GET returns raw DB null for EVENTS integrations with an unset exportFieldGroups column, while PUT always writes the full default list. Any legacy EVENTS row would expose this gap to API consumers.
The write-side logic (defaulting, masking, validation) is solid and tests cover it well. The inconsistency lives in the GET handler and the PUT response builder, both of which skip the null-to-all-groups normalization that the write path applies. For organizations with legacy EVENTS integrations (created before exportFieldGroups was populated), the GET response would return null for a field that should carry the full 11-group list, which could mislead consumers and break SDK round-trips.
web/src/pages/api/public/integrations/blob-storage/index.ts — both response-building blocks (GET and PUT) need the same null-defaulting logic that the write path already has for EVENTS/TRACES_OBSERVATIONS_EVENTS sources.
</details>
<details><summary><h3>Flowchart</h3></summary>
```mermaid
%%{init: {'theme': 'neutral'}}%%
flowchart TD
A[PUT /integrations/blob-storage] --> B{Zod validation}
B -->|exportSource = TRACES_OBSERVATIONS\nexportFieldGroups != null| C[400 not applicable]
B -->|exportSource = EVENTS/TOE\nexportFieldGroups provided without 'core'| D[400 core required]
B -->|valid| E{exportSource?}
E -->|TRACES_OBSERVATIONS| F[pass exportFieldGroups: undefined\nPrisma skips column on update]
E -->|EVENTS / TRACES_OBSERVATIONS_EVENTS| G[pass exportFieldGroups ?? all 11 groups]
F --> H[upsertBlobStorageIntegration]
G --> H
H --> I{Build response}
I -->|TRACES_OBSERVATIONS| J[exportFieldGroups: null]
I -->|EVENTS/TOE| K[exportFieldGroups: DB value as-is\nno null → all-groups default]
L[GET /integrations/blob-storage] --> M[fetch all org integrations]
M --> N{for each integration}
N -->|TRACES_OBSERVATIONS| O[exportFieldGroups: null]
N -->|EVENTS/TOE| P[exportFieldGroups: DB value as-is\nno null → all-groups default]
O --> Q[200 response array]
P --> Q
```
</details>
<details><summary>Prompt To Fix All With AI</summary>
`````markdown
Fix the following 2 code review issues. Work through them one at a time, proposing concise fixes.
---
### Issue 1 of 2
web/src/pages/api/public/integrations/blob-storage/index.ts:94-98
**GET doesn't normalize null `exportFieldGroups` for EVENTS sources**
The PUT handler defaults a null/omitted `exportFieldGroups` to all 11 groups for `EVENTS` and `TRACES_OBSERVATIONS_EVENTS` sources (`validatedData.exportFieldGroups ?? [...BLOB_EXPORT_FIELD_GROUPS]`). The GET handler here simply passes the raw DB value through, so a legacy `EVENTS` row where the column was never populated returns `null` instead of the full group list. A consumer reading the GET response on such a row sees `null`—indistinguishable from `TRACES_OBSERVATIONS`'s "not applicable" null—while a subsequent PUT would return the full list. The same cast-without-defaulting pattern repeats on the PUT response block at lines 213–217.
### Issue 2 of 2
web/src/__tests__/server/blob-storage-integration-api.servertest.ts:31-38
**Test schema marks `exportSource` optional — weaker than production contract**
The production `BlobStorageIntegrationResponse` schema requires `exportSource` as non-optional (it's always present in the response). Marking it `.optional()` in the test schema means the tests would pass even if the field were accidentally dropped from the API response, making the test suite weaker than intended for this new field.
`````
</details>
<sub>Reviews (1): Last reviewed commit: ["feat(blob-export): expose exportSource a..."](https://github.com/langfuse/langfuse/commit/8601e56bd7e996def93d14761a4419b607b77923) | [Re-trigger Greptile](https://app.greptile.com/api/retrigger?id=31769478)</sub>
> Greptile also left **2 inline comments** on this PR.
<!-- /greptile_comment -->
Emit langfuse.queue.clickhouse_writer.rows_dropped with entity_type tag
when rows are discarded after max flush attempts, alongside existing
error increments and logs.
Co-authored-by: Cursor <cursoragent@cursor.com>
Some Entra tenants emit an external or personal address in the `email`
claim while the tenant UPN sits in `preferred_username` / `upn`. When the
email domain doesn't match the configured SSO domain, fall back to either
of those if they're valid emails on the configured domain so the
reverse-domain check in `auth.ts` doesn't reject the user.
## Summary
[LFE-9456](https://linear.app/langfuse/issue/LFE-9456) — stacked on top of the DB migration PR.
Adds configurable field groups for the events blob export, covering the query layer, worker wiring, and UI.
**Query layer (`packages/shared`)**
- Rewrites `getEventsForBlobStorageExport` to select only the requested field groups instead of hardcoding all fields
- Adds two new field sets to the query builder: `trace_context` (tags, release, traceName, usagePricingTierName) and `model_export` (providedModelName, modelId, modelParameters — uses `model_id` alias for blob consumers)
- Extends `OBSERVATION_FIELD_GROUPS` from 9 → 11 groups (adds `tools`, `trace_context`)
**Worker (`worker`)**
- Wires `exportFieldGroups` from the Prisma record through `processBlobStorageExport` into the query function and `enrichObservationStream`
- Gates pricing enrichment (input/output/total price) on the `usage` field group — skips model lookup entirely when `usage` is not selected
- Drops `provided_model_name` and `model_parameters` from enrichment output when `model` group is not selected
- Default = all 11 groups, so output is identical to the previous hardcoded path
**UI (`web`)**
- Adds a multi-checkbox field group selector to the blob storage settings form, visible when `exportSource` is `EVENTS` or `TRACES_OBSERVATIONS_EVENTS`
- Resets `exportFieldGroups` to all groups on source switch to prevent silent validation failures
- Surfaces tRPC mutation errors via toast
**Validation**
- `core` group is required and non-deselectable in the UI
- Schema enforces `core` must be present; `exportFieldGroups` must be non-empty when `exportSource` is `EVENTS`
- Extracts `validateExportFieldGroups` as a reusable validator shared between tRPC and schema
The tools popover capped visible cards at ~4 with no working scrollbar
because `<ScrollArea max-h-[...]>` lands the height constraint on the
Radix Root. The Viewport's `h-full` doesn't resolve against a parent
with only `max-height` (CSS resolves `height: 100%` against parent
`height`, not `max-height`), so the Viewport sized itself to content,
never reported overflow, and Radix never activated its scrollbar.
Meanwhile the Root's `overflow: hidden` clipped the rest.
Apply the height constraint to the Viewport via a Tailwind arbitrary
child variant so Radix sees the overflow and renders its scrollbar.
Closes#13433
Co-authored-by: Nimar <l.nimar.b@gmail.com>
Harden secure outbound fetches against DNS rebinding by validating
connection-time lookup results with the existing outbound URL blocklist and
whitelist policy.
Co-authored-by: Codex Opus 4.6 (1M context) <noreply@anthropic.com>
Part 2 of [LFE-9456](https://linear.app/langfuse/issue/LFE-9456) — stacked on top of the test-coverage PR.
- Adds `export_field_groups TEXT[] NOT NULL DEFAULT ARRAY[...]` column to `blob_storage_integrations`
- Adds `BLOB_EXPORT_FIELD_GROUPS` client-safe constant to `@langfuse/shared` (analytics-integrations module)
- Adds `exportFieldGroups` to the Zod form schema (`types.ts`) with `min(1)` validation and all-groups default
- Wires `exportFieldGroups` through `upsertBlobStorageIntegration` service and tRPC `update` mutation
- No behaviour change: default = all 11 groups = today's output
The 11-group default includes `tools` and `trace_context` which the query builder doesn't handle yet (PR 3). These values are stored but ignored by the worker until PR 3 is merged.
Part 1 of 4 for [LFE-9456](https://linear.app/langfuse/issue/LFE-9456) (export field group selection for blob storage integrations). No implementation changes — establishes regression baselines before the refactor.
* refactor(trace): rename folder from `trace2` to `trace`
* docs: rm `trace2` from code comments
* fixup: delete trace and observation preview files
* refactor(trace): update badge rendering logic in Observation and Trace detail views to conditionally display based on annotation mode
* chore: push
* fix: ensure observation id is selected
* fix(scim): block removing last organization owner
SCIM DELETE, PUT(active:false), and PATCH(active:false) deprovisioning paths
unconditionally removed the target user's organization membership, allowing
the last OWNER to be deleted and orphaning the organization.
Mirror the tRPC `deleteMembership` invariant: count remaining OWNERs and
reject with 403 when the request would remove the final OWNER. The error
body uses the SCIM error schema with the same message as the tRPC path.
Tests cover all three deprovisioning verbs against a sole owner plus a
positive control where a second OWNER exists.
Resolves INT-1223.
* fix(scim): wrap last-owner check + delete in serializable txn
Closes a TOCTOU race where two concurrent SCIM deprovision requests with
exactly two OWNERs could both pass the owner-count guard and both delete,
leaving the org with zero owners. The check and delete now run inside a
single Prisma transaction with Serializable isolation; on a serialization
failure (P2034) the endpoint returns 409 so the SCIM client retries.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
ADMIN previously held organization:CRUD_apiKeys, allowing any ADMIN
to mint organization-scoped API keys. Combined with SCIM PUT not
enforcing role hierarchy, this enabled ADMIN -> OWNER privilege
escalation (INT-1222). Removing the scope from ADMIN closes the
escalation primitive at the key-creation boundary; OWNER remains
the only role that can create, list, update, or delete org API keys.
Adds RateLimitService check (after auth + admin-api entitlement gates)
to the three org-scoped admin handlers so that compromised
organization-scoped API keys can no longer issue unbounded writes or
probe global user existence at full request rate.
Resolves INT-1270.
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
fix(scim): normalize userName casing in user POST flow (INT-1320)
The SCIM POST flow checked for an existing user with the case-preserved
userName but performed the upsert with a lowercased email. With case-sensitive
uniqueness on User.email, a case-variant userName slipped past the duplicate
check and either linked to an unrelated existing user or hit a unique
constraint instead of returning 409.
Lowercase userName once and reuse it for both the existing-user lookup and
the upsert. Adds a server test covering case-variant duplicate detection.
The signIn callback consulted only the cloud-only multi-tenant SSO
provider check, which is a no-op on self-hosted instances. This left
the password-reset OTP path as an alternate authentication channel
for users on domains that AUTH_DOMAINS_WITH_SSO_ENFORCEMENT was
meant to lock down. Block the email provider for enforced domains
the same way the credentials authorize() and signup handler do.
* feat(clickhouse): add analytics_events_core view for project-level analytics (LFE-8734)
Adds a ClickHouse VIEW on events_core with per-project, per-hour aggregations
including type/source/scope/SDK counts via sumMap, unique counts via uniqIf
and uniqArray, and has_* boolean flags for feature detection.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat(auth): add email verification on signup gated behind AUTH_EMAIL_VERIFICATION_REQUIRED (LFE-8709)
Add an optional OTP email verification step before user creation during
email/password signup. When AUTH_EMAIL_VERIFICATION_REQUIRED=true (and
SMTP is configured), the signup flow becomes: enter email+name → receive
OTP → verify code → set password. Self-hosters without SMTP or without
the flag keep the current direct signup behavior. SSO/social logins are
unaffected.
Key changes:
- New env var AUTH_EMAIL_VERIFICATION_REQUIRED
- New POST /api/auth/signup-verify endpoint (creates passwordless user)
- New /auth/setup-password page for initial password setup
- Merged set/reset password into one ResetPasswordPage component
- Context-aware email template (welcome vs reset wording)
- hasPassword added to session for mode detection
- Direct /api/auth/signup blocked when verification is required
- Parameterized email verification cutoff (default 10 minutes)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* fix(dashboards): Use correct units for charts
* Fix view version logic
* Support value formatter in `BigNumber` and `HistogramChart`
* Fix value formatter usage
* Resolve PR comments
* Improve formatting single digit millisecond values
* Introduce `formatMetric`
* Fix chart label in `LatencyChart`
* Remove unit form latency label in `score-analytics-utils.ts`
* fix rounding to m for 1000k
* more compact tests
* compact
---------
Co-authored-by: Nimar <l.nimar.b@gmail.com>
* fix(shared): reject DNS-failing hostnames in outbound URL validation
The LLM base URL validator silently bypassed the IP blocklist whenever
DNS resolution failed (NXDOMAIN/SERVFAIL/timeouts/split-horizon), which
enabled DNS-rebinding SSRF against cloud metadata and internal services.
Treat DNS failure as a hard error and drop the per-caller opt-out flag;
self-hosters must use LANGFUSE_LLM_CONNECTION_WHITELISTED_HOST for
gateways the validator cannot resolve. Closes INT-1226.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test(web): use resolvable placeholders in llm-api-key servertest
The new strict DNS validation rejects custom.openai.com / new-custom.openai.com
/ new-endpoint.example.com because they NXDOMAIN. Swap to IANA-reserved
example.com / example.org / example.net which always resolve to public IPs
and pass the IP blocklist.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(sso): add DNS-based verified domains
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(sso): add ssoConfig tRPC router
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(sso): require https:// on user-supplied OIDC issuer urls
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(sso): show zone-relative host and query fresh DNS for domain verification
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(sso): add per-domain self-service SSO config UI
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(sso): pre-flight OIDC discovery on ssoConfig.save and the legacy support endpoint
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(sso): enforce verified-domain invariant on SsoConfig lifecycle
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(sso): include name field on custom OIDC provider form and payload
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(sso): translate P2002 race on verifiedDomain.create to CONFLICT
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(sso): allow pending verified-domain claims to coexist across orgs
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(sso): require non-empty Azure AD tenantId on save
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(sso): restore URL grammar validation on OIDC issuer and GH Enterprise baseUrl
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(sso): refuse redirects on OIDC discovery fetch (SSRF defense)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(sso): surface schema errors for github-enterprise baseUrl in the form
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(sso): gate verifiedDomain mutations on the SSO entitlement
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(sso): preserve advanced authConfig fields when re-saving the same provider
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(sso): skip orphan-prevention check on pending verified-domain deletes
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(sso): accept Azure AD multi-tenant {tenantid} placeholder in OIDC discovery
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(sso): require non-empty name on custom OIDC provider
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(sso): clarify verified-domain delete dialog copy when SSO config exists
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(sso): satisfy codespell and clean up validation messages
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(scim): write audit log on user creation via SCIM POST
POST /api/public/scim/Users now emits an auditLog entry for the
created organizationMembership, matching the tRPC members.create
behavior. Without this, an ADMIN using SCIM to add users with
arbitrary roles left no audit trail (INT-1250).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(projects): persist parsed metadata on project create/update
handleCreateProject and handleUpdateProject parsed string metadata
via JSON.parse for validation but discarded the parsed value and
wrote the raw string into Prisma. As a result, metadata sent as a
JSON string was stored as a string-typed JSON value instead of the
intended object (INT-1338).
Capture the parsed value and persist it.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(projects): reject non-object project metadata
After parsing the metadata JSON string the value can still be null,
a primitive, or an array. Persisting those in the Json? column
violated the API contract, and JS null in particular wrote SQL NULL
to Prisma — silently wiping any existing metadata on update.
Add a shape check after JSON.parse on both create and update paths
to reject non-object metadata with 400. Adds regression tests for
"null", arrays, numbers, strings, and a direct JS null.
Addresses review feedback on PR #13497.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test(projects): explicit coverage for omitted metadata
Document the contract that omitting metadata is valid:
- create without metadata returns {} and stores NULL
- update without metadata preserves the existing object
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three legacy Next.js handlers authenticate directly via ApiAuthService
without going through createAuthedProjectAPIRoute and were not invoking
RateLimitService:
- projects/[projectId]/apiKeys (GET/POST) — backdoor-credential minting
via caller-chosen publicKey/secretKey was unbounded with a leaked
org-scoped key (INT-1271)
- projects/[projectId]/apiKeys/[apiKeyId] (DELETE) — unbounded enumeration
/ churn of project API keys (INT-1265)
- prompts POST — unlimited prompt-version writes; GET already used the
"prompts" rate-limit bucket but POST silently bypassed it (INT-1260)
Add RateLimitService.rateLimitRequest with isRateLimited() short-circuit
after auth/entitlement checks. Uses "public-api" for the apiKeys admin
endpoints and "prompts" for the prompt POST, matching the bucket the GET
branch already consumes.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
POST /api/public/scim/Users now emits an auditLog entry for the
created organizationMembership, matching the tRPC members.create
behavior. Without this, an ADMIN using SCIM to add users with
arbitrary roles left no audit trail (INT-1250).
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(worker): add secondary otel ingestion queue
Mirrors the existing secondary ingestion queue pattern for the OTel pipeline
so high-throughput projects can be redirected to a dedicated processing pool
via LANGFUSE_SECONDARY_OTEL_INGESTION_QUEUE_ENABLED_PROJECT_IDS.
Refs LFE-6579.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore: remove unused values
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The node:24-alpine base image pre-populates /root/.cache/node/corepack/
when corepack is enabled during the build. Removing the module directory
alone left this cache intact in the final image, giving Snyk something
to scan. Add it to the rm -rf in both web and worker runtime-base stages.
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
The "Number of Observations (7d)" column on the Prompts table was passing
toTimestamp = end of yesterday, which excluded every observation with a
start_time during the current day. New prompt calls therefore never
appeared to increment the counter, and prompts only used today showed 0.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: Nimar <l.nimar.b@gmail.com>
* feat(widgets): use latency formatter for millisecond measure units
Adds getMeasureUnit helper to dataModel and wires latencyFormatter into
both DashboardWidget and WidgetForm preview when the selected measure
unit is "millisecond".
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* feat(widgets): add day unit to latency formatter
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(widgets): apply latency formatter to pivot table values
Threads the existing valueFormatter prop from Chart through to PivotTable
so latency metrics render in auto-scaled units (ms/s/min/hr/day) instead
of raw milliseconds, matching the other chart widget types.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(widgets): memoize valueFormatter with unit switch
Replaces the inline measureUnit ternary with a memoized switch keyed on
measureUnit, making it trivial to add more unit-to-formatter mappings
(usd, tokens, etc.) as they arrive.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(widgets): apply latency formatter to histogram bin labels
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(widgets): apply value formatter to vertical bar y-axis ticks
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(widgets): apply value formatter to pie center label and pad tooltip rows
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(widgets): trim latency formatter
Drop unused latencyFormatterParts export and stop padding latency labels
with trailing zeros (1.00s -> 1s).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(widgets): pick formatter from agg-aware result unit and add USD
Add getResultUnit alongside getMeasureUnit so count/uniq aggregations
resolve to "integer" instead of inheriting the source measure's unit
(e.g. count(latency) no longer renders as milliseconds). Both
DashboardWidget and WidgetForm now derive the formatter from
getResultUnit and switch on the result, with a new USD branch routing
to usdFormatter alongside the existing millisecond -> latencyFormatter.
WidgetForm's inline ternary becomes a useMemo to mirror DashboardWidget.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(widgets): per-column pivot formatting via units overlay
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(widgets): drive chart formatting from chartConfig.unit
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* improvement(widgets): cleaned up some dead code
* improvement(formatting): reduced allocations of time duration formatters
* perf(widgets): memoize chart valueFormatter
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(widgets): route sub-millesimal values through compactSmallNumberFormatter
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(widgets): merge sanitized defaultSort back into pivot chartConfig
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(widgets): scale negative latencies in latencyFormatter
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(widgets): preserve precision for sub-unit magnitudes
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
uuid v7+ ships bundled TypeScript declarations ("types": "./dist/index.d.ts"),
so @types/uuid is redundant after the v9→v14 upgrade and risks the stale
v9-era DefinitelyTyped types shadowing the bundled ones.
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
chore(deps): upgrade uuid from v9 to v14 to resolve Snyk alert
Snyk flagged uuid@9 for improper index validation. The package is only
used for v4() random ID generation with no untrusted input, so not
exploitable, but upgrading clears the alert cleanly.
uuid v14 requires Node 20+ (we run Node 24) and keeps the same
import API (import { v4 } from "uuid").
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
* ci: disable web test sharding
* test: isolate comment fixtures
* test: make score update fixture deterministic
* test: report slowest vitest tests
* ci: install only chromium for e2e tests
* test: speed up slow server tests
* test: show slowest vitest tests only in ci
* ci: re-enable web test sharding
* test: reduce ingestion and rate limit test latency
* test: report top 50 slowest vitest tests
* test: parallelize prompt name validation cases
* ci: limit vitest server workers
* test: stabilize score comparison sampling test
* test: show slowest worker tests only in ci
* ci: disable web test sharding
* test: reduce slow trace and annotation fixtures
* test: report slowest vitest files
* test: reduce slow trace and score fixtures
* test: reuse score and prompt list fixtures
* test: streamline slow server fixtures
* test: use valid dataset list limit
* test: stabilize and speed up worker tests
* test: isolate dataset item backfill assertions
* test: speed up API key fixture creation
* test: keep legacy api auth coverage explicit
* ci: skip duplicate next typecheck in test builds
* ci: build dependencies before typecheck
* ci: install playwright headless shell
* test: retry flaky vitest tests in ci
* ci: run prisma generate without turbo cache
* test: isolate dataset schema fixtures for retries
* refactor(web): Simplify `TablePeekView` props
* fix(traces): Show trace id in trace peek view title
* fix(web): Align observation peek title with trace detail view title
* fix(web): Align trace peek title with trace detail view title
* fix(web): Apply same fix as 2f235d831 to trace peek detail view
---------
Co-authored-by: Nimar <l.nimar.b@gmail.com>
* fix(clickhouse): set alter_sync/mutations_sync on multi-ALTER clustered migrations
Back-to-back ALTERs on the same table in a single migration file race the
replicated metadata-version update on ReplicatedMergeTree / SharedMergeTree
(ClickHouse Cloud), where alter_sync defaults to 0 and the second statement
can land on a replica whose metadata version still lags Keeper, producing
CANNOT_ASSIGN_ALTER (517) during initial bootstrap.
Add the per-statement settings to every clustered migration that issues
multiple ALTERs on the same table:
- alter_sync = 2 on metadata ALTERs (ADD/DROP/MODIFY COLUMN, ADD/DROP INDEX)
- mutations_sync = 2 on mutation-creating ALTERs (MATERIALIZE INDEX) so the
index is fully built on all replicas before the migration returns
Covers 0005, 0006, 0008, 0025, 0026, 0031. The unclustered/ mirror runs on
plain MergeTree where these settings are no-ops, so it is left untouched.
Document the rule and the metadata-vs-mutation distinction in the
clickhouse-best-practices skill so future migrations follow the convention.
Validated end-to-end by bootstrapping migrations 1-34 against a fresh
ClickHouse Cloud instance with no 517 errors.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(clickhouse): use alter_sync on ADD INDEX in 0013/0015/0016/0018
These four pre-existing migrations applied SETTINGS mutations_sync = 2 to
both ALTERs, but mutations_sync is a no-op on metadata ALTERs like
ADD INDEX, so the metadata-version race that this PR is fixing in
0005/0006/0026 still applied here. Switch the ADD INDEX line in each to
SETTINGS alter_sync = 2 (the MATERIALIZE INDEX line stays on
mutations_sync = 2). Caught by review on PR #13398.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* cleanup
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(events): use release field for trace release in events table adapter
eventsToTraceAdapter was using earliest.version for both version and
release fields when synthesizing trace data from the events table
(SDK v5+ direct-write path). This caused langfuse.release span
attribute values to be overwritten with the langfuse.version value.
Also adds 'release' to base/baseWithoutTools/byIdBase field sets in
EventsQueryBuilder so it is actually selected from ClickHouse, and
adds release to EventsObservationSchema so the TypeScript type includes it.
Fixes#13273
* test(events): verify release field is returned from events table queries
Adds two tests to event-repository.servertest.ts:
- getObservationsWithModelDataFromEventsTable returns release separately from version
- getObservationByIdFromEventsTable returns release separately from version
These confirm the fix in event-query-builder.ts (adding 'release' to
base/byIdBase field sets) and EventsObservationSchema (adding the release
field) are wired up end-to-end.
* fix(events): include release field in EventsObservationRecordReadType and converter
Add `release` to `eventsObservationRecordReadSchema` so the field is
typed and preserved through ClickHouse deserialization, and map it in
`convertEventsObservation` so it reaches the domain object.
Previously the field was selected by the query builder but silently
dropped because neither the Zod schema nor the converter passed it
through.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(events): add release field to APIObservation schema
The release field is now returned in GET /api/public/observations
responses (via EventsObservation ...rest spread), but APIObservation
was .strict() and did not declare release, causing makeZodVerifiedAPICall
to fail with "Unrecognized key: release" in all useEventsTable=true tests.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Steffen Schmitz <steffen@langfuse.com>
Adds a shared agent skill for triaging Linear/GitHub issues and incident
reports against Datadog APM, logs, and metrics, with a repo-debug map and
output template that produces a structured root-cause analysis.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore(env): add LANGFUSE_ENABLE_EVENTS_TABLE_UI flag for UI events table support
* refactor: rm env variable from wrong usage
* tests: remove env flag
The ClickHouse index analyzer cannot extract has(names, k) semantics
from
values[indexOf(names, k)] OP v (cross-array arrayElement form), so a
bloom_filter skipping index on metadata_names cannot prune granules for
this shape. Wrap every operator branch in StringObjectFilter.apply() for
events_core / events_full / events_proto with an explicit
has(names, k) AND (...) conjunct.
Also corrects a latent semantic bug: today arr[indexOf(names, missing)]
resolves to the empty string (Array(String) default), making
"does not contain V" match rows where the key was never written, and
"OP empty-value" match similarly. The has(...) conjunct fixes this for
all five operators uniformly.
Add bloom_filter on events_core.metadata_names. events_core is the
table that takes filters in the V2 split-query pattern; events_full
reads by ID after the base CTE narrows things down, so the symmetric
index is intentionally omitted for now.
Pre-filters the trace JOIN in a CTE so the trace timestamp window prunes
partitions directly instead of living alongside the LEFT JOIN where the
planner cannot push it down. Applied to the score and generation analytics
queries that drive the PostHog and Mixpanel exports.
Switches `grace_hash` from unconditional to retry-gated: first attempt uses
ClickHouse's `auto` algorithm, retries fall back to `grace_hash` so an OOM
recovers without manual intervention while healthy syncs stay fast.
Note: the generation analytics query previously used `LEFT JOIN ... WHERE
t.project_id = {projectId}` which silently dropped generations whose trace
was missing or outside the 7-day window. With the CTE-based LEFT JOIN those
generations now ship with NULL trace fields instead.
Refs: LFE-9475
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The observations_agg CTE in getTracesForAnalyticsIntegrations only had a
lower bound on o.start_time, so ClickHouse scanned observations from
minTimestamp - 1h to the end of the table on every run. For long-lived
projects or repeated retries this is an unbounded scan.
Cap the CTE at maxTimestamp + OBSERVATIONS_TO_TRACE_INTERVAL (2 days),
matching the existing observation-to-trace join convention elsewhere in
the file. Traces outside [minTimestamp, maxTimestamp) are already
filtered in the outer SELECT, and in steady state the 30-min now-buffer
ensures a trace's observations have settled well before the window
boundary advances, so the cap does not truncate data that would
otherwise be emitted.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(worker): cap PostHog export window at next UTC day boundary (LFE-9475)
When a PostHog integration failed repeatedly, lastSyncAt never advanced
and each hourly retry re-scanned an ever-growing window against
ClickHouse. Cap maxTimestamp at the next UTC day boundary after
lastSyncAt so per-run work is bounded and aligned with the toDate(...)
partition/ordering keys. Healthy integrations are unaffected because
now - 30min wins whenever the sync is within a day of present. Initial
backfills (no lastSyncAt) skip the cap to avoid pathological
day-by-day stepping from 2000-01-01.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(worker): use project createdAt as PostHog sync floor on first run
Replace the 2000-01-01 fallback for minTimestamp with the project's
createdAt. No trace data can precede it, so this is a tight lower bound
that lets the day-cap also apply to initial backfills. Old projects
still catch up incrementally (one UTC day per hourly run) instead of
re-scanning all of history in one shot.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore: update comments
* fix(shared): use half-open upper bound for analytics integration queries
Switch the primary event-timestamp filter from `<=` to `<` in the four
getXForAnalyticsIntegrations queries (traces, generations, scores,
events). Since the PostHog and Mixpanel schedulers advance
`lastSyncAt` to the previous run's `maxTimestamp`, a row whose
timestamp falls exactly on a window boundary was previously emitted
once per run on both sides. Half-open semantics ensure each row is
emitted exactly once across consecutive runs. Secondary trace-join
upper bounds (7-day lookback for metadata) stay `<=` because they are
range optimizations, not emission bounds.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(scores-api): allow source=ANNOTATION on POST /api/public/scores
Expose the `source` field on the create-score request so callers can
post scores as `ANNOTATION` (or `EVAL`). This unblocks LLM-prefilled
scores that show up in an annotation queue for a human reviewer.
When `source` is `ANNOTATION`, `configId` is required unless
`dataType` is `CORRECTION` (matches the existing tRPC annotation
contract).
Ref: LFE-9330
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(scores-api): narrow source to API/ANNOTATION and enforce rule on ingestion path
Addresses review feedback:
- Drop EVAL from the public create-score surface. EVAL remains a valid
stored/query value but is reserved for internal evaluator outputs; external
callers should use API (default) or ANNOTATION. Narrowed via a new
`CreateScoreSource` enum in Fern and inline zod enum at `PostScoresBody`.
- Move the "ANNOTATION requires configId unless CORRECTION" constraint into
`validateAndInflateScore` so it applies to both the REST `POST /scores` path
and the ingestion/SDK path (`POST /ingestion` → score-create event), closing
the gap flagged on the PR. The zod refine on `PostScoresBody` is kept so REST
callers still get a synchronous 400 instead of an async drop.
- Fix the stale path in the `PostScoresBody` comment that pointed to a
non-existent file.
- Add a servertest covering the ingestion path: HTTP returns 207, worker drops
the ANNOTATION-without-configId event, sentinel score confirms the batch was
processed.
Ref: LFE-9330
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(scores-api): replace magic strings with ScoreSourceEnum and a named subset schema
Follow-up to review feedback on stringly-typed source checks.
- Add PublicApiCreateScoreSourceArray / Domain / Type in domain/scores.ts
with `satisfies readonly ScoreSourceType[]` so the public-API subset stays
provably ⊂ ScoreSourceArray at compile time. Dropping a value from
ScoreSourceArray would now break this declaration first.
- Use PublicApiCreateScoreSourceDomain on PostScoresBody instead of the
inline z.enum(["API", "ANNOTATION"]).
- Reference ScoreSourceEnum.ANNOTATION / ScoreSourceEnum.API and
ScoreDataTypeEnum.CORRECTION instead of raw string literals in the
PostScoresBody refine and in validateAndInflateScore.
No behavior change; all source-field tests continue to pass.
Ref: LFE-9330
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs(scores-api): clarify which entry points trigger the annotation/configId check
The "REST create-score path vs ingestion/SDK path" phrasing was misleading:
both are REST, just different HTTP endpoints (/scores vs /ingestion). Spell
that out and note that both funnel through this function in the worker.
Ref: LFE-9330
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(scores-api): consolidate annotation/configId rule into a shared predicate
- Drop unused PublicApiCreateScoreSourceArray/Type exports; inline the
subset tuple into the Domain declaration.
- Add `isAnnotationScoreMissingConfigId` + ANNOTATION_SCORE_REQUIRES_CONFIG_ID_MESSAGE
in domain/scores.ts. Both the zod refine on PostScoresBody and the throw in
validateAndInflateScore now call the same predicate with the same message,
removing the copy-pasted rule and its comments.
- Trim redundant prose comments that duplicated the predicate's intent.
Net -18 lines. No behavior change; all 6 source-field tests still pass.
Ref: LFE-9330
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test(scores-api): unit-test validateAndInflateScore and expose source on client Fern
Addresses PR review feedback:
- Add unit tests for validateAndInflateScore covering the configId tenancy
check (the reviewer's specific ask) plus the ANNOTATION/configId rule.
Direct function calls — no HTTP, no queue, no sentinel waiting; each test
runs in <400ms.
- Drop the flaky ingestion-endpoint sentinel test that asserted the same
ANNOTATION drop behavior; coverage is now more precise at the function
level where the rule actually lives.
- Add `source` + `CreateScoreSource` to the client Fern definition so the
public client spec matches the server Fern surface. Regenerated the
corresponding OpenAPI.
Ref: LFE-9330
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Wrap each day of the usage aggregation loop in its own span with
langfuse.free_tier.dayStart/dayEnd/dayDate/daysAgo attributes so daily
work is visible individually in traces. Extend the ClickHouse
request_timeout to 120s on the three per-day count queries, and retry
each query up to two additional times on failure — a single re-run is
cheap compared to a full job retry.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(web): prevent crash on invalid JSONPath in dataset mapping editor
CustomMappingEditor crashed the dialog when a user entered a malformed
JSONPath (e.g. `$..[?(@.x=)]`), because MappingPreviewPanel called
applyFieldMappingConfig in a useMemo and jsonpath-plus threw during
render. applyFieldMappingConfig now wraps evaluateJsonPath in a safe
helper, reports failures via a new onJsonPathError callback, and returns
undefined for that entry/field instead of throwing. applyFullMapping
propagates the callback so json_path_error entries carry the actual
source field and mapping key.
On the frontend, MappingPreviewPanel surfaces the error as a destructive
banner and invalidates validation when a schema is present. Notice boxes
are deduplicated into a small IssueList/IssueItem helper.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(web): surface invalid JSON path in evaluator prompt preview
renderPromptPreviewFromObservation used to discard the error returned by
extractValueFromObject, so an invalid jsonSelector silently rendered the
raw column value. Surface it inline as `<invalid JSON path "X": …>` so
the user sees which mapping is broken instead of getting a misleading
preview (or a generic "Unexpected Error" toast via useExtractVariables).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(web): show real error message when evaluator variable extraction fails
useExtractVariables was forwarding plain Errors (e.g. invalid JSON path
thrown by jsonpath-plus) to trpcErrorToast, whose non-TRPC fallback
renders a generic "Unexpected Error" toast and drops the actual message.
Use showErrorToast directly so the user sees "Invalid JSON path: …"
instead of a useless generic error.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(web): label evaluator extraction error as JSON path issue
"Failed to extract variable" read like a semantic failure. The only
throw path in extractValueFromObject is the jsonpath-plus call, so any
error surfaced here is a JSON path syntax problem — reflect that in the
toast title so users know where to look.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(web): include JSON path in evaluator extraction error toast
Match the phrasing of the dataset mapping banner and preview inline
message so all three surfaces show both the offending path and the
underlying parser message.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore(web,shared): standardize on "JSONPath" in user-facing strings
"JSON path" / "JSON Path" / "json path" were used interchangeably in
dataset mapping, evaluator preview, and their shared helpers. Use the
canonical "JSONPath" everywhere these strings surface to the user so the
three error surfaces read consistently.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Revert "fix(web): surface invalid JSON path in evaluator prompt preview"
This reverts commit f943fd87f. Scope creep relative to LFE-9336 — the
inline <invalid JSONPath …> marker inside a rendered prompt is noisy,
and the batch-actions preview is a separate concern that deserves its
own pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(web): distinguish JSONPath syntax errors from misses in final preview
FinalPreviewStep collapsed both json_path_error and json_path_miss into
a single amber "did not match" banner, so a syntax error reaching this
step (possible when the field has no schema or for metadata mappings)
was rendered as a soft warning with misleading wording. Split the
errors by type and render syntax errors with destructive styling and
"invalid syntax" wording, while keeping misses as amber warnings. When
a card has both, prefer the destructive treatment and combine the two
counts into one line. Extract the banner chrome into a small IssueBanner
helper to share variant styling between the top banner and per-card
footer.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(shared): return errors/misses from applyFieldMappingConfig
Replace the onJsonPathMiss/onJsonPathError callback parameters on
applyFieldMappingConfig with a direct FieldMappingResult return shape
of { value, misses, errors }. Callers no longer mutate external arrays
via closures — applyFullMapping consumes result.misses / result.errors
directly and MappingPreviewPanel destructures them out of the return.
Also tightens per-field fault isolation semantics in applyFullMapping
and cleans up a couple of low-value comments.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(web): extract shared IssueBanner from mapping preview panels
MappingPreviewPanel and FinalPreviewStep were each carrying their own
variant lookups, notice-box markup, and icon pickers for the error /
warning JSONPath chrome. Consolidate into a single
AddObservationsToDatasetDialog/components/IssueBanner module that exports:
- IssueBanner, IssueList, IssueItem components
- issueChromeVariants (border + bg + text for banner / list / card footer)
- issueCardVariants (outer card border with an explicit "none" variant)
- issueTextVariants (text color for children that override CSS inheritance)
- issueIcons (variant -> lucide icon)
Both callers now compose cva output with their layout classes via cn().
No visual change; colors and spacing are identical.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(web): address PR review nits on JSONPath extraction + preview copy
useExtractVariables was labelling every error that reached the toast
effect as "Invalid JSONPath in variable mapping", including unrelated
runtime errors that hit the outer Promise.all().catch() branch. Replace
the plain Error state with a discriminated ExtractionError so only
errors surfaced by extractValueFromObject use the JSONPath title;
unexpected failures fall back to a generic "Failed to extract variable".
FinalPreviewStep's per-card footer rendered "1 path have invalid
syntax" for a single error; move the verb into the ternary so the
singular reads "1 path has invalid syntax".
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(experiments): add enabled toggle for remote dataset run trigger (#13221)
* feat(experiments): add enabled toggle for remote dataset run trigger
Allows users to temporarily disable the remote trigger without losing
the configured URL and default payload.
- Add `remoteExperimentEnabled` boolean column (default true) to the
`datasets` table
- Add an "Enable trigger" switch to the upsert form
- Hide the "Run" button in the experiment dialog when disabled
- Server-side early return in `triggerRemoteExperiment` when disabled
(returns `{ success: true, skipped: true }`)
Motivation: once a URL is configured, every Run click fetches the URL
and shows a "Failed to trigger remote experiment" error toast if the
endpoint is unreachable. The only way to silence it today is to delete
the URL, which is lossy. This adds a simple toggle to pause the trigger
while keeping the config.
Backward compatible: default is `true` at both column and form levels,
so existing datasets behave identically.
* fix(public-api): include remoteExperimentEnabled in Dataset v2 response schema
Without this, POST/GET/PUT /api/public/v2/datasets responses fail strict
Zod validation with "Unrecognized key: remoteExperimentEnabled" since the
tRPC/API layer now returns this field for the new toggle.
* fix(experiments): address review feedback on enabled toggle
- deleteRemoteExperiment: reset remoteExperimentEnabled back to true so
a later upsert without the optional enabled flag does not silently
inherit the previously disabled state
- RemoteExperimentTriggerModal.onSuccess: distinguish the
{ success: true, skipped: true } response and show a "Remote trigger
is disabled" toast instead of the misleading success toast
- allDatasets: add remoteExperimentEnabled to the Omit exclusion list
so the $queryRaw return type matches the actual SQL SELECT (the
field is not fetched, so the type annotation would otherwise lie)
* fix(public-api): select remoteExperimentEnabled in list dataset handlers
GET /api/public/datasets (v1) and GET /api/public/v2/datasets (v2) both
use an explicit Prisma select that narrows the return type. The APIDataset
response schema now requires remoteExperimentEnabled, so the select needs
to include it or the build fails with "Property 'remoteExperimentEnabled'
is missing".
The single-dataset GETs (v1 /datasets/[name] and v2 /datasets/[datasetName])
use the default all-fields findFirst, so they're already fine.
---------
Co-authored-by: Nimar <l.nimar.b@gmail.com>
* chore: fix migration order
* refactor: remove remoteExperimentEnabled field from API dataset types and endpoints
* refactor: wording
* test: ensure remote experiment fields are not exposed in public API responses
* chore: wording nits
* fix: invalidate remote experiment cache on dataset removal in RemoteExperimentUpsertForm
---------
Co-authored-by: Yuto Toya <97585904+toyayuto@users.noreply.github.com>
Co-authored-by: Nimar <l.nimar.b@gmail.com>
* chore(observability): upgrade opentelemetry and datadog SDKs
* fix(ci): disable tracing in tests-web startup
* fix(ci): remove local codex files from pr
* test(worker): retry bedrock llm connection smoke tests
* chore; revert pipeline changes
* revert(test): remove bedrock retry logic from llm connection tests
Reverts the retry/backoff additions from cdaacaf45 — these should be
handled in a separate PR since they are unrelated to the OTel/DD upgrade.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* chore; disable DD tracing for web tests
* chore: downgrade dd-trace to 5.82.0
* chore: downgrade prisma instrumentation
* chore: bump dd-trace-js
* chore: release v3.170.0-0
* revert release push
---------
Co-authored-by: steffen911 <steffen@langfuse.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Steffen Schmitz <steffenschmitz@hotmail.de>
* feat: add 5-minute and 20-minute blob storage export frequency options
Add sub-hourly export frequency options (every 5 minutes, every 20 minutes)
to the blob storage integration, in addition to the existing hourly/daily/weekly.
The scheduler cron is updated from hourly to every 5 minutes to support the
new intervals.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: remove 5-minute export option, reduce lag buffer from 30 to 20 minutes
Per reviewer feedback, 5-minute export frequency is too granular given
internal data paths that may take longer in the worst case. Keeps only
the 20-minute option as the new minimum frequency, adjusts the lag
buffer to 20 minutes, and updates the queue schedule accordingly.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: address PR review feedback for blob storage export frequency changes
- Add removeRepeatable for old hourly cron pattern to prevent duplicate schedules
- Extract lag buffer duration into BLOB_STORAGE_LAG_BUFFER_MS constant
- Add every_20_minutes to Fern BlobStorageExportFrequency enum
- Update lag buffer docs from 30 to 20 minutes
- Fix test comment referencing old 30-min lag buffer
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix: update test timing assertion from 30-min to 20-min lag buffer
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix: export BLOB_STORAGE_LAG_BUFFER_MS and use it in tests
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix: prettier formatting for exportFrequency enum in types.ts
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* feat(ui): decode unicode escapes in PrettyJsonView for trace detail
Apply decodeUnicodeEscapesOnly() recursively to parsed JSON in the trace
detail PrettyJsonView so that \uXXXX sequences (produced by Python SDK's
json.dumps with ensure_ascii=True) are decoded to their original
characters when viewing traces in the UI.
This is a follow-up to PR #12882 which applied the same decoder to the
batch export pipeline, and to the earlier IOTableCell change (PR #9686).
Together these ensure non-ASCII content (e.g. Japanese, Chinese, Korean)
renders correctly everywhere a user encounters it in Langfuse.
Uses greedy mode to cover both \uXXXX (single-escaped) and \\uXXXX
(double-escaped) ingest paths. Only \uXXXX escapes are decoded; other
escapes (\n, \t, \", etc.) are left untouched.
Refs #10972
* test(ui): add unit tests for decodeUnicodeInJson in PrettyJsonView
Cover primitive passthrough, string decoding (including double-escaped
greedy mode), recursive decoding of arrays / nested objects, surrogate
pairs, and mixed already-decoded input. Same style as unicode.clienttest
from PR #12882.
* refactor(ui): make decodeUnicodeInJson iterative and bounded
Address review feedback from @nimarb on PR #13223: very large or deeply
nested JSON payloads could blow the call stack or freeze the browser tab.
- Replace recursion with an explicit-stack iterative walk, so stack depth
is no longer bounded by JS engine limits.
- Add two guards:
- DECODE_UNICODE_MAX_NODES (50,000): stop decoding once the total number
of visited entries exceeds the budget; remaining values are kept as-is.
- DECODE_UNICODE_MAX_DEPTH (200): do not descend into subtrees past this
depth; the subtree is returned undecoded.
- Export the caps so tests can assert behavior at the boundary.
Tests: two new cases in PrettyJsonView.clienttest.ts -- one for chains
~10x deeper than MAX_DEPTH (must not throw), one for arrays larger than
MAX_NODES (first entries decoded, tail preserved verbatim).
* fix(ui): clone parsedJson for JSONView and decode escaped object keys
Address follow-up review feedback on PR #13223.
1. JSONView was receiving `parsedJson` directly, but JSONView internally
calls `deepParseJson` which mutates nested string fields in place. Since
baseTableData[].rawChildData holds references back into `parsedJson`,
sharing the same reference corrupted the table's lazy-loaded children
(parsed sub-objects replaced the original maxDepth:2 strings). Pass a
`structuredClone` of `parsedJson` to JSONView via a `useMemo` so the two
views stay independent without cloning on every render.
2. `decodeUnicodeInJson` was decoding values but leaving object keys as-is.
Payloads like `{"\\u4f60\\u597d": "value"}` ended up with escaped keys
alongside decoded values. Apply `decodeUnicodeEscapesOnly` to keys too.
Two new tests cover key decoding (flat + nested); existing tests cover the
value path.
---------
Co-authored-by: Nimar <l.nimar.b@gmail.com>
Track event size distributions across projects via a MergeTree table
fed by two materialized views (one from traces, one from observations).
Every insert including updates gets its own row for full ingestion
visibility.
* fix(evals): remove broken score value filter from evaluator runs page
The score value filter on the evaluator runs page never worked because
it LEFT JOINed the PostgreSQL scores table, but scores live in
ClickHouse. Remove the filter from the UI and strip it in the backend
for backward compatibility with bookmarked URLs.
Closes LFE-9279
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(evals): remove session ID filter and column from evaluator runs page
The session ID filter/column depended on the Postgres `traces` table via
LEFT JOIN, but traces now live in ClickHouse so the join never resolved.
Drop the UI filter facet, table column, and the unused traces JOIN.
Also generalize the bookmarked-URL stripping into a DEPRECATED_FILTER_COLUMNS
constant (currently scoreValue + sessionId).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(prompts): Add time window filtering to prompt metrics
* Normalize time window for prompt metrics to start of day / end of day
* Add explanatory tooltip to "last used" and "first used" columns
* refactor(web): migrate test framework from Jest to Vitest
Replace Jest with Vitest in the web package for faster test execution
and native ESM/TypeScript support without requiring next/jest SWC transforms.
- Replace jest/jest-environment-jsdom/@types/jest with vitest/@vitejs/plugin-react/vite-tsconfig-paths
- Add vitest.config.mts with 3 projects (client/server/e2e-server) matching the original Jest config
- Convert all jest.* API calls to vi.* equivalents across ~28 test files
- Convert jest.requireActual() to async vi.importActual() with async factory functions
- Remove @jest-environment docblock pragmas (environment set in vitest config)
- Add resolveWorkspaceDeps vite plugin for pnpm strict mode compatibility
- Set testTimeout to 30s to match previous Jest/next-jest default
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(web): fix type error in jumptoplayground test mock
Cast vi.importActual("zod") to any to satisfy both the TypeScript
compiler and the consistent-type-imports lint rule.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: remove jest from shared tsconfig types
The nextjs.json shared tsconfig included "jest" in the types array,
which required @types/jest to be installed. Since we migrated to vitest,
vitest globals are provided via web/vitest-env.d.ts instead.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: update CI pipeline to use vitest instead of jest
Replace jest CLI calls with vitest equivalents in the GitHub Actions
pipeline. Also regenerate pnpm-lock.yaml to remove stale jest entries.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(web): include client project in test:watch
The old Jest test:watch ran all projects. The Vitest migration narrowed
it to only --project server, silently excluding *.clienttest.{ts,tsx}
files from watch mode.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* refactor(web): simplify vitest shared aliases and add web to root workspace
- Auto-generate @langfuse/shared resolve aliases from its package.json
exports instead of hardcoding each subpath
- Add web to root vitest.workspace.ts alongside worker
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(web): address PR review feedback
- Update AGENTS.md quick commands to use vitest positional patterns
instead of Jest --testPathPatterns flag
- Move admin-access-webhook.servertest.ts from src/__tests__/async/ to
src/__tests__/server/async/ so it matches the server project include
pattern (was silently orphaned under both Jest and Vitest)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(web): fix admin-access-webhook test for 5-minute dedupe window
The dedupe window was increased from 60s to 5 minutes in bbd209194 but
the test was never updated because it was orphaned (not picked up by any
test project). Now that it runs, fix the test to advance the clock past
the actual 5-minute window.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* refactor(web): simplify vitest config using deps.inline
Replace the custom resolve aliases, resolveWorkspaceDeps plugin, and
esbuild.tsconfigRaw workaround with a single deps.inline config — the
same approach used by the worker package. This tells Vitest to process
@langfuse/* packages through its transform pipeline using normal Node
resolution (following pnpm symlinks) instead of Vite's strict exports
resolver.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* chore(web): remove unnecessary vitest-env.d.ts
Test files are excluded from the build tsconfig, and vitest injects
global types at runtime when globals: true is set. The declaration
file is not needed.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(web): add vitest/globals to tsconfig types and update docs
- Add "vitest/globals" to web/tsconfig.json types so Next.js build
can resolve describe/it/expect in test files (fixes CI build error)
- Move @testing-library/jest-dom/vitest to client project setupFiles
instead of per-file imports
- Update CONTRIBUTING.md, AGENTS.md, and skill reference docs to
replace Jest references with Vitest
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(web): add @testing-library/jest-dom to tsconfig types
Without this, IDE shows type errors on jest-dom matchers like
toBeInTheDocument() in client test files. The setupFiles config
registers the matchers at runtime but doesn't provide the types.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: correct file location paths in testing-guide.md
Update three embedded file location annotations from async/ to server/
to match the actual directory structure.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(web): enable parallel server tests, clean up CONTRIBUTING.md diff, revert turborepo skill
- Remove fileParallelism: false from server/e2e-server projects to
enable parallel test execution (faster CI)
- Rewrite CONTRIBUTING.md changes preserving original CRLF line endings
to minimize diff noise
- Revert turborepo dependencies.md change (generic skill, not repo-specific)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(web): restore sequential server tests, clean up docs
- Add maxWorkers: 1 to server/e2e-server projects to match Jest's
--runInBand (shared DB requires sequential execution)
- Clean CONTRIBUTING.md diff (preserve CRLF line endings)
- Revert turborepo skill change (generic skill, not repo-specific)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: align CONTRIBUTING.md test command description with example
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(web): convert remaining jest.* calls in controller.clienttest.ts
New test code merged from main still used jest.mock/jest.fn/jest.mocked
instead of vi.* equivalents.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Nimar <l.nimar.b@gmail.com>
Tightens `# v1`/`# v2`/`# v6` floating-major comments on SHA-pinned
actions to their exact patch-level tags (`v1.8.4`, `v2.1.0`, `v2.6.1`,
`v6.1.1`). Same SHAs, no behavior change — just removes ambiguity about
which release the pin corresponds to, and keeps the comment truthful if
the upstream major ever moves to a new SHA (which is exactly what bit
us on actions/setup-node in langfuse-js).
Left alone:
- winterjung/split — only publishes a `v2` floating tag; no patch-level
tag exists to tighten to.
- orange-buffalo/dependabot-auto-rebase — pins a `v1` branch head (not
a tag). Switching off a mutable ref is a separate decision.
- .github/workflows/ci.yml.template — not an active workflow.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(api): add DELETE endpoint for LLM connections
Adds `DELETE /api/public/llm-connections/{id}` so API clients (e.g. the
Terraform provider) can fully manage the lifecycle of LLM connections.
Mirrors the tRPC delete behavior by pausing dependent evaluator configs
when the connection is removed.
Refs: langfuse/terraform-provider-langfuse#19,
langfuse/terraform-provider-langfuse#20
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore: remove admin api key auth
* chore: revert
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(web): Add region selector to user menu
* Only show region switcher in cloud region
* Create `isProductionRegion` function
* Use same regions for auth and user navigation
* feat: detect SDK version from langfuse events table
* chore: push
* feat(events): set preferred Clickhouse service for SDK metadata retrieval
* refactor: rename SDK metadata functions for clarity and update documentation
* refactor(events): update metadata selection to use selectMetadataExpanded for full values
* tests: ai sdk test case
* refactor: enhance SDK metadata extraction to include telemetrySdkName for improved identification
---------
Co-authored-by: Nimar <l.nimar.b@gmail.com>
AWS SDK v3 >= 3.729 sends a composite CRC32 header on
CompleteMultipartUpload by default, which GCS's S3-compat layer rejects
with a 412 PreconditionFailed. Set requestChecksumCalculation and
responseChecksumValidation to WHEN_REQUIRED so buffered multipart and
lib-storage Upload both work against GCS HMAC endpoints.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- `updateOrgMembership` now passes the updated record as `after` so org-level
role changes log both before and after in the audit trail.
- `updateProjectRole` passes `updatedProjectMembership` as `after`, uses
`action: "create"` when the upsert creates a new row, and standardises the
`resourceId` to `projectId--userId` across create/update/delete so one
membership can be tracked end-to-end.
Fixes LFE-9056.
fix(batch-actions): allow dialog dismissal on status step and fix Go to Dataset 404
Previously the add-observations-to-dataset dialog blocked ESC / outside-click
on the status step, and the "Go to Dataset" link 404'd for datasets whose ids
contain slashes (e.g. seed datasets named `folder/simple-dataset`). Allow
ambient dismissal except while the batch action is in flight, and navigate
via `router.push` with an encoded dataset id so Next.js doesn't decode `%2F`
back to `/` on render.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replace the scores-numeric segment's "does not contain CATEGORICAL"
exclusion with a positive allow-list of data_type IN ('NUMERIC', 'BOOLEAN').
Previously, free-text (TEXT) and correction scores leaked into the
scores-numeric view because only CATEGORICAL was excluded; their null
numeric values also distorted avg/min/max aggregations.
The positive allow-list is safer by default — any future data_type value
added to the enum is excluded unless explicitly opted in.
Refs https://linear.app/langfuse/issue/LFE-9399
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(web): Stale search highlights
* refactor(web): Do not compute search match ranges twice
* Fix PR review issue about active match not being restored
* Resolve PR comment
* Split `useSyncMessageSearchMessages` useEffects
Emit the S3 fileKey on every OTEL ingestion failure path so operators can
scan their log platform for dropped or errored batches and feed the list
into worker/src/scripts/replayIngestionEventsV2. Covers: masking
fail-closed drops, observation parse failures, per-event record
creation/eval/write failures, and the job-level ForbiddenError/catch
fallthrough. The masking drop log additionally carries orgId and the
propagated callback headers to support zero-trust audit trails.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Avoids rate limit conflicts when VertexAI and GoogleAIStudio tests
run concurrently against the same model.
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(datasets): downgrade schema example error to warn to avoid Sentry noise
json-schema-faker can crash on certain user-provided schemas (e.g. arrays
without items). The function already gracefully returns "" and the UI hides the
example section, but console.error was captured by Sentry's capture_console
integration. Downgrade to console.warn so this expected edge case no longer
triggers alerts.
Fixes LANGFUSE-4S5
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(datasets): upgrade json-schema-faker to 0.6.1 and migrate to async API
- Upgrade json-schema-faker from 0.5.9 to 0.6.1 (ESM-only, zero deps,
TypeScript rewrite) which fixes the array-without-items crash
- Migrate from deprecated synchronous default-export API to the new named
async `generateJson` export with per-call options
- Update callers (DatasetSchemaHoverCard, NewDatasetItemForm) to handle
async generation with proper cancellation cleanup
- Add Jest moduleNameMapper + transformIgnorePatterns for ESM-only package
- Refactor jest.config.mjs to pre-resolve configs and avoid duplicate calls
Fixes LANGFUSE-4S5
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* refactor(datasets): revert jest.config.mjs changes, use virtual mock in test
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* chore(datasets): remove generateSchemaExample test file
ESM-only json-schema-faker@0.6.1 can't be resolved by Jest's CJS
resolver without jest.config changes. The wrapper is trivial — drop the
test rather than adding workarounds.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* test(datasets): add generateSchemaExample tests with Jest ESM resolution
Add moduleNameMapper for json-schema-faker (ESM-only package) so Jest
can resolve it, and restore the client tests.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(datasets): add type assertion for JsonSchema compatibility
Prisma.JsonValue narrows to JsonObject | JsonArray which isn't
assignable to json-schema-faker's JsonSchema type. Cast explicitly
since we already guard for non-objects.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* refactor(datasets): remove unnecessary cancellation in schema example effects
Schema generation is near-instant — cancellation cleanup adds
complexity for no practical benefit.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(datasets): merge moduleNameMapper with Next.js defaults in jest config
Spread sharedOverrides was overwriting Next.js's built-in
moduleNameMapper (CSS, assets, server-only). Use a helper that merges
our ESM mapper with the resolved config's existing mappings instead.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(datasets): add type annotation to withEsmMapper parameter
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(datasets): add cancellation guard to NewDatasetItemForm schema effect
Rapidly switching datasets could let a stale promise overwrite the
correct placeholder. Add cancelled flag for the multi-dependency effect.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(datasets): prevent cascading re-runs and user input overwrite in schema effect
- Remove inputValue/expectedOutputValue from the effect dependency array
to prevent form.setValue triggering cascading effect re-runs
- Use form.getValues() at both dispatch and resolution time to avoid
overwriting content the user typed while generation was in-flight
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(datasets): add cancellation guard to DatasetSchemaHoverCard effect
For consistency with NewDatasetItemForm's async pattern.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* refactor(web): Make `useSidebarFilterState` state location more explicit
* Use discriminated union for state location & further cleanup
* Remove incorrect if condition
* Update peek readme
* Remove unneccessary usePeekTableState hooks
The SHA 5241b2e9 resolves to v1.6.0, not the floating v1 tag. Fixes
zizmor ref-version-mismatch alerts #500 and #501.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Fork PRs don't receive security-events: write on GITHUB_TOKEN, so the
SARIF upload path is unavailable. Previously the whole job was skipped
on fork PRs, meaning contributor changes to workflows bypassed zizmor.
Add a fork-PR step without advanced-security that fails the job on
findings so contributors see the error directly; keep the SARIF upload
step for trusted events so the code-scanning ruleset still blocks merges.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(web): warn about unencoded special characters in DATABASE_URL on migration failure
When Prisma migrations fail, the error message now detects whether the
DATABASE_URL credentials contain special characters that need percent-encoding
and prints an actionable hint with an example and documentation link.
Closes#3923
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(web): add ClickHouse password special character detection on migration failure
Extends the migration failure diagnostics to also check CLICKHOUSE_PASSWORD
for characters (&, =, #, ?, %, +, @) that would break the query-string
interpolation in the ClickHouse migration script.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* chore: feedback
* chore: patch
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Prefix every SCIM API log with [SCIM] so they can be filtered out of
aggregate logs. Add user-id-based confirmation logs after PUT/PATCH
provisioning and deprovisioning and after DELETE, mirroring the
existing POST assignment log, so operations can be traced without
emitting userName/email. Drop the email from the 409 already-exists
log in POST /Users and log the existing membership userId instead.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(security): use constant-time comparison for admin API key auth
Replace direct string comparison (!==) with crypto.timingSafeEqual for
ADMIN_API_KEY verification to prevent timing side-channel attacks
(CWE-208). This aligns with the secure pattern already used for project
API key authentication in createAuthedProjectAPIRoute.ts.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(security): handle timingSafeEqual byte-length mismatch in admin auth
Align AdminApiAuthService with the project-scoped admin-key check in
createAuthedProjectAPIRoute.ts: drop the JS-string-length pre-check (which
misreads UTF-8 byte length and can crash on multibyte tokens) and wrap
timingSafeEqual in try/catch. Remove the dead !env.ADMIN_API_KEY guard
already handled by the early-return. Replace the duplicated inline admin
key comparison in createNewSsoConfigHandler with AdminApiAuthService so
both admin paths share one timing-safe implementation, and add
cross-reference comments between the two remaining call sites.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(security): sanitize score config names to prevent CSS injection
Score config names were interpolated unsanitized into a <style
dangerouslySetInnerHTML> block in ChartStyle, allowing stored CSS
injection via crafted config names (e.g. `x}*{background:red}/*`).
- Sanitize ChartConfig keys at the render site in chart.tsx (defense in
depth for existing data)
- Add ScoreConfigNameSchema in shared domain with regex validation
(^[\w\s.()-]+$) for use at input boundaries
- Apply name validation to tRPC create/update and public API POST/PUT
score-config endpoints
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* chore: score config docs
* chore: udnerscores
* chore: adjust chart.tsx schema
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The zizmor ref-version-mismatch rule flags hash-pinned actions whose
version comment references a moving major tag (e.g. `# v2`) instead of
the exact release tag the hash belongs to. Update the three flagged
actions:
- aws-actions/amazon-ecr-login: `# v2` → `# v2.1.2`
- github/codeql-action/upload-sarif (snyk-worker): `# v4` → `# v4.35.1`
- github/codeql-action/upload-sarif (snyk-web): `# v4` → `# v4.35.1`
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(evals): return full JSONPath slice result and deduplicate eval JSONPath logic
JSONPath slice expressions (e.g. $[1:]) returned only the first matched
element due to an unconditional result[0] in parseJsonDefault. Now
multi-match results return the full array while single-match results
remain unwrapped for backward compatibility.
Also consolidates three separate JSONPath evaluation paths (UI preview,
trace eval, observation eval) into the shared extractValueFromObject,
removing duplicated logic from the worker. The snakeToCamel column ID
fallback and parseUnknownToString remain in the worker since they are
database-specific concerns.
Closes LFE-8416
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(evals): address PR review — scope parseMultiEncodedJson, fix error logging and test expectations
- Only call parseMultiEncodedJson when a jsonSelector is present to avoid
mutating formatting on the no-selector passthrough path.
- Preserve the raw original value (not the parsed one) in the error
fallback.
- Add error logging in extractObservationVariables (was already done in
parseDatabaseRowToString but missed here).
- Update three pre-existing test expectations to match the new unwrap
semantics: single-match results are unwrapped, non-matching paths
return empty string instead of "[]".
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(evals): update evalService test expectations for single-match unwrap
Four more test assertions in evalService.test.ts still expected
array-wrapped JSONPath results (e.g. '["Hello world"]'). Updated to
match the new unwrap semantics for single-match queries.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* test(evals): remove duplicate slice/single-element tests from extractObservationVariables
These cases are already covered by extractValueFromObject.test.ts.
The pre-existing tests ($.prompt, $.response, non-matching path) remain
as integration tests for the observation eval path.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* style(evals): use @langfuse/shared alias instead of relative path in test import
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* chore(ci): replace fkirc/skip-duplicate-actions with inline gh script
Remove third-party action dependency and replicate the tree-hash
deduplication logic using gh api. Compares the current commit's git
tree SHA against recent successful workflow runs to skip redundant CI.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* chore(ci): replace ravsamhq/notify-slack-action with slackapi/slack-github-action
Switch to the official Slack GitHub Action (v3.0.1) for failure
notifications. Uses Block Kit payload for richer messages with
branch/tag, actor, and a direct link to the workflow run.
Also removes the now-unnecessary SLACK_WEBHOOK_URL entry from the
zizmor secrets-outside-env allowlist since the webhook is now passed
via action input rather than env var.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Crafted OTel attribute keys like `gen_ai.prompt.__proto__.POLLUTED` could
pollute Object.prototype via the nested-object construction in
convertKeyPathToNestedObject. Guard against dangerous keys (__proto__,
constructor, prototype) and use Object.create(null) for result objects.
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(worker): sync managed evaluator vars on template updates
* chore: timestamp
* chore: filter based on project id
* Revert "chore: filter based on project id"
This reverts commit 132ac226ad68c3dec25194433e99f95261974294.
* fix: update log message for managed evaluators upsert completion
* perf(dual-write): clamp min start time to past day and optimize trace sorting
* chore: exclude project_id 'cmbktgdyf0059ad07yexqm2gp' from dual write
* chore: introducing LANGFUSE_EVENT_PROPAGATION_EXCLUDE_PROJECT_IDS
---------
Co-authored-by: Valery Meleshkin <valeriy@langfuse.com>
fix(ci): handle null/undefined security-severity in Snyk SARIF output
Snyk emits invalid security-severity values (null, "undefined", "null")
that cause codeql-action/upload-sarif to reject the file. Replace the
sed-based fix with jq to handle all non-numeric values.
See: https://github.com/github/codeql-action/issues/2187
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Slack natively timestamps every message. Our custom footer showed the
worker's server timezone which confused users in different timezones.
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(slack): show change author in Slack prompt notification
Display the user who made the change in the Slack notification message
for prompt version events. Falls back to email when name is unavailable,
and shows "API User" for API key-initiated changes.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(slack): escape mrkdwn in change author and use || for empty string fallback
Escape &, <, > in user name/email to prevent Slack mrkdwn injection
(e.g. <!channel> triggering mass notifications). Use || instead of ??
so empty string names fall back to email correctly.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(slack): escape all user-controlled mrkdwn fields in prompt notification
Apply escapeSlackMrkdwn to prompt.name, prompt.tags, and
prompt.commitMessage to prevent injection via those fields too.
Labels are safe (validated by PROMPT_LABEL_REGEX).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Add zizmor workflow and skip forked Claude review PRs
* ci: test if workflow breaks
* ci: add dependabot cooldown
* ci: zizmor auto-fixes
* ci: harden GitHub Actions workflows
* ci: refine zizmor workflow configuration
* ci: scope sdk and snyk secrets to environments
* ci: address zizmor workflow review feedback
* ci: fix license check
* ci: align sdk workflow secret handling
* ci: enable snyk checks on pull requests
* ci: remove temporary snyk pull request trigger
* ci: bump back to the zizmor minimum of 7 days
* style: move to nicer config syntax for secrets-outside-of-env
* ci: fix template-injection warnings in pipeline digest step
Move step outputs and matrix values from ${{ }} interpolation in run
blocks to env variables, preventing potential shell code injection.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(ci): fix broken heredoc expansion in Docker publish steps
The publish-manifest steps used <<'EOF' (single-quoted heredoc) which
suppresses bash variable expansion, and the env vars were never defined
in those steps. This meant every tag-triggered release would fail with
literal ${VAR} strings passed as Docker tags.
- Change <<'EOF' to <<EOF to enable variable expansion
- Add env: blocks defining STEPS_META_*_OUTPUTS_TAGS from step outputs
- Move remaining ${{ matrix.* }} interpolations to env vars
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* ci: rename reserved GITHUB_ env var prefix to INPUT_
Rename GITHUB_EVENT_INPUTS_CONFIRM to INPUT_CONFIRM. GitHub reserves
the GITHUB_ prefix for built-in runner variables.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* ci: use environment secret instead of inherit
* ci: switch to latest action
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(shared): treat end-of-life model errors as non-retryable
Extract non-retryable error patterns into a shared constant and add
"reached the end of its life" to the list so that Bedrock end-of-life
model errors surface immediately instead of being retried.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* refactor(shared): keep isNonRetryableLLMErrorMessage private
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(shared): extract status code from AWS SDK $metadata.httpStatusCode
The AWS SDK puts the HTTP status on `$metadata.httpStatusCode`, not on
`.status` or `.response.status`. Without this, the fallback defaulted to
500, making 4xx errors appear retryable.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* test: use real ResourceNotFoundException from AWS SDK
Instead of hardcoding the error shape, resolve and instantiate the real
`ResourceNotFoundException` from `@aws-sdk/client-bedrock-runtime` via
`@langchain/aws`'s dependency tree.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* OCI Object Storage Native SDK client Integration for StorageService with identity access Management options.
Updated tests accordingly.
Updated docker file with env variables and build options to build from code.
Updated package json file with OCI libraries used for Object Storage.
Added a sample .env file with instructions on how to use workload_identity' | 'instance_principal' | 'resource_principal' | 'oci_profile' | 'session_token.
Added !.env.dev-oci.example to gitignore to commit the file.
* Update StorageService.ts
Fixed Lint errors
* Added the pnpm lock file
* Add uploadFileBuffered per upstream PR requirements; rename env vars to langfuse_ prefix
Implemented uploadFileBuffered to satisfy requirements introduced by an upstream pull request.
Updated environment variable names to use the langfuse_ prefix for consistency/alignment
* Remove prisma-extension-kysely dependency
* Remove prisma-extension-kysely from pnpm-lock.yaml
Removed prisma-extension-kysely dependency and related entries.
---------
Co-authored-by: Steffen Schmitz <steffen@langfuse.com>
* feat(experiments): direct-write prompt experiment root events
* feat(tracing): centralize internal direct event writes
* push
* chore: move asRecord function to utils and update references in experiment service
* chore: move type coercion functions to utils for better organization and reuse
* chore(tracing): refactor internal tracing to use new events writer interface
* fix(experimentService): fix dataset item version conversion
* fix: remove invalid dependency
* fixup(tracing): ensure ordering key parity with experiment backfill
* fix: do not write to events table for self-hosters
* test: fix
* test: fix
* fix: do not write to events-table for self-hosters
* fix: rebase
* chore: type
* fix: skip remapping of IDs for self-hosters
* fix: test
* chore: push
* chore: move away from de-duplication approach
* chore: push
---------
Co-authored-by: Marlies Mayerhofer <74332854+marliessophie@users.noreply.github.com>
* fix(web): Create new `TableCellWithCopyButton` for `ApiKeyList`
* Fix react re-rendering issue after copying text
* Handle rejections when copying to clipboard
* Simplify useCopyToClipboard tests
* fix(web): Prevent toast error when toggling v4 with selected saved view
* Prevent table being rendered until flag is initialized
* Add mistakenly removed "as const" to StringParam
* chore(experiments): rewrite metrics aggregation for total cost and latency to skip trace-level aggregation
* fix(experiments): handle null values in latency and total cost cells in ExperimentsTable
* fix: typo
* feat(web): add support for AWS Bedrock API Keys (Bearer Tokens)
Add Bedrock API key authentication as an alternative to AWS access keys
(SigV4) for Amazon Bedrock LLM connections. Users can now choose between
AWS access keys and Bedrock API keys via a tab-based selector in the UI.
- Add BedrockApiKeySchema and BedrockAccessKeysSchema as a discriminated
union in shared credential schemas
- Add resolveBedrockAuth() to route between bearer token and SigV4 auth
- Add server-side validation of Bedrock credentials on create and update
- Derive and expose a safe authMethod enum (api-key, access-keys,
default-credentials) in the tRPC list response without leaking secrets
- Add auth method tab selector to the create/update LLM API key form
- Add Bedrock credential validation to the public API PUT endpoint
- Add comprehensive unit, integration, and e2e tests
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* ci: add secret
* fix(web): fix Bedrock DefaultCredentials test for cloud environment
The test assumed updating to BEDROCK_USE_DEFAULT_CREDENTIALS would
succeed, but the test env sets NEXT_PUBLIC_LANGFUSE_CLOUD_REGION="DEV"
which makes the server reject default credentials. Updated the test to
assert the expected rejection on cloud deployments.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(web): add self-hosted happy path test for DefaultCredentials update
The previous fix only asserted cloud rejection. Add back the original
happy-path test that temporarily sets NEXT_PUBLIC_LANGFUSE_CLOUD_REGION
to undefined (simulating self-hosted) so the update-to-DefaultCredentials
path is actually exercised.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(web): add .min(1) to BedrockAccessKeysSchema, guard default creds in public API
- Add .min(1) to accessKeyId and secretAccessKey in BedrockAccessKeysSchema
to reject empty-string credentials at validation time
- Add cloud guard in PUT /api/public/llm-connections rejecting the
BEDROCK_USE_DEFAULT_CREDENTIALS sentinel on Langfuse Cloud
- Fix DefaultCredentials tests: use per-test env override with try/finally
to simulate self-hosted deployments
- Add public API tests for sentinel rejection and invalid credential JSON
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The backfill cursor was only advanced inside the chunk-processing loop,
which is never reached when the query returns zero dataset run items.
This caused last_run_delay_seconds to grow indefinitely in environments
with no recent experiment activity.
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: validate Azure blob storage container names
Azure requires container names to be 3-63 chars, lowercase alphanumeric
and hyphens only. Add Zod superRefine validation to the form schema,
tRPC router, and public API schema so invalid names like "Feedback N8N Bot"
are rejected at submission time with a clear error message.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: address PR review feedback for Azure container name validation
- Add empty-string guard in validateAzureContainerName to avoid double
error when bucketName is blank
- Add .min(1) to public API bucketName schema to match tRPC form schema
- Add Fern docs note describing Azure container naming constraints
- Add server test for invalid Azure container name rejection
- Add client test for empty-string guard behavior
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(web): Table padding issues
* Increase cell padding in `SelectDashboardDialog` and `SelectWidgetDialog`
* Fix memoization comparison for cellPadding in DataTable
* Set cellPadding="comfortable" for `MembersTable` in org settings
* fix(web): allow unicode letters in signup name validation
* refactor(web): share name schema between signup and display name
* fix(web): enforce 100-char limit in shared name schema
* fix(web): allow hyphens, apostrophes, and periods in name validation
The nameSchema regex was too strict, rejecting common name characters
like O'Brien, Smith-Jones, and Dr. Smith. Also align the backend
updateDisplayName schema with the shared nameSchema for consistency.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(web): normalize smart quotes and require letter in name validation
Normalize curly/smart apostrophes (U+2018, U+2019, U+02BC) from mobile
autocorrect to straight apostrophe before validation. Require at least
one letter to reject degenerate punctuation-only names like "---".
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(web): require base letter not combining mark in name validation
The "must contain at least one letter" refine accepted standalone
combining marks (\p{M}) without an actual letter (\p{L}), allowing
inputs like "\u0301\u0301" to pass as valid names.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(web): require base letter not combining mark in name validation
Add NFC normalization before validation so decomposed characters merge
into precomposed form, and add a negative lookahead (?!\p{M}) to reject
names that still start with a combining mark after normalization.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(web): revert display name form to permissive schema and simplify nameSchema
Revert settings and userAccount display name validation back to
StringNoHTML.min(1).max(100) — the signup-oriented nameSchema is too
restrictive for existing display names containing underscores, ampersands, etc.
Simplify nameSchema: merge transforms, combine regex constraints into a single
refine that requires names start with a letter, and remove U+02BC from
smart-quote normalization (it's a linguistic letter, not a typographic quote).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: make Slack integration more robust
* refactor: deduplicate scopes
* fix(slack): make SlackChannel isPrivate and isMember optional
These fields are only known for channels from the fetched list, not for
manually-typed channel names. Making them optional avoids placeholder
booleans and fixes a type error when constructing partial channel objects.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* chore: gracefully handle missing scopes
* fix(slack): cap rate-limit retry, resolve manual channel IDs, add empty state
- Cap retryAfter to 60s max to avoid gateway timeouts on large Slack values
- Add onSuccess handler in SlackActionForm to resolve #channel names to real IDs
- Show empty state message in ChannelSelector when bot has no accessible channels
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(slack): resolve manual channel IDs, virtualize list, fix audit log
- Use resolved Slack channel ID in audit log instead of #-prefixed input
- Replace VirtualizedList with cmdk Command + @tanstack/react-virtual
for keyboard navigation and DOM-efficient rendering of ~5k channels
- Move "Use typed name" fallback to separate CommandGroup so it stays
visible when the virtualized group has zero height
- Import SlackChannel type from @langfuse/shared instead of redeclaring
- Add getChannelInfo mock and #-prefixed channelId test
- Use .concat() instead of spread for channel pagination (repo convention)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* refactor: switch to SDK retry policies
* fix(slack): strip duplicate # prefix and use functional setState
Strip leading # from channelId fallback in test message block to avoid
displaying ##general for manually-typed channel names. Use functional
setSelectedChannel form in slack.tsx to match SlackActionForm.tsx pattern.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(slack): guard CommandEmpty on filteredChannels length
Prevent flash of "No channels available." on popover open by explicitly
guarding CommandEmpty rendering on filteredChannels.length === 0 instead
of relying on cmdk's internal item count, which is 0 on the first
render before the virtualizer scroll container mounts.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* chore: review comments
* chore: another round of review feedback
* chore: more review comments
* fix(slack): improve channel selector search
* fix comment
* fix(slack): refine channel selector search
* fix(slack): sync manifest scopes
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
-Install Playwright Chromium: `pnpm run playwright:install`
Minimum verification matrix:
## Verification
| Change scope | Minimum verification |
| --- | --- |
|`web/**` only | `pnpm --filter web run lint` + targeted web tests |
| `worker/**` only | `pnpm --filter worker run lint`+ targeted worker tests |
|`packages/shared/**` (non-schema) | `pnpm --filter @langfuse/shared run lint` + one targeted web check + one targeted worker check |
| `packages/shared/prisma/**` or `packages/shared/clickhouse/**` | `pnpm --filter @langfuse/shared run lint` +`pnpm run db:generate` + targeted web/worker regressions |
| Public API contract (`web/src/pages/api/public/**`, `web/src/features/public-api/types/**`, `fern/apis/**`) | web lint + targeted server API tests + Fern update/regeneration; never hand-edit `generated/**` |
| Cross-package refactor (`web` + `worker` + `shared`) | `pnpm run lint` + `pnpm run typecheck` + targeted tests per impacted package |
-`web/**`: `pnpm run lint` plus targeted web tests.
-`worker/**`: `pnpm run lint` plus targeted worker tests.
-`packages/shared/**` non-schema changes:
`pnpm run lint`plus one targeted web check and one targeted worker check.
-`packages/shared/prisma/**` or `packages/shared/clickhouse/**`:
`pnpm run lint`,`pnpm run db:generate`, and targeted web/worker
regressions.
- Public API contracts in `web/src/pages/api/public/**`,
`web/src/features/public-api/types/**`, or `fern/apis/**`: `pnpm run lint`,
targeted server API tests, and Fern update/regeneration.
- Cross-package refactors: `pnpm run lint`, `pnpm run typecheck`, and targeted
tests for impacted packages.
## Repo Rules
## Generated Files
- Keep changes scoped; avoid unrelated refactors.
- Prefer package-local implementation details in package `AGENTS.md` files.
-Do not hand-edit generated/build artifacts:
-`generated/*`
-`web/.next/*`
-`web/.next-check/*`
-`*/dist/*`
-`packages/shared/prisma/generated/*`
-Public API contract changes must update Fern sources in `fern/apis/**` and
regenerated outputs; never hand-edit `generated/**`.
- Keep tests independent and parallel-safe.
- For bug fixes, write the failing test first, confirm it fails, then fix the
bug.
- For user-visible frontend changes in `web/**`, review the affected flow in a
real browser with the Playwright MCP server before signoff. Use
`skills/frontend-browser-review/SKILL.md` and `../web/AGENTS.md` for the
browser-review loop.
- Never commit secrets or credentials. Keep `.env*.example` files in sync with
required env vars.
Do not hand-edit generated or build artifacts:
-`generated/*`
-`web/.next/*`
-`web/.next-check/*`
-`*/dist/*`
-`packages/shared/prisma/generated/*`
Public API contract changes must update Fern sources in `fern/apis/**` and
regenerated outputs. Never hand-edit `generated/**`.
## Shared Agent Setup
-`.agents/AGENTS.md` is the canonical root guide.
- Root `AGENTS.md` is a symlink to `.agents/AGENTS.md`.
- Root `CLAUDE.md` is a compatibility symlink to `AGENTS.md`.
-Shared agent/tool config lives in `config.json` and shared skills live in
`skills/`.
- Project-scoped provider discovery files are generated local artifacts. Edit
the canonical files under `.agents/` instead of editing generated tool
directories by hand.
-If you change `.agents/config.json`, `skills/**`, or the shim-generation
workflow, run:
-`pnpm run agents:sync`
-`pnpm run agents:check`
- Do not commit generated provider config or shim outputs under `.claude/`,
`.cursor/`, `.codex/`, `.vscode/`, or `.mcp.json`.
- Durable cross-tool guidance belongs in root/package `AGENTS.md` files or
`skills/**`, not only in tool-specific config directories.
## Commit, PR, and Release Rules
- Commit messages and PR titles must follow Conventional Commits:
`type(scope): description` or `type: description`.
- PR titles are validated by `.github/workflows/validate-pr-title.yml`.
- In PR descriptions, list impacted packages and executed verification commands.
- Release workflow is managed at root with `pnpm run release`.
- Promote `main` to `production` via
`.github/workflows/promote-main-to-production.yml` or
`pnpm run release:cloud`.
- Do not change release/versioning flow without updating this file and impacted
package guides.
## Git and Tooling Notes
- Use `gh search issues` for GitHub issue search.
- Do not use destructive git commands such as `reset --hard` unless explicitly
requested.
- Do not revert unrelated working-tree changes.
- Keep commits focused and atomic.
- Remaining `.cursor/rules/*.mdc` files should stay thin wrappers around shared
docs or skills rather than owning durable repo guidance directly.
-When creating or editing`.agents/skills/**`, use
`.agents/skills/skill-creator/SKILL.md`; keep skills concise with
progressive disclosure.
- After changing shared agent setup, run `pnpm run agents:sync` and
`pnpm run agents:check`.
-Generated provider config and shim outputs under `.claude/`, `.cursor/`,
`.codex/`, `.vscode/`, or `.mcp.json` are local artifacts, not source of
| Schema and tier rules | You need the entry shape or pricing-tier invariants | [references/schema-and-tiers.md](references/schema-and-tiers.md) |
| Provider sources and price keys | You need official pricing URLs, per-token conversion, or provider-specific usage keys | [references/provider-sources-and-price-keys.md](references/provider-sources-and-price-keys.md) |
| Match patterns | You are editing `matchPattern` regexes or provider coverage | [references/match-patterns.md](references/match-patterns.md) |
| Workflow and validation | You are applying the end-to-end edit process or checking common mistakes | [references/workflow-and-validation.md](references/workflow-and-validation.md) |
| Match patterns | You are editing `matchPattern` regexes or provider coverage | [references/match-patterns.md](references/match-patterns.md) |
| Workflow and validation | You are applying the end-to-end edit process or checking common mistakes | [references/workflow-and-validation.md](references/workflow-and-validation.md) |
Establish consistency and best practices across Langfuse's backend packages (web, worker, packages/shared) using Next.js 14, tRPC, BullMQ, and TypeScript patterns.
## When to Use This Skill
Use this guide when working on:
- Creating or modifying tRPC routers and procedures
- Creating or modifying public API endpoints (REST)
- Creating or modifying BullMQ queue consumers and producers
- Building services with business logic
- Authenticating API requests
- Accessing resources based on entitlements
- Implementing middleware (tRPC, NextAuth, public API)
- Database operations with Prisma (PostgreSQL) or ClickHouse
- Observability with OpenTelemetry, DataDog, logger, and traceException
- Input validation with Zod v4
- Environment configuration from env variables
- Backend testing and refactoring
---
## Quick Start
### UI: New tRPC Feature Checklist (Web)
- [ ]**Router**: Define in `features/[feature]/server/*Router.ts`
- [ ]**Procedures**: Use appropriate procedure type (protected, public)
- [ ]**Authentication**: Use JWT authorization via middlewares.
- [ ]**Entitlement check**: Access resources based on resource and role
- [ ]**Validation**: Zod v4 schema for input
- [ ]**Service**: Business logic in service file
- [ ]**Error handling**: Use traceException wrapper
- [ ]**Tests**: Unit + integration tests in `__tests__/`
- [ ]**Config**: Access via env.mjs
### SDKs: New Public API Endpoint Checklist (Web)
- [ ]**Route file**: Create in `pages/api/public/`
- [ ]**Wrapper**: Use `withMiddlewares` + `createAuthedProjectAPIRoute`
- [ ]**Types**: Define in `features/public-api/types/`
- [ ]**Authentication**: Authorization via basic auth
- [ ]**Validation**: Zod schemas for query/body/response
- [ ]**Versioning**: Versioning in API path and Zod schemas for query/body/response
- [ ]**Fern API Docs**: Update `fern/apis/server/definition/` to match TypeScript types
- [ ]**Tests**: Add end-to-end test in `__tests__/async/`
### New Queue Processor Checklist (Worker)
- [ ]**Processor**: Create in `worker/src/queues/`
- [ ]**Queue types**: Create queue types in `packages/shared/src/server/queues`
- [ ]**Service**: Business logic in `features/` or `worker/src/features/`
- [ ]**Error handling**: Distinguish between errors which should fail queue processing and errors which should result in a succeeded event.
- [ ]**Queue registration**: Add to WorkerManager in app.ts
- [ ]**Tests**: Add vitest tests in worker
---
## Architecture Overview
### Layered Architecture
```
# Web Package (Next.js 14)
┌─ tRPC API ──────────────────┐ ┌── Public REST API ──────────┐
The shared package provides types, utilities, and server code used by both web and worker packages. It has **5 export paths** that control frontend vs backend access:
- Use unique IDs (`randomUUID()`) to avoid test interference
- Clean up test data or use unique project IDs
- Tests must be independent and runnable in any order
- Prefer scoped cleanup or unique project IDs over global reset helpers
### 8. Always Filter by projectId for Tenant Isolation
```typescript
// ✅ CORRECT: Filter by projectId for tenant isolation
consttrace=awaitprisma.trace.findUnique({
where:{id: traceId,projectId},// Required for multi-tenant data isolation
});
// ✅ CORRECT: ClickHouse queries also require projectId
consttraces=awaitqueryClickhouse({
query:`
SELECT * FROM traces
WHERE project_id = {projectId: String}
AND timestamp >= {startTime: DateTime64(3)}
`,
params:{projectId,startTime},
});
```
### 9. Keep Fern API Definitions in Sync with TypeScript Types
When modifying public API types in `web/src/features/public-api/types/`, the corresponding Fern API definitions in `fern/apis/server/definition/` must be updated to match.
Service layer overview, dependency injection patterns, singleton patterns, repository pattern for data access, service design principles, caching strategies, testing services
Dual database architecture (PostgreSQL via Prisma + ClickHouse via direct client), PostgreSQL CRUD operations, ClickHouse query patterns (queryClickhouse, queryClickhouseStream, upsertClickhouse), repository pattern for complex queries, tenant isolation with projectId filtering, when to use which database
Environment variable validation with Zod, package-specific configs (web/env.mjs with t3-oss/env-nextjs, worker/env.ts, shared/env.ts), NEXT_PUBLIC_LANGFUSE_CLOUD_REGION usage, LANGFUSE_EE_LICENSE_KEY for enterprise features, best practices for env management
Integration tests (Public API with makeZodVerifiedAPICall), tRPC tests (createInnerTRPCContext, appRouter.createCaller), service-level tests (repository/service functions), worker tests (vitest with streams), test isolation principles, running tests (Jest for web, vitest for worker)
description:Shared backend guide for Langfuse's Next.js 14, tRPC, BullMQ, and TypeScript monorepo. Use when creating or reviewing tRPC routers, public REST endpoints, BullMQ queue processors, backend services, middleware, Prisma or ClickHouse data access, OpenTelemetry instrumentation, Zod validation, env configuration, or backend tests across web, worker, or packages/shared.
description:Shared backend guide for Langfuse's Next.js, tRPC, BullMQ, and TypeScript monorepo. Use when creating or reviewing tRPC routers, public REST endpoints, BullMQ queue processors, backend services, middleware, Prisma or ClickHouse data access, OpenTelemetry instrumentation, Zod validation, env configuration, or backend tests across web, worker, or packages/shared.
---
# Backend Development Guidelines
@@ -20,24 +20,97 @@ Use this skill for backend and API work across `web/`, `worker/`, and
## How to Read This Skill
-Start with [AGENTS.md](AGENTS.md) when the task spans multiple backend areas
or you need the end-to-end checklists.
-Use this `SKILL.md` when the task spans multiple backend areas or you need the
end-to-end reference map.
- Read only the specific reference file that matches the work when the scope is
narrower.
- If the task introduces a user-supplied URL, an outbound HTTP request, a new
integration, or touches secrets, RBAC, or redirect handling, also load the
shared [`security-review`](../security-review/SKILL.md) skill before
designing or implementing the change.
## Quick Start Checklists
### UI: New tRPC Feature
- Define the router in `features/[feature]/server/*Router.ts`.
- Use the appropriate protected or public procedure.
- Authenticate with JWT-aware middleware.
- Check project/resource access and entitlements.
- Validate input with Zod v4.
- Put business logic in a service file.
- Use `traceException` for error handling where relevant.
- Add unit or integration tests in `__tests__/`.
- Access config via `env.mjs`.
### SDKs: New Public API Endpoint
- Create the route in `pages/api/public/`.
- Wrap it with `withMiddlewares` and `createAuthedProjectAPIRoute`.
- Define types in `features/public-api/types/`.
- Authenticate with basic auth.
- Validate query, body, and response with Zod schemas.
- Include API versioning in paths and schemas.
- Update Fern API definitions to match TypeScript types.
- Add end-to-end tests in `__tests__/async/`.
### Worker: New Queue Processor
- Create the processor in `worker/src/queues/`.
- Define queue types in `packages/shared/src/server/queues`.
- Place business logic in `features/` or `worker/src/features/`.
- Distinguish failed jobs from jobs that should succeed with a recorded error.
- Register the queue in `WorkerManager` in `app.ts`.
- Add worker vitest coverage.
## Core Principles
- tRPC procedures, public API routes, and queue processors delegate business
logic to services.
- Access configuration through `env.mjs`; do not read `process.env` directly
outside env setup.
- Validate all external input with Zod v4.
- Use Prisma directly for simple CRUD and repositories for complex query access.
- Use OpenTelemetry and DataDog for backend observability.
- Always filter project-scoped database queries by `projectId`.
- Keep Fern API definitions in sync with public TypeScript API contracts.
- Keep backend tests independent and parallel-safe.
## Live Examples
- tRPC router with project auth and Zod input:
`web/src/features/events/server/eventsRouter.ts`.
- Public API route with middleware and typed request/response schemas:
`web/src/pages/api/public/datasets/index.ts`.
- Worker queue processor with typed jobs, logging, and retry behavior:
`worker/src/queues/evalQueue.ts`.
- Tenant filters for Prisma and ClickHouse:
`references/database-patterns.md`.
## Naming Conventions
- tRPC routers: `camelCaseRouter.ts`, for example `datasetRouter.ts`.
- Services: `service.ts` in the feature server directory.
- Queue processors: `camelCaseQueue.ts`, for example `evalQueue.ts`.
- Public API routes: kebab-case filenames, for example `dataset-items.ts`.
## Anti-Patterns to Avoid
- Business logic in routes or procedures.
- Direct `process.env` usage instead of `env.mjs` / `env.ts`.
- Missing error handling.
- Missing input validation.
- Missing `projectId` filters on tenant-scoped queries.
-`console.log` instead of `logger` / `traceException`.
## Reference Map
| Topic | Read this when | File |
| --- | --- | --- |
| Architecture and package boundaries | You need the web/worker/shared split, request flow, or queue lifecycle | [references/architecture-overview.md](references/architecture-overview.md) |
| Routing and controllers | You are writing tRPC procedures, public API routes, or queue entrypoints | [references/routing-and-controllers.md](references/routing-and-controllers.md) |
| Middleware and auth | You are changing request auth, permissions, or middleware composition | [references/middleware-guide.md](references/middleware-guide.md) |
| Services and repositories | You are placing business logic, repository code, or DI patterns | [references/services-and-repositories.md](references/services-and-repositories.md) |
| Database access | You are touching Prisma, ClickHouse, tenant filters, or query patterns | [references/database-patterns.md](references/database-patterns.md) |
| Configuration | You are adding env vars, startup config, or runtime toggles | [references/configuration.md](references/configuration.md) |
| Testing | You are adding or updating backend tests | [references/testing-guide.md](references/testing-guide.md) |
## Full Compiled Guide
Read [AGENTS.md](AGENTS.md) for the complete backend guide with checklists,
directory conventions, imports, architecture, and cross-cutting practices.
| Architecture and package boundaries | You need the web/worker/shared split, request flow, or queue lifecycle | [references/architecture-overview.md](references/architecture-overview.md) |
| Routing and controllers | You are writing tRPC procedures, public API routes, or queue entrypoints | [references/routing-and-controllers.md](references/routing-and-controllers.md) |
| Middleware and auth | You are changing request auth, permissions, or middleware composition | [references/middleware-guide.md](references/middleware-guide.md) |
| Services and repositories | You are placing business logic, repository code, or DI patterns | [references/services-and-repositories.md](references/services-and-repositories.md) |
| Database access | You are touching Prisma, ClickHouse, tenant filters, or query patterns | [references/database-patterns.md](references/database-patterns.md) |
| Configuration | You are adding env vars, startup config, or runtime toggles | [references/configuration.md](references/configuration.md) |
| Testing | You are adding or updating backend tests | [references/testing-guide.md](references/testing-guide.md) |
Complete guide to the layered architecture pattern used in Langfuse's Next.js 14/tRPC/Express monorepo.
Complete guide to the layered architecture pattern used in Langfuse's Next.js/tRPC/Express monorepo. Check package manifests such as `web/package.json` for current framework versions before version-sensitive work.
## Table of Contents
@@ -20,7 +20,7 @@ Langfuse uses a **three-layer architecture** with two primary entry points (tRPC
### The Three Layers
```
# Web Package (Next.js 14)
# Web Package (Next.js)
┌─ tRPC API ──────────────────┐ ┌── Public REST API ──────────┐
1.**Test Isolation**: Each test should be independent and runnable in any order
2.**Unique IDs**: Use `randomUUID()` or unique project IDs to avoid test interference
3.**Cleanup**: Always clean up test data in service tests (or use unique project IDs)
4.**Avoid Global Resets**: Prefer scoped cleanup or unique project IDs over global reset helpers
5.**Flags and Fallbacks**: When code branches on env flags, feature flags, or fallback data paths, test both branches and ensure fixtures are written to the same store the branch reads from
**Why rules take priority:** ClickHouse has specific behaviors (columnar storage, sparse indexes, merge tree mechanics) where general database intuition can be misleading. The rules encode validated, ClickHouse-specific guidance.
### For Formal Reviews
## Langfuse-Specific Rules
When performing a formal review of schemas, queries, or data ingestion:
- Use `packages/shared/src/server/queries/clickhouse-sql/event-query-builder.ts`
for queries against the `events` table. Do not hand-roll `events` SQL unless
you first confirm the query builder cannot express the query.
- Never use `FINAL` on the `events` table; it is designed so `FINAL` is not
required and the keyword hurts performance.
- Any migration in `packages/shared/clickhouse/migrations/clustered/**` with
more than one `ALTER` on the same table must end every metadata `ALTER`
(`ADD/DROP/MODIFY COLUMN`, `ADD/DROP INDEX`) with `SETTINGS alter_sync = 2`,
and every mutation-creating `ALTER` (`MATERIALIZE …`, `UPDATE`, `DELETE`)
with `SETTINGS mutations_sync = 2`. The matching `unclustered/` file runs
against plain `MergeTree` and does not need (and should not duplicate)
these settings.
---
@@ -48,11 +59,13 @@ When performing a formal review of schemas, queries, or data ingestion:
- [ ] ReplacingMergeTree has version column if used
- [ ] Clustered migration files with multiple ALTERs on the same table use `SETTINGS alter_sync = 2` (metadata) and `SETTINGS mutations_sync = 2` (`MATERIALIZE …`, `UPDATE`, `DELETE`); unclustered mirror has none
### For Query Reviews (SELECT, JOIN, aggregations)
@@ -65,6 +78,7 @@ When performing a formal review of schemas, queries, or data ingestion:
5.`rules/schema-pk-filter-on-orderby.md` - Filter alignment with ORDER BY
**Check for:**
- [ ] Filters use ORDER BY prefix columns
- [ ] JOINs filter tables before joining (not after)
- [ ] Correct JOIN algorithm for table sizes
@@ -81,6 +95,7 @@ When performing a formal review of schemas, queries, or data ingestion:
env:<env> (service:worker OR service:worker-cpu) operation_name:bullmq.process
```
Then group by `resource_name`, queue facets such as `bullmq.queue` or
`messaging.*`, and error fields. Facet names can differ between Datadog sites,
so inspect one sample span before relying on a specific facet.
Queue-specific starter query:
```text
env:<env> (service:worker OR service:worker-cpu) operation_name:bullmq.process (resource_name:"process otel-ingestion-queue" OR resource_name:"Worker.run otel-ingestion-queue" OR bullmq.queue:otel-ingestion-queue)
```
For sharded queues, query the base queue and shard suffixes:
```text
env:<env> (service:worker OR service:worker-cpu) operation_name:bullmq.process resource_name:"*otel-ingestion-queue*"
```
If a queue file wraps the handler with `instrumentAsync`, also search the
| Logger / instrumentation | `packages/shared/src/server/logger.ts`, `packages/shared/src/server/instrumentation.ts` | log silently dropped because `LANGFUSE_LOG_LEVEL` set wrong, or span missing because handler doesn't call `instrumentAsync` |
| Webhook URL validation | `packages/shared/src/server/validateWebhookURL.ts` | rejects with messages that *look* like DNS errors but are SSRF guard rejections |
| Encryption | `packages/shared/encryption` | bad keys → 403/auth-style failures masquerading as upstream errors |
## Common Symptoms → First Files To Read
- **"403 from upstream":** check the per-integration credentials table in
short_description:"Build, change, or refactor React features"
default_prompt:"Use $frontend-large-feature-architecture to build, change, or refactor this large frontend surface with clear state ownership, stable subscriptions, and view-only render boundaries."
default_prompt:"Use $pnpm-upgrade-package to upgrade a package in this pnpm workspace, asking me for the package or version if I did not provide them."
description: Security review patterns for Langfuse. Use during code review, design, or planning whenever a change accepts user-supplied URLs, host/endpoint/baseURL fields, secrets, cross-tenant data, new outbound HTTP requests, new integrations (webhooks, blob storage, LLM connections, image proxies), redirect-following behavior, or new auth/permission scopes. Covers SSRF/outbound URL validation today and is intentionally extensible to other recurring security findings (tenant isolation, secret handling, redirect mishandling, file upload, RBAC scope drift).
---
# Security Review
Use this skill when reviewing or planning code that touches a security-sensitive
surface in Langfuse. It collects the recurring findings the team has seen in
external security reports so that future agents catch them at design and review
time rather than after the fact.
## When to Apply
Apply this skill when the change touches any of:
- a user-supplied URL, host, endpoint, `baseURL`, or webhook target
- a new outbound HTTP request (`fetch`, `axios`, AWS SDK client init with a
custom `endpoint`, OpenAI/Anthropic/Bedrock client init with a custom
`baseURL`, etc.)
- a new integration form under Settings -> Integrations or any
admin-configurable network destination
- a new tRPC procedure or public API route that mutates project-scoped data or
changes who can access it
- secrets, API keys, signing secrets, or encryption-at-rest fields
- redirect-following or cross-origin header handling
- file uploads, image proxies, or other binary data flowing in or out
Apply this skill during **plan mode** when designing a new integration so the
correct validation surfaces land in the plan, not in a follow-up CVE.
## How to Read This Skill
1. Open [references/checklist.md](references/checklist.md) and run the mental
sweep against the change.
2. For each bullet that fires, open the matching topic reference.
| Topic | Open when | File |
| --- | --- | --- |
| SSRF and outbound URL validation | The change accepts or fetches a user-supplied URL, host, or endpoint | [references/outbound-url-validation.md](references/outbound-url-validation.md) |
The catalog is intentionally short today. New topic files are added as new
finding classes recur (see "Extending This Skill").
## Output Expectations (Review Mode)
When this skill is used during code review:
- List findings first, ordered by severity, with file and line references.
- For each finding, name the canonical helper or known-good call site the
author should copy.
- For SSRF-class findings, point at [references/outbound-url-validation.md](references/outbound-url-validation.md)
rather than re-deriving the fix.
- Call out missing **negative tests** (private-IP, cross-tenant, missing-scope)
as findings, not as nice-to-haves.
## Output Expectations (Design / Plan Mode)
When this skill is used while planning:
- Restate which surfaces the new feature exposes (forms, public API routes,
worker entrypoints).
- For each surface that matches a checklist trigger, name the validator or
helper that must be invoked and at which layer (save-time, use-time,
connection-time, redirect-time).
- Treat "we will validate later" as a design defect: validation belongs in the
same change that introduces the surface.
## Extending This Skill
Add a new `references/<topic>.md` whenever a security finding recurs across
features or PR reviews. Keep each reference narrow and concrete:
description: Guide for creating effective skills. This skill should be used when users want to create a new skill (or update an existing skill) that extends Codex's capabilities with specialized knowledge, workflows, or tool integrations.
metadata:
short-description: Create or update Codex skills
---
# Skill Creator
This skill provides guidance for creating effective skills.
## About Skills
Skills are modular, self-contained folders that extend Codex's capabilities by providing
specialized knowledge, workflows, and tools. Think of them as "onboarding guides" for specific
domains or tasks—they transform Codex from a general-purpose agent into a specialized agent
equipped with procedural knowledge that no model can fully possess.
### What Skills Provide
1. Specialized workflows - Multi-step procedures for specific domains
2. Tool integrations - Instructions for working with specific file formats or APIs
3. Domain expertise - Company-specific knowledge, schemas, business logic
4. Bundled resources - Scripts, references, and assets for complex and repetitive tasks
## Core Principles
### Concise is Key
The context window is a public good. Skills share the context window with everything else Codex needs: system prompt, conversation history, other Skills' metadata, and the actual user request.
**Default assumption: Codex is already very smart.** Only add context Codex doesn't already have. Challenge each piece of information: "Does Codex really need this explanation?" and "Does this paragraph justify its token cost?"
Prefer concise examples over verbose explanations.
### Set Appropriate Degrees of Freedom
Match the level of specificity to the task's fragility and variability:
**High freedom (text-based instructions)**: Use when multiple approaches are valid, decisions depend on context, or heuristics guide the approach.
**Medium freedom (pseudocode or scripts with parameters)**: Use when a preferred pattern exists, some variation is acceptable, or configuration affects behavior.
**Low freedom (specific scripts, few parameters)**: Use when operations are fragile and error-prone, consistency is critical, or a specific sequence must be followed.
Think of Codex as exploring a path: a narrow bridge with cliffs needs specific guardrails (low freedom), while an open field allows many routes (high freedom).
### Require Human Review for Ticket Writes
For skills that create Linear tickets, update Linear tickets, or add evidence to
existing tickets, require human review before any write. The skill must present
all findings in a table, ask the human which findings to create or update in
Linear, and wait for an explicit selection before making changes.
Use this table structure unless the domain needs additional columns:
| ID | Finding | Evidence | Impact / Scope | Existing Ticket Match | Proposed Linear Action | Confidence | Human Decision |
| --- | --- | --- | --- | --- | --- | --- | --- |
| F1 | Concise symptom or bug claim | Measured counts, deltas, links, traces, logs, or "No measurements found" | Affected env, service, route, customer segment, or blast radius supported by evidence | Existing issue key/link, duplicate candidate, or "None found" | Create new ticket, add evidence comment, update status/labels, or no action | High/medium/low plus one short reason | Leave blank for the human to choose |
In the skill instructions, state that Codex must not create tickets, comment on
tickets, edit ticket fields, or add evidence until the human chooses one or more
row IDs and actions. If the human asks for an automated sweep, still pause at
this review table before writing to Linear.
### Protect Validation Integrity
You may use subagents during iteration to validate whether a skill works on realistic tasks or whether a suspected problem is real. This is most useful when you want an independent pass on the skill's behavior, outputs, or failure modes after a revision. Only do this when it is possible to start new subagents.
When using subagents for validation, treat that as an evaluation surface. The goal is to learn whether the skill generalizes, not whether another agent can reconstruct the answer from leaked context.
Prefer raw artifacts such as example prompts, outputs, diffs, logs, or traces. Give the minimum task-local context needed to perform the validation. Avoid passing the intended answer, suspected bug, intended fix, or your prior conclusions unless the validation explicitly requires them.
### Require Valid Markdown Output
When a skill instructs Codex to return Markdown, require the final output to be
valid Markdown, not just Markdown-like text.
In the skill instructions, explicitly require Codex to:
- use valid Markdown syntax for headings, lists, links, tables, and code fences;
- include a space after list markers such as `-`, `*`, and `1.`;
- close every Markdown link and parenthesis correctly;
- avoid malformed tables, dangling backticks, or partially opened fenced code
blocks;
- prefer plain paragraphs over complex formatting when the structure would be
fragile.
If a skill produces structured reports, include a short output-format section
that says the response must be valid Markdown and should be checked for basic
syntax mistakes before returning it.
### Anatomy of a Skill
Every skill consists of a required SKILL.md file and optional bundled resources:
```
skill-name/
├── SKILL.md (required)
│ ├── YAML frontmatter metadata (required)
│ │ ├── name: (required)
│ │ └── description: (required)
│ └── Markdown instructions (required)
├── agents/ (recommended)
│ └── openai.yaml - UI metadata for skill lists and chips
└── Bundled Resources (optional)
├── scripts/ - Executable code (Python/Bash/etc.)
├── references/ - Documentation intended to be loaded into context as needed
└── assets/ - Files used in output (templates, icons, fonts, etc.)
```
#### SKILL.md (required)
Every SKILL.md consists of:
- **Frontmatter** (YAML): Contains `name` and `description` fields. These are the only fields that Codex reads to determine when the skill gets used, thus it is very important to be clear and comprehensive in describing what the skill is, and when it should be used.
- **Body** (Markdown): Instructions and guidance for using the skill. Only loaded AFTER the skill triggers (if at all).
#### Agents metadata (recommended)
- UI-facing metadata for skill lists and chips
- Read references/openai_yaml.md before generating values and follow its descriptions and constraints
- Create: human-facing `display_name`, `short_description`, and `default_prompt` by reading the skill
- Generate deterministically by passing the values as `--interface key=value` to `scripts/generate_openai_yaml.py` or `scripts/init_skill.py`
- On updates: validate `agents/openai.yaml` still matches SKILL.md; regenerate if stale
- Only include other optional interface fields (icons, brand color) if explicitly provided
- See references/openai_yaml.md for field definitions and examples
#### Bundled Resources (optional)
##### Scripts (`scripts/`)
Executable code (Python/Bash/etc.) for tasks that require deterministic reliability or are repeatedly rewritten.
- **When to include**: When the same code is being rewritten repeatedly or deterministic reliability is needed
- **Example**: `scripts/rotate_pdf.py` for PDF rotation tasks
- **Benefits**: Token efficient, deterministic, may be executed without loading into context
- **Note**: Scripts may still need to be read by Codex for patching or environment-specific adjustments
##### References (`references/`)
Documentation and reference material intended to be loaded as needed into context to inform Codex's process and thinking.
- **When to include**: For documentation that Codex should reference while working
- **Examples**: `references/finance.md` for financial schemas, `references/mnda.md` for company NDA template, `references/policies.md` for company policies, `references/api_docs.md` for API specifications
- **Use cases**: Database schemas, API documentation, domain knowledge, company policies, detailed workflow guides
- **Benefits**: Keeps SKILL.md lean, loaded only when Codex determines it's needed
- **Best practice**: If files are large (>10k words), include grep search patterns in SKILL.md
- **Avoid duplication**: Information should live in either SKILL.md or references files, not both. Prefer references files for detailed information unless it's truly core to the skill—this keeps SKILL.md lean while making information discoverable without hogging the context window. Keep only essential procedural instructions and workflow guidance in SKILL.md; move detailed reference material, schemas, and examples to references files.
##### Assets (`assets/`)
Files not intended to be loaded into context, but rather used within the output Codex produces.
- **When to include**: When the skill needs files that will be used in the final output
- **Examples**: `assets/logo.png` for brand assets, `assets/slides.pptx` for PowerPoint templates, `assets/frontend-template/` for HTML/React boilerplate, `assets/font.ttf` for typography
- **Use cases**: Templates, images, icons, boilerplate code, fonts, sample documents that get copied or modified
- **Benefits**: Separates output resources from documentation, enables Codex to use files without loading them into context
#### What to Not Include in a Skill
A skill should only contain essential files that directly support its functionality. Do NOT create extraneous documentation or auxiliary files, including:
- README.md
- INSTALLATION_GUIDE.md
- QUICK_REFERENCE.md
- CHANGELOG.md
- etc.
The skill should only contain the information needed for an AI agent to do the job at hand. It should not contain auxiliary context about the process that went into creating it, setup and testing procedures, user-facing documentation, etc. Creating additional documentation files just adds clutter and confusion.
### Progressive Disclosure Design Principle
Skills use a three-level loading system to manage context efficiently:
2. **SKILL.md body** - When skill triggers (<5k words)
3. **Bundled resources** - As needed by Codex (Unlimited because scripts can be executed without reading into context window)
#### Progressive Disclosure Patterns
Keep SKILL.md body to the essentials and under 500 lines to minimize context bloat. Split content into separate files when approaching this limit. When splitting out content into other files, it is very important to reference them from SKILL.md and describe clearly when to read them, to ensure the reader of the skill knows they exist and when to use them.
**Key principle:** When a skill supports multiple variations, frameworks, or options, keep only the core workflow and selection guidance in SKILL.md. Move variant-specific details (patterns, examples, configuration) into separate reference files.
**Pattern 1: High-level guide with references**
```markdown
# PDF Processing
## Quick start
Extract text with pdfplumber:
[code example]
## Advanced features
- **Form filling**: See [FORMS.md](FORMS.md) for complete guide
- **API reference**: See [REFERENCE.md](REFERENCE.md) for all methods
- **Examples**: See [EXAMPLES.md](EXAMPLES.md) for common patterns
```
Codex loads FORMS.md, REFERENCE.md, or EXAMPLES.md only when needed.
**Pattern 2: Domain-specific organization**
For Skills with multiple domains, organize content by domain to avoid loading irrelevant context:
```
bigquery-skill/
├── SKILL.md (overview and navigation)
└── reference/
├── finance.md (revenue, billing metrics)
├── sales.md (opportunities, pipeline)
├── product.md (API usage, features)
└── marketing.md (campaigns, attribution)
```
When a user asks about sales metrics, Codex only reads sales.md.
Similarly, for skills supporting multiple frameworks or variants, organize by variant:
```
cloud-deploy/
├── SKILL.md (workflow + provider selection)
└── references/
├── aws.md (AWS deployment patterns)
├── gcp.md (GCP deployment patterns)
└── azure.md (Azure deployment patterns)
```
When the user chooses AWS, Codex only reads aws.md.
**Pattern 3: Conditional details**
Show basic content, link to advanced content:
```markdown
# DOCX Processing
## Creating documents
Use docx-js for new documents. See [DOCX-JS.md](DOCX-JS.md).
## Editing documents
For simple edits, modify the XML directly.
**For tracked changes**: See [REDLINING.md](REDLINING.md)
**For OOXML details**: See [OOXML.md](OOXML.md)
```
Codex reads REDLINING.md or OOXML.md only when the user needs those features.
**Important guidelines:**
- **Avoid deeply nested references** - Keep references one level deep from SKILL.md. All reference files should link directly from SKILL.md.
- **Structure longer reference files** - For files longer than 100 lines, include a table of contents at the top so Codex can see the full scope when previewing.
## Skill Creation Process
Skill creation involves these steps:
1. Understand the skill with concrete examples
2. Plan reusable skill contents (scripts, references, assets)
3. Initialize the skill (run init_skill.py)
4. Edit the skill (implement resources and write SKILL.md)
5. Validate the skill (run quick_validate.py)
6. Iterate based on real usage and forward-test complex skills.
Follow these steps in order, skipping only if there is a clear reason why they are not applicable.
### Skill Naming
- Use lowercase letters, digits, and hyphens only; normalize user-provided titles to hyphen-case (e.g., "Plan Mode" -> `plan-mode`).
- When generating names, generate a name under 64 characters (letters, digits, hyphens).
- Prefer short, verb-led phrases that describe the action.
- Namespace by tool when it improves clarity or triggering (e.g., `gh-address-comments`, `linear-address-issue`).
- Name the skill folder exactly after the skill name.
### Step 1: Understanding the Skill with Concrete Examples
Skip this step only when the skill's usage patterns are already clearly understood. It remains valuable even when working with an existing skill.
To create an effective skill, clearly understand concrete examples of how the skill will be used. This understanding can come from either direct user examples or generated examples that are validated with user feedback.
For example, when building an image-editor skill, relevant questions include:
- "What functionality should the image-editor skill support? Editing, rotating, anything else?"
- "Can you give some examples of how this skill would be used?"
- "I can imagine users asking for things like 'Remove the red-eye from this image' or 'Rotate this image'. Are there other ways you imagine this skill being used?"
- "What would a user say that should trigger this skill?"
- "Where should I create this skill? If you do not have a preference, I will place it in `$CODEX_HOME/skills` (or `~/.codex/skills` when `CODEX_HOME` is unset) so Codex can discover it automatically."
To avoid overwhelming users, avoid asking too many questions in a single message. Start with the most important questions and follow up as needed for better effectiveness.
Conclude this step when there is a clear sense of the functionality the skill should support.
### Step 2: Planning the Reusable Skill Contents
To turn concrete examples into an effective skill, analyze each example by:
1. Considering how to execute on the example from scratch
2. Identifying what scripts, references, and assets would be helpful when executing these workflows repeatedly
Example: When building a `pdf-editor` skill to handle queries like "Help me rotate this PDF," the analysis shows:
1. Rotating a PDF requires re-writing the same code each time
2. A `scripts/rotate_pdf.py` script would be helpful to store in the skill
Example: When designing a `frontend-webapp-builder` skill for queries like "Build me a todo app" or "Build me a dashboard to track my steps," the analysis shows:
1. Writing a frontend webapp requires the same boilerplate HTML/React each time
2. An `assets/hello-world/` template containing the boilerplate HTML/React project files would be helpful to store in the skill
Example: When building a `big-query` skill to handle queries like "How many users have logged in today?" the analysis shows:
1. Querying BigQuery requires re-discovering the table schemas and relationships each time
2. A `references/schema.md` file documenting the table schemas would be helpful to store in the skill
To establish the skill's contents, analyze each concrete example to create a list of the reusable resources to include: scripts, references, and assets.
### Step 3: Initializing the Skill
At this point, it is time to actually create the skill.
Skip this step only if the skill being developed already exists. In this case, continue to the next step.
Before running `init_skill.py`, ask where the user wants the skill created. If they do not specify a location, default to `$CODEX_HOME/skills`; when `CODEX_HOME` is unset, fall back to `~/.codex/skills` so the skill is auto-discovered.
When creating a new skill from scratch, always run the `init_skill.py` script. The script conveniently generates a new template skill directory that automatically includes everything a skill requires, making the skill creation process much more efficient and reliable.
- Creates the skill directory at the specified path
- Generates a SKILL.md template with proper frontmatter and TODO placeholders
- Creates `agents/openai.yaml` using agent-generated `display_name`, `short_description`, and `default_prompt` passed via `--interface key=value`
- Optionally creates resource directories based on `--resources`
- Optionally adds example files when `--examples` is set
After initialization, customize the SKILL.md and add resources as needed. If you used `--examples`, replace or delete placeholder files.
Generate `display_name`, `short_description`, and `default_prompt` by reading the skill, then pass them as `--interface key=value` to `init_skill.py` or regenerate with:
Only include other optional interface fields when the user explicitly provides them. For full field descriptions and examples, see references/openai_yaml.md.
### Step 4: Edit the Skill
When editing the (newly-generated or existing) skill, remember that the skill is being created for another instance of Codex to use. Include information that would be beneficial and non-obvious to Codex. Consider what procedural knowledge, domain-specific details, or reusable assets would help another Codex instance execute these tasks more effectively.
After substantial revisions, or if the skill is particularly tricky, you should use subagents to forward-test the skill on realistic tasks or artifacts. When doing so, pass the artifact under validation rather than your diagnosis of what is wrong, and keep the prompt generic enough that success depends on transferable reasoning rather than hidden ground truth.
#### Start with Reusable Skill Contents
To begin implementation, start with the reusable resources identified above: `scripts/`, `references/`, and `assets/` files. Note that this step may require user input. For example, when implementing a `brand-guidelines` skill, the user may need to provide brand assets or templates to store in `assets/`, or documentation to store in `references/`.
Added scripts must be tested by actually running them to ensure there are no bugs and that the output matches what is expected. If there are many similar scripts, only a representative sample needs to be tested to ensure confidence that they all work while balancing time to completion.
If you used `--examples`, delete any placeholder files that are not needed for the skill. Only create resource directories that are actually required.
#### Update SKILL.md
**Writing Guidelines:** Always use imperative/infinitive form.
##### Frontmatter
Write the YAML frontmatter with `name` and `description`:
- `name`: The skill name
- `description`: This is the primary triggering mechanism for your skill, and helps Codex understand when to use the skill.
- Include both what the Skill does and specific triggers/contexts for when to use it.
- Include all "when to use" information here - Not in the body. The body is only loaded after triggering, so "When to Use This Skill" sections in the body are not helpful to Codex.
- Example description for a `docx` skill: "Comprehensive document creation, editing, and analysis with support for tracked changes, comments, formatting preservation, and text extraction. Use when Codex needs to work with professional documents (.docx files) for: (1) Creating new documents, (2) Modifying or editing content, (3) Working with tracked changes, (4) Adding comments, or any other document tasks"
Do not include any other fields in YAML frontmatter.
##### Body
Write instructions for using the skill and its bundled resources.
If the skill expects Markdown output, add an explicit output-format section that
requires valid Markdown syntax in the final response.
### Step 5: Validate the Skill
Once development of the skill is complete, validate the skill folder to catch basic issues early:
```bash
scripts/quick_validate.py <path/to/skill-folder>
```
The validation script checks YAML frontmatter format, required fields, and naming rules. If validation fails, fix the reported issues and run the command again.
### Step 6: Iterate
After testing the skill, you may detect the skill is complex enough that it requires forward-testing; or users may request improvements.
User testing often this happens right after using the skill, with fresh context of how the skill performed.
**Forward-testing and iteration workflow:**
1. Use the skill on real tasks
2. Notice struggles or inefficiencies
3. Identify how SKILL.md or bundled resources should be updated
4. Implement changes and test again
5. Forward-test if it is reasonable and appropriate
## Forward-testing
To forward-test, launch subagents as a way to stress test the skill with minimal context.
Subagents should *not* know that they are being asked to test the skill. They should be treated as
an agent asked to perform a task by the user. Prompts to subagents should look like:
`Use $skill-x at /path/to/skill-x to solve problem y`
Not:
`Review the skill at /path/to/skill-x; pretend a user asks you to...`
Decision rule for forward-testing:
- Err on the side of forward-testing
- Ask for approval if you think there's a risk that forward-testing would:
* take a long time,
* require additional approvals from the user, or
* modify live production systems
In these cases, show the user your proposed prompt and request (1) a yes/no decision, and
(2) any suggested modifictions.
Considerations when forward-testing:
- use fresh threads for independent passes
- pass the skill, and a request in a similar way the user would.
- pass raw artifacts, not your conclusions
- avoid showing expected answers or intended fixes
- rebuild context from source artifacts after each iteration
- review the subagent's output and reasoning and emitted artifacts
- avoid leaving artifacts the agent can find on disk between iterations;
clean up subagents' artifacts to avoid additional contamination.
If forward-testing only succeeds when subagents see leaked context, tighten the skill or the
# openai.yaml fields (full example + descriptions)
`agents/openai.yaml` is an extended, product-specific config intended for the machine/harness to read, not the agent. Other product-specific config can also live in the `agents/` folder.
default_prompt: "Optional surrounding prompt to use the skill with"
dependencies:
tools:
- type: "mcp"
value: "github"
description: "GitHub MCP server"
transport: "streamable_http"
url: "https://api.githubcopilot.com/mcp/"
policy:
allow_implicit_invocation: true
```
## Field descriptions and constraints
Top-level constraints:
- Quote all string values.
- Keep keys unquoted.
- For `interface.default_prompt`: generate a helpful, short (typically 1 sentence) example starting prompt based on the skill. It must explicitly mention the skill as `$skill-name` (e.g., "Use $skill-name-here to draft a concise weekly status update.").
- `interface.display_name`: Human-facing title shown in UI skill lists and chips.
- `interface.short_description`: Human-facing short UI blurb (25–64 chars) for quick scanning.
- `interface.icon_small`: Path to a small icon asset (relative to skill dir). Default to `./assets/` and place icons in the skill's `assets/` folder.
- `interface.icon_large`: Path to a larger logo asset (relative to skill dir). Default to `./assets/` and place icons in the skill's `assets/` folder.
- `interface.brand_color`: Hex color used for UI accents (e.g., badges).
- `interface.default_prompt`: Default prompt snippet inserted when invoking the skill.
- `dependencies.tools[].type`: Dependency category. Only `mcp` is supported for now.
- `dependencies.tools[].value`: Identifier of the tool or dependency.
- `dependencies.tools[].description`: Human-readable explanation of the dependency.
- `dependencies.tools[].transport`: Connection type when `type` is `mcp`.
- `dependencies.tools[].url`: MCP server URL when `type` is `mcp`.
- `policy.allow_implicit_invocation`: When false, the skill is not injected into
the model context by default, but can still be invoked explicitly via `$skill`.
description: Use when writing or reviewing Storybook stories (`.stories.tsx`) for React components.
---
# Storybook Component Stories
## Which Components Can Have Stories?
Only create stories for components that **do not** do any of the following:
- Depend on context.
- Fetch data via any private API, including Langfuse’s own tRPC API.
Stories should follow the `ComponentName.stories.tsx` filename pattern. Components covered by stories should have exactly one exported or public component.
If this is not the case, suggest splitting up the file that includes the component to be covered first.
Be mindful of the breadth a story covers. A story should show a component in isolation. Page-level compositions should be rare and intentional.
## What to Do If a Component Violates the Criteria
Suggest abstracting a presentational component that does not violate the criteria and receives relevant data via props.
Make sure the props are well-defined using TypeScript.
Keep the existing component, but update it to use the newly created component for rendering. These presentational components are easier to test and easier to reuse.
## How Stories Should Be Written
- Use "CSF Next" format by default.
- Cover only the relevant component by default.
- Avoid custom render functions by default.
- Use `satisfies` and typed Storybook metadata so invalid args, decorators, and play functions are type-checked.
- Use play functions to test user-relevant interactions after render, not to compensate for complex setup or hidden dependencies.
- Name stories after the state they represent, not the implementation. Also do not include the component name in the story name.
- Set callbacks up as Storybook Actions by default:
```ts
import { fn } from "storybook/test";
```
- Avoid large fixtures.
- Use the smallest meaningful data shape needed to render the state.
- If fixtures are required and may be shared, check whether a reusable helper function exists. Otherwise, create one for defining the fixture.
## Variant and Design Showcase Stories
If a component has many variants, and the point of the story is to showcase the design of a component rather than its functionality, stories may render the component multiple times.
For example, a `Button` with a `size: "sm" | "md" | "lg"` prop may have a story that shows three buttons side by side.
If the button also has a `variant: "primary" | "secondary"` prop, consider using a matrix-like UI that showcases all possible combinations.
These compositional stories **should not** contain Storybook play functions. They should also not allow the Storybook user to customize the predefined args, such as `size` and `variant`, via Storybook args. Having an arg for non-bound props, such as `text`, may be acceptable.
## Additional Information
- We do not use MSW and are not planning to add it.
short_description:"Summarize bugs, pages, and incidents"
default_prompt:"Use $weekly-production-review to prepare an event-centric weekly production review from Linear bugs, Datadog alert/page signals, and status-page or incident data."
# your codebase is analyzed, see https://docs.github.com/en/code-security/code-scanning/creating-an-advanced-setup-for-code-scanning/codeql-code-scanning-for-compiled-languages
# Fern token is generation-only; GitHub write actions are handled by GH_ACCESS_TOKEN in a protected environment.
- FERN_TOKEN
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
