* Checkpoint before follow-up message
Co-authored-by: max <max@langfuse.com>
* fix: apply prettier formatting to batchExport test
- Format array to single line per prettier style guide
* fix(test): remove search query that was filtering out all results
The searchQuery "###" was causing the test to fail because no traces
matched this pattern. Removed it to test the filter behavior correctly.
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
* chore: remove peek view from compare data table
* style(io-table-cell): support content expand on hover
* feat: make compare cell digestable
* feat: open trace peek view from compare view
* style: adjust cell for many scores, IO and metadata
* style: allow untoggling output
* chore: fix
* fix: enable score column query based on runIds length
* feat: add ScoreWriteCache context for optimistic UI updates on score edits
* feat: add ActiveCellContext for managing active dataset run item cell state
* feat: enhance SidePanel component with controlled/uncontrolled modes
* feat: add annotation functionality to DatasetAggregateCell component
* feat: integrate annotation panel and context providers in DatasetCompare component
* chore: add access control for annotation button in DatasetAggregateCell component
* fixup: add annotation panel as sidepanel child
* chore: extract useConfigSelection
* chore: refactor AnnotateDrawerContent to use new useAnnotationFormHandlers for score management
* refactor: simplify score mutation handlers by consolidating query invalidation logic
* chore: refactor to simplify AnnotateDrawer props
* chore: extract useEmptyConfigs hook
* chore: transform score aggregates to annotation scores
* feat: enhance annotation score handling with new caching and filtering mechanisms
* test: transform and filter scores
* chore: delete
* style: paddings and margins
* fix: ensure cached updates apply to cached creates
* chore: do not allow closing annotation panel if has draft comment
* chore: refactor mergeScoreAggregateWithCache for readability
* fix: update dependency in useEffect to use router.query for accurate data handling
* refactor: optimize DatasetAggregateCell for performance using MemoizedIOTableCell and useMemo
* fix: test
* chore: fix build
* chore: move files
* chore: fix pending creates logic
* refactor: enhance mergeScoreAggregateWithCache to prioritize cached creates and streamline aggregate handling
* fix: add comment field to annotation form handlers
* refactor: update AnnotationPanel and related components to utilize filterSingleValueAggregates for improved score handling and disabled config management
* refactor: optimize score field rendering in AnnotateDrawerContent by sorting scores alphabetically
* refactor: merge score columns in DatasetAggregateCell
* test: fix
* style: unify compare cell output heights
* refactor(scores): replace useEmptyConfigs with scoreMetadata in annotation components
- Removed useEmptyConfigs hook and its related state from multiple components including SessionPage, ObservationPreview, and TracePreview.
- Introduced scoreMetadata prop to pass projectId and environment directly to annotation components.
- Updated AnnotateDrawer and AnnotationForm to accommodate the new scoreMetadata structure.
- Refactored related components to streamline score handling and improve performance by eliminating unnecessary state management.
* refactor(scores): update score schemas and cache handling
- Made `id` and `configId` fields mandatory in `CreateAnnotationScoreBase` and `AnnotationScoreDataSchema` for consistency.
- Removed `useEmptyConfigs` imports from components to streamline code.
- Enhanced `ScoreCacheContext` to manage cache operations more effectively, including adding, updating, and deleting scores.
- Updated hooks and components to utilize the new cache methods, improving performance and reducing unnecessary state management.
- Added TODO comments for future schema reviews and adjustments.
* chore: remove unused type
* refactor(scores): enhance score handling and component structure
- Made `id` optional in `CreateAnnotationScoreBase` for backward compatibility.
- Updated `Trace` component to utilize `useMergedScores` for improved score management.
- Refactored `SpanItem` to use loose equality check for `observationId`.
- Changed `AnnotationDrawerSection` to accept `configSelection` prop for better config handling.
- Simplified `AnnotationPanel` by removing unnecessary API calls and directly using active cell data.
- Renamed `useEmptyConfigs` to `useEmptyScoreConfigs` for clarity and updated related components.
- Introduced `useAnnotationScoreConfigs` hook to manage score config selection logic.
- Removed unused `useScoreCustomOptimistic` and `useScoreValues` hooks to clean up the codebase.
* chore: types
* chore: types
* chore: types
* chore: types
* refactor(scores): integrate score column merging and caching
* fix(scores): add error handling to score mutations
- Introduced an `onError` callback to reload the page upon mutation errors, ensuring cache invalidation and fresh data retrieval.
- Updated import for `ScoreTarget` type from `@langfuse/shared` for consistency.
* chore: types
* chore: change color of highlight cell
* style: margins
* fixup: remove alphabetic sorting
* fix: sorting
* chore: drop context provider from traces component
* chore: updated `useMergedScores` and related functions to accept a `mode` parameter for better control over score merging.
- Adjusted components to utilize the new `displayScores` for improved performance and user experience.
* chore: resolve cached score from score domain
* chore: lint
* chore: fix build
* fix: null vs undefined comparison
* Docs: Add link to Langfuse metrics API documentation
Co-authored-by: marc <marc@langfuse.com>
* fern generate
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
* put new api key button to the top
* allow adding note at creation time
* fix
---------
Co-authored-by: Nimar <l.nimar.b@gmail.com>
Co-authored-by: Max Deichmann <m.deichmann@tum.de>
* add banner for unpaid or overdue subscriptions
* remove logging statement
* update cloudConfig to be more flexible and robust
* add pseudo-code for enabling only for admins
* fix comment drawer layout
* add explanatory comment
* extract new tailwind classes
* only show payment banner to admins and owners
* chore: add batch-based observation to event propagation
* chore: adjust timestamp and partition handling
* chore: update metadata handling and disable by default
* chore: align comment with reality
* chore: do not set default value for ingestionservice
* chore: handle staging table write more precisely using environments
* chore: query memory optimizations
* chore: spell check
* chore: set global concurrency limit to 1 and increase timeout
* chore: run event propagation in a new trace
* chore: set request timeout
* chore: pass correct timeout
* chore: backlog handling for event backfill
* feat: introducing events repository with observations-compatible iface.
* feat: wiring the new events repository to the observations table
* feat: add getObservationByIdFromEventsTable to events repository
* chore: run ch:dev-tables in CI to allow tests for experimental events table pass
* fix(dataset-run-item): enhance metadata conversion to handle various data types
* fix(experiment-service): remove unused input and expectedOutput fields from processItem function
I am leaving measureAndReturn in place because it emits a lot of useful
tracing metadata and it remains a good place to conduct experiments in
the future.
* feat(onboarding): add DatasetItemsOnboarding component for dataset item management
- Introduced a new onboarding component to facilitate the addition of items to datasets, including options for CSV uploads and manual entry.
- Updated SplashScreen to accept children for enhanced flexibility.
- Modified DatasetItemsTable and related components to conditionally render onboarding based on dataset item count.
* chore: references
* Checkpoint before follow-up message
Co-authored-by: michael <michael@langfuse.com>
* fix(spend-alerts): format code and fix toast imports
* feat(spend-alerts): add test script for spend alert emails
- Follows same pattern as send-test-threshold-emails.ts
- Includes 3 test scenarios with different thresholds
- Requires manual email configuration for safety
* feat(spend-alerts): add Prisma migration for CloudSpendAlert table
- Creates cloud_spend_alerts table with proper schema
- Adds foreign key constraint to organizations table
- Includes index on org_id for performance
- Supports decimal thresholds and trigger tracking
* add concurrently keyword to index
* fix linter error
* fix linter error
* remove old usage alerts
* update syling
* refatcor alerts jobs
* fix build errors
* update rate limit for stripe
* fix migration
* update email template and tracking
* update text
* add review comments
* add queue body deifniton
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: michael <michael@langfuse.com>
* shuffle toolbar
* controls sidebar
* setup for filter attributes
* test with environment filter
* new filter state management
* name filter
* add reset filter button
* add tags filter
* add bookmarked filter
* unify column id v display handling
* rename to bookmarked
* polish, fix bugs
* merge hooks, efficiency
* support dual-value in slider
* latency filter
* make filters generic
* add sidebar to observations, sessions, prompts, scores and evals
* simplify table-controls component
* allow resetting individual filters
* clean up layout
* show "all" when only selected item
* text facets
* add key-value filter
* add numerical key-value filter
* add metadata filter
* disable accordion animations
* add filtering for values
* fix observation filters not being applied
* add ai filters
* represent no range filter with empty inputs
* update look of reset button
* fix header shrinking
* tidy up vertical spacing
* clean up
* fix type and lint issues
* fix none-of operator not being used when it should
* fix missing env filter
* a few last fixes
* another type error
* oops
* Fix: Prevent duplicate job scheduling in queues
Co-authored-by: michael <michael@langfuse.com>
* Fix: Deduplicate queue jobs across multiple worker instances
Co-authored-by: michael <michael@langfuse.com>
* Checkpoint before follow-up message
Co-authored-by: michael <michael@langfuse.com>
* Checkpoint before follow-up message
Co-authored-by: michael <michael@langfuse.com>
* fix: prevent duplicate queue job scheduling across multiple containers
- Add unique jobIds to CloudFreeTierUsageThresholdQueue for deduplication
- Add comprehensive logging for job scheduling and execution tracking
- Apply best practices pattern with descriptive job data and comments
- Remove investigation documentation files
This resolves the issue where multiple worker containers were creating
duplicate recurring and bootstrap jobs, causing 10x more executions
than expected.
* add logging statements
* remove job id and bootstrap execution
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: michael <michael@langfuse.com>
* fix(billing): implement chunked updates for free tier usage tracking
Reduces DB load by 95% via transaction batching (50,000 → 50 chunks).
Each chunk processes 1,000 orgs with proper error handling.
Failed chunks reported to Datadog without killing the job.
Changes:
- Refactored processThresholds() to return update data instead of executing immediately
- Created bulkUpdates.ts with chunked transaction processing (1000 orgs per batch)
- Modified usageAggregation.ts to collect updates and execute in bulk
- Updated tests to verify returned data instead of mock calls
- Added error handling with traceException for failed chunks
- Structured for easy swap to raw SQL (Option 1) if needed
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* test: update cache invalidation tests to use bulkUpdateOrganizations
The tests now call bulkUpdateOrganizations() to complete the update flow,
including cache invalidation. This reflects the refactored architecture where
processThresholds() returns update data and bulkUpdateOrganizations() executes it.
* fix: reduce transaction timeout from 60s to 15s per chunk
60 seconds was excessive for 1000 orgs. Even at 10ms per update,
that's only 10 seconds. 15 seconds provides a reasonable buffer.
* refactor: use Promise.allSettled instead of transaction wrapper
Benefits over previous () approach:
- Better resilience: One failed org doesn't fail the entire 1000-org chunk
- Concurrent execution: Much faster than sequential transaction
- Granular error tracking: Track exactly which orgs failed
- Better error handling: Each org failure reported to Datadog individually
Trade-off: No atomicity per chunk, but we don't need it for this use case.
Each org update is independent and idempotent.
* fix: remove unused chunkOrgIds variable
* remove unused code
* refactor transaction update and add rawsql update
* Update worker/src/ee/usageThresholds/bulkUpdates.ts
Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
* Update worker/src/ee/usageThresholds/bulkUpdates.ts
Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
* make rawsql query default
---------
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
* fix(billing): enhance free tier usage emails with pricing, reset dates, and CRM integration
- Add getBillingCycleEnd() helper to calculate when usage limits reset
- Add comprehensive tests for billing cycle end date calculations
- Add optional USAGE_THRESHOLD_EMAIL_BCC env variable for CRM integration (e.g., HubSpot)
- Include reset date in both warning and suspension emails
- Add Core plan pricing ($29/month) and key benefits to email templates
- Mention startup program (50% off for first year) with link to langfuse.com/startups
- Update email templates to include:
- When usage limit resets
- Pricing information from stripeCatalogue
- Key upgrade benefits: unlimited users, 90-day retention, email/chat support
- Startup program callout
- Update test script to include reset date and BCC configuration
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* refactor(billing): rename USAGE_THRESHOLD_EMAIL_BCC to CLOUD_CRM_EMAIL
Rename environment variable to better reflect its purpose as a general
cloud CRM integration endpoint rather than being specific to email BCC.
Changes:
- Renamed env variable in both .env.dev.example and .env.prod.example
- Updated email sending functions to use new variable name
- Updated test script with new variable name
- Regenerated TypeScript declarations
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* security(billing): add email validation for CLOUD_CRM_EMAIL
Add Zod email validation before using CLOUD_CRM_EMAIL in BCC field
to prevent potential email header injection attacks.
Changes:
- Import and use zod/v4 for email validation in both email functions
- Validate CLOUD_CRM_EMAIL format before assigning to BCC
- Log warning if invalid email format is detected
- Add CLOUD_CRM_EMAIL to worker env schema
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
When LANGFUSE_FREE_TIER_USAGE_THRESHOLD_ENFORCEMENT_ENABLED is false,
the paid plan check was never reached due to early return. This caused
ALL organizations (paid and free) to be incorrectly counted as free_tier_orgs.
Fix: Move paid plan check before enforcement check to ensure paid orgs
always return "PAID_PLAN" regardless of enforcement status.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude <noreply@anthropic.com>
* fix: show input/output/total columns in traces table correctly
* fix: show input/output/total columns in traces table correctly
* fix: show input/output/total columns in traces table correctly
* Add current datetime to AI prompt context
Co-authored-by: marc <marc@langfuse.com>
* Refactor datetime formatting for AI prompt
Co-authored-by: marc <marc@langfuse.com>
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
* fixup(datasets-compare): support front-end column filtering
- Added column filtering capabilities to the DatasetCompareRunsTable and DatasetRunAggregateColumnHelpers.
- Introduced useColumnFilterState hook for managing filter state with URL persistence.
- Updated DatasetRunItemsByRunTable to accept multiple dataset run IDs.
- Enhanced PopoverFilterBuilder to support different button variants for filter actions.
- Refactored related components to integrate new filtering features.
* chore: remove fetching of run items data in compare view
* chore: move filter logic into useDatasetRunAggregateColumns
* chore: extend useColumnFilterState
* chore(scores): add ScoreDataType and ScoreDataTypeDomain
* chore: move state out of useDatasetRunAggregateColumns
* chore(dataset-run-items): enhance DatasetRunItem types and converters for optional IO support
* chore: refactor compare data fetching to support one API route
* feat: support filtering on compare view
* feat: add FilteredRunPills component for enhanced filtering display in compare view
* chore: eslint
* chore: simplify and refactor
* chore: push
* fix import
* chore: remove console log
* chore: drop first approach of pulling dataset item ids from postgres
* chore: rename trpc route
* tests: adjust non filter tests
* chore: rename dataset repository helper functions
* chore: add intersection query
* chore: push
* chore: debounce updateRunFilters
* chore: add filter synchronization for run IDs in DatasetCompareRunsTable
* feat: better relative date range selects
* fix: date picker styles
fixes LFE-6702
* add TimeRangePicker
* use TimeRangePicker in traces view
* update dashboards and tables to use TimeRangePicker
* missed fixes due to updated time range utils
* simplify
* update remaining
* remove old timestamp filter
* address comment
* silence codespell false-positive
* fix badge heights
ClickHouse has a quirk when it comes to handling exceptions mid response.
It will simply output a row with "exception" key inside, which is indistinguishable from
a query like `SELECT "my lovely string" AS exception;` may return.
This PR makes the best effort to convert such rows into errors and throws them.
See:
- https://github.com/ClickHouse/clickhouse-js/issues/332
- https://github.com/ClickHouse/ClickHouse/issues/75175
Ideally this should get fixed in the future versions of ClickHouse.
* fix(otel-ingest): don't overwrite trace metadata from new observation
* update test
* skip test, not good
* skip test because it's not getting the entire trace
* Refactor: Add last used auth method persistence
Co-authored-by: leo <leo@langfuse.com>
* Refactor: Use useLocalStorage hook for auth method persistence
Co-authored-by: leo <leo@langfuse.com>
* Refactor SSOButtons to use parent-managed last used method
Co-authored-by: leo <leo@langfuse.com>
* Refactor: Conditionally show "Last used" badge on sign-in
Co-authored-by: leo <leo@langfuse.com>
* refactor styles
* pass lint
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
* Michael/lfe 6645 (#9046)
* refactor billing settings and cancel button
* refactor billing components
* chore(stripe): bump stripe from 17.4.0 to 18.5.0.basil (#9050)
* bump stripe api version to 18.5.0.basil
* update stripeAPIHandler to 18.5.0
* update cloud billing router and add log statements
* Show stripe buttons only with active subscription
* remove logging statements
* feat(billing): show plan end in billing settings (#9052)
* update cloudConfigSchema with metadata field for scheduled cancellations
* add function to unpack cancellation metadata in subscription.update webhook
* store cancellation metadata in db
* add and aggegregation option to LocalIsoDate component
* add BillingCurrentPlanLabel component to indicate cancelled plans
* add title prop to StripeCustomerPortalButton for better accessibility
* remove text from label
* fix(billing): cancel pending cancellations on plan switches (#9076)
cancel pending cancellations on plan switches
* feat(billing): distinguish between new subscriptions and legacy subscriptions (#9078)
* add orderKeys to distinguish between upgrades and downgrades of subcriptions
* add optional activeUsageProductId to cloudConfig schema
* add legacy Plan flag on cloudConfig.stripe schema
* set usageProductId in webhook callback
* feat(billing): Add new billing logic and update stripe integration (#9096)
* update stripe usage product id to match sandbox
* update to cloudBillinRouter.createCheckoutSession to deal with legacy and new prices
* add orderKeys and replace them with monthly price to distinguish upgrade and downgrade paths
* update cloudBillingRouter.changePlan mutation to deal with different upgrade paths
* update stripeWebHookApiHandler to listen for subscriptionSchedule events
* update cloudConfig schema to store subscription schedule metadata
* extract formatLocalIsoDate into own function
* add useBillingInformation hook for common ui-billing operations
* Add BillingScheduleNotification component to indicate to user when their subscription is cancelled or scheduled to change
* refactor billing component to use new hook
* add managed cancellation button
* fix planSwitch info overrides on schedule.update callbacks
* remove logging statements
* add buttons to isolate strip cancelation, plan continuation, and switching
* refacor billingSwitchPlanDialog
* fix linter errors
* fix bug where released schedules where released
* fix Billing Portal ui issue
* add support for legacy plans to enable testing on staging
* clear cancellationInfo and planSwitchScheduleInfo on customer.subscription.deleted webhook
* extract condition into variable
* fix typo
* Add invoice table
* fix unstable ref causing rerenders
* fix uneccessary cast
* feat(billing): invoice table in billing settings (#9134)
* Add invoice table
* fix unstable ref causing rerenders
* fix uneccessary cast
* update webhooks to allow externally triggered subscription schedules
* nit
* feat(billing): Rewrite Billing Service to Reduce Complexity and Minimize Data Drift Risks (#9169)
* refactor code for more effective refetches
* Refactor cloudBillingRouter into BillingService
* Add Idempotency keys for unique stripe ops, add logging and auditLogs, Refactor for cleaner dx
* Add env checks to stripe webhook handler
* Add env checks to stripe webhook handler
* add review comments
* standardize logger statements
* Add new readme
* update plan switch/cance/reactivate explanation messages
* Add fallbacks in org resolution to webhook, to deal with susbcriptions created from the dashboard
* fix linter errors
* remove comment
* fix build errors
* implement review comments
* add discount column to invoice table
* update orderkey of core plan to reflect new price
* fix(billing): apply existing discounts to new subscription phase when updating via subscription schedule (#9199)
* safeguard against illegal invoice.createPreview call and apply discounts on subscription schedule change
* fix typo
* add docstrings to stripeIdempotencyKey.ts
* update productId for prod
---------
Co-authored-by: Marc Klingen <git@marcklingen.com>
* fix: fetch environment options from raw data
* chore: drop stuff
* chore: patch
* chore: always add default
* chore: adjust mapping logic for correctness
* chore: drop materialized views to fill project_environments
* chore: remove migration files
* perf: drop final on check trace exists
* chore: only apply final on eval check if non-id filter is used
* chore: skip full query if not needed
* chore: revert query changes
* chore: simplify condition
* chore: add metrics
* feat: add api endpoint to configure blob storage integration
* chore: add test suite and generated API docs
* chore: add implementation
* chore: simplify delete endpoint
* chore: PUT route cleanups
* feat(dataset-run-items-ui): support score filters in UI table
* chore: add dataset run item scores to seeder
* chore: make datasetid optional
* chore: drop dataset item id from scores CTE
* feat: Add HIPAA region and improve region selection
Co-authored-by: marc <marc@langfuse.com>
* Refactor: Remove unused state and simplify region logic
Co-authored-by: marc <marc@langfuse.com>
* Fix: Update BAA link to HIPAA security page
Co-authored-by: marc <marc@langfuse.com>
* Refactor AuthCloudRegionSwitch component for clarity
Co-authored-by: marc <marc@langfuse.com>
* delete weird file and prettier
* replace contact support on sign in with mailto link
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
* Refactor: Use signIn with redirect: false for OTP verification
Co-authored-by: marc <marc@langfuse.com>
* nit
* revert button changes, not the focus of this pr
* push
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
* fixup(peek): isolate data fetching for peek view to depend on trpc and be independent of table state
* fix: rely only on url for peek state management
* fixup: drop complex peek logic
* refactor: replace createPeekHandler with usePeekNavigation for improved URL management in table components
* chore: all peek logic fixed except compare view and expand
* chore: refactor navigation logic
* chore: docs
* chore: clear url params also in row click navigation
* chore(table): rename isPinned to isFixedPosition
* chore(table): do not allow re-ordering fixed position columns
* refactor(table): update column visibility and order keys for dataset runs
* feat(dataset-runs): fix first two columns on y-/x-scroll
* chore: fixup
* chore: add comment
* Add support form and drawer scaffolding
* style changes
* fix flickering
* style support drawer into section
* update form
* refactor to use useForm hook
* add initial trpc router for creating chats
* add plainRouter and remove legacy chat
* refactor plain trpc route to fire and forget non-critical updates
* update success section
* add mobile support view
* make interaction type conditional
* add community section
* replace support-menu-dropdown with supportmenu-button
* add entitlement in-app-support to all cloud plans
* make plain call synchronous
* add file upload
* fix lint error
* remove unused code
* add first implementation of threadEvents
* remove auto-open on project create
* remove auto-open on tracing api generation
* remove comments
* remove old trpc route
* remove unused function
* simplify function return
* reduce input surface of createThread procedure
* simplified layout
* make drawer props more explicit
* remove duplicate threadFields
* remove misleading button
* refactor trpc router and plain client
* add support for uiCustomization
* remove plain chat from csp
* strict equality
* replace radio buttons
* perf: move tokenization in worker to worker thread
* chore: linting
* chore: test timeouts
* duplicate worker file
* chore: linting
* chore: make async tokenization rate configurable
* chore: add note in worker-thread files on keeping them same
* chore: make pool size dynamic and add error fallback
* chore: pass text as is to tokenization
* chore: patch tracing behaviour
* chore: bump to 4 workers
* chore: linting
* chore: update span prop names
* chore: start with 2 workers
* feat(table-ui): filter score columns to show score only if has value for given table
* chore: refactor analytics
* chore: include dataset id in filter condition for DRI scores
* chore: fix tests
* chore: await data
* fixup-remove: upgrade to nextjs v15 (#8795)
* chore: simplify query
* chore: remove getRunScoresGroupedByNameSourceType
* chore: push
* fix: stabilize hook dependency
* Revert "fixup-remove: upgrade to nextjs v15 (#8795)"
This reverts commit 50b742325a82057f5dbfbe3f548d0610d3343f13.
* chore: eslint
---------
Co-authored-by: Nimar <l.nimar.b@gmail.com>
* chore: validate IP adresses
* chore: validate IP adresses
* chore: validate IP adresses
* chore: validate IP adresses
* chore: validate IP adresses
* chore: validate IP adresses
* chore: validate IP adresses
* chore: validate IP adresses
* chore: validate IP adresses
* chore: validate IP adresses
* chore: validate IP adresses
* chore: validate IP adresses
* chore: validate IP adresses
* chore: validate IP adresses
* docs: add spec
* feat: add filter and date range saving to dashboard
* feat: take out url params hook for date ranges
* docs: remove specs
* move migration
* move migration
* fixes
* fixes
---------
Co-authored-by: Max Deichmann <m.deichmann@tum.de>
* feat(dataset-items-ui): allow filtering by metadata keys
* chore: push
* chore: push
* chore: fix after rebase
* feat(datasets): full text search for dataset items
* chore: remove checking for trace and observation existence before returning dataset items
* chore: eslint
* chore: pass search parameters
* perf: use aggregations to calculate traces all API result
* chore: adjust AMT settings for redis-cluster tests
* chore: patch test cases
* chore: handle prefix and filter differentl
* chore: move trace column def to shared server to use env imports
* fix: use dynamic asset path for Ragas logo with custom basePath
- Fixes hardcoded /assets/ragas-logo.png path that breaks with NEXT_PUBLIC_BASE_PATH
- Uses router.basePath to construct correct asset URL dynamically
- Resolves broken Ragas evaluator icons for custom base path deployments
- Backward compatible with root path deployments
Fixes#8611
* fix: prettier formatting for ragas-logo.tsx
- Add missing newline at end of file
- Ensure consistent formatting
* fix: apply prettier formatting to ragas-logo.tsx
* fix: use env.NEXT_PUBLIC_BASE_PATH directly for Ragas logo
Addresses reviewer feedback to use environment variable directly
instead of useRouter().basePath, matching pattern used throughout
the codebase (LangfuseLogo, auth redirects, API endpoints, etc.)
* chore: upgrade to trpc v11, react-query v5
* equalize ts version
* onSuccess -> useEffect
* isPending
* restore some
* fix
* fixx
* fix again
* fix
* ugly working state
* make the TS concise types a bit more beautifyul
* fix one more
* more loading
* mock
* moar pending
* moar
* use queryclient instead
* upgrade superjson
* pot fix
* more specific types
* fix build
* clean up
* undo
* cleanup
* cleanup
* wip small scores
* fix: root node missing selected state
* refactor: only use command for search results
* chore: more consistent styling between search and tree
* fix: rerendering issue
* fix: more styling issues
* fix: lint warnings and eslint vscode plugin config
* more cleanup
* fix: selecting root node from search
* turn timeline toggle into dropdown
* improve collapse / expand all
* fix missing title on view options
* fix observation name overflow
* allow resizing tree view
* better tree indicators and smaller font size
* integrate new feedback
* add some spacing to scores
* fix timeline toggle label
* fix timeline view dynamic sizing
* resizable improvements
* last tree connector fixes
* switching views shouldn't push to history
* even more density
* polish
* fix type errors
* update dataset compare detail view
* fix missing vertical padding in tree node without metrics
* rebase cleanup
* fix: implement a recursive version of getChannels
SlackService.getChannels only fetched 200 channels, which after
filtering might get even less. This doesn't work for large
organizations, as there are usually more than 200 active channels,
with possibly more archived (which counted into the limit)
Signed-off-by: Łukasz Jernaś <lukasz.jernas@allegro.com>
* chore: Add limit on how many records we can download from Slack API
Signed-off-by: Łukasz Jernaś <lukasz.jernas@allegro.com>
* fix: Actually respect the fetch limit
Signed-off-by: Łukasz Jernaś <lukasz.jernas@allegro.com>
* fix: Implement review suggestions
Signed-off-by: Łukasz Jernaś <lukasz.jernas@allegro.com>
---------
Signed-off-by: Łukasz Jernaś <lukasz.jernas@allegro.com>
Co-authored-by: Steffen Schmitz <steffen@langfuse.com>
* feat(traces): add more observation types
* observations table default filter for all types
* fix test running instructions
* add seeder
* add basic tests
* add to evaluator object selector
* add test to verify we default to span-create for all types
* allow generation like attributes on all types
* check if something is generation LIKE not strictly a GENERATION
* fix test
* ensure that GENERATION-like type's fields can also be null
* simplify create- logic
* chore(dataset-run-items): return CH result
* chore: sunset dual-write phase for dri in CH execution path; do not fall back on PG for reads
* chore: eslint
* chore: limit traces to trace AMT migration to only write into traces_all_amt
* chore: revert validation changes
* chore: simplify query
* chore: tune both queries
* chore: avoid full aggregation during migration
* chore: handle IO coalescing to skip aggregations fully
* chore: remove obsolete query parts
* fix(evals): ensure default model supports langfuse evals at set up
* fix: await upsertDefaultModel
* chore: provide actionable error messages in case of misconfiguration
- LLM connections: surface required model selection
for Azure/Bedrock; simplify advanced settings and validation
- Move custom model names to main form for Azure and Bedrock; require at least one model
- Remove default models toggle for Azure/Bedrock (they don’t support defaults)
- Keep Azure base URL and extra headers in main form; remove Azure advanced panel entirely
- Show advanced settings only for OpenAI, Anthropic, Vertex AI, and Google AI Studio
- Improve validation order to avoid confusing errors when defaults aren’t supported
- Add adapter-based placeholder for provider name; clarify copy
- Consolidate adapter logic to a single helper used in UI and schema
* chore: skip S3 list and legacy trace insert based on env flag
* chore: adjust skip s3 list conditions
* chore; refactor traceUpserQueue entity assoication
* chore: reverse
* chore(annotation-queue): add AnnotationQueueMembership model and related database schema updates
* chore(annotation-queue): add AnnotationQueueMembership POST and DEL APIs
* feat(annotation-queue): implement user assignment functionality for annotation queues
* fix: role logic for users
* chore: rn to assignments
* fixup: rn to assignments
* chore(annotation-queue): enhance user assignment section with debounced search and display of assigned users
* feat(annotation-queue): add MultiSelectCombobox for user assignment and enhance DataTable with row class name functionality
* refactor(rbac): replace generateUserQuery with generateUserProjectRolesQuery and introduce utility functions for project role resolution
* fixup: filter query for user ids
* fixup: resolve project role test
* fix: validate user project role before adding to annotation assignment
* chore: remove code
* chore: fix
* chore: fix filter
* chore: fix filter
* tests for auth logic
* push: fix api route test
* tests for auth logic
* fix
* fix
* chore: push
* chore: rename
* docs: api types
* fix: formatting
* fix: formatting
* chore: add disabled prop to MultiSelectCombobox
* chore: rename migration
---------
Co-authored-by: Max Deichmann <m.deichmann@tum.de>
* feat(evaluator-form): add tooltip for existing evaluator usage in edit mode
* feat(evaluator-form): enhance target data label with tooltip for edit mode
* chore: lint
* chore(dataset-run-items): GET /dataset-run-items
* fixup(dataset-run-items): GET /dataset-run-items
* chore(dataset-run-items): GET datasetRun/[runName]
* fix(dataset-run-items): return 0 for count when no records found
* fix(dataset-run-items): ensure offset is calculated only when page and limit are present
* feat(dataset-run-items): enhance dataset filtering by adding datasetId support and refactor related functions
* chore: rm comment
* refactor(dataset-run-items): rename dataset run name to ID across API and table definitions
* feat(dataset-run-items): add dataset_run_items to Clickhouse table names and refactor dataset filtering to use camelCase
* fix(dataset-run-items): update dataset filtering to ensure only the latest version of each dataset run item is retrieved
* chore: imports
* chore: fix de-duplication
* chore(dataset-router): mark runitemsByRunIdOrItemId as deprecated
* chore: refactor only, split out routes into byItemId and byRunId
* Revert "chore: refactor only, split out routes into byItemId and byRunId"
This reverts commit 2e1f2e3c14b9ed1173e2fb73b7f721c72d57bfbb.
* Revert "chore(dataset-router): mark runitemsByRunIdOrItemId as deprecated"
This reverts commit 5cb2232198d84baf7f027b00603c6fb3b9489c52.
* fixup: runitemsbyrunidoritemid
* chore: rewrite runitemsByRunIdOrItemId
* chore: rewrite runsByDatasetIdMetrics
* chore: ordering
* chore: order by created at and remove base data from query
* chore: add back name
* chore: push
* chore(dataset-run-items): GET /dataset-run-items
* fixup(dataset-run-items): GET /dataset-run-items
* chore(dataset-run-items): GET datasetRun/[runName]
* fix(dataset-run-items): return 0 for count when no records found
* fix(dataset-run-items): ensure offset is calculated only when page and limit are present
* feat(dataset-run-items): enhance dataset filtering by adding datasetId support and refactor related functions
* chore: rm comment
* refactor(dataset-run-items): rename dataset run name to ID across API and table definitions
* feat(dataset-run-items): add dataset_run_items to Clickhouse table names and refactor dataset filtering to use camelCase
* fix(dataset-run-items): update dataset filtering to ensure only the latest version of each dataset run item is retrieved
* chore: imports
* chore: fix de-duplication
* feat(dataset-run-items): implement experiment service writes
* chore: add
* fix(experiments): update error messages for missing API key using constants
* chore(experimentCreateQueue): increase backoff delay to 10 seconds
* chore(auth): update ingestion types for improved flexibility
* chore: create traces & error-level generations for DRIs
* chore: unify timestamps
* chore: revert test to PG implementation until we add DRI migration across environments
* chore: typo
* chore: implement unified trace ID generation for ClickHouse and PostgreSQL executions to prevent duplicate traces
* chore: refactor trace creation logic for ClickHouse and PostgreSQL executions to use a unified approach
* chore: eslint
* chore: comment
* chore: rebase
* chore: rebase
* chore: rename function for clarity and fix typo in comments
* refactor: remove fetchDatasetRun function and replace with direct Prisma query for dataset run retrieval
* chore: eslint
* chore: fix test
* chore: fix test
* refactor: rename function and update comments for clarity in experimentServiceClickhouse
* feat(dataset): implement dataset run items deletion queue and processing
* feat(dataset): add dataset run items deletion functionality and integrate with deletion queue
* chore: revert imports
* chore: handle DRI deletion on project deletion conditionally
* fix(dataset-router): await Promise.all for dataset run items deletion
* refactor(dataset-router): streamline dataset deletion process and ensure async handling of dataset run items
* fix(env): add new environment variable for dataset run items deletion concurrency duration
* chore: rm comment
* refactor(dataset-run-items): update delete functions to use object destructuring for parameters
* fix(dataset-run-items): replace hardcoded request timeout with environment variable for deletion timeout
* refactor(dataset): rename and restructure dataset deletion functionality, replacing dataset run items deletion with a unified dataset deletion queue
* fixup(experiments): support trigger for remote experiment run
* chore: remove instructions
* chore: rename webhook > remote experiment server side
* chore: rename webhook > remote experiment client side
* style: design review
* chore: lint
* fix(experiments): change URL validation to use z.url() in RemoteExperimentUpsertForm
* chore: rename
* feat(datasets): add transformation function for datasets and update API responses
* docs: add technical specs for multi window/prompt playground
* docs: add requirements and tech summary for sharing
* feat: add multi-window playground architecture and state isolation
* feat: make playground UI multi-window capable
* fix: multi window playground caching
* fix: jump to playground button adds windows to playground using stable id
* fix: avoid race condition in writing to cache before navigating to playground
* feat: add collapsible, more compact sections to non-model config
* feat: add compact version of model picker/params component
* feat: add ability to copy window states to multi playground
* fix: solve header crowding for thin windows in playground
* feat: make entire playground config section collapsible
* feat: change SaveToPrompt button to align with other window buttons
* feat: switch buttons in page header to icon-only when too small
* fix: adjust button labels to fit button size
* fix: update the execution window status correctly
* docs: complete remaining task items
* refactor: simplify playground state management hooks
* fix: make the model configuration section prettier
* docs: remove agent work files
* refactor: call register functions on the registry directly
* chore: refactored some components and compacted the design even more
* chore: move add message buttons
* chore: start align model param settings popover when compact
* chore: small UI spacing improvements
* fix: scrolling bug
* fix: handle config headers when window gets small
* chore: rename button hover
* feat: move CMD+Enter to execute all button
* feat: give user the choice to jump to fresh/existing playground
* push
* chore: fix
* chore: fix
* push
* push
* fixes
---------
Co-authored-by: Max Deichmann <m.deichmann@tum.de>
* fix: use drop down for prompt full text search
* fix: use drop down for prompt full text search
* fix: use drop down for prompt full text search
* fix: use drop down for prompt full text search
* fix: use drop down for prompt full text search
* feat(migrations): add dataset_run_items table in clickhouse
* feat(dataset): add environment variables and utility for pg/ch execution
* chore: re-write `runitemsByRunIdOrItemId` to support both clickhouse and postgres execution
* chore: use `runItemsByRunId` or `runItemsByItemId` rather than `runitemsByRunIdOrItemId`
* chore: rename env variables
* chore: reflect dri as ch data is seeder
* fixup: dataset run item types and converters
* chore: background migration
* chore: rewrite dataset-router routes
* chore(dataset): dual-write strategy for dataset run items to support ClickHouse and PostgreSQL
* chore: add utilities and converters
* chore: rewrite GET api/public/dataset-run-items
* feat(dataset): implement dataset run items deletion queue and processing
* chore: rewrite DELETE api/public/datasets/{datasetName}/runs/{runName}
* chore: rewrite GET api/public/datasets/{datasetName}/runs/{runName}
* fixup: rewrite GET api/public/datasets/{datasetName}/runs/{runName}
* chore(schema): add error to DRI schema
* fixup(ingestion): support dataset run items
* fixup: rewrite POST /api/public/dataset-run-items
* fixup: rewrite POST /api/public/dataset-run-items --amend
* fixup(ingestion): support dataset run items --amend
* fixup(experiments): preliminary implementation, for illustrative purposes only
* fixup(schema): indexes
* fixup(schema): remove indexes, add `output`
* chore: support DRI ingestion
* chore: eslint, writes related
* chore: rewrite experiment service
* chore: rm file
* chore: cascade delete dataset run items
* chore: add clustered migrations
* chore: update background migration
* chore: handle experiment job errors
* chore: remove CH read implementation
* chore: remove md file
* chore: remove CH read implementation II
* chore: fix typing
* chore: fix migration types
* fix: lint/imports
* fix: import errors
* chore: imports and comments
* chore: remove only from test
* fix: validateDatasetRunAndFetch
* fix: validateDatasetRunAndFetch id
* fixup: attempt fix of test
* fix: test
* chore: test validate logging
* chore: adjust logs
* chore: revert cascade delete
* Revert "feat(dataset): implement dataset run items deletion queue and processing"
This reverts commit 9f725f053386ded41e3617efaacee128d5ac023f.
* chore: delete imports
* revert: experiment service changes
* chore: adjust comment
* rename: env variables to incl experiment
* chore: remove unused variable eslint disables from datasetExecution and types files
* refactor: remove enrichedDatasetRunItem function and integrate its logic directly into IngestionService
* chore: remove TODO comment regarding dataset run item authorization
* refactor: replace Date constructor with parseClickhouseUTCDateTimeFormat for date parsing in dataset run item conversion
* chore: remove deprecated flag for runitemsByRunIdOrItemId method
* refactor: remove getDatasetRunItemsByRunId function to streamline dataset run item retrieval
* refactor: add back run properties in dataset runs mapping
* chore: revert changes to delete-dataset-run API
* refactor: remove validateDatasetItemAndFetch function and replace its usage with direct Prisma query in IngestionService
* refactor: delete validateCreateDatasetRunItemBodyAndFetch function and replace its logic with direct Prisma queries in dataset-run-items API
* refactor: move createOrFetchDatasetRun function to a new dataset-runs API file and maintain unique constraint handling
* refactor: remove validateDatasetRunAndFetch function and replace its usage with direct Prisma queries in dataset-run-items and IngestionService
* chore: nits
* refactor: streamline dataset run item creation by consolidating validation and execution logic
* chore: fix imports
* fix: types
* fix: simplify ingestion schema
* refactor: enhance ingestion schema creation to support public and internal environments
* chore: extend immutable keys list DRI
* chore: use same id for clickhouse and postgres
* chore: remove background migration trigger
* chore: move dataset run items migration into readme
* chore: remove creation of DRI in seeder
* chore: lint
* feat(tracing): pretty display json as collapsible table
* show empty list and unwrap single item containing objects
* enable toggle for code or pretty view
* clean up
* fix type casting
* some more fixes
* expand row with click anywhere
* fix alignment
* remove dead code
* cleanup
* make it more clean
* don't unwrap initial items
* collapse expand all
* fixing collapse button
* don't render for chatml
* better markdown check
* reduce col width
* make table borderless
* display null / empty string for empty value
* smaller text
* make table more compact
* make preview items grey & italic
* linebreak path column
* table headers inherit color
* cleanup
* external collapsed state; also move button into code view
* remove code view switch
* show empty dicts also as empty
* always show objects as tables
* extract markdown into helper function
* review fixes
* update
* Update turbo.json to use @langfuse/shared#dev instead of #build
Co-authored-by: max <max@langfuse.com>
* Add incremental build script and update turbo.json dev dependencies
Co-authored-by: max <max@langfuse.com>
* Remove build:dev script and update turbo.json dependencies
Co-authored-by: max <max@langfuse.com>
* push
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
* perf: enable API Key and Prompt caching in redis by default
* chore: update sync prompt test
* chore: disable prompt caching for sync web tests
* chore: enable prompt caching in sync tests
* chore: disable cachking for v1 pipeline
* chore Use cacheEnabled as pure test overwrite
* chore; undefined check
* chore: log body
* chore: patch promptCache test
* chore: fmt
* remove console log
* chore(experiments): enhance error handling in LLM call and implement retry logic for experiment creation jobs
- Added try-catch block in callLLM function to handle specific API errors and return appropriate ApiError responses.
- Implemented retry logic for experiment creation jobs in experimentQueue, allowing retries for rate-limited and server errors, with a delay mechanism based on job age.
* chore: push
* refactor: introduce delay utility for job processing
* chore: abstract standardized error handling for LLM operations
* refactor: reorganize utilities and enhance error handling for job processing
* chore: push
* chore: typing
* chore: unify error message
* fix(retry-handler): change executeTakeFirst to executeTakeFirstOrThrow for better error handling
* chore(prompt-experiments): set langfuse-native environment
* refactor: no longer export ingestionEvent schema directly
* chore: rm line number
* chore(ingestion): refactor processEventBatch to accept options object
* chore(ingestion): refactor ingestion schemas for public and internal environments
- Introduced separate schemas for public and internal environments.
- Updated environment name validation logic and error messages.
- Refactored event schemas to utilize the new schema structure.
- Deprecated direct export of `ingestionEvent` schema in favor of factory method for better environment handling.
* chore: push
* chore: type environment schema as string
* chore: push
* chore: fix import statements
* chore: update langfuse-langchain to version 3.38.3
* chore: prettier
* chore: revert prettier
* add prettier command
* add pre-commit hook
* sync prettier config with .vscode config
* add line
* fix husky
* update for less changes
* upgrade to prettier 3.4 for bug fixes
* ignore no-undef in TS (also worker!)
* changes to prettier
* adhere to style
* make prettier 3.4 default for bug fix
* add prettier ignore
* reset to standard
* update to 3.6.2
---------
Co-authored-by: Max Deichmann <m.deichmann@tum.de>
* chore(dx): unit tests run against their own db
* fix clickhouse
* clean up a bit
* close
* update only use different postgres
* update with command
* remove CH
* fix: create virtual timestamp column in AMT query to support all filters
* chore: add test logging
* chore: skip database prune for async tests
* chore: cleanup
* chore: drop console logs
* perf: start to experimentally shift reads to new aggregatingmergetrees
* chore: linting
* chore: more linting
* chore: create checklist for trace table view migrations
* feat: shift by id queries onto new AMTs
* chore: cover traces for public api
* chore: add notes for maxMap
* chore: incorporate feedback into query declarations
* chore: cleanup queries
* chore: adjust reads to new layout
* chore: update ingestion pipeline to make use of nullable fields
* chore: add sampling
* chore: update sampling decision attribute
* chore: fix limit for getTracesByIds
* chore: spacing
* chore: linting
* Disable import button during processing and show loading state
Co-authored-by: marc <marc@langfuse.com>
* prettier
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
* run experiments with placeholders as well
* add missing file
* fix experiements
* run outside of loop
* add tests
* fix tests
* make tests clean again
* also test that variables resovle first
* Enhance prompt search to include tags with case-insensitive matching
Co-authored-by: marc <marc@langfuse.com>
* drop md
* prettier
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
* fix(prisma): update JobExecution and DefaultLlmModel relations to include onUpdate and onDelete behaviors to match DB constraints
* chore(prisma): update foreign key onUpdate behavior for job_template_id in JobExecution model
* Revert "chore(prisma): update foreign key onUpdate behavior for job_template_id in JobExecution model"
This reverts commit e65c12c9c36a1f8732560e665b1d8d7422c530f0.
* Fix typos: GitHub brand name and "check out" verb phrase
Co-authored-by: marc <marc@langfuse.com>
* Checkpoint before follow-up message
* Fix GitHub brand name and typos across documentation and UI
Co-authored-by: marc <marc@langfuse.com>
* Changes from background composer bc-bdbd909e-6932-4489-b125-e66b1d0f5ebe
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
* no naming collisions between prompt and placeholder
* tests for variable placeholder name collision
* fix test spelling
* also show conflict in playhground
* Add environment to PostHog event tracking for traces and observations
Co-authored-by: marc <marc@langfuse.com>
* use object envs instead of trace env
* fix
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
* upgrade turbo
* add web dev turbo runner
* upgrade docker
* fix linter
* set env mode to loose (as in turbo v1) so we don't have to double specify env vars
* feat(nav): integrate HoverCard component for improved navigation item display
* refactor(nav): enhance HoverCard integration and sidebar styles for improved UX
* chore(sidebar): replace cookie-based state management with local storage for sidebar state persistence
* chore: push
* refactor(nav): wrap HoverCardContent in Portal for improved rendering context
* style: mark hovered item as active
* chore(CreateLLMApiKeyForm): replace Input with PasswordInput for improved UX
* chore: revert
* chore(data): add seed data and constants for datasets, prompts, and traces
- Introduced new JSON files for chat ML and nested JSON data.
- Created seed constants for datasets and prompts to streamline data generation.
- Updated seed scripts to utilize new constants and improve dataset creation logic.
- Enhanced Clickhouse preparation scripts to support new dataset structures and trace generation.
- Added utility functions for generating unique trace IDs.
This commit lays the groundwork for improved data handling and observability in the Langfuse application.
* chore(data): remove deprecated prompt files and enhance seed constants
* chore(prisma): remove unique constraint on projectId, datasetId, and input in DatasetItem model
* chore: push
* chore(seeder): introduce seeder abstraction
* chore: typos
* chore: restructure files
* chore: restructure files
* chore: fix imports
* chore: fix imports
* chore: lint
* chore: simplify
* chore: ensure eval data integrity
* chore: update metric name calculation in ClickHouse query builder
* refactor: use test-utils for clickhouse inserts
* chore: add otel conventional attributes to clickhouse db spans
* chore: use db.system instead of db.system.name
* chore: set span kind client for clickhouse queries
* fix(NewDatasetItemForm): add JSON formatting utility for input, output, and metadata fields
* perf(NewDatasetItemFromExistingObject): conditionally render NewDatasetItemForm based on form state
* fix(NewDatasetItemForm): update JSON value check to handle undefined case
* fix(TracesTable): stabilize control column definition
- Ensured conditional rendering of the bookmarked column remains intact while simplifying the structure.
* refactor(TracesTable): enhance column visibility and order identifiers for control states
- Updated dialog header to use a title component and improved button layout for clarity.
- Adjusted positioning of the close button for better alignment.
- Enhanced header and footer with rounded corners for a more polished look.
* fixup: limit render duration of traces table
* fixup: reduce render cycles through data flows and memoization
* refactor: enhance performance and structure of table components
- Removed unnecessary console logs in `data-table.tsx`.
- Refactored `TablePeekView` to use a more structured props type and memoization for improved performance.
- Updated `usePeekData` to reduce stale time from 5 minutes to 1 minute.
- Replaced `IOTableCell` with `MemoizedIOTableCell` in various components to optimize rendering.
- Adjusted `peekView` configurations in `observations.tsx` and `traces.tsx` for consistency and clarity.
* chore: eslint
* chore: move features
* chore: move features
* chore: move features
* chore: move features
* chore: move features
* chore: move features
* chore: move features
* chore: move features
* chore: move features
* chore: move features
* chore: move features
* chore: move features
* chore: move features
* chore: move features
* chore: move features
* changes to pro plan
* chore: move features
* truncate title and description to prevent wraps
* remove unnecessary padding from charts
* improve bignumber chart
* add min h/w
* mobile resizing
* fix
2025-05-29 15:58:38 +00:00
Max DeichmannGitHubellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
* perf: add exact timestamp match for eval batch actions
* perf: add exact timestamp match for eval batch actions
* perf: add exact timestamp match for eval batch actions
* chore: make `project_id` nullable on `eval_templates`
* feat: add script for managed langfuse evaluators
* feat: restyle running evaluator configs table
* feat: add peek view for running evals
* fixup: add peek view for evaluator library
* fixup: add configure button to table
* feat: rework eval creation flow
* feat: add option to edit eval from table
* fixup: configure evaluator flow with traces data preview
* style: eval configuration UI elements
* chore: add support for current langfuse templates
* chore: reference templateId on jobExecution
* style: add maintainer icon across
* chore: extract WorkerEvaluationVariableService
* fixup: support live data mapping interface
* fixup: revamp datasets eval ui
* fixup: add clone functionality
* chore: rename online evaluation -> llm-as-a-judge
* feat: add evaluator selection in experiment creation flow
* chore: refactor into hooks in experiment form
* feat: add ragas evaluator prompt templates
* chore: show evaluator prompt preview by default
* fix: eval table filtering
* feat: add input box to slider
* fix: clean up running eval peek view and logs
* fix: ensure preview variables are not wrapped in quotes in evaluation prompt preview
* feat: support navigating through traces table
* feat: support default model workflow
* fix: typos in managed evaluators
* fix: do not allow selection of evaluators requiring default to run
* fix: type and build errors
* chore: rename table to default_llm_model
* chore: rm empty file
* fixup: complete rename
* chore: push
* style: eval set up form
* style: separate new evaluator form out into two steps
* chore: ensure we only run evals if default model is set
* chore: add `partner` col to `eval_templates`
* style: new eval template creation
* chore: add proper mapping for dataset evals
* feat: also put evaluation selection on datasets page
* feat: datasets also run on historic experiments after creating mapping
* revert: 09ab242dc12c8647bc716c84d180bc8e631d4919
* chore: fix eslint and ts
* chore: fix warnings
* fix: ensure we inactivate all job configs that rely on default eval model
* fix: tests
* chore: fix eslint
* chore: wrap in tx
* chore: fix default model checkbox logic
* chore: improve wording on delete model
* fix: eslint
* chore: adjust migration order
* style: push ui improvements
* style: sort templates by partner
* fix: eslint
* chore(dashboards): expand measures for dashboards
* chore: add testcase
* chore: define inputTokens and outputTokens
* chore: add timePerOutputToken and costs
* chore: fix definition and handling of tag based breakdowns
* feat(dashboard): make dashboard widgets draggable
* chore: do not update on layout change
* chore: fix size
* chore: remove the preventCollision flag
* chore: linting
* chore: get row height dynamically
* chore: make chart resizable
* chore: styling for resizes
* chore: add migration to align y values
* chore: add owner to project and return langfuse dashboards
* chore: show owner for dashborads in table
* chore: snapshot
* chore: snapshot
* chore: allow widget clone
* chore: snapshot
* chore: linting
* chore: use project instead of user
* chore: show error message on clone failure
* chore: indicate existing dashboard widget relationship
* chore: show only project dashboards in selection for adding widgets
* chore: make full dashboard row clickable
* chore: add clone button to langfuse dashboards
* feat(dashboards): add support for cloning existing dashboards
* chore: move buttons into a dedicated dropdown menu
* chore: allow updates to dashboard name and description
* chore: lint
* chore: removal double capturing of dashboard delete
* chore: track deletions
* feat: full text search
* push
* feat: full text search
* feat: full text search
* feat: full text search
* feat: full text search
* feat: full text search
* fix
* push
* fix: fix json limits and scope to single bodies only
* fix
* fix
* jumping UI
* jumping UI
* jumping UI
* push
* Update web/src/components/table/data-table-toolbar.tsx
Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
---------
Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
* push
* fix: fix json limits and scope to single bodies only
* push
* fix: fix json limits and scope to single bodies only
* fix: fix json limits and scope to single bodies only
* fix: fix json limits and scope to single bodies only
* fix: fix json limits and scope to single bodies only
* fix: fix json limits and scope to single bodies only
* chore: memoize table body for enhanced front end performance
* chore: push
* chore: refactor
* style: speed up animation
* chore: re-render on table state changes
* chore: import function directly
* chore: refactor to use timestamp
* chore: push
* chore: push
* feat: add dlx retry queue and service
* feat: add dlx retry queue and service
* feat: add dlx retry queue and service
* feat: add dlx retry queue and service
* feat: add dlx retry queue and service
* feat: add new organizations apiKey route
* chore: implement additional org routes
* chore: add API docs
* chore: checkin openapi spec
* chore: update tests
* feat: allow org api key management via admin api
* feat: allow the creation of projects using an org api key
* chore: allow deletion of projects via API
* feat: create outline of SCIM Users support
* feat: implement SCIM style user endpoints
* chore: allow set password on user
* chore: lint
* feat: add endpoints to manage project and organization memberships
* chore: add memberships api docs
* chore: reject project membership calls if entitlement is missing
* chore: add team plan to test org
* chore: add reference to full Langfuse API spec
* feat: add new organizations apiKey route
* chore: implement additional org routes
* chore: add API docs
* chore: checkin openapi spec
* chore: update tests
* feat: allow org api key management via admin api
* feat: allow the creation of projects using an org api key
* chore: allow deletion of projects via API
* feat: create outline of SCIM Users support
* feat: implement SCIM style user endpoints
* chore: allow set password on user
* chore: lint
* feat: add endpoints to manage project and organization memberships
* chore: add memberships api docs
* chore: reject project membership calls if entitlement is missing
* chore: add team plan to test org
* feat: add new organizations apiKey route
* chore: implement additional org routes
* chore: add API docs
* chore: checkin openapi spec
* chore: update tests
* feat: allow org api key management via admin api
* feat: allow the creation of projects using an org api key
* chore: allow deletion of projects via API
* feat: create outline of SCIM Users support
* feat: implement SCIM style user endpoints
* chore: allow set password on user
* chore: lint
* chore: add scim api docs
* chore: remove outdated generated artifacts
* feat: add new organizations apiKey route
* chore: implement additional org routes
* chore: add API docs
* chore: checkin openapi spec
* chore: update tests
* feat: allow org api key management via admin api
* feat: allow the creation of projects using an org api key
* chore: allow deletion of projects via API
* chore: add documentation for new project routes
* chore: cleanup
* chore: error updates and handling
* chore: reduce reduntant error handling
* chore(evals): fall back to bullmq `job.data.timestamp` allowing for successful retries
* chore: test
* chore: test
* docs: add diagram and docs
* chore: typo
* chore: typo
* chore: refactor authentication for admin apis
* chore: lint
* chore: address feedback
* feat: add post organizations endpoint
* chore: update tests
* chore: update api key data model
* chore: align code with new scoped api keys
* chore: typing
* chore: align tests with new logic
* chore: remove ratelimitoverrides
* chore: remove accesslevel from queue message
* chore: add additional check to ingestion endpoint
* chore: add test for legacy api key format in redis
* chore: map all api keys in auditlog
* added environment filter option to traces
* WIP on multi-select ui inconsistency
* fixed overflow issue in multi select where an item would not take the full width automatically; also changed to overflow-x-auto
* fixed invalid DOM nesting of div within p (FormDescription) that resulted in a hydration error
* fix multiselect to cut overflowing text with ellipsis and take full name as titlte so you can hover over it, removed unnecessary css classes
* WIP on create evaluator page ui issue with filter builder
* Revert "WIP on create evaluator page ui issue with filter builder"
This reverts commit fd6dd507f2df3c5fb404f4e38a2983e8ef6405ef.
* hide empty string from multi-select options, as supporting empty values would require further work - out of scope for now; also checked for side effects, with current usage all good
* Added test suite for eval filtering
* Resolved comment: removed getEnvironmentsForProject from traces router and instead fetched environments inside evaluator-form via api.projects.environmentFilterOptions
* move eval filtering tests into separate test file
* removed redundancy via fixtures, prepared tests for concurrency
* filtering tests run concurrently
* extracted traceFilterOptions fetching into separate hook as it is used in both the evaluator form and the traces table, also fixed traces table since the environment was now missing there
* useTraceFilterOptions Hook after merge
* Make TraceOptions keys partial so that we can update filter options asynchronously
* default to empty object if response undefined
* exclude environment filter builder column from traces table since toolbar already has env
* removed useTraceFilterOptions hook as use case changed such that extraction into hook is no longer warranted
* chore: pin dev clickhouse version to 24.3
* chore: only add input format flag for cloud
* chore: whitelist accepted cloud regions
* chore: disable test
* fix: (workaround) evals page overflows when rendering right hand side metadata bar
This is more of a workaround, we should probably disable the overflow on the main wrapper div in layout.tsx and enable it only where needed - or find out why the overflow inside the metadata sidebar trigggers the scrollbar to pop up - some info on what I found:
- The body and other divs close to the root do not increase their size.
- The navigation sidebar does increase its vertical height, but limiting that does not resolve the main scrollbar.
- Limiting specifically the wrapper div in layout.tsx with overflow-hidden does resolve this, but globally disabling it requires most pages to be refactored.
* Revert "fix: (workaround) evals page overflows when rendering right hand side metadata bar"
This reverts commit 313a1c0d8a885a04974399f850f7f9aee58fd714.
* fix(ui): TableWithMetadataWrapper uses contain-layout on its root div so that EvaluatorDetail overflow issue is mitigated
Checked for side effects on the other components using TableWithMetadataWrapper and all work as expected.
---------
Co-authored-by: Marlies Mayerhofer <74332854+marliessophie@users.noreply.github.com>
* chore(annotation-scores): revert to synchronous delete; upsert on create route
* fix: ensure scores data not only marked as stale but refreshed on score update
* feat: add support for GCS buckets
* chore: cleanup
* chore: trim gcs credentials before parsing
* chore: add logging for redis shutdown in tests
* chore: update timeout for trace delete test
* chore: bump trace delete concurrency for tests
* feat(evals): enable deletion of evaluators and templates via UI
* Added tests for the evals job and template deletion trpc endpoints
* evals-trpc test refactor, it had a side effect on other tests since it was pruning the db, now instead creates a new org and project per tests and cleans them up after all tests are done
* removed test comments as those are now implemented
* fix wrong wording
* added template version to delete confirmation on template details delete action
* DeleteButton refactoring and moved deletion of templates to LFE-4573
* moving the call to captureDeleteSuccess inside the successful branch of executeDeleteMutation
* Display lock icon when action button in icon mode is unauthorized
* fix: quietly delete scheduled evals for deleted job executions
---------
Co-authored-by: Max Deichmann <m.deichmann@tum.de>
* fix(trace-timeline): overflow and scroll behaviour
* fix(trace-timeline): show maximum of 3 scores and remainder in hover card
* push
* chore: refactor
* chore
* chore
* feat: add self-serve dashboard backend poc
* chore: data model thoughts
* chore: create initial query builder example and test case
* chore: handle empty dimensions and metrics
* chore: lint
* chore: extend filter conditions to account for timestamps
* chore: refactor query builder into sub-functions
* chore: use template queries for user-supplied values
* chore: add query builder tests
* chore: handle time dimension
* chore: add trpc endpoint to execute custom clickhouse query
* chore: add a query playground component to test a couple of queries
* chore: handle multiple joins
* chore: add observations table to data model
* chore: fix bug in data model
* chore: add test case to compare with old dashboard results
* chore: add scaffold for additional views
* expand the users view and add segments to filter scores subviews
* chore: add test cases for score views
* chore: drop users and sessions for now
* chore: add scores aggregate and observations cost tests
* chore: add sql injection tests
* chore: lint
* chore: fix dashboard test cases
* chore: drop users query test
* chore: add order by logic
* chore: fill timeseries values
* chore: use new query function on dashboard
* chore: convert TracesBarListChart.tsx to new query endpoint
* chore: make chart data compile
* chore: update tests
* chore: add tags to custom queries
* chore: typing
* chore: suffix join condition with sql
* chore: add typing in queryBuidler
* chore: limit playground to cloud admin users
* chore: add util to map legacy dashboard columns to new model
* chore: separate time filter state for TracesBarListChart.tsx
* chore: pass timestamps directly into chart components
* chore: add userId and sessionId on observations
* chore: pick auto time granularity based on hours
* fix(prompt-experiments): show form error if name is duplicated
* chore: fix typo
* Update web/src/ee/features/experiments/hooks/useExperimentNameValidation.tsx
Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
* eslint
---------
Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
* feat(evals): preview number of historic items to be evaluated
* refactor: split up large evaluator form file
* chore: drop optional pagination
* chore: fix eslint
* chore: rename test
* fix: safely parse big numbers in api responses
* feat: add additional input/output parsing for pydantic via OTel
* Revert "feat: add additional input/output parsing for pydantic via OTel"
This reverts commit 71d0f3abd951702393aac7ab79a875b888f089cf.
* pin node to problematic version for working on a fix
* added cross-env and disabled the feature flag causing the testing issue - long term fix should probably be switching to vitest
* unpin node version since problematic feature is now disabled fo affected tests
* fix(ui): IOPreview now renders messages where content is an empty string instead of ignoring them
* removed debug logs
* OpenAiMessageView first filters valid messages to render and then executes the "show x more ..." logic, this way we ensure that there can never be a mismatch between messages items and the amount of fields we support displaying
* Render empty string quotation marks if content is empty but not null or undefined
* Change length check to falsy check for conditional display of empty quotes if content is empty
* cleaner falsy check for rendering empty string quotation marks
* fix(ui): stop propagation on traces table checkbox
* fix(peek-ui): handle onRowClick action correctly for peek view
* feat(peek): do not close upon any screen button or checkbox clicks
* push
* wip
* simplify
* actionbutton for upgrade prompt
* count events for free plan limit
* add sso settings
* rename lite to core
* add product id
* nit
* nits
* fix rate limit test
* ci run on teams plan
* fix
* fix names of pipeline steps
* test tmp
* add better error logs to makeAPICall
* push
* clone to throw better errors
* improve error
* improve error
* fix
* remove comment
* refactor
* push
* next attempt
* nit
* create new auth for failing test suites
* push
* fix rate limit test
* feat: add media and export external endpoint config
* chore: add docs string
* chore: add external endpoint support for azure
* chore: drop feature for media
* Unified design of grouped score badge with trace detail score badges, except that text is no longer medium/semibold
* Made padding of root item of ObservationTree consistent with other items
* fixed alignment issues of badges wrapped in links in trace and observation detail views
* Fixed tooltip of aggregated usage data badge (in trace details) rendering even if there is no underlying data
* chore(cloud): Make UI recognize cloud:enterprise plan and add corresponding access rights and rate limits
- label on project selector
- rate limits same as team
- access rights same as team
- no stripe product id for this (yet), cannot be purchased self-serve in product
* remove comments
---------
Co-authored-by: Marc Klingen <git@marcklingen.com>
* feat(auth): add workos as idp
* add button for org and connection based sign in
* fix
* remove client auth method and checks
* remove unnecessary diff
* fix: header
* fix: search always show current version
* chore: adjust code block
* chore: re-add copy in code editor if no title
* chore: do not match current version always
* adjust margins
* chore: remove breadcrumb
* fix: typo
* fix: final margin
* style: eval template
* small design nits
* fix: do not allow to deactivate eval configs which rano on historic traces only
* fix: do not allow to deactivate eval configs which rano on historic traces only
* fix: do not allow to deactivate eval configs which rano on historic traces only
* fix: do not allow to deactivate eval configs which rano on historic traces only
* fix: do not allow to deactivate eval configs which rano on historic traces only
* fix: do not return all executions for eval config list trpc
* fix: do not return all executions for eval config list trpc
* fix: do not return all executions for eval config list trpc
* fix: do not return all executions for eval config list trpc
* chore: fix demo banner
* fix(page-header): info icon relative positioning
* fix: header title wrap behaviour
* fix(page-header): have info icon follow title
* style: move dataset level buttons to more menu
* push
* push
* abstract editor to support "text" mode in addition to "json" mode
* always use CodeMirrorEditor for prompts
* minheight on editor
* upgrade codemirror and add language package
* add linting for prompts and switch to githubDark theme
* package
* use codemirror for chat prompts
* imrpove scroll behavior of new dataset form
* improve dataset item editor view
* push
* remove dep pinning
* align editor linting with current variable definition
* defauklt value to value
* rename existing command exports
* move existing to input-command
* add comment
* wip cmd k menu for navigation
* fix types and paths of nested menu items
* add support for nested menu items
* filter correctly
* feat(prompts): support in UI prompt duplication
* chore(ui-ds): make `children` in `ActionButton` optional
* fix: extract custom hook for prompt name validation
* eslint
* fix: insert correct prompt count
* chore: validate prompt name uniqueness; attach latest label for single version
* chore: refactor duplication approach to use createMany and save on network-roundtrips
* chore: revert changes on action button
* push
* fix: allow field overwrites with `null` in ingestion api
* chore: add undefined check
* chore: check undefined does not overwrite value
* chore: explicit undefined check
* feat: convert generations to observations table
* feat: add type col to generations table
* feat: add type col to generations table
* feat: add type col to generations table
* feat: add type col to generations table
* feat: add type col to generations table
* fix: fix
* fix: update ordering for get traces calls to avoid stale results
* chore: explanation
* chore: remove stale test
* chore: lint
* chore: update comment
* chore: cleanup
* fix(ui): docs menu item should open in new tab
* Update web/src/components/nav-main.tsx
Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
---------
Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
* chore: stream clickhouse results to posthog to reduce memory consumption
* chore: end span after streaming all results
* chore: tracing for clickhouse streaming
* fix(prompt-experiments): do not restrict to evalModelsOnly in order to use anthropic, bedrock, vertex
* chore(cloud): disable deletion of traces via UI as it caused performance issues (#4873)
chore(cloud): disable deletion of traces via the UI as it caused performance issues
* docs: add link to data model documentation to api reference of ingestion endpoint (#4877)
* fix: use database upserts correctly (#4884)
* fix: use database upserts correctly
* fix: use database upserts correctly
* push
* fix: use database upserts correctly
* fix: use database upserts correctly
* push
* fix: use database upserts correctly
* chore(deps): bump next from 14.2.15 to 14.2.21 (#4887)
Bumps [next](https://github.com/vercel/next.js) from 14.2.15 to 14.2.21.
- [Release notes](https://github.com/vercel/next.js/releases)
- [Changelog](https://github.com/vercel/next.js/blob/canary/release.js)
- [Commits](https://github.com/vercel/next.js/compare/v14.2.15...v14.2.21)
---
updated-dependencies:
- dependency-name: next
dependency-type: direct:production
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* chore: release v3.5.2
* chore: remove postgres from ingestion merge (#4467)
* chore: remove postgres from ingestion merge
* chore: remove pg tests
* chore: remove postgres tracing
* remove evalModelsOnly
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Max Deichmann <m.deichmann@tum.de>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Steffen Schmitz <steffen@langfuse.com>
Co-authored-by: Hassieb Pakzad <68423100+hassiebp@users.noreply.github.com>
* fix(annotation): fail postgres upsert, delete silentley while Clickhouse is in lead
* chore: disable deletion of score while update operation is in progress
* chore(eval/experiments): support job retries via bullmq instead of within worker
* chore: timeout model completion calls after 2 minutes
* chore: set retries on llm completion calls to 1
* chore: do not propogate expected error cases in experiment creation
* eslint
* wip: optimistic updates on annotation form
* refactor to handle sequence of create, update operations correctly
* fix: don't check postgres for score existence when running on clickhouse
* disable annotation button while score updates are saving
* fix: await for all mutations to complete before opening modal again
* rm console log
* fix: update api response clickhouse
* chore: remove reading trace and score from prisma, only read from clickhouse
* fixup: keep create logic behind flag
* chore: only read from postgres for updating annoation score if `LANGFUSE_POSTGRES_INGESTION_ENABLED`
* chore: only read from postgres for deleting annoation score if `LANGFUSE_POSTGRES_INGESTION_ENABLED`
* chore: throw errors if scores not found in CH
* chore: add audit logs in clickhouse scenario
* only create audit log once on cloud
* feat(comments): add support for comment button on each trace preview in single session view
* perf: manually timeout comment count query at 500ms without failing overall operation
* push
* chore: query comment reference objects in clickhouse if `env.clickhouse_url`
* remove: deprecated code
* chore: support fetching traces per session from clickhouse
* push
* refactor: adjust tests to populate traces and observations in clickhouse
* refactor: adjust tests and remove redundant code
* push
* chore: drop FK v3 constraints
* make sleep great again
* make sleep great again
* make sleep great again
* make sleep great again
---------
Co-authored-by: steffen911 <steffen@langfuse.com>
* chore: make project deletion asynchronous
* chore: delete projects on the worker
* chore: indicate in UI overview that project is being deleted
* chore: remove API keys from Postgres in project deletion call
* wip
* push
* add migration
* render correct usage event type in settings
* render number of seats in settings
* wip
* add entitlement limits
* add new actionbutton
* migrate annotation queue create
* add to table date range selectors
* add limit to dashboard date range
* move migration
* fix usage preview in billing settings
* get seats from subscription instead of next invoice
* remove old console.log
* ad pro plan self-hosted
* fix
* readme
* rename / fixes
* push
* only alert on over 90% usage
* push
* push
* push
* push
* add new self host entitlements for v3
* drop batch-export entitlement
* push
* move logic to clickhouse
* push
* fix padding
* fix clickhouse logic
description:Use this agent when a feature branch is complete and ready to merge to main, and you need to create a changelog entry documenting the new feature or changes. This agent should be invoked proactively after significant feature work is completed and before merging.\n\nExamples:\n\n<example>\nContext:User has just completed implementing a new tracing visualization feature and the code has been reviewed.\nuser:"I've finished the trace timeline view feature. Can you help me prepare this for merge?"\nassistant:"Let me use the changelog-writer agent to create a changelog entry for this feature."\n<commentary>\nThe feature is complete and ready for merge, so we should use the changelog-writer agent to document it in the changelog.\n</commentary>\n</example>\n\n<example>\nContext:User mentions they're done with a feature implementation.\nuser:"The prompt versioning feature is done and tested. What's next?"\nassistant:"Great! Let me use the changelog-writer agent to create a changelog entry documenting this new feature before we merge."\n<commentary>\nSince the feature is complete, proactively use the changelog-writer agent to create documentation.\n</commentary>\n</example>\n\n<example>\nContext:User explicitly requests changelog creation.\nuser:"Can you create a changelog post for the new dataset export functionality?"\nassistant:"I'll use the changelog-writer agent to analyze the changes and create an appropriate changelog entry."\n<commentary>\nDirect request to create changelog, use the changelog-writer agent.\n</commentary>\n</example>
model:inherit
color:pink
---
You are an expert technical writer specializing in creating clear, user-focused changelog entries for developer tools and SaaS platforms. Your role is to document completed features in a way that helps users understand what's new, why it matters, and how to use it.
## Your Process
### Step 1: Understand the Changes
1. Extract the Linear issue number from the current branch name (format: lfe-XXXX)
2. Use the Linear MCP to fetch the issue details for additional context about the feature's purpose and requirements
3. Compare the current branch to main using git diff to understand the scope of changes at a high level
4. Identify the core feature or improvement that was implemented
5. Determine which parts of the codebase were affected (frontend, backend, API, database, etc.)
### Step 2: Study Recent Changelog Patterns
1. Read 3-5 of the most recent changelog posts in `../langfuse-docs/pages/changelog`
2. Analyze their structure, tone, and formatting conventions
3. Note how they:
- Title features (concise, benefit-focused)
- Explain the "why" (user problems solved)
- Describe the "what" (feature capabilities)
- Link to relevant documentation
- Use images/screenshots
- Format code examples or technical details
### Step 3: Identify Documentation Links
1. Check if there is relevant documentation in `../langfuse-docs/pages` that relates to this feature
2. If the feature is new, note that documentation may need to be created
3. If the feature extends existing functionality, identify which docs pages should be referenced
### Step 4: Draft the Changelog Entry
Create a changelog post that includes:
**Required Elements:**
- **Title**: Clear, benefit-focused headline (not just the feature name)
- **Date**: Use the current date in the format used by existing changelogs
- **Summary**: 1-2 sentences explaining what changed and why it matters to users
- **Description**: Detailed explanation of the feature, its capabilities, and use cases
- **Documentation Links**: References to relevant docs pages (if applicable)
**Style Guidelines:**
- Write in second person ("you can now...")
- Focus on user benefits, not implementation details
- Be concise but complete
- Use active voice
- Include technical details only when they help users understand the feature
- Match the tone and style of recent changelog entries
**Formatting:**
- Follow the exact file structure and frontmatter format of existing changelog posts
- Use appropriate markdown formatting (headings, lists, code blocks, links)
- Ensure proper spacing and readability
### Step 5: Assess Visual Needs
After drafting the changelog, explicitly tell the user:
- Whether a screenshot or image would enhance understanding of this feature
- What specific aspect should be captured in the screenshot (if applicable)
- Where in the changelog the image should be placed
## Quality Standards
**Before presenting your changelog:**
- Verify it follows the structure and style of recent entries
- Ensure all links are correctly formatted
- Check that technical terms match those used in the codebase and docs
- Confirm the feature description is accurate based on the code changes
- Validate that the user benefit is clear and compelling
## Output Format
Present your work in this order:
1. Brief summary of what you learned from the branch comparison and Linear issue
2. The complete changelog post content (ready to be saved as a new file)
3. Recommendation on whether to add an image/screenshot and what it should show
4. List of any documentation pages that should be referenced or created
## Important Notes
- The changelog lives in `../langfuse-docs/pages/changelog`
- Always check the Linear issue via the branch name (lfe-XXXX format) for context
- Compare against main branch to understand the full scope of changes
- Study recent changelogs before writing to maintain consistency
- Focus on user value, not technical implementation details
- Be thorough in your analysis before drafting
- If you're unsure about any aspect of the feature, ask clarifying questions before proceeding
description: Guidelines for positioning elements with banner awareness
globs:
- "**/*.tsx"
- "**/*.jsx"
alwaysApply: true
---
## Banner Height System
When positioning elements with `sticky`, `fixed`, or `absolute` that have a global reference point (like `top-0`), **always use `top-banner-offset` instead of `top-0`**.
### Why?
The application can display system banners (payment issues, maintenance notices, etc.) that push content down. Using `top-banner-offset` ensures your positioned elements appear below the banner rather than being covered by it.
- `h-screen-with-banner` / `min-h-screen-with-banner` - Use these for full-height containers that should account for banner space
- `pt-banner-offset` - Use for padding-top to offset content below the banner
- The system uses CSS variables `--banner-height` and `--banner-offset` defined in `web/src/styles/globals.css`
- Banner components dynamically update `--banner-height` using ResizeObserver to track their actual rendered height, ensuring accuracy across all screen sizes
### When NOT to use `top-banner-offset`
- Elements positioned relative to their parent container (not globally)
- Elements that should intentionally appear at the very top of the viewport
- Elements inside modals/dialogs that are already z-indexed above banners
- We build two containers, web (./web) and worker (./worker)
- We have shared code between these two in the shared package (./packages/shared). For that package, we have different entry points [package.json](mdc:packages/shared/package.json).
## Domain layer
The most important domain objects are in [observations.ts](mdc:packages/shared/src/domain/observations.ts), [traces.ts](mdc:packages/shared/src/domain/traces.ts), [scores.ts](mdc:packages/shared/src/domain/scores.ts).
## Database schema
We use Postgres and Clickhouse.
- The postgres schema is in [schema.prisma](mdc:packages/shared/prisma/schema.prisma)
- The clickhouse schema is in [0001_traces.up.sql](mdc:packages/shared/clickhouse/migrations/clustered/0001_traces.up.sql), [0002_observations.up.sql](mdc:packages/shared/clickhouse/migrations/clustered/0002_observations.up.sql), [0003_scores.up.sql](mdc:packages/shared/clickhouse/migrations/clustered/0003_scores.up.sql)
- When writing tests, focus on decoupling each `it` or `test` block to ensure that they can run independently and concurrently. Tests must never depend on the action or outcome of previous or subsequent tests.
- When writing tests, especially in the __tests__/async directory, ensure that you avoid `pruneDatabase` calls.
description:Please describe the question you have as clear and concise as possible.
validations:
required:true
- type:dropdown
id:hosting
attributes:
label:Are you using Langfuse Cloud or self-host Langfuse?
options:
- Langfuse Cloud
- Self-hosted Langfuse
validations:
required:true
- type:checkboxes
attributes:
label:I checked if there is already an [issue](https://github.com/langfuse/langfuse/issues) or [discussion](https://github.com/orgs/langfuse/discussions) for my question and asked the [Langfuse AI](https://langfuse.com/docs/ask-ai) for help.
description:Create a bug report to help us improve
title:"bug: "
description:Report a bug to help us improve
title:"bug: <short description>"
labels:["🐞❔ unconfirmed bug"]
body:
- type:textarea
attributes:
label:Describe the bug
description:A clear and concise description of the bug, as well as what you expected to happen when encountering it.
description:A clear and concise description of the bug, and what you expected to happen when you encountered it.
validations:
required:true
- type:textarea
attributes:
label:To reproduce
description:Describe how to reproduce your bug. Please provide detailed steps, code snippets, reproduction repos etc.
label:Steps to reproduce
description:Describe how to reproduce the bug. Please provide detailed steps, code snippets, a minimal reproduction repository, etc.
validations:
required:true
- type:dropdown
attributes:
label:Langfuse Cloud or self-hosted?
options:
- "Langfuse Cloud"
- "Self-hosted"
validations:
required:true
- type:input
attributes:
label:If self-hosted, what version are you running?
description:We may ask you to upgrade to the latest version, as many issues are continuously being fixed.
- type:textarea
attributes:
label:SDK and container versions
description:If you're experiencing an issue with an integration or SDK, please ensure you're using the latest version. If you're self-hosting Langfuse, check that you're running the most recent version. If updating isn't an option, please provide the specific versions you're currently using.
label:SDK and integration versions
description:If you're experiencing an issue with an integration or SDK, please share all package versions you're using. If you are not on the latest version, try upgrading, as this will often resolve the issue.
- type:textarea
attributes:
label:Additional information
description:Add any other information related to the bug here, screenshots if applicable.
description:Add any other information related to the bug here, including screenshots if applicable.
- type:dropdown
id:contribute
attributes:
label:Are you interested to contribute a fix for this bug?
description:If this is a confirmed bug, the maintainers are happy to support with guidance and review.
label:Are you interested in contributing a fix for this bug?
description:If this is a confirmed bug, the maintainers are happy to provide guidance and review.
# Without this patch, the JS SDK will not build due to
# "Error: packages/core build: src/api/core/auth/index.ts(1,10): error TS1205: Re-exporting a type when 'isolatedModules' is enabled requires using 'export type'."
if [ -f "packages/core/src/api/core/auth/index.ts" ]; then
if grep -q '^export { AuthProvider } from "./AuthProvider.js";$' packages/core/src/api/core/auth/index.ts; then
sed -i '1s/^export { AuthProvider }/export { type AuthProvider }/' packages/core/src/api/core/auth/index.ts
fi
fi
# Install dependencies and format
npm install -g pnpm
pnpm install
pnpm format
# Check for changes and show diff for debugging
if git diff --quiet; then
echo "No changes detected in TypeScript SDK"
exit 0
fi
# Close existing api-spec-bot PRs
gh pr list --author langfuse-bot --state open --json number --jq '.[].number' | xargs -I {} gh pr close {}
# Get the GitHub username of the original commit author
cd ../langfuse
ORIGINAL_AUTHOR=$(gh api repos/langfuse/langfuse/commits/${GITHUB_SHA} --jq '.author.login')
cd ../langfuse-js
# Create new branch and push changes
BRANCH_NAME="api-spec-bot-${GITHUB_SHA::7}"
git checkout -b "$BRANCH_NAME"
git add .
git commit -m "feat(api): update API spec from langfuse/langfuse ${GITHUB_SHA::7}" --no-verify
git push origin "$BRANCH_NAME"
# Create PR with original author as reviewer
gh pr create --title "feat(api): update API spec from langfuse/langfuse ${GITHUB_SHA::7}" --body "" --reviewer "$ORIGINAL_AUTHOR"
Langfuse is an **open source LLM engineering** platform for developing, monitoring, evaluating and debugging AI applications. See the README for more details.
## Linting
- Run `pnpm run lint` to lint all packages.
- Fix issues automatically with `pnpm run lint:fix`.
## Tests
- Codex cannot run the test suite because it depends on Docker-based infrastructure that is unavailable in this environment.
- When writing tests, focus on decoupling each `it` or `test` block to ensure that they can run independently and concurrently. Tests must never depend on the action or outcome of previous or subsequent tests.
- When writing tests, especially in the __tests__/async directory, ensure that you avoid `pruneDatabase` calls.
## Cursor Rules
- Additional folder-specific rules live in `.cursor/rules/`.
## Commits
- Follow [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/) when crafting commit messages.
Langfuse is an open-source LLM engineering platform that helps teams collaboratively develop, monitor, evaluate, and debug AI applications.
The main feature areas are tracing, evals and prompt management. Langfuse consists of the web application (this repo), documentation, python SDK and javascript/typescript SDK.
This repo contains the web application, worker, and supporting packages but notably not the JS nor Python client SDKs.
## Repository Structure
High level structure. There are more folders (eg for hooks etc).
pnpm run dev:web # Web app only (localhost:3000) - **used in most cases!**
pnpm run dev:worker # Worker only
pnpm run dx # Full initial setup: install deps, reset DBs, resets node modules, seed data, start dev. USE SPARINGLY AS IT WIPES THE DATABASE & node_modules
```
### Database Management
database commands are to be run in the `packages/shared/` folder.
- Use `golang-migrate` CLI for database migrations
- All database operations go through Prisma ORM for PostgreSQL
- Foreign key relationships may not be enforced in schema to allow unordered ingestion
### Testing
- Jest for API tests, Playwright for E2E tests
- For backend/API changes, tests must pass before pushes
- Add tests for new API endpoints and features
- When writing tests, focus on decoupling each `it` or `test` block to ensure that they can run independently and concurrently. Tests must never depend on the action or outcome of previous or subsequent tests.
- When writing tests, especially in the __tests__/async directory, ensure that you avoid `pruneDatabase` calls.
### Code Conventions
- **Pages Router** (not App Router)
- Follow conventional commits on main branch
- Use CSS variables for theming (supports auto dark/light mode)
- TypeScript throughout
- Zod v4 for all input validation
## Environment Setup
- **Node.js**: Version 24 (specified in `.nvmrc`)
- **Package Manager**: pnpm v9.5.0
- **Database Dependencies**: Docker for local PostgreSQL, ClickHouse, Redis, MinIO
- **Environment**: Copy `.env.dev.example` to `.env`
@@ -27,7 +27,7 @@ The maintainers are available on [Discord](https://langfuse.com/discord) in case
## Making a change
_Before making any significant changes, please [open an issue](https://github.com/langfuse/langfuse/issues)._ Discussing your proposed changes ahead of time will make the contribution process smooth for everyone. Large changes that were not discussed in an issue may be rejected.
_Before making any significant changes, please [open an issue](https://github.com/langfuse/langfuse/issues)._ Discussing your proposed changes ahead of time will make the contribution process smooth for everyone. Changes that were not discussed in an issue may be rejected.
Once we've discussed your changes and you've got your code ready, make sure that tests are passing and open your pull request.
@@ -35,6 +35,10 @@ A good first step is to search for open [issues](https://github.com/langfuse/lan
## Project Overview
We recommend checking out DeepWiki to familiarize yourself with the project:
@@ -186,7 +101,7 @@ Full database schema: [packages/shared/prisma/schema.prisma](packages/shared/pri
We built a monorepo using [pnpm](https://pnpm.io/motivation) and [turbo](https://turbo.build/repo/docs) to manage the dependencies and build process. The monorepo contains the following packages:
-`web`: is the main application package providing Frontend and Backend APIs for Langfuse.
-`worker` (no production yet): contains an application for asynchronous processing of tasks. This package is not yet used in production.
-`worker`: contains an application for asynchronous processing of tasks.
-`packages`:
-`shared`: contains shared code between the above packages.
-`config-eslint`: contains eslint configurations which are shared between the above packages.
@@ -197,61 +112,61 @@ We built a monorepo using [pnpm](https://pnpm.io/motivation) and [turbo](https:/
Requirements
- Node.js 20 as specified in the [.nvmrc](.nvmrc)
- Node.js 24 as specified in the [.nvmrc](.nvmrc)
- Pnpm v.9.5.0
- Docker to run the database locally
- Clickhouse client
**Note:** You can also simply run Langfuse in a **GitHub Codespace** via the provided devcontainer. To do this, click on the green "Code" button in the top right corner of the repository and select "Open with Codespaces".
**Steps**
1.Fork the repository and clone it locally
2. Run the development database
1.Install development dependencies:
- [golang-migrate](https://github.com/golang-migrate/migrate/tree/master/cmd/migrate#migrate-cli) as CLI
- [clickhouse binary](https://clickhouse.com/docs/install) on macOS with brew: `brew install --cask clickhouse`
3. Install dependencies and set up pre-commit hooks
```bash
pnpm install
pnpm run prepare # Sets up Husky pre-commit hooks for code formatting
```
4. Create an env file
```bash
cp .env.dev.example .env
```
4. Install dependencies
5. Run the entire infrastructure in dev mode. **Note**: if you have an existing database, this command wipes it. Also, this will fail on the very first run. Please run it again.
```bash
pnpm install
pnpm run dx # first run only (resets db, docker containers, etc...)
pnpm run dev # any subsequent runs
```
5. Run the migrations
You will be asked whether you want to reset Postgres and ClickHouse. Confirm both with 'Y' and press enter.
All database migrations and configs are in the `shared` package.
6. Open the web app in your browser to start using Langfuse:
- [Sign up page, http://localhost:3000](http://localhost:3000)
To get comprehensive example data, you can use the `seed` command:
6. Start the development server
```bash
pnpm run dev
```
7. Open the web app in the browser:
http://localhost:3000
8. Log in as a test user (after you ran `db:seed` command):
Username: demo@langfuse.com
Password: password
```sh
pnpm run db:seed:examples
```
## Monorepo quickstart
@@ -302,23 +217,67 @@ Requirements
On the main branch, we adhere to the best practices of [conventional commits](https://www.conventionalcommits.org/en/v1.0.0/). All pull requests and branches are squash-merged to maintain a clean and readable history. This approach ensures the addition of a conventional commit message when merging contributions.
## Test the public API
## Running Unit Tests
The API is tested using Jest. With the development server running, you can run the tests with:
All tests run in the CI and must pass before merging.
All tests run against a running langfuse instance and **write/delete real data from the database**.
Run all
### Test Database Setup
Per default, the tests use the local development database. Therefore, wiping your data in the process.
For proper test isolation, create a `.env.test` file in the root directory:
```bash
npm run test
cp .env.test.example .env.test
```
Run interactively in watch mode
Then, a different PostgreSQL and Redis are used for the tests.
The `.env.test` file only overrides the set values and falls back on `.env` for all undefined values.
```bash
npm run test:watch
- **PostgreSQL**: Uses separate `langfuse_test` database for isolation
- **ClickHouse**: Uses shared `default` database for now
- **Redis**: Uses database 1 instead of 0 for isolation (Redis data is not cleaned between tests)
Tests automatically create the PostgreSQL test database if it doesn't exist and clean up data between runs.
### Tests in the `web` package (public API)
We're using Jest with in the `web` package. Therefore, if you want to provide an argument to the test runner, do it directly without an intermittent `--`.
There are three types of unit tests:
- `test-sync`
- `test` (for async folder tests)
- `test-client`
To run a specific test, for example the test: `"should handle special characters in prompt names"` in `prompts.v2.servertest.ts`, run:
```sh
cd web # or with --filter=web
pnpm test-sync --testPathPattern="prompts\.v2\.servertest" --testNamePattern="should handle special characters in prompt names"
# for async folder tests:
pnpm test -- --testPathPattern="observations-api" --testNamePattern="should fetch all observations"
```
These tests are also run in CI.
To run all tests:
```sh
pnpm run test
```
Run interactively in watch mode (not recommended!)
```sh
pnpm run test:watch
```
### Tests in the `worker` package
For the `worker` package, we're using `vitest` to run unit tests.
```sh
pnpm run test --filter=worker -- FILE_YOU_WANT_TO_TEST.ts -t "test name"
```
## CI/CD
@@ -397,7 +356,7 @@ The background color of the following component will be `hsl(var(--primary))` an
| --primary-accent | Primary accent color used for branding | Layout |
| --hover-primary-accent | Primary accent color used for hover effects for links | SignIn and AuthCloudRegionSwitch |
| --muted-green | Muted green for Event label | ObservationTree |
| --muted-blue | Muted blue for Span label | ObservationTree |
| --muted-gray | Muted gray for disabled status badges | StatusBadge |
| --accent-light-green | Light green accent for background of output and assistant messages | IOPreview, Generations, Traces |
@@ -439,15 +398,30 @@ You can update the default AI models and prices by adding or updating an entry i
Please note that
- prices are in USD
- the list is ordered by ID, so make sure to keep this order
- the list is ordered by ID, so make sure to keep this order and insert new models at the end of the list
- the `updated_at` field must be updated with the current date in ISO 8601 format. Otherwise, the change will be ignored.
### Transition period until V3 release
Until the V3 release, both the JSON record must be updated **and** a migration must be created to continue supporting self-hosted users. Note that the migration must updated both the `models` as well as the `prices` table accordingly.
## Updating the OpenAPI Specs & fern SDKs
We maintain the API specifications manually to guarantee a high degree of understandability. If you made changes to the API, please update the respective `.yml` files in `fern/apis/...`.
To generate the respective `openapi.yml` files which power the online API reference & SDKs, run:
```sh
npx fern-api generate --api server # for the server API
npx fern-api generate --api client # for the client API
npx fern-api generate --api organizations # for the organizations API
```
**Note:** You need a signed in fern account to run those commands.
## License
Langfuse is MIT licensed, except for `ee/` folder. See [LICENSE](LICENSE) and [docs](https://langfuse.com/docs/open-source) for more details.
When contributing to the Langfuse codebase, you need to agree to the [Contributor License Agreement](https://cla-assistant.io/langfuse/langfuse). You only need to do this once and the CLA bot will remind you if you haven't signed it yet.
Portions of this software are licensed as follows:
- All content that resides under the "ee/" and/or "web/src/ee" directories of this repository, if these directories exist, is licensed under the license defined in "ee/LICENSE".
- All third party components incorporated into the Finto Technologies Software are licensed under the original license provided by the owner of the applicable component.
- All content that resides under the "ee/", "web/src/ee/", and/or "worker/src/ee/" directories of this repository, if these directories exist, is licensed under the license defined in "ee/LICENSE".
- All third party components incorporated into the Langfuse Software are licensed under the original license provided by the owner of the applicable component.
- Content outside of the above mentioned directories or restrictions above is available under the "MIT Expat" license as defined below.
Permission is hereby granted, free of charge, to any person obtaining a copy
앱에 계측(instrumentation)을 추가하여 Langfuse로 trace 데이터를 수집함으로써, 검색, 임베딩, 또는 에이전트 동작과 같은 LLM 호출 및 기타 관련 로직을 추적할 수 있습니다. 복잡한 로그와 사용자 세션을 확인 및 디버깅 해보세요. 인터랙티브 데모를 통해 실제 작동 예를 확인할 수 있습니다.
- **프롬프트 관리**
프롬프트를 중앙에서 관리하고 버전 관리하며 협업으로 수정할 수 있도록 도와줍니다. 서버와 클라이언트 측의 강력한 캐싱 덕분에 애플리케이션에 지연(latency)을 추가하지 않고도 프롬프트를 반복 개선할 수 있습니다.
- **평가**
LLM 애플리케이션 개발 워크플로우에서 핵심적인 역할을 하며, Langfuse는 여러분의 필요에 맞게 유연하게 대응합니다. LLM을 심사자로 활용하는 기능, 사용자 피드백 수집, 수동 라벨링 및 API/SDK를 통한 맞춤 평가 파이프라인을 지원합니다.
- **데이터셋**
LLM 애플리케이션 평가를 위한 테스트 세트와 벤치마크를 제공하여, 지속적인 개선, 배포 전 테스트, 구조화된 실험, 유연한 평가 및 LangChain과 LlamaIndex와 같은 프레임워크와의 원활한 통합을 지원합니다.
- **LLM 플레이그라운드**
프롬프트와 모델 구성에 대해 테스트 및 반복 개선할 수 있는 도구로, 피드백 루프를 단축하여 개발 속도를 높여줍니다. trace에서 이상한 결과가 발생하면 플레이그라운드로 바로 이동해 개선할 수 있습니다.
- **종합 API**
Langfuse는 API를 통해 제공되는 구성 요소들을 활용하여 맞춤형 LLMOps 워크플로우를 강화하는 데 자주 사용됩니다. OpenAPI 명세, Postman 컬렉션, Python 및 JS/TS용 타입드 SDK가 제공됩니다.
| [LlamaIndex](https://langfuse.com/docs/integrations/llama-index/get-started) | Python | LlamaIndex 콜백 시스템을 통한 자동 계측을 지원합니다. |
| [Haystack](https://langfuse.com/docs/integrations/haystack) | Python | Haystack 콘텐츠 추적 시스템을 통한 자동 계측을 지원합니다. |
| [LiteLLM](https://langfuse.com/docs/integrations/litellm) | Python, JS/TS (proxy only) | GPT의 드롭인 대체품으로 어떤 LLM도 사용할 수 있습니다. Azure, OpenAI, Cohere, Anthropic, Ollama, VLLM, Sagemaker, HuggingFace, Replicate 등 100개 이상의 LLM 지원. |
| [Vercel AI SDK](https://langfuse.com/docs/integrations/vercel-ai-sdk) | JS/TS | React, Next.js, Vue, Svelte, Node.js와 함께 AI 기반 애플리케이션 구축을 돕는 TypeScript 툴킷입니다. |
| [API](https://langfuse.com/docs/api) | | 공개 API를 직접 호출합니다. OpenAPI 명세가 제공됩니다. |
| [Ollama](https://langfuse.com/docs/integrations/ollama) | 모델 (로컬) | 자신의 컴퓨터에서 오픈 소스 LLM을 손쉽게 실행할 수 있습니다. |
| [Amazon Bedrock](https://langfuse.com/docs/integrations/amazon-bedrock) | 모델 | AWS에서 기본 및 파인튜닝된 모델을 실행합니다. |
| [Google VertexAI and Gemini](https://langfuse.com/docs/integrations/google-vertex-ai) | 모델 | Google에서 기본 및 파인튜닝된 모델을 실행합니다. |
| [AutoGen](https://langfuse.com/docs/integrations/autogen) | 에이전트 프레임워크 | 분산 에이전트 구축을 위한 오픈 소스 LLM 플랫폼입니다. |
| [Flowise](https://langfuse.com/docs/integrations/flowise) | 채팅/에이전트 UI | 맞춤형 LLM 플로우를 위한 JS/TS 코드 없는(no-code) 빌더입니다. |
| [Langflow](https://langfuse.com/docs/integrations/langflow) | 채팅/에이전트 UI | react-flow를 활용하여 실험 및 프로토타이핑을 손쉽게 할 수 있도록 디자인된 LangChain용 Python 기반 UI입니다. |
| [Dify](https://langfuse.com/docs/integrations/dify) | 채팅/에이전트 UI | 코드 없는 빌더와 함께 제공되는 오픈 소스 LLM 애플리케이션 개발 플랫폼입니다. |
| [OpenWebUI](https://langfuse.com/docs/integrations/openwebui) | 채팅/에이전트 UI | 셀프 호스팅 및 로컬 모델 등 다양한 LLM 실행기를 지원하는 셀프 호스팅 LLM 채팅 웹 UI입니다. |
| [Promptfoo](https://langfuse.com/docs/integrations/promptfoo) | 도구 | 오픈 소스 LLM 테스트 플랫폼입니다. |
| [LobeChat](https://langfuse.com/docs/integrations/lobechat) | 채팅/에이전트 UI | 오픈 소스 챗봇 플랫폼입니다. |
| [Vapi](https://langfuse.com/docs/integrations/vapi) | 플랫폼 | 오픈 소스 음성 AI 플랫폼입니다. |
| [Inferable](https://langfuse.com/docs/integrations/other/inferable) | 에이전트 | 분산 에이전트 구축을 위한 오픈 소스 LLM 플랫폼입니다. |
| [Gradio](https://langfuse.com/docs/integrations/other/gradio) | 채팅/에이전트 UI | 채팅 UI와 같은 웹 인터페이스 구축을 위한 오픈 소스 Python 라이브러리입니다. |
| [Goose](https://langfuse.com/docs/integrations/goose) | 에이전트 | 분산 에이전트 구축을 위한 오픈 소스 LLM 플랫폼입니다. |
| [smolagents](https://langfuse.com/docs/integrations/smolagents) | 에이전트 | 오픈 소스 AI 에이전트 프레임워크입니다. |
| [CrewAI](https://langfuse.com/docs/integrations/crewai) | 에이전트 | 에이전트 간 협업 및 도구 사용을 위한 다중 에이전트 프레임워크입니다. |
## 🚀 빠른 시작
앱에 계측을 추가하고 Langfuse에 trace 데이터를 수집하여, LLM 호출 및 검색, 임베딩, 에이전트 동작과 같은 애플리케이션 로직을 추적해보세요. 복잡한 로그와 사용자 세션을 확인하여 디버깅할 수 있습니다.
### 1️⃣ 새 프로젝트 생성
1. [Langfuse 계정 생성](https://cloud.langfuse.com/auth/sign-up) 또는 [셀프 호스트](https://langfuse.com/self-hosting)
2. 새 프로젝트를 생성합니다.
3. 프로젝트 설정에서 새로운 API 자격 증명을 생성합니다.
### 2️⃣ 첫 번째 LLM 호출 기록하기
[`@observe()` 데코레이터](https://langfuse.com/docs/sdk/python/decorators)를 사용하면 Python LLM 애플리케이션의 추적이 매우 간편해집니다. 이 빠른 시작 예제에서는 Langfuse [OpenAI 통합](https://langfuse.com/integrations/model-providers/openai-py)을 사용하여 모든 모델 파라미터를 자동으로 캡처합니다.
> [!TIP]
> OpenAI를 사용하지 않으시다면, 다른 모델 및 프레임워크의 로그 기록 방법은 [문서](https://langfuse.com/docs/get-started#log-your-first-llm-call-to-langfuse)를 참조하세요.
```bash
pip install langfuse openai
```
```bash filename=".env"
LANGFUSE_SECRET_KEY="sk-lf-..."
LANGFUSE_PUBLIC_KEY="pk-lf-..."
LANGFUSE_HOST="https://cloud.langfuse.com" # 🇪🇺 EU region
# LANGFUSE_HOST="https://us.cloud.langfuse.com" # 🇺🇸 US region
```
```python:main.py
from langfuse import observe
from langfuse.openai import openai # OpenAI integration
@observe()
def story():
return openai.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": "What is Langfuse?"}],
).choices[0].message.content
@observe()
def main():
return story()
main()
```
### 3️⃣ Langfuse에서 trace 확인하기
Langfuse에서 LLM 호출 및 애플리케이션의 기타 로직에 대한 trace를 확인할 수 있습니다.

_[Langfuse의 공개 예제 trace](https://cloud.langfuse.com/project/cloramnkj0002jz088vzn1ja4/traces/2cec01e3-3dc2-472f-afcf-3b968cf0c1f4?timestamp=2025-02-10T14%3A27%3A30.275Z&observation=cb5ff844-07ef-41e6-b8e2-6c64344bc13b)_
> [!TIP]
>
> Langfuse의 trace에 대해 더 알아보거나 [인터랙티브 데모](https://langfuse.com/docs/demo)에서 직접 체험해보세요.
- 우리의 [문서](https://langfuse.com/docs)는 답을 찾기 위한 최적의 장소입니다. 문서가 매우 포괄적이며, 유지보수에 많은 노력을 기울이고 있습니다. GitHub를 통해 문서 수정 제안도 가능합니다.
- [Langfuse FAQ](https://langfuse.com/faq)에서는 가장 흔한 질문에 대해 답변하고 있습니다.
- 질문에 즉각적인 답변이 필요하다면 [Ask AI](https://langfuse.com/docs/ask-ai)를 사용해보세요.
지원 채널:
- **GitHub Discussions의 [공개 Q&A](https://github.com/orgs/langfuse/discussions/categories/support)** 에 질문을 남겨주세요. 가능한 한 많은 세부 사항(예: 코드 스니펫, 스크린샷, 배경 정보)을 포함해 질문해 주시기 바랍니다.
<span>Langfuse uses <a href="https://github.com/orgs/langfuse/discussions"><strong>Github Discussions</strong></a> for Support and Feature Requests.</span>
<br/>
<span>We're hiring. <a href="https://langfuse.com/careers"><strong>Join us</strong></a> in Product Engineering and Developer Relations.</span>
<span>Langfuse uses <a href="https://github.com/orgs/langfuse/discussions"><strong>GitHub Discussions</strong></a> for Support and Feature Requests.</span>
<br/>
<span><b>We're hiring.</b> <a href="https://langfuse.com/careers"><strong>Join us</strong></a> in product engineering and technical go-to-market roles.</span>
- **LLM Observability:** Instrument your app and start ingesting traces to Langfuse ([Quickstart](https://langfuse.com/docs/get-started), [Integrations](https://langfuse.com/docs/integrations) [Tracing](https://langfuse.com/docs/tracing))
- **Langfuse UI:** Inspect and debug complex logs ([Demo](https://langfuse.com/docs/demo), [Tracing](https://langfuse.com/docs/tracing))
- **Prompt Management:** Manage, version and deploy prompts from within Langfuse ([Prompt Management](https://langfuse.com/docs/prompts/get-started))
- **Prompt Engineering:** Test and iterate on your prompts with the [LLM Playground](https://langfuse.com/docs/playground)
-**LLM Analytics:** Track metrics (cost, latency, quality) and gain insights from dashboards & data exports ([Analytics](https://langfuse.com/docs/analytics))
- **LLM Evaluations:** Collect and calculate scores for your LLM completions ([Scores & Evaluations](https://langfuse.com/docs/scores))
- Run ([Model-based evaluations](https://langfuse.com/docs/scores/model-based-evals)) and LLM-as-a-Judge within Langfuse
- Collect user feedback ([User Feedback](https://langfuse.com/docs/scores/user-feedback))
- Manually score LLM outputs in Langfuse ([Manual Scores](https://langfuse.com/docs/scores/manually))
-[LLM Application Observability](https://langfuse.com/docs/tracing): Instrument your app and start ingesting traces to Langfuse, thereby tracking LLM calls and other relevant logic in your app such as retrieval, embedding, or agent actions. Inspect and debug complex logs and user sessions. Try the interactive [demo](https://langfuse.com/docs/demo) to see this in action.
### Test
- [Prompt Management](https://langfuse.com/docs/prompt-management/get-started) helps you centrally manage, version control, and collaboratively iterate on your prompts. Thanks to strong caching on server and client side, you can iterate on prompts without adding latency to your application.
-**Experiments:** Track and test app behaviour before deploying a new version
- Datasets let you test expected in and output pairs and benchmark performance before deploying ([Datasets](https://langfuse.com/docs/datasets))
- Track versions and releases in your application ([Experimentation](https://langfuse.com/docs/experimentation), [Prompt Management](https://langfuse.com/docs/prompts))
-[Evaluations](https://langfuse.com/docs/evaluation/overview) are key to the LLM application development workflow, and Langfuse adapts to your needs. It supports LLM-as-a-judge, user feedback collection, manual labeling, and custom evaluation pipelines via APIs/SDKs.
## Get started
- [Datasets](https://langfuse.com/docs/evaluation/dataset-runs/datasets) enable test sets and benchmarks for evaluating your LLM application. They support continuous improvement, pre-deployment testing, structured experiments, flexible evaluation, and seamless integration with frameworks like LangChain and LlamaIndex.
- [LLM Playground](https://langfuse.com/docs/playground) is a tool for testing and iterating on your prompts and model configurations, shortening the feedback loop and accelerating development. When you see a bad result in tracing, you can directly jump to the playground to iterate on it.
- [Comprehensive API](https://langfuse.com/docs/api): Langfuse is frequently used to power bespoke LLMOps workflows while using the building blocks provided by Langfuse via the API. OpenAPI spec, Postman collection, and typed SDKs for Python, JS/TS are available.
[→ Learn more about deploying locally](https://langfuse.com/docs/deployment/local)
# Run the langfuse docker compose
docker compose up
```
### Self-host (docker)
- [VM](https://langfuse.com/self-hosting/docker-compose): Run Langfuse on a single Virtual Machine using Docker Compose.
- [Kubernetes (Helm)](https://langfuse.com/self-hosting/kubernetes-helm): Run Langfuse on a Kubernetes cluster using Helm. This is the preferred production deployment.
You need a Langfuse public and secret key to get started. Sign up [here](https://cloud.langfuse.com) and find them in your project settings.
### Ingesting Data · Instrumenting Your Application · LLM Observability with Langfuse
Note: We recommend using our fully async, typed [SDKs](https://langfuse.com/docs/sdk) that allow you to instrument any LLM application with any underlying model. They are available in [Python (Decorators)](https://langfuse.com/docs/sdk/python) & [JS/TS](https://langfuse.com/docs/sdk/typescript). The SDKs will always be the most fully featured and stable way to ingest data into Langfuse.
See the [→ Quickstart](https://langfuse.com/docs/get-started) to integrate Langfuse.
@@ -122,50 +144,233 @@ See the [→ Quickstart](https://langfuse.com/docs/get-started) to integrate Lan
| [Vercel AI SDK](https://langfuse.com/docs/integrations/vercel-ai-sdk) | JS/TS | TypeScript toolkit designed to help developers build AI-powered applications with React, Next.js, Vue, Svelte, Node.js. |
| [API](https://langfuse.com/docs/api) | | Directly call the public API. OpenAPI spec available. |
| [Langflow](https://langfuse.com/docs/integrations/langflow) | Python-based UI for LangChain, designed with react-flow to provide an effortless way to experiment and prototype flows. |
| [Langflow](https://langfuse.com/docs/integrations/langflow) | Chat/Agent UI | Python-based UI for LangChain, designed with react-flow to provide an effortless way to experiment and prototype flows. |
| [Dify](https://langfuse.com/docs/integrations/dify) | Chat/Agent UI | Open source LLM app development platform with no-code builder. |
| [OpenWebUI](https://langfuse.com/docs/integrations/openwebui) | Chat/Agent UI | Self-hosted LLM Chat web ui supporting various LLM runners including self-hosted and local models. |
| [LobeChat](https://langfuse.com/docs/integrations/lobechat) | Chat/Agent UI | Open source chatbot platform. |
| [Vapi](https://langfuse.com/docs/integrations/vapi) | Platform | Open source voice AI platform. |
| [Inferable](https://langfuse.com/docs/integrations/other/inferable) | Agents | Open source LLM platform for building distributed agents. |
| [Gradio](https://langfuse.com/docs/integrations/other/gradio) | Chat/Agent UI | Open source Python library to build web interfaces like Chat UI. |
| [Goose](https://langfuse.com/docs/integrations/goose) | Agents | Open source LLM platform for building distributed agents. |
| [smolagents](https://langfuse.com/docs/integrations/smolagents) | Agents | Open source AI agents framework. |
| [CrewAI](https://langfuse.com/docs/integrations/crewai) | Agents | Multi agent framework for agent collaboration and tool use. |
## Questions and feedback
## 🚀 Quickstart
### Ideas and roadmap
Instrument your app and start ingesting traces to Langfuse, thereby tracking LLM calls and other relevant logic in your app such as retrieval, embedding, or agent actions. Inspect and debug complex logs and user sessions.
- [Discord](https://langfuse.com/discord): community support
- Privately: contact at langfuse dot com
The [`@observe()` decorator](https://langfuse.com/docs/sdk/python/decorators) makes it easy to trace any Python LLM application. In this quickstart we also use the Langfuse [OpenAI integration](https://langfuse.com/integrations/model-providers/openai-py) to automatically capture all model parameters.
## Contributing to Langfuse
> [!TIP]
> Not using OpenAI? Visit [our documentation](https://langfuse.com/docs/get-started#log-your-first-llm-call-to-langfuse) to learn how to log other models and frameworks.
- Vote on [Ideas](https://github.com/orgs/langfuse/discussions/categories/ideas)
- Raise and comment on [Issues](https://github.com/langfuse/langfuse/issues)
```bash
pip install langfuse openai
```
```bash filename=".env"
LANGFUSE_SECRET_KEY="sk-lf-..."
LANGFUSE_PUBLIC_KEY="pk-lf-..."
LANGFUSE_HOST="https://cloud.langfuse.com" # 🇪🇺 EU region
# LANGFUSE_HOST="https://us.cloud.langfuse.com" # 🇺🇸 US region
from langfuse.openai import openai # OpenAI integration
@observe()
def story():
return openai.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": "What is Langfuse?"}],
).choices[0].message.content
@observe()
def main():
return story()
main()
```
### 3️⃣ See traces in Langfuse
See your language model calls and other application logic in Langfuse.
<img width="1787" height="674" alt="Example trace in Langfuse" src="https://github.com/user-attachments/assets/f796eb78-dfb5-4570-b236-bdb4b67d4d55" />
_[Public example trace in Langfuse](https://cloud.langfuse.com/project/cloramnkj0002jz088vzn1ja4/traces/2cec01e3-3dc2-472f-afcf-3b968cf0c1f4?timestamp=2025-02-10T14%3A27%3A30.275Z&observation=cb5ff844-07ef-41e6-b8e2-6c64344bc13b)_
> [!TIP]
>
> [Learn more](https://langfuse.com/docs/tracing) about tracing in Langfuse or play with the [interactive demo](https://langfuse.com/docs/demo).
- Our [documentation](https://langfuse.com/docs) is the best place to start looking for answers. It is comprehensive, and we invest significant time into maintaining it. You can also suggest edits to the docs via GitHub.
- [Langfuse FAQs](https://langfuse.com/faq) where the most common questions are answered.
- Use "[Ask AI](https://langfuse.com/docs/ask-ai)" to get instant answers to your questions.
Support Channels:
- **Ask any question in our [public Q&A](https://github.com/orgs/langfuse/discussions/categories/support) on GitHub Discussions.** Please include as much detail as possible (e.g. code snippets, screenshots, background information) to help us understand your question.
- [Request a feature](https://github.com/orgs/langfuse/discussions/categories/ideas) on GitHub Discussions.
- [Report a Bug](https://github.com/langfuse/langfuse/issues) on GitHub Issues.
- For time-sensitive queries, ping us via the in-app chat widget.
## 🤝 Contributing
Your contributions are welcome!
- Vote on [Ideas](https://github.com/orgs/langfuse/discussions/categories/ideas) in GitHub Discussions.
- Raise and comment on [Issues](https://github.com/langfuse/langfuse/issues).
- Open a PR - see [CONTRIBUTING.md](CONTRIBUTING.md) for details on how to setup a development environment.
## License
## 🥇 License
This repository is MIT licensed, except for the `ee` folders. See [LICENSE](LICENSE) and [docs](https://langfuse.com/docs/open-source) for more details.
<img alt="Star History Chart" src="https://api.star-history.com/svg?repos=langfuse/langfuse&type=Date" style="border-radius: 15px;" />
</picture>
</a>
[**GET routes**](https://langfuse.com/docs/integrations/api) to use data in downstream applications (e.g. embedded analytics). You can also access them conveniently via the SDKs ([docs](https://langfuse.com/docs/query-traces)).
## ❤️ Open Source Projects Using Langfuse
### Security & Privacy
Top open-source Python projects that use Langfuse, ranked by stars ([Source](https://github.com/langfuse/langfuse-docs/blob/main/components-mdx/dependents)):
We take data security and privacy seriously. Please refer to our [Security and Privacy](https://langfuse.com/security) page for more information.
@@ -181,13 +386,3 @@ This helps us to:
None of the data is shared with third parties and does not include any sensitive information. We want to be super transparent about this and you can find the exact data we collect [here](/web/src/features/telemetry/index.ts).
You can opt-out by setting `TELEMETRY_ENABLED=false`.
- ClickHouse migrations in the `packages/shared/clickhouse/migrations/clustered` directory should include `ON CLUSTER default` and should use `Replicated` merge tree table types.
- E.g. `ReplacingMergeTree` is likely an error while `ReplicatedReplacingMergeTree` would be correct in most cases.
- ClickHouse migrations in the `packages/shared/clickhouse/migrations/unclustered` directory must not include `ON CLUSTER` statements and must not use `Replicated` merge tree table types.
- Migrations in `packages/shared/clickhouse/migrations/clustered` should match their counterparts in `packages/shared/clickhouse/migrations/unclustered` aside from the restrictions listed above.
- When adding new indexes on ClickHouse, ensure that there is a corresponding `MATERIALIZE INDEX` statement in the same migration. The materialization can use `SETTINGS mutations_sync = 2` if they operate on smaller tables, but may timeout otherwise.
### Postgres
- Most `schema.prisma` changes should produce a change in `packages/shared/prisma/migrations`.
### Environment Variables
- Environment variables should be imported from the `env.mjs/ts` file of the respective package and not from `process.env.*` to ensure validation and typing.
## Redis Invocations
- Highlight usage of `redis.call` invocations. Those may have suboptimal redis cluster routing and will raise errors. Instead, use the native call patterns.
Example: `await redis?.call("SET", key, "1", "NX", "EX", TTLSeconds);` should use `await redis?.set(key, "1", "EX", TTLSeconds, "NX");` instead.
## Langfuse Cloud
- When attempting to confirm if the current environment is Langfuse Cloud in the frontend, use the `useLangfuseCloudRegion` hook and never environment variables directly.
## Banner Height System
- Use `top-banner-offset` instead of `top-0` for any elements that are positioned `sticky`, `fixed`, or `absolute` with a global reference point (e.g., `top-0`). This ensures proper spacing when system banners (payment, maintenance, etc.) are displayed.
- The banner height is managed through CSS variables (`--banner-height` and `--banner-offset`) defined in `web/src/styles/globals.css`.
- Banner components (like PaymentBanner) dynamically update `--banner-height` using ResizeObserver to track their actual height, ensuring accurate positioning even when banners resize (e.g., on mobile wrapping).
- Available Tailwind utilities:
-`top-banner-offset` / `pt-banner-offset` - For sticky/fixed/absolute positioning and padding
-`h-screen-with-banner` / `min-h-screen-with-banner` - For full-height containers accounting for banners
docs:The value of the score. Must be passed as string for categorical scores, and numeric for boolean and numeric scores. Boolean score values must equal either 1 or 0 (true or false)
observationId:optional<string>
comment:optional<string>
metadata:optional<unknown>
dataType:
type:optional<ScoreDataType>
docs:When set, must match the score value's type. If not set, will be inferred from the score value or config
@@ -64,62 +67,15 @@ types:
dataType:"BOOLEAN"
configId:"1234-5678-90ab-cdef"
traceId:"cdef-1234-5678-90ab"
BaseScore:
properties:
id:string
traceId:string
name:string
source:ScoreSource
observationId:optional<string>
timestamp:datetime
createdAt:datetime
updatedAt:datetime
authorUserId:optional<string>
comment:optional<string>
configId:
type:optional<string>
docs:Reference a score config on a score. When set, config and score name must be equal and value must comply to optionally defined numerical range
NumericScore:
extends:BaseScore
properties:
value:
type:double
docs:The numeric value of the score
BooleanScore:
extends:BaseScore
properties:
value:
type:double
docs:The numeric value of the score. Equals 1 for "True" and 0 for "False"
stringValue:
type:string
docs:The string representation of the score value. Is inferred from the numeric value and equals "True" or "False"
CategoricalScore:
extends:BaseScore
properties:
value:
type:optional<double>
docs:Only defined if a config is linked. Represents the numeric category mapping of the stringValue
stringValue:
type:string
docs:The string representation of the score value. If no config is linked, can be any string. Otherwise, must map to a config category
docs:Get all blob storage integrations for the organization (requires organization-scoped API key)
method:GET
path:""
response:BlobStorageIntegrationsResponse
upsertBlobStorageIntegration:
docs:Create or update a blob storage integration for a specific project (requires organization-scoped API key). The configuration is validated by performing a test upload to the bucket.
method:PUT
path:""
request:CreateBlobStorageIntegrationRequest
response:BlobStorageIntegrationResponse
deleteBlobStorageIntegration:
docs:Delete a blob storage integration by ID (requires organization-scoped API key)
method:DELETE
path:"/{id}"
path-parameters:
id:string
response:BlobStorageIntegrationDeletionResponse
types:
BlobStorageIntegrationType:
enum:
- S3
- S3_COMPATIBLE
- AZURE_BLOB_STORAGE
BlobStorageIntegrationFileType:
enum:
- JSON
- CSV
- JSONL
BlobStorageExportMode:
enum:
- FULL_HISTORY
- FROM_TODAY
- FROM_CUSTOM_DATE
BlobStorageExportFrequency:
enum:
- hourly
- daily
- weekly
CreateBlobStorageIntegrationRequest:
properties:
projectId:
type:string
docs:ID of the project in which to configure the blob storage integration
type:BlobStorageIntegrationType
bucketName:
type:string
docs:Name of the storage bucket
endpoint:
type:optional<string>
docs:Custom endpoint URL (required for S3_COMPATIBLE type)
region:
type:string
docs:Storage region
accessKeyId:
type:optional<string>
docs:Access key ID for authentication
secretAccessKey:
type:optional<string>
docs:Secret access key for authentication (will be encrypted when stored)
prefix:
type:optional<string>
docs:Path prefix for exported files (must end with forward slash if provided)
exportFrequency:BlobStorageExportFrequency
enabled:
type:boolean
docs:Whether the integration is active
forcePathStyle:
type:boolean
docs:Use path-style URLs for S3 requests
fileType:BlobStorageIntegrationFileType
exportMode:BlobStorageExportMode
exportStartDate:
type:optional<datetime>
docs:Custom start date for exports (required when exportMode is FROM_CUSTOM_DATE)
docs:Public traces are accessible via url without login
environment:
type:optional<string>
docs:The environment from which this trace originated. Can be any lowercase alphanumeric string with hyphens and underscores that does not start with 'langfuse'.
TraceWithDetails:# GET /traces
extends:Trace
properties:
@@ -72,13 +75,16 @@ types:
type:list<ObservationsView>
docs:List of observations
scores:
type:list<Score>
type:list<ScoreV1>
docs:List of scores
Session:
properties:
id:string
createdAt:datetime
projectId:string
environment:
type:optional<string>
docs:The environment from which this session originated.
SessionWithTraces:
extends:Session
properties:
@@ -126,7 +132,7 @@ types:
docs:The output data of the observation
usage:
type:optional<Usage>
docs:The usage data of the observation
docs:(Deprecated. Use usageDetails and costDetails instead.) The usage data of the observation
level:
type:ObservationLevel
docs:The level of the observation
@@ -139,6 +145,15 @@ types:
promptId:
type:optional<string>
docs:The prompt ID associated with the observation
usageDetails:
type:optional<map<string, integer>>
docs:The usage details of the observation. Key is the name of the usage metric, value is the number of units consumed. The total key is the sum of all (non-total) usage metrics or the total value ingested.
costDetails:
type:optional<map<string, double>>
docs:The cost details of the observation. Key is the name of the cost metric, value is the cost in USD. The total key is the sum of all (non-total) cost metrics or the total value ingested.
environment:
type:optional<string>
docs:The environment from which this observation originated. Can be any lowercase alphanumeric string with hyphens and underscores that does not start with 'langfuse'.
ObservationsView:
extends:Observation
@@ -163,13 +178,13 @@ types:
docs:The total price in USD.
calculatedInputCost:
type:optional<double>
docs:The calculated cost of the input in USD
docs:(Deprecated. Use usageDetails and costDetails instead.) The calculated cost of the input in USD
calculatedOutputCost:
type:optional<double>
docs:The calculated cost of the output in USD
docs:(Deprecated. Use usageDetails and costDetails instead.) The calculated cost of the output in USD
calculatedTotalCost:
type:optional<double>
docs:The calculated total cost in USD
docs:(Deprecated. Use usageDetails and costDetails instead.) The calculated total cost in USD
latency:
type:optional<double>
docs:The latency in seconds.
@@ -178,7 +193,7 @@ types:
docs:The time to the first token in seconds
Usage:
docs:Standard interface for usage and cost
docs:(Deprecated. Use usageDetails and costDetails instead.) Standard interface for usage and cost
properties:
input:
docs:Number of input units (e.g. tokens)
@@ -225,7 +240,7 @@ types:
properties:
value:double
label:string
BaseScore:
BaseScoreV1:
properties:
id:string
traceId:string
@@ -237,12 +252,77 @@ types:
updatedAt:datetime
authorUserId:optional<string>
comment:optional<string>
metadata:optional<unknown>
configId:
type:optional<string>
docs:Reference a score config on a score. When set, config and score name must be equal and value must comply to optionally defined numerical range
queueId:
type:optional<string>
docs:Reference an annotation queue on a score. Populated if the score was initially created in an annotation queue.
docs:The annotation queue referenced by the score. Indicates if score was initially created while processing annotation queue.
environment:
type:optional<string>
docs:The environment from which this score originated. Can be any lowercase alphanumeric string with hyphens and underscores that does not start with 'langfuse'.
NumericScoreV1:
extends:BaseScoreV1
properties:
value:
type:double
docs:The numeric value of the score
BooleanScoreV1:
extends:BaseScoreV1
properties:
value:
type:double
docs:The numeric value of the score. Equals 1 for "True" and 0 for "False"
stringValue:
type:string
docs:The string representation of the score value. Is inferred from the numeric value and equals "True" or "False"
CategoricalScoreV1:
extends:BaseScoreV1
properties:
value:
type:optional<double>
docs:Only defined if a config is linked. Represents the numeric category mapping of the stringValue
stringValue:
type:string
docs:The string representation of the score value. If no config is linked, can be any string. Otherwise, must map to a config category
ScoreV1:
discriminant:"dataType"
union:
NUMERIC:
type:NumericScoreV1
docs:"Score with NUMERIC data type"
CATEGORICAL:
type:CategoricalScoreV1
docs:"Score with CATEGORICAL data type"
BOOLEAN:
type:BooleanScoreV1
docs:"Score with BOOLEAN data type"
BaseScore:
properties:
id:string
traceId:optional<string>
sessionId:optional<string>
observationId:optional<string>
datasetRunId:optional<string>
name:string
source:ScoreSource
timestamp:datetime
createdAt:datetime
updatedAt:datetime
authorUserId:optional<string>
comment:optional<string>
metadata:optional<unknown>
configId:
type:optional<string>
docs:Reference a score config on a score. When set, config and score name must be equal and value must comply to optionally defined numerical range
queueId:
type:optional<string>
docs:The annotation queue referenced by the score. Indicates if score was initially created while processing annotation queue.
environment:
type:optional<string>
docs:The environment from which this score originated. Can be any lowercase alphanumeric string with hyphens and underscores that does not start with 'langfuse'.
NumericScore:
extends:BaseScore
properties:
@@ -372,18 +452,18 @@ types:
type:string
startDate:
docs:Apply only to generations which are newer than this ISO date.
type:optional<date>
type:optional<datetime>
unit:
docs:Unit used by this model.
type:ModelUsageUnit
type:optional<ModelUsageUnit>
inputPrice:
docs:Price (USD) per input unit
docs:Deprecated. See 'prices' instead. Price (USD) per input unit
type:optional<double>
outputPrice:
docs:Price (USD) per output unit
docs:Deprecated. See 'prices' instead. Price (USD) per output unit
type:optional<double>
totalPrice:
docs:Price (USD) per total unit. Cannot be set if input or output price is set.
docs:Deprecated. See 'prices' instead. Price (USD) per total unit. Cannot be set if input or output price is set.
type:optional<double>
tokenizerId:
docs:Optional. Tokenizer to be applied to observations which match to this model. See docs for more details.
message:"Use the OpenTelemetry endpoint at /api/public/otel/v1/traces instead. Learn more: https://langfuse.com/integrations/native/opentelemetry"
docs:|
Batched ingestion for Langfuse Tracing. If you want to use tracing via the API, such as to build your own Langfuse client implementation, this is the only API route you need to implement.
**Legacy endpoint for batch ingestion for Langfuse Observability.**
-> Please use the OpenTelemetry endpoint (`/api/public/otel/v1/traces`). Learn more: https://langfuse.com/integrations/native/opentelemetry
Within each batch, there can be multiple events.
Each event has a type, an id, a timestamp, metadata and a body.
Internally, we refer to this as the "event envelope" as it tells us something about the event but not the trace.
We use the event id within this envelope to deduplicate messages to avoid processing the same event twice, i.e. the event id should be unique per request.
The event.body.id is the ID of the actual trace and will be used for updates and will be visible within the Langfuse App.
I.e. if you want to update a trace, you'd use the same body id, but separate event IDs.
Notes:
- Introduction to data model: https://langfuse.com/docs/observability/data-model
- Batch sizes are limited to 3.5 MB in total. You need to adjust the number of events per batch accordingly.
- The API does not return a 4xx status code for input errors. Instead, it responds with a 207 status code, which includes a list of the encountered errors.
docs:The annotation queue referenced by the score. Indicates if score was initially created while processing annotation queue.
value:
type:commons.CreateScoreValue
docs:The value of the score. Must be passed as string for categorical scores, and numeric for boolean and numeric scores. Boolean score values must equal either 1 or 0 (true or false)
observationId:optional<string>
comment:optional<string>
metadata:optional<unknown>
dataType:
type:optional<commons.ScoreDataType>
docs:When set, must match the score value's type. If not set, will be inferred from the score value or config
@@ -233,6 +331,14 @@ types:
dataType:"BOOLEAN"
configId:"1234-5678-90ab-cdef"
traceId:"cdef-1234-5678-90ab"
- value:
name:"contextrelevant"
value:"not relevant"
sessionId:"abyt-1234-5678-80ab"
- value:
name:"hallucination"
value:0
datasetRunId:"7891-5678-90ab-hijk"
BaseEvent:
properties:
@@ -312,3 +418,28 @@ types:
properties:
successes:list<IngestionSuccess>
errors:list<IngestionError>
OpenAICompletionUsageSchema:
docs:OpenAI Usage schema from (Chat-)Completion APIs
docs:Get all memberships for the organization associated with the API key (requires organization-scoped API key)
method:GET
path:/organizations/memberships
response:MembershipsResponse
updateOrganizationMembership:
docs:Create or update a membership for the organization associated with the API key (requires organization-scoped API key)
method:PUT
path:/organizations/memberships
request:MembershipRequest
response:MembershipResponse
deleteOrganizationMembership:
docs:Delete a membership from the organization associated with the API key (requires organization-scoped API key)
method:DELETE
path:/organizations/memberships
request:DeleteMembershipRequest
response:MembershipDeletionResponse
getProjectMemberships:
docs:Get all memberships for a specific project (requires organization-scoped API key)
method:GET
path:/projects/{projectId}/memberships
path-parameters:
projectId:string
response:MembershipsResponse
updateProjectMembership:
docs:Create or update a membership for a specific project (requires organization-scoped API key). The user must already be a member of the organization.
method:PUT
path:/projects/{projectId}/memberships
path-parameters:
projectId:string
request:MembershipRequest
response:MembershipResponse
deleteProjectMembership:
docs:Delete a membership from a specific project (requires organization-scoped API key). The user must be a member of the organization.
method:DELETE
path:/projects/{projectId}/memberships
path-parameters:
projectId:string
request:DeleteMembershipRequest
response:MembershipDeletionResponse
getOrganizationProjects:
docs:Get all projects for the organization associated with the API key (requires organization-scoped API key)
docs:Get SCIM Service Provider Configuration (requires organization-scoped API key)
method:GET
path:/ServiceProviderConfig
response:ServiceProviderConfig
getResourceTypes:
docs:Get SCIM Resource Types (requires organization-scoped API key)
method:GET
path:/ResourceTypes
response:ResourceTypesResponse
getSchemas:
docs:Get SCIM Schemas (requires organization-scoped API key)
method:GET
path:/Schemas
response:SchemasResponse
listUsers:
docs:List users in the organization (requires organization-scoped API key)
method:GET
path:/Users
request:
name:ListUsersRequest
query-parameters:
filter:
docs:Filter expression (e.g. userName eq "value")
type:optional<string>
startIndex:
docs:1-based index of the first result to return (default 1)
type:optional<integer>
count:
docs:Maximum number of results to return (default 100)
type:optional<integer>
response:ScimUsersListResponse
createUser:
docs:Create a new user in the organization (requires organization-scoped API key)
method:POST
path:/Users
request:
name:CreateUserRequest
body:
properties:
userName:
docs:User's email address (required)
type:string
name:
docs:User's name information
type:ScimName
emails:
docs:User's email addresses
type:optional<list<ScimEmail>>
active:
docs:Whether the user is active
type:optional<boolean>
password:
docs:Initial password for the user
type:optional<string>
response:ScimUser
getUser:
docs:Get a specific user by ID (requires organization-scoped API key)
method:GET
path:/Users/{userId}
path-parameters:
userId:string
response:ScimUser
deleteUser:
docs:Remove a user from the organization (requires organization-scoped API key). Note that this only removes the user from the organization but does not delete the user entity itself.
docs:The unique langfuse identifier of a score config
response:commons.ScoreConfig
update:
docs:Update a score config
method:PATCH
path:/score-configs/{configId}
path-parameters:
configId:
type:string
docs:The unique langfuse identifier of a score config
request:UpdateScoreConfigRequest
response:commons.ScoreConfig
types:
ScoreConfigs:
properties:
@@ -56,3 +68,23 @@ types:
description:
type:optional<string>
docs:Description is shown across the Langfuse UI and can be used to e.g. explain the config categories in detail, why a numeric range was set, or provide additional context on config name or usage
UpdateScoreConfigRequest:
properties:
isArchived:
type:optional<boolean>
docs:The status of the score config showing if it is archived or not
name:
type:optional<string>
docs:The name of the score config
categories:
type:optional<list<commons.ConfigCategory>>
docs:Configure custom categories for categorical scores. Pass a list of objects with `label` and `value` properties. Categories are autogenerated for boolean configs and cannot be passed
minValue:
type:optional<double>
docs:Configure a minimum value for numerical scores. If not set, the minimum value defaults to -∞
maxValue:
type:optional<double>
docs:Configure a maximum value for numerical scores. If not set, the maximum value defaults to +∞
description:
type:optional<string>
docs:Description is shown across the Langfuse UI and can be used to e.g. explain the config categories in detail, why a numeric range was set, or provide additional context on config name or usage
docs:Create a score (supports both trace and session scores)
method:POST
path:/scores
request:CreateScoreRequest
response:CreateScoreResponse
get:
docs:Get a list of scores
method:GET
path:/scores
request:
name:GetScoresRequest
query-parameters:
page:
type:optional<integer>
docs:Page number, starts at 1.
limit:
type:optional<integer>
docs:Limit of items per page. If you encounter api issues due to too large page sizes, try to reduce the limit.
userId:
type:optional<string>
docs:Retrieve only scores with this userId associated to the trace.
name:
type:optional<string>
docs:Retrieve only scores with this name.
fromTimestamp:
type:optional<datetime>
docs:Optional filter to only include scores created on or after a certain datetime (ISO 8601)
toTimestamp:
type:optional<datetime>
docs:Optional filter to only include scores created before a certain datetime (ISO 8601)
source:
type:optional<commons.ScoreSource>
docs:Retrieve only scores from a specific source.
operator:
type:optional<string>
docs:Retrieve only scores with <operator> value.
value:
type:optional<double>
docs:Retrieve only scores with <operator> value.
scoreIds:
type:optional<string>
docs:Comma-separated list of score IDs to limit the results to.
configId:
type:optional<string>
docs:Retrieve only scores with a specific configId.
queueId:
type:optional<string>
docs:Retrieve only scores with a specific annotation queueId.
dataType:
type:optional<commons.ScoreDataType>
docs:Retrieve only scores with a specific dataType.
traceTags:
type:optional<list<string>>
allow-multiple:true
docs:Only scores linked to traces that include all of these tags will be returned.
response:GetScoresResponse
get-by-id:
docs:Get a score
method:GET
path:/scores/{scoreId}
path-parameters:
scoreId:
type:string
docs:The unique langfuse identifier of a score
response:commons.Score
delete:
docs:Delete a score
docs:Delete a score (supports both trace and session scores)
method:DELETE
path:/scores/{scoreId}
path-parameters:
@@ -84,13 +24,22 @@ types:
CreateScoreRequest:
properties:
id:optional<string>
traceId:string
traceId:optional<string>
sessionId:optional<string>
observationId:optional<string>
datasetRunId:optional<string>
name:string
value:
type:commons.CreateScoreValue
docs:The value of the score. Must be passed as string for categorical scores, and numeric for boolean and numeric scores. Boolean score values must equal either 1 or 0 (true or false)
observationId:optional<string>
comment:optional<string>
metadata:optional<unknown>
environment:
type:optional<string>
docs:The environment of the score. Can be any lowercase alphanumeric string with hyphens and underscores that does not start with 'langfuse'.
queueId:
type:optional<string>
docs:The annotation queue referenced by the score. Indicates if score was initially created while processing annotation queue.
dataType:
type:optional<commons.ScoreDataType>
docs:The data type of the score. When passing a configId this field is inferred. Otherwise, this field must be passed or will default to numeric.
@@ -113,10 +62,12 @@ types:
dataType:"NUMERIC"
configId:"9203-4567-89ab-cdef"
traceId:"cdef-1234-5678-90ab"
environment:"test"
- value:
name:"toxicity"
value:"not toxic"
traceId:"cdef-1234-5678-90ab"
environment:"production"
- value:
name:"correctness"
value:"partially correct"
@@ -139,38 +90,3 @@ types:
id:
type:string
docs:The id of the created object in Langfuse
GetScoresResponseTraceData:
properties:
userId:
type:optional<string>
docs:The user ID associated with the trace referenced by score
tags:
type:optional<list<string>>
docs:A list of tags associated with the trace referenced by score
docs:Optional filter to only include sessions created before a certain datetime (ISO 8601)
environment:
type:optional<string>
allow-multiple:true
docs:Optional filter for sessions where the environment is one of the provided values.
response:PaginatedSessions
get:
docs:Get a session. Please note that `traces` on this endpoint are not paginated, if you plan to fetch large sessions, consider `GET /api/public/traces?sessionId=<sessionId>`
docs:The unique langfuse identifier of the trace to delete
response:DeleteTraceResponse
list:
docs:Get list of traces
method:GET
@@ -50,13 +59,61 @@ service:
release:
type:optional<string>
docs:Optional filter to only include traces with a certain release.
environment:
type:optional<string>
allow-multiple:true
docs:Optional filter for traces where the environment is one of the provided values.
fields:
type:optional<string>
docs:"Comma-separated list of fields to include in the response. Available field groups: 'core' (always included), 'io' (input, output, metadata), 'scores', 'observations', 'metrics'. If not specified, all fields are returned. Example: 'core,scores,metrics'. Note: Excluded 'observations' or 'scores' fields return empty arrays; excluded 'metrics' returns -1 for 'totalCost' and 'latency'."
filter:
type:optional<string>
docs:|
JSON string containing an array of filter conditions. When provided, this takes precedence over legacy filter parameters (userId, name, sessionId, tags, version, release, environment, fromTimestamp, toTimestamp).
Each filter condition has the following structure:
"dx":"pnpm i && pnpm run infra:dev:prune && pnpm run infra:dev:up --pull always && pnpm --filter=shared run db:reset && pnpm --filter=shared run ch:reset && pnpm --filter=shared run db:seed:examples && pnpm run dev",
"dx-f":"pnpm i && pnpm run infra:dev:prune && pnpm run infra:dev:up --pull always && pnpm --filter=shared run db:reset -f && pnpm --filter=shared run ch:reset && pnpm --filter=shared run db:seed:examples && pnpm run dev",
"dx:skip-infra":"pnpm i && pnpm --filter=shared run db:reset && pnpm --filter=shared run ch:reset && pnpm --filter=shared run db:seed:examples && pnpm run dev",
"dx":"pnpm i && pnpm run infra:dev:prune && pnpm run infra:dev:up --pull always && pnpm --filter=shared run db:reset:test && pnpm --filter=shared run db:reset && pnpm --filter=shared run ch:reset && pnpm --filter=shared run db:seed:examples && pnpm run dev",
"dx-f":"pnpm i && pnpm run infra:dev:prune && pnpm run infra:dev:up --pull always && pnpm --filter=shared run db:reset:test && pnpm --filter=shared run db:reset -f && SKIP_CONFIRM=1 pnpm --filter=shared run ch:reset && pnpm --filter=shared run db:seed:examples && pnpm run dev",
"dx:skip-infra":"pnpm i && pnpm --filter=shared run db:reset:test && pnpm --filter=shared run db:reset && pnpm --filter=shared run ch:reset && pnpm --filter=shared run db:seed:examples && pnpm run dev",
"build":"turbo run build",
"start":"turbo run start",
"dev":"turbo run dev",
"dev:worker":"turbo run dev --filter=worker",
"dev:web":"turbo run dev --filter=web -- --turbo",
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.