* upgrade turbo
* add web dev turbo runner
* upgrade docker
* fix linter
* set env mode to loose (as in turbo v1) so we don't have to double specify env vars
* feat(nav): integrate HoverCard component for improved navigation item display
* refactor(nav): enhance HoverCard integration and sidebar styles for improved UX
* chore(sidebar): replace cookie-based state management with local storage for sidebar state persistence
* chore: push
* refactor(nav): wrap HoverCardContent in Portal for improved rendering context
* style: mark hovered item as active
* chore(CreateLLMApiKeyForm): replace Input with PasswordInput for improved UX
* chore: revert
* chore(data): add seed data and constants for datasets, prompts, and traces
- Introduced new JSON files for chat ML and nested JSON data.
- Created seed constants for datasets and prompts to streamline data generation.
- Updated seed scripts to utilize new constants and improve dataset creation logic.
- Enhanced Clickhouse preparation scripts to support new dataset structures and trace generation.
- Added utility functions for generating unique trace IDs.
This commit lays the groundwork for improved data handling and observability in the Langfuse application.
* chore(data): remove deprecated prompt files and enhance seed constants
* chore(prisma): remove unique constraint on projectId, datasetId, and input in DatasetItem model
* chore: push
* chore(seeder): introduce seeder abstraction
* chore: typos
* chore: restructure files
* chore: restructure files
* chore: fix imports
* chore: fix imports
* chore: lint
* chore: simplify
* chore: ensure eval data integrity
* chore: update metric name calculation in ClickHouse query builder
* refactor: use test-utils for clickhouse inserts
* chore: add otel conventional attributes to clickhouse db spans
* chore: use db.system instead of db.system.name
* chore: set span kind client for clickhouse queries
* fix(NewDatasetItemForm): add JSON formatting utility for input, output, and metadata fields
* perf(NewDatasetItemFromExistingObject): conditionally render NewDatasetItemForm based on form state
* fix(NewDatasetItemForm): update JSON value check to handle undefined case
* fix(TracesTable): stabilize control column definition
- Ensured conditional rendering of the bookmarked column remains intact while simplifying the structure.
* refactor(TracesTable): enhance column visibility and order identifiers for control states
- Updated dialog header to use a title component and improved button layout for clarity.
- Adjusted positioning of the close button for better alignment.
- Enhanced header and footer with rounded corners for a more polished look.
* fixup: limit render duration of traces table
* fixup: reduce render cycles through data flows and memoization
* refactor: enhance performance and structure of table components
- Removed unnecessary console logs in `data-table.tsx`.
- Refactored `TablePeekView` to use a more structured props type and memoization for improved performance.
- Updated `usePeekData` to reduce stale time from 5 minutes to 1 minute.
- Replaced `IOTableCell` with `MemoizedIOTableCell` in various components to optimize rendering.
- Adjusted `peekView` configurations in `observations.tsx` and `traces.tsx` for consistency and clarity.
* chore: eslint
* chore: move features
* chore: move features
* chore: move features
* chore: move features
* chore: move features
* chore: move features
* chore: move features
* chore: move features
* chore: move features
* chore: move features
* chore: move features
* chore: move features
* chore: move features
* chore: move features
* chore: move features
* changes to pro plan
* chore: move features
* truncate title and description to prevent wraps
* remove unnecessary padding from charts
* improve bignumber chart
* add min h/w
* mobile resizing
* fix
2025-05-29 15:58:38 +00:00
Max DeichmannGitHubellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
* perf: add exact timestamp match for eval batch actions
* perf: add exact timestamp match for eval batch actions
* perf: add exact timestamp match for eval batch actions
* chore: make `project_id` nullable on `eval_templates`
* feat: add script for managed langfuse evaluators
* feat: restyle running evaluator configs table
* feat: add peek view for running evals
* fixup: add peek view for evaluator library
* fixup: add configure button to table
* feat: rework eval creation flow
* feat: add option to edit eval from table
* fixup: configure evaluator flow with traces data preview
* style: eval configuration UI elements
* chore: add support for current langfuse templates
* chore: reference templateId on jobExecution
* style: add maintainer icon across
* chore: extract WorkerEvaluationVariableService
* fixup: support live data mapping interface
* fixup: revamp datasets eval ui
* fixup: add clone functionality
* chore: rename online evaluation -> llm-as-a-judge
* feat: add evaluator selection in experiment creation flow
* chore: refactor into hooks in experiment form
* feat: add ragas evaluator prompt templates
* chore: show evaluator prompt preview by default
* fix: eval table filtering
* feat: add input box to slider
* fix: clean up running eval peek view and logs
* fix: ensure preview variables are not wrapped in quotes in evaluation prompt preview
* feat: support navigating through traces table
* feat: support default model workflow
* fix: typos in managed evaluators
* fix: do not allow selection of evaluators requiring default to run
* fix: type and build errors
* chore: rename table to default_llm_model
* chore: rm empty file
* fixup: complete rename
* chore: push
* style: eval set up form
* style: separate new evaluator form out into two steps
* chore: ensure we only run evals if default model is set
* chore: add `partner` col to `eval_templates`
* style: new eval template creation
* chore: add proper mapping for dataset evals
* feat: also put evaluation selection on datasets page
* feat: datasets also run on historic experiments after creating mapping
* revert: 09ab242dc12c8647bc716c84d180bc8e631d4919
* chore: fix eslint and ts
* chore: fix warnings
* fix: ensure we inactivate all job configs that rely on default eval model
* fix: tests
* chore: fix eslint
* chore: wrap in tx
* chore: fix default model checkbox logic
* chore: improve wording on delete model
* fix: eslint
* chore: adjust migration order
* style: push ui improvements
* style: sort templates by partner
* fix: eslint
* chore(dashboards): expand measures for dashboards
* chore: add testcase
* chore: define inputTokens and outputTokens
* chore: add timePerOutputToken and costs
* chore: fix definition and handling of tag based breakdowns
* feat(dashboard): make dashboard widgets draggable
* chore: do not update on layout change
* chore: fix size
* chore: remove the preventCollision flag
* chore: linting
* chore: get row height dynamically
* chore: make chart resizable
* chore: styling for resizes
* chore: add migration to align y values
* chore: add owner to project and return langfuse dashboards
* chore: show owner for dashborads in table
* chore: snapshot
* chore: snapshot
* chore: allow widget clone
* chore: snapshot
* chore: linting
* chore: use project instead of user
* chore: show error message on clone failure
* chore: indicate existing dashboard widget relationship
* chore: show only project dashboards in selection for adding widgets
* chore: make full dashboard row clickable
* chore: add clone button to langfuse dashboards
* feat(dashboards): add support for cloning existing dashboards
* chore: move buttons into a dedicated dropdown menu
* chore: allow updates to dashboard name and description
* chore: lint
* chore: removal double capturing of dashboard delete
* chore: track deletions
* feat: full text search
* push
* feat: full text search
* feat: full text search
* feat: full text search
* feat: full text search
* feat: full text search
* fix
* push
* fix: fix json limits and scope to single bodies only
* fix
* fix
* jumping UI
* jumping UI
* jumping UI
* push
* Update web/src/components/table/data-table-toolbar.tsx
Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
---------
Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
* push
* fix: fix json limits and scope to single bodies only
* push
* fix: fix json limits and scope to single bodies only
* fix: fix json limits and scope to single bodies only
* fix: fix json limits and scope to single bodies only
* fix: fix json limits and scope to single bodies only
* fix: fix json limits and scope to single bodies only
* chore: memoize table body for enhanced front end performance
* chore: push
* chore: refactor
* style: speed up animation
* chore: re-render on table state changes
* chore: import function directly
* chore: refactor to use timestamp
* chore: push
* chore: push
* feat: add dlx retry queue and service
* feat: add dlx retry queue and service
* feat: add dlx retry queue and service
* feat: add dlx retry queue and service
* feat: add dlx retry queue and service
* feat: add new organizations apiKey route
* chore: implement additional org routes
* chore: add API docs
* chore: checkin openapi spec
* chore: update tests
* feat: allow org api key management via admin api
* feat: allow the creation of projects using an org api key
* chore: allow deletion of projects via API
* feat: create outline of SCIM Users support
* feat: implement SCIM style user endpoints
* chore: allow set password on user
* chore: lint
* feat: add endpoints to manage project and organization memberships
* chore: add memberships api docs
* chore: reject project membership calls if entitlement is missing
* chore: add team plan to test org
* chore: add reference to full Langfuse API spec
* feat: add new organizations apiKey route
* chore: implement additional org routes
* chore: add API docs
* chore: checkin openapi spec
* chore: update tests
* feat: allow org api key management via admin api
* feat: allow the creation of projects using an org api key
* chore: allow deletion of projects via API
* feat: create outline of SCIM Users support
* feat: implement SCIM style user endpoints
* chore: allow set password on user
* chore: lint
* feat: add endpoints to manage project and organization memberships
* chore: add memberships api docs
* chore: reject project membership calls if entitlement is missing
* chore: add team plan to test org
* feat: add new organizations apiKey route
* chore: implement additional org routes
* chore: add API docs
* chore: checkin openapi spec
* chore: update tests
* feat: allow org api key management via admin api
* feat: allow the creation of projects using an org api key
* chore: allow deletion of projects via API
* feat: create outline of SCIM Users support
* feat: implement SCIM style user endpoints
* chore: allow set password on user
* chore: lint
* chore: add scim api docs
* chore: remove outdated generated artifacts
* feat: add new organizations apiKey route
* chore: implement additional org routes
* chore: add API docs
* chore: checkin openapi spec
* chore: update tests
* feat: allow org api key management via admin api
* feat: allow the creation of projects using an org api key
* chore: allow deletion of projects via API
* chore: add documentation for new project routes
* chore: cleanup
* chore: error updates and handling
* chore: reduce reduntant error handling
* chore(evals): fall back to bullmq `job.data.timestamp` allowing for successful retries
* chore: test
* chore: test
* docs: add diagram and docs
* chore: typo
* chore: typo
* chore: refactor authentication for admin apis
* chore: lint
* chore: address feedback
* feat: add post organizations endpoint
* chore: update tests
* chore: update api key data model
* chore: align code with new scoped api keys
* chore: typing
* chore: align tests with new logic
* chore: remove ratelimitoverrides
* chore: remove accesslevel from queue message
* chore: add additional check to ingestion endpoint
* chore: add test for legacy api key format in redis
* chore: map all api keys in auditlog
* added environment filter option to traces
* WIP on multi-select ui inconsistency
* fixed overflow issue in multi select where an item would not take the full width automatically; also changed to overflow-x-auto
* fixed invalid DOM nesting of div within p (FormDescription) that resulted in a hydration error
* fix multiselect to cut overflowing text with ellipsis and take full name as titlte so you can hover over it, removed unnecessary css classes
* WIP on create evaluator page ui issue with filter builder
* Revert "WIP on create evaluator page ui issue with filter builder"
This reverts commit fd6dd507f2df3c5fb404f4e38a2983e8ef6405ef.
* hide empty string from multi-select options, as supporting empty values would require further work - out of scope for now; also checked for side effects, with current usage all good
* Added test suite for eval filtering
* Resolved comment: removed getEnvironmentsForProject from traces router and instead fetched environments inside evaluator-form via api.projects.environmentFilterOptions
* move eval filtering tests into separate test file
* removed redundancy via fixtures, prepared tests for concurrency
* filtering tests run concurrently
* extracted traceFilterOptions fetching into separate hook as it is used in both the evaluator form and the traces table, also fixed traces table since the environment was now missing there
* useTraceFilterOptions Hook after merge
* Make TraceOptions keys partial so that we can update filter options asynchronously
* default to empty object if response undefined
* exclude environment filter builder column from traces table since toolbar already has env
* removed useTraceFilterOptions hook as use case changed such that extraction into hook is no longer warranted
* chore: pin dev clickhouse version to 24.3
* chore: only add input format flag for cloud
* chore: whitelist accepted cloud regions
* chore: disable test
* fix: (workaround) evals page overflows when rendering right hand side metadata bar
This is more of a workaround, we should probably disable the overflow on the main wrapper div in layout.tsx and enable it only where needed - or find out why the overflow inside the metadata sidebar trigggers the scrollbar to pop up - some info on what I found:
- The body and other divs close to the root do not increase their size.
- The navigation sidebar does increase its vertical height, but limiting that does not resolve the main scrollbar.
- Limiting specifically the wrapper div in layout.tsx with overflow-hidden does resolve this, but globally disabling it requires most pages to be refactored.
* Revert "fix: (workaround) evals page overflows when rendering right hand side metadata bar"
This reverts commit 313a1c0d8a885a04974399f850f7f9aee58fd714.
* fix(ui): TableWithMetadataWrapper uses contain-layout on its root div so that EvaluatorDetail overflow issue is mitigated
Checked for side effects on the other components using TableWithMetadataWrapper and all work as expected.
---------
Co-authored-by: Marlies Mayerhofer <74332854+marliessophie@users.noreply.github.com>
* chore(annotation-scores): revert to synchronous delete; upsert on create route
* fix: ensure scores data not only marked as stale but refreshed on score update
* feat: add support for GCS buckets
* chore: cleanup
* chore: trim gcs credentials before parsing
* chore: add logging for redis shutdown in tests
* chore: update timeout for trace delete test
* chore: bump trace delete concurrency for tests
* feat(evals): enable deletion of evaluators and templates via UI
* Added tests for the evals job and template deletion trpc endpoints
* evals-trpc test refactor, it had a side effect on other tests since it was pruning the db, now instead creates a new org and project per tests and cleans them up after all tests are done
* removed test comments as those are now implemented
* fix wrong wording
* added template version to delete confirmation on template details delete action
* DeleteButton refactoring and moved deletion of templates to LFE-4573
* moving the call to captureDeleteSuccess inside the successful branch of executeDeleteMutation
* Display lock icon when action button in icon mode is unauthorized
* fix: quietly delete scheduled evals for deleted job executions
---------
Co-authored-by: Max Deichmann <m.deichmann@tum.de>
* fix(trace-timeline): overflow and scroll behaviour
* fix(trace-timeline): show maximum of 3 scores and remainder in hover card
* push
* chore: refactor
* chore
* chore
* feat: add self-serve dashboard backend poc
* chore: data model thoughts
* chore: create initial query builder example and test case
* chore: handle empty dimensions and metrics
* chore: lint
* chore: extend filter conditions to account for timestamps
* chore: refactor query builder into sub-functions
* chore: use template queries for user-supplied values
* chore: add query builder tests
* chore: handle time dimension
* chore: add trpc endpoint to execute custom clickhouse query
* chore: add a query playground component to test a couple of queries
* chore: handle multiple joins
* chore: add observations table to data model
* chore: fix bug in data model
* chore: add test case to compare with old dashboard results
* chore: add scaffold for additional views
* expand the users view and add segments to filter scores subviews
* chore: add test cases for score views
* chore: drop users and sessions for now
* chore: add scores aggregate and observations cost tests
* chore: add sql injection tests
* chore: lint
* chore: fix dashboard test cases
* chore: drop users query test
* chore: add order by logic
* chore: fill timeseries values
* chore: use new query function on dashboard
* chore: convert TracesBarListChart.tsx to new query endpoint
* chore: make chart data compile
* chore: update tests
* chore: add tags to custom queries
* chore: typing
* chore: suffix join condition with sql
* chore: add typing in queryBuidler
* chore: limit playground to cloud admin users
* chore: add util to map legacy dashboard columns to new model
* chore: separate time filter state for TracesBarListChart.tsx
* chore: pass timestamps directly into chart components
* chore: add userId and sessionId on observations
* chore: pick auto time granularity based on hours
* fix(prompt-experiments): show form error if name is duplicated
* chore: fix typo
* Update web/src/ee/features/experiments/hooks/useExperimentNameValidation.tsx
Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
* eslint
---------
Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
* feat(evals): preview number of historic items to be evaluated
* refactor: split up large evaluator form file
* chore: drop optional pagination
* chore: fix eslint
* chore: rename test
* fix: safely parse big numbers in api responses
* feat: add additional input/output parsing for pydantic via OTel
* Revert "feat: add additional input/output parsing for pydantic via OTel"
This reverts commit 71d0f3abd951702393aac7ab79a875b888f089cf.
* pin node to problematic version for working on a fix
* added cross-env and disabled the feature flag causing the testing issue - long term fix should probably be switching to vitest
* unpin node version since problematic feature is now disabled fo affected tests
* fix(ui): IOPreview now renders messages where content is an empty string instead of ignoring them
* removed debug logs
* OpenAiMessageView first filters valid messages to render and then executes the "show x more ..." logic, this way we ensure that there can never be a mismatch between messages items and the amount of fields we support displaying
* Render empty string quotation marks if content is empty but not null or undefined
* Change length check to falsy check for conditional display of empty quotes if content is empty
* cleaner falsy check for rendering empty string quotation marks
* fix(ui): stop propagation on traces table checkbox
* fix(peek-ui): handle onRowClick action correctly for peek view
* feat(peek): do not close upon any screen button or checkbox clicks
* push
* wip
* simplify
* actionbutton for upgrade prompt
* count events for free plan limit
* add sso settings
* rename lite to core
* add product id
* nit
* nits
* fix rate limit test
* ci run on teams plan
* fix
* fix names of pipeline steps
* test tmp
* add better error logs to makeAPICall
* push
* clone to throw better errors
* improve error
* improve error
* fix
* remove comment
* refactor
* push
* next attempt
* nit
* create new auth for failing test suites
* push
* fix rate limit test
* feat: add media and export external endpoint config
* chore: add docs string
* chore: add external endpoint support for azure
* chore: drop feature for media
* Unified design of grouped score badge with trace detail score badges, except that text is no longer medium/semibold
* Made padding of root item of ObservationTree consistent with other items
* fixed alignment issues of badges wrapped in links in trace and observation detail views
* Fixed tooltip of aggregated usage data badge (in trace details) rendering even if there is no underlying data
* chore(cloud): Make UI recognize cloud:enterprise plan and add corresponding access rights and rate limits
- label on project selector
- rate limits same as team
- access rights same as team
- no stripe product id for this (yet), cannot be purchased self-serve in product
* remove comments
---------
Co-authored-by: Marc Klingen <git@marcklingen.com>
* feat(auth): add workos as idp
* add button for org and connection based sign in
* fix
* remove client auth method and checks
* remove unnecessary diff
* fix: header
* fix: search always show current version
* chore: adjust code block
* chore: re-add copy in code editor if no title
* chore: do not match current version always
* adjust margins
* chore: remove breadcrumb
* fix: typo
* fix: final margin
* style: eval template
* small design nits
* fix: do not allow to deactivate eval configs which rano on historic traces only
* fix: do not allow to deactivate eval configs which rano on historic traces only
* fix: do not allow to deactivate eval configs which rano on historic traces only
* fix: do not allow to deactivate eval configs which rano on historic traces only
* fix: do not allow to deactivate eval configs which rano on historic traces only
* fix: do not return all executions for eval config list trpc
* fix: do not return all executions for eval config list trpc
* fix: do not return all executions for eval config list trpc
* fix: do not return all executions for eval config list trpc
* chore: fix demo banner
* fix(page-header): info icon relative positioning
* fix: header title wrap behaviour
* fix(page-header): have info icon follow title
* style: move dataset level buttons to more menu
* push
* push
* abstract editor to support "text" mode in addition to "json" mode
* always use CodeMirrorEditor for prompts
* minheight on editor
* upgrade codemirror and add language package
* add linting for prompts and switch to githubDark theme
* package
* use codemirror for chat prompts
* imrpove scroll behavior of new dataset form
* improve dataset item editor view
* push
* remove dep pinning
* align editor linting with current variable definition
* defauklt value to value
* rename existing command exports
* move existing to input-command
* add comment
* wip cmd k menu for navigation
* fix types and paths of nested menu items
* add support for nested menu items
* filter correctly
* feat(prompts): support in UI prompt duplication
* chore(ui-ds): make `children` in `ActionButton` optional
* fix: extract custom hook for prompt name validation
* eslint
* fix: insert correct prompt count
* chore: validate prompt name uniqueness; attach latest label for single version
* chore: refactor duplication approach to use createMany and save on network-roundtrips
* chore: revert changes on action button
* push
* fix: allow field overwrites with `null` in ingestion api
* chore: add undefined check
* chore: check undefined does not overwrite value
* chore: explicit undefined check
* feat: convert generations to observations table
* feat: add type col to generations table
* feat: add type col to generations table
* feat: add type col to generations table
* feat: add type col to generations table
* feat: add type col to generations table
* fix: fix
* fix: update ordering for get traces calls to avoid stale results
* chore: explanation
* chore: remove stale test
* chore: lint
* chore: update comment
* chore: cleanup
* fix(ui): docs menu item should open in new tab
* Update web/src/components/nav-main.tsx
Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
---------
Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
* chore: stream clickhouse results to posthog to reduce memory consumption
* chore: end span after streaming all results
* chore: tracing for clickhouse streaming
* fix(prompt-experiments): do not restrict to evalModelsOnly in order to use anthropic, bedrock, vertex
* chore(cloud): disable deletion of traces via UI as it caused performance issues (#4873)
chore(cloud): disable deletion of traces via the UI as it caused performance issues
* docs: add link to data model documentation to api reference of ingestion endpoint (#4877)
* fix: use database upserts correctly (#4884)
* fix: use database upserts correctly
* fix: use database upserts correctly
* push
* fix: use database upserts correctly
* fix: use database upserts correctly
* push
* fix: use database upserts correctly
* chore(deps): bump next from 14.2.15 to 14.2.21 (#4887)
Bumps [next](https://github.com/vercel/next.js) from 14.2.15 to 14.2.21.
- [Release notes](https://github.com/vercel/next.js/releases)
- [Changelog](https://github.com/vercel/next.js/blob/canary/release.js)
- [Commits](https://github.com/vercel/next.js/compare/v14.2.15...v14.2.21)
---
updated-dependencies:
- dependency-name: next
dependency-type: direct:production
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* chore: release v3.5.2
* chore: remove postgres from ingestion merge (#4467)
* chore: remove postgres from ingestion merge
* chore: remove pg tests
* chore: remove postgres tracing
* remove evalModelsOnly
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Max Deichmann <m.deichmann@tum.de>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Steffen Schmitz <steffen@langfuse.com>
Co-authored-by: Hassieb Pakzad <68423100+hassiebp@users.noreply.github.com>
* fix(annotation): fail postgres upsert, delete silentley while Clickhouse is in lead
* chore: disable deletion of score while update operation is in progress
* chore(eval/experiments): support job retries via bullmq instead of within worker
* chore: timeout model completion calls after 2 minutes
* chore: set retries on llm completion calls to 1
* chore: do not propogate expected error cases in experiment creation
* eslint
* wip: optimistic updates on annotation form
* refactor to handle sequence of create, update operations correctly
* fix: don't check postgres for score existence when running on clickhouse
* disable annotation button while score updates are saving
* fix: await for all mutations to complete before opening modal again
* rm console log
* fix: update api response clickhouse
* chore: remove reading trace and score from prisma, only read from clickhouse
* fixup: keep create logic behind flag
* chore: only read from postgres for updating annoation score if `LANGFUSE_POSTGRES_INGESTION_ENABLED`
* chore: only read from postgres for deleting annoation score if `LANGFUSE_POSTGRES_INGESTION_ENABLED`
* chore: throw errors if scores not found in CH
* chore: add audit logs in clickhouse scenario
* only create audit log once on cloud
* feat(comments): add support for comment button on each trace preview in single session view
* perf: manually timeout comment count query at 500ms without failing overall operation
* push
* chore: query comment reference objects in clickhouse if `env.clickhouse_url`
* remove: deprecated code
* chore: support fetching traces per session from clickhouse
* push
* refactor: adjust tests to populate traces and observations in clickhouse
* refactor: adjust tests and remove redundant code
* push
* chore: drop FK v3 constraints
* make sleep great again
* make sleep great again
* make sleep great again
* make sleep great again
---------
Co-authored-by: steffen911 <steffen@langfuse.com>
* chore: make project deletion asynchronous
* chore: delete projects on the worker
* chore: indicate in UI overview that project is being deleted
* chore: remove API keys from Postgres in project deletion call
* wip
* push
* add migration
* render correct usage event type in settings
* render number of seats in settings
* wip
* add entitlement limits
* add new actionbutton
* migrate annotation queue create
* add to table date range selectors
* add limit to dashboard date range
* move migration
* fix usage preview in billing settings
* get seats from subscription instead of next invoice
* remove old console.log
* ad pro plan self-hosted
* fix
* readme
* rename / fixes
* push
* only alert on over 90% usage
* push
* push
* push
* push
* add new self host entitlements for v3
* drop batch-export entitlement
* push
* move logic to clickhouse
* push
* fix padding
* fix clickhouse logic
# Snyk cannot upload results in merge group. Hence, we only run on PRs and when pushingon the main branch https://github.com/github/codeql-action/issues/1572
Langfuse is an **open source LLM engineering** platform for developing, monitoring, evaluating and debugging AI applications. See the README for more details.
## Linting
- Run `pnpm run lint` to lint all packages.
- Fix issues automatically with `pnpm run lint:fix`.
## Tests
- Codex cannot run the test suite because it depends on Docker-based infrastructure that is unavailable in this environment.
## Cursor Rules
- Additional folder-specific rules live in `.cursor/rules/`.
## Commits
- Follow [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/) when crafting commit messages.
Langfuse is an open-source LLM engineering platform that helps teams collaboratively develop, monitor, evaluate, and debug AI applications.
The main feature areas are tracing, evals and prompt management. Langfuse consists of the web application (this repo), documentation, python SDK and javascript/typescript SDK.
This repo contains the web application, worker, and supporting packages but notably not the JS nor Python client SDKs.
## Repository Structure
High level structure. There are more folders (eg for hooks etc).
pnpm run dev:web # Web app only (localhost:3000) - **used in most cases!**
pnpm run dev:worker # Worker only
pnpm run dx # Full initial setup: install deps, reset DBs, resets node modules, seed data, start dev. USE SPARINGLY AS IT WIPES THE DATABASE & node_modules
```
### Database Management
database commands are to be run in the `packages/shared/` folder.
@@ -27,7 +27,7 @@ The maintainers are available on [Discord](https://langfuse.com/discord) in case
## Making a change
_Before making any significant changes, please [open an issue](https://github.com/langfuse/langfuse/issues)._ Discussing your proposed changes ahead of time will make the contribution process smooth for everyone. Large changes that were not discussed in an issue may be rejected.
_Before making any significant changes, please [open an issue](https://github.com/langfuse/langfuse/issues)._ Discussing your proposed changes ahead of time will make the contribution process smooth for everyone. Changes that were not discussed in an issue may be rejected.
Once we've discussed your changes and you've got your code ready, make sure that tests are passing and open your pull request.
@@ -42,7 +42,7 @@ A good first step is to search for open [issues](https://github.com/langfuse/lan
- NextAuth.js / Auth.js
- tRPC: Frontend APIs
- Prisma ORM
- Zod
- Zod v4
- Tailwind CSS
- shadcn/ui tailwind components (using Radix and tanstack)
- Fern: generate OpenAPI spec and Pydantic models
@@ -53,124 +53,35 @@ A good first step is to search for open [issues](https://github.com/langfuse/lan
@@ -186,7 +97,7 @@ Full database schema: [packages/shared/prisma/schema.prisma](packages/shared/pri
We built a monorepo using [pnpm](https://pnpm.io/motivation) and [turbo](https://turbo.build/repo/docs) to manage the dependencies and build process. The monorepo contains the following packages:
-`web`: is the main application package providing Frontend and Backend APIs for Langfuse.
-`worker` (no production yet): contains an application for asynchronous processing of tasks. This package is not yet used in production.
-`worker`: contains an application for asynchronous processing of tasks.
-`packages`:
-`shared`: contains shared code between the above packages.
-`config-eslint`: contains eslint configurations which are shared between the above packages.
@@ -205,11 +116,12 @@ Requirements
**Steps**
1.Fork the repository and clone it locally
2.Run the development database
1.Install [golang-migrate](https://github.com/golang-migrate/migrate/tree/master/cmd/migrate#migrate-cli) as CLI
8. Log in as a test user (after you ran `db:seed` command):
Username: demo@langfuse.com
Password: password
To get comprehensive example data, you can use the `seed` command:
```sh
pnpm run db:seed:examples
```
## Monorepo quickstart
@@ -302,23 +204,41 @@ Requirements
On the main branch, we adhere to the best practices of [conventional commits](https://www.conventionalcommits.org/en/v1.0.0/). All pull requests and branches are squash-merged to maintain a clean and readable history. This approach ensures the addition of a conventional commit message when merging contributions.
## Test the public API
## Running Unit Tests
The API is tested using Jest. With the development server running, you can run the tests with:
All tests run in the CI and must pass before merging.
All tests run against a running langfuse instance and **write/delete real data from the database**.
Run all
### Tests in the `web` package (public API)
We're using Jest with in the `web` package. Therefore, if you want to provide an argument to the test runner, do it directly without an intermittent ` -- `.
```bash
npm run test
There are three types of unit tests:
- `test-sync`
- `test-async`
- `test-client`
To run a specific test, for example the test: `"should handle special characters in prompt names"` in `prompts.v2.servertest.ts`, run:
```sh
cd web # or with --filter=web
pnpm test-sync --testPathPattern="prompts\.v2\.servertest" --testNamePattern="should handle special characters in prompt names"
```
Run interactively in watch mode
```bash
npm run test:watch
To run all tests:
```sh
pnpm run test
```
These tests are also run in CI.
Run interactively in watch mode (not recommended!)
```sh
pnpm run test:watch
```
### Tests in the `worker` package
For the `worker` package, we're using `vitest` to run unit tests.
```sh
pnpm run test --filter=worker -- FILE_YOU_WANT_TO_TEST.ts -t "test name"
```
## CI/CD
@@ -397,7 +317,7 @@ The background color of the following component will be `hsl(var(--primary))` an
| --primary-accent | Primary accent color used for branding | Layout |
| --hover-primary-accent | Primary accent color used for hover effects for links | SignIn and AuthCloudRegionSwitch |
| --muted-green | Muted green for Event label | ObservationTree |
| --muted-blue | Muted blue for Span label | ObservationTree |
| --muted-gray | Muted gray for disabled status badges | StatusBadge |
| --accent-light-green | Light green accent for background of output and assistant messages | IOPreview, Generations, Traces |
@@ -439,13 +359,27 @@ You can update the default AI models and prices by adding or updating an entry i
Please note that
- prices are in USD
- the list is ordered by ID, so make sure to keep this order
- the list is ordered by ID, so make sure to keep this order and insert new models at the end of the list
- the `updated_at` field must be updated with the current date in ISO 8601 format. Otherwise, the change will be ignored.
### Transition period until V3 release
Until the V3 release, both the JSON record must be updated **and** a migration must be created to continue supporting self-hosted users. Note that the migration must updated both the `models` as well as the `prices` table accordingly.
## Updating the OpenAPI Specs & fern SDKs
We maintain the API specifications manually to guarantee a high degree of understandability. If you made changes to the API, please update the respective `.yml` files in `fern/apis/...`.
To generate the respective `openapi.yml` files which power the online API reference & SDKs, run:
```sh
npx fern-api generate --api server # for the server API
npx fern-api generate --api client # for the client API
npx fern-api generate --api organizations # for the organizations API
```
**Note:** You need a signed in fern account to run those commands.
## License
Langfuse is MIT licensed, except for `ee/` folder. See [LICENSE](LICENSE) and [docs](https://langfuse.com/docs/open-source) for more details.
Portions of this software are licensed as follows:
- All content that resides under the "ee/" and/or "web/src/ee" directories of this repository, if these directories exist, is licensed under the license defined in "ee/LICENSE".
- All third party components incorporated into the Finto Technologies Software are licensed under the original license provided by the owner of the applicable component.
- All content that resides under the "ee/", "web/src/ee/", and/or "worker/src/ee/" directories of this repository, if these directories exist, is licensed under the license defined in "ee/LICENSE".
- All third party components incorporated into the Langfuse Software are licensed under the original license provided by the owner of the applicable component.
- Content outside of the above mentioned directories or restrictions above is available under the "MIT Expat" license as defined below.
Permission is hereby granted, free of charge, to any person obtaining a copy
앱에 계측(instrumentation)을 추가하여 Langfuse로 trace 데이터를 수집함으로써, 검색, 임베딩, 또는 에이전트 동작과 같은 LLM 호출 및 기타 관련 로직을 추적할 수 있습니다. 복잡한 로그와 사용자 세션을 확인 및 디버깅 해보세요. 인터랙티브 데모를 통해 실제 작동 예를 확인할 수 있습니다.
- **프롬프트 관리**
프롬프트를 중앙에서 관리하고 버전 관리하며 협업으로 수정할 수 있도록 도와줍니다. 서버와 클라이언트 측의 강력한 캐싱 덕분에 애플리케이션에 지연(latency)을 추가하지 않고도 프롬프트를 반복 개선할 수 있습니다.
- **평가**
LLM 애플리케이션 개발 워크플로우에서 핵심적인 역할을 하며, Langfuse는 여러분의 필요에 맞게 유연하게 대응합니다. LLM을 심사자로 활용하는 기능, 사용자 피드백 수집, 수동 라벨링 및 API/SDK를 통한 맞춤 평가 파이프라인을 지원합니다.
- **데이터셋**
LLM 애플리케이션 평가를 위한 테스트 세트와 벤치마크를 제공하여, 지속적인 개선, 배포 전 테스트, 구조화된 실험, 유연한 평가 및 LangChain과 LlamaIndex와 같은 프레임워크와의 원활한 통합을 지원합니다.
- **LLM 플레이그라운드**
프롬프트와 모델 구성에 대해 테스트 및 반복 개선할 수 있는 도구로, 피드백 루프를 단축하여 개발 속도를 높여줍니다. trace에서 이상한 결과가 발생하면 플레이그라운드로 바로 이동해 개선할 수 있습니다.
- **종합 API**
Langfuse는 API를 통해 제공되는 구성 요소들을 활용하여 맞춤형 LLMOps 워크플로우를 강화하는 데 자주 사용됩니다. OpenAPI 명세, Postman 컬렉션, Python 및 JS/TS용 타입드 SDK가 제공됩니다.
| [LlamaIndex](https://langfuse.com/docs/integrations/llama-index/get-started) | Python | LlamaIndex 콜백 시스템을 통한 자동 계측을 지원합니다. |
| [Haystack](https://langfuse.com/docs/integrations/haystack) | Python | Haystack 콘텐츠 추적 시스템을 통한 자동 계측을 지원합니다. |
| [LiteLLM](https://langfuse.com/docs/integrations/litellm) | Python, JS/TS (proxy only) | GPT의 드롭인 대체품으로 어떤 LLM도 사용할 수 있습니다. Azure, OpenAI, Cohere, Anthropic, Ollama, VLLM, Sagemaker, HuggingFace, Replicate 등 100개 이상의 LLM 지원. |
| [Vercel AI SDK](https://langfuse.com/docs/integrations/vercel-ai-sdk) | JS/TS | React, Next.js, Vue, Svelte, Node.js와 함께 AI 기반 애플리케이션 구축을 돕는 TypeScript 툴킷입니다. |
| [API](https://langfuse.com/docs/api) | | 공개 API를 직접 호출합니다. OpenAPI 명세가 제공됩니다. |
| [Ollama](https://langfuse.com/docs/integrations/ollama) | 모델 (로컬) | 자신의 컴퓨터에서 오픈 소스 LLM을 손쉽게 실행할 수 있습니다. |
| [Amazon Bedrock](https://langfuse.com/docs/integrations/amazon-bedrock) | 모델 | AWS에서 기본 및 파인튜닝된 모델을 실행합니다. |
| [AutoGen](https://langfuse.com/docs/integrations/autogen) | 에이전트 프레임워크 | 분산 에이전트 구축을 위한 오픈 소스 LLM 플랫폼입니다. |
| [Flowise](https://langfuse.com/docs/integrations/flowise) | 채팅/에이전트 UI | 맞춤형 LLM 플로우를 위한 JS/TS 코드 없는(no-code) 빌더입니다. |
| [Langflow](https://langfuse.com/docs/integrations/langflow) | 채팅/에이전트 UI | react-flow를 활용하여 실험 및 프로토타이핑을 손쉽게 할 수 있도록 디자인된 LangChain용 Python 기반 UI입니다. |
| [Dify](https://langfuse.com/docs/integrations/dify) | 채팅/에이전트 UI | 코드 없는 빌더와 함께 제공되는 오픈 소스 LLM 애플리케이션 개발 플랫폼입니다. |
| [OpenWebUI](https://langfuse.com/docs/integrations/openwebui) | 채팅/에이전트 UI | 셀프 호스팅 및 로컬 모델 등 다양한 LLM 실행기를 지원하는 셀프 호스팅 LLM 채팅 웹 UI입니다. |
| [Promptfoo](https://langfuse.com/docs/integrations/promptfoo) | 도구 | 오픈 소스 LLM 테스트 플랫폼입니다. |
| [LobeChat](https://langfuse.com/docs/integrations/lobechat) | 채팅/에이전트 UI | 오픈 소스 챗봇 플랫폼입니다. |
| [Vapi](https://langfuse.com/docs/integrations/vapi) | 플랫폼 | 오픈 소스 음성 AI 플랫폼입니다. |
| [Inferable](https://langfuse.com/docs/integrations/other/inferable) | 에이전트 | 분산 에이전트 구축을 위한 오픈 소스 LLM 플랫폼입니다. |
| [Gradio](https://langfuse.com/docs/integrations/other/gradio) | 채팅/에이전트 UI | 채팅 UI와 같은 웹 인터페이스 구축을 위한 오픈 소스 Python 라이브러리입니다. |
| [Goose](https://langfuse.com/docs/integrations/goose) | 에이전트 | 분산 에이전트 구축을 위한 오픈 소스 LLM 플랫폼입니다. |
| [smolagents](https://langfuse.com/docs/integrations/smolagents) | 에이전트 | 오픈 소스 AI 에이전트 프레임워크입니다. |
| [CrewAI](https://langfuse.com/docs/integrations/crewai) | 에이전트 | 에이전트 간 협업 및 도구 사용을 위한 다중 에이전트 프레임워크입니다. |
## 🚀 빠른 시작
앱에 계측을 추가하고 Langfuse에 trace 데이터를 수집하여, LLM 호출 및 검색, 임베딩, 에이전트 동작과 같은 애플리케이션 로직을 추적해보세요. 복잡한 로그와 사용자 세션을 확인하여 디버깅할 수 있습니다.
### 1️⃣ 새 프로젝트 생성
1. [Langfuse 계정 생성](https://cloud.langfuse.com/auth/sign-up) 또는 [셀프 호스트](https://langfuse.com/self-hosting)
2. 새 프로젝트를 생성합니다.
3. 프로젝트 설정에서 새로운 API 자격 증명을 생성합니다.
### 2️⃣ 첫 번째 LLM 호출 기록하기
[`@observe()` 데코레이터](https://langfuse.com/docs/sdk/python/decorators)를 사용하면 Python LLM 애플리케이션의 추적이 매우 간편해집니다. 이 빠른 시작 예제에서는 Langfuse [OpenAI 통합](https://langfuse.com/docs/integrations/openai)을 사용하여 모든 모델 파라미터를 자동으로 캡처합니다.
> [!TIP]
> OpenAI를 사용하지 않으시다면, 다른 모델 및 프레임워크의 로그 기록 방법은 [문서](https://langfuse.com/docs/get-started#log-your-first-llm-call-to-langfuse)를 참조하세요.
```bash
pip install langfuse openai
```
```bash filename=".env"
LANGFUSE_SECRET_KEY="sk-lf-..."
LANGFUSE_PUBLIC_KEY="pk-lf-..."
LANGFUSE_HOST="https://cloud.langfuse.com" # 🇪🇺 EU region
# LANGFUSE_HOST="https://us.cloud.langfuse.com" # 🇺🇸 US region
```
```python:main.py
from langfuse import observe
from langfuse.openai import openai # OpenAI integration
@observe()
def story():
return openai.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": "What is Langfuse?"}],
).choices[0].message.content
@observe()
def main():
return story()
main()
```
### 3️⃣ Langfuse에서 trace 확인하기
Langfuse에서 LLM 호출 및 애플리케이션의 기타 로직에 대한 trace를 확인할 수 있습니다.

_[Langfuse의 공개 예제 trace](https://cloud.langfuse.com/project/cloramnkj0002jz088vzn1ja4/traces/2cec01e3-3dc2-472f-afcf-3b968cf0c1f4?timestamp=2025-02-10T14%3A27%3A30.275Z&observation=cb5ff844-07ef-41e6-b8e2-6c64344bc13b)_
> [!TIP]
>
> Langfuse의 trace에 대해 더 알아보거나 [인터랙티브 데모](https://langfuse.com/docs/demo)에서 직접 체험해보세요.
- 우리의 [문서](https://langfuse.com/docs)는 답을 찾기 위한 최적의 장소입니다. 문서가 매우 포괄적이며, 유지보수에 많은 노력을 기울이고 있습니다. GitHub를 통해 문서 수정 제안도 가능합니다.
- [Langfuse FAQ](https://langfuse.com/faq)에서는 가장 흔한 질문에 대해 답변하고 있습니다.
- 질문에 즉각적인 답변이 필요하다면 [Ask AI](https://langfuse.com/docs/ask-ai)를 사용해보세요.
지원 채널:
- **GitHub Discussions의 [공개 Q&A](https://github.com/orgs/langfuse/discussions/categories/support)** 에 질문을 남겨주세요. 가능한 한 많은 세부 사항(예: 코드 스니펫, 스크린샷, 배경 정보)을 포함해 질문해 주시기 바랍니다.
<span>Langfuse uses <a href="https://github.com/orgs/langfuse/discussions"><strong>Github Discussions</strong></a> for Support and Feature Requests.</span>
<br/>
<span>We're hiring. <a href="https://langfuse.com/careers"><strong>Join us</strong></a> in Product Engineering and Developer Relations.</span>
<span><b>We're hiring.</b> <a href="https://langfuse.com/careers"><strong>Join us</strong></a> in product engineering and technical go-to-market roles.</span>
- **LLM Observability:** Instrument your app and start ingesting traces to Langfuse ([Quickstart](https://langfuse.com/docs/get-started), [Integrations](https://langfuse.com/docs/integrations) [Tracing](https://langfuse.com/docs/tracing))
- **Langfuse UI:** Inspect and debug complex logs ([Demo](https://langfuse.com/docs/demo), [Tracing](https://langfuse.com/docs/tracing))
- **Prompt Management:** Manage, version and deploy prompts from within Langfuse ([Prompt Management](https://langfuse.com/docs/prompts/get-started))
- **Prompt Engineering:** Test and iterate on your prompts with the [LLM Playground](https://langfuse.com/docs/playground)
-**LLM Analytics:** Track metrics (cost, latency, quality) and gain insights from dashboards & data exports ([Analytics](https://langfuse.com/docs/analytics))
- **LLM Evaluations:** Collect and calculate scores for your LLM completions ([Scores & Evaluations](https://langfuse.com/docs/scores))
- Run ([Model-based evaluations](https://langfuse.com/docs/scores/model-based-evals)) and LLM-as-a-Judge within Langfuse
- Collect user feedback ([User Feedback](https://langfuse.com/docs/scores/user-feedback))
- Manually score LLM outputs in Langfuse ([Manual Scores](https://langfuse.com/docs/scores/manually))
-[LLM Application Observability](https://langfuse.com/docs/tracing): Instrument your app and start ingesting traces to Langfuse, thereby tracking LLM calls and other relevant logic in your app such as retrieval, embedding, or agent actions. Inspect and debug complex logs and user sessions. Try the interactive [demo](https://langfuse.com/docs/demo) to see this in action.
### Test
- [Prompt Management](https://langfuse.com/docs/prompts/get-started) helps you centrally manage, version control, and collaboratively iterate on your prompts. Thanks to strong caching on server and client side, you can iterate on prompts without adding latency to your application.
-**Experiments:** Track and test app behaviour before deploying a new version
- Datasets let you test expected in and output pairs and benchmark performance before deploying ([Datasets](https://langfuse.com/docs/datasets))
- Track versions and releases in your application ([Experimentation](https://langfuse.com/docs/experimentation), [Prompt Management](https://langfuse.com/docs/prompts))
-[Evaluations](https://langfuse.com/docs/scores/overview) are key to the LLM application development workflow, and Langfuse adapts to your needs. It supports LLM-as-a-judge, user feedback collection, manual labeling, and custom evaluation pipelines via APIs/SDKs.
## Get started
- [Datasets](https://langfuse.com/docs/datasets/overview) enable test sets and benchmarks for evaluating your LLM application. They support continuous improvement, pre-deployment testing, structured experiments, flexible evaluation, and seamless integration with frameworks like LangChain and LlamaIndex.
- [LLM Playground](https://langfuse.com/docs/playground) is a tool for testing and iterating on your prompts and model configurations, shortening the feedback loop and accelerating development. When you see a bad result in tracing, you can directly jump to the playground to iterate on it.
- [Comprehensive API](https://langfuse.com/docs/api): Langfuse is frequently used to power bespoke LLMOps workflows while using the building blocks provided by Langfuse via the API. OpenAPI spec, Postman collection, and typed SDKs for Python, JS/TS are available.
[→ Learn more about deploying locally](https://langfuse.com/docs/deployment/local)
# Run the langfuse docker compose
docker compose up
```
- [VM](https://langfuse.com/self-hosting/docker-compose): Run Langfuse on a single Virtual Machine using Docker Compose.
- [Kubernetes (Helm)](https://langfuse.com/self-hosting/kubernetes-helm): Run Langfuse on a Kubernetes cluster using Helm. This is the preferred production deployment.
You need a Langfuse public and secret key to get started. Sign up [here](https://cloud.langfuse.com) and find them in your project settings.
### Ingesting Data · Instrumenting Your Application · LLM Observability with Langfuse
Note: We recommend using our fully async, typed [SDKs](https://langfuse.com/docs/sdk) that allow you to instrument any LLM application with any underlying model. They are available in [Python (Decorators)](https://langfuse.com/docs/sdk/python) & [JS/TS](https://langfuse.com/docs/sdk/typescript). The SDKs will always be the most fully featured and stable way to ingest data into Langfuse.
See the [→ Quickstart](https://langfuse.com/docs/get-started) to integrate Langfuse.
@@ -122,50 +141,180 @@ See the [→ Quickstart](https://langfuse.com/docs/get-started) to integrate Lan
| [Vercel AI SDK](https://langfuse.com/docs/integrations/vercel-ai-sdk) | JS/TS | TypeScript toolkit designed to help developers build AI-powered applications with React, Next.js, Vue, Svelte, Node.js. |
| [API](https://langfuse.com/docs/api) | | Directly call the public API. OpenAPI spec available. |
| [Langflow](https://langfuse.com/docs/integrations/langflow) | Python-based UI for LangChain, designed with react-flow to provide an effortless way to experiment and prototype flows. |
| [Langflow](https://langfuse.com/docs/integrations/langflow) | Chat/Agent UI | Python-based UI for LangChain, designed with react-flow to provide an effortless way to experiment and prototype flows. |
| [Dify](https://langfuse.com/docs/integrations/dify) | Chat/Agent UI | Open source LLM app development platform with no-code builder. |
| [OpenWebUI](https://langfuse.com/docs/integrations/openwebui) | Chat/Agent UI | Self-hosted LLM Chat web ui supporting various LLM runners including self-hosted and local models. |
| [LobeChat](https://langfuse.com/docs/integrations/lobechat) | Chat/Agent UI | Open source chatbot platform. |
| [Vapi](https://langfuse.com/docs/integrations/vapi) | Platform | Open source voice AI platform. |
| [Inferable](https://langfuse.com/docs/integrations/other/inferable) | Agents | Open source LLM platform for building distributed agents. |
| [Gradio](https://langfuse.com/docs/integrations/other/gradio) | Chat/Agent UI | Open source Python library to build web interfaces like Chat UI. |
| [Goose](https://langfuse.com/docs/integrations/goose) | Agents | Open source LLM platform for building distributed agents. |
| [smolagents](https://langfuse.com/docs/integrations/smolagents) | Agents | Open source AI agents framework. |
| [CrewAI](https://langfuse.com/docs/integrations/crewai) | Agents | Multi agent framework for agent collaboration and tool use. |
## Questions and feedback
## 🚀 Quickstart
### Ideas and roadmap
Instrument your app and start ingesting traces to Langfuse, thereby tracking LLM calls and other relevant logic in your app such as retrieval, embedding, or agent actions. Inspect and debug complex logs and user sessions.
- [Discord](https://langfuse.com/discord): community support
- Privately: contact at langfuse dot com
The [`@observe()` decorator](https://langfuse.com/docs/sdk/python/decorators) makes it easy to trace any Python LLM application. In this quickstart we also use the Langfuse [OpenAI integration](https://langfuse.com/docs/integrations/openai) to automatically capture all model parameters.
## Contributing to Langfuse
> [!TIP]
> Not using OpenAI? Visit [our documentation](https://langfuse.com/docs/get-started#log-your-first-llm-call-to-langfuse) to learn how to log other models and frameworks.
- Vote on [Ideas](https://github.com/orgs/langfuse/discussions/categories/ideas)
- Raise and comment on [Issues](https://github.com/langfuse/langfuse/issues)
```bash
pip install langfuse openai
```
```bash filename=".env"
LANGFUSE_SECRET_KEY="sk-lf-..."
LANGFUSE_PUBLIC_KEY="pk-lf-..."
LANGFUSE_HOST="https://cloud.langfuse.com" # 🇪🇺 EU region
# LANGFUSE_HOST="https://us.cloud.langfuse.com" # 🇺🇸 US region
from langfuse.openai import openai # OpenAI integration
@observe()
def story():
return openai.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": "What is Langfuse?"}],
).choices[0].message.content
@observe()
def main():
return story()
main()
```
### 3️⃣ See traces in Langfuse
See your language model calls and other application logic in Langfuse.

_[Public example trace in Langfuse](https://cloud.langfuse.com/project/cloramnkj0002jz088vzn1ja4/traces/2cec01e3-3dc2-472f-afcf-3b968cf0c1f4?timestamp=2025-02-10T14%3A27%3A30.275Z&observation=cb5ff844-07ef-41e6-b8e2-6c64344bc13b)_
> [!TIP]
>
> [Learn more](https://langfuse.com/docs/tracing) about tracing in Langfuse or play with the [interactive demo](https://langfuse.com/docs/demo).
- Our [documentation](https://langfuse.com/docs) is the best place to start looking for answers. It is comprehensive, and we invest significant time into maintaining it. You can also suggest edits to the docs via GitHub.
- [Langfuse FAQs](https://langfuse.com/faq) where the most common questions are answered.
- Use "[Ask AI](https://langfuse.com/docs/ask-ai)" to get instant answers to your questions.
Support Channels:
- **Ask any question in our [public Q&A](https://github.com/orgs/langfuse/discussions/categories/support) on GitHub Discussions.** Please include as much detail as possible (e.g. code snippets, screenshots, background information) to help us understand your question.
- [Request a feature](https://github.com/orgs/langfuse/discussions/categories/ideas) on GitHub Discussions.
- [Report a Bug](https://github.com/langfuse/langfuse/issues) on GitHub Issues.
- For time-sensitive queries, ping us via the in-app chat widget.
## 🤝 Contributing
Your contributions are welcome!
- Vote on [Ideas](https://github.com/orgs/langfuse/discussions/categories/ideas) in GitHub Discussions.
- Raise and comment on [Issues](https://github.com/langfuse/langfuse/issues).
- Open a PR - see [CONTRIBUTING.md](CONTRIBUTING.md) for details on how to setup a development environment.
## License
## 🥇 License
This repository is MIT licensed, except for the `ee` folders. See [LICENSE](LICENSE) and [docs](https://langfuse.com/docs/open-source) for more details.
<img alt="Star History Chart" src="https://api.star-history.com/svg?repos=langfuse/langfuse&type=Date" style="border-radius: 15px;" />
</picture>
</a>
[**GET routes**](https://langfuse.com/docs/integrations/api) to use data in downstream applications (e.g. embedded analytics). You can also access them conveniently via the SDKs ([docs](https://langfuse.com/docs/query-traces)).
## ❤️ Open Source Projects Using Langfuse
### Security & Privacy
Top open-source Python projects that use Langfuse, ranked by stars ([Source](https://github.com/langfuse/langfuse-docs/blob/main/components-mdx/dependents)):
We take data security and privacy seriously. Please refer to our [Security and Privacy](https://langfuse.com/security) page for more information.
@@ -181,13 +330,3 @@ This helps us to:
None of the data is shared with third parties and does not include any sensitive information. We want to be super transparent about this and you can find the exact data we collect [here](/web/src/features/telemetry/index.ts).
You can opt-out by setting `TELEMETRY_ENABLED=false`.
docs:The value of the score. Must be passed as string for categorical scores, and numeric for boolean and numeric scores. Boolean score values must equal either 1 or 0 (true or false)
observationId:optional<string>
comment:optional<string>
metadata:optional<unknown>
dataType:
type:optional<ScoreDataType>
docs:When set, must match the score value's type. If not set, will be inferred from the score value or config
@@ -64,62 +67,15 @@ types:
dataType:"BOOLEAN"
configId:"1234-5678-90ab-cdef"
traceId:"cdef-1234-5678-90ab"
BaseScore:
properties:
id:string
traceId:string
name:string
source:ScoreSource
observationId:optional<string>
timestamp:datetime
createdAt:datetime
updatedAt:datetime
authorUserId:optional<string>
comment:optional<string>
configId:
type:optional<string>
docs:Reference a score config on a score. When set, config and score name must be equal and value must comply to optionally defined numerical range
NumericScore:
extends:BaseScore
properties:
value:
type:double
docs:The numeric value of the score
BooleanScore:
extends:BaseScore
properties:
value:
type:double
docs:The numeric value of the score. Equals 1 for "True" and 0 for "False"
stringValue:
type:string
docs:The string representation of the score value. Is inferred from the numeric value and equals "True" or "False"
CategoricalScore:
extends:BaseScore
properties:
value:
type:optional<double>
docs:Only defined if a config is linked. Represents the numeric category mapping of the stringValue
stringValue:
type:string
docs:The string representation of the score value. If no config is linked, can be any string. Otherwise, must map to a config category
docs:Public traces are accessible via url without login
environment:
type:optional<string>
docs:The environment from which this trace originated. Can be any lowercase alphanumeric string with hyphens and underscores that does not start with 'langfuse'.
TraceWithDetails:# GET /traces
extends:Trace
properties:
@@ -72,13 +75,16 @@ types:
type:list<ObservationsView>
docs:List of observations
scores:
type:list<Score>
type:list<ScoreV1>
docs:List of scores
Session:
properties:
id:string
createdAt:datetime
projectId:string
environment:
type:optional<string>
docs:The environment from which this session originated.
SessionWithTraces:
extends:Session
properties:
@@ -126,7 +132,7 @@ types:
docs:The output data of the observation
usage:
type:optional<Usage>
docs:The usage data of the observation
docs:(Deprecated. Use usageDetails and costDetails instead.) The usage data of the observation
level:
type:ObservationLevel
docs:The level of the observation
@@ -139,6 +145,15 @@ types:
promptId:
type:optional<string>
docs:The prompt ID associated with the observation
usageDetails:
type:optional<map<string, integer>>
docs:The usage details of the observation. Key is the name of the usage metric, value is the number of units consumed. The total key is the sum of all (non-total) usage metrics or the total value ingested.
costDetails:
type:optional<map<string, double>>
docs:The cost details of the observation. Key is the name of the cost metric, value is the cost in USD. The total key is the sum of all (non-total) cost metrics or the total value ingested.
environment:
type:optional<string>
docs:The environment from which this observation originated. Can be any lowercase alphanumeric string with hyphens and underscores that does not start with 'langfuse'.
ObservationsView:
extends:Observation
@@ -163,13 +178,13 @@ types:
docs:The total price in USD.
calculatedInputCost:
type:optional<double>
docs:The calculated cost of the input in USD
docs:(Deprecated. Use usageDetails and costDetails instead.) The calculated cost of the input in USD
calculatedOutputCost:
type:optional<double>
docs:The calculated cost of the output in USD
docs:(Deprecated. Use usageDetails and costDetails instead.) The calculated cost of the output in USD
calculatedTotalCost:
type:optional<double>
docs:The calculated total cost in USD
docs:(Deprecated. Use usageDetails and costDetails instead.) The calculated total cost in USD
latency:
type:optional<double>
docs:The latency in seconds.
@@ -178,7 +193,7 @@ types:
docs:The time to the first token in seconds
Usage:
docs:Standard interface for usage and cost
docs:(Deprecated. Use usageDetails and costDetails instead.) Standard interface for usage and cost
properties:
input:
docs:Number of input units (e.g. tokens)
@@ -225,7 +240,7 @@ types:
properties:
value:double
label:string
BaseScore:
BaseScoreV1:
properties:
id:string
traceId:string
@@ -237,12 +252,77 @@ types:
updatedAt:datetime
authorUserId:optional<string>
comment:optional<string>
metadata:optional<unknown>
configId:
type:optional<string>
docs:Reference a score config on a score. When set, config and score name must be equal and value must comply to optionally defined numerical range
queueId:
type:optional<string>
docs:Reference an annotation queue on a score. Populated if the score was initially created in an annotation queue.
environment:
type:optional<string>
docs:The environment from which this score originated. Can be any lowercase alphanumeric string with hyphens and underscores that does not start with 'langfuse'.
NumericScoreV1:
extends:BaseScoreV1
properties:
value:
type:double
docs:The numeric value of the score
BooleanScoreV1:
extends:BaseScoreV1
properties:
value:
type:double
docs:The numeric value of the score. Equals 1 for "True" and 0 for "False"
stringValue:
type:string
docs:The string representation of the score value. Is inferred from the numeric value and equals "True" or "False"
CategoricalScoreV1:
extends:BaseScoreV1
properties:
value:
type:optional<double>
docs:Only defined if a config is linked. Represents the numeric category mapping of the stringValue
stringValue:
type:string
docs:The string representation of the score value. If no config is linked, can be any string. Otherwise, must map to a config category
ScoreV1:
discriminant:"dataType"
union:
NUMERIC:
type:NumericScoreV1
docs:"Score with NUMERIC data type"
CATEGORICAL:
type:CategoricalScoreV1
docs:"Score with CATEGORICAL data type"
BOOLEAN:
type:BooleanScoreV1
docs:"Score with BOOLEAN data type"
BaseScore:
properties:
id:string
traceId:optional<string>
sessionId:optional<string>
observationId:optional<string>
datasetRunId:optional<string>
name:string
source:ScoreSource
timestamp:datetime
createdAt:datetime
updatedAt:datetime
authorUserId:optional<string>
comment:optional<string>
metadata:optional<unknown>
configId:
type:optional<string>
docs:Reference a score config on a score. When set, config and score name must be equal and value must comply to optionally defined numerical range
queueId:
type:optional<string>
docs:Reference an annotation queue on a score. Populated if the score was initially created in an annotation queue.
environment:
type:optional<string>
docs:The environment from which this score originated. Can be any lowercase alphanumeric string with hyphens and underscores that does not start with 'langfuse'.
NumericScore:
extends:BaseScore
properties:
@@ -372,18 +452,18 @@ types:
type:string
startDate:
docs:Apply only to generations which are newer than this ISO date.
type:optional<date>
type:optional<datetime>
unit:
docs:Unit used by this model.
type:ModelUsageUnit
type:optional<ModelUsageUnit>
inputPrice:
docs:Price (USD) per input unit
docs:Deprecated. See 'prices' instead. Price (USD) per input unit
type:optional<double>
outputPrice:
docs:Price (USD) per output unit
docs:Deprecated. See 'prices' instead. Price (USD) per output unit
type:optional<double>
totalPrice:
docs:Price (USD) per total unit. Cannot be set if input or output price is set.
docs:Deprecated. See 'prices' instead. Price (USD) per total unit. Cannot be set if input or output price is set.
type:optional<double>
tokenizerId:
docs:Optional. Tokenizer to be applied to observations which match to this model. See docs for more details.
Batched ingestion for Langfuse Tracing. If you want to use tracing via the API, such as to build your own Langfuse client implementation, this is the only API route you need to implement.
Batched ingestion for Langfuse Tracing.
If you want to use tracing via the API, such as to build your own Langfuse client implementation, this is the only API route you need to implement.
Within each batch, there can be multiple events.
Each event has a type, an id, a timestamp, metadata and a body.
Internally, we refer to this as the "event envelope" as it tells us something about the event but not the trace.
We use the event id within this envelope to deduplicate messages to avoid processing the same event twice, i.e. the event id should be unique per request.
The event.body.id is the ID of the actual trace and will be used for updates and will be visible within the Langfuse App.
I.e. if you want to update a trace, you'd use the same body id, but separate event IDs.
Notes:
- Introduction to data model: https://langfuse.com/docs/tracing-data-model
- Batch sizes are limited to 3.5 MB in total. You need to adjust the number of events per batch accordingly.
- The API does not return a 4xx status code for input errors. Instead, it responds with a 207 status code, which includes a list of the encountered errors.
docs:The value of the score. Must be passed as string for categorical scores, and numeric for boolean and numeric scores. Boolean score values must equal either 1 or 0 (true or false)
observationId:optional<string>
comment:optional<string>
metadata:optional<unknown>
dataType:
type:optional<commons.ScoreDataType>
docs:When set, must match the score value's type. If not set, will be inferred from the score value or config
@@ -233,6 +317,14 @@ types:
dataType:"BOOLEAN"
configId:"1234-5678-90ab-cdef"
traceId:"cdef-1234-5678-90ab"
- value:
name:"contextrelevant"
value:"not relevant"
sessionId:"abyt-1234-5678-80ab"
- value:
name:"hallucination"
value:0
datasetRunId:"7891-5678-90ab-hijk"
BaseEvent:
properties:
@@ -312,3 +404,28 @@ types:
properties:
successes:list<IngestionSuccess>
errors:list<IngestionError>
OpenAICompletionUsageSchema:
docs:OpenAI Usage schema from (Chat-)Completion APIs
docs:Get all memberships for the organization associated with the API key (requires organization-scoped API key)
method:GET
path:/organizations/memberships
response:MembershipsResponse
updateOrganizationMembership:
docs:Create or update a membership for the organization associated with the API key (requires organization-scoped API key)
method:PUT
path:/organizations/memberships
request:MembershipRequest
response:MembershipResponse
getProjectMemberships:
docs:Get all memberships for a specific project (requires organization-scoped API key)
method:GET
path:/projects/{projectId}/memberships
path-parameters:
projectId:string
response:MembershipsResponse
updateProjectMembership:
docs:Create or update a membership for a specific project (requires organization-scoped API key). The user must already be a member of the organization.
method:PUT
path:/projects/{projectId}/memberships
path-parameters:
projectId:string
request:MembershipRequest
response:MembershipResponse
getOrganizationProjects:
docs:Get all projects for the organization associated with the API key (requires organization-scoped API key)
docs:Get SCIM Service Provider Configuration (requires organization-scoped API key)
method:GET
path:/ServiceProviderConfig
response:ServiceProviderConfig
getResourceTypes:
docs:Get SCIM Resource Types (requires organization-scoped API key)
method:GET
path:/ResourceTypes
response:ResourceTypesResponse
getSchemas:
docs:Get SCIM Schemas (requires organization-scoped API key)
method:GET
path:/Schemas
response:SchemasResponse
listUsers:
docs:List users in the organization (requires organization-scoped API key)
method:GET
path:/Users
request:
name:ListUsersRequest
query-parameters:
filter:
docs:Filter expression (e.g. userName eq "value")
type:optional<string>
startIndex:
docs:1-based index of the first result to return (default 1)
type:optional<integer>
count:
docs:Maximum number of results to return (default 100)
type:optional<integer>
response:ScimUsersListResponse
createUser:
docs:Create a new user in the organization (requires organization-scoped API key)
method:POST
path:/Users
request:
name:CreateUserRequest
body:
properties:
userName:
docs:User's email address (required)
type:string
name:
docs:User's name information
type:ScimName
emails:
docs:User's email addresses
type:optional<list<ScimEmail>>
active:
docs:Whether the user is active
type:optional<boolean>
password:
docs:Initial password for the user
type:optional<string>
response:ScimUser
getUser:
docs:Get a specific user by ID (requires organization-scoped API key)
method:GET
path:/Users/{userId}
path-parameters:
userId:string
response:ScimUser
deleteUser:
docs:Remove a user from the organization (requires organization-scoped API key). Note that this only removes the user from the organization but does not delete the user entity itself.
docs:Create a score (supports both trace and session scores)
method:POST
path:/scores
request:CreateScoreRequest
response:CreateScoreResponse
get:
docs:Get a list of scores
method:GET
path:/scores
request:
name:GetScoresRequest
query-parameters:
page:
type:optional<integer>
docs:Page number, starts at 1.
limit:
type:optional<integer>
docs:Limit of items per page. If you encounter api issues due to too large page sizes, try to reduce the limit.
userId:
type:optional<string>
docs:Retrieve only scores with this userId associated to the trace.
name:
type:optional<string>
docs:Retrieve only scores with this name.
fromTimestamp:
type:optional<datetime>
docs:Optional filter to only include scores created on or after a certain datetime (ISO 8601)
toTimestamp:
type:optional<datetime>
docs:Optional filter to only include scores created before a certain datetime (ISO 8601)
source:
type:optional<commons.ScoreSource>
docs:Retrieve only scores from a specific source.
operator:
type:optional<string>
docs:Retrieve only scores with <operator> value.
value:
type:optional<double>
docs:Retrieve only scores with <operator> value.
scoreIds:
type:optional<string>
docs:Comma-separated list of score IDs to limit the results to.
configId:
type:optional<string>
docs:Retrieve only scores with a specific configId.
queueId:
type:optional<string>
docs:Retrieve only scores with a specific annotation queueId.
dataType:
type:optional<commons.ScoreDataType>
docs:Retrieve only scores with a specific dataType.
traceTags:
type:optional<list<string>>
allow-multiple:true
docs:Only scores linked to traces that include all of these tags will be returned.
response:GetScoresResponse
get-by-id:
docs:Get a score
method:GET
path:/scores/{scoreId}
path-parameters:
scoreId:
type:string
docs:The unique langfuse identifier of a score
response:commons.Score
delete:
docs:Delete a score
docs:Delete a score (supports both trace and session scores)
method:DELETE
path:/scores/{scoreId}
path-parameters:
@@ -84,13 +24,19 @@ types:
CreateScoreRequest:
properties:
id:optional<string>
traceId:string
traceId:optional<string>
sessionId:optional<string>
observationId:optional<string>
datasetRunId:optional<string>
name:string
value:
type:commons.CreateScoreValue
docs:The value of the score. Must be passed as string for categorical scores, and numeric for boolean and numeric scores. Boolean score values must equal either 1 or 0 (true or false)
observationId:optional<string>
comment:optional<string>
metadata:optional<unknown>
environment:
type:optional<string>
docs:The environment of the score. Can be any lowercase alphanumeric string with hyphens and underscores that does not start with 'langfuse'.
dataType:
type:optional<commons.ScoreDataType>
docs:The data type of the score. When passing a configId this field is inferred. Otherwise, this field must be passed or will default to numeric.
@@ -113,10 +59,12 @@ types:
dataType:"NUMERIC"
configId:"9203-4567-89ab-cdef"
traceId:"cdef-1234-5678-90ab"
environment:"test"
- value:
name:"toxicity"
value:"not toxic"
traceId:"cdef-1234-5678-90ab"
environment:"production"
- value:
name:"correctness"
value:"partially correct"
@@ -139,38 +87,3 @@ types:
id:
type:string
docs:The id of the created object in Langfuse
GetScoresResponseTraceData:
properties:
userId:
type:optional<string>
docs:The user ID associated with the trace referenced by score
tags:
type:optional<list<string>>
docs:A list of tags associated with the trace referenced by score
docs:Optional filter to only include sessions created before a certain datetime (ISO 8601)
environment:
type:optional<string>
allow-multiple:true
docs:Optional filter for sessions where the environment is one of the provided values.
response:PaginatedSessions
get:
docs:Get a session. Please note that `traces` on this endpoint are not paginated, if you plan to fetch large sessions, consider `GET /api/public/traces?sessionId=<sessionId>`
docs:The unique langfuse identifier of the trace to delete
response:DeleteTraceResponse
list:
docs:Get list of traces
method:GET
@@ -50,13 +59,35 @@ service:
release:
type:optional<string>
docs:Optional filter to only include traces with a certain release.
environment:
type:optional<string>
allow-multiple:true
docs:Optional filter for traces where the environment is one of the provided values.
fields:
type:optional<string>
docs:"Comma-separated list of fields to include in the response. Available field groups are 'core' (always included), 'io' (input, output, metadata), 'scores', 'observations', 'metrics'. If not provided, all fields are included. Example: 'core,scores,metrics'"
"dx":"pnpm i && pnpm run infra:dev:prune && pnpm run infra:dev:up --pull always && pnpm --filter=shared run db:reset && pnpm --filter=shared run ch:reset && pnpm --filter=shared run db:seed:examples && pnpm run dev",
"dx-f":"pnpm i && pnpm run infra:dev:prune && pnpm run infra:dev:up --pull always && pnpm --filter=shared run db:reset -f && pnpm --filter=shared run ch:reset && pnpm --filter=shared run db:seed:examples && pnpm run dev",
"dx-f":"pnpm i && pnpm run infra:dev:prune && pnpm run infra:dev:up --pull always && pnpm --filter=shared run db:reset -f && SKIP_CONFIRM=1 pnpm --filter=shared run ch:reset && pnpm --filter=shared run db:seed:examples && pnpm run dev",
"dx:skip-infra":"pnpm i && pnpm --filter=shared run db:reset && pnpm --filter=shared run ch:reset && pnpm --filter=shared run db:seed:examples && pnpm run dev",
"build":"turbo run build",
"start":"turbo run start",
"dev":"turbo run dev",
"dev:worker":"turbo run dev --filter=worker",
"dev:web":"turbo run dev --filter=web",
"dev:web-turbo":"turbo run dev --filter=web -- --turbo",
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.