* style(ui): align styling of comment button on sessions page
* style(dashboard): score analytics charts full width for mobile
* style(markdown): fix overflow behavior on sessions page for mobile
* chore(ui): populate annotation drawer default form state on render
* style(ui): add width and quality properties to markdown image loader
* style(ui): markdown fix list marker alignment
* fix: more state management enhancement for annotation drawer empty keys
* less padding for code blocks
* fix corners with overflow-hidden
* corner of regular json view
* use non-mono font for md viewer
* support inline code and code blocks without language
* style(annotation): validate numbers in annotation onKeyUp
* style(drawer): round top left corner
* style: mark discard comment as destructive
* refactor: extract config categories to type in shared
* style: improve annotation range display in configs table
* style: ensure form caters to a lot of categories
* style: show entire description on screen
* style(scores_table): allow hiding author
* push
* push
* push
* push
* perf(sessions): improve `filterOptions` api
* limit to 1000 results
* fix: remove unintentional line from `traces.filterOptions`
* feat(table_filters): add free text search option
* push
* show custom select option in `userIds` column on sessions table
* style: fix padding
* style: improve styling
* simplify
* push
* feat(multi-select): show `customSelect` also when "no results found"
* improve ux across delete buttons
* updated delete button on models screen
* trpc errors are logged to console by default
* revert to pointer-events-none on disabled button
---------
Co-authored-by: Mish Ushakov <mishushakov@users.noreply.github.com>
Co-authored-by: Marc Klingen <git@marcklingen.com>
* feat: add timestamp filter to user table
* users table definition instead of reusing traces
* add filter to total as well to make table state consistent
---------
Co-authored-by: Marc Klingen <git@marcklingen.com>
* feat: add author user id column to scores table
* feat: add author user id to scores created via trpc router
* feat: add author user id to scores created in seeder
* fixup
* add: index to author user id on scores
* feat: annotation config schema
* add: migration
* add: migrations in order
* fixup after merge
* fix: replicate to worker/types
* fix: address feedback
* rn: to scoreConfig
* push
* refactor: rn `ScoreDataType` to `ScoreConfigDataType`
* fix
* fix: scores server test
* context: order of returned data can vary based on how data is retrieved. Likely re-ordered due to new index on configId in scores table
* fix: re-generate types for correct alphabetical order
* fix: remodel migrations to match new schema
* feat: rm backfill migration for data_type on scores
* make data_type column optional on scores
* push: regenerate types
* push
* feat: make dataType on scores required again
* rename: scores add data type migration
* re-run migrations
* push
* feat: add author user id column to scores table
* feat: add author user id to scores created via trpc router
* feat: add author user id to scores created in seeder
* fixup
* add: index to author user id on scores
* replicate to worker/types
---------
Co-authored-by: Marc Klingen <git@marcklingen.com>
* feat: move scores on trace and observation preview to different tab
* refactor: extract current table logic into `ScoresTablePreview`
* fix: rm unused footer
* refactor: scores preview table
* fix: tab state needs to be reset upon changing currentObservationId
* refactor: reorder for readability
* refactor: rm column visibility as not needed
* fix: rm pagination
* refactor: rn details to preview
* feat: restyle tabs
* feat: reuse scores table
* feat: hide columns
* feat: add source column to scores table
* feat: add user name and avatar to scores table
* adjust seeder
* feat: show score value in preview tab
* fix: don't show comment if empty string
* feat: add score source to preview UI
* feat: add tooltip
* feat: don't use user image and name in scores table
* feat: hide userId column in trace and observation preview
* adjust seeder to reuse image in bucket
* fix: pass table name to save column visibility in local storage
* revert: changes to seed file
* feat: misc UI improvements
* fixup: first draft add `is not` filter to table
* feat: show all nested scores for trace scores
* fix: misc UI
* ui: use `primary-accent` color
* adjust padding/margins
* adjust paddings
---------
Co-authored-by: Marc Klingen <git@marcklingen.com>
App["Langfuse Application (Docker or Serverless)"]
App["Langfuse Application"]
end
subgraph i2["Database Network"]
DB["Postgres Database"]
@@ -149,8 +198,11 @@ We built a monorepo using [pnpm](https://pnpm.io/motivation) and [turbo](https:/
Requirements
- Node.js 20 as specified in the [.nvmrc](.nvmrc)
- Pnpm v.9.5.0
- Docker to run the database locally
**Note:** You can also simply run Langfuse in a **GitHub Codespace** via the provided devcontainer. To do this, click on the green "Code" button in the top right corner of the repository and select "Open with Codespaces".
**Steps**
1. Fork the repository and clone it locally
@@ -191,6 +243,16 @@ Requirements
pnpm run dev
```
7. Open the web app in the browser:
http://localhost:3000
8. Log in as a test user (after you ran `db:seed` command):
Username: demo@langfuse.com
Password: password
## Monorepo quickstart
- Available packages and their dependencies
@@ -290,7 +352,77 @@ When a new release is tagged on the `main` branch (excluding prereleases), it tr
1. The Docker image is published to GitHub Packages with the version number and `latest` tag.
2. The deployment is carried out on Langfuse Cloud. This is done by force pushing the `main` branch to the `production` branch during every release, using the [`release.yml`](.github/workflows/release.yml) GitHub Action.
## Using secrets stored in 1Password
## Theming
At Langfuse, we utilize CSS variables to manage our theme settings across the platform.
Our approach leverages separate CSS variables for backgrounds (--background) and foregrounds (--foreground), fully adhering to the [shadcn/ui](https://ui.shadcn.com/docs/theming) color conventions. The background suffix can be omitted if the variable is used for the background color of the component. We recommend using HSL values for these colors to enhance consistency and customization. There is no need to manually handle dark mode styling with "dark:" prefixes, as next-themes automatically manages the theme switching.
Given the following CSS variables:
```
--primary: 222.2 47.4% 11.2%; // e.g. background-color
--primary-foreground: 210 40% 98%; // e.g. text-color
```
The background color of the following component will be `hsl(var(--primary))` and the foreground color will be `hsl(var(--primary-foreground))`.
<span>Langfuse uses <a href="https://github.com/orgs/langfuse/discussions"><strong>Github Discussions</strong></a> for Support and Feature Requests.</span>
<br/>
<span>We're hiring. <a href="https://langfuse.com/careers"><strong>Join us</strong></a> in Backend Engineering, Product Engineering, and Developer Relations.</span>
- **Observability:** Instrument your app and start ingesting traces to Langfuse ([Quickstart](https://langfuse.com/docs/get-started), [Integrations](https://langfuse.com/docs/integrations) [Tracing](https://langfuse.com/docs/tracing))
- **LLM Observability:** Instrument your app and start ingesting traces to Langfuse ([Quickstart](https://langfuse.com/docs/get-started), [Integrations](https://langfuse.com/docs/integrations) [Tracing](https://langfuse.com/docs/tracing))
- **Langfuse UI:** Inspect and debug complex logs ([Demo](https://langfuse.com/docs/demo), [Tracing](https://langfuse.com/docs/tracing))
- **Prompt Management:** Manage, version and deploy prompts from within Langfuse ([Prompt Management](https://langfuse.com/docs/prompts/get-started))
- **Prompt Engineering:** Test and iterate on your prompts with the [LLM Playground](https://langfuse.com/docs/playground)
### Monitor
- **Analytics:** Track metrics (cost, latency, quality) and gain insights from dashboards & data exports ([Analytics](https://langfuse.com/docs/analytics))
- **Evals:** Collect and calculate scores for your LLM completions ([Scores & Evaluations](https://langfuse.com/docs/scores))
- Run model-based evaluations ([Model-based evaluations](https://langfuse.com/docs/scores/model-based-evals)) within Langfuse
- **LLM Analytics:** Track metrics (cost, latency, quality) and gain insights from dashboards & data exports ([Analytics](https://langfuse.com/docs/analytics))
- **LLM Evaluations:** Collect and calculate scores for your LLM completions ([Scores & Evaluations](https://langfuse.com/docs/scores))
- Run ([Model-based evaluations](https://langfuse.com/docs/scores/model-based-evals)) and LLM-as-a-Judge within Langfuse
- Collect user feedback ([User Feedback](https://langfuse.com/docs/scores/user-feedback))
- Manually score observations in Langfuse ([Manual Scores](https://langfuse.com/docs/scores/manually))
- Manually score LLM outputs in Langfuse ([Manual Scores](https://langfuse.com/docs/scores/manually))
### Test
@@ -70,6 +76,8 @@ Managed deployment by the Langfuse team, generous free-tier (hobby plan), no cre
You need a Langfuse public and secret key to get started. Sign up [here](https://cloud.langfuse.com) and find them in your project settings.
### Ingesting Data · Instrumenting Your Application
### Ingesting Data · Instrumenting Your Application · LLM Observability with Langfuse
Note: We recommend using our fully async, typed [SDKs](https://langfuse.com/docs/sdk) that allow you to instrument any LLM application with any underlying model. They are available in [Python (Decorators)](https://langfuse.com/docs/sdk/python) & [JS/TS](https://langfuse.com/docs/sdk/typescript). The SDKs will always be the most fully featured and stable way to ingest data into Langfuse.
@@ -104,32 +112,35 @@ You may want to use another integration to get started quickly or implement a us
See the [→ Quickstart](https://langfuse.com/docs/get-started) to integrate Langfuse.
| [LiteLLM](/docs/integrations/litellm) | Python, JS/TS (proxy only) | Use any LLM as a drop in replacement for GPT. Use Azure, OpenAI, Cohere, Anthropic, Ollama, VLLM, Sagemaker, HuggingFace, Replicate (100+ LLMs). |
| [API](/docs/api) | | Directly call the public API. OpenAPI spec available. |
External projects/packages that integrate with Langfuse:
| [LiteLLM](https://langfuse.com/docs/integrations/litellm) | Use any LLM as a drop in replacement for GPT. Use Azure, OpenAI, Cohere, Anthropic, Ollama, VLLM, Sagemaker, HuggingFace, Replicate (100+ LLMs). |
| [Langflow](https://langfuse.com/docs/integrations/langflow) | Python-based UI for LangChain, designed with react-flow to provide an effortless way to experiment and prototype flows. |
| [Superagent](https://langfuse.com/docs/integrations/superagent) | Open Source AI Assistant Framework & API for prototyping and deployment of agents. |
| [Langflow](/docs/integrations/langflow) | Python-based UI for LangChain, designed with react-flow to provide an effortless way to experiment and prototype flows. |
@@ -153,7 +164,7 @@ This repository is MIT licensed, except for the `ee` folders. See [LICENSE](LICE
### GET API to export your data
[**GET routes**](https://langfuse.com/docs/integrations/api) to use data in downstream applications (e.g. embedded analytics).
[**GET routes**](https://langfuse.com/docs/integrations/api) to use data in downstream applications (e.g. embedded analytics). You can also access them conveniently via the SDKs ([docs](https://langfuse.com/docs/query-traces)).
docs:The value of the score. Must be passed as string for categorical scores, and numeric for boolean and numeric scores. Boolean score values must equal either 1 or 0 (true or false)
observationId:optional<string>
comment:optional<string>
Score:
dataType:
type:optional<ScoreDataType>
docs:When set, must match the score value's type. If not set, will be inferred from the score value or config
configId:
type:optional<string>
docs:Reference a score config on a score. When set, the score name must equal the config name and scores must comply with the config's range and data type. For categorical scores, the value must map to a config category. Numeric scores might be constrained by the score config's max and min values
examples:
- value:
name:"novelty"
value:0.9
traceId:"cdef-1234-5678-90ab"
- value:
name:"consistency"
value:1.2
dataType:"NUMERIC"
traceId:"cdef-1234-5678-90ab"
- value:
name:"accuracy"
value:0.9
dataType:"NUMERIC"
configId:"9203-4567-89ab-cdef"
traceId:"cdef-1234-5678-90ab"
- value:
name:"toxicity"
value:"not toxic"
traceId:"cdef-1234-5678-90ab"
- value:
name:"correctness"
value:"partially correct"
dataType:"CATEGORICAL"
configId:"1234-5678-90ab-cdef"
traceId:"cdef-1234-5678-90ab"
- value:
name:"hallucination"
value:0
dataType:"BOOLEAN"
traceId:"cdef-1234-5678-90ab"
- value:
name:"helpfulness"
value:1
dataType:"BOOLEAN"
configId:"1234-5678-90ab-cdef"
traceId:"cdef-1234-5678-90ab"
BaseScore:
properties:
id:string
traceId:string
name:string
value:double
source:ScoreSource
observationId:optional<string>
timestamp:datetime
createdAt:datetime
updatedAt:datetime
authorUserId:optional<string>
comment:optional<string>
configId:
type:optional<string>
docs:Reference a score config on a score. When set, config and score name must be equal and value must comply to optionally defined numerical range
NumericScore:
extends:BaseScore
properties:
value:
type:double
docs:The numeric value of the score
BooleanScore:
extends:BaseScore
properties:
value:
type:double
docs:The numeric value of the score. Equals 1 for "True" and 0 for "False"
stringValue:
type:string
docs:The string representation of the score value. Is inferred from the numeric value and equals "True" or "False"
CategoricalScore:
extends:BaseScore
properties:
value:
type:optional<double>
docs:Only defined if a config is linked. Represents the numeric category mapping of the stringValue
stringValue:
type:string
docs:The string representation of the score value. If no config is linked, can be any string. Otherwise, must map to a config category
Score:
discriminant:"dataType"
union:
NUMERIC:
type:NumericScore
docs:"Score with NUMERIC data type"
CATEGORICAL:
type:CategoricalScore
docs:"Score with CATEGORICAL data type"
BOOLEAN:
type:BooleanScore
docs:"Score with BOOLEAN data type"
ScoreSource:
enum:
- ANNOTATION
- API
- EVAL
ScoreDataType:
enum:
- NUMERIC
- CATEGORICAL
- BOOLEAN
CreateScoreValue:
discriminated:false
union:
- double
- string
docs:The value of the score. Must be passed as string for categorical scores, and numeric for boolean and numeric scores
docs:The input data of the trace. Can be any JSON.
output:
type:optional<unknown>
docs:The output data of the trace. Can be any JSON.
sessionId:
type:optional<string>
docs:The session identifier associated with the trace
release:
type:optional<string>
docs:The release version of the application when the trace was created
version:
type:optional<string>
docs:The version of the trace
userId:
type:optional<string>
docs:The user identifier associated with the trace
metadata:
type:optional<unknown>
docs:The metadata associated with the trace. Can be any JSON.
tags:
type:optional<list<string>>
docs:The tags associated with the trace. Can be an array of strings or null.
public:
type:optional<boolean>
docs:Public traces are accessible via url without login
@@ -42,11 +62,18 @@ types:
htmlPath:
type:string
docs:Path of trace in Langfuse UI
latency:
type:double
docs:Latency of trace in seconds
totalCost:
type:double
docs:Cost of trace in USD
observations:list<ObservationsView>
scores:list<Score>
observations:
type:list<ObservationsView>
docs:List of observations
scores:
type:list<Score>
docs:List of scores
Session:
properties:
id:string
@@ -58,36 +85,97 @@ types:
traces:list<Trace>
Observation:
properties:
id:string
traceId:optional<string>
type:string
name:optional<string>
startTime:datetime
endTime:optional<datetime>
completionStartTime:optional<datetime>
model:optional<string>
modelParameters:optional<map<string, MapValue>>
input:optional<unknown>
version:optional<string>
metadata:optional<unknown>
output:optional<unknown>
usage:optional<Usage>
level:ObservationLevel
statusMessage:optional<string>
parentObservationId:optional<string>
promptId:optional<string>
id:
type:string
docs:The unique identifier of the observation
traceId:
type:optional<string>
docs:The trace ID associated with the observation
type:
type:string
docs:The type of the observation
name:
type:optional<string>
docs:The name of the observation
startTime:
type:datetime
docs:The start time of the observation
endTime:
type:optional<datetime>
docs:The end time of the observation.
completionStartTime:
type:optional<datetime>
docs:The completion start time of the observation
model:
type:optional<string>
docs:The model used for the observation
modelParameters:
type:optional<map<string, MapValue>>
docs:The parameters of the model used for the observation
input:
type:optional<unknown>
docs:The input data of the observation
version:
type:optional<string>
docs:The version of the observation
metadata:
type:optional<unknown>
docs:Additional metadata of the observation
output:
type:optional<unknown>
docs:The output data of the observation
usage:
type:optional<Usage>
docs:The usage data of the observation
level:
type:ObservationLevel
docs:The level of the observation
statusMessage:
type:optional<string>
docs:The status message of the observation
parentObservationId:
type:optional<string>
docs:The parent observation ID
promptId:
type:optional<string>
docs:The prompt ID associated with the observation
ObservationsView:
extends:Observation
properties:
modelId:optional<string>
inputPrice:optional<double>
outputPrice:optional<double>
totalPrice:optional<double>
calculatedInputCost:optional<double>
calculatedOutputCost:optional<double>
calculatedTotalCost:optional<double>
latency:optional<double>
promptName:
type:optional<string>
docs:The name of the prompt associated with the observation
promptVersion:
type:optional<integer>
docs:The version of the prompt associated with the observation
modelId:
type:optional<string>
docs:The unique identifier of the model
inputPrice:
type:optional<double>
docs:The price of the input in USD
outputPrice:
type:optional<double>
docs:The price of the output in USD.
totalPrice:
type:optional<double>
docs:The total price in USD.
calculatedInputCost:
type:optional<double>
docs:The calculated cost of the input in USD
calculatedOutputCost:
type:optional<double>
docs:The calculated cost of the output in USD
calculatedTotalCost:
type:optional<double>
docs:The calculated total cost in USD
latency:
type:optional<double>
docs:The latency in seconds.
timeToFirstToken:
type:optional<double>
docs:The time to the first token in seconds
Usage:
docs:Standard interface for usage and cost
@@ -111,18 +199,91 @@ types:
totalCost:
docs:USD total cost, defaults to input+output
type:optional<double>
Score:
ScoreConfig:
docs:Configuration for a score
properties:
id:string
name:string
createdAt:datetime
updatedAt:datetime
projectId:string
dataType:ScoreDataType
isArchived:
type:boolean
docs:Whether the score config is archived. Defaults to false
minValue:
type:optional<double>
docs:Sets minimum value for numerical scores. If not set, the minimum value defaults to -∞
maxValue:
type:optional<double>
docs:Sets maximum value for numerical scores. If not set, the maximum value defaults to +∞
categories:
type:optional<list<ConfigCategory>>
docs:Configures custom categories for categorical scores
description:optional<string>
ConfigCategory:
properties:
value:double
label:string
BaseScore:
properties:
id:string
traceId:string
name:string
value:double
source:ScoreSource
observationId:optional<string>
timestamp:datetime
createdAt:datetime
updatedAt:datetime
authorUserId:optional<string>
comment:optional<string>
DatasetCore:
configId:
type:optional<string>
docs:Reference a score config on a score. When set, config and score name must be equal and value must comply to optionally defined numerical range
NumericScore:
extends:BaseScore
properties:
value:
type:double
docs:The numeric value of the score
BooleanScore:
extends:BaseScore
properties:
value:
type:double
docs:The numeric value of the score. Equals 1 for "True" and 0 for "False"
stringValue:
type:string
docs:The string representation of the score value. Is inferred from the numeric value and equals "True" or "False"
CategoricalScore:
extends:BaseScore
properties:
value:
type:optional<double>
docs:Only defined if a config is linked. Represents the numeric category mapping of the stringValue
stringValue:
type:string
docs:The string representation of the score value. If no config is linked, can be any string. Otherwise, must map to a config category
Score:
discriminant:"dataType"
union:
NUMERIC:
type:NumericScore
docs:"Score with NUMERIC data type"
CATEGORICAL:
type:CategoricalScore
docs:"Score with CATEGORICAL data type"
BOOLEAN:
type:BooleanScore
docs:"Score with BOOLEAN data type"
CreateScoreValue:
discriminated:false
union:
- double
- string
docs:The value of the score. Must be passed as string for categorical scores, and numeric for boolean and numeric scores
Dataset:
properties:
id:string
name:string
@@ -131,21 +292,6 @@ types:
projectId:string
createdAt:datetime
updatedAt:datetime
DatasetWithReferences:# GET datasets
extends:DatasetCore
properties:
items:
type:list<string>
docs:list of dataset item ids
runs:
type:list<string>
docs:list of dataset run names
Dataset:# GET datasets/[name]
docs:Dataset including all items
extends:DatasetCore
properties:
items:list<DatasetItem>
runs:list<string>
DatasetItem:
properties:
id:string
@@ -171,15 +317,67 @@ types:
updatedAt:datetime
DatasetRun:
properties:
id:string
name:string
description:optional<string>
metadata:optional<unknown>
datasetId:string
datasetName:string
createdAt:datetime
updatedAt:datetime
id:
type:string
docs:Unique identifier of the dataset run
name:
type:string
docs:Name of the dataset run
description:
type:optional<string>
docs:Description of the run
metadata:
type:optional<unknown>
docs:Metadata of the dataset run
datasetId:
type:string
docs:Id of the associated dataset
datasetName:
type:string
docs:Name of the associated dataset
createdAt:
type:datetime
docs:The date and time when the dataset run was created
updatedAt:
type:datetime
docs:The date and time when the dataset run was last updated
DatasetRunWithItems:
extends:DatasetRun
properties:
datasetRunItems:list<DatasetRunItem>
Model:
docs:Model definition used for transforming usage into USD cost and/or tokenization.
properties:
id:string
modelName:
docs:"Name of the model definition. If multiple with the same name exist, they are applied in the following order: (1) custom over built-in, (2) newest according to startTime where model.startTime<observation.startTime"
type:string
matchPattern:
docs:"Regex pattern which matches this model definition to generation.model. Useful in case of fine-tuned models. If you want to exact match, use `(?i)^modelname$`"
type:string
startDate:
docs:Apply only to generations which are newer than this ISO date.
type:optional<date>
unit:
docs:Unit used by this model.
type:ModelUsageUnit
inputPrice:
docs:Price (USD) per input unit
type:optional<double>
outputPrice:
docs:Price (USD) per output unit
type:optional<double>
totalPrice:
docs:Price (USD) per total unit. Cannot be set if input or output price is set.
type:optional<double>
tokenizerId:
docs:Optional. Tokenizer to be applied to observations which match to this model. See docs for more details.
type:optional<string>
tokenizerConfig:
docs:Optional. Configuration for the selected tokenizer. Needs to be JSON. See docs for more details.
Batched ingestion for Langfuse Tracing. If you want to use tracing via the API, such as to build your own Langfuse client implementation, this is the only API route you need to implement.
Notes:
- Batch sizes are limited to 3.5 MB in total. You need to adjust the number of events per batch accordingly.
- The API does not return a 4xx status code for input errors. Instead, it responds with a 207 status code, which includes a list of the encountered errors.
method:POST
path:/ingestion
request:
name:IngestionRequest
body:
properties:
batch:list<IngestionEvent>
response:IngestionResponse# will be reportet as 200 response, but endpoint returns 207
batch:
type:list<IngestionEvent>
docs:"Batch of tracing events to be ingested. Discriminated by attribute `type`."
metadata:
type:optional<unknown>
docs:Optional. Metadata field used by the Langfuse SDKs for debugging.
response:
type:IngestionResponse
status-code:207
types:
IngestionEvent:
discriminant:"type"
union:
trace-create:TraceEvent
score-create:ScoreEvent
event-create:CreateEventEvent
generation-create:CreateGenerationEvent
generation-update:UpdateGenerationEvent
span-create:CreateSpanEvent
span-update:UpdateSpanEvent
sdk-log:SDKLogEvent
trace-create:
type:TraceEvent
docs:Creates a new trace. Upserts on id for updates if trace with id exists.
score-create:
type:ScoreEvent
docs:Creates a new score. Upserts on id for updates if score with id exists.
span-create:
type:CreateSpanEvent
docs:Creates a new span.
span-update:
type:UpdateSpanEvent
docs:Updates span based on id.
generation-create:
type:CreateGenerationEvent
docs:Creates a new generation.
generation-update:
type:UpdateGenerationEvent
docs:Updates a generation based on id.
event-create:
type:CreateEventEvent
docs:Creates an event.
sdk-log:
type:SDKLogEvent
docs:Langfuse SDKs only, used for debugging purposes.
# both are legacy
observation-create:
@@ -156,15 +185,66 @@ types:
id:optional<string>
traceId:string
name:string
value:double
value:
type:commons.CreateScoreValue
docs:The value of the score. Must be passed as string for categorical scores, and numeric for boolean and numeric scores. Boolean score values must equal either 1 or 0 (true or false)
observationId:optional<string>
comment:optional<string>
dataType:
type:optional<commons.ScoreDataType>
docs:When set, must match the score value's type. If not set, will be inferred from the score value or config
configId:
type:optional<string>
docs:Reference a score config on a score. When set, the score name must equal the config name and scores must comply with the config's range and data type. For categorical scores, the value must map to a config category. Numeric scores might be constrained by the score config's max and min values
examples:
- value:
name:"novelty"
value:0.9
traceId:"cdef-1234-5678-90ab"
- value:
name:"consistency"
value:1.2
dataType:"NUMERIC"
traceId:"cdef-1234-5678-90ab"
- value:
name:"accuracy"
value:0.9
dataType:"NUMERIC"
configId:"9203-4567-89ab-cdef"
traceId:"cdef-1234-5678-90ab"
- value:
name:"toxicity"
value:"not toxic"
traceId:"cdef-1234-5678-90ab"
- value:
name:"correctness"
value:"partially correct"
dataType:"CATEGORICAL"
configId:"1234-5678-90ab-cdef"
traceId:"cdef-1234-5678-90ab"
- value:
name:"hallucination"
value:0
dataType:"BOOLEAN"
traceId:"cdef-1234-5678-90ab"
- value:
name:"helpfulness"
value:1
dataType:"BOOLEAN"
configId:"1234-5678-90ab-cdef"
traceId:"cdef-1234-5678-90ab"
BaseEvent:
properties:
id:string
timestamp:string
metadata:unknown
id:
type:string
docs:UUID v4 that identifies the event
timestamp:
type:string
docs:"Datetime (ISO 8601) of event creation in client. Should be as close to actual event creation in client as possible, this timestamp will be used for ordering of events in future release. Resolution: milliseconds (required), microseconds (optimal)."
metadata:
type:optional<unknown>
docs:Optional. Metadata field used by the Langfuse SDKs for debugging.
docs:Delete a model. Cannot delete models managed by Langfuse. You can create your own definition with the same modelName to override the definition though.
path:/models/{id}
path-parameters:
id:string
types:
PaginatedModels:
properties:
data:list<commons.Model>
meta:pagination.MetaResponse
CreateModelRequest:
properties:
modelName:
docs:"Name of the model definition. If multiple with the same name exist, they are applied in the following order: (1) custom over built-in, (2) newest according to startTime where model.startTime<observation.startTime"
type:string
matchPattern:
docs:"Regex pattern which matches this model definition to generation.model. Useful in case of fine-tuned models. If you want to exact match, use `(?i)^modelname$`"
type:string
startDate:
docs:Apply only to generations which are newer than this ISO date.
type:optional<date>
unit:
docs:Unit used by this model.
type:commons.ModelUsageUnit
inputPrice:
docs:Price (USD) per input unit
type:optional<double>
outputPrice:
docs:Price (USD) per output unit
type:optional<double>
totalPrice:
docs:Price (USD) per total units. Cannot be set if input or output price is set.
type:optional<double>
tokenizerId:
docs:Optional. Tokenizer to be applied to observations which match to this model. See docs for more details.
type:optional<string>
tokenizerConfig:
docs:Optional. Configuration for the selected tokenizer. Needs to be JSON. See docs for more details.
docs:Create a score configuration (config). Score configs are used to define the structure of scores
method:POST
path:/score-configs
request:CreateScoreConfigRequest
response:commons.ScoreConfig
get:
docs:Get all score configs
method:GET
path:/score-configs
request:
name:GetScoreConfigsRequest
query-parameters:
page:
type:optional<integer>
docs:Page number, starts at 1.
limit:
type:optional<integer>
docs:Limit of items per page. If you encounter api issues due to too large page sizes, try to reduce the limit
response:ScoreConfigs
get-by-id:
docs:Get a score config
method:GET
path:/score-configs/{configId}
path-parameters:
configId:
type:string
docs:The unique langfuse identifier of a score config
response:commons.ScoreConfig
types:
ScoreConfigs:
properties:
data:list<commons.ScoreConfig>
meta:pagination.MetaResponse
CreateScoreConfigRequest:
properties:
name:string
dataType:commons.ScoreDataType
categories:
type:optional<list<commons.ConfigCategory>>
docs:Configure custom categories for categorical scores. Pass a list of objects with `label` and `value` properties. Categories are autogenerated for boolean configs and cannot be passed
minValue:
type:optional<double>
docs:Configure a minimum value for numerical scores. If not set, the minimum value defaults to -∞
maxValue:
type:optional<double>
docs:Configure a maximum value for numerical scores. If not set, the maximum value defaults to +∞
description:
type:optional<string>
docs:Description is shown across the Langfuse UI and can be used to e.g. explain the config categories in detail, why a numeric range was set, or provide additional context on config name or usage
docs:Limit of items per page. If you encounter api issues due to too large page sizes, try to reduce the limit.
userId:optional<string>
name:optional<string>
userId:
type:optional<string>
docs:Retrieve only scores with this userId associated to the trace.
name:
type:optional<string>
docs:Retrieve only scores with this name.
fromTimestamp:
type:optional<datetime>
docs:Retrieve only scores newer than this datetime (ISO 8601).
docs:Optional filter to only include scores created on or after a certain datetime (ISO 8601)
toTimestamp:
type:optional<datetime>
docs:Optional filter to only include scores created before a certain datetime (ISO 8601)
source:
type:commons.ScoreSource
type:optional<commons.ScoreSource>
docs:Retrieve only scores from a specific source.
operator:
type:optional<string>
@@ -39,6 +46,15 @@ service:
value:
type:optional<double>
docs:Retrieve only scores with <operator> value.
scoreIds:
type:optional<string>
docs:Comma-separated list of score IDs to limit the results to.
configId:
type:optional<string>
docs:Retrieve only scores with a specific configId.
dataType:
type:optional<commons.ScoreDataType>
docs:Retrieve only scores with a specific dataType.
response:Scores
get-by-id:
docs:Get a score
@@ -63,9 +79,59 @@ types:
id:optional<string>
traceId:string
name:string
value:double
value:
type:commons.CreateScoreValue
docs:The value of the score. Must be passed as string for categorical scores, and numeric for boolean and numeric scores. Boolean score values must equal either 1 or 0 (true or false)
observationId:optional<string>
comment:optional<string>
dataType:
type:optional<commons.ScoreDataType>
docs:The data type of the score. When passing a configId this field is inferred. Otherwise, this field must be passed or will default to numeric.
configId:
type:optional<string>
docs:Reference a score config on a score. The unique langfuse identifier of a score config. When passing this field, the dataType and stringValue fields are automatically populated.
docs:Limit of items per page. If you encounter api issues due to too large page sizes, try to reduce the limit.
fromTimestamp:
type:optional<datetime>
docs:Optional filter to only include sessions created on or after a certain datetime (ISO 8601)
toTimestamp:
type:optional<datetime>
docs:Optional filter to only include sessions created before a certain datetime (ISO 8601)
response:PaginatedSessions
get:
docs:Get a session
docs:Get a session. Please note that `traces` on this endpoint are not paginated, if you plan to fetch large sessions, consider `GET /api/public/traces?sessionId=<sessionId>`
"dx":"pnpm i && pnpm --filter=shared run db:reset && pnpm --filter=shared run db:seed:examples && pnpm run dev",
"dx":"pnpm i && pnpm run infra:dev:up && pnpm --filter=shared run db:reset && pnpm --filter=shared run ch:reset && pnpm --filter=shared run db:seed:examples && pnpm run dev",
"dx-f":"pnpm i && pnpm run infra:dev:up && pnpm --filter=shared run db:reset -f && pnpm --filter=shared run ch:reset && pnpm --filter=shared run db:seed:examples && pnpm run dev",
"dx:skip-infra":"pnpm i && pnpm --filter=shared run db:reset && pnpm --filter=shared run ch:reset && pnpm --filter=shared run db:seed:examples && pnpm run dev",
"build":"turbo run build",
"start":"turbo run start",
"dev":"turbo run dev",
"lint":"turbo run lint",
"test":"turbo run test",
"models:migrate":"turbo run models:migrate",
"release":"dotenv -e ../.env -- release-it"
"release":"dotenv -e ../.env -- release-it",
"prepare":"husky"
},
"devDependencies":{
"@release-it/bumper":"^6.0.1",
"dotenv-cli":"^7.4.1",
"braces":"3.0.3",
"dotenv-cli":"^7.4.2",
"husky":"^9.0.11",
"prettier":"^3.2.5",
"release-it":"^17.2.1",
"turbo":"^1.13.3"
"release-it":"^17.3.0",
"turbo":"^1.13.4"
},
"release-it":{
"git":{
"commitMessage":"chore: release v${version}",
"tagName":"v${version}"
"tagName":"v${version}",
"commitArgs":[
"--no-verify"
],
"pushArgs":[
"--no-verify"
]
},
"plugins":{
"@release-it/bumper":{
@@ -69,5 +79,6 @@
"pr":":rocket: _This pull request is included in v${version}. See [${releaseName}](${releaseUrl}) for release notes._"
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Marc Klingen