Compare commits

...
12 Commits
Author SHA1 Message Date
Marc Klingen f2c792d3dc chore: release v2.3.0
CI/CD / lint (push) Waiting to run
CI/CD / test-docker-build (push) Waiting to run
CI/CD / tests (18) (push) Waiting to run
CI/CD / tests (20) (push) Waiting to run
CI/CD / e2e-tests (push) Waiting to run
CI/CD / all-ci-passed (push) Blocked by required conditions
CI/CD / push-docker-image (push) Blocked by required conditions
release.yml / release (push) Waiting to run
2024-02-09 21:38:20 +01:00
Marc Klingen 0527fcf097 fix(ui): add space 2024-02-09 21:30:06 +01:00
Max DeichmannandGitHub 85737221d9 feat: send sdk logs to betterstack (#1125) 2024-02-09 20:01:46 +00:00
ffe95cf427 feat(ui): add request dashboard button (#1122)
Co-authored-by: Max Deichmann <m.deichmann@tum.de>
2024-02-09 19:25:08 +00:00
adec1476ea feat(ui): add quickstart examples to api key creation popup (#1119)
---------

Co-authored-by: Marc Klingen <git@marcklingen.com>
2024-02-09 19:10:21 +00:00
c5d7772fe6 chore(ui): convert model pricing in UI to per 1000 units (#1123)
Co-authored-by: Max Deichmann <m.deichmann@tum.de>
2024-02-09 19:05:04 +00:00
Max DeichmannandGitHub a88a6b8026 fix: upgrade API keys (#1120) 2024-02-09 16:44:48 +00:00
Marc Klingen 5d77e09eed docs: update readme and contributing 2024-02-09 15:38:22 +01:00
Hassieb PakzadandGitHub 85e5ea4661 fix: allow bearer auth for score ingestion via /api/public/ingestion (#1117)
* fix: allow bearer auth for score ingestion

* tests: add test for opening ingestion endpoint for scoring

* test: adds db check for bearer auth tests in ingestion
2024-02-09 13:51:22 +01:00
366df5a5c9 Fix decimals in scores filter on traces + generations table (#1012)
Co-authored-by: gitstart-langfuse <gitstart-langfuse@users.noreply.github.com>
Co-authored-by: GitStart-LangFuse <151670644+gitstart-langfuse@users.noreply.github.com>
Co-authored-by: Max Deichmann <m.deichmann@tum.de>
2024-02-09 08:44:12 +01:00
4441383dfd docs: update readme (#1107)
docs: update and significantly shorten readme
---------

Co-authored-by: Clemo <121163007+clemra@users.noreply.github.com>
2024-02-08 22:18:02 +00:00
Richard KrümmelandGitHub 7f7696e9ff feat(ui): add in-product notifications (#1057) 2024-02-08 19:41:30 +00:00
27 changed files with 1096 additions and 351 deletions
+12 -4
View File
@@ -2,22 +2,30 @@
First off, thanks for taking the time to contribute! ❤️
Langfuse is an open-source observability and analytics solution for LLM-based applications. We welcome contributions through GitHub pull requests. This document outlines our conventions regarding development workflow, commit message formatting, contact points, and other resources. Our goal is to simplify the process and ensure that your contributions are easily accepted.
The best ways to contribute to Langfuse:
We gratefully welcome improvements to documentation as well as to code.
- Submit and vote on [Ideas](https://github.com/orgs/langfuse/discussions/categories/ideas)
- Create and comment on [Issues](https://github.com/langfuse/langfuse/issues)
- Open a PR.
We welcome contributions through GitHub pull requests. This document outlines our conventions regarding development workflow, commit message formatting, contact points, and other resources. Our goal is to simplify the process and ensure that your contributions are easily accepted.
We gratefully welcome improvements to documentation ([docs repo](https://github.com/langfuse/langfuse-docs)), the core application (this repo) and the SDKs ([Python](https://github.com/langfuse/langfuse-python), [JS](https://github.com/langfuse/langfuse-js)).
The maintainers are available on [Discord](https://langfuse.com/discord) in case you have any questions.
> And if you like the project, but just don't have time to contribute, that's fine. There are other easy ways to support the project and show your appreciation, which we would also be very happy about:
> And if you like the project, but just don't have time to contribute code, that's fine. There are other easy ways to support the project and show your appreciation, which we would also be very happy about:
>
> - Star the project;
> - Tweet about it;
> - Refer to this project in your project's readme;
> - Submit and vote on [Ideas](https://github.com/orgs/langfuse/discussions/categories/ideas);
> - Create and comment on [Issues](https://github.com/langfuse/langfuse/issues);
> - Mention the project at local meetups and tell your friends/colleagues.
## Making a change
_Before making any significant changes, please [open an issue](https://github.com/langfuse/langfuse/issues)._ Discussing your proposed changes ahead of time will make the contribution process smooth for everyone.
_Before making any significant changes, please [open an issue](https://github.com/langfuse/langfuse/issues)._ Discussing your proposed changes ahead of time will make the contribution process smooth for everyone. Large changes that were not discussed in an issue may be rejected.
Once we've discussed your changes and you've got your code ready, make sure that tests are passing and open your pull request.
+90 -174
View File
@@ -2,51 +2,43 @@
<a href="https://langfuse.com">
<h1>🪢 Langfuse</h1>
</a>
<div>
<h3> <a href="https://cloud.langfuse.com">
<strong>Sign up</strong>
</a> ·
<a href="https://langfuse.com/docs/deployment/self-host">
<strong>Self Host Langfuse</strong>
</a> ·
<a href="https://langfuse.com/demo">
<strong>Demo Project (live data)</strong>
</a>
</h3>
<h3>
Open source observability & analytics for LLM-based applications
Langfuse is the open source LLM engineering platform.
</h3>
<div>
<strong>Observability:</strong> Explore and debug complex logs & traces in a visual UI
</div>
<div>
<strong>Analytics:</strong> Measure & improve costs, latency and response quality
<div>
Debug, analyze and iterate - together
</div>
</br>
<div>
<a href="https://cloud.langfuse.com">
<strong>Langfuse Cloud »</strong>
</a>
</br>
<a href="https://langfuse.com/docs/deployment/self-host">
<strong>Self Host Langfuse »</strong>
</a>
</br>
<a href="https://discord.gg/7NXusRtqYU">
<strong>Join the Langfuse Discord »</strong>
</a>
</br>
<a href="https://langfuse.com">
<strong>langfuse.com</strong>
</a> ·
<a href="https://langfuse.com/docs">
<strong>Docs</strong>
</a> ·
<a href="https://github.com/langfuse/langfuse/issues/new?labels=%F0%9F%90%9E%E2%9D%94+unconfirmed+bug&projects=&template=bug_report.yml&title=bug%3A+">
<a href="https://langfuse.com/issue">
<strong>Report Bug</strong>
</a> ·
<a href="https://github.com/orgs/langfuse/discussions/new?category=ideas">
<a href="https://langfuse.com/idea">
<strong>Feature Request</strong>
</a>
</a> ·
<a href="https://langfuse.com/changelog">
<strong>Changelog</strong>
</a> ·
<a href="https://langfuse.com/discord">
<strong>Discord</strong>
</a>
</div>
</br>
</br>
<div>
<img src="https://img.shields.io/badge/License-MIT-red.svg?style=flat-square" alt="MIT License">
<a href="https://discord.gg/7NXusRtqYU"><img src="https://img.shields.io/discord/1111061815649124414?style=flat-square&logo=Discord&logoColor=white&label=Discord&color=%23434EE4" alt="Discord"></a>
<a href="https://github.com/langfuse/langfuse"><img src="https://img.shields.io/github/stars/langfuse/langfuse?style=flat-square&logo=GitHub&label=langfuse%2Flangfuse" alt="Github Repo Stars"></a>
<a href="https://github.com/langfuse/langfuse/releases"><img src="https://img.shields.io/github/v/release/langfuse/langfuse?include_prereleases&style=flat-square" alt="langfuse releases"></a>
<a href="https://github.com/langfuse/langfuse/actions/workflows/pipeline.yml?query=branch:main"><img src="https://img.shields.io/github/actions/workflow/status/langfuse/langfuse/pipeline.yml?style=flat-square&label=All%20tests" alt="CI test status"></a>
<a href="https://status.langfuse.com"><img src="https://uptime.betterstack.com/status-badges/v1/monitor/udlc.svg" alt="Uptime Status"/></a>
<a href="https://www.ycombinator.com/companies/langfuse"><img src="https://img.shields.io/badge/Y%20Combinator-W23-orange?style=flat-square" alt="Y Combinator W23"></a>
<a href="https://github.com/langfuse/langfuse/pkgs/container/langfuse"><img alt="Docker Image" src="https://img.shields.io/badge/docker-langfuse-blue?logo=Docker&logoColor=white&style=flat-square"></a>
<a href="https://www.npmjs.com/package/langfuse"><img src="https://img.shields.io/npm/v/langfuse?style=flat-square&label=npm+langfuse" alt="langfuse npm package"></a>
@@ -54,61 +46,44 @@
</div>
</div>
</br>
</div>
</br>
## What is Langfuse?
## Overview
Langfuse is an open source observability & analytics solution for LLM-based applications. It is mostly geared towards production usage but some users also use it for local development of their LLM applications.
### Develop
Langfuse is focused on applications built on top of LLMs. Many new abstractions and common best practices evolved recently, e.g. agents, chained prompts, embedding-based retrieval, LLM access to REPLs & APIs. These make applications more powerful but also unpredictable for developers as they cannot fully anticipate how changes impact the quality, cost and overall latency of their application. Thus Langfuse helps to monitor and debug these applications.
- **Observability:** Instrument your app and start ingesting traces to Langfuse ([Quickstart](https://langfuse.com/docs/get-started), [Integrations](https://langfuse.com/docs/integrations) [Tracing](https://langfuse.com/docs/tracing))
- **Langfuse UI:** Inspect and debug complex logs ([Demo](https://langfuse.com/docs/demo), [Tracing](https://langfuse.com/docs/tracing))
- **Prompts:** Manage, version and deploy prompts from within Langfuse ([Prompt Management](https://langfuse.com/docs/prompts))
**Demo (2 min)**
### Monitor
- **Analytics:** Track metrics (cost, latency, quality) and gain insights from dashboards & data exports ([Analytics](https://langfuse.com/docs/analytics))
- **Evals:** Collect and calculate scores for your LLM completions ([Scores & Evaluations](https://langfuse.com/docs/scores))
- Run model-based evaluations ([Model-based evaluations](https://langfuse.com/docs/scores/model-based-evals))
- Collect user feedback ([User Feedback](https://langfuse.com/docs/scores/user-feedback))
- Manually score observations in Langfuse ([Manual Scores](https://langfuse.com/docs/scores/manually))
### Test
- **Experiments:** Track and test app behaviour before deploying a new version
- Datasets let you test expected in and output pairs and benchmark performance before deployiong ([Datasets](https://langfuse.com/docs/datasets))
- Track versions and releases in your application ([Experimentation](https://langfuse.com/docs/experimentation), [Prompt Management](https://langfuse.com/docs/prompts))
### Video: Langfuse in two minutes
https://github.com/langfuse/langfuse/assets/2834609/6041347a-b517-4a11-8737-93ef8f8af49f
_Muted by default, enable sound for voice-over_
Explore demo project in Langfuse here (free account required): https://langfuse.com/demo
### Observability
Langfuse offers an admin UI to explore the ingested data.
- Nested view of LLM app executions; detailed information along the traces on: latency, cost, scores
- Segment execution traces by user feedback, to e.g. identify production issues
### Analytics
Reporting on
- Token usage by model
- Volume of traces
- Scores/evals
Broken down by
- Users
- Releases
- Prompt/chain versions
- Prompt/chain types
- Time
→ Expect releases with more ways to analyze the data over the next weeks.
## Get started
### Step 1: Run Server
### Langfuse Cloud
#### Langfuse Cloud
Managed deployment by the Langfuse team, generous free-tier (hobby plan), no credit card required.
Managed deployment by the Langfuse team, generous free-tier (hobby plan) available, no credit card required.
**[» Langfuse Cloud](https://cloud.langfuse.com)**
Links: [Create account](https://cloud.langfuse.com), [learn more](https://cloud.langfuse.com)
#### Localhost
Requirements: docker, docker compose (e.g. using Docker Desktop)
### Localhost (docker)
```bash
# Clone repository
@@ -119,143 +94,84 @@ cd langfuse
docker compose up -d
```
#### Self-host (Docker)
[→ Learn more about deploying locally](https://langfuse.com/docs/deployment/local)
[→ Instructions](https://langfuse.com/docs/deployment/self-host)
### Self-host (docker)
[![Deploy on Railway](https://railway.app/button.svg)](https://railway.app/template/gmbqa_)
Langfuse is simple to self-host and keep updated. It currently requires only a single docker container.
[→ Self Hosting Instructions](https://langfuse.com/docs/deployment/self-host)
### Step 2: Data ingestion
Templated deployments: [Railway, GCP Cloud Run, AWS Fargate, Kubernetes and others](https://langfuse.com/docs/deployment/self-host#platform-specific-information)
#### SDKs to instrument application
## Get Started
Fully async, typed SDKs to instrument any LLM application. Currently available for Python & JS/TS.
### API Keys
[Guide](https://langfuse.com/docs/guides/sdk-integration) with an example of how the SDK can be used
You require a Langfuse public and secret key to get started. Sign up [here](https://cloud.langfuse.com) and find them in your project settings.
| Package | Description | Links |
| --------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------- | -------------------------------------------------------------------------------------------------------------- |
| [![PyPI Version](https://img.shields.io/pypi/v/langfuse.svg?style=flat-square&label=pypi+langfuse)](https://pypi.python.org/pypi/langfuse) | Python | [docs](https://langfuse.com/docs/integrations/sdk/python), [repo](https://github.com/langfuse/langfuse-python) |
| [![npm Version](https://img.shields.io/npm/v/langfuse?style=flat-square&label=npm+langfuse)](https://www.npmjs.com/package/langfuse) | JS/TS: Node >= 18, Edge runtimes | [docs](https://langfuse.com/docs/integrations/sdk/typescript), [repo](https://github.com/langfuse/langfuse-js) |
| [![npm package](https://img.shields.io/npm/v/langfuse-node?style=flat-square&label=npm+langfuse-node)](https://www.npmjs.com/package/langfuse-node) | JS/TS: Node <18 | [docs](https://langfuse.com/docs/integrations/sdk/typescript), [repo](https://github.com/langfuse/langfuse-js) |
### Ingesting Data · Instrumenting Your Application
#### Langchain applications
Note: We recommend using our fully async, typed [SDKs](https://langfuse.com/docs/sdk) that allow you to instrument any LLM application with any underlying model. They are available in [Python](https://langfuse.com/docs/sdk/python) & [JS/TS](https://langfuse.com/docs/sdk/typescript). The SDKs will always be the most fully featured and stable way to ingest data into Langfuse.
The Langfuse callback handler automatically instruments Langchain applications. Currently available for Python and JS/TS.
You may want to use another integration to get started quickly or implement a use case that we do not yet support. However, we recommend to migrate to the Langfuse SDKs over time to ensure performance and stability.
**Python**
See our the [→ Quickstart](https://langfuse.com/docs/get-started) to get started in integrating Langfuse.
```shell
pip install langfuse
```
### Integrations
```python
# Initialize Langfuse handler
from langfuse.callback import CallbackHandler
handler = CallbackHandler(PUBLIC_KEY, SECRET_KEY)
| Integration | Supports | Description |
| -------------------------------------------------------- | ------------- | ------------------------------------------------------------------------------- |
| [**SDK** - _recommended_](https://langfuse.com/docs/sdk) | Python, JS/TS | Manual instrumentation using the SDKs for full flexibility. |
| [OpenAI](https://langfuse.com/docs/openai) | Python | Automated instrumentation using drop-in replacement of OpenAI SDK. |
| [Langchain](https://langfuse.com/docs/langchain) | Python, JS/TS | Automated instrumentation by passing callback handler to Langchain application. |
| [API](https://langfuse.com/docs/api) | | Directly call the public API. OpenAPI spec available. |
# Setup Langchain
from langchain.chains import LLMChain
...
chain = LLMChain(llm=llm, prompt=prompt)
External projects/packages that integrate with Langfuse:
# Add Langfuse handler as callback
chain.run(input="<user_input", callbacks=[handler])
```
| Name | Description |
| ---------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ |
| [LiteLLM](/https://langfuse.comdocs/litellm) | Use any LLM as a drop in replacement for GPT. Use Azure, OpenAI, Cohere, Anthropic, Ollama, VLLM, Sagemaker, HuggingFace, Replicate (100+ LLMs). |
| [Flowise](https://langfuse.com/docs/flowise) | JS/TS no-code builder for customized LLM flows. |
| [Langflow](https://langfuse.com/docs/langflow) | Python-based UI for LangChain, designed with react-flow to provide an effortless way to experiment and prototype flows. |
→ [Langchain integration docs for Python](https://langfuse.com/docs/integrations/langchain/python)
## Questions and feedback
**JS/TS**
### Ideas and roadmap
[Langchain integration docs for JS/TS](https://langfuse.com/docs/integrations/langchain/typescript)
- [GitHub Discussions](https://github.com/orgs/langfuse/discussions)
- [Feature Requests](https://langfuse.com/idea)
#### Add scores/evaluations to traces (optional)
### Support and feedback
Quality/evaluation of traces is tracked via scores ([docs](https://langfuse.com/docs/scores)). Scores are related to traces and optionally to observations. Scores can be added via:
In order of preference the best way to communicate with us:
- **Backend SDKs** (see docs above): `{trace, event, span, generation}.score()`
- **API** (see docs below): `POST /api/public/scores`
- **Client-side using Web SDK**, e.g. to capture user feedback or other user-based quality metrics:
```sh
npm install langfuse
```
```ts
// Client-side (browser)
import { LangfuseWeb } from "langfuse";
const langfuseWeb = new LangfuseWeb({
publicKey: process.env.LANGFUSE_PUBLIC_KEY,
});
// frontend handler (example: React)
export function UserFeedbackComponent(props: { traceId: string }) {
const handleUserFeedback = async (value: number) => {
await langfuseWeb.score({
traceId: props.traceId,
name: "user_feedback",
value,
});
};
return (
<div>
<button onClick={() => handleUserFeedback(1)}>👍</button>
<button onClick={() => handleUserFeedback(-1)}>👎</button>
</div>
);
}
```
#### API
[**Api reference**](https://langfuse.com/docs/integrations/api)
- POST/PATCH routes to ingest data
- GET routes to use data in downstream applications (e.g. embedded analytics)
## Questions / Feedback
The maintainers are very active in the Langfuse [Discord](https://langfuse.com/discord) and are happy to answer questions or discuss feedback/ideas regarding the future of the project.
- [GitHub Discussions](https://github.com/orgs/langfuse/discussions): Contribute [ideas](https://langfuse.com/idea) [support requests](https://github.com/orgs/langfuse/discussions/categories/support) and [report bugs](https://github.com/langfuse/langfuse/issues/new?labels=%F0%9F%90%9E%E2%9D%94+unconfirmed+bug&projects=&template=bug_report.yml&title=bug%3A+) (preferred as we create a permanent, indexed artifact for other community members)
- [Discord](https://langfuse.com/discord): For community support and to chat directly with maintainers
- Privately: Email contact at langfuse dot com
## Contributing to Langfuse
Join the community [on Discord](https://discord.gg/7NXusRtqYU).
To contribute, send us a PR, raise a GitHub issue, or email at contributing@langfuse.com
### Development setup
See [CONTRIBUTING.md](CONTRIBUTING.md) for details on how to setup a development environment.
- Vote on [Ideas](https://github.com/orgs/langfuse/discussions/categories/ideas)
- Raise and comment on [Issues](https://github.com/langfuse/langfuse/issues)
- Open a PR - see [CONTRIBUTING.md](CONTRIBUTING.md) for details on how to setup a development environment.
## License
Langfuse is MIT licensed, except for `ee/` folder. See [LICENSE](LICENSE) and [docs](https://langfuse.com/docs/open-source) for more details.
This repository is MIT licensed, except for the `ee/` folder. See [LICENSE](LICENSE) and [docs](https://langfuse.com/docs/open-source) for more details.
## Misc
### Upgrade Langfuse (localhost)
### GET API to export your data
```bash
# Stop server and db
docker compose down
[**GET routes**](https://langfuse.com/docs/integrations/api) to use data in downstream applications (e.g. embedded analytics).
# Pull latest changes
git pull
docker-compose pull
### Security & Privacy
# Run server and db
docker compose up -d
```
### Run Langfuse in CI for integration tests
Checkout GitHub Actions workflows of [Python SDK](https://github.com/langfuse/langfuse-python/blob/main/.github/workflows/ci.yml) and [JS/TS SDK](https://github.com/langfuse/langfuse-js/blob/main/.github/workflows/ci.yml).
We take data security and privacy seriously. Please refer to our [Security and Privacy](https://langfuse.com/security) page for more information.
### Telemetry
By default, Langfuse automatically reports basic usage statistics to a centralized server (PostHog).
By default, Langfuse automatically reports basic usage statistics of self-hosted instances to a centralized server (PostHog).
This helps us to:
+12 -2
View File
@@ -1,12 +1,12 @@
{
"name": "langfuse-core",
"version": "2.2.0",
"version": "2.3.0",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "langfuse-core",
"version": "2.2.0",
"version": "2.3.0",
"hasInstallScript": true,
"dependencies": {
"@anthropic-ai/tokenizer": "^0.0.4",
@@ -77,6 +77,7 @@
"react-icons": "^5.0.1",
"react-responsive": "^9.0.2",
"react18-json-view": "^0.2.7",
"sonner": "^1.4.0",
"superjson": "2.2.1",
"tailwind-merge": "^2.2.1",
"tailwindcss-animate": "^1.0.7",
@@ -16872,6 +16873,15 @@
"integrity": "sha512-WKa+XuLG1A1R0UWhl2+1XQSi+fZWMsYKffMZTTYsiZaUD8k2yDAj5atimTUD2TZkyCkNEeYE5NhFZmupOGtjYQ==",
"dev": true
},
"node_modules/sonner": {
"version": "1.4.0",
"resolved": "https://registry.npmjs.org/sonner/-/sonner-1.4.0.tgz",
"integrity": "sha512-nvkTsIuOmi9e5Wz5If8ldasJjZNVfwiXYijBi2dbijvTQnQppvMcXTFNxL/NUFWlI2yJ1JX7TREDsg+gYm9WyA==",
"peerDependencies": {
"react": "^18.0.0",
"react-dom": "^18.0.0"
}
},
"node_modules/source-map-js": {
"version": "1.0.2",
"resolved": "https://registry.npmjs.org/source-map-js/-/source-map-js-1.0.2.tgz",
+2 -1
View File
@@ -1,6 +1,6 @@
{
"name": "langfuse-core",
"version": "2.2.0",
"version": "2.3.0",
"private": true,
"scripts": {
"prebuild": "cp generated/openapi-client/openapi.yml public/openapi-client.yml && cp generated/openapi-server/openapi.yml public/openapi-server.yml",
@@ -97,6 +97,7 @@
"react-icons": "^5.0.1",
"react-responsive": "^9.0.2",
"react18-json-view": "^0.2.7",
"sonner": "^1.4.0",
"superjson": "2.2.1",
"tailwind-merge": "^2.2.1",
"tailwindcss-animate": "^1.0.7",
+357 -6
View File
@@ -795,10 +795,10 @@ describe("/api/public/ingestion API Endpoint", () => {
expect("errors" in responseOne.body).toBe(true);
// eslint-disable-next-line @typescript-eslint/no-unsafe-member-access
expect(responseOne.body?.errors.length).toBe(1);
expect(responseOne.body.errors.length).toBe(1);
expect("successes" in responseOne.body).toBe(true);
// eslint-disable-next-line @typescript-eslint/no-unsafe-member-access
expect(responseOne.body?.successes.length).toBe(1);
expect(responseOne.body.successes.length).toBe(1);
const dbTrace = await prisma.trace.findMany({
where: {
@@ -847,10 +847,10 @@ describe("/api/public/ingestion API Endpoint", () => {
expect("errors" in responseOne.body).toBe(true);
// eslint-disable-next-line @typescript-eslint/no-unsafe-member-access
expect(responseOne.body?.errors.length).toBe(1);
expect(responseOne.body.errors.length).toBe(1);
expect("successes" in responseOne.body).toBe(true);
// eslint-disable-next-line @typescript-eslint/no-unsafe-member-access
expect(responseOne.body?.successes.length).toBe(1);
expect(responseOne.body.successes.length).toBe(1);
const dbTrace = await prisma.trace.findMany({
where: {
@@ -897,10 +897,10 @@ describe("/api/public/ingestion API Endpoint", () => {
expect("errors" in responseOne.body).toBe(true);
// eslint-disable-next-line @typescript-eslint/no-unsafe-member-access
expect(responseOne.body?.errors.length).toBe(1);
expect(responseOne.body.errors.length).toBe(1);
expect("successes" in responseOne.body).toBe(true);
// eslint-disable-next-line @typescript-eslint/no-unsafe-member-access
expect(responseOne.body?.successes.length).toBe(1);
expect(responseOne.body.successes.length).toBe(1);
const dbTrace = await prisma.trace.findMany({
where: {
@@ -1414,4 +1414,355 @@ IB Home / . . . / News / News about the IB / Why ChatGPT is an o
expect(cleanedEvent).toStrictEqual(expected);
});
});
it("should allow score ingestion via Basic auth", async () => {
const projectId = "7a88fb47-b4e2-43b8-a06c-a5ce950dc53a";
const traceId = "trace_id";
const scoreId = "score_id";
const scoreEventId = "score_event_id";
const scoreName = "score-name";
const scoreValue = 100.5;
// Seed db with a trace to be scored
await prisma.trace.create({
data: {
id: traceId,
name: "trace-name",
project: { connect: { id: projectId } },
},
});
const response = await makeAPICall("POST", "/api/public/ingestion", {
batch: [
{
id: scoreEventId,
type: "score-create",
timestamp: new Date().toISOString(),
body: {
id: scoreId,
name: scoreName,
value: scoreValue,
traceId: traceId,
},
},
],
});
expect(response.status).toBe(207);
expect(response.body.successes.length).toBe(1);
expect(response.body.successes[0]?.id).toBe(scoreEventId);
expect(response.body.errors.length).toBe(0);
const dbScore = await prisma.score.findUnique({
where: {
id: scoreId,
},
});
expect(dbScore?.id).toBe(scoreId);
expect(dbScore?.traceId).toBe(traceId);
expect(dbScore?.name).toBe(scoreName);
expect(dbScore?.value).toBe(scoreValue);
});
it("should allow score ingestion via Bearer auth", async () => {
const projectId = "7a88fb47-b4e2-43b8-a06c-a5ce950dc53a";
const traceId = "trace_id";
const bearerAuth = "Bearer pk-lf-1234567890";
const scoreId = "score_id";
const scoreEventId = "score_event_id";
const scoreName = "score-name";
const scoreValue = 100.5;
// Seed db with a trace to be scored
await prisma.trace.create({
data: {
id: traceId,
name: "trace-name",
project: { connect: { id: projectId } },
},
});
const response = await makeAPICall(
"POST",
"/api/public/ingestion",
{
batch: [
{
id: scoreEventId,
type: "score-create",
timestamp: new Date().toISOString(),
body: {
id: scoreId,
name: scoreName,
value: scoreValue,
traceId: traceId,
},
},
],
},
bearerAuth,
);
expect(response.status).toBe(207);
expect(response.body.successes.length).toEqual(1);
expect(response.body.successes[0]?.id).toBe(scoreEventId);
expect(response.body.errors.length).toBe(0);
const dbScore = await prisma.score.findUnique({
where: {
id: scoreId,
},
});
expect(dbScore?.id).toBe(scoreId);
expect(dbScore?.traceId).toBe(traceId);
expect(dbScore?.name).toBe(scoreName);
expect(dbScore?.value).toBe(scoreValue);
});
it("should throw an Auth error on Bearer Auth for all events that are NOT 'score-create'", async () => {
const projectId = "7a88fb47-b4e2-43b8-a06c-a5ce950dc53a";
const traceId = "trace_id";
const bearerAuth = "Bearer pk-lf-1234567890";
const scoreId = "score_id";
const scoreEventId = "score_event_id";
const scoreName = "score-name";
const scoreValue = 100.5;
const generationId = v4();
const spanId = v4();
const anotherTraceId = "another_trace_id";
// Seed db with a trace to be scored
await prisma.trace.create({
data: {
id: traceId,
name: "trace-name",
project: { connect: { id: projectId } },
},
});
const response = await makeAPICall(
"POST",
"/api/public/ingestion",
{
metadata: {
sdk_verion: "1.0.0",
sdk_name: "python",
},
batch: [
{
id: v4(),
type: "trace-create",
timestamp: new Date().toISOString(),
body: {
id: anotherTraceId,
name: "trace-name",
userId: "user-1",
metadata: { key: "value" },
release: "1.0.0",
version: "2.0.0",
tags: ["tag-1", "tag-2"],
},
},
{
id: v4(),
type: "observation-create",
timestamp: new Date().toISOString(),
body: {
id: generationId,
traceId: traceId,
type: "GENERATION",
name: "generation-name",
startTime: "2021-01-01T00:00:00.000Z",
endTime: "2021-01-01T00:00:00.000Z",
modelParameters: { key: "value" },
input: { key: "value" },
metadata: { key: "value" },
version: "2.0.0",
},
},
{
id: v4(),
type: "observation-update",
timestamp: new Date().toISOString(),
body: {
id: generationId,
type: "GENERATION",
output: { key: "this is a great gpt output" },
},
},
{
id: v4(),
type: "observation-create",
timestamp: new Date().toISOString(),
body: {
id: spanId,
traceId: traceId,
type: "SPAN",
name: "span-name",
startTime: "2021-01-01T00:00:00.000Z",
endTime: "2021-01-01T00:00:00.000Z",
input: { input: "value" },
metadata: { meta: "value" },
version: "2.0.0",
},
},
{
id: scoreEventId,
type: "score-create",
timestamp: new Date().toISOString(),
body: {
id: scoreId,
name: "score-name",
value: 100.5,
traceId: traceId,
},
},
],
},
bearerAuth,
);
expect(response.status).toBe(207);
expect(response.body.successes.length).toEqual(1);
expect(response.body.successes[0]?.id).toEqual(scoreEventId);
expect(response.body.errors.length).toEqual(4);
const dbScore = await prisma.score.findUnique({
where: {
id: scoreId,
},
});
expect(await prisma.trace.count()).toBe(1);
expect(await prisma.trace.count({ where: { id: traceId } })).toBe(1);
expect(await prisma.observation.count()).toBe(0);
expect(dbScore?.id).toBe(scoreId);
expect(dbScore?.traceId).toBe(traceId);
expect(dbScore?.name).toBe(scoreName);
expect(dbScore?.value).toBe(scoreValue);
});
it("should error on Bearer Auth for a trace from different project", async () => {
const otherProjectId = "other_project_id";
const traceId = "trace_id";
const bearerAuth = "Bearer pk-lf-1234567890";
const scoreId = "score_id";
const scoreEventId = "score_event_id";
const scoreName = "score-name";
const scoreValue = 100.5;
// Seed db with a trace to be scored
try {
await prisma.project.create({
data: {
id: otherProjectId,
name: "another-project",
},
});
await prisma.trace.create({
data: {
id: traceId,
name: "trace-name",
project: { connect: { id: otherProjectId } },
},
});
const response = await makeAPICall(
"POST",
"/api/public/ingestion",
{
batch: [
{
id: scoreEventId,
type: "score-create",
timestamp: new Date().toISOString(),
body: {
id: scoreId,
name: scoreName,
value: scoreValue,
traceId: traceId,
},
},
],
},
bearerAuth,
);
expect(response.status).toBe(207);
expect(response.body.successes.length).toBe(0);
expect(response.body).toHaveProperty("errors");
expect(response.body.errors.length).toEqual(1);
expect(response.body.errors[0]?.id).toEqual(scoreEventId);
expect(await prisma.trace.count()).toBe(1);
expect(await prisma.trace.count({ where: { id: traceId } })).toBe(1);
const dbScore = await prisma.score.findUnique({
where: {
id: scoreId,
},
});
expect(dbScore).toBeNull();
} finally {
await prisma.project.delete({ where: { id: otherProjectId } });
}
});
it("should error on Bearer Auth with a trace that does not exist", async () => {
const traceId = "trace_id";
const bearerAuth = "Bearer pk-lf-1234567890";
const scoreId = "score_id";
const scoreEventId = "score_event_id";
const scoreName = "score-name";
const scoreValue = 100.5;
const response = await makeAPICall(
"POST",
"/api/public/ingestion",
{
batch: [
{
id: scoreEventId,
type: "score-create",
timestamp: new Date().toISOString(),
body: {
id: scoreId,
name: scoreName,
value: scoreValue,
traceId: traceId,
},
},
],
},
bearerAuth,
);
expect(response.status).toBe(207);
expect(response.body.successes.length).toBe(0);
expect(response.body.errors.length).toBe(1);
expect(response.body.errors[0]?.id).toBe(scoreEventId);
expect(await prisma.trace.count()).toBe(0);
const dbScore = await prisma.score.findUnique({
where: {
id: scoreId,
},
});
expect(dbScore).toBeNull();
});
});
+22 -5
View File
@@ -27,21 +27,38 @@ export function createBasicAuthHeader(
return `Basic ${base64Credentials}`;
}
export type IngestionAPIResponse = {
errors: ErrorIngestion[];
successes: SuccessfulIngestion[];
};
export type SuccessfulIngestion = {
id: string;
status: number;
};
export type ErrorIngestion = {
id: string;
status: number;
message: string;
error: string;
};
export async function makeAPICall(
method: "POST" | "GET" | "PUT" | "DELETE" | "PATCH",
url: string,
body?: unknown,
auth?: string,
) {
const finalUrl = `http://localhost:3000/${url}`;
const authorization =
auth || createBasicAuthHeader("pk-lf-1234567890", "sk-lf-1234567890");
const options = {
method: method,
headers: {
Accept: "application/json",
"Content-Type": "application/json;charset=UTF-8",
Authorization: createBasicAuthHeader(
"pk-lf-1234567890",
"sk-lf-1234567890",
),
Authorization: authorization,
},
// Conditionally include the body property if the method is not "GET"
...(method !== "GET" &&
@@ -50,7 +67,7 @@ export async function makeAPICall(
const a = await fetch(finalUrl, options);
// eslint-disable-next-line @typescript-eslint/no-unsafe-assignment
return { body: await a.json(), status: a.status };
return { body: (await a.json()) as IngestionAPIResponse, status: a.status };
}
export const setupUserAndProject = async () => {
+23 -27
View File
@@ -8,7 +8,6 @@ import { useRouter } from "next/router";
import clsx from "clsx";
import { Code, MessageSquarePlus, Info, ChevronRightIcon } from "lucide-react";
import { signOut, useSession } from "next-auth/react";
import { ChevronDownIcon } from "@heroicons/react/20/solid";
import { cn } from "@/src/utils/tailwind";
import {
Avatar,
@@ -24,6 +23,13 @@ import { env } from "@/src/env.mjs";
import { LangfuseLogo } from "@/src/components/LangfuseLogo";
import { Spinner } from "@/src/components/layouts/spinner";
import { hasAccess } from "@/src/features/rbac/utils/checkAccess";
import { Toaster } from "@/src/components/ui/sonner";
import {
NOTIFICATIONS,
useCheckNotification,
} from "@/src/features/notifications/checkNotifications";
import { ChevronDownIcon } from "@heroicons/react/20/solid";
import useLocalStorage from "@/src/components/useLocalStorage";
const userNavigation = [
{
@@ -46,6 +52,9 @@ export default function Layout(props: PropsWithChildren) {
const [sidebarOpen, setSidebarOpen] = useState(false);
const router = useRouter();
const session = useSession();
useCheckNotification(NOTIFICATIONS, session.status === "authenticated");
const enableExperimentalFeatures =
api.environment.enableExperimentalFeatures.useQuery().data ?? false;
@@ -503,6 +512,7 @@ export default function Layout(props: PropsWithChildren) {
</div>
) : null}
<main className="p-4">{props.children}</main>
<Toaster visibleToasts={1} />
</div>
</div>
</>
@@ -522,29 +532,11 @@ const MainNavigation: React.FC<{
nav: NavigationItem[];
onNavitemClick?: () => void;
}> = ({ nav, onNavitemClick }) => {
const STORAGE_KEY = "sidebar-tracing-default-open";
const getDefaultOpen = () => {
const savedState = localStorage.getItem(STORAGE_KEY);
if (savedState !== null) {
try {
return JSON.parse(savedState) as boolean;
} catch (e) {
console.error("Error parsing saved state: ", e);
}
}
return false;
};
const [isOpen, setIsOpen] = useLocalStorage(
"sidebar-tracing-default-open",
false,
);
const handleDropDownClick = () => {
const savedState = localStorage.getItem(STORAGE_KEY);
const isOpen =
savedState !== null ? (JSON.parse(savedState) as boolean) : false;
const newState = !isOpen;
localStorage.setItem(
"sidebar-tracing-default-open",
JSON.stringify(newState),
);
};
return (
<li>
<ul role="list" className="-mx-2 space-y-1">
@@ -590,15 +582,14 @@ const MainNavigation: React.FC<{
<Disclosure
as="div"
defaultOpen={
item.children.some((child) => child.current) ||
getDefaultOpen()
item.children.some((child) => child.current) || isOpen
}
>
{({ open }) => (
<>
<Disclosure.Button
className="group flex w-full items-center gap-x-3 rounded-md p-2 text-left text-sm font-semibold leading-6 hover:bg-gray-50 hover:text-indigo-600"
onClick={handleDropDownClick}
onClick={() => setIsOpen(!isOpen)}
>
{item.icon && (
<item.icon
@@ -656,7 +647,12 @@ const MainNavigation: React.FC<{
) : null}
</li>
))}
<FeedbackButtonWrapper className="w-full">
<FeedbackButtonWrapper
className="w-full"
title="Provide feedback"
description="What do you think about this project? What can be improved?"
type="feedback"
>
<li className="group flex cursor-pointer gap-x-3 rounded-md p-2 text-sm font-semibold leading-6 text-gray-700 hover:bg-gray-50 hover:text-indigo-600">
<MessageSquarePlus
className="h-6 w-6 shrink-0 text-gray-400 group-hover:text-indigo-600"
+30 -9
View File
@@ -31,10 +31,10 @@ const modelConfigDescriptions = {
"Regex pattern to match `model` parameter of generations to model pricing",
startDate:
"Date to start pricing model. If not set, model is active unless a more recent version exists.",
inputPrice: "Price per unit of input",
outputPrice: "Price per unit of output",
inputPrice: "Price per 1000 units of input",
outputPrice: "Price per 1000 units of output",
totalPrice:
"Price per unit, for models that don't have input/output specific prices",
"Price per 1000 units, for models that don't have input/output specific prices",
unit: "Unit of measurement for generative model, can be TOKENS, CHARACTERS, SECONDS, MILLISECONDS, or IMAGES.",
tokenizerId:
"Tokenizer used for this model to calculate token counts if none are ingested. Pick from list of supported tokenizers.",
@@ -107,7 +107,14 @@ export default function ModelTable({ projectId }: { projectId: string }) {
{
accessorKey: "inputPrice",
id: "inputPrice",
header: "Input Price",
header: () => {
return (
<>
Input Price{" "}
<span className="text-xs text-gray-400">/ 1k units</span>
</>
);
},
headerTooltip: {
description: modelConfigDescriptions.inputPrice,
},
@@ -116,7 +123,7 @@ export default function ModelTable({ projectId }: { projectId: string }) {
return value ? (
<span className="text-xs">
{usdFormatter(value.toNumber(), 2, 8)}
{usdFormatter(value.toNumber() * 1000, 2, 8)}
</span>
) : (
<span className="text-xs">-</span>
@@ -129,13 +136,20 @@ export default function ModelTable({ projectId }: { projectId: string }) {
headerTooltip: {
description: modelConfigDescriptions.outputPrice,
},
header: "Output Price",
header: () => {
return (
<>
Output Price{" "}
<span className="text-xs text-gray-400">/ 1k units</span>
</>
);
},
cell: ({ row }) => {
const value: Decimal | undefined = row.getValue("outputPrice");
return value ? (
<span className="text-xs">
{usdFormatter(value.toNumber(), 2, 8)}
{usdFormatter(value.toNumber() * 1000, 2, 8)}
</span>
) : (
<span className="text-xs">-</span>
@@ -145,7 +159,14 @@ export default function ModelTable({ projectId }: { projectId: string }) {
{
accessorKey: "totalPrice",
id: "totalPrice",
header: "Total Price",
header: () => {
return (
<>
Total Price{" "}
<span className="text-xs text-gray-400">/ 1k units</span>
</>
);
},
headerTooltip: {
description: modelConfigDescriptions.totalPrice,
},
@@ -154,7 +175,7 @@ export default function ModelTable({ projectId }: { projectId: string }) {
return value ? (
<span className="text-xs">
{usdFormatter(value.toNumber(), 2, 8)}
{usdFormatter(value.toNumber() * 1000, 2, 8)}
</span>
) : (
<span className="text-xs">-</span>
+29
View File
@@ -0,0 +1,29 @@
"use client";
import { Toaster as Sonner } from "sonner";
type ToasterProps = React.ComponentProps<typeof Sonner>;
const Toaster = ({ ...props }: ToasterProps) => {
return (
<Sonner
theme={"light"}
className="toaster group"
position="top-right"
toastOptions={{
classNames: {
toast:
"group toast group-[.toaster]:bg-background group-[.toaster]:text-foreground group-[.toaster]:border-border group-[.toaster]:shadow-lg",
description: "group-[.toast]:text-muted-foreground",
actionButton:
"group-[.toast]:bg-primary group-[.toast]:text-primary-foreground",
cancelButton:
"group-[.toast]:bg-muted group-[.toast]:text-muted-foreground",
},
}}
{...props}
/>
);
};
export { Toaster };
+3
View File
@@ -20,6 +20,9 @@ import { useState, useEffect } from "react";
*/
function useLocalStorage<T>(localStorageKey: string, initialValue: T) {
const [value, setValue] = useState<T>(() => {
if (typeof window === "undefined") {
return initialValue;
}
try {
const storedValue = localStorage.getItem(localStorageKey);
return storedValue ? (JSON.parse(storedValue) as T) : initialValue;
+1 -1
View File
@@ -1 +1 @@
export const VERSION = "v2.2.0";
export const VERSION = "v2.3.0";
+3
View File
@@ -26,6 +26,7 @@ export const env = createEnv({
),
NEXTAUTH_COOKIE_DOMAIN: z.string().optional(),
LANGFUSE_TEAM_SLACK_WEBHOOK: z.string().url().optional(),
LANGFUSE_TEAM_BETTERSTACK_TOKEN: z.string().optional(),
LANGFUSE_NEW_USER_SIGNUP_WEBHOOK: z.string().url().optional(),
// Add `.min(1) on ID and SECRET if you want to make sure they're not empty
LANGFUSE_ENABLE_EXPERIMENTAL_FEATURES: z.enum(["true", "false"]).optional(),
@@ -91,6 +92,8 @@ export const env = createEnv({
LANGFUSE_ENABLE_EXPERIMENTAL_FEATURES:
process.env.LANGFUSE_ENABLE_EXPERIMENTAL_FEATURES,
LANGFUSE_TEAM_SLACK_WEBHOOK: process.env.LANGFUSE_TEAM_SLACK_WEBHOOK,
LANGFUSE_TEAM_BETTERSTACK_TOKEN:
process.env.LANGFUSE_TEAM_BETTERSTACK_TOKEN,
LANGFUSE_NEW_USER_SIGNUP_WEBHOOK:
process.env.LANGFUSE_NEW_USER_SIGNUP_WEBHOOK,
SALT: process.env.SALT,
@@ -0,0 +1,28 @@
import { env } from "@/src/env.mjs";
export const sendToBetterstack = async (message: unknown) => {
if (!env.LANGFUSE_TEAM_BETTERSTACK_TOKEN)
throw new Error("LANGFUSE_TEAM_BETTERSTACK_TOKEN is not set");
const url = "https://in.logs.betterstack.com";
const headers = new Headers({
"Content-Type": "application/json",
Authorization: `Bearer ${env.LANGFUSE_TEAM_BETTERSTACK_TOKEN}`,
});
const payload = {
dt: new Date().toISOString(), // Gets the current date in ISO format (UTC)
message: JSON.stringify(message, null, 2),
};
const response = await fetch(url, {
method: "POST",
headers: headers,
body: JSON.stringify(payload),
});
if (!response.ok) {
throw new Error(`HTTP error! status: ${response.status}`);
}
};
@@ -1,42 +1,28 @@
import { useState, useEffect } from "react";
import { type VisibilityState } from "@tanstack/react-table";
import { type LangfuseColumnDef } from "@/src/components/table/types";
import useLocalStorage from "@/src/components/useLocalStorage";
import { useEffect } from "react";
function useColumnVisibility<TData>(
localStorageKey: string,
columns: LangfuseColumnDef<TData>[],
) {
const [columnVisibility, setColumnVisibility] = useState<VisibilityState>(
() => {
try {
const savedVisibility = localStorage.getItem(localStorageKey);
const visibilityState: VisibilityState = savedVisibility
? (JSON.parse(savedVisibility) as VisibilityState)
: {};
// set default visibility for columns that are not in the saved state
columns.forEach((column) => {
if (
"accessorKey" in column &&
typeof column.accessorKey === "string"
) {
if (!(column.accessorKey in visibilityState)) {
visibilityState[column.accessorKey] =
column.defaultHidden === true ? false : true;
}
}
});
return visibilityState;
} catch (e) {
console.error("Error while loading saved column visibility", e);
return {};
const initialVisibilityState = () => {
const visibilityState: VisibilityState = {};
columns.forEach((column) => {
if ("accessorKey" in column && typeof column.accessorKey === "string") {
visibilityState[column.accessorKey] =
column.defaultHidden === true ? false : true;
}
},
);
});
return visibilityState;
};
const [columnVisibility, setColumnVisibility] =
useLocalStorage<VisibilityState>(localStorageKey, initialVisibilityState());
useEffect(() => {
const localStorageItem = localStorage.getItem(localStorageKey);
if (!localStorageItem || localStorageItem === "{}") {
if (Object.keys(columnVisibility).length === 0) {
const initialVisibility: VisibilityState = {};
columns.forEach((column) => {
if ("accessorKey" in column && typeof column.accessorKey === "string") {
@@ -45,23 +31,8 @@ function useColumnVisibility<TData>(
}
});
setColumnVisibility(initialVisibility);
} else {
// make sure all columns are in the visibility state
const visibilityState = JSON.parse(localStorageItem) as VisibilityState;
columns.forEach((column) => {
if ("accessorKey" in column && typeof column.accessorKey === "string") {
if (!(column.accessorKey in visibilityState)) {
visibilityState[column.accessorKey] =
column.defaultHidden === true ? false : true;
}
}
});
}
}, [columns, localStorageKey]);
useEffect(() => {
localStorage.setItem(localStorageKey, JSON.stringify(columnVisibility));
}, [columnVisibility, localStorageKey]);
}, [columnVisibility, columns, setColumnVisibility]);
return [columnVisibility, setColumnVisibility] as const;
}
@@ -1,4 +1,3 @@
import * as Sentry from "@sentry/nextjs";
import { Button } from "@/src/components/ui/button";
import {
Dialog,
@@ -18,17 +17,28 @@ import {
FormItem,
FormMessage,
} from "@/src/components/ui/form";
import { type PropsWithChildren, useState } from "react";
import { useState } from "react";
import { useSession } from "next-auth/react";
import { Textarea } from "@/src/components/ui/textarea";
interface FeedbackDialogProps {
className?: string;
children: React.ReactNode;
title: string;
description: string;
type: "feedback" | "dashboard";
}
const formSchema = z.object({
feedback: z.string().min(3, "Must have at least 3 characters"),
});
export function FeedbackButtonWrapper(
props: PropsWithChildren<{ className?: string }>,
) {
export function FeedbackButtonWrapper({
className,
children,
title,
description,
type,
}: FeedbackDialogProps) {
const [open, setOpen] = useState(false);
const session = useSession();
@@ -40,16 +50,6 @@ export function FeedbackButtonWrapper(
});
async function onSubmit(values: z.infer<typeof formSchema>) {
// Add to sentry
if (process.env.NEXT_PUBLIC_SENTRY_DSN) {
const eventId = Sentry.captureMessage(`User submitted feedback`);
Sentry.captureUserFeedback({
event_id: eventId,
email: session.data?.user?.email ?? "",
name: session.data?.user?.name ?? "",
comments: values.feedback,
});
}
try {
const res = await fetch("https://cloud.langfuse.com/api/feedback", {
method: "POST",
@@ -57,6 +57,7 @@ export function FeedbackButtonWrapper(
"Content-Type": "application/json",
},
body: JSON.stringify({
type,
...values,
url: window.location.href,
user: session.data?.user,
@@ -84,12 +85,12 @@ export function FeedbackButtonWrapper(
return (
<Dialog open={open} onOpenChange={setOpen}>
<DialogTrigger className={props.className} asChild>
{props.children}
<DialogTrigger className={className} asChild>
{children}
</DialogTrigger>
<DialogContent>
<DialogHeader>
<DialogTitle className="mb-5">Provide feedback</DialogTitle>
<DialogTitle className="mb-5">{title}</DialogTitle>
</DialogHeader>
<Form {...form}>
<form
@@ -102,9 +103,7 @@ export function FeedbackButtonWrapper(
name="feedback"
render={({ field }) => (
<FormItem>
<FormDescription>
What do you think about this project? What can be improved?
</FormDescription>
<FormDescription>{description}</FormDescription>
<FormControl>
<Textarea {...field} />
</FormControl>
@@ -1,5 +1,5 @@
import { env } from "@/src/env.mjs";
import { runFeedbackCorsMiddleware } from "@/src/features/feedback/server/corsMiddleware";
import { sendToSlack } from "@/src/features/slack/server/slack-webhook";
import { type NextApiRequest, type NextApiResponse } from "next";
// Collects feedack from users that do not use the cloud version of the app
@@ -10,17 +10,7 @@ export default async function feedbackApiHandler(
await runFeedbackCorsMiddleware(req, res);
try {
if (!env.LANGFUSE_TEAM_SLACK_WEBHOOK)
throw new Error("LANGFUSE_TEAM_SLACK_WEBHOOK is not set");
const slackResponse = await fetch(env.LANGFUSE_TEAM_SLACK_WEBHOOK, {
method: "POST",
body: JSON.stringify({ rawBody: JSON.stringify(req.body, null, 2) }),
headers: {
"Content-Type": "application/json",
},
});
const slackResponse = await sendToSlack(req);
if (slackResponse.status === 200) {
res.status(200).json({ status: "OK" });
} else {
@@ -292,18 +292,17 @@ function FilterBuilderForm({
) : filter.type === "number" ||
filter.type === "numberObject" ? (
<Input
value={filter.value?.toString() ?? ""}
value={filter.value ?? undefined}
type="number"
step="0.01"
lang="en-US"
onChange={(e) =>
handleFilterChange(
{
...filter,
value:
isNaN(Number(e.target.value)) ||
e.target.value.endsWith(".")
? e.target.value
: Number(e.target.value),
value: isNaN(Number(e.target.value))
? e.target.value
: Number(e.target.value),
},
i,
)
@@ -58,7 +58,7 @@ export function filterToPrismaSql(
break;
case "number":
case "numberObject":
valuePrisma = Prisma.sql`${filter.value}::DOUBLE PRECISION`;
valuePrisma = Prisma.sql`${filter.value.toString()}::DOUBLE PRECISION`;
break;
case "string":
case "stringObject":
@@ -0,0 +1,47 @@
import { X } from "lucide-react";
export interface TNotification {
id: number;
releaseDate: Date;
message: string | JSX.Element;
description?: JSX.Element | string;
}
interface NotificationProps {
notification: TNotification;
setLastSeenId: (id: number) => void;
dismissToast: (t?: string | number | undefined) => void;
toast: string | number;
}
const Notification: React.FC<NotificationProps> = ({
notification,
setLastSeenId,
dismissToast,
toast,
}) => (
<div className="flex justify-between">
<div className="flex min-w-[300px] flex-1 flex-col justify-center">
<div className="m-0 text-sm font-medium leading-tight text-gray-800">
{notification.message}
</div>
{notification.description && (
<div className="mt-2 flex-1 text-sm leading-tight text-gray-800">
{notification.description}
</div>
)}
</div>
<button
className="flex h-6 w-6 cursor-pointer items-start justify-end border-none bg-transparent p-0 text-gray-800 transition-colors duration-200"
onClick={() => {
setLastSeenId(notification.id);
dismissToast(toast);
}}
aria-label="Close"
>
<X size={14} />
</button>
</div>
);
export default Notification;
@@ -0,0 +1,95 @@
import { useEffect } from "react";
import Link from "next/link";
import { toast } from "sonner";
import useLocalStorage from "@/src/components/useLocalStorage";
import Notification, {
type TNotification,
} from "@/src/features/notifications/Notification";
import { Button } from "@/src/components/ui/button";
import { env } from "@/src/env.mjs";
export const NOTIFICATIONS: TNotification[] = [
{
id: 1,
releaseDate: new Date("2024-01-29"),
message: "New: Custom model prices",
description: (
<div>
<p>
Langfuse now supports any LLM model for usage and cost tracking. The
highlights:
</p>
<ul className="ms-4 mt-2 list-outside list-disc">
<li>Define your model definitions (price, usage).</li>
<li>Optionally, ingest cost via the API/SDK</li>
<li>Support for usage in tokens, seconds and characters</li>
</ul>
{env.NEXT_PUBLIC_LANGFUSE_CLOUD_REGION === undefined && (
<p className="mt-2">
Self-hosted: The upgrade to v2 includes a breaking change. After
upgrading, you must run a migration script to ensure accurate usage
and cost calculation for previously ingested traces. Please refer to
the post for details.
</p>
)}
<Button size="sm" variant="secondary" className="mt-3">
<Link href="https://langfuse.com/changelog/2024-01-29-custom-model-prices">
Changelog post
</Link>
</Button>
</div>
),
},
];
export const useCheckNotification = (
notification: TNotification[],
authenticated: boolean,
) => {
const [lastSeenId, setLastSeenId] = useLocalStorage<number>(
"lastSeenNotificationId",
0,
);
useEffect(() => {
if (!authenticated) {
return;
}
const timeoutId = setTimeout(() => {
notification
.reverse()
.filter(
(n) =>
// only show notifications that are less than 30 days old
(new Date().getTime() - n.releaseDate.getTime()) /
(1000 * 60 * 60 * 24) <=
30,
)
.forEach((n) => {
if (n.id > lastSeenId) {
toast.custom(
(t) => (
<Notification
notification={n}
setLastSeenId={setLastSeenId}
dismissToast={toast.dismiss}
toast={t}
/>
),
{
// needed to upsert toasts in case it is rendered multiple times
id: n.id.toString(),
duration: 600_000, // 10 minutes
style: {
padding: "1rem",
border: "1px solid #e2e8f0",
borderRadius: "0.5rem",
},
},
);
}
});
}, 1500);
return () => clearTimeout(timeoutId);
}, [lastSeenId, notification, setLastSeenId, authenticated]);
};
@@ -11,7 +11,7 @@ import {
import { CodeView } from "@/src/components/ui/code";
import { useHasAccess } from "@/src/features/rbac/utils/checkAccess";
import { usePostHog } from "posthog-js/react";
import { env } from "@/src/env.mjs";
import { QuickstartExamples } from "@/src/features/public-api/components/QuickstartExamples";
export function CreateApiKeyButton(props: { projectId: string }) {
const utils = api.useUtils();
@@ -21,8 +21,7 @@ export function CreateApiKeyButton(props: { projectId: string }) {
scope: "apiKeys:create",
});
const hostname =
env.NEXT_PUBLIC_LANGFUSE_CLOUD_REGION !== "EU" ? window.origin : undefined;
const hostname = window.origin;
const mutCreateApiKey = api.apiKeys.create.useMutation({
onSuccess: () => utils.apiKeys.invalidate(),
@@ -66,28 +65,15 @@ export function CreateApiKeyButton(props: { projectId: string }) {
Create new API keys
</Button>
</DialogTrigger>
<DialogContent onPointerDownOutside={(e) => e.preventDefault()}>
{hostname ? (
<>
<DialogTitle>Hostname</DialogTitle>
<div className="mb-6">
<div className="my-2">
When connecting to Langfuse, use this hostname / baseurl.
</div>
<CodeView content={hostname} />
</div>
</>
) : null}
<DialogContent
onPointerDownOutside={(e) => e.preventDefault()}
className="max-w-full md:max-w-xl"
>
<DialogTitle>API Keys</DialogTitle>
<div className="mb-2">
<div className="text-md font-semibold">Secret Key</div>
<div className="my-2">
Please save this secret key.{" "}
<span className="font-semibold">
You will not be able to view it again
</span>
. If you lose it, you will need to generate a new one.
This key can only be viewed once. You can always generate a new key.
</div>
<CodeView content={generatedKeys?.secretKey ?? "Loading ..."} />
</div>
@@ -95,6 +81,20 @@ export function CreateApiKeyButton(props: { projectId: string }) {
<div className="text-md mb-2 font-semibold">Public Key</div>
<CodeView content={generatedKeys?.publicKey ?? "Loading ..."} />
</div>
<div>
<div className="text-md mb-2 font-semibold">Host</div>
<CodeView content={hostname} />
</div>
{generatedKeys && (
<div className="mb-2">
<div className="text-md my-2 font-semibold">Usage</div>
<QuickstartExamples
secretKey={generatedKeys.secretKey}
publicKey={generatedKeys.publicKey}
host={hostname}
/>
</div>
)}
</DialogContent>
</Dialog>
);
@@ -0,0 +1,210 @@
import { CodeView } from "@/src/components/ui/code";
import {
Tabs,
TabsList,
TabsContent,
TabsTrigger,
} from "@/src/components/ui/tabs";
export const QuickstartExamples = ({
secretKey,
publicKey,
host,
}: {
secretKey: string;
publicKey: string;
host: string;
}) => (
<Tabs defaultValue="python">
<TabsList>
<TabsTrigger value="python">Python</TabsTrigger>
<TabsTrigger value="js">JS/TS</TabsTrigger>
<TabsTrigger value="openai">OpenAI</TabsTrigger>
<TabsTrigger value="langchain">Langchain</TabsTrigger>
<TabsTrigger value="langchain-js">Langchain JS</TabsTrigger>
<TabsTrigger value="other">Other</TabsTrigger>
</TabsList>
<TabsContent value="python">
<CodeView content="pip install langfuse" className="mb-2 bg-blue-50" />
<CodeView
className="bg-blue-50"
content={`from langfuse import Langfuse\n\nlangfuse = Langfuse(\n secret_key="${secretKey}",\n public_key="${publicKey}",\n host="${host}"\n)`}
/>
<p className="mt-3 text-xs text-gray-600">
See{" "}
<a
href="https://langfuse.com/docs/get-started"
className="underline"
target="_blank"
rel="noopener noreferrer"
>
Quickstart
</a>{" "}
and{" "}
<a
href="https://langfuse.com/docs/sdk/python"
className="underline"
target="_blank"
rel="noopener noreferrer"
>
Python docs
</a>{" "}
for more details.
</p>
</TabsContent>
<TabsContent value="js">
<CodeView content="npm install langfuse" className="mb-2 bg-blue-50" />
<CodeView
className="bg-blue-50"
content={`import { Langfuse } from "langfuse";\n\nconst langfuse = new Langfuse({\n secretKey: "${secretKey}",\n publicKey: "${publicKey}",\n baseUrl: "${host}"\n});`}
/>
<p className="mt-3 text-xs text-gray-600">
See{" "}
<a
href="https://langfuse.com/docs/get-started"
className="underline"
target="_blank"
rel="noopener noreferrer"
>
Quickstart
</a>{" "}
and{" "}
<a
href="https://langfuse.com/docs/sdk/typescript"
className="underline"
target="_blank"
rel="noopener noreferrer"
>
JS/TS docs
</a>{" "}
for more details.
</p>
</TabsContent>
<TabsContent value="openai">
<p className="mt-2 text-xs text-gray-600">
The integration is a drop-in replacement for the OpenAI Python SDK. By
changing the import, Langfuse will capture all LLM calls and send them
to Langfuse asynchronously.
</p>
<CodeView content="pip install langfuse" className="my-2 bg-blue-50" />
<CodeView
title=".env"
content={`LANGFUSE_SECRET_KEY=${secretKey};\nLANGFUSE_PUBLIC_KEY=${publicKey};\nLANGFUSE_HOST="${host}";`}
className="my-2 bg-blue-50"
/>
<CodeView
content={`# remove: import openai\n\nfrom langfuse.openai import openai`}
className="my-2 bg-blue-50"
/>
<p className="mt-2 text-xs text-gray-600">
Use the OpenAI SDK as you would normally. See the{" "}
<a
href="https://langfuse.com/docs/integrations/openai"
className="underline"
target="_blank"
rel="noopener noreferrer"
>
OpenAI Integration docs
</a>{" "}
for more details.
</p>
</TabsContent>
<TabsContent value="langchain">
<p className="mt-2 text-xs text-gray-600">
The integration uses the Langchain callback system to automatically
capture detailed traces of your Langchain executions.
</p>
<CodeView content="pip install langfuse" className="my-2 bg-blue-50" />
<CodeView
title=".env"
content={`LANGFUSE_SECRET_KEY=${secretKey};\nLANGFUSE_PUBLIC_KEY=${publicKey};\nLANGFUSE_HOST="${host}";`}
className="my-2 bg-blue-50"
/>
<CodeView content={LANGCHAIN_PYTHON_CODE} className="my-2 bg-blue-50" />
<p className="mt-2 text-xs text-gray-600">
See the{" "}
<a
href="https://langfuse.com/docs/integrations/langchain/python"
className="underline"
target="_blank"
rel="noopener noreferrer"
>
Langchain Integration docs
</a>{" "}
for more details.
</p>
</TabsContent>
<TabsContent value="langchain-js">
<p className="mt-2 text-xs text-gray-600">
The integration uses the Langchain callback system to automatically
capture detailed traces of your Langchain executions.
</p>
<CodeView
content="npm install langfuse-langchain"
className="my-2 bg-blue-50"
/>
<CodeView
title=".env"
content={`LANGFUSE_SECRET_KEY=${secretKey};\nLANGFUSE_PUBLIC_KEY=${publicKey};\LANGFUSE_BASEURL="${host}";`}
className="my-2 bg-blue-50"
/>
<CodeView content={LANGCHAIN_JS_CODE} className="my-2 bg-blue-50" />
<p className="mt-2 text-xs text-gray-600">
See the{" "}
<a
href="https://langfuse.com/docs/integrations/langchain/typescript"
className="underline"
target="_blank"
rel="noopener noreferrer"
>
Langchain Integration docs
</a>{" "}
for more details.
</p>
</TabsContent>
<TabsContent value="other">
<p className="mt-2 text-xs text-gray-600">
Use the{" "}
<a
href="https://api.reference.langfuse.com/"
className="underline"
target="_blank"
rel="noopener noreferrer"
>
API
</a>{" "}
or one of the{" "}
<a
href="https://langfuse.com/docs/integrations"
className="underline"
target="_blank"
rel="noopener noreferrer"
>
native integrations
</a>{" "}
(e.g. LiteLLM, Flowise, and Langflow) to integrate with Langfuse.
</p>
</TabsContent>
</Tabs>
);
const LANGCHAIN_PYTHON_CODE = `from langfuse.callback import CallbackHandler
# <Your Langchain code here>
# Add handler to run/invoke/call/chat
agent.run("<user_input>", callbacks=[handler])`;
const LANGCHAIN_JS_CODE = `import { CallbackHandler } from "langfuse-langchain";
// Initialize Langfuse callback handler
const handler = new CallbackHandler();
// Your Langchain implementation
const chain = new LLMChain(...);
// Add handler as callback when running the Langchain agent
await chain.call(
{ input: "<user_input>" },
{ callbacks: [handler] }
);`;
@@ -0,0 +1,14 @@
import { env } from "@/src/env.mjs";
export const sendToSlack = async (message: unknown) => {
if (!env.LANGFUSE_TEAM_SLACK_WEBHOOK)
throw new Error("LANGFUSE_TEAM_SLACK_WEBHOOK is not set");
return await fetch(env.LANGFUSE_TEAM_SLACK_WEBHOOK, {
method: "POST",
body: JSON.stringify({ rawBody: JSON.stringify(message, null, 2) }),
headers: {
"Content-Type": "application/json",
},
});
};
+2 -1
View File
@@ -9,6 +9,7 @@ import {
PopoverContent,
} from "@/src/components/ui/popover";
import { Command, CommandList, CommandGroup } from "cmdk";
import { cn } from "@/src/utils/tailwind";
type TagManagerProps = {
tags: string[];
@@ -61,7 +62,7 @@ const TagManager = ({
setSelectedTags={setSelectedTags}
/>
<CommandList
className={availableTags.length > 0 ? "mt-2" : undefined}
className={cn("overflow-auto", availableTags.length > 0 && "mt-2")}
>
<CommandGroup>
{availableTags.slice(0, 5).map((value: string) => (
+7 -6
View File
@@ -30,7 +30,7 @@ import * as Sentry from "@sentry/nextjs";
export const config = {
api: {
bodyParser: {
sizeLimit: "3mb",
sizeLimit: "4.5mb",
},
},
};
@@ -56,11 +56,6 @@ export default async function handler(
message: authCheck.error,
});
if (authCheck.scope.accessLevel !== "all")
return res.status(403).json({
message: "Access denied",
});
const batchType = z.object({
batch: z.array(z.unknown()),
metadata: jsonSchema.nullish(),
@@ -246,6 +241,12 @@ const handleSingleEvent = async (
processor = new SdkLogProcessor(cleanedEvent);
}
// Deny access to non-score events if the access level is not "all"
// This is an additional safeguard to auth checks in EventProcessor
if (apiScope.accessLevel !== "all" && type !== eventTypes.SCORE_CREATE) {
throw new AuthenticationError("Access denied. Event type not allowed.");
}
return await processor.process(apiScope);
};
+32 -7
View File
@@ -25,6 +25,9 @@ import {
import { isValidOption } from "@/src/utils/types";
import { api } from "@/src/utils/api";
import { usePostHog } from "posthog-js/react";
import { FeedbackButtonWrapper } from "@/src/features/feedback/component/FeedbackButton";
import { BarChart2 } from "lucide-react";
import { Button } from "@/src/components/ui/button";
export type DashboardDateRange = {
from: Date;
@@ -92,13 +95,35 @@ export default function Start() {
return (
<div className="md:container">
<Header title={project?.name ?? "Dashboard"} />
<DatePickerWithRange
dateRange={dateRange}
setAgg={setAgg}
setDateRangeAndOption={setDateRangeAndOption}
selectedOption={selectedOption}
className=" max-w-full overflow-x-auto"
/>
<div className="flex items-center justify-between">
<DatePickerWithRange
dateRange={dateRange}
setAgg={setAgg}
setDateRangeAndOption={setDateRangeAndOption}
selectedOption={selectedOption}
className=" max-w-full overflow-x-auto"
/>
<FeedbackButtonWrapper
className="border-box"
title="Request Chart"
description="Your feedback matters! Let us know what additional data or metrics you'd like to see in your dashboard."
type="dashboard"
>
<Button
id="date"
variant={"outline"}
className={
"group justify-start gap-x-3 text-left font-semibold text-gray-700 hover:bg-gray-50 hover:text-indigo-600"
}
>
<BarChart2
className="h-6 w-6 shrink-0 text-gray-700 group-hover:text-indigo-600"
aria-hidden="true"
/>
Request Chart
</Button>
</FeedbackButtonWrapper>
</div>
<div className="grid w-full grid-cols-1 gap-4 overflow-hidden lg:grid-cols-2 xl:grid-cols-6">
<TracesBarListChart
className="col-span-1 xl:col-span-2 "
+12 -2
View File
@@ -28,6 +28,7 @@ import {
import { v4 } from "uuid";
import { type z } from "zod";
import { jsonSchema } from "@/src/utils/zod";
import { sendToBetterstack } from "@/src/features/betterstack/server/betterstack-webhook";
export interface EventProcessor {
process(
@@ -562,7 +563,16 @@ export class SdkLogProcessor implements EventProcessor {
this.event = event;
}
process() {
return undefined;
process(apiScope: ApiAccessScope) {
try {
void sendToBetterstack({
type: "sdk-log",
event: this.event,
projectId: apiScope.projectId,
});
return undefined;
} catch (error) {
return undefined;
}
}
}