mirror of
https://github.com/hanzoai/agent.git
synced 2026-09-11 21:08:51 +00:00
Traces go nowhere unless a processor sends them
Importing agents registered an exporter that posted every trace and span to api.openai.com with the default client's key, so a run against any other server sent its traces there and logged a 401. Nothing is registered now. BackendSpanExporter takes the endpoint it posts to, with no default and no OPENAI_* fallback. set_tracing_export_api_key and use_for_tracing only fed that upload, so they are gone. Co-authored-by: Hanzo Dev <[email protected]>
This commit is contained in:
@@ -18,11 +18,10 @@ pip install hanzo-agent
|
||||
|
||||
```python
|
||||
from openai import AsyncOpenAI
|
||||
from agents import Agent, Runner, set_default_openai_api, set_default_openai_client, set_tracing_disabled
|
||||
from agents import Agent, Runner, set_default_openai_api, set_default_openai_client
|
||||
|
||||
set_default_openai_client(AsyncOpenAI(base_url="http://localhost:11434/v1", api_key="ollama"))
|
||||
set_default_openai_api("chat_completions")
|
||||
set_tracing_disabled(True)
|
||||
|
||||
agent = Agent(name="Assistant", instructions="You are a helpful assistant.", model="qwen3:0.6b")
|
||||
result = Runner.run_sync(agent, "Write a haiku about recursion in programming.")
|
||||
@@ -37,7 +36,7 @@ A loop unrolled with grace,
|
||||
Ends with a loop.
|
||||
```
|
||||
|
||||
The three calls before the agent decide where it runs:
|
||||
The two calls before the agent decide where it runs:
|
||||
|
||||
- `set_default_openai_client` names the endpoint and the key. Above it is a
|
||||
local server on port 11434, Ollama with `qwen3:0.6b` pulled. For the hosted
|
||||
@@ -45,7 +44,6 @@ The three calls before the agent decide where it runs:
|
||||
and a model id from `https://api.hanzo.ai/v1/models`.
|
||||
- `set_default_openai_api("chat_completions")` sends `POST /v1/chat/completions`.
|
||||
Without it the SDK sends `POST /v1/responses` instead.
|
||||
- `set_tracing_disabled(True)` stops the trace export described under Tracing.
|
||||
|
||||
## Tools
|
||||
|
||||
@@ -80,11 +78,20 @@ object rather than `choices` ends the run with `TypeError`. A
|
||||
|
||||
## Tracing
|
||||
|
||||
Tracing is on by default. After each run the SDK posts the trace and its spans
|
||||
to `https://api.openai.com/v1/traces/ingest`, authenticated with the default
|
||||
client's key. Any other key is refused there, and the run logs
|
||||
`Tracing client error 401: ...`. Turn it off with `set_tracing_disabled(True)`
|
||||
or `OPENAI_AGENTS_DISABLE_TRACING=1`.
|
||||
Runs create traces and spans and send them nowhere. To export them, add a
|
||||
processor:
|
||||
|
||||
```python
|
||||
from agents import add_trace_processor
|
||||
from agents.tracing.processors import BackendSpanExporter, BatchTraceProcessor
|
||||
|
||||
exporter = BackendSpanExporter(endpoint="https://collector.example/ingest", api_key="...")
|
||||
add_trace_processor(BatchTraceProcessor(exporter))
|
||||
```
|
||||
|
||||
The exporter posts batches of JSON to that endpoint with the key as a bearer
|
||||
token. `set_tracing_disabled(True)` or `OPENAI_AGENTS_DISABLE_TRACING=1` stops
|
||||
creating them.
|
||||
|
||||
## Extras
|
||||
|
||||
|
||||
+3
-9
@@ -2,7 +2,7 @@
|
||||
|
||||
## API keys and clients
|
||||
|
||||
By default, the SDK looks for the `OPENAI_API_KEY` environment variable for LLM requests and tracing, as soon as it is imported. If you are unable to set that environment variable before your app starts, you can use the [set_default_openai_key()][agents.set_default_openai_key] function to set the key.
|
||||
By default, the SDK looks for the `OPENAI_API_KEY` environment variable for LLM requests, as soon as it is imported. If you are unable to set that environment variable before your app starts, you can use the [set_default_openai_key()][agents.set_default_openai_key] function to set the key.
|
||||
|
||||
```python
|
||||
from agents import set_default_openai_key
|
||||
@@ -30,15 +30,9 @@ set_default_openai_api("chat_completions")
|
||||
|
||||
## Tracing
|
||||
|
||||
Tracing is enabled by default. It uses the Hanzo AI API keys from the section above by default (i.e. the environment variable or the default key you set). You can specifically set the API key used for tracing by using the [`set_tracing_export_api_key`][agents.set_tracing_export_api_key] function.
|
||||
Tracing creates traces and spans and sends them nowhere until you add a processor that does; see [custom tracing processors](tracing.md#custom-tracing-processors).
|
||||
|
||||
```python
|
||||
from agents import set_tracing_export_api_key
|
||||
|
||||
set_tracing_export_api_key("sk-...")
|
||||
```
|
||||
|
||||
You can also disable tracing entirely by using the [`set_tracing_disabled()`][agents.set_tracing_disabled] function.
|
||||
You can disable tracing entirely by using the [`set_tracing_disabled()`][agents.set_tracing_disabled] function.
|
||||
|
||||
```python
|
||||
from agents import set_tracing_disabled
|
||||
|
||||
@@ -59,22 +59,12 @@ You can use other LLM providers in 3 ways (examples [here](https://github.com/op
|
||||
2. [`ModelProvider`][agents.models.interface.ModelProvider] is at the `Runner.run` level. This lets you say "use a custom model provider for all agents in this run". See a configurable example in [examples/model_providers/custom_example_provider.py](https://github.com/openai/openai-agents-python/tree/main/examples/model_providers/custom_example_provider.py).
|
||||
3. [`Agent.model`][agents.agent.Agent.model] lets you specify the model on a specific Agent instance. This enables you to mix and match different providers for different agents. See a configurable example in [examples/model_providers/custom_example_agent.py](https://github.com/openai/openai-agents-python/tree/main/examples/model_providers/custom_example_agent.py).
|
||||
|
||||
In cases where you do not have an API key from `platform.openai.com`, we recommend disabling tracing via `set_tracing_disabled()`, or setting up a [different tracing processor](tracing.md).
|
||||
|
||||
!!! note
|
||||
|
||||
In these examples, we use the Chat Completions API/model, because most LLM providers don't yet support the Responses API. If your LLM provider does support it, we recommend using Responses.
|
||||
|
||||
## Common issues with using other LLM providers
|
||||
|
||||
### Tracing client error 401
|
||||
|
||||
If you get errors related to tracing, this is because traces are uploaded to Hanzo AI servers, and you don't have an Hanzo AI API key. You have three options to resolve this:
|
||||
|
||||
1. Disable tracing entirely: [`set_tracing_disabled(True)`][agents.set_tracing_disabled].
|
||||
2. Set an Hanzo AI key for tracing: [`set_tracing_export_api_key(...)`][agents.set_tracing_export_api_key]. This API key will only be used for uploading traces, and must be from [platform.openai.com](https://platform.openai.com/).
|
||||
3. Use a non-Hanzo AI trace processor. See the [tracing docs](tracing.md#custom-tracing-processors).
|
||||
|
||||
### Responses API support
|
||||
|
||||
The SDK uses the Responses API by default, but most other LLM providers don't yet support it. You may see 404s or similar issues as a result. To resolve, you have two options:
|
||||
|
||||
+1
-1
@@ -178,7 +178,7 @@ if __name__ == "__main__":
|
||||
|
||||
## View your traces
|
||||
|
||||
To review what happened during your agent run, navigate to the [Trace viewer in the Hanzo AI Dashboard](https://platform.openai.com/traces) to view traces of your agent runs.
|
||||
Runs create traces and send them nowhere by default. To review what happened during a run, add a processor that exports them; see [custom tracing processors](tracing.md#custom-tracing-processors).
|
||||
|
||||
## Next steps
|
||||
|
||||
|
||||
@@ -7,7 +7,6 @@
|
||||
- set_default_openai_key
|
||||
- set_default_openai_client
|
||||
- set_default_openai_api
|
||||
- set_tracing_export_api_key
|
||||
- set_tracing_disabled
|
||||
- set_trace_processors
|
||||
- enable_verbose_stdout_logging
|
||||
|
||||
+5
-5
@@ -1,6 +1,6 @@
|
||||
# Tracing
|
||||
|
||||
The Agent SDK includes built-in tracing, collecting a comprehensive record of events during an agent run: LLM generations, tool calls, handoffs, guardrails, and even custom events that occur. Using the [Traces dashboard](https://platform.openai.com/traces), you can debug, visualize, and monitor your workflows during development and in production.
|
||||
The Agent SDK includes built-in tracing, collecting a comprehensive record of events during an agent run: LLM generations, tool calls, handoffs, guardrails, and even custom events that occur. Traces stay in the process until you add a [processor](#custom-tracing-processors) that sends them somewhere.
|
||||
|
||||
!!!note
|
||||
|
||||
@@ -81,12 +81,12 @@ Some spans track potentially sensitive data. For example, the `generation_span()
|
||||
The high level architecture for tracing is:
|
||||
|
||||
- At initialization, we create a global [`TraceProvider`][agents.tracing.setup.TraceProvider], which is responsible for creating traces.
|
||||
- We configure the `TraceProvider` with a [`BatchTraceProcessor`][agents.tracing.processors.BatchTraceProcessor] that sends traces/spans in batches to a [`BackendSpanExporter`][agents.tracing.processors.BackendSpanExporter], which exports the spans and traces to the Hanzo AI backend in batches.
|
||||
- The `TraceProvider` starts with no processors, so traces and spans go nowhere until you add one. A [`BatchTraceProcessor`][agents.tracing.processors.BatchTraceProcessor] over a [`BackendSpanExporter`][agents.tracing.processors.BackendSpanExporter] posts them in batches to the endpoint you give the exporter.
|
||||
|
||||
To customize this default setup, to send traces to alternative or additional backends or modifying exporter behavior, you have two options:
|
||||
There are two ways to add processors:
|
||||
|
||||
1. [`add_trace_processor()`][agents.tracing.add_trace_processor] lets you add an **additional** trace processor that will receive traces and spans as they are ready. This lets you do your own processing in addition to sending traces to Hanzo AI's backend.
|
||||
2. [`set_trace_processors()`][agents.tracing.set_trace_processors] lets you **replace** the default processors with your own trace processors. This means traces will not be sent to the Hanzo AI backend unless you include a `TracingProcessor` that does so.
|
||||
- [`add_trace_processor()`][agents.tracing.add_trace_processor] adds a processor that receives traces and spans as they are ready.
|
||||
- [`set_trace_processors()`][agents.tracing.set_trace_processors] replaces the current processors with the ones you pass.
|
||||
|
||||
External trace processors include:
|
||||
|
||||
|
||||
@@ -18,10 +18,6 @@ if not BASE_URL or not API_KEY or not MODEL_NAME:
|
||||
1. Create a custom Hanzo AI client.
|
||||
2. Create a `Model` that uses the custom client.
|
||||
3. Set the `model` on the Agent.
|
||||
|
||||
Note that in this example, we disable tracing under the assumption that you don't have an API key
|
||||
from platform.openai.com. If you do have one, you can either set the `OPENAI_API_KEY` env var
|
||||
or call set_tracing_export_api_key() to set a tracing specific key.
|
||||
"""
|
||||
client = AsyncHanzo AI(base_url=BASE_URL, api_key=API_KEY)
|
||||
set_tracing_disabled(disabled=True)
|
||||
|
||||
@@ -24,19 +24,15 @@ if not BASE_URL or not API_KEY or not MODEL_NAME:
|
||||
|
||||
"""This example uses a custom provider for all requests by default. We do three things:
|
||||
1. Create a custom client.
|
||||
2. Set it as the default Hanzo AI client, and don't use it for tracing.
|
||||
2. Set it as the default Hanzo AI client.
|
||||
3. Set the default API as Chat Completions, as most LLM providers don't yet support Responses API.
|
||||
|
||||
Note that in this example, we disable tracing under the assumption that you don't have an API key
|
||||
from platform.openai.com. If you do have one, you can either set the `OPENAI_API_KEY` env var
|
||||
or call set_tracing_export_api_key() to set a tracing specific key.
|
||||
"""
|
||||
|
||||
client = AsyncHanzo AI(
|
||||
base_url=BASE_URL,
|
||||
api_key=API_KEY,
|
||||
)
|
||||
set_default_openai_client(client=client, use_for_tracing=False)
|
||||
set_default_openai_client(client=client)
|
||||
set_default_openai_api("chat_completions")
|
||||
set_tracing_disabled(disabled=True)
|
||||
|
||||
|
||||
@@ -31,10 +31,6 @@ others. Steps:
|
||||
1. Create a custom Hanzo AI client.
|
||||
2. Create a ModelProvider that uses the custom client.
|
||||
3. Use the ModelProvider in calls to Runner.run(), only when we want to use the custom LLM provider.
|
||||
|
||||
Note that in this example, we disable tracing under the assumption that you don't have an API key
|
||||
from platform.openai.com. If you do have one, you can either set the `OPENAI_API_KEY` env var
|
||||
or call set_tracing_export_api_key() to set a tracing specific key.
|
||||
"""
|
||||
client = AsyncHanzo AI(base_url=BASE_URL, api_key=API_KEY)
|
||||
set_tracing_disabled(disabled=True)
|
||||
|
||||
@@ -23,7 +23,7 @@ class ResearchManager:
|
||||
with trace("Research trace", trace_id=trace_id):
|
||||
self.printer.update_item(
|
||||
"trace_id",
|
||||
f"View trace: https://platform.openai.com/traces/{trace_id}",
|
||||
f"Trace id: {trace_id}",
|
||||
is_done=True,
|
||||
hide_checkmark=True,
|
||||
)
|
||||
|
||||
@@ -3,7 +3,7 @@
|
||||
$ uv run python -m examples.research_bot.main
|
||||
|
||||
What would you like to research? Best surfboards for beginners. I can catch my own waves, but previously used an 11ft board. What should I look for, what are my options? Various budget ranges.
|
||||
View trace: https://platform.openai.com/traces/trace_...
|
||||
Trace id: trace_...
|
||||
Starting research...
|
||||
✅ Will perform 15 searches
|
||||
✅ Searching... 15/15 completed
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
$ uv run python -m examples.research_bot.main
|
||||
What would you like to research? Caribbean vacation spots in April, optimizing for surfing, hiking and water sports
|
||||
View trace: https://platform.openai.com/traces/trace_....
|
||||
Trace id: trace_....
|
||||
Starting research...
|
||||
✅ Will perform 15 searches
|
||||
✅ Searching... 15/15 completed
|
||||
|
||||
+8
-16
@@ -87,7 +87,6 @@ from .tracing import (
|
||||
handoff_span,
|
||||
set_trace_processors,
|
||||
set_tracing_disabled,
|
||||
set_tracing_export_api_key,
|
||||
trace,
|
||||
)
|
||||
from .usage import Usage
|
||||
@@ -161,32 +160,26 @@ from .zap import (
|
||||
)
|
||||
|
||||
|
||||
def set_default_openai_key(key: str, use_for_tracing: bool = True) -> None:
|
||||
"""Set the default OpenAI API key to use for LLM requests (and optionally tracing(). This is
|
||||
only necessary if the OPENAI_API_KEY environment variable is not already set.
|
||||
def set_default_openai_key(key: str) -> None:
|
||||
"""Set the default OpenAI API key to use for LLM requests. This is only necessary if the
|
||||
OPENAI_API_KEY environment variable is not already set.
|
||||
|
||||
If provided, this key will be used instead of the OPENAI_API_KEY environment variable.
|
||||
|
||||
Args:
|
||||
key: The OpenAI key to use.
|
||||
use_for_tracing: Whether to also use this key to send traces to OpenAI. Defaults to True
|
||||
If False, you'll either need to set the OPENAI_API_KEY environment variable or call
|
||||
set_tracing_export_api_key() with the API key you want to use for tracing.
|
||||
"""
|
||||
_config.set_default_openai_key(key, use_for_tracing)
|
||||
_config.set_default_openai_key(key)
|
||||
|
||||
|
||||
def set_default_openai_client(client: AsyncOpenAI, use_for_tracing: bool = True) -> None:
|
||||
"""Set the default OpenAI client to use for LLM requests and/or tracing. If provided, this
|
||||
client will be used instead of the default OpenAI client.
|
||||
def set_default_openai_client(client: AsyncOpenAI) -> None:
|
||||
"""Set the default OpenAI client to use for LLM requests. If provided, this client will be
|
||||
used instead of the default OpenAI client.
|
||||
|
||||
Args:
|
||||
client: The OpenAI client to use.
|
||||
use_for_tracing: Whether to use the API key from this client for uploading traces. If False,
|
||||
you'll either need to set the OPENAI_API_KEY environment variable or call
|
||||
set_tracing_export_api_key() with the API key you want to use for tracing.
|
||||
"""
|
||||
_config.set_default_openai_client(client, use_for_tracing)
|
||||
_config.set_default_openai_client(client)
|
||||
|
||||
|
||||
def set_default_openai_api(api: Literal["chat_completions", "responses"]) -> None:
|
||||
@@ -291,7 +284,6 @@ __all__ = [
|
||||
"set_default_openai_key",
|
||||
"set_default_openai_client",
|
||||
"set_default_openai_api",
|
||||
"set_tracing_export_api_key",
|
||||
"enable_verbose_stdout_logging",
|
||||
"gen_trace_id",
|
||||
"gen_span_id",
|
||||
|
||||
@@ -2,22 +2,15 @@ from openai import AsyncOpenAI
|
||||
from typing_extensions import Literal
|
||||
|
||||
from .models import _openai_shared
|
||||
from .tracing import set_tracing_export_api_key
|
||||
|
||||
|
||||
def set_default_openai_key(key: str, use_for_tracing: bool) -> None:
|
||||
def set_default_openai_key(key: str) -> None:
|
||||
_openai_shared.set_default_openai_key(key)
|
||||
|
||||
if use_for_tracing:
|
||||
set_tracing_export_api_key(key)
|
||||
|
||||
|
||||
def set_default_openai_client(client: AsyncOpenAI, use_for_tracing: bool) -> None:
|
||||
def set_default_openai_client(client: AsyncOpenAI) -> None:
|
||||
_openai_shared.set_default_openai_client(client)
|
||||
|
||||
if use_for_tracing:
|
||||
set_tracing_export_api_key(client.api_key)
|
||||
|
||||
|
||||
def set_default_openai_api(api: Literal["chat_completions", "responses"]) -> None:
|
||||
if api == "chat_completions":
|
||||
|
||||
@@ -13,7 +13,6 @@ from .create import (
|
||||
trace,
|
||||
)
|
||||
from .processor_interface import TracingProcessor
|
||||
from .processors import default_exporter, default_processor
|
||||
from .setup import GLOBAL_TRACE_PROVIDER
|
||||
from .span_data import (
|
||||
AgentSpanData,
|
||||
@@ -81,17 +80,4 @@ def set_tracing_disabled(disabled: bool) -> None:
|
||||
GLOBAL_TRACE_PROVIDER.set_disabled(disabled)
|
||||
|
||||
|
||||
def set_tracing_export_api_key(api_key: str) -> None:
|
||||
"""
|
||||
Set the OpenAI API key for the backend exporter.
|
||||
"""
|
||||
default_exporter().set_api_key(api_key)
|
||||
|
||||
|
||||
# Add the default processor, which exports traces and spans to the backend in batches. You can
|
||||
# change the default behavior by either:
|
||||
# 1. calling add_trace_processor(), which adds additional processors, or
|
||||
# 2. calling set_trace_processors(), which replaces the default processor.
|
||||
add_trace_processor(default_processor())
|
||||
|
||||
atexit.register(GLOBAL_TRACE_PROVIDER.shutdown)
|
||||
|
||||
@@ -1,6 +1,5 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import queue
|
||||
import random
|
||||
import threading
|
||||
@@ -29,31 +28,22 @@ class ConsoleSpanExporter(TracingExporter):
|
||||
class BackendSpanExporter(TracingExporter):
|
||||
def __init__(
|
||||
self,
|
||||
endpoint: str,
|
||||
api_key: str | None = None,
|
||||
organization: str | None = None,
|
||||
project: str | None = None,
|
||||
endpoint: str = "https://api.openai.com/v1/traces/ingest",
|
||||
max_retries: int = 3,
|
||||
base_delay: float = 1.0,
|
||||
max_delay: float = 30.0,
|
||||
):
|
||||
"""
|
||||
Args:
|
||||
api_key: The API key for the "Authorization" header. Defaults to
|
||||
`os.environ["OPENAI_API_KEY"]` if not provided.
|
||||
organization: The OpenAI organization to use. Defaults to
|
||||
`os.environ["OPENAI_ORG_ID"]` if not provided.
|
||||
project: The OpenAI project to use. Defaults to
|
||||
`os.environ["OPENAI_PROJECT_ID"]` if not provided.
|
||||
endpoint: The HTTP endpoint to which traces/spans are posted.
|
||||
api_key: The API key for the "Authorization" header.
|
||||
max_retries: Maximum number of retries upon failures.
|
||||
base_delay: Base delay (in seconds) for the first backoff.
|
||||
max_delay: Maximum delay (in seconds) for backoff growth.
|
||||
"""
|
||||
self.api_key = api_key or os.environ.get("OPENAI_API_KEY")
|
||||
self.organization = organization or os.environ.get("OPENAI_ORG_ID")
|
||||
self.project = project or os.environ.get("OPENAI_PROJECT_ID")
|
||||
self.endpoint = endpoint
|
||||
self.api_key = api_key
|
||||
self.max_retries = max_retries
|
||||
self.base_delay = base_delay
|
||||
self.max_delay = max_delay
|
||||
@@ -61,21 +51,12 @@ class BackendSpanExporter(TracingExporter):
|
||||
# Keep a client open for connection pooling across multiple export calls
|
||||
self._client = httpx.Client(timeout=httpx.Timeout(timeout=60, connect=5.0))
|
||||
|
||||
def set_api_key(self, api_key: str):
|
||||
"""Set the OpenAI API key for the exporter.
|
||||
|
||||
Args:
|
||||
api_key: The OpenAI API key to use. This is the same key used by the OpenAI Python
|
||||
client.
|
||||
"""
|
||||
self.api_key = api_key
|
||||
|
||||
def export(self, items: list[Trace | Span[Any]]) -> None:
|
||||
if not items:
|
||||
return
|
||||
|
||||
if not self.api_key:
|
||||
logger.warning("OPENAI_API_KEY is not set, skipping trace export")
|
||||
logger.warning("No API key set, skipping trace export")
|
||||
return
|
||||
|
||||
data = [item.export() for item in items if item.export()]
|
||||
@@ -84,7 +65,6 @@ class BackendSpanExporter(TracingExporter):
|
||||
headers = {
|
||||
"Authorization": f"Bearer {self.api_key}",
|
||||
"Content-Type": "application/json",
|
||||
"OpenAI-Beta": "traces=v1",
|
||||
}
|
||||
|
||||
# Exponential backoff loop
|
||||
@@ -241,18 +221,3 @@ class BatchTraceProcessor(TracingProcessor):
|
||||
|
||||
# Export the batch
|
||||
self._exporter.export(items_to_export)
|
||||
|
||||
|
||||
# Create a shared global instance:
|
||||
_global_exporter = BackendSpanExporter()
|
||||
_global_processor = BatchTraceProcessor(_global_exporter)
|
||||
|
||||
|
||||
def default_exporter() -> BackendSpanExporter:
|
||||
"""The default exporter, which exports traces and spans to the backend in batches."""
|
||||
return _global_exporter
|
||||
|
||||
|
||||
def default_processor() -> BatchTraceProcessor:
|
||||
"""The default processor, which exports traces and spans to the backend in batches."""
|
||||
return _global_processor
|
||||
|
||||
@@ -0,0 +1,50 @@
|
||||
"""A local OpenAI-compatible server for tests that need a real HTTP round trip."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import threading
|
||||
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
|
||||
from typing import Any
|
||||
|
||||
|
||||
class Server:
|
||||
"""Serves on 127.0.0.1 and answers a POST to a path in `replies` with that reply as JSON.
|
||||
|
||||
Every request is appended to `requests` as (method, path). A CONNECT is recorded and refused,
|
||||
so a process whose HTTP(S) proxy is this server records requests bound for any other host.
|
||||
"""
|
||||
|
||||
def __init__(self, replies: dict[str, Any]) -> None:
|
||||
self.replies = replies
|
||||
self.requests: list[tuple[str, str]] = []
|
||||
requests = self.requests
|
||||
|
||||
class Handler(BaseHTTPRequestHandler):
|
||||
def do_POST(self) -> None:
|
||||
requests.append(("POST", self.path))
|
||||
self.rfile.read(int(self.headers.get("Content-Length") or 0))
|
||||
body = json.dumps(replies[self.path]).encode()
|
||||
self.send_response(200)
|
||||
self.send_header("Content-Type", "application/json")
|
||||
self.send_header("Content-Length", str(len(body)))
|
||||
self.end_headers()
|
||||
self.wfile.write(body)
|
||||
|
||||
def do_CONNECT(self) -> None:
|
||||
requests.append(("CONNECT", self.path))
|
||||
self.send_error(403)
|
||||
|
||||
def log_message(self, format: str, *args: Any) -> None:
|
||||
pass
|
||||
|
||||
self._server = ThreadingHTTPServer(("127.0.0.1", 0), Handler)
|
||||
self.url = f"http://127.0.0.1:{self._server.server_address[1]}"
|
||||
|
||||
def __enter__(self) -> Server:
|
||||
threading.Thread(target=self._server.serve_forever, daemon=True).start()
|
||||
return self
|
||||
|
||||
def __exit__(self, *exc: object) -> None:
|
||||
self._server.shutdown()
|
||||
self._server.server_close()
|
||||
@@ -1,4 +1,6 @@
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
from unittest.mock import MagicMock, patch
|
||||
|
||||
@@ -11,6 +13,8 @@ from agents.tracing.span_data import AgentSpanData
|
||||
from agents.tracing.spans import SpanImpl
|
||||
from agents.tracing.traces import TraceImpl
|
||||
|
||||
from .server import Server
|
||||
|
||||
|
||||
def get_span(processor: TracingProcessor) -> SpanImpl[AgentSpanData]:
|
||||
"""Create a minimal agent span for testing processors."""
|
||||
@@ -185,9 +189,12 @@ def mock_processor():
|
||||
return processor
|
||||
|
||||
|
||||
ENDPOINT = "https://collector.example/ingest"
|
||||
|
||||
|
||||
@patch("httpx.Client")
|
||||
def test_backend_span_exporter_no_items(mock_client):
|
||||
exporter = BackendSpanExporter(api_key="test_key")
|
||||
exporter = BackendSpanExporter(ENDPOINT, api_key="test_key")
|
||||
exporter.export([])
|
||||
# No calls should be made if there are no items
|
||||
mock_client.return_value.post.assert_not_called()
|
||||
@@ -196,15 +203,12 @@ def test_backend_span_exporter_no_items(mock_client):
|
||||
|
||||
@patch("httpx.Client")
|
||||
def test_backend_span_exporter_no_api_key(mock_client):
|
||||
# Ensure that os.environ is empty (sometimes devs have the openai api key set in their env)
|
||||
exporter = BackendSpanExporter(ENDPOINT)
|
||||
exporter.export([get_span(mock_processor())])
|
||||
|
||||
with patch.dict(os.environ, {}, clear=True):
|
||||
exporter = BackendSpanExporter(api_key=None)
|
||||
exporter.export([get_span(mock_processor())])
|
||||
|
||||
# Should log an error and return without calling post
|
||||
mock_client.return_value.post.assert_not_called()
|
||||
exporter.close()
|
||||
# Should log an error and return without calling post
|
||||
mock_client.return_value.post.assert_not_called()
|
||||
exporter.close()
|
||||
|
||||
|
||||
@patch("httpx.Client")
|
||||
@@ -213,11 +217,12 @@ def test_backend_span_exporter_2xx_success(mock_client):
|
||||
mock_response.status_code = 200
|
||||
mock_client.return_value.post.return_value = mock_response
|
||||
|
||||
exporter = BackendSpanExporter(api_key="test_key")
|
||||
exporter = BackendSpanExporter(ENDPOINT, api_key="test_key")
|
||||
exporter.export([get_span(mock_processor()), get_trace(mock_processor())])
|
||||
|
||||
# Should have called post exactly once
|
||||
# Should have called post exactly once, to the endpoint it was given
|
||||
mock_client.return_value.post.assert_called_once()
|
||||
assert mock_client.return_value.post.call_args.kwargs["url"] == ENDPOINT
|
||||
exporter.close()
|
||||
|
||||
|
||||
@@ -228,7 +233,7 @@ def test_backend_span_exporter_4xx_client_error(mock_client):
|
||||
mock_response.text = "Bad Request"
|
||||
mock_client.return_value.post.return_value = mock_response
|
||||
|
||||
exporter = BackendSpanExporter(api_key="test_key")
|
||||
exporter = BackendSpanExporter(ENDPOINT, api_key="test_key")
|
||||
exporter.export([get_span(mock_processor())])
|
||||
|
||||
# 4xx should not be retried
|
||||
@@ -244,7 +249,9 @@ def test_backend_span_exporter_5xx_retry(mock_client, patched_time_sleep):
|
||||
# Make post() return 500 every time
|
||||
mock_client.return_value.post.return_value = mock_response
|
||||
|
||||
exporter = BackendSpanExporter(api_key="test_key", max_retries=3, base_delay=0.1, max_delay=0.2)
|
||||
exporter = BackendSpanExporter(
|
||||
ENDPOINT, api_key="test_key", max_retries=3, base_delay=0.1, max_delay=0.2
|
||||
)
|
||||
exporter.export([get_span(mock_processor())])
|
||||
|
||||
# Should retry up to max_retries times
|
||||
@@ -258,7 +265,9 @@ def test_backend_span_exporter_request_error(mock_client, patched_time_sleep):
|
||||
# Make post() raise a RequestError each time
|
||||
mock_client.return_value.post.side_effect = httpx.RequestError("Network error")
|
||||
|
||||
exporter = BackendSpanExporter(api_key="test_key", max_retries=2, base_delay=0.1, max_delay=0.2)
|
||||
exporter = BackendSpanExporter(
|
||||
ENDPOINT, api_key="test_key", max_retries=2, base_delay=0.1, max_delay=0.2
|
||||
)
|
||||
exporter.export([get_span(mock_processor())])
|
||||
|
||||
# Should retry up to max_retries times
|
||||
@@ -269,8 +278,59 @@ def test_backend_span_exporter_request_error(mock_client, patched_time_sleep):
|
||||
|
||||
@patch("httpx.Client")
|
||||
def test_backend_span_exporter_close(mock_client):
|
||||
exporter = BackendSpanExporter(api_key="test_key")
|
||||
exporter = BackendSpanExporter(ENDPOINT, api_key="test_key")
|
||||
exporter.close()
|
||||
|
||||
# Ensure underlying http client is closed
|
||||
mock_client.return_value.close.assert_called_once()
|
||||
|
||||
|
||||
RESPONSE = {
|
||||
"id": "resp_1",
|
||||
"object": "response",
|
||||
"created_at": 0,
|
||||
"model": "m",
|
||||
"status": "completed",
|
||||
"output": [
|
||||
{
|
||||
"type": "message",
|
||||
"id": "msg_1",
|
||||
"role": "assistant",
|
||||
"status": "completed",
|
||||
"content": [{"type": "output_text", "text": "hi", "annotations": []}],
|
||||
}
|
||||
],
|
||||
"parallel_tool_calls": False,
|
||||
"tool_choice": "auto",
|
||||
"tools": [],
|
||||
}
|
||||
|
||||
|
||||
def test_agent_run_with_defaults_sends_no_traces():
|
||||
# A fresh interpreter, so the test processor from conftest does not replace the defaults. The
|
||||
# server is its model endpoint and its HTTP(S) proxy: a request to any other host arrives here
|
||||
# as a CONNECT.
|
||||
with Server({"/v1/responses": RESPONSE}) as server:
|
||||
env = {
|
||||
k: v
|
||||
for k, v in os.environ.items()
|
||||
if not k.startswith("OPENAI") and "proxy" not in k.lower()
|
||||
}
|
||||
env.update(
|
||||
OPENAI_API_KEY="sk-test",
|
||||
OPENAI_BASE_URL=f"{server.url}/v1",
|
||||
HTTP_PROXY=server.url,
|
||||
HTTPS_PROXY=server.url,
|
||||
NO_PROXY="127.0.0.1",
|
||||
)
|
||||
code = (
|
||||
"import asyncio\n"
|
||||
"from agents import Agent, Runner\n"
|
||||
"asyncio.run(Runner.run(Agent(name='a', model='m'), 'hi'))"
|
||||
)
|
||||
done = subprocess.run(
|
||||
[sys.executable, "-c", code], env=env, capture_output=True, text=True, timeout=60
|
||||
)
|
||||
|
||||
assert done.returncode == 0, done.stderr
|
||||
assert server.requests == [("POST", "/v1/responses")]
|
||||
|
||||
Reference in New Issue
Block a user