* feat(hooks): opt-in prediction hooks to observe or shape every decision Add on_predict_start/on_predict_end and a Hook protocol to Agent, Router and ONNXAgent, plus on_route/on_load/on_evict on the Router. Hooks receive a mutable PredictContext: a start hook can redact or rewrite the state/questions, or call ctx.skip(...) to serve a cached result and skip the forward pass; an end hook can rewrite the results. Unset hooks are a no-op, so the default path is unchanged. Adds tests/test_hooks.py (50 checks, no weights), examples/hooks/ and docs/hooks.md. * fix(hooks): run start/error/end on start-hook failure; expose model id; per-call on_route Review pass: - A failing on_predict_start hook now still runs on_error and on_predict_end, instead of aborting before the try/finally. - Agent and ONNXAgent populate ctx.model from the checkpoint id, matching the documented context. - Router per-call hooks= now apply to on_route too (and route() takes hooks and hooks_raise), so a per-call hook covers the whole call. - normalise_hooks rejects a class instead of an instance with a clear error. Adds 9 checks to tests/test_hooks.py (59 total). * test(hooks): regression + API-stability tests; fix router cache-hit routing - tests/test_hooks_api.py pins the hook surface (parameter names/defaults, PredictContext fields, lifecycle events, exports, class defaults) so an accidental API change fails CI. - Regression checks for the review-pass fixes, plus hooks_concurrent storage. - Router.predict now adds routing on a cache-hit skip, keeping its documented return contract. - PredictContext uses identity equality/hash so a hook can store it in a set. - normalise_hooks rejects non-callable lifecycle attributes. * feat(hooks): add run_id for tracing; document real-world coverage - PredictContext.run_id is a unique id shared by every hook of one call, so a tracer can correlate start/end/error spans without its own bookkeeping. - docs/hooks.md: prior-art mapping (browser-use on_step_*, OpenAI Agents SDK RunHooks/AgentHooks), a tracing example, and a note that hooks are synchronous and must not block (laya.serve runs inference on a single worker). - tests: run_id regression + API field guard; prod scenario correlating spans. * docs(hooks): expand hooks.md into a docs/hooks/ folder Replace the single hooks.md with a seven-page reference: overview, full API, lifecycle flowcharts, error handling, patterns and anti-patterns, examples and tracing. Every public hook API is covered, links and anchors are checked by test_packaging.py, and the README/examples links point at docs/hooks/index.md. * feat(hooks): runtime registration, scoped install, per-call token budget - HookRegistry mixin: add_hook / remove_hook / hooks_installed on Agent, Router and ONNXAgent. Thread-safe; a call reads a snapshot, so mutation never disturbs a call in flight. - PredictContext.max_len / head_max_len: a start hook, or a per-call max_len= / head_max_len= argument, shapes the token budget for one call without touching shared agent config. Honored by Agent and ONNXAgent, propagated from Router. - Tests: 86 hook checks, 101 API-stability checks; docs updated (API, lifecycle, patterns, examples). --------- Co-authored-by: Nandakishor <[email protected]>
5.2 KiB
Tracing
Every hook of one call receives the same PredictContext.run_id, so a tracer can correlate the
start, end and error events (and any spans it opens) without keeping its own bookkeeping.
- run_id
- A minimal tracer
- Span lifetime
- Error spans
- OpenTelemetry
- Distributed propagation
- Nested calls
run_id
- A
uuid4().hexstring, created once per public call (predict_batch,system_one,Router.predict,ONNXAgent.system_one). - Shared by every hook of that call, including
on_errorandon_predict_end. - Not global and not persisted: it identifies a call within the process. Put it in your logs and outbound payloads to correlate across systems.
- Distinct per call, so two calls never collide.
call A: run_id=1a2b... start ──┐
├─ end
error ─┘
call B: run_id=9f8e... start ─── end
A minimal tracer
Keep a map from run_id to the span you opened at start, and close it at end or error.
import time
class Tracer:
def __init__(self):
self.spans = {}
def on_predict_start(self, ctx):
self.spans[ctx.run_id] = {
"model": ctx.model,
"started_at": ctx.started_at,
}
def on_predict_end(self, ctx):
span = self.spans.pop(ctx.run_id, None)
if span is None:
return
emit_span(
name="laya.predict",
run_id=ctx.run_id,
model=ctx.model,
duration_ms=ctx.elapsed_ms,
usage=ctx.usage,
ok=ctx.error is None,
)
def on_error(self, ctx):
# on_error runs before on_predict_end; leaving the span for on_predict_end is fine,
# or close it here if you prefer.
pass
agent = laya.load("convaiinnovations/laya", hooks=[Tracer()])
Because on_predict_end always runs, it is the natural place to close a span, and it can see
ctx.error on the failure path.
Span lifetime
on_predict_start ──► open span (run_id, model, started_at)
│
├─ inference
│
on_error ──► record ctx.error on the span
│
on_predict_end ──► close span (elapsed_ms, usage, ok)
Error spans
on_predict_end runs on the failure path with ctx.error set, so one close site handles both:
def on_predict_end(self, ctx):
span = self.spans.pop(ctx.run_id, None)
if span is None:
return
if ctx.error is not None:
span["status"] = "error"
span["error_type"] = type(ctx.error).__name__
span["error_message"] = str(ctx.error)
span["duration_ms"] = ctx.elapsed_ms
emit(span)
If you only implement on_error, remember it fires before on_predict_end; do not close the
span in both or you will double-count.
OpenTelemetry
The example examples/hooks/otel.py records counters and a
histogram. For real spans, drive the OTel API from the tracer. Hooks are synchronous, so use the
synchronous exporter (or enqueue and export from a worker).
from opentelemetry import trace
tracer = trace.get_tracer("laya")
class OTelHooks:
def __init__(self):
self.spans = {}
def on_predict_start(self, ctx):
span = tracer.start_span("laya.predict", attributes={"laya.run_id": ctx.run_id, "laya.model": ctx.model})
self.spans[ctx.run_id] = span
def on_predict_end(self, ctx):
span = self.spans.pop(ctx.run_id, None)
if span is None:
return
if ctx.usage:
span.set_attribute("laya.input_tokens", ctx.usage["input_tokens"])
if ctx.error is not None:
span.record_exception(ctx.error)
span.set_status(trace.Status(trace.StatusCode.ERROR))
span.end()
laya.load("convaiinnovations/laya", hooks=[OTelHooks()], hooks_raise=False)
Set hooks_raise=False so a tracer outage never fails a request.
Distributed propagation
run_id is a plain string, so include it in whatever leaves the process: log lines, the payload
sent to an audit service, or an HTTP header if a decision triggers a downstream call.
def audit(ctx):
requests.post(
"https://audit.example/decisions",
json=record(ctx),
headers={"X-Laya-Run-Id": ctx.run_id},
timeout=2,
)
Remember that a blocking call like this stalls the calling thread; enqueue it instead when serving concurrently. See patterns.
Nested calls
A hook that calls predict again starts a new call with a new run_id. The parent and child are
independent unless you link them yourself. Capture the parent id and pass it along:
def enrich(ctx):
child = enricher.predict(ctx.states[0], EXTRA_QUESTIONS)
record_child_span(parent_run_id=ctx.run_id, child_run_id=child.get("run_id"))
Guard against recursion (see anti-patterns); the easiest guard
is a separate enricher agent with no hooks.
See also
- API reference: the full
PredictContext. - Patterns and anti-patterns: non-blocking tracing.