Files
b668a1b6bc feat(hooks): opt-in prediction hooks to observe or shape every decision (#270)
* feat(hooks): opt-in prediction hooks to observe or shape every decision

Add on_predict_start/on_predict_end and a Hook protocol to Agent, Router and
ONNXAgent, plus on_route/on_load/on_evict on the Router. Hooks receive a mutable
PredictContext: a start hook can redact or rewrite the state/questions, or call
ctx.skip(...) to serve a cached result and skip the forward pass; an end hook can
rewrite the results. Unset hooks are a no-op, so the default path is unchanged.

Adds tests/test_hooks.py (50 checks, no weights), examples/hooks/ and
docs/hooks.md.

* fix(hooks): run start/error/end on start-hook failure; expose model id; per-call on_route

Review pass:
- A failing on_predict_start hook now still runs on_error and on_predict_end,
  instead of aborting before the try/finally.
- Agent and ONNXAgent populate ctx.model from the checkpoint id, matching the
  documented context.
- Router per-call hooks= now apply to on_route too (and route() takes hooks and
  hooks_raise), so a per-call hook covers the whole call.
- normalise_hooks rejects a class instead of an instance with a clear error.

Adds 9 checks to tests/test_hooks.py (59 total).

* test(hooks): regression + API-stability tests; fix router cache-hit routing

- tests/test_hooks_api.py pins the hook surface (parameter names/defaults,
  PredictContext fields, lifecycle events, exports, class defaults) so an
  accidental API change fails CI.
- Regression checks for the review-pass fixes, plus hooks_concurrent storage.
- Router.predict now adds routing on a cache-hit skip, keeping its documented
  return contract.
- PredictContext uses identity equality/hash so a hook can store it in a set.
- normalise_hooks rejects non-callable lifecycle attributes.

* feat(hooks): add run_id for tracing; document real-world coverage

- PredictContext.run_id is a unique id shared by every hook of one call, so a
  tracer can correlate start/end/error spans without its own bookkeeping.
- docs/hooks.md: prior-art mapping (browser-use on_step_*, OpenAI Agents SDK
  RunHooks/AgentHooks), a tracing example, and a note that hooks are synchronous
  and must not block (laya.serve runs inference on a single worker).
- tests: run_id regression + API field guard; prod scenario correlating spans.

* docs(hooks): expand hooks.md into a docs/hooks/ folder

Replace the single hooks.md with a seven-page reference: overview, full API,
lifecycle flowcharts, error handling, patterns and anti-patterns, examples and
tracing. Every public hook API is covered, links and anchors are checked by
test_packaging.py, and the README/examples links point at docs/hooks/index.md.

* feat(hooks): runtime registration, scoped install, per-call token budget

- HookRegistry mixin: add_hook / remove_hook / hooks_installed on Agent, Router
  and ONNXAgent. Thread-safe; a call reads a snapshot, so mutation never disturbs
  a call in flight.
- PredictContext.max_len / head_max_len: a start hook, or a per-call max_len= /
  head_max_len= argument, shapes the token budget for one call without touching
  shared agent config. Honored by Agent and ONNXAgent, propagated from Router.
- Tests: 86 hook checks, 101 API-stability checks; docs updated (API, lifecycle,
  patterns, examples).

---------

Co-authored-by: Nandakishor <[email protected]>
2026-09-23 23:24:56 +05:30

5.2 KiB

Tracing

Every hook of one call receives the same PredictContext.run_id, so a tracer can correlate the start, end and error events (and any spans it opens) without keeping its own bookkeeping.

run_id

  • A uuid4().hex string, created once per public call (predict_batch, system_one, Router.predict, ONNXAgent.system_one).
  • Shared by every hook of that call, including on_error and on_predict_end.
  • Not global and not persisted: it identifies a call within the process. Put it in your logs and outbound payloads to correlate across systems.
  • Distinct per call, so two calls never collide.
call A:  run_id=1a2b...  start ──┐
                                 ├─ end
                          error ─┘
call B:  run_id=9f8e...  start ─── end

A minimal tracer

Keep a map from run_id to the span you opened at start, and close it at end or error.

import time

class Tracer:
    def __init__(self):
        self.spans = {}

    def on_predict_start(self, ctx):
        self.spans[ctx.run_id] = {
            "model": ctx.model,
            "started_at": ctx.started_at,
        }

    def on_predict_end(self, ctx):
        span = self.spans.pop(ctx.run_id, None)
        if span is None:
            return
        emit_span(
            name="laya.predict",
            run_id=ctx.run_id,
            model=ctx.model,
            duration_ms=ctx.elapsed_ms,
            usage=ctx.usage,
            ok=ctx.error is None,
        )

    def on_error(self, ctx):
        # on_error runs before on_predict_end; leaving the span for on_predict_end is fine,
        # or close it here if you prefer.
        pass

agent = laya.load("convaiinnovations/laya", hooks=[Tracer()])

Because on_predict_end always runs, it is the natural place to close a span, and it can see ctx.error on the failure path.

Span lifetime

on_predict_start ──► open span (run_id, model, started_at)
      │
      ├─ inference
      │
on_error ──► record ctx.error on the span
      │
on_predict_end ──► close span (elapsed_ms, usage, ok)

Error spans

on_predict_end runs on the failure path with ctx.error set, so one close site handles both:

def on_predict_end(self, ctx):
    span = self.spans.pop(ctx.run_id, None)
    if span is None:
        return
    if ctx.error is not None:
        span["status"] = "error"
        span["error_type"] = type(ctx.error).__name__
        span["error_message"] = str(ctx.error)
    span["duration_ms"] = ctx.elapsed_ms
    emit(span)

If you only implement on_error, remember it fires before on_predict_end; do not close the span in both or you will double-count.

OpenTelemetry

The example examples/hooks/otel.py records counters and a histogram. For real spans, drive the OTel API from the tracer. Hooks are synchronous, so use the synchronous exporter (or enqueue and export from a worker).

from opentelemetry import trace

tracer = trace.get_tracer("laya")

class OTelHooks:
    def __init__(self):
        self.spans = {}

    def on_predict_start(self, ctx):
        span = tracer.start_span("laya.predict", attributes={"laya.run_id": ctx.run_id, "laya.model": ctx.model})
        self.spans[ctx.run_id] = span

    def on_predict_end(self, ctx):
        span = self.spans.pop(ctx.run_id, None)
        if span is None:
            return
        if ctx.usage:
            span.set_attribute("laya.input_tokens", ctx.usage["input_tokens"])
        if ctx.error is not None:
            span.record_exception(ctx.error)
            span.set_status(trace.Status(trace.StatusCode.ERROR))
        span.end()

laya.load("convaiinnovations/laya", hooks=[OTelHooks()], hooks_raise=False)

Set hooks_raise=False so a tracer outage never fails a request.

Distributed propagation

run_id is a plain string, so include it in whatever leaves the process: log lines, the payload sent to an audit service, or an HTTP header if a decision triggers a downstream call.

def audit(ctx):
    requests.post(
        "https://audit.example/decisions",
        json=record(ctx),
        headers={"X-Laya-Run-Id": ctx.run_id},
        timeout=2,
    )

Remember that a blocking call like this stalls the calling thread; enqueue it instead when serving concurrently. See patterns.

Nested calls

A hook that calls predict again starts a new call with a new run_id. The parent and child are independent unless you link them yourself. Capture the parent id and pass it along:

def enrich(ctx):
    child = enricher.predict(ctx.states[0], EXTRA_QUESTIONS)
    record_child_span(parent_run_id=ctx.run_id, child_run_id=child.get("run_id"))

Guard against recursion (see anti-patterns); the easiest guard is a separate enricher agent with no hooks.

See also