* feat(hooks): opt-in prediction hooks to observe or shape every decision Add on_predict_start/on_predict_end and a Hook protocol to Agent, Router and ONNXAgent, plus on_route/on_load/on_evict on the Router. Hooks receive a mutable PredictContext: a start hook can redact or rewrite the state/questions, or call ctx.skip(...) to serve a cached result and skip the forward pass; an end hook can rewrite the results. Unset hooks are a no-op, so the default path is unchanged. Adds tests/test_hooks.py (50 checks, no weights), examples/hooks/ and docs/hooks.md. * fix(hooks): run start/error/end on start-hook failure; expose model id; per-call on_route Review pass: - A failing on_predict_start hook now still runs on_error and on_predict_end, instead of aborting before the try/finally. - Agent and ONNXAgent populate ctx.model from the checkpoint id, matching the documented context. - Router per-call hooks= now apply to on_route too (and route() takes hooks and hooks_raise), so a per-call hook covers the whole call. - normalise_hooks rejects a class instead of an instance with a clear error. Adds 9 checks to tests/test_hooks.py (59 total). * test(hooks): regression + API-stability tests; fix router cache-hit routing - tests/test_hooks_api.py pins the hook surface (parameter names/defaults, PredictContext fields, lifecycle events, exports, class defaults) so an accidental API change fails CI. - Regression checks for the review-pass fixes, plus hooks_concurrent storage. - Router.predict now adds routing on a cache-hit skip, keeping its documented return contract. - PredictContext uses identity equality/hash so a hook can store it in a set. - normalise_hooks rejects non-callable lifecycle attributes. * feat(hooks): add run_id for tracing; document real-world coverage - PredictContext.run_id is a unique id shared by every hook of one call, so a tracer can correlate start/end/error spans without its own bookkeeping. - docs/hooks.md: prior-art mapping (browser-use on_step_*, OpenAI Agents SDK RunHooks/AgentHooks), a tracing example, and a note that hooks are synchronous and must not block (laya.serve runs inference on a single worker). - tests: run_id regression + API field guard; prod scenario correlating spans. * docs(hooks): expand hooks.md into a docs/hooks/ folder Replace the single hooks.md with a seven-page reference: overview, full API, lifecycle flowcharts, error handling, patterns and anti-patterns, examples and tracing. Every public hook API is covered, links and anchors are checked by test_packaging.py, and the README/examples links point at docs/hooks/index.md. * feat(hooks): runtime registration, scoped install, per-call token budget - HookRegistry mixin: add_hook / remove_hook / hooks_installed on Agent, Router and ONNXAgent. Thread-safe; a call reads a snapshot, so mutation never disturbs a call in flight. - PredictContext.max_len / head_max_len: a start hook, or a per-call max_len= / head_max_len= argument, shapes the token budget for one call without touching shared agent config. Honored by Agent and ONNXAgent, propagated from Router. - Tests: 86 hook checks, 101 API-stability checks; docs updated (API, lifecycle, patterns, examples). --------- Co-authored-by: Nandakishor <[email protected]>
5.7 KiB
Error handling
Hooks run inside the request they observe, so how their failures are handled matters. This page is the exact policy.
- The two policies
- What runs when something fails
- Exception chaining
- BaseException
- Configuration errors
- Warnings
- Choosing a policy
The two policies
hooks_raise controls what happens when a hook raises.
hooks_raise |
behaviour |
|---|---|
True (default) |
the hook exception propagates out of the call. |
False |
the hook is skipped with a RuntimeWarning and the call continues. |
It is set per instance and can be overridden per call (hooks_raise= on predict_batch,
system_one, Router.route, Router.predict). Per-call None means "use the instance value".
# strict: a broken audit hook fails the request
laya.load("convaiinnovations/laya", on_predict_end=audit, hooks_raise=True)
# lenient: telemetry must never take down a served request
laya.load("convaiinnovations/laya", on_predict_end=metrics, hooks_raise=False)
dispatch catches Exception. Anything that is not an Exception (see
BaseException) is never swallowed, even with hooks_raise=False.
What runs when something fails
The predict lifecycle is wrapped in try / except / finally, so cleanup hooks run on failure.
try:
on_predict_start
inference
except BaseException as exc:
ctx.error = exc
on_error (best effort; cannot mask exc)
raise
finally:
elapsed_ms, usage
on_predict_end (best effort; cannot mask exc on the failure path)
Failure matrix, per event and runtime:
| event | runtime | if it raises |
|---|---|---|
on_predict_start |
Agent / Router | hooks_raise=True: on_error and on_predict_end still run, then the exception propagates. False: warn and continue (mutations made before the raise remain). |
| inference | Agent / Router | ctx.error set, on_error runs, on_predict_end runs, exception propagates. |
on_error |
Agent / Router | never masks the original exception; chained as __context__. |
on_predict_end (success path) |
Agent / Router | hooks_raise=True: propagates (the result is computed but the call fails). False: warn. |
on_predict_end (failure path) |
Agent / Router | never masks the original exception; chained as __context__. |
on_route |
Router | propagates directly; there is no predict context yet. |
on_load |
Router | propagates directly; the checkpoint stays built and resident. |
on_evict |
Router | propagates directly; the checkpoint is already freed. |
Consequences worth knowing:
- A failing
on_loadleaves the model cached, so the nextloadreturns it without firingon_loadagain. - A failing
on_predict_endon the success path means the caller gets an exception instead of a result, even though inference succeeded. Usehooks_raise=Falsefor end hooks that are pure side effects.
Exception chaining
When a hook fails while another exception is already propagating, the original exception is
re-raised and the hook's exception is attached as __context__. The root cause is never lost.
class BadTelemetry:
def on_error(self, ctx):
raise RuntimeError("telemetry down")
try:
agent.system_one(state, questions, hooks=[BadTelemetry()])
except RuntimeError as exc:
assert exc.__context__ is not None # the telemetry failure
The same rule applies to a failing on_predict_end on the failure path.
BaseException
dispatch catches Exception, not BaseException, so KeyboardInterrupt and SystemExit
always propagate. They still trigger the except BaseException branch of the predict lifecycle,
which means on_error and on_predict_end run before the process unwinds. Keep those hooks
fast and non-blocking if you care about interrupt latency.
Configuration errors
Bad configuration fails fast with TypeError, before any inference:
| case | raised at | example |
|---|---|---|
| class instead of instance | construction | hooks=[MyHook] |
| no lifecycle method | construction | hooks=[object()] |
| non-callable event | construction | on_predict_start = 5 |
| non-callable convenience hook | construction | on_predict_start=123 |
plain callable in hooks= |
construction | hooks=[lambda ctx: None] |
Per-call hooks are validated when the call is made, so a bad per-call hook raises TypeError
from predict/system_one rather than at construction.
Warnings
With hooks_raise=False, each failing hook emits one RuntimeWarning naming the hook type and
event:
laya: hook Metrics.on_predict_end failed: connection reset
The warning is emitted once per failure, not once per hook definition, so a flaky hook under load can be noisy. Aggregate or rate-limit inside the hook if that matters.
Choosing a policy
| hook kind | recommended | why |
|---|---|---|
| policy / guardrail / redaction | hooks_raise=True |
a policy that silently fails is a security hole. |
| audit / logging | hooks_raise=True in tests, often False in production |
losing an audit record should be loud, but not necessarily fatal. |
| metrics / tracing | hooks_raise=False |
observability must not fail the request. |
| cache read/write | hooks_raise=True |
a broken cache should surface, not silently miss. |
You can mix: install a guardrail with the instance default and give a telemetry hook its own
try/except, or use a separate hooks_raise per call.
See also
- Lifecycle: the
try/except/finallyshape in context. - Patterns and anti-patterns: common mistakes with error handling.