- BaseHook: a no-op base class for subclassing, exported at the top level next to the Hook protocol. - laya.hooks gains a process-wide registry: default_hooks, set_default_hooks, add_default_hook, clear_default_hooks, and compose_hooks. Defaults run before installed and per-call hooks, and cover the Router lifecycle events too. - Tests and docs updated (api, index, patterns, examples).
14 KiB
API reference
Everything on this page is importable from laya (the common names) or laya.hooks (the
whole surface).
from laya import PredictContext, PredictHook, Hook
from laya.hooks import HOOK_EVENTS, normalise_hooks, dispatch, aggregate_usage
- PredictContext
- Hook protocol
- Convenience types
- Configuration surface
- Event payloads
- Validation
- Advanced helpers
PredictContext
A PredictContext is created once per public call and passed to every hook of that call. It is
mutable: hooks may rewrite states, questions and results, and on_route may rewrite
decision. It uses identity equality (eq=False), so a context is hashable and two contexts are
never equal.
@dataclass(eq=False)
class PredictContext:
states: List[Any]
questions: Dict[str, Any]
run_id: str = <uuid4 hex>
results: Optional[List[Dict[str, Any]]] = None
decision: Optional[Dict[str, Any]] = None
model: Optional[str] = None
agent: Any = None
router: Any = None
max_len: Optional[int] = None
head_max_len: Optional[int] = None
usage: Optional[Dict[str, int]] = None
started_at: float = <perf_counter()>
elapsed_ms: Optional[float] = None
error: Optional[BaseException] = None
| field | type | set when | mutable | meaning |
|---|---|---|---|---|
states |
list |
always | yes (start) | the states for this call. system_one/Router.predict pass one; predict_batch passes many. A start hook may replace the list. |
questions |
dict |
always | yes (start) | the questions. A start hook may replace the dict. |
run_id |
str |
always | no | a unique id shared by every hook of this call. Use it to correlate events and spans. |
results |
list | None |
end (and on a skip) | yes (end) | per-state result dicts, each shaped like system_one's return. None until inference finishes. |
decision |
dict | None |
Router only | yes (route) | the RouteDecision (a dict) that selected the checkpoint. |
model |
str | None |
always | no | the checkpoint id: Agent.model_id for an Agent, the resolved alias (for example "english") for a Router. |
agent |
Agent | ONNXAgent | None |
predict events | no | the runtime answering the call. |
router |
Router | None |
Router events | no | the Router, when one is in play. |
max_len |
int | None |
always | yes (start) | per-call token budget for the encoder. None uses the agent config. |
head_max_len |
int | None |
always | yes (start) | per-call token budget for the question head. None uses the agent config. |
usage |
dict | None |
end | yes (end) | {"input_tokens", "output_tokens"}, summed over the states of the call. |
started_at |
float |
always | no | time.perf_counter() when the call began. |
elapsed_ms |
float | None |
end | no | wall time for the whole call, milliseconds. |
error |
BaseException | None |
failure path | no | the exception, set before on_error and on_predict_end. |
PredictContext.skip(results)
Short-circuits inference. Called from on_predict_start, it sets ctx.results so the forward
pass is skipped; on_predict_end still runs and the supplied results are returned.
def cache_read(ctx):
hit = CACHE.get(key(ctx.states[0], ctx.questions))
if hit is not None:
ctx.skip([hit]) # list of per-state results, same shape as predict_batch's return
On the Router, a skipped payload gets a routing key added (without overwriting one it
already has), so Router.predict keeps its documented return shape.
Hook protocol
Hook is a typing.Protocol. Implement any subset of the methods; the rest are skipped.
class Hook(Protocol):
def on_predict_start(self, ctx: PredictContext) -> None: ...
def on_predict_end(self, ctx: PredictContext) -> None: ...
def on_route(self, ctx: PredictContext) -> None: ...
def on_load(self, ctx: PredictContext) -> None: ...
def on_evict(self, ctx: PredictContext) -> None: ...
def on_error(self, ctx: PredictContext) -> None: ...
| event | where | runs | can change |
|---|---|---|---|
on_predict_start |
Agent, Router | before tokenization/forward | states, questions, or skip() |
on_predict_end |
Agent, Router | after results exist, success or failure | results |
on_route |
Router | after detection, before loading | decision |
on_load |
Router | after a checkpoint is built | nothing (observe) |
on_evict |
Router | after a checkpoint is freed | nothing (observe) |
on_error |
Agent, Router | when a predict call fails | nothing (observe) |
A hook is free to define extra attributes and methods; only the six event names are consulted. If a hook defines one of the six as a non-callable, configuration fails fast (see Validation).
BaseHook
BaseHook is the concrete counterpart to the protocol: a class with a no-op body for every event.
Subclass it and override only the events you need.
from laya import BaseHook
class Audit(BaseHook):
def on_predict_end(self, ctx):
ship(ctx.run_id, ctx.results)
Hook is best when you want structural typing (any object with the right methods); BaseHook is
best when you want an explicit base to subclass and call super() on.
Convenience types
PredictHook = Callable[[PredictContext], None]
PredictHook is the type of a plain callable used with on_predict_start= / on_predict_end=.
Pass a single callable or a sequence of them; each is wrapped into a minimal hook.
Runtime registration
Every runtime mixes in HookRegistry, so hooks can be added, removed or scoped after
construction. Mutation is thread-safe; a call reads a snapshot of the list, so adding or
removing a hook never disturbs a call in flight.
agent.add_hook(tracer) # one hook or a sequence; returns self for chaining
agent.remove_hook(tracer) # by identity; True if it was installed
with agent.hooks_installed(debug): # installed for the block, removed on exit
agent.system_one(state, questions)
add_hook accepts the same objects as hooks= (not plain callables). hooks_installed takes
any number of hook objects or sequences and restores the previous list on exit, including when
the block raises.
Process-wide defaults
laya.hooks keeps a small process-wide registry, so a tracer, metrics hook or tenant tagger does
not have to be threaded through every Agent and Router. Defaults run first, then the
hooks installed on the instance, then per-call hooks.
from laya import hooks
hooks.set_default_hooks(hooks=[Tracer()]) # replaces the set, accepts the hooks= arguments
hooks.add_default_hook(Metrics()) # appends
hooks.clear_default_hooks() # removes everything
hooks.default_hooks() # a copy of the current list
hooks.compose_hooks(agent.hooks) # defaults + installed (advanced)
Defaults apply to every event, including the Router lifecycle events on_load and on_evict.
The registry is read at call time, so hooks set after an Agent or Router is built still apply.
There is no per-instance opt-out; call clear_default_hooks() to turn the process-wide set off.
Configuration surface
Every entry point accepts the same five hook parameters. hooks takes an object or a sequence
of objects; on_predict_start / on_predict_end take a callable or a sequence.
| parameter | type | default | meaning |
|---|---|---|---|
hooks |
Hook | Sequence[Hook] | None |
None |
lifecycle hooks (any of the six events). |
on_predict_start |
PredictHook | Sequence[PredictHook] | None |
None |
convenience callables for one event. |
on_predict_end |
PredictHook | Sequence[PredictHook] | None |
None |
convenience callables for one event. |
hooks_raise |
bool |
True |
True: a hook exception propagates. False: warn and continue. |
hooks_concurrent |
bool |
True |
False: dispatch hooks under a lock, one at a time. |
Agent
Agent(
model_id_or_path="convaiinnovations/laya",
device=None, token=None, subfolder=None, fast=False, compile=False,
hooks=None, on_predict_start=None, on_predict_end=None,
hooks_raise=True, hooks_concurrent=True,
)
load(..., hooks=None, on_predict_start=None, on_predict_end=None,
hooks_raise=True, hooks_concurrent=True)
agent.predict_batch(states, questions, batch_size=None,
hooks=None, on_predict_start=None, on_predict_end=None, hooks_raise=None,
max_len=None, head_max_len=None, sort_by_length=False)
agent.system_one(state, questions,
hooks=None, on_predict_start=None, on_predict_end=None, hooks_raise=None,
max_len=None, head_max_len=None)
agent.predict(...) # alias of system_one
hooks_raiseon a per-call method defaults toNone, meaning "use the instance value".hooks_concurrentis instance-level only.
Router
Router(
models=None, device=None, token=None, max_loaded=2, default="english",
auto_task_detection=False, standalone_repos=False, preload=False, lang_guess=None,
hooks=None, on_predict_start=None, on_predict_end=None,
hooks_raise=True, hooks_concurrent=True,
)
router.route(state, questions=None, model=None, task=None, lang=None, lang_guess=None,
hooks=None, hooks_raise=None)
router.predict(state, questions, model=None, task=None, lang=None, lang_guess=None,
hooks=None, on_predict_start=None, on_predict_end=None, hooks_raise=None,
max_len=None, head_max_len=None)
router.system_one(...) # alias of predict
router.load(name) # builds on first use; fires on_load
router.preload(names=None) # builds several; fires on_load per build
router.unload(name=None) # frees one or all; fires on_evict
router.attach(name, agent) # registers an existing agent; does not fire on_load
router.loaded # list of resident checkpoint names
- Per-call
hooks=onrouteandpredictapply to the whole call, includingon_route. route()is public: calling it dispatcheson_routewith the installed hooks plus any per-callhooks.
ONNXAgent
ONNXAgent(model_id_or_path, onnx_path="laya.onnx", subfolder=None,
hooks=None, on_predict_start=None, on_predict_end=None,
hooks_raise=True, hooks_concurrent=True)
onnx_agent.system_one(state, questions,
hooks=None, on_predict_start=None, on_predict_end=None, hooks_raise=None,
max_len=None, head_max_len=None)
onnx_agent.predict(...) # alias of system_one
ONNXAgent has no Router, so it exposes only the predict-level events.
Event payloads
Which fields are populated, per event and runtime:
| event | runtime | states |
questions |
decision |
model |
agent |
router |
results |
usage |
elapsed_ms |
error |
|---|---|---|---|---|---|---|---|---|---|---|---|
on_predict_start |
Agent | ✓ | ✓ | – | ✓ | ✓ | – | – | – | – | – |
on_predict_start |
Router | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | – | – | – | – |
on_predict_end |
Agent | ✓ | ✓ | – | ✓ | ✓ | – | ✓ | on success | ✓ | on failure |
on_predict_end |
Router | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | on success | ✓ | on failure |
on_error |
both | ✓ | ✓ | ✓ (Router) | ✓ | ✓ | ✓ (Router) | – | – | – | ✓ |
on_route |
Router | ✓ | ✓ | ✓ | – | – | ✓ | – | – | – | – |
on_load |
Router | [] |
{} |
– | ✓ | ✓ | ✓ | – | – | – | – |
on_evict |
Router | [] |
{} |
– | ✓ | – | ✓ | – | – | – | – |
Timing details:
on_predict_endseesresultson the success path. On the failure pathresultsisNoneunless a start hook had set them viaskip(), andusageis thereforeNonetoo (it is derived fromresults);elapsed_msis always set.on_errorruns before thefinallyblock that computeselapsed_msandusage, so both areNonethere. Read timing and usage fromon_predict_endinstead.run_idis always populated.
Validation
Configuration is validated when hooks are normalised, which happens at construction for
installed hooks and at call time for per-call hooks. These raise TypeError:
| case | message |
|---|---|
| a class is passed instead of an instance | hooks entries must be instances, not classes; ... |
| an object implements none of the six events | hooks entries must implement at least one of ... |
| an event attribute is not callable | hooks entry X.on_predict_start must be callable, got int |
on_predict_start= / on_predict_end= is not callable |
on_predict_start must be callable, got int |
hooks= does not accept plain callables, because a bare callable does not say which event it
is for. Use on_predict_start= / on_predict_end= for those.
Advanced helpers
These are used internally and are stable, but most users do not need them.
HOOK_EVENTS # tuple of the six event names, in dispatch order
normalise_hooks(hooks=None, on_predict_start=None, on_predict_end=None) -> list
dispatch(hooks, event, ctx, *, raise_errors=True, lock=None) -> None
aggregate_usage(results) -> {"input_tokens": int, "output_tokens": int}
normalise_hooks flattens a hooks object/sequence and the two callables into one ordered list.
dispatch calls event on every hook that implements it, applying the raise policy and lock.
aggregate_usage sums per-state usage blocks.
from laya.hooks import normalise_hooks, dispatch, PredictContext
hooks = normalise_hooks(on_predict_start=[log, redact])
ctx = PredictContext(states=["..."], questions={...})
dispatch(hooks, "on_predict_start", ctx)
See also
- Lifecycle: when each event runs, with flowcharts.
- Errors: the failure matrix and chaining rules.
- Patterns and anti-patterns: how to structure hooks well.
- Examples: recipes for every use case.