Files
engine/docs/ENGINE.md
Eric BuehlerandGitHub e9f5d3c943 feat(core): code execution, file outputs and /v1/files, strict tool calling, new docs and ui (#2130)
* Add code exec

* Fixes

* Fixes

* Fixes

* Fixes

* Tweak and fix

* Tweak and fix

* Tweak and fix

* Tweak and fix

* Nicer ux

* Nicer ux

* Nicer ux

* Docs

* Docs

* Docs

* nName

* ux in cli

* Server rework

* Fixes

* Fixes

* Fixes

* Fixes

* Fixes

* Fixes

* Fixes

* Fixes

* Fixes

* Session id

* Session id

* Fixes

* Fixes

* Fixes

* Add agentic session

* Add agentic session

* Clarify if supports vision

* Add video support

* Add ui support

* Salt cache

* Salt cache

* Download

* Finish reason

* Video rendering

* Tweak

* Tweak

* Order

* Agentic session serde

* Agentic session ui

* Cleanup

* Cleanup

* Docs

* Docs

* Docs

* Tweaks to lang

* Rpogress

* Cleaner

* Docs

* Env var

* Nit

* Nit

* Better python docs

* Nit

* Nit

* Nit

* Support python

* Improve?

* STart here page

* Docs

* Files: typed outputs from code execution

Code execution now produces first-class File objects (text, image, video,
binary) with stable ids and OpenAI-compatible Files endpoints. The model
declares outputs in the execute_python tool args; the runtime reads them,
applies inline/preview/fetch policy by size, and surfaces them to the app
without inflating the transcript.

The model gets read_file / list_files tools for working with large or
prior-turn files. Request-level `files: [...]` is a contract — required
files surface even if the model forgets to declare them.

This separates the model's context budget from the app's output channel,
which previously shared one bloated path through assistant text.

* Add support for files/typeed outputs

* Fix concurrency

* Fix

* Refactor

* Codequa;

* Codequal

* Fix

* Fixes

* Cleaner design for injection

* Cleaner design for injection

* Add copy and line count

* Add copy and line count

* Update ui

* Update ui

* Coppy button

* Coppy button

* Coppy button

* Allow forking sessions

* Allow forking sessions

* Prep website

* Prep website

* Fix

* Fix

* Fix

* Fix

* Fix

* Fix

* Fix

* Fix

* Fix

* Fix

* Fix

* Fix

* Fix

* Tweak

* Tweak

* Tweak

* Fix

* Improvements

* Improvements

* Improvements

* Fix downloading

* Readme

* Readme

* Format

* Fix

* CUDA pin to 13.0 in CI

* Fix docs
2026-05-15 21:34:33 -04:00

2.1 KiB

Engine Internals

This document describes internal engine behaviors in Hanzo Engine.

Overview

The Hanzo Engine engine manages model inference through a background thread pool. Each loaded model runs in its own engine thread, which handles request queuing, batching, and execution.

Warmup Run

When a text or multimodal model is loaded in a multi-threaded runtime, Hanzo Engine automatically performs a warmup ("dummy") run:

  • Sends a short completion request ("hello" with max 1 token) to initialize CUDA kernels and caches
  • Logs "Beginning dummy run." when starting and "Dummy run completed in Xs." when finished
  • Helps ensure more consistent performance for the first real user request
  • Only runs for text and multimodal models (not diffusion/speech)

This warmup ensures that CUDA kernel compilation and memory allocation happens during model loading rather than during the first user request.

Automatic Engine Recovery

If the inference engine thread dies unexpectedly (e.g., due to a panic), Hanzo Engine can automatically recover:

  • Detects dead engine threads when sending requests
  • Automatically reboots the engine using saved configuration
  • Logs "Engine {model_id} is dead, rebooting" followed by "Successfully rebooted engine {model_id}"
  • Preserves all original configuration including KV cache settings, prefix cache, and tool callbacks

This ensures high availability without manual intervention.

Thread Model

Each model loaded in Hanzo Engine runs in its own dedicated engine thread:

  1. Main Thread: Handles HTTP requests, CLI interaction, and dispatches work to engine threads
  2. Engine Threads: Each loaded model has a dedicated thread for inference
  3. Background Workers: Tokenization and other preprocessing can run in parallel

For multi-model setups, each model gets its own engine thread, allowing true parallel inference across different models.

See Also