mirror of
https://github.com/hanzoai/laya.git
synced 2026-10-30 19:33:31 +00:00
* feat: self-hostable server + Nix flake (Jev-compatible /v1/systemone) Laya's predict() output is already schema-compatible with TypeSafe Jev's decision API, so this adds only the HTTP surface a Jev client needs to talk to a self-hosted Laya: - laya/serve.py: FastAPI app exposing POST /v1/systemone (+ /health), calling Router.predict, with optional LAYA_API_KEY bearer auth. Config via env (LAYA_HOST/PORT/DEVICE/PRELOAD/MODELS/AUTO_TASK). Heavy imports are deferred so `import laya.serve` stays GPU- and web-dep-free. - laya-serve console script + laya[serve] extra (fastapi, uvicorn). - flake.nix: overlay building laya from torch-bin (prebuilt CUDA), a laya-serve runner, a dev shell, and nixosModules.default. - nix/laya-serve.nix: hardened DynamicUser systemd service (services.laya-serve) with CUDA device access and a HF weight cache. - tests/test_serve.py: shim tests (injected router, real FastAPI TestClient). Verified end-to-end on an RTX 3090: all three checkpoints preload on CUDA, English/Hindi requests route correctly and return Jev-shaped payloads, 6/6 tests pass. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_019NnoUnPhBMoS4S7eYWHYfe * fix(nix): build the service package from the host pkgs, not an overlay Hosts that inject `pkgs` via specialArgs (read-only nixpkgs) ignore module-level `nixpkgs.overlays`, so `pkgs.laya-serve` was missing and the NixOS toplevel failed to evaluate. Factor the package into nix/package.nix and have services.laya-serve.package default to `(pkgs.callPackage ./package.nix {}).laya-serve`, built from the host's own pkgs. The flake overlay/packages now reuse the same file. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_019NnoUnPhBMoS4S7eYWHYfe * feat(serve): LAYA_THREADS knob + services.laya-serve.threads for CPU tuning Cap torch intra-op threads for CPU inference. serve.py reads LAYA_THREADS and calls torch.set_num_threads (torch imported only when a limit is set); the NixOS module exposes `threads` and wires it to LAYA_THREADS + OMP_NUM_THREADS. Motivated by a thread sweep on a 32-core/64-thread EPYC: oversubscribing the logical core count is a ~4x latency regression, and single-request latency is often best below the physical core count (e.g. 8-16), while batched throughput peaks at the physical count. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_019NnoUnPhBMoS4S7eYWHYfe * chore(nix): commit flake.lock pinning nixpkgs to ccad53c Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_019NnoUnPhBMoS4S7eYWHYfe --------- Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
34 lines
1.2 KiB
Nix
34 lines
1.2 KiB
Nix
# The laya python package + a laya-serve runner, built from whatever `pkgs` is
|
|
# in scope. Factored out so both the flake's packages/overlay AND the NixOS
|
|
# module can build it the same way -- the module cannot rely on an overlay,
|
|
# because hosts that inject `pkgs` via specialArgs ignore module-level
|
|
# `nixpkgs.overlays`.
|
|
{ lib, python3, python3Packages, writeShellScriptBin }:
|
|
let
|
|
laya = python3Packages.buildPythonPackage {
|
|
pname = "laya";
|
|
version = "0.3.4";
|
|
src = ../.;
|
|
format = "setuptools";
|
|
propagatedBuildInputs = with python3Packages; [
|
|
torch-bin # prebuilt CUDA wheel -- no source build
|
|
transformers
|
|
safetensors
|
|
huggingface-hub
|
|
numpy
|
|
];
|
|
# Every test loads a checkpoint from the Hub -> needs network + a GPU.
|
|
doCheck = false;
|
|
# serve.py defers its fastapi/uvicorn imports, so this stays honest without
|
|
# dragging the web stack into the base library.
|
|
pythonImportsCheck = [ "laya" "laya.serve" ];
|
|
};
|
|
pyEnv = python3.withPackages (ps: [ laya ps.fastapi ps.uvicorn ]);
|
|
in
|
|
{
|
|
inherit laya;
|
|
laya-serve = writeShellScriptBin "laya-serve" ''
|
|
exec ${pyEnv}/bin/python -m laya.serve "$@"
|
|
'';
|
|
}
|