zeekay 5d52064f4f export: quantized deployment path (llama.cpp embeddings + numpy heads) ...

Heads-only checkpoint -> heads.safetensors (1.2MB) + router_config.json.
Backbone runs quantized in llama.cpp embedding mode (--pooling last
--embd-normalize -1 = raw post-norm hidden state); heads applied in numpy.

Parity vs torch fp32/MPS (n=100 holdout): Q4_K_M route-argmax 95% / top-3
100% / cosine 0.969; Q8_0 97% / 100% / 0.988. Latency (Q4_K_M resident
llama-server, prompt->model id, n=200): mean 132ms p50 97ms p99 351ms.

Files: export/{export_heads,route_gguf,verify_parity}.py, heads.safetensors,
QUANTIZED.md, router_config.json; README Quantized deployment section.

2026-07-07 18:14:54 -07:00

..

export_heads.py

export: quantized deployment path (llama.cpp embeddings + numpy heads)

2026-07-07 18:14:54 -07:00

heads.safetensors

export: quantized deployment path (llama.cpp embeddings + numpy heads)

2026-07-07 18:14:54 -07:00

QUANTIZED.md

export: quantized deployment path (llama.cpp embeddings + numpy heads)

2026-07-07 18:14:54 -07:00

route_gguf.py

export: quantized deployment path (llama.cpp embeddings + numpy heads)

2026-07-07 18:14:54 -07:00

router_config.json

export: quantized deployment path (llama.cpp embeddings + numpy heads)

2026-07-07 18:14:54 -07:00

verify_parity.py

export: quantized deployment path (llama.cpp embeddings + numpy heads)

2026-07-07 18:14:54 -07:00