mirror of
https://github.com/hanzoai/ml.git
synced 2026-11-20 23:08:29 +00:00
`Config::yarn` (s) gives a sequence longer than max_position_embeddings rotary tables over s times as many positions: its global layers' are YaRN's (Peng et al. 2023, NTK-by-parts with beta 32 and 1, attention factor 0.1 ln s + 1, angles in f64), its local layers' are their own frequencies extended, since a 128-token window spans no more than it was trained on. A sequence within max_position_embeddings reads the trained tables through the same calls as before, whatever it is batched beside: the packed pass runs each run of short or long sequences in its own attention call, and the padded and masked passes place a row past the trained positions in the trained tables followed by the stretched ones. Tested: with yarn, every pass of a sequence within the trained positions is the pass without it, bit for bit; a short sequence beside a long one is the short one alone, bit for bit, in the packed pass; the long one's packed pass is its padded pass to 1e-5 and its masked pass bit for bit; past the stretched positions, and past the trained ones without yarn, a sequence is refused. hanzo-transformers 0.11.14. Claude-Session: https://claude.ai/code/session_01HdGTLzC2y2CATAiF1pD1KE