Files
zeekay 1224ff8282 ModernBert reads past its trained positions by YaRN, and only there
`Config::yarn` (s) gives a sequence longer than max_position_embeddings
rotary tables over s times as many positions: its global layers' are
YaRN's (Peng et al. 2023, NTK-by-parts with beta 32 and 1, attention
factor 0.1 ln s + 1, angles in f64), its local layers' are their own
frequencies extended, since a 128-token window spans no more than it was
trained on.  A sequence within max_position_embeddings reads the trained
tables through the same calls as before, whatever it is batched beside:
the packed pass runs each run of short or long sequences in its own
attention call, and the padded and masked passes place a row past the
trained positions in the trained tables followed by the stretched ones.

Tested: with yarn, every pass of a sequence within the trained positions
is the pass without it, bit for bit; a short sequence beside a long one
is the short one alone, bit for bit, in the packed pass; the long one's
packed pass is its padded pass to 1e-5 and its masked pass bit for bit;
past the stretched positions, and past the trained ones without yarn, a
sequence is refused.

hanzo-transformers 0.11.14.

Claude-Session: https://claude.ai/code/session_01HdGTLzC2y2CATAiF1pD1KE
2026-10-02 17:30:36 -07:00
..
2026-01-16 13:01:08 -08:00