papers/kai/data/diagnosis/rag2f.log

corpus: 100000 train rows, 2345 val rows, suites ["rag_relevance"] (2s)
init: 187 tensors carried, fresh: ["ordinal.location.center", "retrieval.key.center", "retrieval.query.center"]
plan: stage rag2, 1 epochs, 771 batches, 23 warmup, 154 cooldown; rows on a sample of their options, by epoch: [0]
cluster: coordinating on 127.0.0.1:59138
cluster: ra (127.0.0.1:59139) joins round 0 as worker 0 (Metal { gpu_id: 4294970014 }, F32)
worker 0 (ra): joined round 0
round 0: 31/771 lr×1.000 eta 13m | ra 31 steps 4039 tok/s nll 0.6957 ret 0.0000 | ret 0.0000 | 31.3s, closed 1.3s past the deadline, outer step 0.1s
worker 0 round 0: 31 steps 4039 tok/s (4065 padded; contexts 4019 of 4043) nll 0.6957 ret 0.0000 gpu 97% free 64% | delta sent 0.2s, waited 0.1s, update 0.1s
round 1: 60/771 lr×1.000 eta 12m | ra 29 steps 3852 tok/s nll 0.6991 ret 0.0000 | ret 0.0000 | 30.7s, closed 0.7s past the deadline, outer step 0.2s
worker 0 round 1: 29 steps 3852 tok/s (3887 padded; contexts 3833 of 3866) nll 0.6991 ret 0.0000 gpu 97% free 64% | delta sent 0.3s, waited 0.2s, update 0.1s
round 2: 91/771 lr×1.000 eta 12m | ra 31 steps 4029 tok/s nll 0.6978 ret 0.0000 | ret 0.0000 | 30.7s, closed 0.7s past the deadline, outer step 0.1s
worker 0 round 2: 31 steps 4029 tok/s (4057 padded; contexts 4009 of 4035) nll 0.6978 ret 0.0000 gpu 98% free 63% | delta sent 0.2s, waited 0.1s, update 0.1s
round 3: 120/771 lr×1.000 eta 11m | ra 29 steps 3741 tok/s nll 0.6939 ret 0.0000 | ret 0.0000 | 30.6s, closed 0.6s past the deadline, outer step 0.1s
worker 0 round 3: 29 steps 3741 tok/s (3773 padded; contexts 3721 of 3752) nll 0.6939 ret 0.0000 gpu 97% free 63% | delta sent 0.3s, waited 0.1s, update 0.1s
round 4: 150/771 lr×1.000 eta 11m | ra 30 steps 3915 tok/s nll 0.6946 ret 0.0000 | ret 0.0000 | 31.2s, closed 1.2s past the deadline, outer step 0.0s
worker 0 round 4: 30 steps 3915 tok/s (3932 padded; contexts 3896 of 3910) nll 0.6946 ret 0.0000 gpu 98% free 63% | delta sent 0.2s, waited 0.0s, update 0.1s, state kept
val @150: rag_relevance 0.522/0.694
saved runs/rag2f at 150 batches, 0 epochs
round 5: 175/771 lr×1.000 eta 11m | ra 25 steps 3337 tok/s nll 0.6933 ret 0.0000 | ret 0.0000 | 31.2s, closed 1.2s past the deadline, outer step 0.1s
worker 0 round 5: 25 steps 3337 tok/s (3358 padded; contexts 3320 of 3339) nll 0.6933 ret 0.0000 gpu 98% free 61% | delta sent 0.2s, waited 0.1s, update 0.1s
round 6: 206/771 lr×1.000 eta 10m | ra 31 steps 3908 tok/s nll 0.6922 ret 0.0000 | ret 0.0000 | 30.7s, closed 0.7s past the deadline, outer step 0.0s
worker 0 round 6: 31 steps 3908 tok/s (3930 padded; contexts 3887 of 3908) nll 0.6922 ret 0.0000 gpu 98% free 62% | delta sent 0.2s, waited 0.0s, update 0.1s