papers/kai/data/diagnosis/rag2.log

corpus: 100000 train rows, 2345 val rows, suites ["rag_relevance"] (2s)
init: 187 tensors carried, fresh: ["ordinal.location.center", "retrieval.key.center", "retrieval.query.center"]
plan: stage rag2, 1 epochs, 771 batches, 23 warmup, 154 cooldown; rows on a sample of their options, by epoch: [0]
cluster: coordinating on 127.0.0.1:58705
cluster: ra (127.0.0.1:58706) joins round 0 as worker 0 (Metal { gpu_id: 4294970014 }, BF16)
worker 0 (ra): joined round 0
round 0: 35/771 lr×1.000 eta 11m | ra 35 steps 4581 tok/s nll 0.6938 ret 0.0000 | ret 0.0000 | 31.4s, closed 1.4s past the deadline, outer step 0.2s
worker 0 round 0: 35 steps 4581 tok/s (4624 padded; contexts 4558 of 4599) nll 0.6938 ret 0.0000 gpu 94% free 63% | delta sent 0.5s, waited 0.2s, update 0.1s
round 1: 69/771 lr×1.000 eta 11m | ra 34 steps 4552 tok/s nll 0.7043 ret 0.0000 | ret 0.0000 | 30.6s, closed 0.6s past the deadline, outer step 0.1s
worker 0 round 1: 34 steps 4552 tok/s (4577 padded; contexts 4529 of 4552) nll 0.7043 ret 0.0000 gpu 97% free 62% | delta sent 0.2s, waited 0.1s, update 0.1s
round 2: 105/771 lr×1.000 eta 10m | ra 36 steps 4716 tok/s nll 0.6946 ret 0.0000 | ret 0.0000 | 30.5s, closed 0.5s past the deadline, outer step 0.0s
worker 0 round 2: 36 steps 4716 tok/s (4752 padded; contexts 4692 of 4725) nll 0.6946 ret 0.0000 gpu 97% free 63% | delta sent 0.2s, waited 0.0s, update 0.1s
round 3: 142/771 lr×1.000 eta 9m | ra 37 steps 4720 tok/s nll 0.6934 ret 0.0000 | ret 0.0000 | 31.1s, closed 1.1s past the deadline, outer step 0.0s
worker 0 round 3: 37 steps 4720 tok/s (4750 padded; contexts 4696 of 4724) nll 0.6934 ret 0.0000 gpu 98% free 63% | delta sent 0.2s, waited 0.0s, update 0.1s
round 4: 175/771 lr×1.000 eta 9m | ra 33 steps 4370 tok/s nll 0.6933 ret 0.0000 | ret 0.0000 | 30.6s, closed 0.6s past the deadline, outer step 0.0s
worker 0 round 4: 33 steps 4370 tok/s (4393 padded; contexts 4348 of 4369) nll 0.6933 ret 0.0000 gpu 98% free 63% | delta sent 0.2s, waited 0.0s, update 0.1s, state kept
val @175: rag_relevance 0.497/0.693
saved runs/rag2 at 175 batches, 0 epochs
round 5: 205/771 lr×1.000 eta 9m | ra 30 steps 3781 tok/s nll 0.6918 ret 0.0000 | ret 0.0000 | 31.4s, closed 1.4s past the deadline, outer step 0.0s
worker 0 round 5: 30 steps 3781 tok/s (3802 padded; contexts 3761 of 3781) nll 0.6918 ret 0.0000 gpu 98% free 62% | delta sent 0.2s, waited 0.0s, update 0.1s
round 6: 236/771 lr×1.000 eta 8m | ra 31 steps 4062 tok/s nll 0.6939 ret 0.0000 | ret 0.0000 | 31.1s, closed 1.1s past the deadline, outer step 0.0s
worker 0 round 6: 31 steps 4062 tok/s (4082 padded; contexts 4041 of 4060) nll 0.6939 ret 0.0000 gpu 98% free 59% | delta sent 0.2s, waited 0.0s, update 0.1s
round 7: 268/771 lr×1.000 eta 8m | ra 32 steps 4207 tok/s nll 0.6935 ret 0.0000 | ret 0.0000 | 30.5s, closed 0.5s past the deadline, outer step 0.0s
worker 0 round 7: 32 steps 4207 tok/s (4253 padded; contexts 4186 of 4229) nll 0.6935 ret 0.0000 gpu 98% free 56% | delta sent 0.2s, waited 0.0s, update 0.1s
round 8: 302/771 lr×1.000 eta 7m | ra 34 steps 4349 tok/s nll 0.6927 ret 0.0000 | ret 0.0000 | 31.1s, closed 1.1s past the deadline, outer step 0.0s
worker 0 round 8: 34 steps 4349 tok/s (4387 padded; contexts 4327 of 4362) nll 0.6927 ret 0.0000 gpu 98% free 57% | delta sent 0.2s, waited 0.0s, update 0.1s, state kept
val @302: rag_relevance 0.495/0.693
saved runs/rag2 at 302 batches, 0 epochs
round 9: 332/771 lr×1.000 eta 7m | ra 30 steps 3926 tok/s nll 0.6933 ret 0.0000 | ret 0.0000 | 32.2s, closed 2.2s past the deadline, outer step 0.2s
worker 0 round 9: 30 steps 3926 tok/s (3942 padded; contexts 3907 of 3920) nll 0.6933 ret 0.0000 gpu 96% free 60% | delta sent 0.3s, waited 0.2s, update 0.1s