papers/kai/data/diagnosis/h2.log

corpus: 100000 train rows, 2345 val rows, suites ["rag_relevance"] (2s)
init: 190 tensors carried, fresh: []
plan: stage h2, 1 epochs, 771 batches, 23 warmup, 154 cooldown; rows on a sample of their options, by epoch: [0]
cluster: coordinating on 127.0.0.1:60230
cluster: ra (127.0.0.1:60231) joins round 0 as worker 0 (Metal { gpu_id: 4294970014 }, BF16)
worker 0 (ra): joined round 0
round 0: 37/300 lr×1.000 eta 4m | ra 37 steps 4811 tok/s nll 0.6362 ret 0.0000 | ret 0.0000 | 31.4s, closed 1.4s past the deadline, outer step 0.1s
worker 0 round 0: 37 steps 4811 tok/s (4855 padded; contexts 4787 of 4828) nll 0.6362 ret 0.0000 gpu 97% free 68% | delta sent 0.3s, waited 0.1s, update 0.1s
round 1: 70/300 lr×1.000 eta 3m | ra 33 steps 4323 tok/s nll 0.5858 ret 0.0000 | ret 0.0000 | 31.2s, closed 1.2s past the deadline, outer step 0.1s
worker 0 round 1: 33 steps 4323 tok/s (4349 padded; contexts 4302 of 4325) nll 0.5858 ret 0.0000 gpu 97% free 68% | delta sent 0.2s, waited 0.1s, update 0.1s
round 2: 104/300 lr×1.000 eta 3m | ra 34 steps 4365 tok/s nll 0.5487 ret 0.0000 | ret 0.0000 | 31.1s, closed 1.1s past the deadline, outer step 0.1s
worker 0 round 2: 34 steps 4365 tok/s (4397 padded; contexts 4343 of 4373) nll 0.5487 ret 0.0000 gpu 98% free 68% | delta sent 0.2s, waited 0.1s, update 0.1s, state kept
val @104: rag_relevance 0.662/0.677
saved runs/h2run at 104 batches, 0 epochs
round 3: 134/300 lr×1.000 eta 3m | ra 30 steps 3785 tok/s nll 0.5366 ret 0.0000 | ret 0.0000 | 32.5s, closed 2.5s past the deadline, outer step 0.2s
worker 0 round 3: 30 steps 3785 tok/s (3812 padded; contexts 3766 of 3791) nll 0.5366 ret 0.0000 gpu 97% free 63% | delta sent 0.5s, waited 0.2s, update 0.2s
round 4: 165/300 lr×1.000 eta 2m | ra 31 steps 4057 tok/s nll 0.4745 ret 0.0000 | ret 0.0000 | 31.2s, closed 1.2s past the deadline, outer step 0.1s
worker 0 round 4: 31 steps 4057 tok/s (4077 padded; contexts 4037 of 4054) nll 0.4745 ret 0.0000 gpu 97% free 58% | delta sent 0.3s, waited 0.1s, update 0.1s
round 5: 200/300 lr×1.000 eta 2m | ra 35 steps 4402 tok/s nll 0.4480 ret 0.0000 | ret 0.0000 | 30.8s, closed 0.8s past the deadline, outer step 0.0s
worker 0 round 5: 35 steps 4402 tok/s (4429 padded; contexts 4379 of 4404) nll 0.4480 ret 0.0000 gpu 98% free 59% | delta sent 0.2s, waited 0.0s, update 0.1s, state kept
val @200: rag_relevance 0.632/0.810
saved runs/h2run at 200 batches, 0 epochs
round 6: 231/300 lr×1.000 eta 1m | ra 31 steps 4114 tok/s nll 0.4483 ret 0.0000 | ret 0.0000 | 31.5s, closed 1.5s past the deadline, outer step 0.0s
worker 0 round 6: 31 steps 4114 tok/s (4136 padded; contexts 4093 of 4114) nll 0.4483 ret 0.0000 gpu 96% free 66% | delta sent 0.2s, waited 0.0s, update 0.1s
round 7: 267/300 lr×1.000 eta 1m | ra 36 steps 4657 tok/s nll 0.4446 ret 0.0000 | ret 0.0000 | 31.1s, closed 1.1s past the deadline, outer step 0.0s
worker 0 round 7: 36 steps 4657 tok/s (4694 padded; contexts 4634 of 4668) nll 0.4446 ret 0.0000 gpu 98% free 66% | delta sent 0.2s, waited 0.0s, update 0.1s
round 8: 300/300 lr×1.000 eta 0m | ra 33 steps 4586 tok/s nll 0.3530 ret 0.0000 | ret 0.0000 | 28.5s, closed 0.0s past the deadline, outer step 0.0s
worker 0 round 8: 33 steps 4586 tok/s (4638 padded; contexts 4563 of 4612) nll 0.3530 ret 0.0000 gpu 98% free 66% | delta sent 0.2s, waited 0.0s; done
val @300: rag_relevance 0.635/0.828
saved runs/h2run at 300 batches, 0 epochs