papers/kai/data/diagnosis/rag1.log

corpus: 100000 train rows, 2345 val rows, suites ["rag_relevance"] (2s)
init: 176 tensors carried, fresh: ["ordinal.cut.bias", "ordinal.cut.weight", "ordinal.gap.bias", "ordinal.gap.weight", "ordinal.location.bias", "ordinal.location.center", "ordinal.location.weight", "retrieval.key.bias", "retrieval.key.center", "retrieval.key.weight", "retrieval.query.bias", "retrieval.query.center", "retrieval.query.weight", "retrieval.scale"]
plan: stage rag1, 1 epochs, 771 batches, 23 warmup, 154 cooldown; rows on a sample of their options, by epoch: [0]
cluster: coordinating on 127.0.0.1:57622
cluster: ra (127.0.0.1:57623) joins round 0 as worker 0 (Metal { gpu_id: 4294970014 }, BF16)
worker 0 (ra): joined round 0
round 0: 27/771 lr×1.000 eta 15m | ra 27 steps 3547 tok/s nll 0.7139 ret 0.0000 | ret 0.0000 | 31.3s, closed 1.3s past the deadline, outer step 0.1s
worker 0 round 0: 27 steps 3547 tok/s (3562 padded; contexts 3530 of 3542) nll 0.7139 ret 0.0000 gpu 70% free 76% | delta sent 0.3s, waited 0.1s, update 0.1s
round 1: 67/771 lr×1.000 eta 11m | ra 40 steps 5213 tok/s nll 0.6818 ret 0.0000 | ret 0.0000 | 31.1s, closed 1.1s past the deadline, outer step 0.0s
worker 0 round 1: 40 steps 5213 tok/s (5266 padded; contexts 5187 of 5237) nll 0.6818 ret 0.0000 gpu 98% free 74% | delta sent 0.2s, waited 0.0s, update 0.1s
round 2: 108/771 lr×1.000 eta 10m | ra 41 steps 5244 tok/s nll 0.6455 ret 0.0000 | ret 0.0000 | 30.6s, closed 0.6s past the deadline, outer step 0.0s
worker 0 round 2: 41 steps 5244 tok/s (5283 padded; contexts 5217 of 5253) nll 0.6455 ret 0.0000 gpu 98% free 73% | delta sent 0.2s, waited 0.0s, update 0.1s
round 3: 148/771 lr×1.000 eta 9m | ra 40 steps 5284 tok/s nll 0.6430 ret 0.0000 | ret 0.0000 | 30.7s, closed 0.7s past the deadline, outer step 0.0s
worker 0 round 3: 40 steps 5284 tok/s (5315 padded; contexts 5258 of 5286) nll 0.6430 ret 0.0000 gpu 98% free 73% | delta sent 0.2s, waited 0.0s, update 0.1s
round 4: 188/771 lr×1.000 eta 8m | ra 40 steps 5219 tok/s nll 0.6056 ret 0.0000 | ret 0.0000 | 30.5s, closed 0.5s past the deadline, outer step 0.0s
worker 0 round 4: 40 steps 5219 tok/s (5250 padded; contexts 5192 of 5221) nll 0.6056 ret 0.0000 gpu 98% free 73% | delta sent 0.2s, waited 0.0s, update 0.1s
round 5: 229/771 lr×1.000 eta 7m | ra 41 steps 5254 tok/s nll 0.6245 ret 0.0000 | ret 0.0000 | 31.1s, closed 1.1s past the deadline, outer step 0.0s
worker 0 round 5: 41 steps 5254 tok/s (5282 padded; contexts 5227 of 5253) nll 0.6245 ret 0.0000 gpu 98% free 73% | delta sent 0.2s, waited 0.0s, update 0.1s, state kept
val @229: rag_relevance 0.645/0.644
saved runs/rag1 at 229 batches, 0 epochs
round 6: 266/771 lr×1.000 eta 7m | ra 37 steps 4791 tok/s nll 0.6527 ret 0.0000 | ret 0.0000 | 31.8s, closed 1.8s past the deadline, outer step 0.0s
worker 0 round 6: 37 steps 4791 tok/s (4830 padded; contexts 4767 of 4803) nll 0.6527 ret 0.0000 gpu 97% free 65% | delta sent 0.2s, waited 0.0s, update 0.1s
round 7: 306/771 lr×1.000 eta 6m | ra 40 steps 5236 tok/s nll 0.6235 ret 0.0000 | ret 0.0000 | 30.4s, closed 0.4s past the deadline, outer step 0.0s
worker 0 round 7: 40 steps 5236 tok/s (5287 padded; contexts 5209 of 5258) nll 0.6235 ret 0.0000 gpu 98% free 66% | delta sent 0.2s, waited 0.0s, update 0.1s
round 8: 346/771 lr×1.000 eta 6m | ra 40 steps 5320 tok/s nll 0.6217 ret 0.0000 | ret 0.0000 | 30.6s, closed 0.6s past the deadline, outer step 0.0s
worker 0 round 8: 40 steps 5320 tok/s (5347 padded; contexts 5294 of 5318) nll 0.6217 ret 0.0000 gpu 98% free 66% | delta sent 0.2s, waited 0.0s, update 0.1s
round 9: 386/771 lr×1.000 eta 5m | ra 40 steps 5279 tok/s nll 0.6174 ret 0.0000 | ret 0.0000 | 30.7s, closed 0.7s past the deadline, outer step 0.0s
worker 0 round 9: 40 steps 5279 tok/s (5312 padded; contexts 5253 of 5283) nll 0.6174 ret 0.0000 gpu 98% free 66% | delta sent 0.2s, waited 0.0s, update 0.1s
round 10: 426/771 lr×1.000 eta 5m | ra 40 steps 5317 tok/s nll 0.6277 ret 0.0000 | ret 0.0000 | 30.5s, closed 0.5s past the deadline, outer step 0.0s
worker 0 round 10: 40 steps 5317 tok/s (5350 padded; contexts 5291 of 5321) nll 0.6277 ret 0.0000 gpu 98% free 66% | delta sent 0.2s, waited 0.0s, update 0.1s, state kept
val @426: rag_relevance 0.640/0.643