papers/kai/data/diagnosis/rag3b.log

corpus: 100000 train rows, 2345 val rows, suites ["rag_relevance"] (2s)
init: 176 tensors carried, fresh: ["ordinal.cut.bias", "ordinal.cut.weight", "ordinal.gap.bias", "ordinal.gap.weight", "ordinal.location.bias", "ordinal.location.center", "ordinal.location.weight", "retrieval.key.bias", "retrieval.key.center", "retrieval.key.weight", "retrieval.query.bias", "retrieval.query.center", "retrieval.query.weight", "retrieval.scale"]
plan: stage rag3, 2 epochs, 1540 batches, 46 warmup, 308 cooldown; rows on a sample of their options, by epoch: [0, 0]
cluster: coordinating on 127.0.0.1:61652
cluster: ra (127.0.0.1:61653) joins round 0 as worker 0 (Metal { gpu_id: 4294970014 }, BF16)
worker 0 (ra): joined round 0
round 0: 150/1540 lr×1.000 eta 19m | ra 150 steps 4973 tok/s nll 0.6607 ret 0.0000 | ret 0.0000 | 120.7s, closed 0.7s past the deadline, outer step 0.1s
worker 0 round 0: 150 steps 4973 tok/s (5008 padded; contexts 4948 of 4981) nll 0.6607 ret 0.0000 gpu 98% free 65% | delta sent 0.3s, waited 0.1s, update 0.1s
round 1: 301/1540 lr×1.000 eta 17m | ra 151 steps 4947 tok/s nll 0.6271 ret 0.0000 | ret 0.0000 | 121.0s, closed 1.0s past the deadline, outer step 0.0s
worker 0 round 1: 151 steps 4947 tok/s (4983 padded; contexts 4922 of 4956) nll 0.6271 ret 0.0000 gpu 98% free 68% | delta sent 0.2s, waited 0.0s, update 0.1s
round 2: 451/1540 lr×1.000 eta 15m | ra 150 steps 5006 tok/s nll 0.6365 ret 0.0000 | ret 0.0000 | 120.7s, closed 0.7s past the deadline, outer step 0.0s
worker 0 round 2: 150 steps 5006 tok/s (5043 padded; contexts 4981 of 5015) nll 0.6365 ret 0.0000 gpu 98% free 69% | delta sent 0.2s, waited 0.0s, update 0.1s
round 3: 601/1540 lr×1.000 eta 13m | ra 150 steps 5002 tok/s nll 0.6320 ret 0.0000 | ret 0.0000 | 120.8s, closed 0.8s past the deadline, outer step 0.0s
worker 0 round 3: 150 steps 5002 tok/s (5035 padded; contexts 4977 of 5008) nll 0.6320 ret 0.0000 gpu 98% free 69% | delta sent 0.2s, waited 0.0s, update 0.1s
round 4: 752/1540 lr×1.000 eta 11m | ra 151 steps 5040 tok/s nll 0.6294 ret 0.0000 | ret 0.0000 | 120.7s, closed 0.7s past the deadline, outer step 0.0s
worker 0 round 4: 151 steps 5040 tok/s (5077 padded; contexts 5015 of 5050) nll 0.6294 ret 0.0000 gpu 98% free 69% | delta sent 0.2s, waited 0.0s, update 0.1s
round 5: 907/1540 lr×1.000 eta 8m | ra 155 steps 5133 tok/s nll 0.6094 ret 0.0000 | ret 0.0000 | 120.7s, closed 0.7s past the deadline, outer step 0.0s
worker 0 round 5: 155 steps 5133 tok/s (5162 padded; contexts 5107 of 5134) nll 0.6094 ret 0.0000 gpu 98% free 69% | delta sent 0.2s, waited 0.0s, update 0.1s, state kept
val @907: rag_relevance 0.623/0.636
saved runs/rag3b at 907 batches, 1 epochs
round 6: 1054/1540 lr×1.000 eta 7m | ra 147 steps 4917 tok/s nll 0.6156 ret 0.0000 | ret 0.0000 | 121.3s, closed 1.3s past the deadline, outer step 0.2s
worker 0 round 6: 147 steps 4917 tok/s (4952 padded; contexts 4893 of 4925) nll 0.6156 ret 0.0000 gpu 98% free 65% | delta sent 0.3s, waited 0.2s, update 0.1s
round 7: 1214/1540 lr×1.000 eta 4m | ra 160 steps 5338 tok/s nll 0.6251 ret 0.0000 | ret 0.0000 | 120.7s, closed 0.7s past the deadline, outer step 0.0s
worker 0 round 7: 160 steps 5338 tok/s (5384 padded; contexts 5312 of 5355) nll 0.6251 ret 0.0000 gpu 98% free 65% | delta sent 0.2s, waited 0.0s, update 0.1s, state kept
val @1214: rag_relevance 0.580/0.725
Error: Resource temporarily unavailable (os error 35)