papers/kai/data/diagnosis/a2.log

corpus: 3686494 train rows, 139612 val rows, suites ["ag_news", "emotion", "banking77", "support_triage", "email_spam", "phishing", "guardrails_jailbreak", "moderation_toxicity", "rag_relevance", "model_routing_domain", "typed_decisions", "massive", "tool", "risk", "done", "repo", "function", "resolve", "action", "nemotron", "clinc150", "xnli", "sst5", "shuffle", "alias", "near", "subset", "swap", "codeswitch", "boolq", "mnli", "code.file.gold", "code.affected.gold", "code.symbol.gold", "code.suspect.gold", "code.test.silver", "control.context.gold", "control.user.gold", "control.continue.gold", "control.commit.gold", "control.rollback.gold", "control.strategy.gold", "control.repeat.gold", "control.done.gold", "control.retry.gold", "control.user.silver", "code.failure.gold", "code.failure.silver", "skills.select.silver", "graph_typed", "turbofan", "rig", "sysml", "readiness", "trade", "testplan", "arch"] (62s)
init: 187 tensors carried, fresh: ["retrieval.key.center", "retrieval.query.center"]
plan: stage a2, 1 epochs, 32801 batches, 984 warmup, 9840 cooldown; rows on a sample of their options, by epoch: [10220]
cluster: coordinating on 0.0.0.0:7070
cluster: dbc (127.0.0.1:57605) joins round 0 as worker 0 (Metal { gpu_id: 4294968649 }, BF16)
worker 0 (dbc): joined round 0
cluster: evo (10.0.0.57:3292) joins round 0 as worker 1 (Rocm { gpu_id: 0 }, BF16)
worker 0 round 0: 766 steps 5872 tok/s (6449 padded; contexts 2771 of 2783) nll 2.6453 ret 1.8067 gpu 92% free 80% | delta sent 0.1s, waited 10.0s, update 0.0s
round 0: 809/32801 lr×0.823 eta 222m | dbc 766 steps 5872 tok/s nll 2.6453 ret 1.8067 | evo 43 steps 1140 tok/s nll 1.9829 ret 1.1796 | ret 1.7685, recall@32 0.078 (code.file.gold 0.750, code.suspect.gold 0.000, code.symbol.gold 0.000, code.test.silver 0.000, skills.select.silver 0.040) (6.5s) | 310.4s, closed 10.4s past the deadline, outer step 0.1s
worker 0 round 1: 780 steps 5824 tok/s (6400 padded; contexts 2882 of 2904) nll 2.5649 ret 1.7482 gpu 94% free 80% | delta sent 0.1s, waited 5.4s, update 0.0s
round 1: 1815/32801 lr×1.000 eta 183m | dbc 780 steps 5824 tok/s nll 2.5649 ret 1.7482 | evo 226 steps 2048 tok/s nll 2.5012 ret 1.8045 | ret 1.7590, recall@32 0.047 (code.file.gold 0.250, code.suspect.gold 0.000, code.symbol.gold 0.000, code.test.silver 0.000, skills.select.silver 0.040) (6.8s) | 306.0s, closed 6.0s past the deadline, outer step 0.0s
worker 0 round 2: 775 steps 5911 tok/s (6477 padded; contexts 2891 of 2914) nll 2.5782 ret 1.6768 gpu 94% free 80% | delta sent 0.1s, waited 4.7s, update 0.0s
round 2: 2793/32801 lr×1.000 eta 170m | dbc 775 steps 5911 tok/s nll 2.5782 ret 1.6768 | evo 203 steps 1844 tok/s nll 2.5816 ret 1.8302 | ret 1.7100, recall@32 0.062 (code.file.gold 0.250, code.suspect.gold 1.000, code.symbol.gold 0.000, code.test.silver 0.000, skills.select.silver 0.020) (6.4s) | 305.2s, closed 5.2s past the deadline, outer step 0.0s
worker 0 round 3: 756 steps 5811 tok/s (6374 padded; contexts 2851 of 2868) nll 2.6168 ret 1.6745 gpu 91% free 80% | delta sent 0.1s, waited 15.4s, update 0.0s
round 3: 3760/32801 lr×1.000 eta 163m | dbc 756 steps 5811 tok/s nll 2.6168 ret 1.6745 | evo 211 steps 1976 tok/s nll 2.4401 ret 1.7131 | ret 1.6845, recall@32 0.047 (code.file.gold 0.250, code.suspect.gold 0.500, code.symbol.gold 0.000, code.test.silver 0.000, skills.select.silver 0.020) (6.7s) | 315.7s, closed 15.7s past the deadline, outer step 0.0s
worker 0 round 4: 757 steps 5911 tok/s (6500 padded; contexts 2870 of 2888) nll 2.5739 ret 1.6241 gpu 93% free 81% | delta sent 0.1s, waited 5.3s, update 0.0s
round 4: 4724/32801 lr×1.000 eta 155m | dbc 757 steps 5911 tok/s nll 2.5739 ret 1.6241 | evo 207 steps 1903 tok/s nll 2.5638 ret 1.6050 | ret 1.6202, recall@32 0.078 (code.file.gold 0.000, code.suspect.gold 0.500, code.symbol.gold 0.000, code.test.silver 0.000, skills.select.silver 0.080) (6.5s) | 305.5s, closed 5.5s past the deadline, outer step 0.0s
worker 0 round 5: 773 steps 5967 tok/s (6536 padded; contexts 3020 of 3039) nll 2.4620 ret 1.5786 gpu 94% free 81% | delta sent 0.1s, waited 5.1s, update 0.0s
round 5: 5689/32801 lr×1.000 eta 149m | dbc 773 steps 5967 tok/s nll 2.4620 ret 1.5786 | evo 192 steps 1716 tok/s nll 2.6544 ret 1.6208 | ret 1.5889, recall@32 0.078 (code.file.gold 0.500, code.suspect.gold 1.000, code.symbol.gold 0.000, code.test.silver 0.000, skills.select.silver 0.020) (6.3s) | 305.3s, closed 5.3s past the deadline, outer step 0.0s
worker 0 round 6: 762 steps 5956 tok/s (6507 padded; contexts 2991 of 3007) nll 2.5758 ret 1.5476 gpu 93% free 81% | delta sent 0.1s, waited 4.9s, update 0.0s
round 6: 6648/32801 lr×1.000 eta 143m | dbc 762 steps 5956 tok/s nll 2.5758 ret 1.5476 | evo 197 steps 1715 tok/s nll 2.7762 ret 1.7974 | ret 1.5967, recall@32 0.078 (code.file.gold 0.500, code.suspect.gold 1.000, code.symbol.gold 0.250, code.test.silver 0.000, skills.select.silver 0.000) (6.3s) | 305.1s, closed 5.1s past the deadline, outer step 0.0s
worker 0 round 7: 780 steps 5801 tok/s (6364 padded; contexts 2862 of 2886) nll 2.5884 ret 1.6188 gpu 93% free 81% | delta sent 0.1s, waited 4.9s, update 0.0s
round 7: 7618/32801 lr×1.000 eta 137m | dbc 780 steps 5801 tok/s nll 2.5884 ret 1.6188 | evo 190 steps 1747 tok/s nll 2.6648 ret 1.5457 | ret 1.6013, recall@32 0.047 (code.file.gold 0.500, code.suspect.gold 0.500, code.symbol.gold 0.000, code.test.silver 0.000, skills.select.silver 0.000) (6.5s) | 305.1s, closed 5.1s past the deadline, outer step 0.0s
worker 0 round 8: 773 steps 5912 tok/s (6481 padded; contexts 2869 of 2886) nll 2.6126 ret 1.5523 gpu 94% free 80% | delta sent 0.1s, waited 4.8s, update 0.0s
round 8: 8579/32801 lr×1.000 eta 131m | dbc 773 steps 5912 tok/s nll 2.6126 ret 1.5523 | evo 188 steps 1734 tok/s nll 2.5469 ret 1.5640 | ret 1.5547, recall@32 0.062 (code.file.gold 0.500, code.suspect.gold 0.500, code.symbol.gold 0.000, code.test.silver 0.000, skills.select.silver 0.020) (6.6s) | 305.1s, closed 5.1s past the deadline, outer step 0.1s
worker 0 round 9: 763 steps 5796 tok/s (6346 padded; contexts 2834 of 2849) nll 2.5516 ret 1.4962 gpu 93% free 80% | delta sent 0.1s, waited 6.5s, update 0.1s
round 9: 9527/32801 lr×1.000 eta 126m | dbc 763 steps 5796 tok/s nll 2.5516 ret 1.4962 | evo 185 steps 1659 tok/s nll 2.7267 ret 1.5438 | ret 1.5043, recall@32 0.094 (code.file.gold 0.250, code.suspect.gold 0.500, code.symbol.gold 0.000, code.test.silver 0.000, skills.select.silver 0.080) (6.3s) | 306.8s, closed 6.8s past the deadline, outer step 0.0s
worker 0 round 10: 768 steps 5909 tok/s (6443 padded; contexts 2942 of 2958) nll 2.6012 ret 1.4786 gpu 92% free 80% | delta sent 0.1s, waited 5.6s, update 0.0s
round 10: 10490/32801 lr×1.000 eta 121m | dbc 768 steps 5909 tok/s nll 2.6012 ret 1.4786 | evo 195 steps 1798 tok/s nll 2.7726 ret 1.6269 | ret 1.5082, recall@32 0.047 (code.file.gold 0.250, code.suspect.gold 0.000, code.symbol.gold 0.250, code.test.silver 0.000, skills.select.silver 0.020) (6.6s) | 306.0s, closed 6.0s past the deadline, outer step 0.0s
worker 0 round 11: 752 steps 5763 tok/s (6348 padded; contexts 2841 of 2860) nll 2.6669 ret 1.5628 gpu 94% free 80% | delta sent 0.1s, waited 4.9s, update 0.0s
round 11: 11427/32801 lr×1.000 eta 116m | dbc 752 steps 5763 tok/s nll 2.6669 ret 1.5628 | evo 185 steps 1728 tok/s nll 2.5053 ret 1.4501 | ret 1.5372, recall@32 0.062 (code.file.gold 0.250, code.suspect.gold 0.000, code.symbol.gold 0.000, code.test.silver 0.500, skills.select.silver 0.020) (6.7s) | 305.3s, closed 5.3s past the deadline, outer step 0.0s