papers/kai/data/diagnosis/a3.log

corpus: 3691398 train rows, 139807 val rows, suites ["ag_news", "emotion", "banking77", "support_triage", "email_spam", "phishing", "guardrails_jailbreak", "moderation_toxicity", "rag_relevance", "model_routing_domain", "typed_decisions", "massive", "tool", "risk", "done", "repo", "function", "resolve", "action", "nemotron", "clinc150", "xnli", "sst5", "shuffle", "alias", "near", "subset", "swap", "codeswitch", "boolq", "mnli", "code.file.gold", "code.affected.gold", "code.symbol.gold", "code.suspect.gold", "code.test.silver", "control.context.gold", "control.user.gold", "control.continue.gold", "control.commit.gold", "control.rollback.gold", "control.strategy.gold", "control.repeat.gold", "control.done.gold", "control.retry.gold", "control.user.silver", "code.failure.gold", "code.failure.silver", "skills.select.silver", "graph_typed", "turbofan", "rig", "sysml", "readiness", "trade", "testplan", "arch"] (63s)
init: 187 tensors carried, fresh: ["ordinal.location.center", "retrieval.key.center", "retrieval.query.center"]
plan: stage a3, 2 epochs, 32681 batches, 980 warmup, 9804 cooldown; rows on a sample of their options, by epoch: [623, 654]
cluster: coordinating on 0.0.0.0:7070
cluster: dbc (127.0.0.1:60409) joins round 0 as worker 0 (Metal { gpu_id: 4294968649 }, BF16)
worker 0 (dbc): joined round 0
cluster: evo (10.0.0.57:15458) joins round 0 as worker 1 (Rocm { gpu_id: 0 }, BF16)
worker 0 round 0: 830 steps 6717 tok/s (7058 padded; contexts 5183 of 5225) nll 1.2814 ret 1.3340 gpu 93% free 81% | delta sent 0.1s, waited 7.6s, update 0.0s
round 0: 926/32681 lr×0.946 eta 190m | dbc 830 steps 6717 tok/s nll 1.2814 ret 1.3340 | evo 96 steps 2577 tok/s nll 1.3115 ret 1.2605 | ret 1.3254, recall@32 0.062 (code.file.gold 0.250, code.suspect.gold 0.500, code.symbol.gold 0.000, code.test.silver 0.250, skills.select.silver 0.020) (7.4s) | 307.8s, closed 7.8s past the deadline, outer step 0.1s
worker 0 round 1: 811 steps 6605 tok/s (6953 padded; contexts 5097 of 5154) nll 1.3088 ret 1.2676 gpu 94% free 80% | delta sent 0.1s, waited 4.6s, update 0.1s
round 1: 1948/32681 lr×1.000 eta 168m | dbc 811 steps 6605 tok/s nll 1.3088 ret 1.2676 | evo 211 steps 2326 tok/s nll 1.2393 ret 1.2945 | ret 1.2734, recall@32 0.047 (code.file.gold 0.250, code.suspect.gold 0.500, code.symbol.gold 0.000, code.test.silver 0.250, skills.select.silver 0.000) (7.8s) | 304.9s, closed 4.9s past the deadline, outer step 0.1s
worker 0 round 2: 805 steps 6634 tok/s (6972 padded; contexts 5192 of 5236) nll 1.2024 ret 1.1647 gpu 94% free 81% | delta sent 0.1s, waited 5.6s, update 0.0s
round 2: 2963/32681 lr×1.000 eta 158m | dbc 805 steps 6634 tok/s nll 1.2024 ret 1.1647 | evo 210 steps 2305 tok/s nll 1.3081 ret 1.2863 | ret 1.1919, recall@32 0.078 (code.file.gold 0.250, code.suspect.gold 1.000, code.symbol.gold 0.000, code.test.silver 0.250, skills.select.silver 0.020) (6.6s) | 306.1s, closed 6.1s past the deadline, outer step 0.0s
worker 0 round 3: 825 steps 6734 tok/s (7082 padded; contexts 5257 of 5313) nll 1.2541 ret 1.1725 gpu 94% free 80% | delta sent 0.1s, waited 6.2s, update 0.0s
round 3: 3989/32681 lr×1.000 eta 150m | dbc 825 steps 6734 tok/s nll 1.2541 ret 1.1725 | evo 201 steps 2257 tok/s nll 1.2135 ret 1.1401 | ret 1.1659, recall@32 0.062 (code.file.gold 0.250, code.suspect.gold 0.500, code.symbol.gold 0.000, code.test.silver 0.250, skills.select.silver 0.020) (7.3s) | 306.7s, closed 6.7s past the deadline, outer step 0.0s
worker 0 round 4: 794 steps 6493 tok/s (6829 padded; contexts 5028 of 5075) nll 1.1956 ret 1.1391 gpu 94% free 80% | delta sent 0.1s, waited 5.7s, update 0.0s
round 4: 4964/32681 lr×1.000 eta 145m | dbc 794 steps 6493 tok/s nll 1.1956 ret 1.1391 | evo 181 steps 2067 tok/s nll 1.2064 ret 1.1321 | ret 1.1378, recall@32 0.125 (code.file.gold 0.250, code.suspect.gold 1.000, code.symbol.gold 0.000, code.test.silver 0.250, skills.select.silver 0.080) (6.8s) | 306.2s, closed 6.2s past the deadline, outer step 0.0s
worker 0 round 5: 821 steps 6689 tok/s (7040 padded; contexts 5189 of 5237) nll 1.1482 ret 1.0956 gpu 93% free 81% | delta sent 0.1s, waited 4.1s, update 0.0s
round 5: 5966/32681 lr×1.000 eta 139m | dbc 821 steps 6689 tok/s nll 1.1482 ret 1.0956 | evo 181 steps 2011 tok/s nll 1.2778 ret 1.2575 | ret 1.1270, recall@32 0.078 (code.file.gold 0.250, code.suspect.gold 1.000, code.symbol.gold 0.000, code.test.silver 0.250, skills.select.silver 0.020) (6.9s) | 304.8s, closed 4.8s past the deadline, outer step 0.0s
worker 0 round 6: 813 steps 6798 tok/s (7152 padded; contexts 5375 of 5443) nll 1.1061 ret 1.0930 gpu 93% free 81% | delta sent 0.1s, waited 5.8s, update 0.0s
round 6: 6932/32681 lr×1.000 eta 134m | dbc 813 steps 6798 tok/s nll 1.1061 ret 1.0930 | evo 153 steps 1701 tok/s nll 1.1162 ret 1.0231 | ret 1.0794, recall@32 0.109 (code.file.gold 0.250, code.suspect.gold 1.000, code.symbol.gold 0.000, code.test.silver 0.250, skills.select.silver 0.060) (6.8s) | 306.3s, closed 6.3s past the deadline, outer step 0.0s
worker 0 round 7: 828 steps 6778 tok/s (7147 padded; contexts 5222 of 5277) nll 1.1586 ret 1.0969 gpu 93% free 81% | delta sent 0.1s, waited 4.9s, update 0.0s
round 7: 7912/32681 lr×1.000 eta 129m | dbc 828 steps 6778 tok/s nll 1.1586 ret 1.0969 | evo 152 steps 1735 tok/s nll 1.1280 ret 1.0951 | ret 1.0966, recall@32 0.062 (code.file.gold 0.250, code.suspect.gold 0.500, code.symbol.gold 0.000, code.test.silver 0.250, skills.select.silver 0.020) (6.8s) | 305.4s, closed 5.4s past the deadline, outer step 0.0s
worker 0 round 8: 839 steps 6860 tok/s (7215 padded; contexts 5323 of 5377) nll 1.1120 ret 1.0534 gpu 94% free 81% | delta sent 0.1s, waited 7.8s, update 0.0s
round 8: 8891/32681 lr×1.000 eta 124m | dbc 839 steps 6860 tok/s nll 1.1120 ret 1.0534 | evo 140 steps 1582 tok/s nll 1.1663 ret 1.2461 | ret 1.0765, recall@32 0.062 (code.file.gold 0.250, code.suspect.gold 1.000, code.symbol.gold 0.000, code.test.silver 0.000, skills.select.silver 0.020) (6.9s) | 308.1s, closed 8.1s past the deadline, outer step 0.0s
worker 0 round 9: 836 steps 6805 tok/s (7166 padded; contexts 5260 of 5317) nll 1.1413 ret 1.0820 gpu 93% free 81% | delta sent 0.1s, waited 5.8s, update 0.0s
round 9: 9834/32681 lr×1.000 eta 120m | dbc 836 steps 6805 tok/s nll 1.1413 ret 1.0820 | evo 107 steps 1225 tok/s nll 1.0136 ret 1.0293 | ret 1.0748, recall@32 0.062 (code.file.gold 0.250, code.suspect.gold 0.500, code.symbol.gold 0.000, code.test.silver 0.250, skills.select.silver 0.020) (6.8s) | 306.0s, closed 6.0s past the deadline, outer step 0.0s
worker 0 round 10: 820 steps 6806 tok/s (7165 padded; contexts 5273 of 5323) nll 1.1404 ret 1.0721 gpu 94% free 80% | delta sent 0.1s, waited 5.3s, update 0.0s
round 10: 10801/32681 lr×1.000 eta 115m | dbc 820 steps 6806 tok/s nll 1.1404 ret 1.0721 | evo 147 steps 1639 tok/s nll 1.0105 ret 0.9811 | ret 1.0568, recall@32 0.078 (code.file.gold 0.250, code.suspect.gold 0.500, code.symbol.gold 0.250, code.test.silver 0.250, skills.select.silver 0.020) (7.2s) | 305.6s, closed 5.6s past the deadline, outer step 0.0s
worker 0 round 11: 839 steps 6772 tok/s (7130 padded; contexts 5242 of 5296) nll 1.0595 ret 1.0365 gpu 94% free 81% | delta sent 0.1s, waited 6.0s, update 0.0s
round 11: 11774/32681 lr×1.000 eta 110m | dbc 839 steps 6772 tok/s nll 1.0595 ret 1.0365 | evo 134 steps 1522 tok/s nll 1.1925 ret 0.9237 | ret 1.0187, recall@32 0.078 (code.file.gold 0.250, code.suspect.gold 0.500, code.symbol.gold 0.250, code.test.silver 0.250, skills.select.silver 0.020) (6.7s) | 306.3s, closed 6.3s past the deadline, outer step 0.0s
worker 0 round 12: 824 steps 6715 tok/s (7073 padded; contexts 5143 of 5193) nll 1.0902 ret 1.0136 gpu 94% free 80% | delta sent 0.1s, waited 4.2s, update 0.0s
round 12: 12733/32681 lr×1.000 eta 105m | dbc 824 steps 6715 tok/s nll 1.0902 ret 1.0136 | evo 135 steps 1523 tok/s nll 0.9276 ret 1.0268 | ret 1.0156, recall@32 0.141 (code.file.gold 0.250, code.suspect.gold 0.500, code.symbol.gold 0.500, code.test.silver 0.250, skills.select.silver 0.080) (7.0s) | 304.7s, closed 4.7s past the deadline, outer step 0.0s
worker 0 round 13: 829 steps 6683 tok/s (7029 padded; contexts 5213 of 5262) nll 1.0655 ret 1.0331 gpu 94% free 81% | delta sent 0.1s, waited 5.9s, update 0.0s
round 13: 13702/32681 lr×1.000 eta 100m | dbc 829 steps 6683 tok/s nll 1.0655 ret 1.0331 | evo 140 steps 1597 tok/s nll 1.1236 ret 0.9700 | ret 1.0236, recall@32 0.094 (code.file.gold 0.250, code.suspect.gold 0.500, code.symbol.gold 0.250, code.test.silver 0.250, skills.select.silver 0.040) (6.8s) | 306.1s, closed 6.1s past the deadline, outer step 0.0s
worker 0 round 14: 829 steps 6791 tok/s (7137 padded; contexts 5272 of 5313) nll 1.0702 ret 1.0072 gpu 94% free 81% | delta sent 0.1s, waited 5.9s, update 0.0s
round 14: 14678/32681 lr×1.000 eta 94m | dbc 829 steps 6791 tok/s nll 1.0702 ret 1.0072 | evo 147 steps 1645 tok/s nll 1.0185 ret 0.9782 | ret 1.0027, recall@32 0.078 (code.file.gold 0.250, code.suspect.gold 0.500, code.symbol.gold 0.500, code.test.silver 0.250, skills.select.silver 0.000) (6.7s) | 306.3s, closed 6.3s past the deadline, outer step 0.0s
worker 0 round 15: 836 steps 6854 tok/s (7184 padded; contexts 5393 of 5439) nll 0.9997 ret 1.0064 gpu 94% free 80% | delta sent 0.1s, waited 5.4s, update 0.0s
round 15: 15659/32681 lr×1.000 eta 89m | dbc 836 steps 6854 tok/s nll 0.9997 ret 1.0064 | evo 145 steps 1592 tok/s nll 1.1441 ret 1.0206 | ret 1.0082, recall@32 0.094 (code.file.gold 0.500, code.suspect.gold 0.500, code.symbol.gold 0.000, code.test.silver 0.250, skills.select.silver 0.040) (7.3s) | 306.0s, closed 6.0s past the deadline, outer step 0.0s
worker 0 round 16: 796 steps 6451 tok/s (6792 padded; contexts 4972 of 5020) nll 1.0032 ret 0.9811 gpu 93% free 80% | delta sent 0.1s, waited 6.4s, update 0.1s, state kept
round 16: 16598/32681 lr×1.000 eta 84m | dbc 796 steps 6451 tok/s nll 1.0032 ret 0.9811 | evo 143 steps 1619 tok/s nll 0.8975 ret 0.9620 | ret 0.9780, recall@32 0.125 (code.file.gold 0.500, code.suspect.gold 0.500, code.symbol.gold 0.250, code.test.silver 0.000, skills.select.silver 0.080) (7.9s) | 306.6s, closed 6.6s past the deadline, outer step 0.1s
val: 344 rows over 255218 options, each answered over all of them, retrieval's top 160 by the head (156s); 13214 rows by the head (367s)
val @16598: action 0.282/1.857  ag_news 0.890/0.407  arch 0.667/0.830  banking77 0.693/1.678  boolq 0.638/0.658  clinc150 0.657/1.592  code.affected.gold 0.477/0.693  code.failure.silver 0.316/1.789  code.file.gold 0.083/4.535  code.suspect.gold 0.000/5.088  code.symbol.gold 0.089/4.298  code.test.silver 0.020/3.860  control.commit.gold 0.286/1.488  control.context.gold 0.806/0.805  control.continue.gold 0.952/0.479  control.done.gold 0.364/0.720  control.repeat.gold 1.000/0.650  control.retry.gold 0.000/0.955  control.rollback.gold 0.429/0.695  control.strategy.gold 0.750/0.581  control.user.silver 0.875/0.395  done 0.970/0.433  email_spam 0.983/0.042  emotion 0.831/0.561  function 0.593/1.669  graph_typed 0.548/1.085  guardrails_jailbreak 0.940/0.192  massive 0.787/0.769  mnli 0.583/0.913  model_routing_domain 0.995/0.027  moderation_toxicity 0.885/0.322  nemotron 0.312/2.612  phishing 0.969/0.143  rag_relevance 0.497/0.693  readiness 0.502/1.123  resolve 0.985/0.388  rig 0.468/1.166  risk 0.072/2.488  skills.select.silver 0.000/7.109  sst5 0.207/1.630  support_triage 0.400/1.651  sysml 0.600/0.715  testplan 0.608/1.010  tool 0.565/1.987  trade 0.583/0.807  turbofan 0.480/1.025  typed_decisions 0.537/1.035  xnli 0.600/0.961
saved /Users/a/scratch/decision/runs/a3 at 16598 batches, 1 epochs
worker 0 round 17: 512 steps 4148 tok/s (4377 padded; contexts 3214 of 3255) nll 1.0080 ret 0.9522 gpu 93% free 77% | delta sent 0.2s, waited 231.0s, update 0.1s
round 17: 17255/32681 lr×1.000 eta 86m | dbc 512 steps 4148 tok/s nll 1.0080 ret 0.9522 | evo 145 steps 1625 tok/s nll 1.0574 ret 0.8483 | ret 0.9301, recall@32 0.078 (code.file.gold 0.500, code.suspect.gold 0.500, code.symbol.gold 0.250, code.test.silver 0.250, skills.select.silver 0.000) (7.1s) | 532.2s, closed 232.2s past the deadline, outer step 0.0s
worker 0 round 18: 788 steps 6497 tok/s (6834 padded; contexts 5050 of 5100) nll 0.9992 ret 0.9596 gpu 94% free 79% | delta sent 0.1s, waited 5.1s, update 0.0s
round 18: 18184/32681 lr×1.000 eta 81m | dbc 788 steps 6497 tok/s nll 0.9992 ret 0.9596 | evo 141 steps 1538 tok/s nll 0.9785 ret 0.9185 | ret 0.9540, recall@32 0.047 (code.file.gold 0.250, code.suspect.gold 0.500, code.symbol.gold 0.000, code.test.silver 0.250, skills.select.silver 0.000) (7.0s) | 305.4s, closed 5.4s past the deadline, outer step 0.1s