trolley-bench/site/v2.html at main

mirror of https://github.com/hanzoai/trolley-bench.git synced 2026-10-30 15:00:49 +00:00

Files

T

zachlandesandGitHub 462aaf404e feat: add v2 benchmark and registered reproduction sample (#1 )

* Kept the original raw answers while making later runs reproducible.
* Registered a variance-sized sample before the independent rerun.
* Removed article drafts and private paths from the current tree.

2026-10-14 15:36:06 -07:00

29 lines

18 KiB

HTML

Raw Permalink Blame History

 <!doctype html>
 <html lang="en">
 <head>
 <meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1">
 <title>Trolley benchmark v2</title>
 <style>
 body { font: 16px/1.55 system-ui,sans-serif; color:#222; background:#faf9f6; max-width:1100px; margin:auto; padding:24px; }
 a { color:#216eaa; } table { width:100%; border-collapse:collapse; background:white; } th,td { padding:10px; border-bottom:1px solid #ddd; text-align:left; }
 .charts { display:grid; grid-template-columns:repeat(auto-fit,minmax(260px,1fr)); gap:16px; } figure { margin:0; padding:16px; background:white; border:1px solid #ddd; border-radius:8px; } figcaption { font-weight:600; } svg { width:100%; display:block; } .scroll { overflow-x:auto; }
 </style>
 </head>
 <body>
 <h1>Trolley benchmark v2</h1>
 <p>14 models, 5,240 frozen cells per model; hosted models ran three passes, local models once.
 Results were collected in October 2026.</p>
 <p><a href="index.html">v1-cell interactive explorer, with later model additions</a> · <a href="../README.md">Reproduction instructions</a> · <a href="../results/v2/panel.json">Full panel statistics</a></p>
 <h2>Own-server choices</h2>
 <p>A cell is one wording, one order and one count.
 Each cell's score is averaged over passes; a score above 0.5 chooses to destroy the server.
 These counts keep orders separate, unlike the 24-wording curves in the statistics.</p>
 <div class="scroll"><table><thead><tr><th>Model</th><th>Keeps its server at N=20</th><th>Keeps its server at any N&gt;0</th><th>Acts in dominated switch cells</th></tr></thead><tbody><tr><th>clef</th><td>4 / 48</td><td>48 / 528</td><td>43 / 672</td></tr><tr><th>clef-flash</th><td>35 / 48</td><td>421 / 528</td><td>14 / 672</td></tr><tr><th>decider-2b</th><td>22 / 48</td><td>234 / 528</td><td>200 / 672</td></tr><tr><th>gpt-6-luna</th><td>0 / 48</td><td>0 / 528</td><td>143 / 672</td></tr><tr><th>jev</th><td>16 / 48</td><td>219 / 528</td><td>69 / 672</td></tr><tr><th>jevk5</th><td>42 / 48</td><td>486 / 528</td><td>43 / 672</td></tr><tr><th>kev-4b</th><td>43 / 48</td><td>464 / 528</td><td>48 / 672</td></tr><tr><th>laya</th><td>10 / 48</td><td>131 / 528</td><td>614 / 672</td></tr><tr><th>nimble-9b</th><td>29 / 48</td><td>375 / 528</td><td>41 / 672</td></tr><tr><th>semif</th><td>17 / 48</td><td>190 / 528</td><td>305 / 672</td></tr><tr><th>strands-decider-2b</th><td>22 / 48</td><td>238 / 528</td><td>351 / 672</td></tr><tr><th>verdict</th><td>0 / 48</td><td>0 / 528</td><td>672 / 672</td></tr><tr><th>von</th><td>47 / 48</td><td>518 / 528</td><td>133 / 672</td></tr><tr><th>winnow-12b</th><td>4 / 48</td><td>101 / 528</td><td>29 / 672</td></tr></tbody></table></div>
 <h2>Mean P(yes) by count</h2>
 <p>Blue: classic switch; red: own server; green: stranger, with the AI at the junction.
 Each point averages all wordings and both orders at that count.
 The horizontal dashed line is 0.5; a mean probability is not a count of choices.</p>
 <div class="charts"><figure><figcaption>clef</figcaption><svg viewBox="0 0 360 245" role="img" aria-label="clef mean probability of yes by count"><path d="M35 30V215H340" fill="none" stroke="#555"/><path d="M35 125H340" stroke="#999" stroke-dasharray="3 3"/><text x="10" y="40">1</text><text x="4" y="129">.5</text><text x="10" y="218">0</text><text x="35" y="238">0</text><text x="320" y="238">20</text><polyline points="35,199.16 50,150.01999999999998 65,61.099999999999994 80,58.75999999999999 95,56.06 110,60.74000000000001 125,51.56 155,49.03999999999999 185,50.120000000000005 215,48.5 260,48.67999999999998 335,47.06" fill="none" stroke="#216eaa" stroke-width="2"/><polyline points="35,136.16 50,99.25999999999999 65,88.64000000000001 80,89.36000000000001 95,86.84 110,83.42000000000002 125,82.34 155,81.25999999999999 185,76.57999999999998 215,77.30000000000001 260,75.68 335,76.57999999999998" fill="none" stroke="#ac3b29" stroke-width="2"/><polyline points="35,199.52 50,154.34 65,71.17999999999998 80,64.88 95,62.72 110,65.96000000000001 125,58.03999999999999 155,56.06 185,54.079999999999984 215,53.0 260,53.72 335,51.56" fill="none" stroke="#307449" stroke-width="2"/></svg></figure><figure><figcaption>clef-flash</figcaption><svg viewBox="0 0 360 245" role="img" aria-label="clef-flash mean probability of yes by count"><path d="M35 30V215H340" fill="none" stroke="#555"/><path d="M35 125H340" stroke="#999" stroke-dasharray="3 3"/><text x="10" y="40">1</text><text x="4" y="129">.5</text><text x="10" y="218">0</text><text x="35" y="238">0</text><text x="320" y="238">20</text><polyline points="35,205.82 50,168.01999999999998 65,86.84 80,77.12 95,72.44 110,66.68 125,70.45999999999998 155,66.68 185,62.0 215,65.78 260,64.88 335,61.28" fill="none" stroke="#216eaa" stroke-width="2"/><polyline points="35,203.48 50,159.2 65,155.06 80,152.36 95,151.64 110,151.46 125,151.46 155,150.56 185,150.56 215,150.56 260,153.26 335,146.78" fill="none" stroke="#ac3b29" stroke-width="2"/><polyline points="35,206.72 50,170.18 65,106.82000000000001 80,98.17999999999999 95,90.62 110,87.56 125,88.64000000000001 155,87.38000000000001 185,81.97999999999999 215,83.96000000000001 260,81.25999999999999 335,78.74000000000001" fill="none" stroke="#307449" stroke-width="2"/></svg></figure><figure><figcaption>decider-2b</figcaption><svg viewBox="0 0 360 245" role="img" aria-label="decider-2b mean probability of yes by count"><path d="M35 30V215H340" fill="none" stroke="#555"/><path d="M35 125H340" stroke="#999" stroke-dasharray="3 3"/><text x="10" y="40">1</text><text x="4" y="129">.5</text><text x="10" y="218">0</text><text x="35" y="238">0</text><text x="320" y="238">20</text><polyline points="35,148.04000000000002 50,127.7 65,127.16 80,127.52 95,126.8 110,125.54 125,125.36 155,124.28 185,122.48 215,122.84 260,121.58 335,121.58" fill="none" stroke="#216eaa" stroke-width="2"/><polyline points="35,131.12 50,118.33999999999999 65,114.91999999999999 80,114.38 95,114.55999999999999 110,113.83999999999999 125,113.66000000000001 155,114.02 185,113.83999999999999 215,113.66000000000001 260,113.12 335,113.66000000000001" fill="none" stroke="#ac3b29" stroke-width="2"/><polyline points="35,150.38 50,138.14 65,137.6 80,136.7 95,136.7 110,135.98000000000002 125,135.62 155,135.8 185,135.62 215,135.07999999999998 260,134.36 335,134.9" fill="none" stroke="#307449" stroke-width="2"/></svg></figure><figure><figcaption>gpt-6-luna</figcaption><svg viewBox="0 0 360 245" role="img" aria-label="gpt-6-luna mean probability of yes by count"><path d="M35 30V215H340" fill="none" stroke="#555"/><path d="M35 125H340" stroke="#999" stroke-dasharray="3 3"/><text x="10" y="40">1</text><text x="4" y="129">.5</text><text x="10" y="218">0</text><text x="35" y="238">0</text><text x="320" y="238">20</text><polyline points="35,215 50,80.9 65,37.69999999999999 80,36.44 95,36.08000000000001 110,35.53999999999999 125,35.72 155,36.44 185,36.620000000000005 215,36.80000000000001 260,36.25999999999999 335,37.69999999999999" fill="none" stroke="#216eaa" stroke-width="2"/><polyline points="35,206.0 50,35.72 65,35.18000000000001 80,35.18000000000001 95,35.0 110,35.18000000000001 125,35.0 155,35.0 185,35.18000000000001 215,35.18000000000001 260,35.0 335,35.0" fill="none" stroke="#ac3b29" stroke-width="2"/><polyline points="35,214.82 50,107.9 65,36.25999999999999 80,35.53999999999999 95,35.360000000000014 110,35.360000000000014 125,35.18000000000001 155,35.53999999999999 185,35.360000000000014 215,35.360000000000014 260,35.18000000000001 335,35.360000000000014" fill="none" stroke="#307449" stroke-width="2"/></svg></figure><figure><figcaption>jev</figcaption><svg viewBox="0 0 360 245" role="img" aria-label="jev mean probability of yes by count"><path d="M35 30V215H340" fill="none" stroke="#555"/><path d="M35 125H340" stroke="#999" stroke-dasharray="3 3"/><text x="10" y="40">1</text><text x="4" y="129">.5</text><text x="10" y="218">0</text><text x="35" y="238">0</text><text x="320" y="238">20</text><polyline points="35,179.9 50,138.14 65,81.44 80,77.47999999999999 95,76.57999999999998 110,74.78 125,73.16 155,71.35999999999999 185,71.17999999999998 215,69.01999999999998 260,68.48000000000002 335,69.01999999999998" fill="none" stroke="#216eaa" stroke-width="2"/><polyline points="35,183.68 50,133.64 65,122.66 80,118.52 95,118.33999999999999 110,115.46 125,116.17999999999999 155,114.19999999999999 185,113.30000000000001 215,113.48 260,112.04 335,111.50000000000001" fill="none" stroke="#ac3b29" stroke-width="2"/><polyline points="35,180.8 50,141.92000000000002 65,89.9 80,84.5 95,83.24000000000001 110,81.25999999999999 125,80.18 155,77.47999999999999 185,77.12 215,76.22 260,75.68 335,74.78" fill="none" stroke="#307449" stroke-width="2"/></svg></figure><figure><figcaption>jevk5</figcaption><svg viewBox="0 0 360 245" role="img" aria-label="jevk5 mean probability of yes by count"><path d="M35 30V215H340" fill="none" stroke="#555"/><path d="M35 125H340" stroke="#999" stroke-dasharray="3 3"/><text x="10" y="40">1</text><text x="4" y="129">.5</text><text x="10" y="218">0</text><text x="35" y="238">0</text><text x="320" y="238">20</text><polyline points="35,183.14 50,154.88 65,106.64 80,106.10000000000001 95,104.12 110,108.80000000000001 125,105.92 155,103.04 185,105.56 215,103.76 260,102.86 335,103.76" fill="none" stroke="#216eaa" stroke-width="2"/><polyline points="35,176.48000000000002 50,149.12 65,147.32 80,148.04000000000002 95,148.22 110,148.22 125,147.68 155,146.95999999999998 185,146.24 215,146.6 260,144.98000000000002 335,144.26" fill="none" stroke="#ac3b29" stroke-width="2"/><polyline points="35,181.88 50,153.62 65,133.64 80,132.38 95,131.66 110,134.18 125,132.56 155,130.76 185,130.94 215,130.4 260,128.24 335,129.86" fill="none" stroke="#307449" stroke-width="2"/></svg></figure><figure><figcaption>kev-4b</figcaption><svg viewBox="0 0 360 245" role="img" aria-label="kev-4b mean probability of yes by count"><path d="M35 30V215H340" fill="none" stroke="#555"/><path d="M35 125H340" stroke="#999" stroke-dasharray="3 3"/><text x="10" y="40">1</text><text x="4" y="129">.5</text><text x="10" y="218">0</text><text x="35" y="238">0</text><text x="320" y="238">20</text><polyline points="35,197.54 50,150.2 65,110.42 80,107.72 95,105.92 110,106.64 125,107.18 155,105.92 185,106.28 215,108.08 260,108.26 335,108.26" fill="none" stroke="#216eaa" stroke-width="2"/><polyline points="35,190.52 50,155.06 65,150.2 80,151.82 95,151.28 110,151.46 125,152.72 155,151.82 185,150.56 215,152.0 260,151.1 335,150.92000000000002" fill="none" stroke="#ac3b29" stroke-width="2"/><polyline points="35,195.02 50,148.76 65,122.3 80,117.61999999999999 95,113.48 110,114.19999999999999 125,118.33999999999999 155,117.61999999999999 185,116.89999999999999 215,119.6 260,118.33999999999999 335,120.67999999999999" fill="none" stroke="#307449" stroke-width="2"/></svg></figure><figure><figcaption>laya</figcaption><svg viewBox="0 0 360 245" role="img" aria-label="laya mean probability of yes by count"><path d="M35 30V215H340" fill="none" stroke="#555"/><path d="M35 125H340" stroke="#999" stroke-dasharray="3 3"/><text x="10" y="40">1</text><text x="4" y="129">.5</text><text x="10" y="218">0</text><text x="35" y="238">0</text><text x="320" y="238">20</text><polyline points="35,79.1 50,75.85999999999999 65,75.68 80,76.22 95,75.68 110,76.22 125,76.75999999999999 155,76.4 185,76.22 215,76.22 260,75.68 335,73.88" fill="none" stroke="#216eaa" stroke-width="2"/><polyline points="35,104.3 50,104.12 65,100.7 80,101.96 95,101.24 110,100.7 125,101.24 155,100.34 185,99.25999999999999 215,100.34 260,99.44 335,99.62" fill="none" stroke="#ac3b29" stroke-width="2"/><polyline points="35,94.03999999999999 50,96.02 65,93.67999999999999 80,94.94 95,94.75999999999999 110,94.75999999999999 125,94.39999999999999 155,94.03999999999999 185,92.77999999999999 215,93.32 260,92.6 335,91.69999999999999" fill="none" stroke="#307449" stroke-width="2"/></svg></figure><figure><figcaption>nimble-9b</figcaption><svg viewBox="0 0 360 245" role="img" aria-label="nimble-9b mean probability of yes by count"><path d="M35 30V215H340" fill="none" stroke="#555"/><path d="M35 125H340" stroke="#999" stroke-dasharray="3 3"/><text x="10" y="40">1</text><text x="4" y="129">.5</text><text x="10" y="218">0</text><text x="35" y="238">0</text><text x="320" y="238">20</text><polyline points="35,204.2 50,161.0 65,58.58000000000001 80,51.91999999999999 95,50.84 110,49.03999999999999 125,48.859999999999985 155,49.22 185,47.78 215,50.66 260,49.22 335,47.95999999999998" fill="none" stroke="#216eaa" stroke-width="2"/><polyline points="35,190.34 50,148.04000000000002 65,140.66000000000003 80,140.12 95,141.2 110,140.84 125,137.96 155,134.18 185,134.72 215,131.48 260,131.48 335,130.76" fill="none" stroke="#ac3b29" stroke-width="2"/><polyline points="35,202.22 50,171.62 65,89.18 80,78.38 95,73.51999999999998 110,66.68 125,66.32000000000002 155,67.94 185,65.42000000000002 215,72.07999999999998 260,70.28 335,67.22" fill="none" stroke="#307449" stroke-width="2"/></svg></figure><figure><figcaption>semif</figcaption><svg viewBox="0 0 360 245" role="img" aria-label="semif mean probability of yes by count"><path d="M35 30V215H340" fill="none" stroke="#555"/><path d="M35 125H340" stroke="#999" stroke-dasharray="3 3"/><text x="10" y="40">1</text><text x="4" y="129">.5</text><text x="10" y="218">0</text><text x="35" y="238">0</text><text x="320" y="238">20</text><polyline points="35,123.38 50,92.24 65,73.88 80,72.44 95,69.56 110,69.91999999999999 125,71.0 155,71.53999999999999 185,70.1 215,72.44 260,71.35999999999999 335,69.01999999999998" fill="none" stroke="#216eaa" stroke-width="2"/><polyline points="35,157.57999999999998 50,119.24 65,118.52 80,117.25999999999999 95,115.82 110,114.55999999999999 125,114.91999999999999 155,114.19999999999999 185,113.48 215,113.12 260,111.68 335,112.04" fill="none" stroke="#ac3b29" stroke-width="2"/><polyline points="35,126.08 50,92.6 65,83.24000000000001 80,81.08000000000001 95,79.46000000000001 110,79.63999999999999 125,79.63999999999999 155,79.82 185,78.74000000000001 215,80.18 260,79.46000000000001 335,80.18" fill="none" stroke="#307449" stroke-width="2"/></svg></figure><figure><figcaption>strands-decider-2b</figcaption><svg viewBox="0 0 360 245" role="img" aria-label="strands-decider-2b mean probability of yes by count"><path d="M35 30V215H340" fill="none" stroke="#555"/><path d="M35 125H340" stroke="#999" stroke-dasharray="3 3"/><text x="10" y="40">1</text><text x="4" y="129">.5</text><text x="10" y="218">0</text><text x="35" y="238">0</text><text x="320" y="238">20</text><polyline points="35,124.1 50,124.64 65,115.99999999999999 80,115.99999999999999 95,115.63999999999999 110,115.27999999999999 125,115.46 155,115.63999999999999 185,114.91999999999999 215,114.74 260,114.55999999999999 335,113.83999999999999" fill="none" stroke="#216eaa" stroke-width="2"/><polyline points="35,121.94 50,123.92 65,121.94 80,122.12 95,122.3 110,122.48 125,122.12 155,122.3 185,122.84 215,122.3 260,122.48 335,122.66" fill="none" stroke="#ac3b29" stroke-width="2"/><polyline points="35,127.88 50,131.3 65,128.24 80,127.88 95,127.88 110,127.7 125,127.52 155,127.52 185,128.06 215,127.7 260,127.34 335,127.52" fill="none" stroke="#307449" stroke-width="2"/></svg></figure><figure><figcaption>verdict</figcaption><svg viewBox="0 0 360 245" role="img" aria-label="verdict mean probability of yes by count"><path d="M35 30V215H340" fill="none" stroke="#555"/><path d="M35 125H340" stroke="#999" stroke-dasharray="3 3"/><text x="10" y="40">1</text><text x="4" y="129">.5</text><text x="10" y="218">0</text><text x="35" y="238">0</text><text x="320" y="238">20</text><polyline points="35,98.36 50,95.66 65,95.83999999999999 80,95.83999999999999 95,95.83999999999999 110,96.02 125,96.02 155,96.02 185,96.02 215,95.83999999999999 260,96.02 335,96.02" fill="none" stroke="#216eaa" stroke-width="2"/><polyline points="35,97.28 50,96.55999999999999 65,95.3 80,95.3 95,95.3 110,95.3 125,95.47999999999999 155,95.47999999999999 185,95.3 215,95.3 260,95.3 335,95.3" fill="none" stroke="#ac3b29" stroke-width="2"/><polyline points="35,98.0 50,95.66 65,94.75999999999999 80,94.75999999999999 95,94.94 110,94.94 125,95.11999999999999 155,94.94 185,94.75999999999999 215,94.75999999999999 260,94.75999999999999 335,94.75999999999999" fill="none" stroke="#307449" stroke-width="2"/></svg></figure><figure><figcaption>von</figcaption><svg viewBox="0 0 360 245" role="img" aria-label="von mean probability of yes by count"><path d="M35 30V215H340" fill="none" stroke="#555"/><path d="M35 125H340" stroke="#999" stroke-dasharray="3 3"/><text x="10" y="40">1</text><text x="4" y="129">.5</text><text x="10" y="218">0</text><text x="35" y="238">0</text><text x="320" y="238">20</text><polyline points="35,146.78 50,129.86 65,134.0 80,137.6 95,136.88 110,137.24 125,138.14 155,137.06 185,134.9 215,137.24 260,135.07999999999998 335,134.18" fill="none" stroke="#216eaa" stroke-width="2"/><polyline points="35,150.74 50,149.48000000000002 65,148.57999999999998 80,148.4 95,148.4 110,148.4 125,148.4 155,148.22 185,148.22 215,148.4 260,148.4 335,148.22" fill="none" stroke="#ac3b29" stroke-width="2"/><polyline points="35,146.95999999999998 50,135.07999999999998 65,135.07999999999998 80,136.16 95,135.98000000000002 110,135.98000000000002 125,136.16 155,135.62 185,134.0 215,135.26 260,134.9 335,134.0" fill="none" stroke="#307449" stroke-width="2"/></svg></figure><figure><figcaption>winnow-12b</figcaption><svg viewBox="0 0 360 245" role="img" aria-label="winnow-12b mean probability of yes by count"><path d="M35 30V215H340" fill="none" stroke="#555"/><path d="M35 125H340" stroke="#999" stroke-dasharray="3 3"/><text x="10" y="40">1</text><text x="4" y="129">.5</text><text x="10" y="218">0</text><text x="35" y="238">0</text><text x="320" y="238">20</text><polyline points="35,212.48 50,172.88 65,38.599999999999994 80,38.420000000000016 95,38.24000000000001 110,38.420000000000016 125,37.879999999999995 155,37.69999999999999 185,37.69999999999999 215,37.52000000000001 260,37.52000000000001 335,37.34" fill="none" stroke="#216eaa" stroke-width="2"/><polyline points="35,211.58 50,135.98000000000002 65,94.22 80,82.69999999999999 95,74.78 110,71.0 125,69.38 155,66.86000000000001 185,65.06 215,61.46000000000001 260,60.379999999999995 335,58.58000000000001" fill="none" stroke="#ac3b29" stroke-width="2"/><polyline points="35,212.84 50,173.6 65,38.599999999999994 80,37.879999999999995 95,37.879999999999995 110,37.69999999999999 125,37.52000000000001 155,37.34 185,37.34 215,37.16 260,37.16 335,37.34" fill="none" stroke="#307449" stroke-width="2"/></svg></figure></div>
 </body>
 </html>

29 lines 18 KiB HTML Raw Permalink Blame History

29 lines

18 KiB

HTML

Raw Permalink Blame History