Jose Romero

← All benchmarks← Back to the post

Qwen 3.8 Flash-Next on llama.cpp: 18 coding runs

Agentic benchmark results for local coding models · suite v0.2.4 · 18 runs · rendered 2026-09-16 15:48
how to read this page
17/18
runs resolved, 94%
100/100
mean score
0/0
pass^n cells
11.1 h
agent time
1
model
3
harnesses

The model and how it ran 2026-09-16-v024-flashnext-iq4xs-think-xhigh-3h

modelQwen3.8-Flash-Next-UD-IQ4_XS-00001-of-00003.gguf
labelQwen3.8-Flash-Next UD-IQ4_XS (93.7 GB, the top quant that fits 128 GB), thinking xhigh (the template default), stock llama.cpp master (Aug 28, Vulkan, RADV), flash attention on, ctx 131072, samplers pinned server-side to the model card thinking set 1.0/0.95/20/0, inline think blocks
serverb1-ca3d5a3
context window131072
samplingtemperature=1.0, top_p=0.95, top_k=20, min_p=0.0, repeat_penalty=1.0
thinkingxhigh (template default); every earlier harness campaign ran medium at temp 0.6, so compare with care
timeouts120 min cap on every task (campaign override): xhigh thinks several times longer than medium; n=1
harnessespi 0.73.1 · dsh 0.1.1-rc.2 · opencode 1.18.25
runs18 runs, n=1 per (harness, task), 2026-09-16T03:50:17 to 2026-09-16T13:23:03
platformLinux-7.2.5-1-cachyos-x86_64-with-glibc2.44
13.9
gen tok/s, mean per run
9.7 to 18.2
145
prefill tok/s, mean per run
123 to 158
96.8k
peak context, tokens
42.5k mean
406k
tokens generated
10.2 tok/s over wall time
37 min
wall per run, mean
4 to 100 min
13 GB
lowest available memory
mean 18 GB, swap up to 10 GB
think tokens counted in 12 runs from the harness session store and in the rest from stdout; mean ~7590 per run

Tasks resolved per task, click a card

3/3 resolved
mean 100/100, 6.9 min avg, pass^n yes
3/3 resolved
mean 100/100, 63.0 min avg, pass^n yes
3/3 resolved
mean 100/100, 35.9 min avg, pass^n yes
2/3 resolved
mean 98/100, 28.0 min avg, pass^n no
3/3 resolved
mean 100/100, 9.3 min avg, pass^n yes
3/3 resolved
mean 100/100, 78.4 min avg, pass^n yes

Run map every run at a glance

100 resolved 55 partial 0 nothing earned ? not scored · one square per run, hover for details, click to jump
t1-cold-fix · dsh
1/1
t2-logsift · dsh
1/1
t3-shoebox · dsh
1/1
t4-night-shift · dsh
1/1
t5-regression · dsh
1/1
t6-payroll · dsh
1/1
t1-cold-fix · opencode
1/1
t2-logsift · opencode
1/1
t3-shoebox · opencode
1/1
t4-night-shift · opencode
1/1
t5-regression · opencode
1/1
t6-payroll · opencode
1/1
t1-cold-fix · pi
1/1
t2-logsift · pi
1/1
t3-shoebox · pi
1/1
t4-night-shift · pi
0/1
t5-regression · pi
1/1
t6-payroll · pi
1/1

Speed, context and memory from the model server log, one dot per request

Generation speed against context size, every request
dshopencodepi
0102030020k40k60k80k100kcontext size at the request, tokensgeneration, tok/sdsh / t1-cold-fix: 50 tokens at 24.8 tok/s, context 117dsh / t1-cold-fix: 64 tokens at 24.9 tok/s, context 342dsh / t1-cold-fix: 118 tokens at 18.9 tok/s, context 7890dsh / t1-cold-fix: 175 tokens at 18.7 tok/s, context 8325dsh / t1-cold-fix: 99 tokens at 18.5 tok/s, context 8774dsh / t1-cold-fix: 341 tokens at 18.1 tok/s, context 9707dsh / t1-cold-fix: 233 tokens at 17.6 tok/s, context 12113dsh / t1-cold-fix: 2168 tokens at 14.3 tok/s, context 16749dsh / t1-cold-fix: 1132 tokens at 15.0 tok/s, context 18014dsh / t1-cold-fix: 116 tokens at 16.6 tok/s, context 18213dsh / t1-cold-fix: 196 tokens at 16.8 tok/s, context 18448dsh / t1-cold-fix: 557 tokens at 16.0 tok/s, context 19477dsh / t1-cold-fix: 50 tokens at 24.9 tok/s, context 117dsh / t2-logsift: 50 tokens at 24.9 tok/s, context 117dsh / t2-logsift: 158 tokens at 18.5 tok/s, context 9212dsh / t2-logsift: 112 tokens at 18.0 tok/s, context 10813dsh / t2-logsift: 11719 tokens at 13.4 tok/s, context 23823dsh / t2-logsift: 438 tokens at 15.5 tok/s, context 24322dsh / t2-logsift: 140 tokens at 15.5 tok/s, context 24507dsh / t2-logsift: 187 tokens at 15.5 tok/s, context 24739dsh / t2-logsift: 227 tokens at 15.4 tok/s, context 25011dsh / t2-logsift: 508 tokens at 15.1 tok/s, context 26267dsh / t2-logsift: 10877 tokens at 11.2 tok/s, context 37228dsh / t2-logsift: 1186 tokens at 11.5 tok/s, context 38469dsh / t2-logsift: 159 tokens at 13.9 tok/s, context 38696dsh / t2-logsift: 291 tokens at 13.7 tok/s, context 39200dsh / t2-logsift: 265 tokens at 13.5 tok/s, context 39815dsh / t2-logsift: 10740 tokens at 9.5 tok/s, context 50871dsh / t2-logsift: 424 tokens at 11.5 tok/s, context 51358dsh / t2-logsift: 165 tokens at 11.6 tok/s, context 51656dsh / t2-logsift: 169 tokens at 11.6 tok/s, context 51872dsh / t2-logsift: 1385 tokens at 9.3 tok/s, context 54282dsh / t2-logsift: 809 tokens at 9.5 tok/s, context 55284dsh / t2-logsift: 192 tokens at 11.0 tok/s, context 55497dsh / t2-logsift: 3042 tokens at 9.0 tok/s, context 58568dsh / t2-logsift: 1960 tokens at 9.2 tok/s, context 61169dsh / t2-logsift: 381 tokens at 10.4 tok/s, context 61595dsh / t2-logsift: 640 tokens at 9.3 tok/s, context 62280dsh / t2-logsift: 283 tokens at 10.2 tok/s, context 62608dsh / t2-logsift: 979 tokens at 9.0 tok/s, context 64019dsh / t2-logsift: 121 tokens at 10.1 tok/s, context 64179dsh / t2-logsift: 284 tokens at 9.8 tok/s, context 64595dsh / t2-logsift: 752 tokens at 8.8 tok/s, context 65550dsh / t2-logsift: 109 tokens at 10.2 tok/s, context 65716dsh / t2-logsift: 395 tokens at 9.0 tok/s, context 70343dsh / t2-logsift: 168 tokens at 9.3 tok/s, context 70556dsh / t2-logsift: 910 tokens at 8.4 tok/s, context 71495dsh / t2-logsift: 1103 tokens at 8.4 tok/s, context 73261dsh / t2-logsift: 670 tokens at 8.5 tok/s, context 74351dsh / t2-logsift: 2187 tokens at 8.1 tok/s, context 76765dsh / t2-logsift: 586 tokens at 8.4 tok/s, context 77658dsh / t2-logsift: 213 tokens at 9.1 tok/s, context 78274dsh / t2-logsift: 1183 tokens at 8.2 tok/s, context 79811dsh / t3-shoebox: 50 tokens at 25.1 tok/s, context 117dsh / t3-shoebox: 150 tokens at 18.5 tok/s, context 9007dsh / t3-shoebox: 266 tokens at 18.4 tok/s, context 9308dsh / t3-shoebox: 228 tokens at 18.2 tok/s, context 9693dsh / t3-shoebox: 1935 tokens at 15.2 tok/s, context 11708dsh / t3-shoebox: 1381 tokens at 15.2 tok/s, context 13602dsh / t3-shoebox: 966 tokens at 16.4 tok/s, context 14726dsh / t3-shoebox: 3720 tokens at 13.2 tok/s, context 19334dsh / t3-shoebox: 2213 tokens at 13.2 tok/s, context 23025dsh / t3-shoebox: 1262 tokens at 13.2 tok/s, context 24325dsh / t3-shoebox: 158 tokens at 15.0 tok/s, context 27733dsh / t3-shoebox: 4752 tokens at 10.7 tok/s, context 35546dsh / t3-shoebox: 2826 tokens at 10.6 tok/s, context 39324dsh / t3-shoebox: 1557 tokens at 10.3 tok/s, context 41181dsh / t3-shoebox: 1303 tokens at 10.4 tok/s, context 42642dsh / t3-shoebox: 872 tokens at 10.6 tok/s, context 43764dsh / t3-shoebox: 1083 tokens at 10.2 tok/s, context 45134dsh / t3-shoebox: 1170 tokens at 10.1 tok/s, context 46398dsh / t3-shoebox: 971 tokens at 10.2 tok/s, context 47488dsh / t3-shoebox: 2849 tokens at 9.4 tok/s, context 50583dsh / t3-shoebox: 1423 tokens at 9.8 tok/s, context 52286dsh / t3-shoebox: 493 tokens at 11.1 tok/s, context 53201dsh / t3-shoebox: 1013 tokens at 9.6 tok/s, context 54245dsh / t3-shoebox: 50 tokens at 24.7 tok/s, context 117dsh / t4-night-shift: 50 tokens at 24.7 tok/s, context 117dsh / t4-night-shift: 64 tokens at 22.5 tok/s, context 976dsh / t4-night-shift: 386 tokens at 18.3 tok/s, context 8775dsh / t4-night-shift: 505 tokens at 18.0 tok/s, context 9411dsh / t4-night-shift: 4808 tokens at 13.9 tok/s, context 16349dsh / t4-night-shift: 2714 tokens at 13.1 tok/s, context 19471dsh / t4-night-shift: 735 tokens at 15.7 tok/s, context 20651dsh / t4-night-shift: 127 tokens at 16.2 tok/s, context 20858dsh / t4-night-shift: 586 tokens at 16.0 tok/s, context 21710dsh / t4-night-shift: 4413 tokens at 12.2 tok/s, context 26286dsh / t4-night-shift: 1745 tokens at 12.4 tok/s, context 28463dsh / t4-night-shift: 1220 tokens at 13.2 tok/s, context 29855dsh / t4-night-shift: 676 tokens at 14.2 tok/s, context 30681dsh / t4-night-shift: 344 tokens at 14.5 tok/s, context 31319dsh / t4-night-shift: 752 tokens at 14.1 tok/s, context 32108dsh / t5-regression: 50 tokens at 25.8 tok/s, context 117dsh / t5-regression: 64 tokens at 24.7 tok/s, context 419dsh / t5-regression: 68 tokens at 18.9 tok/s, context 7911dsh / t5-regression: 69 tokens at 18.6 tok/s, context 8408dsh / t5-regression: 89 tokens at 18.3 tok/s, context 9084dsh / t5-regression: 546 tokens at 18.0 tok/s, context 9978dsh / t5-regression: 2566 tokens at 14.3 tok/s, context 13335dsh / t5-regression: 557 tokens at 16.8 tok/s, context 15564dsh / t5-regression: 344 tokens at 16.3 tok/s, context 17223dsh / t5-regression: 209 tokens at 15.9 tok/s, context 19695dsh / t5-regression: 1093 tokens at 14.6 tok/s, context 21799dsh / t5-regression: 821 tokens at 14.8 tok/s, context 23069dsh / t5-regression: 297 tokens at 15.4 tok/s, context 23416dsh / t5-regression: 322 tokens at 15.6 tok/s, context 23873dsh / t5-regression: 90 tokens at 16.0 tok/s, context 24015dsh / t5-regression: 239 tokens at 15.4 tok/s, context 24484dsh / t5-regression: 290 tokens at 15.3 tok/s, context 24819dsh / t5-regression: 184 tokens at 15.4 tok/s, context 25064dsh / t5-regression: 226 tokens at 15.5 tok/s, context 25415dsh / t5-regression: 792 tokens at 14.4 tok/s, context 26372dsh / t6-payroll: 50 tokens at 24.9 tok/s, context 117dsh / t6-payroll: 234 tokens at 18.2 tok/s, context 9908dsh / t6-payroll: 174 tokens at 17.9 tok/s, context 10457dsh / t6-payroll: 111 tokens at 17.8 tok/s, context 11273dsh / t6-payroll: 3470 tokens at 13.2 tok/s, context 25970dsh / t6-payroll: 3012 tokens at 12.4 tok/s, context 29124dsh / t6-payroll: 2778 tokens at 11.8 tok/s, context 31964dsh / t6-payroll: 161 tokens at 14.5 tok/s, context 32187dsh / t6-payroll: 2800 tokens at 11.5 tok/s, context 35985dsh / t6-payroll: 530 tokens at 13.3 tok/s, context 37391dsh / t6-payroll: 3995 tokens at 10.2 tok/s, context 41632dsh / t6-payroll: 1261 tokens at 10.8 tok/s, context 42954dsh / t6-payroll: 185 tokens at 13.2 tok/s, context 43225dsh / t6-payroll: 2082 tokens at 9.9 tok/s, context 45638dsh / t6-payroll: 1024 tokens at 10.5 tok/s, context 46725dsh / t6-payroll: 370 tokens at 12.0 tok/s, context 47142dsh / t6-payroll: 979 tokens at 10.1 tok/s, context 49642dsh / t6-payroll: 1001 tokens at 9.7 tok/s, context 51596dsh / t6-payroll: 287 tokens at 11.3 tok/s, context 51930dsh / t6-payroll: 2817 tokens at 9.2 tok/s, context 54768dsh / t6-payroll: 435 tokens at 10.4 tok/s, context 55848dsh / t6-payroll: 2746 tokens at 8.9 tok/s, context 59368dsh / t6-payroll: 3382 tokens at 8.6 tok/s, context 63640dsh / t6-payroll: 3761 tokens at 8.5 tok/s, context 68415dsh / t6-payroll: 320 tokens at 9.4 tok/s, context 68808dsh / t6-payroll: 296 tokens at 8.8 tok/s, context 69150dsh / t6-payroll: 348 tokens at 9.0 tok/s, context 69545dsh / t6-payroll: 1359 tokens at 8.3 tok/s, context 71236dsh / t6-payroll: 1451 tokens at 8.2 tok/s, context 73383dsh / t6-payroll: 1518 tokens at 8.0 tok/s, context 75981dsh / t6-payroll: 1779 tokens at 7.9 tok/s, context 79411dsh / t6-payroll: 968 tokens at 8.0 tok/s, context 81395dsh / t6-payroll: 468 tokens at 8.1 tok/s, context 81943dsh / t6-payroll: 623 tokens at 7.9 tok/s, context 82634dsh / t6-payroll: 286 tokens at 8.5 tok/s, context 82989dsh / t6-payroll: 1124 tokens at 7.8 tok/s, context 84443opencode / t1-cold-fix: 50 tokens at 26.1 tok/s, context 117opencode / t1-cold-fix: 102 tokens at 22.1 tok/s, context 806opencode / t1-cold-fix: 78 tokens at 18.8 tok/s, context 7326opencode / t1-cold-fix: 84 tokens at 15.4 tok/s, context 9094opencode / t1-cold-fix: 193 tokens at 15.9 tok/s, context 9536opencode / t1-cold-fix: 190 tokens at 17.6 tok/s, context 10795opencode / t1-cold-fix: 140 tokens at 17.3 tok/s, context 12238opencode / t1-cold-fix: 1179 tokens at 15.5 tok/s, context 15542opencode / t1-cold-fix: 2215 tokens at 14.0 tok/s, context 18183opencode / t1-cold-fix: 544 tokens at 15.8 tok/s, context 19322opencode / t1-cold-fix: 101 tokens at 16.2 tok/s, context 19455opencode / t1-cold-fix: 121 tokens at 16.3 tok/s, context 19615opencode / t1-cold-fix: 160 tokens at 16.3 tok/s, context 19859opencode / t1-cold-fix: 115 tokens at 16.1 tok/s, context 20005opencode / t1-cold-fix: 140 tokens at 15.5 tok/s, context 20184opencode / t2-logsift: 50 tokens at 25.0 tok/s, context 117opencode / t2-logsift: 121 tokens at 20.9 tok/s, context 2116opencode / t2-logsift: 62 tokens at 18.4 tok/s, context 8605opencode / t2-logsift: 53 tokens at 18.5 tok/s, context 8806opencode / t2-logsift: 6829 tokens at 14.7 tok/s, context 16903opencode / t2-logsift: 282 tokens at 16.3 tok/s, context 17209opencode / t2-logsift: 124 tokens at 16.3 tok/s, context 17356opencode / t2-logsift: 241 tokens at 16.2 tok/s, context 17620opencode / t2-logsift: 193 tokens at 16.0 tok/s, context 17836opencode / t2-logsift: 444 tokens at 15.8 tok/s, context 19097opencode / t2-logsift: 365 tokens at 15.8 tok/s, context 19485opencode / t2-logsift: 1852 tokens at 13.8 tok/s, context 22081opencode / t2-logsift: 1316 tokens at 14.1 tok/s, context 23530opencode / t2-logsift: 861 tokens at 14.9 tok/s, context 24577opencode / t2-logsift: 1301 tokens at 13.6 tok/s, context 26086opencode / t2-logsift: 763 tokens at 14.2 tok/s, context 29638opencode / t2-logsift: 190 tokens at 13.7 tok/s, context 29851opencode / t2-logsift: 272 tokens at 14.5 tok/s, context 30146opencode / t2-logsift: 497 tokens at 14.2 tok/s, context 30666opencode / t2-logsift: 93 tokens at 14.8 tok/s, context 30852opencode / t3-shoebox: 50 tokens at 25.1 tok/s, context 117opencode / t3-shoebox: 405 tokens at 20.6 tok/s, context 2204opencode / t3-shoebox: 74 tokens at 18.6 tok/s, context 8419opencode / t3-shoebox: 117 tokens at 18.4 tok/s, context 8663opencode / t3-shoebox: 73 tokens at 17.2 tok/s, context 13992opencode / t3-shoebox: 3460 tokens at 14.0 tok/s, context 20325opencode / t3-shoebox: 1721 tokens at 14.1 tok/s, context 22130opencode / t3-shoebox: 512 tokens at 15.6 tok/s, context 22666opencode / t3-shoebox: 473 tokens at 15.4 tok/s, context 23728opencode / t3-shoebox: 591 tokens at 14.9 tok/s, context 24342opencode / t3-shoebox: 132 tokens at 15.5 tok/s, context 24497opencode / t3-shoebox: 214 tokens at 15.2 tok/s, context 25366opencode / t3-shoebox: 161 tokens at 15.4 tok/s, context 25550opencode / t3-shoebox: 210 tokens at 15.0 tok/s, context 26731opencode / t3-shoebox: 475 tokens at 14.7 tok/s, context 27229opencode / t3-shoebox: 108 tokens at 14.9 tok/s, context 27789opencode / t3-shoebox: 970 tokens at 13.5 tok/s, context 29148opencode / t3-shoebox: 1311 tokens at 13.3 tok/s, context 30941opencode / t3-shoebox: 1578 tokens at 12.8 tok/s, context 33137opencode / t3-shoebox: 861 tokens at 13.0 tok/s, context 34112opencode / t3-shoebox: 802 tokens at 12.8 tok/s, context 35621opencode / t3-shoebox: 298 tokens at 13.3 tok/s, context 36570opencode / t3-shoebox: 1454 tokens at 12.0 tok/s, context 38229opencode / t3-shoebox: 725 tokens at 12.1 tok/s, context 39152opencode / t3-shoebox: 2713 tokens at 11.0 tok/s, context 42625opencode / t3-shoebox: 1400 tokens at 10.7 tok/s, context 45878opencode / t3-shoebox: 588 tokens at 11.4 tok/s, context 46550opencode / t3-shoebox: 595 tokens at 11.3 tok/s, context 47306opencode / t3-shoebox: 171 tokens at 12.0 tok/s, context 47683opencode / t3-shoebox: 430 tokens at 11.4 tok/s, context 48685opencode / t3-shoebox: 673 tokens at 10.4 tok/s, context 49381opencode / t3-shoebox: 155 tokens at 11.7 tok/s, context 49585opencode / t3-shoebox: 389 tokens at 11.1 tok/s, context 50118opencode / t3-shoebox: 50 tokens at 24.8 tok/s, context 117opencode / t4-night-shift: 50 tokens at 24.8 tok/s, context 117opencode / t4-night-shift: 108 tokens at 21.5 tok/s, context 1436opencode / t4-night-shift: 100 tokens at 18.5 tok/s, context 7972opencode / t4-night-shift: 369 tokens at 18.2 tok/s, context 8667opencode / t4-night-shift: 144 tokens at 17.8 tok/s, context 10461opencode / t4-night-shift: 5317 tokens at 15.1 tok/s, context 16283opencode / t4-night-shift: 353 tokens at 16.4 tok/s, context 16680opencode / t4-night-shift: 136 tokens at 15.8 tok/s, context 17268opencode / t4-night-shift: 502 tokens at 16.1 tok/s, context 17794opencode / t4-night-shift: 344 tokens at 16.2 tok/s, context 18162opencode / t4-night-shift: 432 tokens at 16.1 tok/s, context 18626opencode / t4-night-shift: 407 tokens at 15.6 tok/s, context 19057opencode / t4-night-shift: 476 tokens at 15.9 tok/s, context 19557opencode / t4-night-shift: 755 tokens at 14.9 tok/s, context 20346opencode / t4-night-shift: 123 tokens at 15.9 tok/s, context 20527opencode / t4-night-shift: 371 tokens at 15.8 tok/s, context 21170opencode / t4-night-shift: 431 tokens at 15.6 tok/s, context 21657opencode / t4-night-shift: 729 tokens at 14.6 tok/s, context 22549opencode / t4-night-shift: 294 tokens at 15.6 tok/s, context 22865opencode / t4-night-shift: 477 tokens at 15.4 tok/s, context 23389opencode / t4-night-shift: 621 tokens at 14.8 tok/s, context 24148opencode / t4-night-shift: 775 tokens at 14.7 tok/s, context 25785opencode / t4-night-shift: 433 tokens at 14.8 tok/s, context 26324opencode / t4-night-shift: 603 tokens at 14.7 tok/s, context 27296opencode / t4-night-shift: 160 tokens at 14.9 tok/s, context 27741opencode / t4-night-shift: 407 tokens at 14.4 tok/s, context 28246opencode / t4-night-shift: 357 tokens at 14.7 tok/s, context 29054opencode / t5-regression: 50 tokens at 25.8 tok/s, context 117opencode / t5-regression: 232 tokens at 21.9 tok/s, context 1012opencode / t5-regression: 84 tokens at 18.6 tok/s, context 7408opencode / t5-regression: 70 tokens at 17.8 tok/s, context 8931opencode / t5-regression: 807 tokens at 17.5 tok/s, context 10124opencode / t5-regression: 278 tokens at 17.6 tok/s, context 11096opencode / t5-regression: 392 tokens at 17.3 tok/s, context 12150opencode / t5-regression: 235 tokens at 17.3 tok/s, context 12442opencode / t5-regression: 310 tokens at 16.9 tok/s, context 12810opencode / t5-regression: 258 tokens at 17.0 tok/s, context 13091opencode / t5-regression: 70 tokens at 17.3 tok/s, context 13185opencode / t5-regression: 169 tokens at 17.0 tok/s, context 13403opencode / t5-regression: 50 tokens at 25.1 tok/s, context 117opencode / t6-payroll: 50 tokens at 25.1 tok/s, context 117opencode / t6-payroll: 203 tokens at 20.3 tok/s, context 2809opencode / t6-payroll: 53 tokens at 18.2 tok/s, context 9201opencode / t6-payroll: 73 tokens at 18.1 tok/s, context 9608opencode / t6-payroll: 58 tokens at 17.8 tok/s, context 10009opencode / t6-payroll: 70 tokens at 17.9 tok/s, context 10190opencode / t6-payroll: 49 tokens at 17.9 tok/s, context 10512opencode / t6-payroll: 79 tokens at 17.8 tok/s, context 10618opencode / t6-payroll: 20793 tokens at 12.0 tok/s, context 42636opencode / t6-payroll: 292 tokens at 12.5 tok/s, context 42952opencode / t6-payroll: 882 tokens at 11.3 tok/s, context 44831opencode / t6-payroll: 2794 tokens at 10.4 tok/s, context 48212opencode / t6-payroll: 4857 tokens at 8.5 tok/s, context 79779opencode / t6-payroll: 3781 tokens at 8.4 tok/s, context 83976opencode / t6-payroll: 114 tokens at 8.8 tok/s, context 84113opencode / t6-payroll: 767 tokens at 8.3 tok/s, context 86114opencode / t6-payroll: 150 tokens at 8.5 tok/s, context 86287opencode / t6-payroll: 9208 tokens at 7.9 tok/s, context 96493opencode / t6-payroll: 224 tokens at 8.1 tok/s, context 96796pi / t1-cold-fix: 50 tokens at 25.0 tok/s, context 117pi / t1-cold-fix: 107 tokens at 21.7 tok/s, context 1767pi / t1-cold-fix: 214 tokens at 21.0 tok/s, context 2291pi / t1-cold-fix: 1255 tokens at 18.0 tok/s, context 8137pi / t1-cold-fix: 784 tokens at 17.9 tok/s, context 9847pi / t1-cold-fix: 92 tokens at 18.4 tok/s, context 10028pi / t1-cold-fix: 142 tokens at 18.5 tok/s, context 10209pi / t1-cold-fix: 175 tokens at 18.3 tok/s, context 11050pi / t1-cold-fix: 350 tokens at 18.3 tok/s, context 11465pi / t1-cold-fix: 190 tokens at 18.1 tok/s, context 11986pi / t1-cold-fix: 570 tokens at 17.0 tok/s, context 12608pi / t2-logsift: 50 tokens at 25.0 tok/s, context 117pi / t2-logsift: 62 tokens at 20.5 tok/s, context 3005pi / t2-logsift: 172 tokens at 19.9 tok/s, context 4379pi / t2-logsift: 7308 tokens at 16.4 tok/s, context 12124pi / t2-logsift: 1470 tokens at 15.5 tok/s, context 13642pi / t2-logsift: 188 tokens at 17.3 tok/s, context 13878pi / t2-logsift: 123 tokens at 17.6 tok/s, context 14049pi / t2-logsift: 4996 tokens at 13.2 tok/s, context 19819pi / t2-logsift: 530 tokens at 16.2 tok/s, context 20397pi / t2-logsift: 388 tokens at 16.1 tok/s, context 20833pi / t2-logsift: 56 tokens at 16.3 tok/s, context 20937pi / t2-logsift: 2217 tokens at 12.9 tok/s, context 23630pi / t2-logsift: 403 tokens at 15.7 tok/s, context 24502pi / t2-logsift: 373 tokens at 15.4 tok/s, context 25793pi / t2-logsift: 505 tokens at 15.1 tok/s, context 26838pi / t2-logsift: 884 tokens at 14.6 tok/s, context 27770pi / t2-logsift: 425 tokens at 14.9 tok/s, context 28305pi / t2-logsift: 1869 tokens at 12.8 tok/s, context 30205pi / t2-logsift: 144 tokens at 14.8 tok/s, context 30460pi / t2-logsift: 132 tokens at 14.9 tok/s, context 30620pi / t2-logsift: 1526 tokens at 12.2 tok/s, context 34655pi / t2-logsift: 132 tokens at 13.8 tok/s, context 34835pi / t2-logsift: 906 tokens at 12.5 tok/s, context 36460pi / t2-logsift: 183 tokens at 13.9 tok/s, context 36691pi / t2-logsift: 379 tokens at 13.9 tok/s, context 37129pi / t2-logsift: 653 tokens at 12.2 tok/s, context 38118pi / t2-logsift: 453 tokens at 13.4 tok/s, context 38847pi / t2-logsift: 338 tokens at 13.3 tok/s, context 39391pi / t2-logsift: 169 tokens at 13.4 tok/s, context 39999pi / t2-logsift: 914 tokens at 11.3 tok/s, context 41558pi / t2-logsift: 300 tokens at 13.0 tok/s, context 41906pi / t2-logsift: 113 tokens at 12.4 tok/s, context 48194pi / t2-logsift: 166 tokens at 12.2 tok/s, context 49551pi / t2-logsift: 391 tokens at 11.4 tok/s, context 50283pi / t2-logsift: 156 tokens at 12.1 tok/s, context 50579pi / t2-logsift: 2567 tokens at 9.4 tok/s, context 53202pi / t2-logsift: 921 tokens at 9.9 tok/s, context 54154pi / t2-logsift: 1449 tokens at 9.4 tok/s, context 55771pi / t2-logsift: 1839 tokens at 9.1 tok/s, context 57778pi / t2-logsift: 86 tokens at 11.3 tok/s, context 57897pi / t2-logsift: 174 tokens at 11.1 tok/s, context 58302pi / t2-logsift: 174 tokens at 11.0 tok/s, context 58682pi / t2-logsift: 2116 tokens at 8.8 tok/s, context 60865pi / t2-logsift: 104 tokens at 10.6 tok/s, context 61002pi / t2-logsift: 540 tokens at 9.5 tok/s, context 61947pi / t2-logsift: 157 tokens at 10.9 tok/s, context 62128pi / t2-logsift: 821 tokens at 9.1 tok/s, context 63697pi / t2-logsift: 1694 tokens at 13.2 tok/s, context 31736pi / t3-shoebox: 50 tokens at 25.1 tok/s, context 117pi / t3-shoebox: 177 tokens at 20.7 tok/s, context 2920pi / t3-shoebox: 159 tokens at 20.4 tok/s, context 3477pi / t3-shoebox: 74 tokens at 19.8 tok/s, context 4833pi / t3-shoebox: 201 tokens at 18.9 tok/s, context 7049pi / t3-shoebox: 92 tokens at 18.6 tok/s, context 8213pi / t3-shoebox: 204 tokens at 18.0 tok/s, context 10648pi / t3-shoebox: 2537 tokens at 14.8 tok/s, context 16271pi / t3-shoebox: 1850 tokens at 14.6 tok/s, context 18335pi / t3-shoebox: 255 tokens at 16.6 tok/s, context 18639pi / t3-shoebox: 991 tokens at 14.6 tok/s, context 20027pi / t3-shoebox: 439 tokens at 16.0 tok/s, context 21230pi / t3-shoebox: 874 tokens at 15.4 tok/s, context 22423pi / t3-shoebox: 2356 tokens at 13.8 tok/s, context 25040pi / t3-shoebox: 1290 tokens at 14.5 tok/s, context 26406pi / t3-shoebox: 105 tokens at 15.9 tok/s, context 26552pi / t3-shoebox: 766 tokens at 15.0 tok/s, context 27707pi / t3-shoebox: 50 tokens at 24.6 tok/s, context 117pi / t4-night-shift: 50 tokens at 24.6 tok/s, context 117pi / t4-night-shift: 122 tokens at 20.9 tok/s, context 2401pi / t4-night-shift: 230 tokens at 20.5 tok/s, context 3175pi / t4-night-shift: 292 tokens at 20.1 tok/s, context 4041pi / t4-night-shift: 172 tokens at 19.5 tok/s, context 4592pi / t4-night-shift: 1240 tokens at 18.8 tok/s, context 6107pi / t4-night-shift: 279 tokens at 19.1 tok/s, context 6460pi / t4-night-shift: 6636 tokens at 15.8 tok/s, context 13185pi / t4-night-shift: 436 tokens at 17.4 tok/s, context 13705pi / t4-night-shift: 594 tokens at 17.1 tok/s, context 14383pi / t4-night-shift: 721 tokens at 16.5 tok/s, context 15153pi / t4-night-shift: 337 tokens at 17.2 tok/s, context 15539pi / t4-night-shift: 984 tokens at 15.3 tok/s, context 16601pi / t4-night-shift: 780 tokens at 16.1 tok/s, context 17429pi / t4-night-shift: 304 tokens at 16.4 tok/s, context 17779pi / t4-night-shift: 448 tokens at 16.6 tok/s, context 18275pi / t4-night-shift: 2685 tokens at 13.7 tok/s, context 21214pi / t4-night-shift: 931 tokens at 15.4 tok/s, context 22229pi / t4-night-shift: 2262 tokens at 12.8 tok/s, context 24539pi / t4-night-shift: 1449 tokens at 12.7 tok/s, context 26036pi / t4-night-shift: 278 tokens at 11.1 tok/s, context 26362pi / t4-night-shift: 189 tokens at 15.5 tok/s, context 26869pi / t4-night-shift: 259 tokens at 15.5 tok/s, context 27191pi / t4-night-shift: 316 tokens at 15.0 tok/s, context 27777pi / t4-night-shift: 185 tokens at 15.1 tok/s, context 28010pi / t4-night-shift: 209 tokens at 15.2 tok/s, context 28259pi / t4-night-shift: 617 tokens at 15.0 tok/s, context 28954pi / t4-night-shift: 762 tokens at 14.8 tok/s, context 29889pi / t4-night-shift: 388 tokens at 14.5 tok/s, context 31211pi / t4-night-shift: 534 tokens at 14.5 tok/s, context 31798pi / t4-night-shift: 874 tokens at 13.3 tok/s, context 32827pi / t4-night-shift: 360 tokens at 14.6 tok/s, context 33235pi / t4-night-shift: 305 tokens at 14.3 tok/s, context 33971pi / t4-night-shift: 1070 tokens at 12.5 tok/s, context 37883pi / t4-night-shift: 823 tokens at 12.6 tok/s, context 38901pi / t5-regression: 50 tokens at 26.3 tok/s, context 117pi / t5-regression: 118 tokens at 21.4 tok/s, context 1850pi / t5-regression: 154 tokens at 21.0 tok/s, context 2527pi / t5-regression: 1863 tokens at 18.3 tok/s, context 6715pi / t5-regression: 834 tokens at 18.0 tok/s, context 9954pi / t5-regression: 249 tokens at 17.9 tok/s, context 10933pi / t5-regression: 554 tokens at 17.4 tok/s, context 13899pi / t5-regression: 316 tokens at 17.4 tok/s, context 14927pi / t5-regression: 913 tokens at 16.3 tok/s, context 16080pi / t5-regression: 144 tokens at 17.2 tok/s, context 16583pi / t5-regression: 528 tokens at 16.7 tok/s, context 17422pi / t5-regression: 122 tokens at 16.8 tok/s, context 17591pi / t5-regression: 181 tokens at 16.7 tok/s, context 18026pi / t5-regression: 246 tokens at 16.6 tok/s, context 18333pi / t5-regression: 182 tokens at 16.7 tok/s, context 18561pi / t5-regression: 281 tokens at 16.4 tok/s, context 19087pi / t5-regression: 110 tokens at 16.7 tok/s, context 19438pi / t5-regression: 934 tokens at 15.3 tok/s, context 20607pi / t5-regression: 293 tokens at 15.9 tok/s, context 21639pi / t5-regression: 148 tokens at 16.2 tok/s, context 21834pi / t5-regression: 599 tokens at 15.8 tok/s, context 22907pi / t6-payroll: 50 tokens at 24.9 tok/s, context 117pi / t6-payroll: 89 tokens at 20.4 tok/s, context 3647pi / t6-payroll: 113 tokens at 20.1 tok/s, context 4144pi / t6-payroll: 56 tokens at 20.0 tok/s, context 4678pi / t6-payroll: 8532 tokens at 14.1 tok/s, context 23685pi / t6-payroll: 531 tokens at 15.8 tok/s, context 24246pi / t6-payroll: 221 tokens at 15.7 tok/s, context 24497pi / t6-payroll: 157 tokens at 14.4 tok/s, context 24910pi / t6-payroll: 250 tokens at 15.5 tok/s, context 25190pi / t6-payroll: 1426 tokens at 13.0 tok/s, context 27614pi / t6-payroll: 693 tokens at 15.0 tok/s, context 29535pi / t6-payroll: 1678 tokens at 13.3 tok/s, context 31535pi / t6-payroll: 1359 tokens at 12.1 tok/s, context 32983pi / t6-payroll: 1439 tokens at 12.4 tok/s, context 34963pi / t6-payroll: 1041 tokens at 12.1 tok/s, context 36087pi / t6-payroll: 3594 tokens at 9.4 tok/s, context 51552pi / t6-payroll: 7054 tokens at 9.0 tok/s, context 58998pi / t6-payroll: 1311 tokens at 9.1 tok/s, context 60416pi / t6-payroll: 639 tokens at 9.5 tok/s, context 61342pi / t6-payroll: 123 tokens at 10.8 tok/s, context 62466pi / t6-payroll: 475 tokens at 9.5 tok/s, context 63120pi / t6-payroll: 1855 tokens at 19.1 tok/s, context 6616pi / t6-payroll: 50 tokens at 24.8 tok/s, context 117dsh · mean 13.6opencode · mean 15.8pi · mean 15.7
Prefill speed against prompt size, every request
dshopencodepi
0100200010k20k30kprompt tokens processed in the requestprefill, tok/sdsh / t1-cold-fix: 279 prompt tokens at 95 tok/sdsh / t1-cold-fix: 7773 prompt tokens at 190 tok/sdsh / t1-cold-fix: 382 prompt tokens at 118 tok/sdsh / t1-cold-fix: 529 prompt tokens at 129 tok/sdsh / t1-cold-fix: 695 prompt tokens at 131 tok/sdsh / t1-cold-fix: 2518 prompt tokens at 165 tok/sdsh / t1-cold-fix: 2705 prompt tokens at 158 tok/sdsh / t1-cold-fix: 2305 prompt tokens at 156 tok/sdsh / t1-cold-fix: 1219 prompt tokens at 137 tok/sdsh / t1-cold-fix: 159 prompt tokens at 60 tok/sdsh / t1-cold-fix: 672 prompt tokens at 116 tok/sdsh / t2-logsift: 9055 prompt tokens at 196 tok/sdsh / t2-logsift: 1651 prompt tokens at 161 tok/sdsh / t2-logsift: 1407 prompt tokens at 154 tok/sdsh / t2-logsift: 11784 prompt tokens at 173 tok/sdsh / t2-logsift: 487 prompt tokens at 109 tok/sdsh / t2-logsift: 189 prompt tokens at 64 tok/sdsh / t2-logsift: 236 prompt tokens at 76 tok/sdsh / t2-logsift: 979 prompt tokens at 132 tok/sdsh / t2-logsift: 596 prompt tokens at 109 tok/sdsh / t2-logsift: 10936 prompt tokens at 149 tok/sdsh / t2-logsift: 1258 prompt tokens at 119 tok/sdsh / t2-logsift: 376 prompt tokens at 95 tok/sdsh / t2-logsift: 645 prompt tokens at 109 tok/sdsh / t2-logsift: 585 prompt tokens at 102 tok/sdsh / t2-logsift: 10807 prompt tokens at 125 tok/sdsh / t2-logsift: 561 prompt tokens at 92 tok/sdsh / t2-logsift: 216 prompt tokens at 63 tok/sdsh / t2-logsift: 1198 prompt tokens at 102 tok/sdsh / t2-logsift: 1582 prompt tokens at 108 tok/sdsh / t2-logsift: 834 prompt tokens at 97 tok/sdsh / t2-logsift: 225 prompt tokens at 67 tok/sdsh / t2-logsift: 3687 prompt tokens at 118 tok/sdsh / t2-logsift: 2009 prompt tokens at 117 tok/sdsh / t2-logsift: 430 prompt tokens at 86 tok/sdsh / t2-logsift: 689 prompt tokens at 93 tok/sdsh / t2-logsift: 719 prompt tokens at 92 tok/sdsh / t2-logsift: 1022 prompt tokens at 101 tok/sdsh / t2-logsift: 257 prompt tokens at 67 tok/sdsh / t2-logsift: 491 prompt tokens at 84 tok/sdsh / t2-logsift: 813 prompt tokens at 99 tok/sdsh / t2-logsift: 4345 prompt tokens at 102 tok/sdsh / t2-logsift: 444 prompt tokens at 83 tok/sdsh / t2-logsift: 201 prompt tokens at 60 tok/sdsh / t2-logsift: 1577 prompt tokens at 92 tok/sdsh / t2-logsift: 1527 prompt tokens at 102 tok/sdsh / t2-logsift: 901 prompt tokens at 91 tok/sdsh / t2-logsift: 2498 prompt tokens at 100 tok/sdsh / t2-logsift: 993 prompt tokens at 93 tok/sdsh / t2-logsift: 571 prompt tokens at 79 tok/sdsh / t3-shoebox: 8858 prompt tokens at 195 tok/sdsh / t3-shoebox: 189 prompt tokens at 76 tok/sdsh / t3-shoebox: 427 prompt tokens at 122 tok/sdsh / t3-shoebox: 312 prompt tokens at 101 tok/sdsh / t3-shoebox: 2452 prompt tokens at 169 tok/sdsh / t3-shoebox: 1543 prompt tokens at 155 tok/sdsh / t3-shoebox: 1858 prompt tokens at 155 tok/sdsh / t3-shoebox: 5202 prompt tokens at 169 tok/sdsh / t3-shoebox: 2255 prompt tokens at 149 tok/sdsh / t3-shoebox: 4516 prompt tokens at 154 tok/sdsh / t3-shoebox: 3223 prompt tokens at 138 tok/sdsh / t3-shoebox: 5708 prompt tokens at 142 tok/sdsh / t3-shoebox: 3130 prompt tokens at 132 tok/sdsh / t3-shoebox: 1719 prompt tokens at 120 tok/sdsh / t3-shoebox: 1557 prompt tokens at 117 tok/sdsh / t3-shoebox: 1163 prompt tokens at 116 tok/sdsh / t3-shoebox: 1181 prompt tokens at 115 tok/sdsh / t3-shoebox: 1293 prompt tokens at 118 tok/sdsh / t3-shoebox: 1221 prompt tokens at 109 tok/sdsh / t3-shoebox: 3133 prompt tokens at 118 tok/sdsh / t3-shoebox: 1849 prompt tokens at 110 tok/sdsh / t3-shoebox: 528 prompt tokens at 95 tok/sdsh / t4-night-shift: 785 prompt tokens at 159 tok/sdsh / t4-night-shift: 8390 prompt tokens at 194 tok/sdsh / t4-night-shift: 521 prompt tokens at 129 tok/sdsh / t4-night-shift: 2639 prompt tokens at 162 tok/sdsh / t4-night-shift: 5220 prompt tokens at 170 tok/sdsh / t4-night-shift: 3163 prompt tokens at 150 tok/sdsh / t4-night-shift: 819 prompt tokens at 124 tok/sdsh / t4-night-shift: 397 prompt tokens at 100 tok/sdsh / t4-night-shift: 753 prompt tokens at 124 tok/sdsh / t4-night-shift: 4849 prompt tokens at 154 tok/sdsh / t4-night-shift: 1921 prompt tokens at 139 tok/sdsh / t4-night-shift: 1374 prompt tokens at 131 tok/sdsh / t4-night-shift: 974 prompt tokens at 130 tok/sdsh / t4-night-shift: 385 prompt tokens at 95 tok/sdsh / t5-regression: 356 prompt tokens at 120 tok/sdsh / t5-regression: 7844 prompt tokens at 190 tok/sdsh / t5-regression: 500 prompt tokens at 136 tok/sdsh / t5-regression: 660 prompt tokens at 133 tok/sdsh / t5-regression: 441 prompt tokens at 123 tok/sdsh / t5-regression: 1341 prompt tokens at 157 tok/sdsh / t5-regression: 4242 prompt tokens at 170 tok/sdsh / t5-regression: 1876 prompt tokens at 148 tok/sdsh / t5-regression: 2611 prompt tokens at 145 tok/sdsh / t5-regression: 1224 prompt tokens at 131 tok/sdsh / t5-regression: 1546 prompt tokens at 144 tok/sdsh / t5-regression: 875 prompt tokens at 128 tok/sdsh / t5-regression: 436 prompt tokens at 111 tok/sdsh / t5-regression: 378 prompt tokens at 99 tok/sdsh / t5-regression: 324 prompt tokens at 88 tok/sdsh / t5-regression: 288 prompt tokens at 84 tok/sdsh / t5-regression: 355 prompt tokens at 96 tok/sdsh / t5-regression: 313 prompt tokens at 89 tok/sdsh / t5-regression: 395 prompt tokens at 98 tok/sdsh / t6-payroll: 9675 prompt tokens at 194 tok/sdsh / t6-payroll: 613 prompt tokens at 129 tok/sdsh / t6-payroll: 883 prompt tokens at 140 tok/sdsh / t6-payroll: 11342 prompt tokens at 169 tok/sdsh / t6-payroll: 3616 prompt tokens at 155 tok/sdsh / t6-payroll: 3078 prompt tokens at 153 tok/sdsh / t6-payroll: 2844 prompt tokens at 141 tok/sdsh / t6-payroll: 1163 prompt tokens at 122 tok/sdsh / t6-payroll: 3680 prompt tokens at 136 tok/sdsh / t6-payroll: 780 prompt tokens at 109 tok/sdsh / t6-payroll: 4060 prompt tokens at 142 tok/sdsh / t6-payroll: 1351 prompt tokens at 124 tok/sdsh / t6-payroll: 520 prompt tokens at 107 tok/sdsh / t6-payroll: 2149 prompt tokens at 123 tok/sdsh / t6-payroll: 1075 prompt tokens at 118 tok/sdsh / t6-payroll: 1895 prompt tokens at 115 tok/sdsh / t6-payroll: 1936 prompt tokens at 122 tok/sdsh / t6-payroll: 1052 prompt tokens at 113 tok/sdsh / t6-payroll: 312 prompt tokens at 79 tok/sdsh / t6-payroll: 3466 prompt tokens at 121 tok/sdsh / t6-payroll: 1213 prompt tokens at 104 tok/sdsh / t6-payroll: 3640 prompt tokens at 118 tok/sdsh / t6-payroll: 4400 prompt tokens at 116 tok/sdsh / t6-payroll: 3838 prompt tokens at 119 tok/sdsh / t6-payroll: 370 prompt tokens at 80 tok/sdsh / t6-payroll: 347 prompt tokens at 78 tok/sdsh / t6-payroll: 684 prompt tokens at 89 tok/sdsh / t6-payroll: 2059 prompt tokens at 103 tok/sdsh / t6-payroll: 2535 prompt tokens at 101 tok/sdsh / t6-payroll: 3173 prompt tokens at 99 tok/sdsh / t6-payroll: 2799 prompt tokens at 100 tok/sdsh / t6-payroll: 1052 prompt tokens at 90 tok/sdsh / t6-payroll: 540 prompt tokens at 77 tok/sdsh / t6-payroll: 696 prompt tokens at 82 tok/sdsh / t6-payroll: 620 prompt tokens at 83 tok/sopencode / t1-cold-fix: 68 prompt tokens at 35 tok/sopencode / t1-cold-fix: 705 prompt tokens at 117 tok/sopencode / t1-cold-fix: 7249 prompt tokens at 193 tok/sopencode / t1-cold-fix: 1766 prompt tokens at 170 tok/sopencode / t1-cold-fix: 337 prompt tokens at 102 tok/sopencode / t1-cold-fix: 1266 prompt tokens at 152 tok/sopencode / t1-cold-fix: 1497 prompt tokens at 158 tok/sopencode / t1-cold-fix: 2269 prompt tokens at 157 tok/sopencode / t1-cold-fix: 1609 prompt tokens at 152 tok/sopencode / t1-cold-fix: 2814 prompt tokens at 158 tok/sopencode / t1-cold-fix: 580 prompt tokens at 112 tok/sopencode / t1-cold-fix: 144 prompt tokens at 56 tok/sopencode / t1-cold-fix: 209 prompt tokens at 74 tok/sopencode / t1-cold-fix: 195 prompt tokens at 72 tok/sopencode / t1-cold-fix: 158 prompt tokens at 61 tok/sopencode / t2-logsift: 1419 prompt tokens at 193 tok/sopencode / t2-logsift: 1811 prompt tokens at 164 tok/sopencode / t2-logsift: 214 prompt tokens at 82 tok/sopencode / t2-logsift: 1325 prompt tokens at 158 tok/sopencode / t2-logsift: 6857 prompt tokens at 180 tok/sopencode / t2-logsift: 309 prompt tokens at 93 tok/sopencode / t2-logsift: 151 prompt tokens at 58 tok/sopencode / t2-logsift: 268 prompt tokens at 84 tok/sopencode / t2-logsift: 1014 prompt tokens at 137 tok/sopencode / t2-logsift: 471 prompt tokens at 116 tok/sopencode / t2-logsift: 1113 prompt tokens at 137 tok/sopencode / t2-logsift: 1989 prompt tokens at 154 tok/sopencode / t2-logsift: 1506 prompt tokens at 145 tok/sopencode / t2-logsift: 1073 prompt tokens at 128 tok/sopencode / t2-logsift: 4094 prompt tokens at 147 tok/sopencode / t2-logsift: 790 prompt tokens at 119 tok/sopencode / t2-logsift: 217 prompt tokens at 70 tok/sopencode / t2-logsift: 299 prompt tokens at 85 tok/sopencode / t2-logsift: 594 prompt tokens at 107 tok/sopencode / t3-shoebox: 1800 prompt tokens at 177 tok/sopencode / t3-shoebox: 8346 prompt tokens at 199 tok/sopencode / t3-shoebox: 205 prompt tokens at 80 tok/sopencode / t3-shoebox: 5377 prompt tokens at 178 tok/sopencode / t3-shoebox: 2950 prompt tokens at 163 tok/sopencode / t3-shoebox: 3548 prompt tokens at 166 tok/sopencode / t3-shoebox: 1749 prompt tokens at 148 tok/sopencode / t3-shoebox: 1105 prompt tokens at 135 tok/sopencode / t3-shoebox: 500 prompt tokens at 122 tok/sopencode / t3-shoebox: 618 prompt tokens at 114 tok/sopencode / t3-shoebox: 791 prompt tokens at 117 tok/sopencode / t3-shoebox: 241 prompt tokens at 74 tok/sopencode / t3-shoebox: 1136 prompt tokens at 120 tok/sopencode / t3-shoebox: 237 prompt tokens at 76 tok/sopencode / t3-shoebox: 931 prompt tokens at 124 tok/sopencode / t3-shoebox: 501 prompt tokens at 110 tok/sopencode / t3-shoebox: 1456 prompt tokens at 136 tok/sopencode / t3-shoebox: 1933 prompt tokens at 138 tok/sopencode / t3-shoebox: 1696 prompt tokens at 139 tok/sopencode / t3-shoebox: 1572 prompt tokens at 129 tok/sopencode / t3-shoebox: 1457 prompt tokens at 128 tok/sopencode / t3-shoebox: 507 prompt tokens at 107 tok/sopencode / t3-shoebox: 1656 prompt tokens at 127 tok/sopencode / t3-shoebox: 1489 prompt tokens at 128 tok/sopencode / t3-shoebox: 4570 prompt tokens at 132 tok/sopencode / t3-shoebox: 1488 prompt tokens at 118 tok/sopencode / t3-shoebox: 753 prompt tokens at 97 tok/sopencode / t3-shoebox: 805 prompt tokens at 105 tok/sopencode / t3-shoebox: 747 prompt tokens at 96 tok/sopencode / t3-shoebox: 457 prompt tokens at 94 tok/sopencode / t3-shoebox: 726 prompt tokens at 96 tok/sopencode / t3-shoebox: 303 prompt tokens at 75 tok/sopencode / t4-night-shift: 1329 prompt tokens at 165 tok/sopencode / t4-night-shift: 7873 prompt tokens at 200 tok/sopencode / t4-night-shift: 430 prompt tokens at 124 tok/sopencode / t4-night-shift: 2023 prompt tokens at 166 tok/sopencode / t4-night-shift: 653 prompt tokens at 126 tok/sopencode / t4-night-shift: 5365 prompt tokens at 171 tok/sopencode / t4-night-shift: 809 prompt tokens at 124 tok/sopencode / t4-night-shift: 164 prompt tokens at 62 tok/sopencode / t4-night-shift: 530 prompt tokens at 115 tok/sopencode / t4-night-shift: 380 prompt tokens at 104 tok/sopencode / t4-night-shift: 460 prompt tokens at 113 tok/sopencode / t4-night-shift: 435 prompt tokens at 106 tok/sopencode / t4-night-shift: 514 prompt tokens at 118 tok/sopencode / t4-night-shift: 817 prompt tokens at 123 tok/sopencode / t4-night-shift: 399 prompt tokens at 102 tok/sopencode / t4-night-shift: 431 prompt tokens at 110 tok/sopencode / t4-night-shift: 598 prompt tokens at 113 tok/sopencode / t4-night-shift: 755 prompt tokens at 123 tok/sopencode / t4-night-shift: 345 prompt tokens at 98 tok/sopencode / t4-night-shift: 619 prompt tokens at 109 tok/sopencode / t4-night-shift: 1487 prompt tokens at 141 tok/sopencode / t4-night-shift: 885 prompt tokens at 130 tok/sopencode / t4-night-shift: 806 prompt tokens at 125 tok/sopencode / t4-night-shift: 892 prompt tokens at 128 tok/sopencode / t4-night-shift: 262 prompt tokens at 77 tok/sopencode / t4-night-shift: 862 prompt tokens at 119 tok/sopencode / t5-regression: 204 prompt tokens at 95 tok/sopencode / t5-regression: 7325 prompt tokens at 195 tok/sopencode / t5-regression: 1541 prompt tokens at 166 tok/sopencode / t5-regression: 460 prompt tokens at 127 tok/sopencode / t5-regression: 1505 prompt tokens at 162 tok/sopencode / t5-regression: 944 prompt tokens at 144 tok/sopencode / t5-regression: 453 prompt tokens at 119 tok/sopencode / t5-regression: 297 prompt tokens at 93 tok/sopencode / t5-regression: 337 prompt tokens at 99 tok/sopencode / t5-regression: 286 prompt tokens at 91 tok/sopencode / t5-regression: 123 prompt tokens at 51 tok/sopencode / t6-payroll: 2030 prompt tokens at 211 tok/sopencode / t6-payroll: 2340 prompt tokens at 171 tok/sopencode / t6-payroll: 391 prompt tokens at 115 tok/sopencode / t6-payroll: 420 prompt tokens at 120 tok/sopencode / t6-payroll: 173 prompt tokens at 67 tok/sopencode / t6-payroll: 347 prompt tokens at 104 tok/sopencode / t6-payroll: 80 prompt tokens at 38 tok/sopencode / t6-payroll: 11308 prompt tokens at 171 tok/sopencode / t6-payroll: 20821 prompt tokens at 154 tok/sopencode / t6-payroll: 1293 prompt tokens at 116 tok/sopencode / t6-payroll: 1473 prompt tokens at 117 tok/sopencode / t6-payroll: 29508 prompt tokens at 107 tok/sopencode / t6-payroll: 5277 prompt tokens at 105 tok/sopencode / t6-payroll: 3808 prompt tokens at 114 tok/sopencode / t6-payroll: 1352 prompt tokens at 96 tok/sopencode / t6-payroll: 794 prompt tokens at 86 tok/sopencode / t6-payroll: 1152 prompt tokens at 88 tok/sopencode / t6-payroll: 9291 prompt tokens at 108 tok/spi / t1-cold-fix: 1661 prompt tokens at 171 tok/spi / t1-cold-fix: 421 prompt tokens at 135 tok/spi / t1-cold-fix: 4809 prompt tokens at 195 tok/spi / t1-cold-fix: 2185 prompt tokens at 164 tok/spi / t1-cold-fix: 877 prompt tokens at 142 tok/spi / t1-cold-fix: 135 prompt tokens at 56 tok/spi / t1-cold-fix: 812 prompt tokens at 133 tok/spi / t1-cold-fix: 244 prompt tokens at 88 tok/spi / t1-cold-fix: 685 prompt tokens at 130 tok/spi / t1-cold-fix: 246 prompt tokens at 87 tok/spi / t2-logsift: 2944 prompt tokens at 210 tok/spi / t2-logsift: 1268 prompt tokens at 178 tok/spi / t2-logsift: 613 prompt tokens at 134 tok/spi / t2-logsift: 7360 prompt tokens at 191 tok/spi / t2-logsift: 1522 prompt tokens at 150 tok/spi / t2-logsift: 240 prompt tokens at 80 tok/spi / t2-logsift: 901 prompt tokens at 141 tok/spi / t2-logsift: 5048 prompt tokens at 165 tok/spi / t2-logsift: 582 prompt tokens at 111 tok/spi / t2-logsift: 440 prompt tokens at 107 tok/spi / t2-logsift: 536 prompt tokens at 116 tok/spi / t2-logsift: 2690 prompt tokens at 148 tok/spi / t2-logsift: 1325 prompt tokens at 136 tok/spi / t2-logsift: 917 prompt tokens at 126 tok/spi / t2-logsift: 557 prompt tokens at 104 tok/spi / t2-logsift: 998 prompt tokens at 125 tok/spi / t2-logsift: 460 prompt tokens at 111 tok/spi / t2-logsift: 1984 prompt tokens at 135 tok/spi / t2-logsift: 176 prompt tokens at 63 tok/spi / t2-logsift: 2645 prompt tokens at 128 tok/spi / t2-logsift: 1578 prompt tokens at 128 tok/spi / t2-logsift: 855 prompt tokens at 109 tok/spi / t2-logsift: 958 prompt tokens at 112 tok/spi / t2-logsift: 246 prompt tokens at 75 tok/spi / t2-logsift: 719 prompt tokens at 112 tok/spi / t2-logsift: 933 prompt tokens at 117 tok/spi / t2-logsift: 663 prompt tokens at 106 tok/spi / t2-logsift: 781 prompt tokens at 105 tok/spi / t2-logsift: 818 prompt tokens at 102 tok/spi / t2-logsift: 966 prompt tokens at 115 tok/spi / t2-logsift: 6479 prompt tokens at 150 tok/spi / t2-logsift: 1308 prompt tokens at 106 tok/spi / t2-logsift: 511 prompt tokens at 92 tok/spi / t2-logsift: 535 prompt tokens at 93 tok/spi / t2-logsift: 216 prompt tokens at 66 tok/spi / t2-logsift: 2602 prompt tokens at 130 tok/spi / t2-logsift: 1093 prompt tokens at 101 tok/spi / t2-logsift: 1621 prompt tokens at 105 tok/spi / t2-logsift: 1876 prompt tokens at 107 tok/spi / t2-logsift: 321 prompt tokens at 74 tok/spi / t2-logsift: 384 prompt tokens at 81 tok/spi / t2-logsift: 245 prompt tokens at 72 tok/spi / t2-logsift: 2153 prompt tokens at 102 tok/spi / t2-logsift: 513 prompt tokens at 90 tok/spi / t2-logsift: 568 prompt tokens at 93 tok/spi / t2-logsift: 909 prompt tokens at 94 tok/spi / t2-logsift: 30043 prompt tokens at 173 tok/spi / t3-shoebox: 2744 prompt tokens at 206 tok/spi / t3-shoebox: 579 prompt tokens at 140 tok/spi / t3-shoebox: 1445 prompt tokens at 181 tok/spi / t3-shoebox: 2093 prompt tokens at 178 tok/spi / t3-shoebox: 1277 prompt tokens at 160 tok/spi / t3-shoebox: 2327 prompt tokens at 165 tok/spi / t3-shoebox: 3294 prompt tokens at 164 tok/spi / t3-shoebox: 2755 prompt tokens at 168 tok/spi / t3-shoebox: 1903 prompt tokens at 153 tok/spi / t3-shoebox: 656 prompt tokens at 119 tok/spi / t3-shoebox: 1759 prompt tokens at 144 tok/spi / t3-shoebox: 762 prompt tokens at 124 tok/spi / t3-shoebox: 1139 prompt tokens at 132 tok/spi / t3-shoebox: 2436 prompt tokens at 149 tok/spi / t3-shoebox: 1335 prompt tokens at 130 tok/spi / t3-shoebox: 498 prompt tokens at 109 tok/spi / t4-night-shift: 2280 prompt tokens at 201 tok/spi / t4-night-shift: 670 prompt tokens at 152 tok/spi / t4-night-shift: 808 prompt tokens at 156 tok/spi / t4-night-shift: 675 prompt tokens at 140 tok/spi / t4-night-shift: 451 prompt tokens at 127 tok/spi / t4-night-shift: 1318 prompt tokens at 176 tok/spi / t4-night-shift: 372 prompt tokens at 122 tok/spi / t4-night-shift: 6724 prompt tokens at 184 tok/spi / t4-night-shift: 524 prompt tokens at 114 tok/spi / t4-night-shift: 647 prompt tokens at 118 tok/spi / t4-night-shift: 774 prompt tokens at 126 tok/spi / t4-night-shift: 419 prompt tokens at 109 tok/spi / t4-night-shift: 1036 prompt tokens at 144 tok/spi / t4-night-shift: 830 prompt tokens at 134 tok/spi / t4-night-shift: 356 prompt tokens at 99 tok/spi / t4-night-shift: 706 prompt tokens at 122 tok/spi / t4-night-shift: 2773 prompt tokens at 150 tok/spi / t4-night-shift: 983 prompt tokens at 130 tok/spi / t4-night-shift: 2314 prompt tokens at 138 tok/spi / t4-night-shift: 1501 prompt tokens at 130 tok/spi / t4-night-shift: 600 prompt tokens at 112 tok/spi / t4-night-shift: 256 prompt tokens at 82 tok/spi / t4-night-shift: 533 prompt tokens at 108 tok/spi / t4-night-shift: 368 prompt tokens at 94 tok/spi / t4-night-shift: 229 prompt tokens at 77 tok/spi / t4-night-shift: 291 prompt tokens at 97 tok/spi / t4-night-shift: 794 prompt tokens at 117 tok/spi / t4-night-shift: 1700 prompt tokens at 132 tok/spi / t4-night-shift: 445 prompt tokens at 104 tok/spi / t4-night-shift: 693 prompt tokens at 108 tok/spi / t4-night-shift: 926 prompt tokens at 119 tok/spi / t4-night-shift: 795 prompt tokens at 122 tok/spi / t4-night-shift: 3151 prompt tokens at 125 tok/spi / t4-night-shift: 1269 prompt tokens at 130 tok/spi / t5-regression: 1733 prompt tokens at 177 tok/spi / t5-regression: 645 prompt tokens at 149 tok/spi / t5-regression: 2483 prompt tokens at 196 tok/spi / t5-regression: 4272 prompt tokens at 181 tok/spi / t5-regression: 1568 prompt tokens at 160 tok/spi / t5-regression: 2665 prompt tokens at 153 tok/spi / t5-regression: 1270 prompt tokens at 143 tok/spi / t5-regression: 560 prompt tokens at 121 tok/spi / t5-regression: 1276 prompt tokens at 148 tok/spi / t5-regression: 459 prompt tokens at 112 tok/spi / t5-regression: 579 prompt tokens at 114 tok/spi / t5-regression: 380 prompt tokens at 104 tok/spi / t5-regression: 246 prompt tokens at 84 tok/spi / t5-regression: 296 prompt tokens at 90 tok/spi / t5-regression: 431 prompt tokens at 108 tok/spi / t5-regression: 526 prompt tokens at 114 tok/spi / t5-regression: 349 prompt tokens at 97 tok/spi / t5-regression: 1677 prompt tokens at 139 tok/spi / t5-regression: 344 prompt tokens at 93 tok/spi / t5-regression: 626 prompt tokens at 111 tok/spi / t6-payroll: 3559 prompt tokens at 218 tok/spi / t6-payroll: 477 prompt tokens at 142 tok/spi / t6-payroll: 595 prompt tokens at 130 tok/spi / t6-payroll: 10535 prompt tokens at 182 tok/spi / t6-payroll: 8566 prompt tokens at 170 tok/spi / t6-payroll: 565 prompt tokens at 109 tok/spi / t6-payroll: 481 prompt tokens at 113 tok/spi / t6-payroll: 191 prompt tokens at 68 tok/spi / t6-payroll: 1252 prompt tokens at 134 tok/spi / t6-payroll: 2658 prompt tokens at 140 tok/spi / t6-payroll: 1019 prompt tokens at 127 tok/spi / t6-payroll: 1771 prompt tokens at 139 tok/spi / t6-payroll: 1904 prompt tokens at 132 tok/spi / t6-payroll: 1526 prompt tokens at 132 tok/spi / t6-payroll: 12916 prompt tokens at 131 tok/spi / t6-payroll: 3990 prompt tokens at 120 tok/spi / t6-payroll: 7165 prompt tokens at 124 tok/spi / t6-payroll: 1602 prompt tokens at 110 tok/spi / t6-payroll: 1644 prompt tokens at 105 tok/spi / t6-payroll: 306 prompt tokens at 79 tok/spi / t6-payroll: 4762 prompt tokens at 211 tok/sdsh · mean 119.9opencode · mean 121.5pi · mean 127.7
Available memory over the campaign
0100010hours into the campaignavailable memory, GB0.0 h: 113.00.1 h: 19.00.2 h: 20.00.2 h: 19.00.3 h: 19.00.4 h: 19.00.5 h: 19.00.6 h: 18.00.7 h: 18.00.8 h: 18.00.8 h: 18.00.9 h: 18.01.0 h: 18.01.1 h: 18.01.2 h: 16.01.3 h: 16.01.3 h: 15.01.4 h: 15.01.5 h: 14.01.6 h: 16.01.7 h: 16.01.8 h: 16.01.8 h: 16.01.9 h: 16.02.0 h: 16.02.1 h: 16.02.2 h: 16.02.3 h: 16.02.3 h: 16.02.4 h: 16.02.5 h: 16.02.6 h: 16.02.7 h: 16.02.8 h: 16.02.8 h: 16.02.9 h: 16.03.0 h: 15.03.1 h: 14.03.2 h: 14.03.3 h: 15.03.3 h: 15.03.4 h: 15.03.5 h: 15.03.6 h: 15.03.7 h: 15.03.8 h: 15.03.8 h: 15.03.9 h: 15.04.0 h: 15.04.1 h: 15.04.2 h: 17.04.3 h: 15.04.3 h: 15.04.4 h: 15.04.5 h: 15.04.6 h: 15.04.7 h: 15.04.8 h: 15.04.8 h: 15.04.9 h: 15.05.0 h: 15.05.1 h: 15.05.2 h: 16.05.3 h: 16.05.3 h: 14.05.4 h: 14.05.5 h: 14.05.6 h: 14.05.7 h: 14.05.8 h: 14.05.8 h: 14.05.9 h: 14.06.0 h: 14.06.1 h: 14.06.2 h: 14.06.3 h: 14.06.3 h: 16.06.4 h: 16.06.5 h: 16.06.6 h: 16.06.7 h: 16.06.8 h: 15.06.8 h: 15.06.9 h: 15.07.0 h: 15.07.1 h: 15.07.2 h: 15.07.3 h: 15.07.3 h: 15.07.4 h: 15.07.5 h: 15.07.6 h: 15.07.7 h: 15.07.8 h: 15.07.8 h: 15.07.9 h: 15.08.0 h: 14.08.1 h: 13.08.2 h: 14.08.3 h: 13.08.3 h: 13.08.4 h: 14.08.5 h: 14.08.6 h: 14.08.7 h: 14.08.8 h: 14.08.8 h: 14.08.9 h: 14.09.0 h: 14.09.1 h: 14.09.2 h: 14.09.3 h: 14.09.3 h: 14.09.4 h: 16.09.5 h: 16.09.6 h: 15.09.7 h: 15.09.8 h: 15.09.8 h: 15.09.9 h: 15.010.0 h: 15.010.1 h: 15.010.2 h: 15.010.3 h: 15.010.3 h: 15.010.4 h: 15.010.5 h: 15.010.6 h: 15.010.7 h: 15.010.8 h: 15.010.8 h: 14.010.9 h: 14.011.0 h: 14.011.1 h: 14.011.2 h: 93.011.3 h: 90.011.3 h: 89.0
Swap in use over the campaign
010010hours into the campaignswap used, GB0.0 h: 0.00.1 h: 1.00.2 h: 1.00.2 h: 1.00.3 h: 1.00.4 h: 1.00.5 h: 1.00.6 h: 2.00.7 h: 3.00.8 h: 4.00.8 h: 4.00.9 h: 5.01.0 h: 5.01.1 h: 5.01.2 h: 7.01.3 h: 8.01.3 h: 8.01.4 h: 9.01.5 h: 10.01.6 h: 8.01.7 h: 8.01.8 h: 8.01.8 h: 8.01.9 h: 8.02.0 h: 9.02.1 h: 9.02.2 h: 9.02.3 h: 9.02.3 h: 9.02.4 h: 9.02.5 h: 9.02.6 h: 9.02.7 h: 9.02.8 h: 9.02.8 h: 9.02.9 h: 9.03.0 h: 6.03.1 h: 7.03.2 h: 7.03.3 h: 7.03.3 h: 7.03.4 h: 7.03.5 h: 8.03.6 h: 8.03.7 h: 9.03.8 h: 9.03.8 h: 9.03.9 h: 10.04.0 h: 10.04.1 h: 10.04.2 h: 7.04.3 h: 7.04.3 h: 8.04.4 h: 8.04.5 h: 7.04.6 h: 7.04.7 h: 8.04.8 h: 8.04.8 h: 8.04.9 h: 8.05.0 h: 8.05.1 h: 8.05.2 h: 7.05.3 h: 7.05.3 h: 7.05.4 h: 7.05.5 h: 7.05.6 h: 7.05.7 h: 7.05.8 h: 7.05.8 h: 7.05.9 h: 7.06.0 h: 7.06.1 h: 7.06.2 h: 6.06.3 h: 6.06.3 h: 6.06.4 h: 6.06.5 h: 6.06.6 h: 6.06.7 h: 6.06.8 h: 6.06.8 h: 6.06.9 h: 6.07.0 h: 6.07.1 h: 6.07.2 h: 6.07.3 h: 6.07.3 h: 6.07.4 h: 6.07.5 h: 6.07.6 h: 6.07.7 h: 6.07.8 h: 6.07.8 h: 6.07.9 h: 6.08.0 h: 6.08.1 h: 6.08.2 h: 6.08.3 h: 6.08.3 h: 6.08.4 h: 6.08.5 h: 7.08.6 h: 7.08.7 h: 8.08.8 h: 8.08.8 h: 8.08.9 h: 6.09.0 h: 7.09.1 h: 7.09.2 h: 7.09.3 h: 7.09.3 h: 7.09.4 h: 6.09.5 h: 6.09.6 h: 6.09.7 h: 6.09.8 h: 6.09.8 h: 6.09.9 h: 6.010.0 h: 6.010.1 h: 6.010.2 h: 6.010.3 h: 6.010.3 h: 6.010.4 h: 6.010.5 h: 6.010.6 h: 6.010.7 h: 6.010.8 h: 6.010.8 h: 6.010.9 h: 6.011.0 h: 6.011.1 h: 6.011.2 h: 4.011.3 h: 4.011.3 h: 4.0
harnesstaskwall mingen tok/sprefill tok/sgen tokenspeak ctxthink tokrequests
dsht1-cold-fix815.61585,29919,477~2,37413
dsht2-logsift10010.412656,16779,811~11,22540
dsht3-shoebox5411.514332,69154,245~10,10224
dsht4-night-shift2713.515719,12532,108~7,06315
dsht5-regression1315.21498,91626,372~3,79820
dsht6-payroll939.712848,18584,443~4,40136
opencodet1-cold-fix815.21545,41220,184~2,60215
opencodet2-logsift2114.614615,90930,852~7,72120
opencodet3-shoebox3712.814123,93950,118~11,11934
opencodet4-night-shift2015.314615,27429,054~7,28027
opencodet5-regression417.81583,00513,403~1,28413
opencodet6-payroll8710.012344,49796,796~19,77719
pit1-cold-fix518.21573,92912,608~1,83311
pit2-logsift6712.414141,71663,697~12,99948
pit3-shoebox1714.915612,47027,707~4,17418
pit4-night-shift3714.913828,12138,901~13,00235
pit5-regression1117.11468,81922,907~4,37521
pit6-payroll5511.614432,73663,120~11,50823

Campaigns test settings per campaign, expand for the full record

Qwen3.8-Flash-Next UD-IQ4_XS (93.7 GB, the top quant that fits 128 GB), thinking xhigh (the template default), stock llama.cpp master (Aug 28, Vulkan, RADV), flash attention on, ctx 131072, samplers pinned server-side to the model card thinking set 1.0/0.95/20/0, inline think blocks · Qwen3.8-Flash-Next-UD-IQ4_XS-00001-of-00003.gguf · dsh, opencode, pi · 17/18 resolved · v0.2.4
folder 2026-09-16-v024-flashnext-iq4xs-think-xhigh-3h
labelQwen3.8-Flash-Next UD-IQ4_XS (93.7 GB, the top quant that fits 128 GB), thinking xhigh (the template default), stock llama.cpp master (Aug 28, Vulkan, RADV), flash attention on, ctx 131072, samplers pinned server-side to the model card thinking set 1.0/0.95/20/0, inline think blocks
model fileQwen3.8-Flash-Next-UD-IQ4_XS-00001-of-00003.gguf
hugging facehttps://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF
suite version0.2.4 current
run dates2026-09-16 03:50 to 2026-09-16 13:23
context length131072
serverb1-ca3d5a3
samplingtemperature=1.0, top_p=0.95, top_k=20, min_p=0.0, repeat_penalty=1.0
thinking modexhigh (template default); every earlier harness campaign ran medium at temp 0.6, so compare with care; ~7590 think tok avg per run
speculative / mtpnone
harnessespi 0.73.1 · dsh 0.1.1-rc.2 · opencode 1.18.25
runs18 runs, n=1 per (harness, task)
platformLinux-7.2.5-1-cachyos-x86_64-with-glibc2.44

t1-cold-fix

3/3 resolved · mean 100 · 6.9 min avg · timeout 20 min
Two planted defects in different layers of a small stock ledger; one shows only through integration tests. Hidden edge tests catch fixes in the wrong layer; the tree must stay clean.
prompt ab20b1a6b1 · commit 35a9b01 LATEST · task content 81be8e28c6 LATEST
the exact prompt these runs saw
The test suite in this project is failing. Run it, find the root causes, and fix them.

Rules:
- Do not modify anything under tests/ or data/. The tests describe the intended behavior and they are correct.
- Fix the underlying defects, not the symptoms. Making a test pass by special-casing it is not a fix.
- Keep the change as small as the fixes allow, and leave the tree clean: no scratch files, no edits outside the modules that are actually defective.
- You are done when the full test suite passes.
scoring key: how the 100 points split
failing tests pass 25full visible suite 20hidden ledger edges 20hidden loader edges 15footprint clean 20
what each checkpoint verifies
  • failing tests pass: the four visible failing tests pass
  • full visible suite: the whole visible suite passes (nothing else broke)
  • hidden ledger edges: hidden ledger tests pass: shipments never move the cost basis, sell-out and restart, loss sales, overship leaves state intact
  • hidden loader edges: hidden loader tests pass: SKU normalization agrees in both directions, mixed-case export replays
  • footprint clean: only stockroom/ledger.py and stockroom/loader.py changed; tests/, data/, README untouched; no stray files (needs c1)
#harness · modelresulttimeturnstoolsgen tokthinktg tok/speak ctxoutput
#1dsh
qwen38-flashnext-iq4xs
RESOLVED 1007.7 min11952992374~15.619kdiff · transcript
checkpoints, tools, judge
✓ failing tests pass 25 ✓ full visible suite 20 ✓ hidden ledger edges 20 ✓ hidden loader edges 15 ✓ footprint clean 20
tools: read ×10, bash ×7, edit ×2 · tokens: · started 2026-09-16 10:01:01 · pp 157.7 tok/s
quality 5/5 · AI-graded by qwen38-27b-q4-judge at 2026-09-16 14:56 · hover a chip for the question and reason: ✓ q1 ✓ q2 ✓ q3 ✓ q4 ✓ q5
#1opencode
qwen38-flashnext-iq4xs
RESOLVED 1008.2 min132054122602~15.220kdiff · transcript
checkpoints, tools, judge
✓ failing tests pass 25 ✓ full visible suite 20 ✓ hidden ledger edges 20 ✓ hidden loader edges 15 ✓ footprint clean 20
tools: read ×11, bash ×7, edit ×2 · tokens: input 20093, output 5260 · started 2026-09-16 03:50:17 · pp 153.6 tok/s
quality 5/5 · AI-graded by qwen38-27b-q4-judge at 2026-09-16 15:00 · hover a chip for the question and reason: ✓ q1 ✓ q2 ✓ q3 ✓ q4 ✓ q5
#1pi
qwen38-flashnext-iq4xs
RESOLVED 1004.9 min101339291833~18.213kdiff · transcript
checkpoints, tools, judge
✓ failing tests pass 25 ✓ full visible suite 20 ✓ hidden ledger edges 20 ✓ hidden loader edges 15 ✓ footprint clean 20
tools: bash ×11, edit ×2 · tokens: input 12075, output 3879, cache_read 73444 · started 2026-09-16 06:48:36 · pp 156.8 tok/s
quality 5/5 · AI-graded by qwen38-27b-q4-judge at 2026-09-16 15:06 · hover a chip for the question and reason: ✓ q1 ✓ q2 ✓ q3 ✓ q4 ✓ q5

t2-logsift

3/3 resolved · mean 100 · 63.0 min avg · timeout 25 min
Write a multi-file log analyzer from a precise spec: strict line validation, calendar-checked timestamps, repeatable filters, per-file JSON rollups, exit codes
prompt e0806ba9b5 · commit 7b98f05 LATEST · task content f45b10dc77 LATEST
the exact prompt these runs saw
Build a command-line log analyzer in this directory as a single file named logsift.py, runnable with Python 3 standard library only:

    python3 logsift.py [OPTIONS] FILE [FILE ...]

One or more log files are given. All of them are read and analyzed together as one stream: matching entries and malformed lines are counted across all files. A file given twice is read twice.

Input format, one entry per line:

    <timestamp> <LEVEL> <component>: <message>

Lines are separated by "\n". A final "\n" at the end of a file does not start another line (an empty file has zero lines). Before a line is matched, one trailing "\r" is removed if present, so files with CRLF line endings behave exactly like LF files. After that, the line must match the shape above exactly, with a single space between the fields:

- <timestamp> is UTC in the form YYYY-MM-DDTHH:MM:SSZ, exactly 20 characters, and it must be a real calendar instant: the date must exist in the Gregorian calendar (Feb 29 only in leap years), hour 00-23, minute 00-59, second 00-59. So 2026-02-30T10:00:00Z and 2026-03-14T24:00:00Z are not valid timestamps. Fractional seconds, a timezone offset instead of Z, a lowercase z, or a space instead of T are not valid either.
- <LEVEL> is exactly one of DEBUG, INFO, WARN, ERROR, in uppercase. Warn and warn are not levels.
- <component> is one or more non-space characters, immediately followed by a literal ": " (colon, space). The component cannot be empty.
- <message> is everything after that ": " and must be non-empty. It may contain any characters, including more colons, and a message made only of spaces still counts as non-empty.

Any line that does not match (including an empty line, a line of only whitespace, a line with a leading space, or a tab between fields) is malformed. Malformed lines are only ever counted as malformed; filters never apply to them.

Options:
- --min-level LEVEL   keep only entries at or above LEVEL (severity order: DEBUG < INFO < WARN < ERROR)
- --since TS          keep only entries with timestamp >= TS (inclusive)
- --until TS          keep only entries with timestamp <= TS (inclusive)
- --component NAME    keep only entries of that component (exact match). May be given more than once: an entry then matches if its component equals any of the given names.
- --top N             how many pairs "top_components" holds in JSON output (default 3)
- --json              JSON output mode (see below)

LEVEL must be exactly one of the four level names, uppercase. TS must be a valid timestamp by the same rule as the log lines. --since greater than --until is allowed and simply matches nothing. N must be a positive integer: one or more decimal digits with a value of at least 1 (so 0, -1, 2.0 and x are all rejected). --top is checked even when --json is not given.

"Matching entries" means well-formed lines that pass every given filter.

Default output (no --json), exactly these seven lines in this order, one per line:

    DEBUG <count>
    INFO <count>
    WARN <count>
    ERROR <count>
    components <number of distinct components among matching entries>
    malformed <count of malformed lines across all files>
    files <number of FILE arguments given, counting repeats>

where each level count is the number of matching entries of that level. So `logsift.py x.log x.log` ends with the line `files 2`.

JSON output (--json): print a single JSON object with exactly these eight keys:
- "counts": object mapping each of the four level names to its matching-entry count (all four keys always present)
- "components": object mapping each component name that appears among matching entries to its matching-entry count
- "malformed": malformed line count across all files
- "malformed_by_file": object mapping each FILE, exactly as written on the command line, to the number of malformed lines read from it. Every FILE given appears, even with 0 malformed lines. A file given twice is read twice and appears once, with both readings summed. "malformed" stays the sum of all readings.
- "first_ts": the smallest (earliest) timestamp among matching entries, as a string; null if there are none
- "last_ts": the largest (latest) timestamp among matching entries, as a string; null if there are none
- "by_hour": object mapping "YYYY-MM-DDTHH" (the first 13 characters of the timestamp) to the number of matching entries in that UTC hour; only hours with at least one matching entry appear
- "top_components": list of up to N [name, count] pairs (N from --top, default 3) for the components with the most matching entries, sorted by count descending, then by name ascending (plain string order). Ties matter: when several components share a count, the alphabetically earlier names are the ones kept. Empty list when nothing matches.

Files are not guaranteed to be in time order, neither within a file nor across files.

Exit codes:
- 0 on success
- 2 if any FILE does not exist or cannot be read, or if no FILE is given at all
- 3 if --min-level is given a value that is not one of the four level names
- 4 if --since or --until is given a value that is not a valid timestamp
- 5 if --top is given a value that is not a positive integer

Checks happen in this order and the first failure decides the exit code: --min-level (3), then --since and --until (4), then --top (5), then the files (2). "No FILE given at all" is a file check, so it comes last as well. Option values are checked before any file is opened, and every file is checked before anything is printed. On any non-zero exit, stdout stays empty and a short message goes to stderr. Error text never goes to stdout.

A sample.log is provided to try against. You are done when logsift.py implements all of the above.
scoring key: how the 100 points split
runs and summary format 10filters correct 20json core 20malformed and validation 20multi file and rankings 20exit codes 10
what each checkpoint verifies
  • runs and summary format: default run on sample.log exits 0 and prints the seven summary lines in order, ending with the files line (shape only)
  • filters correct: --min-level, --component, --since, --until text-mode cases on an unsorted file: the six count lines are exact (the files line is scored in c1, c4, c5)
  • json core: --json parses and counts, components, malformed, first_ts, last_ts are exact (the exact eight-key set and malformed_by_file are scored in c4, c5)
  • malformed and validation: CRLF file with calendar traps, case-sensitive levels, empty messages and blank lines is counted exactly (all seven text lines, all eight JSON keys); invalid --since/--until values exit 4 in the stated order
  • multi file and rankings: two files as one stream, a file given twice, repeated --component, --top 1/2/5 with ties, malformed_by_file keyed by file as written (zero-malformed file included), 50,000-line file within 60 s
  • exit codes: exit 2 for a missing file or no file at all, 3 for a bad level, 4 for a bad --since, 5 for --top 0 or --top x, checked in the stated order, all with empty stdout; exit 0 on success
#harness · modelresulttimeturnstoolsgen tokthinktg tok/speak ctxoutput
#1dsh
qwen38-flashnext-iq4xs
RESOLVED 100100.5 min1395616711225~10.480kdiff · transcript
checkpoints, tools, judge
✓ runs and summary format 10 ✓ filters correct 20 ✓ json core 20 ✓ malformed and validation 20 ✓ multi file and rankings 20 ✓ exit codes 10
tools: bash ×24, edit ×9, write ×3, read ×3 · tokens: · started 2026-09-16 10:08:44 · pp 126.0 tok/s
quality 4/5 · AI-graded by qwen38-27b-q4-judge at 2026-09-16 14:57 · hover a chip for the question and reason: ✓ q1 ✓ q2 ✗ q3 ✓ q4 ✓ q5
#1opencode
qwen38-flashnext-iq4xs
RESOLVED 10021.1 min1818159097721~14.631kdiff · transcript
checkpoints, tools, judge
✓ runs and summary format 10 ✓ filters correct 20 ✓ json core 20 ✓ malformed and validation 20 ✓ multi file and rankings 20 ✓ exit codes 10
tools: bash ×7, edit ×7, read ×3, write ×1 · tokens: input 24095, output 15738 · started 2026-09-16 03:58:32 · pp 145.7 tok/s
quality 3/5 · AI-graded by qwen38-27b-q4-judge at 2026-09-16 15:01 · hover a chip for the question and reason: ✗ q1 ✓ q2 ✓ q3 ✓ q4 ✗ q5
#1pi
qwen38-flashnext-iq4xs
RESOLVED 10067.4 min46454171612999~12.464kdiff · transcript
checkpoints, tools, judge
✓ runs and summary format 10 ✓ filters correct 20 ✓ json core 20 ✓ malformed and validation 20 ✓ multi file and rankings 20 ✓ exit codes 10
tools: bash ×30, edit ×8, write ×4, read ×3 · tokens: input 63057, output 39972, cache_read 1611824 · started 2026-09-16 06:53:33 · pp 141.4 tok/s
quality 4/5 · AI-graded by qwen38-27b-q4-judge at 2026-09-16 15:07 · hover a chip for the question and reason: ✓ q1 ✗ q2 ✓ q3 ✓ q4 ✓ q5

t3-shoebox

3/3 resolved · mean 100 · 35.9 min avg · timeout 25 min
consolidate four messy expense exports into one exact report.json
prompt fc5ac813a0 · commit 35a9b01 LATEST · task content 50e4eb387e LATEST
the exact prompt these runs saw
The data/ directory contains expense exports that need to be consolidated into one report.

Source files (everything else in data/ is not an expense export and must be ignored):
- data/exports/q1_expenses.csv
- data/exports/q1_expenses_reexport.csv
- data/exports/q2_expenses.csv
- data/exports/backfill.json

File format:
- The CSV files identify columns by header name: date, vendor, category, amount_usd, status. Column order differs between files. Extra columns (for example notes) are ignored.
- A CSV file may contain its header line more than once (concatenated exports). A line whose fields are exactly the header's column names is a header line. Header lines are never data rows and are never counted anywhere.
- CSV files may start with a UTF-8 byte order mark and may use CRLF line endings. Fields may be quoted and quoted fields may contain commas.
- backfill.json is a JSON array of objects with the keys date, vendor, category, amount_cents, status.
- Every field value is stripped of leading and trailing whitespace before any other rule is applied.

Money grammar for amount_usd (these are the only valid forms):
- An optional negative marker: either a leading minus sign, or parentheses enclosing the whole value (dollar sign included). Both mean negative. Never both at once.
- Then an optional dollar sign.
- Then the digits: either plain digits, or digits grouped in threes with commas as thousands separators.
- Then a decimal point and exactly two decimal digits.
- Valid: "$1,234.56", "1234.56", "$1234.56", "-$12.00", "-12.00", "(45.10)", "($45.10)", "0.75".
- Not money: "1,234.5", "N/A", "12", "$1.2.3", "12,34.00", "-(12.00)", an empty value.

Money grammar for amount_cents in backfill.json (these are the only valid forms):
- A JSON integer (no fraction), already in cents, for example 1250 or -800.
- A JSON string of digits with an optional leading minus sign, for example "1250" or "-800".
- Anything else is not money, for example "12.5", "$12.50", 12.5, null.

Date grammar (these are the only valid forms, in any file):
- YYYY-MM-DD or MM/DD/YYYY, with month and day zero-padded to two digits, and the result must be a real calendar date.
- Valid: "2026-03-05", "03/05/2026". Not dates: "2026-3-5", "3/5/2026", "02/30/2026", "2026/03/05", "sometime in march".

Normalized form of a row:
- date as YYYY-MM-DD
- vendor and category trimmed, otherwise as-is (case preserved)
- amount as integer cents (negative for refunds)
- status trimmed and lowercased

Processing rules, applied in this order:
1. A row is invalid if its amount is not money or its date is not a date. Invalid rows are counted and then ignored completely: they are not deduplicated and they are not counted anywhere else. Two identical invalid rows are two invalid rows.
2. Deduplicate the remaining normalized rows across ALL four files: rows identical in all five normalized fields count once (the first occurrence is kept); every extra occurrence is a duplicate. Two rows that differ only in status are not duplicates.
3. Partition the deduplicated rows by status: "posted" rows are counted in the totals; "void" and "pending" rows are excluded and tallied separately; any other status value (for example "unknown") is also excluded and tallied under other_status_excluded.

Write the result to report.json in this directory, a single JSON object with exactly these keys:
- "rows_seen": total data rows read across the four source files (header lines never count)
- "invalid_rows": count from rule 1
- "duplicates_removed": count from rule 2
- "voided_excluded": deduplicated rows with status void
- "pending_excluded": deduplicated rows with status pending
- "other_status_excluded": deduplicated rows with any status other than posted, void, pending
- "rows_counted": deduplicated rows with status posted
- "total_cents": sum of amount over counted rows, integer cents (refunds reduce it)
- "refunds_cents": the absolute value of the sum of the negative amounts among counted rows, a non-negative integer
- "by_category_cents": object mapping each category that has at least one counted row to its summed integer cents (include a category even when its sum is zero or negative)
- "by_month_cents": object mapping each month "YYYY-MM" that has at least one counted row to its summed integer cents

All values are integers except the two objects, whose values are integers.

Do not modify anything under data/. Leave the tree clean: report.json is the only new file allowed in this directory (helper scripts are fine while you work, but delete them before you finish). You are done when report.json is written and correct.
scoring key: how the 100 points split
report schema 10accounting counts 20totals exact 30months and refunds 20tree clean 20
what each checkpoint verifies
  • report schema: report.json exists, is one object with exactly the eleven required keys, integers and integer-valued objects
  • accounting counts: rows_seen, invalid_rows, duplicates_removed, voided/pending/other excluded and rows_counted all exact
  • totals exact: total_cents and by_category_cents exact (parentheses refunds, trimmed dupes, zero and negative categories)
  • months and refunds: by_month_cents and refunds_cents exact
  • tree clean: data/ byte-identical to the original and no file other than report.json added (only awarded if c1 passed)
#harness · modelresulttimeturnstoolsgen tokthinktg tok/speak ctxoutput
#1dsh
qwen38-flashnext-iq4xs
RESOLVED 10053.7 min1213269110102~11.554kdiff · transcript
checkpoints, tools, judge
✓ report schema 10 ✓ accounting counts 20 ✓ totals exact 30 ✓ months and refunds 20 ✓ tree clean 20
tools: bash ×21 · tokens: · started 2026-09-16 11:49:18 · pp 142.7 tok/s
quality 2/5 · AI-graded by qwen38-27b-q4-judge at 2026-09-16 14:58 · hover a chip for the question and reason: ✗ q1 ✗ q2 ✗ q3 ✓ q4 ✓ q5
#1opencode
qwen38-flashnext-iq4xs
RESOLVED 10037.2 min31302393911119~12.850kdiff · transcript
checkpoints, tools, judge
✓ report schema 10 ✓ accounting counts 20 ✓ totals exact 30 ✓ months and refunds 20 ✓ tree clean 20
tools: bash ×22, edit ×5, read ×2, write ×1 · tokens: input 49850, output 23434 · started 2026-09-16 04:19:45 · pp 140.9 tok/s
quality 5/5 · AI-graded by qwen38-27b-q4-judge at 2026-09-16 15:02 · hover a chip for the question and reason: ✓ q1 ✓ q2 ✓ q3 ✓ q4 ✓ q5
#1pi
qwen38-flashnext-iq4xs
RESOLVED 10016.7 min1618124704174~14.928kdiff · transcript
checkpoints, tools, judge
✓ report schema 10 ✓ accounting counts 20 ✓ totals exact 30 ✓ months and refunds 20 ✓ tree clean 20
tools: bash ×17, write ×1 · tokens: input 27002, output 12370, cache_read 220414 · started 2026-09-16 08:01:03 · pp 156.0 tok/s
quality 2/5 · AI-graded by qwen38-27b-q4-judge at 2026-09-16 15:07 · hover a chip for the question and reason: ✗ q1 ✗ q2 ✗ q3 ✓ q4 ✓ q5

t4-night-shift

2/3 resolved · mean 98 · 28.0 min avg · timeout 30 min
a maintenance shift: ten small chores on one tiny service, every one verified by running the code, retry tests checked against mutants
prompt d312b55227 · commit 4a0b56a LATEST · task content 6bdac0aea6 LATEST
the exact prompt these runs saw
This project needs a maintenance pass. Complete every numbered item below.

1. `python3 -m webhookd --version` prints a hardcoded old number. Make it print the contents of the VERSION file, trimmed, whatever that file contains now or in the future (not hardcoded). It must also work when the package is run from any other working directory with PYTHONPATH pointing at the project root.

2. load_config in webhookd/config.py accepts nonsense. Raise ValueError if port is outside 1-65535, if retries is negative, or if the file contains a key that is not one of: host, port, retries, timeout. timeout is a new key: parse it as a float, it must be >= 0. Every ValueError message must contain the offending key name. Valid configs keep working exactly as before (tests/test_config.py stays green).

3. legacy_parse in webhookd/parser.py is deprecated and drops multi-word payloads. Remove the function entirely and switch its one caller to parse_line.

4. webhookd/retry.py has no tests. Add tests/test_retry.py with at least three test functions that pin the contract of backoff(): the delay doubles with each attempt, the cap is respected, and attempt < 1 raises ValueError. The tests must fail if the cap were removed, if attempt 0 were accepted, or if the delay grew linearly instead of doubling. They must pass against the current retry.py.

5. Add a "## Unreleased" section at the top of CHANGELOG.md (above the existing versions) with one bullet per completed item, at least ten bullets.

6. Add run.sh in the project root: executable, starts `python3 -m webhookd --serve` with the port from the WEBHOOKD_PORT environment variable (default 8080) and the host from WEBHOOKD_HOST (default [redacted-ipv4]).

7. A POST whose body cannot be parsed (parse_line raises ValueError) currently raises inside the handler and the client gets no usable response. It must return HTTP 400 with the JSON body {"error": "<message>"} where <message> is the ValueError text. Successful POSTs keep returning 200 with the existing JSON.

8. Add a --config PATH flag to the CLI. With --serve, the server takes host and port from that file via load_config. An explicit --port or --host on the command line beats the config value. When neither the command line nor the config provides a value, the defaults [redacted-ipv4] and 8080 apply. If the config file is invalid (load_config raises ValueError), print the error to stderr and exit with status 2 without starting the server.

9. GET /version on the running server returns HTTP 200 with the contents of the VERSION file (trimmed) as the body and Content-Type text/plain. Unknown GET paths keep returning 404.

10. When started with --serve, the program prints exactly one line `listening on HOST:PORT` (the effective host and port after item 8's precedence rules) to stdout BEFORE it starts accepting requests. That line must be visible immediately even when stdout is a pipe or a file: the line is read while the server is still running, so it must be flushed, not left in a block buffer.

Do not break the existing tests. Leave the tree clean: no scratch files. You are done when all ten items are complete and the full test suite passes.
scoring key: how the 100 points split
version reads file 8config validation 12legacy removed 8retry tests mutation 15changelog 5run script 8bad payload 400 12config flag 12suite green 5version endpoint 7listening line 8
what each checkpoint verifies
  • version reads file: --version prints the VERSION file (follows edits) from the project dir and from another cwd via PYTHONPATH
  • config validation: load_config raises ValueError naming the key for port 0/70000, retries -1, timeout -1, unknown key; valid files incl. float timeout load; canonical test_config passes
  • legacy removed: legacy_parse is gone and handle_payload keeps multi-word payloads via parse_line
  • retry tests mutation: tests/test_retry.py has 3+ tests, passes on the real retry.py and fails against each mutant (no cap, attempt 0 accepted, linear growth)
  • changelog: CHANGELOG.md starts with an Unreleased section holding at least 10 bullets
  • run script: run.sh is executable, serves health on WEBHOOKD_PORT with and without WEBHOOKD_HOST set, and carries the 8080 / [redacted-ipv4] defaults
  • bad payload 400: live server answers an unparseable POST with 400 + JSON error and keeps serving 200 for valid lines
  • config flag: --serve --config takes host/port from the file, explicit --port/--host win over it, an invalid file exits 2 with stderr
  • suite green: full suite green with canonical tests restored; no project files deleted; no scratch files (new modules under webhookd/ and new test files are fine)
  • version endpoint: live GET /version answers 200 text/plain with the trimmed VERSION file, follows an edit after restart, and GET /nope still 404s
  • listening line: with stdout redirected to a file, exactly one flushed 'listening on [redacted-ipv4]:PORT' line is visible while the server runs, and it reports the --port that beat the --config port
#harness · modelresulttimeturnstoolsgen tokthinktg tok/speak ctxoutput
#1dsh
qwen38-flashnext-iq4xs
RESOLVED 10027.1 min132191257063~13.532kdiff · transcript
checkpoints, tools, judge
✓ version reads file 8 ✓ config validation 12 ✓ legacy removed 8 ✓ retry tests mutation 15 ✓ changelog 5 ✓ run script 8 ✓ bad payload 400 12 ✓ config flag 12 ✓ suite green 5 ✓ version endpoint 7 ✓ listening line 8
tools: read ×11, bash ×9, write ×9, todo_write ×2, edit ×1 · tokens: · started 2026-09-16 12:43:06 · pp 157.0 tok/s
quality 5/5 · AI-graded by qwen38-27b-q4-judge at 2026-09-16 14:59 · hover a chip for the question and reason: ✓ q1 ✓ q2 ✓ q3 ✓ q4 ✓ q5
#1opencode
qwen38-flashnext-iq4xs
RESOLVED 10020.1 min2538152747280~15.329kdiff · transcript
checkpoints, tools, judge
✓ version reads file 8 ✓ config validation 12 ✓ legacy removed 8 ✓ retry tests mutation 15 ✓ changelog 5 ✓ run script 8 ✓ bad payload 400 12 ✓ config flag 12 ✓ suite green 5 ✓ version endpoint 7 ✓ listening line 8
tools: bash ×15, read ×11, write ×6, edit ×3, todowrite ×2, grep ×1 · tokens: input 28794, output 15116 · started 2026-09-16 04:57:04 · pp 145.7 tok/s
quality 5/5 · AI-graded by qwen38-27b-q4-judge at 2026-09-16 15:03 · hover a chip for the question and reason: ✓ q1 ✓ q2 ✓ q3 ✓ q4 ✓ q5
#1pi
qwen38-flashnext-iq4xs
PARTIAL 9536.7 min34472812113002~14.939kdiff · transcript
missed suite green (5)
  • c9: stray files left in the tree: conftest.py
touched: CHANGELOG.md, conftest.py, run.sh, tests/test_cli.py, tests/test_config_validation.py, tests/test_parser_migration.py, tests/test_retry.py, tests/test_server.py …
checkpoints, tools, judge
✓ version reads file 8 ✓ config validation 12 ✓ legacy removed 8 ✓ retry tests mutation 15 ✓ changelog 5 ✓ run script 8 ✓ bad payload 400 12 ✓ config flag 12 ✗ suite green 5 ✓ version endpoint 7 ✓ listening line 8
tools: bash ×19, write ×13, read ×11, edit ×4 · tokens: input 38211, output 28071, cache_read 659732 · started 2026-09-16 08:17:51 · pp 137.9 tok/s
quality 5/5 · AI-graded by qwen38-27b-q4-judge at 2026-09-16 15:09 · hover a chip for the question and reason: ✓ q1 ✓ q2 ✓ q3 ✓ q4 ✓ q5

t5-regression

3/3 resolved · mean 100 · 9.3 min avg · timeout 30 min
Twelve commits of history; one landed a feature and quietly broke weekly totals. Find it in git, fix only the regression, name the commit.
prompt ad96eca7bf · commit d693e75 LATEST · task content 2efe25828a LATEST
the exact prompt these runs saw
tests/test_report.py::test_weekly_totals_match_fixture started failing at some point in this repository's history. The fixture is correct; the code regressed. Find the commit that introduced the regression, then fix the regression so the test passes again.

Requirements:
1. Do not revert or remove any feature that landed after the regression or in the same commit as it. Every existing test must keep passing.
2. Do not modify anything under tests/.
3. Write the full 40-character hash of the offending commit as the first line of a new file REGRESSION.md, followed by one or two sentences describing the defect.
4. Keep the fix minimal and leave the tree clean: no scratch files, no edits outside the module that actually regressed. You may commit your fix or leave it in the working tree, either is fine.

You are done when the full test suite passes and REGRESSION.md is written.
scoring key: how the 100 points split
regression test passes 30full suite passes 15features intact 20commit identified 20footprint clean 15
what each checkpoint verifies
  • regression test passes: test_weekly_totals_match_fixture passes against the canonical tests
  • full suite passes: the whole visible suite passes (nothing else broke)
  • features intact: hidden probes of --tz, --round, --project, --csv and check still pass, so nothing was reverted (needs c1)
  • commit identified: REGRESSION.md starts with the full 40-character hash of the commit that introduced the defect, followed by a description
  • footprint clean: only tally/report.py changed; tests/ untouched; nothing added besides REGRESSION.md (needs c1)
#harness · modelresulttimeturnstoolsgen tokthinktg tok/speak ctxoutput
#1dsh
qwen38-flashnext-iq4xs
RESOLVED 10012.7 min11789163798~15.226kdiff · transcript
checkpoints, tools, judge
✓ regression test passes 30 ✓ full suite passes 15 ✓ features intact 20 ✓ commit identified 20 ✓ footprint clean 15
tools: bash ×13, edit ×2, read ×1, write ×1 · tokens: · started 2026-09-16 13:10:19 · pp 149.3 tok/s
quality 5/5 · AI-graded by qwen38-27b-q4-judge at 2026-09-16 14:59 · hover a chip for the question and reason: ✓ q1 ✓ q2 ✓ q3 ✓ q4 ✓ q5
#1opencode
qwen38-flashnext-iq4xs
RESOLVED 1004.2 min101030051284~17.813kdiff · transcript
checkpoints, tools, judge
✓ regression test passes 30 ✓ full suite passes 15 ✓ features intact 20 ✓ commit identified 20 ✓ footprint clean 15
tools: bash ×8, edit ×1, write ×1 · tokens: input 13271, output 2673 · started 2026-09-16 05:17:22 · pp 157.6 tok/s
quality 5/5 · AI-graded by qwen38-27b-q4-judge at 2026-09-16 15:04 · hover a chip for the question and reason: ✓ q1 ✓ q2 ✓ q3 ✓ q4 ✓ q5
#1pi
qwen38-flashnext-iq4xs
RESOLVED 10011.2 min202288194375~17.123kdiff · transcript
checkpoints, tools, judge
✓ regression test passes 30 ✓ full suite passes 15 ✓ features intact 20 ✓ commit identified 20 ✓ footprint clean 15
tools: bash ×19, edit ×2, write ×1 · tokens: input 22385, output 8769, cache_read 277779 · started 2026-09-16 08:54:45 · pp 146.3 tok/s
quality 5/5 · AI-graded by qwen38-27b-q4-judge at 2026-09-16 15:09 · hover a chip for the question and reason: ✓ q1 ✓ q2 ✓ q3 ✓ q4 ✓ q5

t6-payroll

3/3 resolved · mean 100 · 78.4 min avg · timeout 45 min
compute a two-week payroll from a messy punch log where a dozen stated rules interact
prompt 560cf7a195 · commit 5952b62 LATEST · task content d58e0eb5a0 LATEST
the exact prompt these runs saw
The data/ directory holds a raw time-clock export for one two-week pay period. Compute the payroll for that period and write the result to report.json in this directory.

Files:
- data/employees.csv: columns employee_id, name, hourly_cents, night_eligible, meal_waiver. hourly_cents is an integer number of cents per hour. night_eligible and meal_waiver are yes or no. Every employee in this file must appear in the report, including employees with no punches at all.
- data/punches.csv: columns employee_id, punch_id, in, out, kind, note. kind is work or correction. note is free text and never affects anything. Rows are in no particular order.
- data/holidays.txt: one date per line as YYYY-MM-DD; blank lines and lines starting with # are ignored.
- Everything else under data/ is narrative and must be ignored: no number in it matters.

Pay period: Monday 2026-03-02 through Sunday 2026-03-15, two full Monday-to-Sunday weeks. Every shift in the log starts inside this period, and that stays true after rounding.

Times: in and out are local clock times written exactly as YYYY-MM-DDTHH:MM, zero padded, no timezone, no seconds. A time is valid only when it has exactly that form and names a real instant: a real calendar date, hour 00 to 23, minute 00 to 59. So 2026-02-30T08:00, 2026-03-10T24:00 and 2026-03-05T8:00 are not valid times. A shift may cross midnight (out on the next calendar day).

Processing rules, applied in this order. All arithmetic is in whole minutes until the money step.

1. Validation, duplicates, corrections, overlaps.
   a. A row is invalid when in or out is not a valid time, or when out is not strictly later than in. Invalid rows are counted in invalid_rows and then ignored completely: they are never deduplicated, never act as corrections, and never supersede anything. Two identical invalid rows are two invalid rows.
   b. Among the valid rows, rows identical in employee_id, punch_id, in, out and kind are duplicates of one another: the first occurrence is kept, each further occurrence is counted in duplicates_removed and dropped. A work row and a correction row with the same times are not duplicates.
   c. Corrections. Rows that share a punch_id always share an employee_id. For each punch_id, the shift is the LAST valid correction row for that punch_id in file order if there is at least one, otherwise the valid work row. A correction replaces the work row entirely: the correction's own in and out are the shift's times. Apart from exact duplicates, no two work rows share a punch_id. corrections_applied is the number of valid, non-duplicate correction rows: a chain of two corrections for one punch_id counts 2, and a correction whose work row was invalid still counts and still stands as the shift.
   d. Overlap. Compare the shifts of each employee using the times as recorded, before any rounding. Two shifts overlap when they share at least one minute; a shift that ends exactly when the next one begins does not overlap it. Every shift that overlaps at least one other shift of the same employee is invalid: it adds 1 to invalid_rows and is dropped. (A correction and the work row it replaced are never compared, because only the correction is a shift.)
   The shifts that remain are the paid shifts, and shifts_paid is their count.

2. Rounding. Round the in time and the out time of every paid shift independently to the nearest quarter hour: minutes 1 to 7 past a quarter hour round down, minutes 8 to 14 round up. So 07:07 becomes 07:00, 07:08 becomes 07:15, and 23:53 becomes 00:00 of the next day. Everything below uses the rounded times only. The length of a shift is its rounded out minus its rounded in, in minutes.

3. Meal deduction. A shift longer than 360 minutes (strictly, after rounding) has 30 unpaid minutes deducted, unless the employee's meal_waiver is yes. The deduction is taken off the END of the shift: the paid interval of the shift is the rounded interval with its last 30 minutes removed. Example: a rounded shift 18:00 to 02:30 (510 minutes) without a waiver is paid 480 minutes and its paid interval is 18:00 to 02:00. A rounded shift of exactly 360 minutes has no deduction.

4. Day and week. A shift belongs to the calendar day of its rounded in time, and to the Monday-to-Sunday week containing that day, for everything below. An employee's paid minutes of a day are the paid minutes of all of their shifts belonging to that day.

5. Daily overtime. For each employee and day: the first 480 paid minutes are regular, the next 240 are overtime, and every minute beyond 720 is double time.

6. Seventh day. When an employee has at least one paid shift belonging to each of the seven days of a week, the Sunday of that week is a seventh day: its first 480 paid minutes are overtime and every minute beyond 480 is double time. This replaces rule 5 for that day, so a seventh day has no regular minutes. Holidays count as days worked.

7. Weekly overtime. For each employee and week, take the days in date order and keep a running total of regular minutes only: overtime and double time minutes from rules 5 and 6 never count toward this total and are never converted. Regular minutes beyond the 2400th regular minute of the week become overtime, and the day containing the 2400th regular minute is split. Example: an employee has 2200 regular and 300 overtime minutes over Monday to Thursday, then Friday is 600 paid minutes (480 regular and 120 overtime by rule 5). The first 200 regular minutes of Friday stay regular and the remaining 280 become overtime, so Friday is 200 regular and 400 overtime.

8. Holidays. Every minute of a day that is a holiday date is paid at 1.5 times what it would otherwise be: regular minutes at 1.5, overtime minutes at 2.25, double time minutes at 3.0 times hourly_cents. The minute categories do not change, only the multiplier does. The night differential of rule 9 is not multiplied.

9. Night differential. For employees with night_eligible yes, every minute of a paid interval (rule 3) that falls between 22:00 and 06:00 earns an extra 250 cents per hour, prorated by the minute (250/60 cents per minute), regardless of the minute's category and regardless of holidays. These minutes are the employee's night_minutes. Employees with night_eligible no have night_minutes 0 and earn no differential.

10. Money. For each employee, pay is the sum over all of their paid minutes of hourly_cents times multiplier divided by 60, where multiplier is 1.0 for regular, 1.5 for overtime and 2.0 for double time, times 1.5 for minutes on a holiday, plus night_minutes times 250 divided by 60. Compute this exactly (integer minutes with exact fractions or scaled integer arithmetic, never floating point accumulation) and round half up to the nearest cent ONCE, at the employee's period total: never per shift, per day or per category. pay_cents is that rounded integer. total_pay_cents is the sum of the rounded pay_cents of all employees. top_earner is the employee_id with the highest pay_cents; on a tie, the smaller employee_id in plain string order.

Output: report.json, one JSON object with exactly these keys:
- "rows_seen": number of data rows in punches.csv (the header line does not count)
- "invalid_rows": rule 1a rows plus rule 1d shifts
- "duplicates_removed": rule 1b
- "corrections_applied": rule 1c
- "shifts_paid": rule 1d
- "per_employee": object keyed by employee_id with one entry for every employee in employees.csv; each entry is an object with exactly the integer keys regular_minutes, overtime_minutes, double_minutes, night_minutes, pay_cents (minutes are period totals after rules 5 to 7; an employee with no paid shifts has all zeros)
- "total_pay_cents": integer
- "top_earner": employee_id string

Do not modify anything under data/. Leave the tree clean: report.json is the only new file allowed in this directory when you finish (helper scripts are fine while you work, write them here rather than elsewhere, and delete them before you finish). You are done when report.json is written and correct.
scoring key: how the 100 points split
schema 10row accounting 15minutes 25pay 30tree clean 20
what each checkpoint verifies
  • schema: report.json exists, is one object with exactly the eight required keys, integer counts, and a per_employee object whose entries carry exactly the five integer fields
  • row accounting: rows_seen, invalid_rows (bad times, out before in, overlaps), duplicates_removed, corrections_applied and shifts_paid all exact
  • minutes: regular, overtime, double and night minutes exact for every roster employee (rounding, meal off the end, day by rounded start, daily/seventh-day/weekly overtime)
  • pay: pay_cents exact for every employee (holiday multiplier, night differential, exact fractions rounded half up once), plus total_pay_cents and top_earner
  • tree clean: data/ byte-identical to the original and no file other than report.json added (only awarded if c1 passed)
#harness · modelresulttimeturnstoolsgen tokthinktg tok/speak ctxoutput
#1dsh
qwen38-flashnext-iq4xs
RESOLVED 10093.4 min137481854401~9.784kdiff · transcript
checkpoints, tools, judge
✓ schema 10 ✓ row accounting 15 ✓ minutes 25 ✓ pay 30 ✓ tree clean 20
tools: bash ×25, read ×4, write ×4, edit ×4 · tokens: · started 2026-09-16 13:23:03 · pp 128.3 tok/s
quality 1/5 · AI-graded by qwen38-27b-q4-judge at 2026-09-16 15:00 · hover a chip for the question and reason: ✗ q1 ✗ q2 ✗ q3 ✗ q4 ✓ q5
#1opencode
qwen38-flashnext-iq4xs
RESOLVED 10086.9 min17164449719777~10.097kdiff · transcript
checkpoints, tools, judge
✓ schema 10 ✓ row accounting 15 ✓ minutes 25 ✓ pay 30 ✓ tree clean 20
tools: bash ×8, read ×5, edit ×2, write ×1 · tokens: input 89828, output 44244 · started 2026-09-16 05:21:39 · pp 123.1 tok/s
quality 5/5 · AI-graded by qwen38-27b-q4-judge at 2026-09-16 15:06 · hover a chip for the question and reason: ✓ q1 ✓ q2 ✓ q3 ✓ q4 ✓ q5
#1pi
qwen38-flashnext-iq4xs
RESOLVED 10054.9 min20223273611508~11.663kdiff · transcript
checkpoints, tools, judge
✓ schema 10 ✓ row accounting 15 ✓ minutes 25 ✓ pay 30 ✓ tree clean 20
tools: bash ×15, read ×4, edit ×2, write ×1 · tokens: input 62722, output 30781, cache_read 592125 · started 2026-09-16 09:06:00 · pp 144.3 tok/s
quality 1/5 · AI-graded by qwen38-27b-q4-judge at 2026-09-16 15:09 · hover a chip for the question and reason: ✗ q1 ✗ q2 ✗ q3 ✗ q4 ✓ q5

How scoring works