Coding task comparison

Attempts
288
Tasks
16
Categories
4
Models
6
Intervals
95% bootstrap, 2000 resamples, seed 20261021

Results describe this recorded run only. The bundled offline demo uses mock adapters, not live models; mock results are not a live-model benchmark. Hidden tests measure only their specified behaviour.

Leaderboard

Pass rates average each observed task equally. Pass@k is shown only when every observed task for that model has at least k attempts. Models with different task sets are not directly comparable. Usage columns sum only what the provider reported and say unknown when nothing was reported.

Models ranked by pass@1; select a column heading to sort.
Rank Model Pass@1 95% CI Pass@k (k > 1) Passes Mean s Median s Input tokens Output tokens Cost USD
1 claude-sonnet-5-5 100.0% 100.0–100.0% @3 100.0% 48/48 9.42 8.81 20727 55573 unknown
2 claude-opus-5-5 100.0% 100.0–100.0% @3 100.0% 48/48 12.92 11.89 20727 62037 unknown
3 claude-fable-5-1 100.0% 100.0–100.0% @3 100.0% 48/48 18.69 17.12 20727 74638 unknown
4 gpt-6-sol 97.9% 93.8–100.0% @3 100.0% 47/48 11.28 10.06 13755 38837 unknown
5 claude-sonnet-5 79.2% 60.4–93.8% @3 87.5% 38/48 27.58 14.91 20631 129892 unknown
6 claude-haiku-4-5 66.7% 47.9–85.4% @3 81.2% 32/48 9.03 8.84 15678 43707 unknown

Retries after transient provider errors (HTTP runners, recorded per attempt): claude-sonnet-5-5 0 · claude-opus-5-5 0 · claude-fable-5-1 0 · gpt-6-sol 0 · claude-sonnet-5 0 · claude-haiku-4-5 0.

Pass@1 with 95% bootstrap intervals

Task-level percentile bootstrap: tasks are resampled with replacement 2000 times (seed 20261021) and the interval is the central 95% of the resampled means. The dot is the observed pass@1; the whiskers span the interval.

  • claude-sonnet-5-5 100.0% (100.0%–100.0%)
  • claude-opus-5-5 100.0% (100.0%–100.0%)
  • claude-fable-5-1 100.0% (100.0%–100.0%)
  • gpt-6-sol 97.9% (93.8%–100.0%)
  • claude-sonnet-5 79.2% (60.4%–93.8%)
  • claude-haiku-4-5 66.7% (47.9%–85.4%)

Paired comparisons

Overlapping intervals do not settle a ranking. For every pair whose intervals overlap, a paired bootstrap over the shared tasks (same seed) estimates the difference directly.

Pass@1 difference (first minus second model) in percentage points.
PairTasksDifferencePaired 95% CIReading
claude-sonnet-5-5 − claude-opus-5-516+0.0+0.0 to +0.0no reliable difference
claude-sonnet-5-5 − claude-fable-5-116+0.0+0.0 to +0.0no reliable difference
claude-sonnet-5-5 − gpt-6-sol16+2.1+0.0 to +6.2no reliable difference
claude-opus-5-5 − claude-fable-5-116+0.0+0.0 to +0.0no reliable difference
claude-opus-5-5 − gpt-6-sol16+2.1+0.0 to +6.2no reliable difference
claude-fable-5-1 − gpt-6-sol16+2.1+0.0 to +6.2no reliable difference
gpt-6-sol − claude-sonnet-516+18.8+4.2 to +37.5differs (interval excludes 0)
claude-sonnet-5 − claude-haiku-4-516+12.5+4.2 to +22.9differs (interval excludes 0)

Pass@1 by category

Task-averaged pass@1 per category. Each cell shows the rate and passes/attempts; the colour band repeats the number.
CategoryTasksclaude-sonnet-5-5claude-opus-5-5claude-fable-5-1gpt-6-solclaude-sonnet-5claude-haiku-4-5
algorithms4100.0%12/12100.0%12/12100.0%12/12100.0%12/1266.7%8/1258.3%7/12
bug-fix4100.0%12/12100.0%12/12100.0%12/12100.0%12/12100.0%12/1283.3%10/12
parsing4100.0%12/12100.0%12/12100.0%12/1291.7%11/1266.7%8/1241.7%5/12
stateful4100.0%12/12100.0%12/12100.0%12/12100.0%12/1283.3%10/1283.3%10/12
  • 0–20%
  • 20–40%
  • 40–60%
  • 60–80%
  • 80–100%

Per-task pass matrix

Passes / attempts per task and model.
TaskCategoryclaude-sonnet-5-5claude-opus-5-5claude-fable-5-1gpt-6-solclaude-sonnet-5claude-haiku-4-5
arithmeticparsing3/33/33/33/30/30/3
fix_binary_searchbug-fix3/33/33/33/33/33/3
fix_memoizebug-fix3/33/33/33/33/32/3
fix_running_statsbug-fix3/33/33/33/33/33/3
fix_slugifybug-fix3/33/33/33/33/32/3
ini_parserparsing3/33/33/33/33/33/3
intervalsalgorithms3/33/33/33/32/31/3
json_pathparsing3/33/33/32/32/30/3
ledgerstateful3/33/33/33/33/33/3
levenshteinalgorithms3/33/33/33/30/30/3
lru_ttlstateful3/33/33/33/33/33/3
rate_limiterstateful3/33/33/33/33/33/3
romanalgorithms3/33/33/33/33/33/3
semverparsing3/33/33/33/33/32/3
topo_sortalgorithms3/33/33/33/33/33/3
undo_bufferstateful3/33/33/33/31/31/3

Timing distribution

One mark per attempt (generation plus grading), on a shared scale from 0 to 96.98 seconds.

  • claude-sonnet-5-5 8.81 s (18.83 s)
  • claude-opus-5-5 11.89 s (23.40 s)
  • claude-fable-5-1 17.12 s (53.26 s)
  • gpt-6-sol 10.06 s (23.94 s)
  • claude-sonnet-5 14.91 s (96.98 s)
  • claude-haiku-4-5 8.84 s (19.05 s)