Leaderboard
Pass rates average each observed task equally. Pass@k is shown only when every observed task for that model has at least k attempts. Models with different task sets are not directly comparable. Usage columns sum only what the provider reported and say unknown when nothing was reported.
| Rank | Model | Pass@1 | 95% CI | Pass@k (k > 1) | Passes | Mean s | Median s | Input tokens | Output tokens | Cost USD |
|---|---|---|---|---|---|---|---|---|---|---|
| 1 | claude-sonnet-5-5 | 100.0% | 100.0–100.0% | @3 100.0% | 48/48 | 9.42 | 8.81 | 20727 | 55573 | unknown |
| 2 | claude-opus-5-5 | 100.0% | 100.0–100.0% | @3 100.0% | 48/48 | 12.92 | 11.89 | 20727 | 62037 | unknown |
| 3 | claude-fable-5-1 | 100.0% | 100.0–100.0% | @3 100.0% | 48/48 | 18.69 | 17.12 | 20727 | 74638 | unknown |
| 4 | gpt-6-sol | 97.9% | 93.8–100.0% | @3 100.0% | 47/48 | 11.28 | 10.06 | 13755 | 38837 | unknown |
| 5 | claude-sonnet-5 | 79.2% | 60.4–93.8% | @3 87.5% | 38/48 | 27.58 | 14.91 | 20631 | 129892 | unknown |
| 6 | claude-haiku-4-5 | 66.7% | 47.9–85.4% | @3 81.2% | 32/48 | 9.03 | 8.84 | 15678 | 43707 | unknown |
Retries after transient provider errors (HTTP runners, recorded per attempt): claude-sonnet-5-5 0 · claude-opus-5-5 0 · claude-fable-5-1 0 · gpt-6-sol 0 · claude-sonnet-5 0 · claude-haiku-4-5 0.
Pass@1 with 95% bootstrap intervals
Task-level percentile bootstrap: tasks are resampled with replacement 2000 times (seed 20261021) and the interval is the central 95% of the resampled means. The dot is the observed pass@1; the whiskers span the interval.
- claude-sonnet-5-5 100.0% (100.0%–100.0%)
- claude-opus-5-5 100.0% (100.0%–100.0%)
- claude-fable-5-1 100.0% (100.0%–100.0%)
- gpt-6-sol 97.9% (93.8%–100.0%)
- claude-sonnet-5 79.2% (60.4%–93.8%)
- claude-haiku-4-5 66.7% (47.9%–85.4%)
Paired comparisons
Overlapping intervals do not settle a ranking. For every pair whose intervals overlap, a paired bootstrap over the shared tasks (same seed) estimates the difference directly.
| Pair | Tasks | Difference | Paired 95% CI | Reading |
|---|---|---|---|---|
| claude-sonnet-5-5 − claude-opus-5-5 | 16 | +0.0 | +0.0 to +0.0 | no reliable difference |
| claude-sonnet-5-5 − claude-fable-5-1 | 16 | +0.0 | +0.0 to +0.0 | no reliable difference |
| claude-sonnet-5-5 − gpt-6-sol | 16 | +2.1 | +0.0 to +6.2 | no reliable difference |
| claude-opus-5-5 − claude-fable-5-1 | 16 | +0.0 | +0.0 to +0.0 | no reliable difference |
| claude-opus-5-5 − gpt-6-sol | 16 | +2.1 | +0.0 to +6.2 | no reliable difference |
| claude-fable-5-1 − gpt-6-sol | 16 | +2.1 | +0.0 to +6.2 | no reliable difference |
| gpt-6-sol − claude-sonnet-5 | 16 | +18.8 | +4.2 to +37.5 | differs (interval excludes 0) |
| claude-sonnet-5 − claude-haiku-4-5 | 16 | +12.5 | +4.2 to +22.9 | differs (interval excludes 0) |
Pass@1 by category
| Category | Tasks | claude-sonnet-5-5 | claude-opus-5-5 | claude-fable-5-1 | gpt-6-sol | claude-sonnet-5 | claude-haiku-4-5 |
|---|---|---|---|---|---|---|---|
| algorithms | 4 | 100.0%12/12 | 100.0%12/12 | 100.0%12/12 | 100.0%12/12 | 66.7%8/12 | 58.3%7/12 |
| bug-fix | 4 | 100.0%12/12 | 100.0%12/12 | 100.0%12/12 | 100.0%12/12 | 100.0%12/12 | 83.3%10/12 |
| parsing | 4 | 100.0%12/12 | 100.0%12/12 | 100.0%12/12 | 91.7%11/12 | 66.7%8/12 | 41.7%5/12 |
| stateful | 4 | 100.0%12/12 | 100.0%12/12 | 100.0%12/12 | 100.0%12/12 | 83.3%10/12 | 83.3%10/12 |
- 0–20%
- 20–40%
- 40–60%
- 60–80%
- 80–100%
Per-task pass matrix
| Task | Category | claude-sonnet-5-5 | claude-opus-5-5 | claude-fable-5-1 | gpt-6-sol | claude-sonnet-5 | claude-haiku-4-5 |
|---|---|---|---|---|---|---|---|
| arithmetic | parsing | 3/3 | 3/3 | 3/3 | 3/3 | 0/3 | 0/3 |
| fix_binary_search | bug-fix | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 |
| fix_memoize | bug-fix | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 | 2/3 |
| fix_running_stats | bug-fix | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 |
| fix_slugify | bug-fix | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 | 2/3 |
| ini_parser | parsing | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 |
| intervals | algorithms | 3/3 | 3/3 | 3/3 | 3/3 | 2/3 | 1/3 |
| json_path | parsing | 3/3 | 3/3 | 3/3 | 2/3 | 2/3 | 0/3 |
| ledger | stateful | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 |
| levenshtein | algorithms | 3/3 | 3/3 | 3/3 | 3/3 | 0/3 | 0/3 |
| lru_ttl | stateful | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 |
| rate_limiter | stateful | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 |
| roman | algorithms | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 |
| semver | parsing | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 | 2/3 |
| topo_sort | algorithms | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 |
| undo_buffer | stateful | 3/3 | 3/3 | 3/3 | 3/3 | 1/3 | 1/3 |
Timing distribution
One mark per attempt (generation plus grading), on a shared scale from 0 to 96.98 seconds.
- claude-sonnet-5-5 8.81 s (18.83 s)
- claude-opus-5-5 11.89 s (23.40 s)
- claude-fable-5-1 17.12 s (53.26 s)
- gpt-6-sol 10.06 s (23.94 s)
- claude-sonnet-5 14.91 s (96.98 s)
- claude-haiku-4-5 8.84 s (19.05 s)