bangla-llm-eval
A small, fixed, reproducible Bangla (Bengali) evaluation of large language models: translation both ways, natural-language inference, and Bengali-digit numerals. 270 items, one live run per model on 2026-10-17. Every number below is recomputed in CI from the committed raw model outputs.
| Model | BN→EN chrF++ | EN→BN chrF++ | NLI acc. | Numerals exact | Median latency (s) | Cost (USD) |
|---|---|---|---|---|---|---|
| Claude Opus 5.5 | 75.40 | 45.39 | 90.0% | 100.0% | 2.24 | 0.978 |
| Claude Sonnet 5 | 64.88 | 47.42 | 77.8% | 100.0% | 1.19 | 0.257 |
| GPT-6 Sol | 61.98 | 42.10 | 77.8% | 100.0% | 1.56 | 0.312 |
| Claude Haiku 4.5 | 62.63 | 48.04 | 74.4% | 60.0% | 0.65 | 0.079 |
chrF++ is corpus-level (sacrebleu, word order 2) on 60 FLORES-200 devtest sentences per direction. NLI accuracy is on 90 balanced items from the Bengali XNLI test split. Numerals is exact match on 60 items (30 arithmetic in Bengali digits, 30 Bengali-to-ASCII digit transcriptions). Cost is at list prices from the token counts the providers returned. Rows are ordered by the sum of the four quality scores.
Limits: one run per model, small per-task samples (differences of a few points are within noise), and FLORES-200 is public, so contamination cannot be ruled out. Details, raw outputs and the one-command repro are in the repository.