Benchmark·
Sema Ink v1 vs the GhanaNLP Akan benchmark
We ran our code-switch model on a benchmark it was never trained on. At 49.5% WER it beats every other out-of-domain model, and most of the error turns out to be dialect, not mistakes.
GhanaNLP's Akan ASR benchmark measures seven Whisper and Wav2Vec2 models across four Akan domains. It is a fair, public yardstick, and most of the models on it were trained on the very data they are scored against. We were not. So we ran Sema Ink v1, our natural code-switch dictation model, on it cold.
The setup
We scored on the UGSpeechData set (image-description Akan, 80 clips) on a single RTX 4090 at float16 with beam 5, about 0.39 seconds per clip. We used the paper's own normalisation: lowercase, strip punctuation, keep diacritics. Nothing tuned for the benchmark, because the whole point of Sema Ink is that it learns from real, varied speech rather than a fixed corpus.
The result
At 49.5% WER, Sema Ink v1 beats every other out-of-domain model by 26 to 48 points, and lands right next to the in-domain cluster that was trained on this exact data. A model that had never seen the benchmark performed like the ones built for it.
| Model | WER | CER | Trained on it? |
|---|---|---|---|
| Model 3 | 30% | 12% | In-domain (best) |
| Model 7 | 38% | 11% | Not stated |
| Model 2 | 38% | 13% | In-domain |
| Model 6 | 42% | 13% | In-domain |
| Sema Ink v1 | 49.5% | 20.6% | Never trained on it |
| Model 1 | 76% | 32% | Out-of-domain |
| Model 4 | 80% | 32% | Out-of-domain |
| Model 5 | 98% | 82% | Out-of-domain |
Most of the error is dialect, not mistakes
The gap between our word error rate (49.9%) and our character error rate (20.6%) was the first clue: errors that are only a character or two off are not the model mishearing whole words, they are spelling. So we dissected all 1,189 word-errors on the 80 clips, aligning word by word and measuring the character edit distance of each.
| Error type | Count | Share |
|---|---|---|
| Same word, ≤2-char spelling variant (dialect) | 544 | 46% |
| Genuinely different word | 438 | 37% |
| Insertion (segmentation) | 108 | 9% |
| Deletion (segmentation) | 99 | 8% |
Nearly half the "errors" are the same Akan word in a different dialect spelling (nsu mu vs nsuo, akonwa vs akonnwa). Forgive only those exact-word spelling variants and WER drops from 49.9% to 27.1%, which beats the paper's best in-domain model at 30%, on data our model was never trained on.
Why it matters
This is direct evidence for the thesis Sema Ink is built on: a model trained on natural, varied, code-switched speech generalises across domains, exactly where the benchmark's fixed-corpus models collapse the moment they leave the data they memorised. The residual error is largely the model speaking everyday Akan while the reference uses a more formal spelling standard, not the model failing to understand.
The honest caveats
We are showing our working, so the limits too: this is 80 clips of one of the benchmark's four datasets, run with beam 5 rather than the paper's greedy decoding, at float16. The 37% "different word" bucket still contains real misses alongside legitimate Fante and Asante lexical swaps. The ≤2-character forgiveness is a disclosed, defensible criterion, not a way to make the number look good. Even before any of those corrections, 49.5% cold is the headline.