Models speak 30 languages but keep failing the culture test
Cohere's 6,200-riddle benchmark finds models hallucinate rather than admit cultural ignorance, while a new preprint shows post-training quietly favors some dialects over others.
- Cohere's CulturalRiddles benchmark contains 6,200 riddles in 51 languages from 45 countries; the company says 28 models were tested and the best reached 79% accuracy. Self-reported m:…3739
- Cohere says models answered "I don't know" only 0.7% of the time while about 31% of their answers were hallucinations. Self-reported m:…3739
- Providing English translations lifted accuracy for the hardest-to-solve communities only from 22% to 32%, and models still defaulted to Western-centric answers, Cohere reports. Self-reported m:…3739
- In a preprint, researchers report that DPO updates favor matching British and Australian spelling over AAE and Singlish across 10 model lineages, and that chat models answer in Standard English regardless of the user's dialect. Self-reported m:…2110
- The Qwen team reports that Qwen-Audio-3.1-ASR, a family of audio-language models built on Qwen-Audio-3.0-ASR, supports 30 languages and 16 Chinese dialects, with speaker-diarized transcription. Self-reported m:…4068
Speaking isn't understanding
Cohere's researchers have developed CulturalRiddles, a benchmark built to test AI models' cultural understanding. The company says 194 volunteers from 62 cultural communities contributed 6,200 riddles in 51 languages from 45 countries, and that 28 models were put through the test. The best score was 79% accuracy. m:…3739
The misses matter more than the score. The models answered "I don't know" only 0.7% of the time, while roughly 31% of their answers were hallucinations. Handing them English translations of the riddles barely helped: accuracy on the hardest-to-solve communities rose only from 22% to 32%. Even with cultural context supplied, the models kept defaulting to familiar, Western-centric answers. m:…3739
The benchmark reaches into languages that have never appeared in AI evaluations before, including Buginese, Mongondow, Konkani and Sudanese Arabic. This suggests the failure is not a language problem at all. A riddle is a compressed cultural artifact: it assumes a shared world. Translation does not supply one, and no amount of translation budget fixes that. m:…3739
The dialect tax
A preprint posted this week finds a second, related bias much closer to home. The researchers report that supervised fine-tuning, DPO and reinforcement learning progressively standardize model replies to informal cues, and that chat models tend to answer in Standard English no matter which dialect the user writes in. m:…2110
The standardizing is not neutral. Across 10 open model lineages, the authors report that DPO updates favored matching British and Australian spelling over AAE and Singlish. They propose Calibrated Accommodation Pairs, training pairs that reward matching a user's dialect density while leaving Standard-English users' output alone. They also warn that uniform replies can hide selective updates, so an audit has to read the training update, not just the output. m:…2110
Two details sharpen the picture. The authors report that CAP raises convergence on dialect features it was not even trained on, without any detectable increase in dialect for Standard-English users. And they found that one-sided preference pairs reach similar gains only by overshooting the user and imposing dialect, which suggests accommodation done badly is just stereotyping. m:…2110
Thirty languages, one blind spot
Meanwhile the language-coverage race continues. A technical report from the Qwen team this week describes Qwen-Audio-3.1-ASR, a family of audio-language models built on its 3.0 predecessor, supporting 30 languages and 16 Chinese dialects. Its speaker-diarized transcription produces structured outputs with speaker labels, timestamps and recognized text, including overlapping speech. m:…4068
Put the three together and the headline writes itself: coverage is winning, comprehension is losing. A model family can support 30 languages and still fail a riddle from a culture it has never read about. And a model that politely answers every user in Standard English is quietly deciding whose speech counts as the default. For anyone reporting on AI in non-Western markets, the question to ask vendors is no longer how many languages they support. It is whose culture the model was measured against, and how. m:…3739 m:…2110 m:…4068
What to watch
The practical edge of all three findings is measurement. CulturalRiddles gives reporters a number to demand: not a language count but a culture-aware score. The CAP paper gives auditors a place to look: the preference updates, not just the chat logs. And the Qwen report is a reminder that the gap is structural. Speech coverage and cultural coverage are different products, built on different data, and only one of them is scaling. m:…3739 m:…2110 m:…4068
What is still unresolved
- Whether training on lived cultural experience is possible at scale, rather than on static facts
- Whether Calibrated Accommodation Pairs generalize beyond the 10 open model lineages studied
- How dialect accommodation interacts with instruction-following quality in production chat products
Provenance by Synorb. Every claim links to its source. This story was written by an agent from the Manifests listed below. Read the methodology and limitations, or send a correction.
Free to reuse, including commercially. Credit “Hanging Context, with provenance by Synorb” and link to this story. Keep the cited Manifest ids. Reuse terms v1.