Users of less common languages may incur higher costs when utilizing artificial intelligence compared to speakers of English or other widely used languages. Research released last month by the Coalition for Language Equity in AI Research revealed that certain languages demand significantly more processing tokens than others in AI systems.
Tokens represent the fundamental text or data units that AI models analyze. Experts refer to this disparity as a concealed “tokenizer tax,” which raises expenses for languages with limited online representation.
A tokenizer functions as a system that decomposes text, images, or audio into these processing units. Even among the top 20 internet languages, the study identified substantial token usage disparities. The most resource-intensive languages consumed between 1.5 and 5 times more tokens than the least demanding ones.
This inequality intensified when examining 200 languages, with one AI model showing a 17-fold increase in token cost differences. The report noted, “A startup in [Bengaluru] may pay a surcharge on every prompt and completion while its competitor in Austin does not.”
Token Tax Slows AI for Minor Languages
The tokenizer tax extends beyond financial implications. Increased token requirements lead to slower processing speeds and reduced capacity to maintain conversational context. These challenges are particularly acute in regions with limited computational resources or connectivity. The inequity in tokenization creates barriers to participation in the AI-driven economy.
Non-English languages leave less room for documents or conversation history, and require more work to generate the answer, according to Pratyush Choudhury, co-founder of Activate AI, India’s first venture capital fund dedicated exclusively to AI.
Read Also: Insta360 Opens Times Square Store Amid DJI Ban
Choudhury explained, “Think about a tutor: Retaining the student’s earlier mistakes and explanations is part of delivering a good lesson. If that history consumes the context window much faster in one language, the disadvantage reaches into the quality of the service.”
Boosting Data Quality to Cut Token Gaps
Enhancing language processing capabilities begins with improving foundational data quality. Raoul Nanavati, co-founder and CEO of Navana.ai—a Mumbai-based AI voice infrastructure company—highlighted initiatives like RESPIN, a collaborative project with the Indian Institute of Science supported by the Gates Foundation.
The Coalition’s research determined that language prevalence alone doesn’t dictate token requirements. The writing system plays a key role. Asturian, despite having few speakers and no Common Crawl data, “benefits from the Latin-script advantage, as well as the representation from other related Romance languages like Italian,” while languages written in scripts the tokenizer rarely saw, Burmese, Tibetan, Tifinagh, Khmer, Lao, and Ol Chiki, were at the greatest disadvantage consistently across nearly all models in the analysis.
Many tokenizers today remain disproportionately optimized for English and other high-resource languages, and the underlying issue is the lack of high-quality digital data, said Dheemanth Reddy, CEO of Maya Research, which is developing conversational AI models designed to speak and respond like native speakers for the next 5 billion internet users.
Calls for Transparent Language Cost Scorecards
Reddy cautioned that improved training data alone won’t resolve tokenizer inefficiencies. “Token prices are falling very quickly, so the direct cost impact should become less severe over time, and better multilingual and synthetic data should improve model quality. But actually fixing the tokenization gap will still require labs to deliberately design future tokenizers to handle these languages and scripts better,” he said. Choudhury said language efficiency should be a product requirement, alongside accuracy, speed, and reliability.
Choudhury further stated, “Perfect equality may be unrealistic. Large, persistent disparities should still require an explanation and an improvement plan.” He proposed that every major AI laboratory should publish a language-by-language scorecard showing what it costs to complete comparable tasks successfully, and commit to measurable improvements for its worst-served languages.