What 200+ languages cost on AI tokenizers

We sent the same 1,012 sentences, in 221 languages and English, through nine tokenizers, including Claude, Gemini 3 and Gemini 4. Saying the same thing costs anywhere from about the same as English to more than seventeen times as many tokens, depending on the language and the model.

By Dr Santosh Kumar, Token Bench research, November 2026. Findings, by writing system, every language, method, download the data.

On the map

Each circle is one language, placed roughly where it is mainly spoken. The number is the average premium on the four commercial tokenizers (Claude, Gemini 3, Gemini 4 and GPT), so a few figures are lower than a single tokenizer’s maximum quoted elsewhere on this page. The maps show 99 of the 221 languages; the table has them all.

What we found

No tokenizer is cheapest everywhere

Among the four commercial tokenizers, OpenAI’s o200k is the cheapest for 97 of the 221 languages, Claude’s for 57, Gemini 3’s for 38 and Gemini 4’s for 29. Google’s two tokenizers share the lowest average premium, 2.11×, while o200k has the lowest median, 1.77×: it is the best choice for a typical language and weaker on rare scripts. Which model is cheapest depends on your language, so the table below shows it for each one.

Gemini 4 reshuffles which languages are cheap

Google’s new Gemini 4 Argon tokenizer uses almost exactly as many tokens as Gemini 3 across all 221 languages combined (0.2% fewer), but it moves individual languages a long way in both directions. It is much cheaper for several rare scripts: Manipuri in Meitei Mayek needs 48% fewer tokens, N’Ko 44% fewer, Mongolian script 37% fewer and Odia 29% fewer. It is dearer for much of South and Southeast Asia and the Horn of Africa: Sinhala 41% more, Burmese 34%, Amharic 33%, Lao 32%, Khmer 30%, Bengali 25% and Tamil 21%. Latin-script languages barely move. Gemini 4’s price doubles to $4 / $20 per million tokens on 1 December 2026.

The highest costs fall on languages with their own scripts

The most expensive languages are written in scripts that few tokenizers have learned: Chakma (up to 17.52×), Mongolian script (up to 16.20×), Santali in Ol Chiki, Manipuri in Meitei Mayek, N’Ko and Tifinagh. Several are languages of India: Santali and Manipuri are both among India’s scheduled languages, and Chakma is spoken in India’s north-east and in Bangladesh.

The script matters more than the language

Some languages appear twice in the corpus, written in two scripts. On Claude, Gemini 3 and GPT, Manipuri costs about 3× English in Bengali script but 9 to 12.5× in Meitei Mayek; Mongolian about 2× in Cyrillic but 11 to 16× in Mongolian script; and Tamasheq about 2× in Latin but 5 to 10× in Tifinagh. Same language, same sentences: the difference is entirely the writing system. Gemini 4 narrows these gaps but doesn’t close them: Meitei Mayek still costs 4.9× and Mongolian script 7.2×.

Chinese, Japanese and Korean are close to English

Mandarin costs only 1.0 to 1.4× English on the four commercial tokenizers, Korean 1.3 to 1.6× and Japanese 1.1 to 1.7×. These scripts are well represented in the text tokenizers were built from, and each character carries a lot of meaning.

The Latin alphabet isn’t free either

The 134 languages written in Latin script average about 1.8× English on the commercial tokenizers. Most of them are smaller languages, such as Yoruba or Quechua, whose words the tokenizers rarely saw, so they are split into more pieces even though the letters are familiar.

Claude is strong on Latin, Cyrillic and Chinese, weakest on Indian scripts

Claude is competitive for Latin, Cyrillic and Chinese-family text. For Indian scripts it costs the most of the four: Punjabi averages 6.47× on Claude against 2.40× to 2.69× on Gemini and 2.62× on GPT. Our Indian languages study looks at this in detail.

By writing system

Each cell is the average premium for the languages written in that script: how many times more tokens they need than English for identical content. Writing systems are ordered from most to least expensive on average, and tokenizers from cheapest to most expensive.

Every language

Search by language, script or code, and click a column heading to sort. The lowest premium in each row is highlighted, and the last column shows which of the four commercial tokenizers is cheapest for that language.

Method

The corpus is the devtest split of FLORES+: 1,012 sentences professionally translated into each language and aligned line by line, so every language expresses identical content. We used every FLORES+ variety whose file had exactly as many sentences as English: 221 varieties plus English. Where FLORES+ gives two varieties the same name, such as Valencian and standard Catalan, we label them separately. For each tokenizer we counted the tokens needed for the whole set in each language and divided by the count for English.

Open tokenizers were run locally, sentence by sentence. Claude and Gemini publish no tokenizer, so we measured them through each provider’s free token-counting endpoint. To keep the run practical at this scale, each language’s sentences were sent as one request with a line break between sentences. For every language we also counted five sentences one by one to measure what a line break costs; in all 666 measurements it was exactly one token, so we subtracted one token per line break, which gives the same totals as counting each sentence separately. The fixed overhead each endpoint adds to every request was measured and subtracted too. As a check, the Indian languages in this study match our Indian languages study exactly.

Measured between 31 October and 1 November 2026.

Tokenizer	How measured	Exact source
Gemini 3	Provider counting API	gemini-3.8-flash (countTokens)
Gemini 4	Provider counting API	gemini-4-argon (countTokens)
Gemma 2	Open tokenizer, run locally	mlx-community/gemma-2-2b
Claude	Provider counting API	claude-sonnet-5-5 (count_tokens)
GPT o200k	Open tokenizer, run locally	tiktoken 0.14.0, o200k_base
Qwen 2.5	Open tokenizer, run locally	Qwen/Qwen2.5-0.5B
Llama 3	Open tokenizer, run locally	Xenova/llama3-tokenizer
Mistral	Open tokenizer, run locally	Xenova/mistral-tokenizer-v1
GPT cl100k	Open tokenizer, run locally	tiktoken 0.14.0, cl100k_base

Limitations

• Gemini was measured on gemini-3.8-flash, which gave counts identical to gemini-3.6-flash on every Indian language, and Claude on claude-sonnet-5-5, which counts identically to claude-sonnet-5. Gemini 4 was measured on gemini-4-argon, which uses a new tokenizer.
• FLORES+ is formal, news-domain text. Everyday writing, transliterated text and code-mixed text may give different premiums.
• Language and script names follow FLORES+ and ISO 15924. The Mongolian-script row is labelled Halh Mongolian, as in FLORES+.
• Community copies of open tokenizers may differ marginally from the versions vendors run in production.

Data

world-languages-premiums.csv has every premium on this page, with total token counts for every language and tokenizer: 1,998 rows. It’s released under CC BY-SA 4.0, matching the corpus it was measured on. It contains measurements only. The FLORES+ sentences themselves aren’t included: the dataset’s terms ask that they not be re-hosted where web crawlers can reach them.