The price of a sentence: why models do worse outside English

Barakaeli Lawuo, Jun 27, 2026

In 2023, Aleksandar Petrov and three colleagues at the University of Oxford took FLORES-200, a set of 2,000 Wikipedia sentences that professional translators rendered into 200 languages, and ran every version through cl100k_base, the tokenizer shared by ChatGPT and GPT-4.¹ Because the sentences say the same thing in every language, any difference in length comes from the language and not from the content. Portuguese came out closest to English and still needed 1.48 times as many tokens. Standard Arabic needed 3.04 times as many. Shan, spoken in the Shan State of Myanmar, needed 15.05 times as many.¹ One Shan word for "you" is a single consonant carrying three diacritics, and cl100k_base splits it into 9 tokens. The English word "you" is 1 token.¹

That ratio is a price. Commercial APIs bill per token, context windows are measured in tokens, and a model writes its answer one token at a time. So the same sentence costs more, fits less, and takes longer depending on the language it is written in. Then, on top of that, the model tends to answer it worse. This post follows the ratio through each of those effects, using the papers that measured them.

Token

The unit a language model reads and writes. A tokenizer cuts text into tokens from a fixed vocabulary; a common English word is often one token, a rare word or an unfamiliar script can take many.

Parallel corpus

The same set of sentences translated into many languages. FLORES-200 is one. It lets you compare lengths across languages while holding the meaning fixed.

Tokenization premium

How many tokens a tokenizer needs for a text in language A, divided by how many it needs for the same text in language B. English is usually B, so a premium of 3 means three times the tokens of English.

Fertility

The average number of tokens a tokenizer produces per word. It is a related measure, but it needs word boundaries, which some languages do not mark with spaces.

Resource level

A rough label for how much text in a language exists in training data. A high-resource language has a large share of the corpus, a low-resource language a tiny one.

Zero-shot and few-shot

Zero-shot means asking the model to do a task from an instruction alone. Few-shot (in-context learning) means putting a handful of solved examples in the prompt first.

Where the extra tokens come from

Petrov et al. define the premium in the plainest way possible. Take a sentence $s_A$ in language A and its translation $s_B$ in language B, tokenize both with the same tokenizer $t$, and divide the lengths.¹

Here $t(s_A)$ is the list of tokens the tokenizer produces for the sentence, and the bars $|\cdot|$ mean its length. The numerator is what language A pays, the denominator is what B pays for the same meaning. Nothing about the model enters the formula. The disparity exists before the model has read a single token, which is the paper's main point.¹

Why would the same meaning need more tokens? The paper measured the same corpus at three levels: Unicode characters (the CANINE model reads one codepoint per token), UTF-8 bytes (ByT5 reads one byte per token), and subword tokens.¹ That suggests a way to split the premium into parts. The split below is my own bookkeeping, not an equation from the paper, but every number that goes into it is from the paper.

The first factor is how many characters a language uses to say something. It is a property of the language and its writing system. For Shan the character premium over English is 1.42; for Simplified Chinese it is 0.34, because each Chinese character carries a lot of meaning.¹ The second factor is how many bytes each character takes in UTF-8, the standard way text is stored. ASCII letters, which cover English, take one byte. Other Latin letters, Greek, Cyrillic, Arabic and Hebrew take two, and Chinese, Japanese and Korean characters take three. Every Shan consonant and diacritic takes three.¹ The byte premium over English that the paper reports is 3.94 for Shan and 0.93 for Chinese.¹

The third factor is how many bytes the tokenizer's vocabulary manages to glue into each token, and this is where training data comes in. A subword vocabulary is learned from a corpus, so it holds long, frequent chunks of whatever that corpus contained most. Dividing the paper's numbers (my arithmetic): cl100k_base spends about 3.8 times as many tokens per byte on Shan as on English (15.05 / 3.94), and about 2.1 times as many on Chinese (1.91 / 0.93). Put the three Shan ratios together, 1.42 × 2.77 × 3.82, and you get back the 15.05.¹ Fewer letters do not save Chinese. Three bytes per character, and a vocabulary that packs fewer Chinese bytes into each token, add back more than the compact script saves.

The corpus effect shows up in odd places. GPT-2's vocabulary contains a dedicated token for BuyableInstoreAndOnline, probably from an online store's backend, while the common Arabic word for "why" is broken into one token per letter.¹ In cl100k_base, more than 65% of Japanese kanji still need three tokens each.¹ Ahia et al. at the University of Washington and Carnegie Mellon checked whether this is only about data. They trained byte-level tokenizers on parallel text, one language per script, with the content and data size held equal and vocabularies from 5,000 to 50,000 entries. The disparity remained at every size, and most scripts suffered more than Latin and Hangul when the vocabulary was small.² Their conclusion: the premium comes partly from each language's share of the tokenizer's training data and partly from properties of the language and its script.²

What a 15x premium does to a bill, a window and a wait

Cost is the direct one. When a service charges per token, the premium is the price multiplier. Petrov et al. put German and Italian at about 50% more than English on ChatGPT and GPT-4, and Dzongkha, Odia, Santali and Shan at more than 12 times.¹ Pricing per character does not fix it, because Burmese, Dzongkha, Shan, Tok Pisin and Tumbuka need more than 4 times as many characters as Yue Chinese for the same text.¹ Ahia et al. went past the corpus and measured what they actually spent running real tasks through gpt-3.5-turbo. On the XL-Sum summarization benchmark, prompt plus generated tokens cost up to about 4 times as much in Telugu and Amharic as in English.²

To make the multiplier concrete, here is some arithmetic of my own using the paper's premiums. Say a support queue spends $100 a month on input tokens in English. The same traffic in Italian costs about $164, in Standard Arabic about $304, in Shan about $1,505. That assumes the answers carry the same premium as the questions, which the corpus numbers suggest but no paper here measured for chat traffic.

Context is the second effect. A window has a fixed number of tokens, so a premium shrinks how much content fits. Petrov et al. note that in Burmese or Dzongkha you can fit less than a tenth of the content you could fit in English.¹ More arithmetic of mine: gpt-3.5-turbo at the time had a 4,096-token limit for prompt and output together.² Divided by the premiums, that holds about 2,770 English tokens' worth of meaning in Portuguese, about 1,350 in Standard Arabic, and about 270 in Shan. Ahia et al. saw the practical result on XL-Sum: for most Telugu and Amharic test articles, not even one worked example fit next to the article, so those languages could only be run zero-shot.² Few-shot examples raised scores for most tasks and languages they could test, so those languages lost that help because of token counts alone.²

Latency is the third. Petrov et al. timed RoBERTa over FLORES-200 and found processing time grew roughly linearly with tokenized length. English sat in the fast, short corner. Shan had the longest encoding and took almost twice as long.¹ That measurement is for reading input. For writing output my reasoning goes further: a model generates one token per decoding step, so an answer with 15 times the tokens needs 15 times the steps. I have not seen that measured across languages in these papers.

The accuracy gap follows the data, mostly

Cost would matter less if the expensive languages got equal answers. They do not. The training mix is lopsided to begin with: the MEGA paper from Microsoft cites documentation that about 93% of the tokens in GPT-3's pre-training data were English.⁴ Lai et al. at the University of Oregon and Adobe Research sorted 37 languages by their share of the CommonCrawl web corpus: high resource above 1%, medium above 0.1%, low above 0.01%, and extremely low below that. English was 45.88%. Hindi, with 602 million first- and second-language speakers, was 0.1588%. Swahili was 0.0077%.³

They then tested ChatGPT zero-shot on seven tasks. One is XNLI, where the model reads two sentences and says whether the first entails, contradicts or is neutral to the second.³ The XNLI data in other languages was translated from English, so the test items mean the same thing everywhere.³

With English instructions, accuracy fell from 70.2% in English to 43.7% in Urdu, and a fine-tuned mT5-XXL model beat ChatGPT in every language by a wide margin.³ Lai et al. report that the gap between ChatGPT and the fine-tuned model looked smaller for high-resource languages.³ The summarization run adds a reliability angle. On XL-Sum, ChatGPT returned a non-empty summary for 99% of English requests but only 42% of Bengali and 43% of Burmese requests.³

The trend is not a clean staircase, and the chart shows one exception: Swahili, extremely low resource, scored 50.3% while Urdu and Thai scored lower. Lai et al. list other cases. ChatGPT tagged parts of speech better in Urdu than in Vietnamese or Thai, and summarized better in Kyrgyz than in Hindi with in-language prompts.³ They conclude that data size is probably not the only factor, and that the task and a language's closeness to the dominant training languages matter too.³

MEGA, which covered 16 datasets and 70 languages, tried to pin down those factors for GPT-3.5 and GPT-4.⁴ It found the OpenAI tokenizers were far worse for low-resource, non-Latin-script languages, with fertility near 10 tokens per word for Malayalam and Tamil, enough that the tokenizer essentially works byte by byte.⁴ On six tasks, GPT-3.5-Turbo's score fell as fertility rose, with statistically significant correlations. On four tasks, scores rose with the language's amount of pre-training data (GPT-3's language mix stood in, since GPT-3.5's is not public).⁴ The two factors are not the same thing. French and Japanese get similar fertility from these tokenizers, yet GPT-3.5-Turbo scored 72.1% in French and 67% in Japanese on the PAWS-X paraphrase task. GPT-3's data had roughly 3.5 billion French tokens and 214 million Japanese ones.⁴ The authors say plainly that these are correlations and may not be causes.⁴ GPT-4 narrowed the gap between English and other languages to some extent, but did not close it.⁴

What the papers found that narrows it

The cheapest fix they measured is to translate the input into English. MEGA calls this translate-test: machine-translate the test example with Bing Translator, then prompt the model in English.⁴ For GPT-3.5-Turbo, the average relative gain over prompting in the original language was more than 30% for Burmese, Tamil and Telugu, while high-resource languages scored about the same either way.⁴ GPT-4 scored 77.6% on Burmese story completion (XStoryCloze) when prompted in Burmese and 93.2% with translate-test.⁴ It does not reach English. GPT-3.5-Turbo on Urdu XNLI went from 49.1% to 54% with translation, against 76.2% in English.⁴ My own addition: translating into English also shortens the prompt, since English has the lowest premium, but MEGA does not measure that saving.

The prompt language matters as well. Lai et al. found English instructions beat in-language instructions for most tasks and languages, sometimes by a lot: Hindi XNLI went from 5.6% with Hindi instructions to 48.8% with English ones.³ The worked examples are a different story. When MEGA gave GPT-3.5-Turbo English examples for Quechua and Haitian Creole items, the model often replied, "I'm sorry, but the premise is not in a language that I understand." With examples in the language itself, that almost never happened.⁴ My reading of the two papers together is that the model should get its instructions in English and see examples in the user's language.

The deeper fixes change the tokenizer. Rust et al. trained models on the same data with two tokenizers, a dedicated one for the language and the multilingual mBERT one. The dedicated tokenizer won in 38 of 48 combinations of task, model and language. Swapping only the tokenizer improved the original mBERT in 20 of 24 language and task settings.⁵ Petrov et al. argue a fair tokenizer has to be built into the model from the start: a separate tokenizer used just for billing would fix the price but leave latency and context untouched.¹ They propose training one tokenizer per language, then merging them, starting from the 256 byte values and repeatedly adding the most frequent token of whichever language currently has the highest premium.¹ English would lose less than you might fear. Keeping only a third of cl100k_base's vocabulary would make English sequences just 10% longer, by their measurement.¹ Ahia et al. also point to balancing languages in tokenizer data, as BLOOMZ does, to longer context windows, and to pricing that accounts for language and region.²

The languages nobody can score yet

Every accuracy number above depends on a benchmark existing in that language, and here the MEGA authors name a limit their method cannot get around. Even if they evaluated every multilingual dataset available, those datasets leave out many typologically diverse, under-resourced languages. There is very little from African languages or the Indigenous languages of the Americas in any benchmark available today.⁴ Petrov et al. add a caution about their own ruler: FLORES-200 contains many English-centric names and institutions, which may tilt the premiums in English's favor.¹ The premium can be computed for any language with a translated sentence. For the languages where it is highest, the matching accuracy number mostly has not been measured.