Abstract

The tokenizer is the least glamorous component of a large language model but one with outsized influence on multilingual performance, inference cost, and model fairness. Tokenizer design decisions made before training begins - vocabulary size, training corpus language balance, subword algorithm, and normalization choices - propagate through every downstream metric. As multilingual LLMs have scaled to cover dozens or hundreds of languages, the tradeoffs in tokenizer design have become more acute and better documented. This article examines the key design dimensions, the empirically observed failure modes in low-resource language handling, and directions the community is exploring to address them.

Vocabulary Size and the Fertility Problem

Fertility, defined as the average number of tokens required to encode a unit of text (typically measured per word or per character), is the central diagnostic metric for tokenizer quality in a given language. High fertility in a target language means that the model must process more tokens to convey the same semantic content, directly increasing inference cost and filling the context window faster. For scripts underrepresented in the tokenizer training corpus, such as Tamil, Amharic, or Mongolian in many current vocabularies, fertility is often 5-10x higher than for English, which was typically dominant in tokenizer training data.

Vocabulary size is the primary lever for controlling fertility across languages, but it involves a direct tradeoff with embedding table size. A vocabulary of 32,000 tokens (standard for early LLaMA models) results in significantly higher fertility for low-resource languages than a vocabulary of 120,000-150,000 tokens (as used in Gemma and some Mistral variants). The embedding table for a 150,000-token vocabulary adds roughly 300-600M parameters at typical embedding dimensions, a non-trivial fraction of a 7B model. Scaling vocabulary too aggressively also risks vocabulary sparsity, where rare tokens receive inadequate training signal.

Corpus Balance During Tokenizer Training

The sentencepiece and tiktoken algorithms learn subword merges from a text corpus. If that corpus contains 90% English text, the resulting vocabulary will reflect English subword structure, and most tokens for other languages will be single-character or near-single-character pieces. Deliberate upsampling of low-resource languages during tokenizer training is well established as beneficial; the question is by how much. Aggressive upsampling of underrepresented scripts can allocate vocabulary slots to language-specific multi-character sequences that have low frequency in the overall pretraining corpus, producing fertile tokens that see little training signal. Balanced upsampling targets, typically aiming for a roughly uniform token-per-byte ratio across languages, have become a common heuristic in multilingual tokenizer design.

Normalization and Script Handling

Unicode normalization (NFC, NFD, NFKC) choices affect how characters with multiple valid encodings are treated before subword segmentation. For languages with complex scripts involving combining characters (Devanagari, Arabic, many Southeast Asian scripts), normalization decisions can split or merge character sequences in ways that affect fertility and semantic coherence of tokens. Some tokenizer implementations apply language-detection-based normalization, applying different normalization rules to different scripts, though this adds complexity and can behave unexpectedly when multiple scripts appear in the same document.

Byte-level fallback, used in tiktoken and GPT-4’s cl100k vocabulary, ensures that any UTF-8 input is representable without an explicit unknown token. This is important for robustness but does not improve fertility for unseen scripts; it simply ensures graceful degradation rather than hard failure.

Current Directions

Vocabulary extension after pretraining, adding new token types and training only the embedding table for a small number of steps on target-language text, is being explored as a lower-cost path to improving fertility for specific deployment languages without retraining from scratch. Dynamic vocabularies, where the token set adapts at inference time based on input language, remain largely at the research stage. The community is also developing better fertility and downstream-task benchmarks specifically for low-resource languages, driven by multilingual NLP evaluation initiatives at CMU, Johns Hopkins, and Masakhane.