Phonemes to the Rescue: Multilingual Tokenization Based on International Phonetic Alphabet

Multilingual language models often exhibit performance disparities across languages that can arise as early as the tokenization stage. Widely-used subword tokenization approaches favor high-resource languages, and tokenizer-free methods still yield longer sequences for scripts with a higher bytes-pe…