minbpe vs turboBPE: Two ways to think about tokenizer training (opens in new tab)

If you have spent time understanding how LLMs process text, you have probably come across Byte Pair Encoding. It is the algorithm sitting quietly under the hood of GPT, Llama, Mistral, and most other major models, turning raw text into a sequence of tokens before anything else happens. The algorithm itself is elegant. Start with a vocabulary of all 256 individual bytes. Find the most frequently occurring adjacent pair of tokens in your training text, merge it into a new single token, and repe...