VihangaFTW/bytetok: A fast BPE tokenizer with a clean Python API. (opens in new tab)
A fast BPE tokenizer with a clean Python API. Contribute to VihangaFTW/bytetok development by creating an account on GitHub.
Read the original articleA fast BPE tokenizer with a clean Python API. Contribute to VihangaFTW/bytetok development by creating an account on GitHub.
Read the original article