github’s rust-gems bpe crate is quite interesting. they use aho-corasick algorithm to make a fast chunking and backtracking tokenizer.