i should have paid a closer look at tokenizer (opens in new tab)

Well, I gave an interview sometime ago, and wasn't able to explain tokenizer properly, missed important points, and got the sweet rejection I deserved (along...