SentencePiece: Unsupervised Tokenization You Can Train on Raw Text
Kudo and Richardson built a language-independent trainer for subword models (BPE and unigram) that sits on raw sentences, not whitespace-tokenized words. Why tokenizer choice is a model API.
2 min read