Pre-trained Models Perform the Best When Token Distributions Follow Zipf's Law

Yanjin He, Qingkai Zeng, Meng Jiang

Tokenization is a fundamental step in natural language processing (NLP) and other sequence modeling domains, where the choice of vocabulary size significantly impacts model perform…