Defragmenting Language Models: An Interpretability-based Approach for Vocabulary Expansion

Maitrey Mehta, Nishant Subramani, Zhichao Xu +2

All languages are equal; when it comes to tokenization, some are more equal than others. Tokens are the hidden currency that dictate the cost and latency of access to contemporary…