50 citations · 51 across the 2 of their papers we have counts for
1 paper · 1 filter
Jinbiao Yang
Tokenization significantly influences language models(LMs)' performance. This paper traces the evolution of tokenizers from word-level to subword-level, analyzing how they balance…