From Characters to Tokens: Dynamic Grouping with Hierarchical BPE

Rares Dolga, Lucas Maystre, Tudor Berariu +1

Subword tokenization methods like Byte Pair Encoding (BPE) are widely used in large language models due to their balance of vocabulary compactness and representational power. Howev…