Enhancing Character-Level Understanding in LLMs through Token Internal Structure Learning

Zhu Xu, Zhiqiang Zhao, Zihan Zhang +6

Tokenization methods like Byte-Pair Encoding (BPE) enhance computational efficiency in large language models (LLMs) but often obscure internal character structures within tokens. T…