1 paper
Shuhao Gu, Mengdi Zhao, Bowen Zhang +3
Tokenizer is an essential component for large language models (LLMs), and a tokenizer with a high compression rate can improve the model's representation and processing efficiency.…