Problematic Tokens: Tokenizer Bias in Large Language Models

Jin Yang, Zhiqiang Wang, Yanbin Lin +1

Recent advancements in large language models(LLMs), such as GPT-4 and GPT-4o, have shown exceptional performance, especially in languages with abundant resources like English, than…