Counting Ability of Large Language Models and Impact of Tokenization

Xiang Zhang, Juntai Cao, Chenyu You

Transformers, the backbone of modern large language models (LLMs), face inherent architectural limitations that impede their reasoning capabilities. Unlike recurrent networks, Tran…