Language Models Encode Numbers Using Digit Representations in Base 10

Amit Arnold Levy, Mor Geva

Large language models (LLMs) frequently make errors when handling even simple numerical problems, such as comparing two small numbers. A natural hypothesis is that these errors ste…