Arithmetic with Language Models: from Memorization to Computation
arXiv:2308.01154 · doi:10.1016/j.neunet.2024.106550
Abstract
A better understanding of the emergent computation and problem-solving capabilities of recent large language models is of paramount importance to further improve them and broaden their applicability. This work investigates how a language model, trained to predict the next token, can perform arithmetic computations generalizing beyond training data. Binary addition and multiplication constitute a good testbed for this purpose, since they require a very small vocabulary and exhibit relevant input/output discontinuities making smooth input interpolation ineffective for novel data. We successfully trained a light language model to learn these tasks and ran a number of experiments to investigate the extrapolation capabilities and internal information processing. Our findings support the hypothesis that the language model works as an Encoding-Regression-Decoding machine where the computation takes place in the value space once the input token representation is mapped to an appropriate internal representation.
The article has been accepted for publication in Elsevier Neural Networks journal. The final version is available on the Elsevier ScienceDirect platform
References in corpus (11)
- Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- Sparks of Artificial General Intelligence: Early experiments with GPT-4
- Scaling Laws for Neural Language Models
- Emergent Abilities of Large Language Models
- On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
- Progress measures for grokking via mechanistic interpretability
- Investigating the Limitations of Transformers with Simple Arithmetic Tasks
- Probing for the Usage of Grammatical Number
- How well do Large Language Models perform in Arithmetic tasks?
- Goat: Fine-tuned LLaMA Outperforms GPT-4 on Arithmetic Tasks
- Teaching Arithmetic to Small Transformers