Convergent Evolution: How Different Language Models Learn Similar Number Representations

Deqing Fu, Tianyi Zhou, Mikhail Belkin +2

Language models trained on natural text learn to represent numbers using periodic features with dominant periods at $T=2,5,10$. In this paper, we identify a two-tiered hierarchy…