As Language Models Scale, Low-order Linear Depth Dynamics Emerge

Buddhika Nettasinghe, Geethu Joseph

Large language models are often viewed as high-dimensional nonlinear systems and treated as black boxes. Here, we show that transformer depth dynamics admit accurate low-order line…