Numerical stability analysis of large language models

Stanislav Budzinskiy, Wenyi Fang, Longbin Zeng +1

Transformers are the state-of-the-art architecture for large language models, and a key to their scalability is the strategic usage of low-precision arithmetic. We develop a mixed-…