5 papers
LampAttention: Look-Ahead Mixed-Precision FlashAttention for Dedicated Accelerators
Stanislav Budzinskiy, Marian Gloser, Tolunay Yilmaz +5
While most attention logits can be computed in low precision without degrading numerical stability, current attention kernels fail to exploit this phenomenon. We introduce a novel…
Dispelling the Curse of Singularities in Neural Network Optimizations
Hengjie Cao, Mengyi Chen, Yifeng Yang +11
This work investigates the optimization instability of deep neural networks from a less-explored yet insightful perspective: the emergence and amplification of singularities in the…
LAMP: Look-Ahead Mixed-Precision Inference of Large Language Models
Stanislav Budzinskiy, Marian Gloser, Tolunay Yilmaz +5
Mixed-precision computations are a hallmark of the current stage of AI, driving the progress in large language models towards efficient, locally deployable solutions. This article…
Transferable text data distillation by trajectory matching
Rong Yao, Hailin Hu, Yifei Fu +5
In the realm of large language model (LLM), as the size of large models increases, it also brings higher training costs. There is a urgent need to minimize the data size in LLM tra…
Numerical stability analysis of large language models
Stanislav Budzinskiy, Wenyi Fang, Longbin Zeng +1
Transformers are the state-of-the-art architecture for large language models, and a key to their scalability is the strategic usage of low-precision arithmetic. We develop a mixed-…