374 citations · 941 across the 27 of their papers we have counts for
28 papers
Gradient Smoothing: Coupling Layer-wise Updates for Improved Optimization
Haoming Meng, Anton Sugolov, Vardan Papyan
Deep neural networks with repeated architectural blocks, such as transformers, often exhibit structured relationships across layers that emerge during training. Motivated by this o…
Laplacian Heads Improve Transformers by Smoothing Token Representations
Yuchong Zhang, Vardan Papyan
Transformers update token representations through multi-head attention and residual connections as , where is the softma…
On the Importance of Gaussianizing Representations
Daniel Eftekhari, Vardan Papyan
The normal distribution plays a central role in information theory - it is at the same time the best-case signal and worst-case noise distribution, has the greatest representationa…
Attention Sinks: A 'Catch, Tag, Release' Mechanism for Embeddings
Stephen Zhang, Mustafa Khan, Vardan Papyan
Large language models (LLMs) often concentrate their attention on a few specific tokens referred to as attention sinks. Common examples include the first token, a prompt-independen…
OATS: Outlier-Aware Pruning Through Sparse and Low Rank Decomposition
Stephen Zhang, Vardan Papyan
The recent paradigm shift to large-scale foundation models has brought about a new era for deep learning that, while has found great success in practice, has also been plagued by p…
Sparsest Models Elude Pruning: An Exposé of Pruning's Current Capabilities
Stephen Zhang, Vardan Papyan
Pruning has emerged as a promising approach for compressing large-scale models, yet its effectiveness in recovering the sparsest of models has not yet been explored. We conducted a…