374 citations · 602 across the 19 of their papers we have counts for
15 papers · 1 filter
Gradient Smoothing: Coupling Layer-wise Updates for Improved Optimization
Haoming Meng, Anton Sugolov, Vardan Papyan
Deep neural networks with repeated architectural blocks, such as transformers, often exhibit structured relationships across layers that emerge during training. Motivated by this o…
Laplacian Heads Improve Transformers by Smoothing Token Representations
Yuchong Zhang, Vardan Papyan
Transformers update token representations through multi-head attention and residual connections as , where is the softma…
On the Importance of Gaussianizing Representations
Daniel Eftekhari, Vardan Papyan
The normal distribution plays a central role in information theory - it is at the same time the best-case signal and worst-case noise distribution, has the greatest representationa…
OATS: Outlier-Aware Pruning Through Sparse and Low Rank Decomposition
Stephen Zhang, Vardan Papyan
The recent paradigm shift to large-scale foundation models has brought about a new era for deep learning that, while has found great success in practice, has also been plagued by p…
Sparsest Models Elude Pruning: An Exposé of Pruning's Current Capabilities
Stephen Zhang, Vardan Papyan
Pruning has emerged as a promising approach for compressing large-scale models, yet its effectiveness in recovering the sparsest of models has not yet been explored. We conducted a…
Transformer Block Coupling and its Correlation with Generalization in LLMs
Murdock Aubry, Haoming Meng, Anton Sugolov +1
Large Language Models (LLMs) have made significant strides in natural language processing, and a precise understanding of the internal mechanisms driving their success is essential…