4 papers · 1 filter
Data-Free Pruning of Self-Attention Layers in LLMs
Dhananjay Saikumar, Blesson Varghese
Many self-attention sublayers in large language models (LLMs) can be removed with little to no loss. We attribute this to the Attention Suppression Hypothesis: during pre-training,…
Signal Collapse in One-Shot Pruning: When Sparse Models Fail to Distinguish Neural Representations
Dhananjay Saikumar, Blesson Varghese
Neural network pruning is essential for reducing model complexity to enable deployment on resource constrained hardware. While performance loss of pruned networks is often attribut…
DRIVE: Dual Gradient-Based Rapid Iterative Pruning
Dhananjay Saikumar, Blesson Varghese
Modern deep neural networks (DNNs) consist of millions of parameters, necessitating high-performance computing during training and inference. Pruning is one solution that significa…
NeuroFlux: Memory-Efficient CNN Training Using Adaptive Local Learning
Dhananjay Saikumar, Blesson Varghese
Efficient on-device Convolutional Neural Network (CNN) training in resource-constrained mobile and edge environments is an open challenge. Backpropagation is the standard approach…