167 citations · 181 across the 12 of their papers we have counts for
13 papers
OptRot: Mitigating Weight Outliers via Data-Free Rotations for Post-Training Quantization
Advait Gadhikar, Riccardo Grazzi, James Hensman
The presence of outliers in Large Language Models (LLMs) weights and activations makes them difficult to quantize. Recent work has leveraged rotations to mitigate these outliers. I…
Pay Attention to Small Weights
Chao Zhou, Tom Jacobs, Advait Gadhikar +1
Finetuning large pretrained neural networks is known to be resource-intensive, both in terms of memory and computational cost. To mitigate this, a common approach is to restrict tr…
Hyperbolic Aware Minimization: Implicit Bias for Sparsity
Tom Jacobs, Advait Gadhikar, Celia Rubio-Madrigal +1
Understanding the implicit bias of optimization algorithms is key to explaining and improving the generalization of deep models. The hyperbolic implicit bias induced by pointwise o…
Sign-In to the Lottery: Reparameterizing Sparse Training From Scratch
Advait Gadhikar, Tom Jacobs, Chao Zhou +1
The performance gap between training sparse neural networks from scratch (PaI) and dense-to-sparse training presents a major roadblock for efficient deep learning. According to the…
Attention Is All You Need For Mixture-of-Depths Routing
Advait Gadhikar, Souptik Kumar Majumdar, Niclas Popp +3
Advancements in deep learning are driven by training models with increasingly larger numbers of parameters, which in turn heightens the computational demands. To address this issue…
Cyclic Sparse Training: Is it Enough?
Advait Gadhikar, Sree Harsha Nelaturu, Rebekka Burkholz
The success of iterative pruning methods in achieving state-of-the-art sparse networks has largely been attributed to improved mask identification and an implicit regularization in…
Masks, Signs, And Learning Rate Rewinding
Advait Gadhikar, Rebekka Burkholz
Learning Rate Rewinding (LRR) has been established as a strong variant of Iterative Magnitude Pruning (IMP) to find lottery tickets in deep overparameterized neural networks. While…
Dynamical Isometry for Residual Networks
Advait Gadhikar, Rebekka Burkholz
The training success, training speed and generalization ability of neural networks rely crucially on the choice of random parameter initialization. It has been shown for multiple a…
Why Random Pruning Is All We Need to Start Sparse
Advait Gadhikar, Sohom Mukherjee, Rebekka Burkholz
Random masks define surprisingly effective sparse neural network models, as has been shown empirically. The resulting sparse networks can often compete with dense architectures and…
Leveraging Spatial and Temporal Correlations in Sparsified Mean Estimation
Divyansh Jhunjhunwala, Ankur Mallick, Advait Gadhikar +2
We study the problem of estimating at a central server the mean of a set of vectors distributed across several nodes (one vector per node). When the vectors are high-dimensional, t…
Lottery Tickets with Nonzero Biases
Jonas Fischer, Advait Gadhikar, Rebekka Burkholz
The strong lottery ticket hypothesis holds the promise that pruning randomly initialized deep neural networks could offer a computationally efficient alternative to deep learning w…
A Field Guide to Federated Optimization
Jianyu Wang, Zachary Charles, Zheng Xu +50
Federated learning and analytics are a distributed approach for collaboratively learning models (or statistics) from decentralized data, motivated by and designed for privacy prote…
Adaptive Quantization of Model Updates for Communication-Efficient Federated Learning
Divyansh Jhunjhunwala, Advait Gadhikar, Gauri Joshi +1
Communication of model updates between client nodes and the central aggregating server is a major bottleneck in federated learning, especially in bandwidth-limited settings and hig…