172 citations · 257 across the 17 of their papers we have counts for
4 papers · 1 filter
Towards Continual Learning: Mitigating Forgetting With Subspace Tuning
Arnav Chavan, Kusha Sareen, Rishabh Tiwari +6
Large language models (LLMs) acquire new skills through fine-tuning, but it can destroy capabilities they already have: catastrophic forgetting remains a central obstacle to contin…
Jacobian-guided Noise Injection for Quantization Robustness in Large Language Models
Deepanshu Pandey, Arnav Chavan, Nahush Lele +2
Quantization of Large Language Models (LLMs) is often hindered by the sensitivity of the self-attention mechanism to discretization errors. We identify the softmax operator as a bo…
DOT-MoE: Differentiable Optimal Transport for MoEfication
Udbhav Bamba, Arnav Chavan, Aryamaan Thakur +2
The scaling of Large Language Models (LLMs) has driven significant performance gains but created substantial challenges in inference efficiency. While Mixture of Experts (MoEs) arc…
S2D: Selective Spectral Decay for Quantization-Friendly Conditioning of Neural Activations
Arnav Chavan, Nahush Lele, Udbhav Bamba +3
Activation outliers in large-scale transformer models pose a fundamental challenge to model quantization, creating excessively large ranges that cause severe accuracy drops during…