172 citations · 257 across the 17 of their papers we have counts for
9 papers · 1 filter
Towards Continual Learning: Mitigating Forgetting With Subspace Tuning
Arnav Chavan, Kusha Sareen, Rishabh Tiwari +6
Large language models (LLMs) acquire new skills through fine-tuning, but it can destroy capabilities they already have: catastrophic forgetting remains a central obstacle to contin…
Jacobian-guided Noise Injection for Quantization Robustness in Large Language Models
Deepanshu Pandey, Arnav Chavan, Nahush Lele +2
Quantization of Large Language Models (LLMs) is often hindered by the sensitivity of the self-attention mechanism to discretization errors. We identify the softmax operator as a bo…
DOT-MoE: Differentiable Optimal Transport for MoEfication
Udbhav Bamba, Arnav Chavan, Aryamaan Thakur +2
The scaling of Large Language Models (LLMs) has driven significant performance gains but created substantial challenges in inference efficiency. While Mixture of Experts (MoEs) arc…
S2D: Selective Spectral Decay for Quantization-Friendly Conditioning of Neural Activations
Arnav Chavan, Nahush Lele, Udbhav Bamba +3
Activation outliers in large-scale transformer models pose a fundamental challenge to model quantization, creating excessively large ranges that cause severe accuracy drops during…
Beyond Uniform Scaling: Exploring Depth Heterogeneity in Neural Architectures
Akash Guna R. T, Arnav Chavan, Deepak Gupta
Conventional scaling of neural networks typically involves designing a base network and growing different dimensions like width, depth, etc. of the same by some predefined scaling…
Faster and Lighter LLMs: A Survey on Current Challenges and Way Forward
Arnav Chavan, Raghav Magazine, Shubham Kushwaha +2
Despite the impressive performance of LLMs, their widespread adoption faces challenges due to substantial computational and memory requirements during inference. Recent advancement…
Rethinking Compression: Reduced Order Modelling of Latent Features in Large Language Models
Arnav Chavan, Nahush Lele, Deepak Gupta
Due to the substantial scale of Large Language Models (LLMs), the direct application of conventional compression methodologies proves impractical. The computational demands associa…
One-for-All: Generalized LoRA for Parameter-Efficient Fine-tuning
Arnav Chavan, Zhuang Liu, Deepak Gupta +2
We present Generalized LoRA (GLoRA), an advanced approach for universal parameter-efficient fine-tuning tasks. Enhancing Low-Rank Adaptation (LoRA), GLoRA employs a generalized pro…
Dynamic Kernel Selection for Improved Generalization and Memory Efficiency in Meta-learning
Arnav Chavan, Rishabh Tiwari, Udbhav Bamba +1
Gradient based meta-learning methods are prone to overfit on the meta-training set, and this behaviour is more prominent with large and complex networks. Moreover, large networks r…