Low-Rank Compression of Language Models via Differentiable Rank Selection

Sidhant Sundrani, Francesco Tudisco, Pasquale Minervini

Approaches for compressing large-language models using low-rank decomposition have made strides, particularly with the introduction of activation and loss-aware SVD, which improves…