LRCC: Generalizing Low-Rank Compression with Conditional Computation

Thomas Vaitses Fontanari, Maximo Eduardo Rulli, Federico Alvetreti +2

Low-rank compression reduces the cost of pretrained language models by replacing linear transformations with low-rank factorizations. However, conventional methods use a fixed rank…