An Empirical Investigation of Matrix Factorization Methods for Pre-trained Transformers

Ashim Gupta, Sina Mahdipour Saravani, P. Sadayappan +1

The increasing size of transformer-based models in NLP makes the question of compressing them important. In this work, we present a comprehensive analysis of factorization based mo…