Complementary Attention Head Pruning for Efficient Transformers

Yaniv Livertovsky, Shahar Somin, Gonen Singer

The remarkable success of Transformer-based models in natural language processing stems from architectural scaling, which leads to a large number of parameters and hinders deployme…