Magnitude Profile Pruning: Calibration-Free Structured Attention Head Removal for Transformer Compression

Kasun Dewage, Marianna Pensky, Heranga K. Rathnasekara +1

Structured pruning of attention heads provides a hardware-friendly way to compress Transformer language models. However, existing methods for measuring head-level importance requir…