Pruning Redundant Mappings in Transformer Models via Spectral-Normalized Identity Prior

Zi Lin, Jeremiah Zhe Liu, Zi Yang +2

Traditional (unstructured) pruning methods for a Transformer model focus on regularizing the individual weights by penalizing them toward zero. In this work, we explore spectral-no…