SAP: Syntactic Attention Pruning for Transformer-based Language Models

Tzu-Yun Lee, Ding-Yong Hong, Jan-Jan Wu

This paper introduces Syntactic Attention Pruning (SAP), a novel method for effectively pruning attention heads in Transformer models. Unlike conventional approaches that rely sole…