Paying More Attention to Self-attention: Improving Pre-trained Language Models via Attention Guiding

Shanshan Wang, Zhumin Chen, Zhaochun Ren +3

Pre-trained language models (PLM) have demonstrated their effectiveness for a broad range of information retrieval and natural language processing tasks. As the core part of PLM, m…