LazyFormer: Self Attention with Lazy Update

Chengxuan Ying, Guolin Ke, Di He +1

Improving the efficiency of Transformer-based language pre-training is an important task in NLP, especially for the self-attention module, which is computationally expensive. In th…