Mask Attention Networks: Rethinking and Strengthen Transformer

Abstract

Transformer is an attention-based neural network, which consists of two sublayers, namely, Self-Attention Network (SAN) and Feed-Forward Network (FFN). Existing research explores to enhance the two sublayers separately to improve the capability of Transformer for text representation. In this paper, we present a novel understanding of SAN and FFN as Mask Attention Networks (MANs) and show that they are two special cases of MANs with static mask matrices. However, their static mask matrices limit the capability for localness modeling in text representation learning. We therefore introduce a new layer named dynamic mask attention network (DMAN) with a learnable mask matrix which is able to model localness adaptively. To incorporate advantages of DMAN, SAN, and FFN, we propose a sequential layered structure to combine the three types of layers. Extensive experiments on various tasks, including neural machine translation and text summarization demonstrate that our model outperforms the original Transformer.

Accepted as a long paper to NAACL 2021

References in corpus (6)

• MASS: Masked Sequence to Sequence Pre-training for Language Generation
• Pay Less Attention with Lightweight and Dynamic Convolutions
• Weighted Transformer Network for Machine Translation
• Understanding and Improving Transformer From a Multi-Particle Dynamic System Point of View
• Augmenting Self-attention with Persistent Memory
• MUSE: Parallel Multi-Scale Attention for Sequence to Sequence Learning