Hybrid Self-Attention Network for Machine Translation

Abstract

The encoder-decoder is the typical framework for Neural Machine Translation (NMT), and different structures have been developed for improving the translation performance. Transformer is one of the most promising structures, which can leverage the self-attention mechanism to capture the semantic dependency from global view. However, it cannot distinguish the relative position of different tokens very well, such as the tokens located at the left or right of the current token, and cannot focus on the local information around the current token either. To alleviate these problems, we propose a novel attention mechanism named Hybrid Self-Attention Network (HySAN) which accommodates some specific-designed masks for self-attention network to extract various semantic, such as the global/local information, the left/right part context. Finally, a squeeze gate is introduced to combine different kinds of SANs for fusion. Experimental results on three machine translation tasks show that our proposed framework outperforms the Transformer baseline significantly and achieves superior results over state-of-the-art NMT systems.

References in corpus (10)

• Sequence to Sequence Learning with Neural Networks
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Convolutional Sequence to Sequence Learning
• Effective Approaches to Attention-based Neural Machine Translation
• Achieving Human Parity on Automatic Chinese to English News Translation
• Layer Normalization
• Neural Machine Translation in Linear Time
• Weighted Transformer Network for Machine Translation
• DiSAN: Directional Self-Attention Network for RNN/CNN-Free Language Understanding
• The University of Edinburgh's Neural MT Systems for WMT17

Cited by in corpus (4)

• Hard but Robust, Easy but Sensitive: How Encoder and Decoder Perform in Neural Machine Translation
• Neural Machine Translation: A Review and Survey
• Controllable Time-Delay Transformer for Real-Time Punctuation Prediction and Disfluency Detection
• Modeling Concentrated Cross-Attention for Neural Machine Translation with Gaussian Mixture Model