Convolutional Self-Attention Networks
arXiv:1904.03107
Abstract
Self-attention networks (SANs) have drawn increasing interest due to their high parallelization in computation and flexibility in modeling dependencies. SANs can be further enhanced with multi-head attention by allowing the model to attend to information from different representation subspaces. In this work, we propose novel convolutional self-attention networks, which offer SANs the abilities to 1) strengthen dependencies among neighboring elements, and 2) model the interaction between features extracted by multiple attention heads. Experimental results of machine translation on different language pairs and model settings show that our approach outperforms both the strong Transformer baseline and other existing models on enhancing the locality of SANs. Comparing with prior studies, the proposed model is parameter free in terms of introducing no more parameters.
NAACL 2019
References in corpus (10)
- Convolutional Sequence to Sequence Learning
- A Structured Self-attentive Sentence Embedding
- Layer Normalization
- QANet: Combining Local Convolution with Global Self-Attention for Reading Comprehension
- Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
- THUMT: An Open Source Toolkit for Neural Machine Translation
- Exploiting Deep Representations for Neural Machine Translation
- Multi-Head Attention with Disagreement Regularization
- Information Aggregation for Multi-Head Attention with Routing-by-Agreement
- Modeling Recurrence for Transformer
Cited by in corpus (9)
- Conformer: Convolution-augmented Transformer for Speech Recognition
- Context-Aware Self-Attention Networks
- Multi-head or Single-head? An Empirical Comparison for Transformer Training
- Porous Lattice-based Transformer Encoder for Chinese NER
- Information Aggregation for Multi-Head Attention with Routing-by-Agreement
- Language Modeling using LMUs: 10x Better Data Efficiency or Improved Scaling Compared to Transformers
- BUT Opensat 2019 Speech Recognition System
- Generating Diverse Translation by Manipulating Multi-Head Attention
- Attention-based Neural Beamforming Layers for Multi-channel Speech Recognition