Pervasive Attention: 2D Convolutional Neural Networks for Sequence-to-Sequence Prediction
arXiv:1808.03867
Abstract
Current state-of-the-art machine translation systems are based on encoder-decoder architectures, that first encode the input sequence, and then generate an output sequence based on the input encoding. Both are interfaced with an attention mechanism that recombines a fixed encoding of the source tokens based on the decoder state. We propose an alternative approach which instead relies on a single 2D convolutional neural network across both sequences. Each layer of our network re-codes source tokens on the basis of the output sequence produced so far. Attention-like properties are therefore pervasive throughout the network. Our model yields excellent results, outperforming state-of-the-art encoder-decoder systems, while being conceptually simpler and having fewer parameters.
arXiv commentAccepted at CoNLL 2018
References in corpus (13)
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- Sequence to Sequence Learning with Neural Networks
- Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
- WaveNet: A Generative Model for Raw Audio
- Convolutional Sequence to Sequence Learning
- Conditional Image Generation with PixelCNN Decoders
- Pixel Recurrent Neural Networks
- Sequence Transduction with Recurrent Neural Networks
- Neural Machine Translation in Linear Time
- Grid Long Short-Term Memory
- Latent Alignment and Variational Attention
- Sequence Modeling via Segmentations
- Towards Neural Phrase-based Machine Translation
Cited by in corpus (9)
- Deep Learning as a Parton Shower
- Revisiting Low-Resource Neural Machine Translation: A Case Study
- You May Not Need Attention
- Transformer to CNN: Label-scarce distillation for efficient text classification
- Convolution, attention and structure embedding
- Joint Source-Target Self Attention with Locality Constraints
- An Attempt towards Interpretable Audio-Visual Video Captioning
- Neural Machine Translation with Joint Representation
- Regularized Context Gates on Transformer for Machine Translation