Attentive Convolution: Equipping CNNs with RNN-style Attention Mechanisms
arXiv:1710.00519
Abstract
In NLP, convolutional neural networks (CNNs) have benefited less than recurrent neural networks (RNNs) from attention mechanisms. We hypothesize that this is because the attention in CNNs has been mainly implemented as attentive pooling (i.e., it is applied to pooling) rather than as attentive convolution (i.e., it is integrated into convolution). Convolution is the differentiator of CNNs in that it can powerfully model the higher-level representation of a word by taking into account its local fixed-size context in the input text t^x. In this work, we propose an attentive convolution network, ATTCONV. It extends the context scope of the convolution operation, deriving higher-level features for a word not only from local context, but also information extracted from nonlocal context by the attention mechanism commonly used in RNNs. This nonlocal context can come (i) from parts of the input text t^x that are distant or (ii) from extra (i.e., external) contexts t^y. Experiments on sentence modeling with zero-context (sentiment analysis), single-context (textual entailment) and multiple-context (claim verification) demonstrate the effectiveness of ATTCONV in sentence representation learning with the incorporation of context. In particular, attentive convolution outperforms attentive pooling and is a strong competitor to popular attentive RNNs.
arXiv commentCamera-ready for TACL. 16 pages
References in corpus (22)
- Natural Language Processing (almost) from Scratch
- Teaching Machines to Read and Comprehend
- Convolutional Sequence to Sequence Learning
- Training Very Deep Networks
- Enhanced LSTM for Natural Language Inference
- SQuAD: 100,000+ Questions for Machine Comprehension of Text
- Effective Approaches to Attention-based Neural Machine Translation
- Dynamic Memory Networks for Visual and Textual Question Answering
- Dynamic Coattention Networks For Question Answering
- A Convolutional Neural Network for Modelling Sentences
- A large annotated corpus for learning natural language inference
- Machine Comprehension Using Match-LSTM and Answer Pointer
- Attentive Pooling Networks
- ReasoNet: Learning to Stop Reading in Machine Comprehension
- Neural Responding Machine for Short-Text Conversation
- A Hierarchical Neural Autoencoder for Paragraphs and Documents
- A simple but tough-to-beat baseline for the Fake News Challenge stance detection task
- Long Short-Term Memory-Networks for Machine Reading
- Question Answering and Question Generation as Dual Tasks
- Learning Recurrent Span Representations for Extractive Question Answering
- Bilateral Multi-Perspective Matching for Natural Language Sentences
- Ruminating Reader: Reasoning with Gated Multi-Hop Attention