Attentive Pooling Networks

Abstract

In this work, we propose Attentive Pooling (AP), a two-way attention mechanism for discriminative model training. In the context of pair-wise ranking or classification with neural networks, AP enables the pooling layer to be aware of the current input pair, in a way that information from the two input items can directly influence the computation of each other's representations. Along with such representations of the paired inputs, AP jointly learns a similarity measure over projected segments (e.g. trigrams) of the pair, and subsequently, derives the corresponding attention vector for each input to guide the pooling. Our two-way attention mechanism is a general framework independent of the underlying representation learning, and it has been applied to both convolutional neural networks (CNNs) and recurrent neural networks (RNNs) in our studies. The empirical results, from three very different benchmark tasks of question answering/answer selection, demonstrate that our proposed models outperform a variety of strong baselines and achieve state-of-the-art performance in all the benchmarks.

10 pages

References in corpus (10)

• Distributed Representations of Words and Phrases and their Compositionality
• Sequence to Sequence Learning with Neural Networks
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Teaching Machines to Read and Comprehend
• Convolutional Neural Network Architectures for Matching Natural Language Sentences
• Reasoning about Entailment with Neural Attention
• LSTM-based Deep Learning Models for Non-factoid Answer Selection
• Deep Learning for Answer Sentence Selection
• Applying Deep Learning to Answer Selection: A Study and An Open Task
• FAQ-based Question Answering via Word Alignment

Cited by in corpus (66)

• Hierarchical Question-Image Co-Attention for Visual Question Answering
• Attention in Natural Language Processing
• Deep Learning Based Text Classification: A Comprehensive Review
• Sentence Similarity Learning by Lexical Decomposition and Composition
• Question Answering and Question Generation as Dual Tasks
• Bilateral Multi-Perspective Matching for Natural Language Sentences
• Emotion Detection on TV Show Transcripts with Sequence-based Convolutional Neural Networks
• A Survey on Text Classification: From Shallow to Deep Learning
• Dataset and Neural Recurrent Sequence Labeling Model for Open-Domain Factoid Question Answering
• Embracing data abundance: BookTest Dataset for Reading Comprehension
• Quantum Language Model with Entanglement Embedding for Question Answering
• End-to-End Answer Chunk Extraction and Ranking for Reading Comprehension
• Multi-View Self-Attention for Interpretable Drug-Target Interaction Prediction
• Towards Scalable and Reliable Capsule Networks for Challenging NLP Applications
• CNM: An Interpretable Complex-valued Network for Matching
• Simple Question Answering by Attentive Convolutional Neural Network
• Sentence Pair Scoring: Towards Unified Framework for Text Comprehension
• Learning Loss Functions for Semi-supervised Learning via Discriminative Adversarial Networks
• Multi-Cast Attention Networks for Retrieval-based Question Answering and Response Prediction
• Learning Visual Knowledge Memory Networks for Visual Question Answering
• Adaptability of Neural Networks on Varying Granularity IR Tasks
• Improving Multilingual Sentence Embedding using Bi-directional Dual Encoder with Additive Margin Softmax
• Multi-Task Learning with Multi-View Attention for Answer Selection and Knowledge Base Question Answering
• Cross Temporal Recurrent Networks for Ranking Question Answer Pairs
• Hierarchical Question Answering for Long Documents
• Video-based Person Re-identification Using Spatial-Temporal Attention Networks
• A Context-Aware User-Item Representation Learning for Item Recommendation
• Sanity Check: A Strong Alignment and Information Retrieval Baseline for Question Answering
• Improving Textual Network Embedding with Global Attention via Optimal Transport
• Graph Neighborhood Attentive Pooling
• M${}^2$S-Net: Multi-Modal Similarity Metric Learning based Deep Convolutional Network for Answer Selection
• Sentiment Analysis with Contextual Embeddings and Self-Attention
• Gated Group Self-Attention for Answer Selection
• Universal Natural Language Processing with Limited Annotations: Try Few-shot Textual Entailment as a Start
• Joint Learning of Answer Selection and Answer Summary Generation in Community Question Answering
• Learning Context-Sensitive Convolutional Filters for Text Processing
• A Zero Attentive Relevance Matching Networkfor Review Modeling in Recommendation System
• BanditRank: Learning to Rank Using Contextual Bandits
• Data-driven Smart Ponzi Scheme Detection
• RAP-Net: Recurrent Attention Pooling Networks for Dialogue Response Selection
• Attentive Convolution: Equipping CNNs with RNN-style Attention Mechanisms
• Contextualized Knowledge-aware Attentive Neural Network: Enhancing Answer Selection with Knowledge
• Why and How to Pay Different Attention to Phrase Alignments of Different Intensities
• Improved Answer Selection with Pre-Trained Word Embeddings
• Zero-shot Text-to-SQL Learning with Auxiliary Task
• A Multi-Size Neural Network with Attention Mechanism for Answer Selection
• MRNN: A Multi-Resolution Neural Network with Duplex Attention for Document Retrieval in the Context of Question Answering
• An Explainable CNN Approach for Medical Codes Prediction from Clinical Text
• Evaluating a Generative Adversarial Framework for Information Retrieval
• CogAlign: Learning to Align Textual Neural Representations to Cognitive Language Processing Signals
• Review-guided Helpful Answer Identification in E-commerce
• Neural Relation Prediction for Simple Question Answering over Knowledge Graph
• Learning to Embed Sentences Using Attentive Recursive Trees
• Cut-Based Graph Learning Networks to Discover Compositional Structure of Sequential Video Data
• On Tree-Based Neural Sentence Modeling
• AAANE: Attention-based Adversarial Autoencoder for Multi-scale Network Embedding
• Hierarchical Gated Recurrent Neural Tensor Network for Answer Triggering
• Asymmetrical Hierarchical Networks with Attentive Interactions for Interpretable Review-Based Recommendation
• Sequential Learning of Convolutional Features for Effective Text Classification
• A Gated Self-attention Memory Network for Answer Selection
• Hashing based Answer Selection
• Two-Headed Monster And Crossed Co-Attention Networks
• Beyond [CLS] through Ranking by Generation
• Hierarchical Ranking for Answer Selection
• Indirect Supervision for Relation Extraction using Question-Answer Pairs
• CIKM AnalytiCup 2017 Lazada Product Title Quality Challenge An Ensemble of Deep and Shallow Learning to predict the Quality of Product Titles