Multi-Cast Attention Networks for Retrieval-based Question Answering and Response Prediction

Abstract

Attention is typically used to select informative sub-phrases that are used for prediction. This paper investigates the novel use of attention as a form of feature augmentation, i.e, casted attention. We propose Multi-Cast Attention Networks (MCAN), a new attention mechanism and general model architecture for a potpourri of ranking tasks in the conversational modeling and question answering domains. Our approach performs a series of soft attention operations, each time casting a scalar feature upon the inner word embeddings. The key idea is to provide a real-valued hint (feature) to a subsequent encoder layer and is targeted at improving the representation learning process. There are several advantages to this design, e.g., it allows an arbitrary number of attention mechanisms to be casted, allowing for multiple attention types (e.g., co-attention, intra-attention) and attention variants (e.g., alignment-pooling, max-pooling, mean-pooling) to be executed simultaneously. This not only eliminates the costly need to tune the nature of the co-attention layer, but also provides greater extents of explainability to practitioners. Via extensive experiments on four well-known benchmark datasets, we show that MCAN achieves state-of-the-art performance. On the Ubuntu Dialogue Corpus, MCAN outperforms existing state-of-the-art models by $9\%$. MCAN also achieves the best performing score to date on the well-studied TrecQA dataset.

arXiv commentAccepted to KDD 2018 (Paper titled only "Multi-Cast Attention Networks" in KDD version)

References in corpus (19)

• Adam: A Method for Stochastic Optimization
• Bidirectional Attention Flow for Machine Comprehension
• Enhanced LSTM for Natural Language Inference
• Convolutional Neural Network Architectures for Matching Natural Language Sentences
• Reasoning about Entailment with Neural Attention
• End-to-End Neural Ad-hoc Ranking with Kernel Pooling
• Highway Networks
• Dynamic Coattention Networks For Question Answering
• Machine Comprehension Using Match-LSTM and Answer Pointer
• Attentive Pooling Networks
• Sentence Similarity Learning by Lexical Decomposition and Composition
• A Deep Architecture for Semantic Matching with Multiple Positional Sentence Representations
• Text Matching as Image Recognition
• Match-SRNN: Modeling the Recursive Matching Structure with Spatial RNN
• Bilateral Multi-Perspective Matching for Natural Language Sentences
• Word-Entity Duet Representations for Document Ranking
• Knowledge Enhanced Hybrid Neural Network for Text Matching
• Cross Temporal Recurrent Networks for Ranking Question Answer Pairs
• Co-PACRR: A Context-Aware Neural IR Model for Ad-hoc Retrieval

Cited by in corpus (4)

• Ensemble Transformer for Efficient and Accurate Ranking Tasks: an Application to Question Answering Systems
• RAP-Net: Recurrent Attention Pooling Networks for Dialogue Response Selection
• BanditRank: Learning to Rank Using Contextual Bandits
• Self-Attentive Neural Collaborative Filtering