Focused Hierarchical RNNs for Conditional Sequence Processing

Abstract

Recurrent Neural Networks (RNNs) with attention mechanisms have obtained state-of-the-art results for many sequence processing tasks. Most of these models use a simple form of encoder with attention that looks over the entire sequence and assigns a weight to each token independently. We present a mechanism for focusing RNN encoders for sequence modelling tasks which allows them to attend to key parts of the input as needed. We formulate this using a multi-layer conditional sequence encoder that reads in one token at a time and makes a discrete decision on whether the token is relevant to the context or question being asked. The discrete gating mechanism takes in the context embedding and the current hidden state as inputs and controls information flow into the layer above. We train it using policy gradient methods. We evaluate this method on several types of tasks with different attributes. First, we evaluate the method on synthetic tasks which allow us to evaluate the model for its generalization ability and probe the behavior of the gates in more controlled settings. We then evaluate this approach on large scale Question Answering tasks including the challenging MS MARCO and SearchQA tasks. Our models shows consistent improvements for both tasks over prior work and our baselines. It has also shown to generalize significantly better on synthetic tasks as compared to the baselines.

arXiv commentTo appear at ICML 2018

References in corpus (15)

• Auto-Encoding Variational Bayes
• Machine Comprehension Using Match-LSTM and Answer Pointer
• SearchQA: A New Q&A Dataset Augmented with Context from a Search Engine
• Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyond
• Adaptive Computation Time for Recurrent Neural Networks
• ReasoNet: Learning to Stop Reading in Machine Comprehension
• A Clockwork RNN
• Hierarchical Multiscale Recurrent Neural Networks
• Ask the Right Questions: Active Question Reformulation with Reinforcement Learning
• Reading Wikipedia to Answer Open-Domain Questions
• S-Net: From Answer Extraction to Answer Generation for Machine Reading Comprehension
• Making Neural QA as Simple as Possible but not Simpler
• Skip RNN: Learning to Skip State Updates in Recurrent Neural Networks
• Learning Efficient Algorithms with Hierarchical Attentive Memory
• Focused Hierarchical RNNs for Conditional Sequence Processing

Cited by in corpus (6)

• Exploring Interpretable LSTM Neural Networks over Multi-Variable Data
• Dynamic Neural Networks: A Survey
• Focused Hierarchical RNNs for Conditional Sequence Processing
• Mixture Content Selection for Diverse Sequence Generation
• Select and Attend: Towards Controllable Content Selection in Text Generation
• HorNet: A Hierarchical Offshoot Recurrent Network for Improving Person Re-ID via Image Captioning