Quasi-Recurrent Neural Networks

Abstract

Recurrent neural networks are a powerful tool for modeling sequential data, but the dependence of each timestep's computation on the previous timestep's output limits parallelism and makes RNNs unwieldy for very long sequences. We introduce quasi-recurrent neural networks (QRNNs), an approach to neural sequence modeling that alternates convolutional layers, which apply in parallel across timesteps, and a minimalist recurrent pooling function that applies in parallel across channels. Despite lacking trainable recurrent layers, stacked QRNNs have better predictive accuracy than stacked LSTMs of the same hidden size. Due to their increased parallelism, they are up to 16 times faster at train and test time. Experiments on language modeling, sentiment classification, and character-level neural machine translation demonstrate these advantages and underline the viability of QRNNs as a basic building block for a variety of sequence tasks.

Submitted to conference track at ICLR 2017

References in corpus (7)

• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Recurrent Neural Network Regularization
• Pointer Sentinel Mixture Models
• Neural Machine Translation in Linear Time
• Effective Use of Word Order for Text Categorization with Convolutional Neural Networks
• Ensemble of Generative and Discriminative Techniques for Sentiment Analysis of Movie Reviews
• A Way out of the Odyssey: Analyzing and Combining Recent Insights for LSTMs

Cited by in corpus (18)

• Regularizing and Optimizing LSTM Language Models
• Deep Voice: Real-time Neural Text-to-Speech
• Weighted Transformer Network for Machine Translation
• Revisiting Activation Regularization for Language RNNs
• Learning to Attend via Word-Aspect Associative Fusion for Aspect-based Sentiment Analysis
• Modeling Vocabulary for Big Code Machine Learning
• Adaptively Truncating Backpropagation Through Time to Control Gradient Bias
• Rethinking Full Connectivity in Recurrent Neural Networks
• Temporal Deformable Convolutional Encoder-Decoder Networks for Video Captioning
• Cross Temporal Recurrent Networks for Ranking Question Answer Pairs
• Hardware-Guided Symbiotic Training for Compact, Accurate, yet Execution-Efficient LSTM
• Applying a Pre-trained Language Model to Spanish Twitter Humor Prediction
• Equilibrated Recurrent Neural Network: Neuronal Time-Delayed Self-Feedback Improves Accuracy and Stability
• Efficient Contextual Representation Learning Without Softmax Layer
• Multi-Scale Quasi-RNN for Next Item Recommendation
• MobiVSR: A Visual Speech Recognition Solution for Mobile Devices
• A Lightweight Recurrent Network for Sequence Modeling
• Improving Neural Language Models by Segmenting, Attending, and Predicting the Future