Quasi-Recurrent Neural Networks

Abstract

Recurrent neural networks are a powerful tool for modeling sequential data, but the dependence of each timestep's computation on the previous timestep's output limits parallelism and makes RNNs unwieldy for very long sequences. We introduce quasi-recurrent neural networks (QRNNs), an approach to neural sequence modeling that alternates convolutional layers, which apply in parallel across timesteps, and a minimalist recurrent pooling function that applies in parallel across channels. Despite lacking trainable recurrent layers, stacked QRNNs have better predictive accuracy than stacked LSTMs of the same hidden size. Due to their increased parallelism, they are up to 16 times faster at train and test time. Experiments on language modeling, sentiment classification, and character-level neural machine translation demonstrate these advantages and underline the viability of QRNNs as a basic building block for a variety of sequence tasks.

Submitted to conference track at ICLR 2017

References in corpus (16)

• Adam: A Method for Stochastic Optimization
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Recurrent Neural Network Regularization
• Densely Connected Convolutional Networks
• Effective Approaches to Attention-based Neural Machine Translation
• Dynamic Memory Networks for Visual and Textual Question Answering
• A C-LSTM Neural Network for Text Classification
• Pointer Sentinel Mixture Models
• Neural Machine Translation in Linear Time
• Adversarial Training Methods for Semi-Supervised Text Classification
• Effective Use of Word Order for Text Categorization with Convolutional Neural Networks
• Efficient Character-level Document Classification by Combining Convolution and Recurrent Layers
• Zoneout: Regularizing RNNs by Randomly Preserving Hidden Activations
• Sequence-to-Sequence Learning as Beam-Search Optimization
• Ensemble of Generative and Discriminative Techniques for Sentiment Analysis of Movie Reviews
• A Way out of the Odyssey: Analyzing and Combining Recent Insights for LSTMs

Cited by in corpus (82)

• An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling
• Convolutional Sequence to Sequence Learning
• Efficient Deep Learning: A Survey on Making Deep Learning Models Smaller, Faster, and Better
• Multimodal Intelligence: Representation Learning, Information Fusion, and Applications
• Regularizing and Optimizing LSTM Language Models
• Deep Voice: Real-time Neural Text-to-Speech
• Efficiently Trainable Text-to-Speech System Based on Deep Convolutional Networks with Guided Attention
• Deep Voice 2: Multi-Speaker Neural Text-to-Speech
• Weighted Transformer Network for Machine Translation
• Recurrent Neural Networks: An Embedded Computing Perspective
• Independently Recurrent Neural Network (IndRNN): Building A Longer and Deeper RNN
• Learning Intrinsic Sparse Structures within Long Short-Term Memory
• Training with Quantization Noise for Extreme Model Compression
• FastGRNN: A Fast, Accurate, Stable and Tiny Kilobyte Sized Gated Recurrent Neural Network
• A Survey on 3D Skeleton-Based Action Recognition Using Learning Method
• Neural Abstractive Text Summarization with Sequence-to-Sequence Models
• Unsupervised Deep Learning for IoT Time Series
• Compressive Transformers for Long-Range Sequence Modelling
• Dual Rectified Linear Units (DReLUs): A Replacement for Tanh Activation Functions in Quasi-Recurrent Neural Networks
• Revisiting Activation Regularization for Language RNNs
• Skip RNN: Learning to Skip State Updates in Recurrent Neural Networks
• Addressing Some Limitations of Transformers with Feedback Memory
• Capitalization and Punctuation Restoration: a Survey
• Learning to Attend via Word-Aspect Associative Fusion for Aspect-based Sentiment Analysis
• The Implicit and Explicit Regularization Effects of Dropout
• Modeling Vocabulary for Big Code Machine Learning
• Parallelizing Linear Recurrent Neural Nets Over Sequence Length
• Adaptively Truncating Backpropagation Through Time to Control Gradient Bias
• Machine Translation of Low-Resource Spoken Dialects: Strategies for Normalizing Swiss German
• Efficient Visual Recognition with Deep Neural Networks: A Survey on Recent Advances and New Directions
• Recurrent Attention Unit
• Distilling Large Language Models into Tiny and Effective Students using pQRNN
• Have convolutions already made recurrence obsolete for unconstrained handwritten text recognition ?
• NAUTILUS: a Versatile Voice Cloning System
• Rethinking Full Connectivity in Recurrent Neural Networks
• Temporal Deformable Convolutional Encoder-Decoder Networks for Video Captioning
• 3D Convolutional Neural Networks for Ultrasound-Based Silent Speech Interfaces
• Simple Recurrent Neural Networks is all we need for clinical events predictions using EHR data
• Representation based meta-learning for few-shot spoken intent recognition
• A Unified Speaker Adaptation Method for Speech Synthesis using Transcribed and Untranscribed Speech with Backpropagation
• Cross Temporal Recurrent Networks for Ranking Question Answer Pairs
• Hardware-Guided Symbiotic Training for Compact, Accurate, yet Execution-Efficient LSTM
• Evaluating neural network explanation methods using hybrid documents and morphological agreement
• Echo: Compiler-based GPU Memory Footprint Reduction for LSTM RNN Training
• Nondeterminism and Instability in Neural Network Optimization
• Voice Activity Detection for Ultrasound-based Silent Speech Interfaces using Convolutional Neural Networks
• Progress and Tradeoffs in Neural Language Models
• Single Stream Parallelization of Recurrent Neural Networks for Low Power and Fast Inference
• An Effective, Robust and Fairness-aware Hate Speech Detection Framework
• Applying a Pre-trained Language Model to Spanish Twitter Humor Prediction
• Multi-task WaveNet: A Multi-task Generative Model for Statistical Parametric Speech Synthesis without Fundamental Frequency Conditions
• End-to-End Text Classification via Image-based Embedding using Character-level Networks
• Learning to Search in Long Documents Using Document Structure
• Anticipating Traffic Accidents with Adaptive Loss and Large-scale Incident DB
• Equilibrated Recurrent Neural Network: Neuronal Time-Delayed Self-Feedback Improves Accuracy and Stability
• NeuronBlocks: Building Your NLP DNN Models Like Playing Lego
• Recognizing Video Events with Varying Rhythms
• Refined Gate: A Simple and Effective Gating Mechanism for Recurrent Units
• Detecting Hostile Posts using Relational Graph Convolutional Network
• Multi-Scale Quasi-RNN for Next Item Recommendation
• Efficient Contextual Representation Learning Without Softmax Layer
• Block-wise Dynamic Sparseness
• 2kenize: Tying Subword Sequences for Chinese Script Conversion
• Deep Learning Brasil -- NLP at SemEval-2020 Task 9: Overview of Sentiment Analysis of Code-Mixed Tweets
• Simplifying Neural Machine Translation with Addition-Subtraction Twin-Gated Recurrent Networks
• MobiVSR: A Visual Speech Recognition Solution for Mobile Devices
• SoPa: Bridging CNNs, RNNs, and Weighted Finite-State Machines
• RIANN -- A Robust Neural Network Outperforms Attitude Estimation Filters
• Persistence pays off: Paying Attention to What the LSTM Gating Mechanism Persists
• $k$-Neighbor Based Curriculum Sampling for Sequence Prediction
• Recurrent Neural Network from Adder's Perspective: Carry-lookahead RNN
• Mixture of Expert/Imitator Networks: Scalable Semi-supervised Learning Framework
• A Lightweight Recurrent Network for Sequence Modeling
• Analysing Dropout and Compounding Errors in Neural Language Models
• Alternating Synthetic and Real Gradients for Neural Language Modeling
• Comparing CNN and LSTM character-level embeddings in BiLSTM-CRF models for chemical and disease named entity recognition
• Semi-tied Units for Efficient Gating in LSTM and Highway Networks
• Multi-axis Attentive Prediction for Sparse EventData: An Application to Crime Prediction
• FastTrees: Parallel Latent Tree-Induction for Faster Sequence Encoding
• Improving Neural Language Models by Segmenting, Attending, and Predicting the Future
• Rumour Detection via Zero-shot Cross-lingual Transfer Learning
• Improving Speech Recognition Accuracy of Local POI Using Geographical Models