Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling

Abstract

In this paper we compare different types of recurrent units in recurrent neural networks (RNNs). Especially, we focus on more sophisticated units that implement a gating mechanism, such as a long short-term memory (LSTM) unit and a recently proposed gated recurrent unit (GRU). We evaluate these recurrent units on the tasks of polyphonic music modeling and speech signal modeling. Our experiments revealed that these advanced recurrent units are indeed better than more traditional recurrent units such as tanh units. Also, we found GRU to be comparable to LSTM.

Presented in NIPS 2014 Deep Learning and Representation Learning Workshop

References in corpus (2)

• Theano: new features and speed improvements
• Modeling Temporal Dependencies in High-Dimensional Sequences: Application to Polyphonic Music Generation and Transcription

Cited by in corpus (194)

• Comparative Study of CNN and RNN for Natural Language Processing
• Skip-Thought Vectors
• Understanding Neural Networks through Representation Erasure
• Deep Voice: Real-time Neural Text-to-Speech
• DyNet: The Dynamic Neural Network Toolkit
• Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books
• Ask the GRU: Multi-Task Learning for Deep Text Recommendations
• LipNet: End-to-End Sentence-level Lipreading
• Deep Learning and Its Applications to Machine Health Monitoring: A Survey
• Tweet2Vec: Learning Tweet Embeddings Using Character-level CNN-LSTM Encoder-Decoder
• Neural Machine Translation and Sequence-to-sequence Models: A Tutorial
• DiSAN: Directional Self-Attention Network for RNN/CNN-Free Language Understanding
• Generating and designing DNA with deep generative models
• Compressing Recurrent Neural Network with Tensor Train
• ATRank: An Attention-Based User Behavior Modeling Framework for Recommendation
• Learning Deep Structured Multi-Scale Features using Attention-Gated CRFs for Contour Prediction
• Neural Map: Structured Memory for Deep Reinforcement Learning
• A Hybrid RNN-HMM Approach for Weakly Supervised Temporal Action Segmentation
• Flexible statistical inference for mechanistic models of neural dynamics
• R-Transformer: Recurrent Neural Network Enhanced Transformer
• THUMT: An Open Source Toolkit for Neural Machine Translation
• Deeply AggreVaTeD: Differentiable Imitation Learning for Sequential Prediction
• Application of generative autoencoder in de novo molecular design
• LFADS - Latent Factor Analysis via Dynamical Systems
• Query Focused Abstractive Summarization: Incorporating Query Relevance, Multi-Document Coverage, and Summary Length Constraints into seq2seq Models
• Effective Quantization Methods for Recurrent Neural Networks
• Semantic Tagging with Deep Residual Networks
• Automatic Detection of Malware-Generated Domains with Recurrent Neural Models
• Benchmark of Deep Learning Models on Large Healthcare MIMIC Datasets
• Neural Networks for Text Correction and Completion in Keyboard Decoding
• EKT: Exercise-aware Knowledge Tracing for Student Performance Prediction
• Deep Learning for Sentiment Analysis : A Survey
• Learning text representation using recurrent convolutional neural network with highway layers
• Graph-based Neural Multi-Document Summarization
• Aspect-Based Sentiment Analysis Using a Two-Step Neural Network Architecture
• Molecular Generation with Recurrent Neural Networks (RNNs)
• Neural Information Retrieval: A Literature Review
• Transferable Multi-Domain State Generator for Task-Oriented Dialogue Systems
• LightRNN: Memory and Computation-Efficient Recurrent Neural Networks
• Delta Networks for Optimized Recurrent Network Computation
• Speaker Diarization using Deep Recurrent Convolutional Neural Networks for Speaker Embeddings
• A Survey of Natural Language Generation Techniques with a Focus on Dialogue Systems - Past, Present and Future Directions
• Latent Attention For If-Then Program Synthesis
• Neural Graph Evolution: Towards Efficient Automatic Robot Design
• Deep Conversational Recommender in Travel
• An End-to-End Architecture for Keyword Spotting and Voice Activity Detection
• Cortical microcircuits as gated-recurrent neural networks
• Gated Recurrent Neural Tensor Network
• Discrete Event, Continuous Time RNNs
• R2N2: Residual Recurrent Neural Networks for Multivariate Time Series Forecasting
• Knowledge as a Teacher: Knowledge-Guided Structural Attention Networks
• A multi-task learning model for malware classification with useful file access pattern from API call sequence
• DivGraphPointer: A Graph Pointer Network for Extracting Diverse Keyphrases
• AMPNet: Asynchronous Model-Parallel Training for Dynamic Neural Networks
• Joint Modeling of Event Sequence and Time Series with Attentional Twin Recurrent Neural Networks
• Neural Machine Translation with Supervised Attention
• Improved Lossy Image Compression with Priming and Spatially Adaptive Bit Rates for Recurrent Networks
• Understanding Hidden Memories of Recurrent Neural Networks
• Transcribing Content from Structural Images with Spotlight Mechanism
• Unsupervised temporal context learning using convolutional neural networks for laparoscopic workflow analysis
• Query-Based Abstractive Summarization Using Neural Networks
• Named Entity Recognition with stack residual LSTM and trainable bias decoding
• Streaming Normalization: Towards Simpler and More Biologically-plausible Normalizations for Online and Recurrent Learning
• From phonemes to images: levels of representation in a recurrent neural model of visually-grounded language learning
• Algorithmic Composition of Melodies with Deep Recurrent Neural Networks
• Large-Scale User Modeling with Recurrent Neural Networks for Music Discovery on Multiple Time Scales
• Multi-Scale Attention with Dense Encoder for Handwritten Mathematical Expression Recognition
• Highrisk Prediction from Electronic Medical Records via Deep Attention Networks
• On the Benefit of Combining Neural, Statistical and External Features for Fake News Identification
• Representation-Aggregation Networks for Segmentation of Multi-Gigapixel Histology Images
• Improving Interpretability of Deep Neural Networks with Semantic Information
• Composing Meta-Policies for Autonomous Driving Using Hierarchical Deep Reinforcement Learning
• Improving Background Based Conversation with Context-aware Knowledge Pre-selection
• Modelling Sentence Pairs with Tree-structured Attentive Encoder
• Are Emojis Predictable?
• The Statistical Recurrent Unit
• From Deterministic to Generative: Multi-Modal Stochastic RNNs for Video Captioning
• Exploration with Unreliable Intrinsic Reward in Multi-Agent Reinforcement Learning
• Style Imitation and Chord Invention in Polyphonic Music with Exponential Families
• Context Gates for Neural Machine Translation
• Separating Answers from Queries for Neural Reading Comprehension
• Aspect-Based Relational Sentiment Analysis Using a Stacked Neural Network Architecture
• End-to-end Learning for 3D Facial Animation from Raw Waveforms of Speech
• Bidirectional Recurrent Neural Networks for Medical Event Detection in Electronic Health Records
• Lyrics-Based Music Genre Classification Using a Hierarchical Attention Network
• Large-scale weakly supervised audio classification using gated convolutional neural network
• Rare Disease Detection by Sequence Modeling with Generative Adversarial Networks
• Code Attention: Translating Code to Comments by Exploiting Domain Features
• A GRU-based Encoder-Decoder Approach with Attention for Online Handwritten Mathematical Expression Recognition
• A Comparison of deep learning methods for environmental sound
• Enabling My Robot To Play Pictionary : Recurrent Neural Networks For Sketch Recognition
• Improving Description-based Person Re-identification by Multi-granularity Image-text Alignments
• Human Action Forecasting by Learning Task Grammars
• Unbounded cache model for online language modeling with open vocabulary
• Sequential Dual Deep Learning with Shape and Texture Features for Sketch Recognition
• Convolutional Residual Memory Networks
• Neural Machine Translation with Word Predictions
• Mining YouTube - A dataset for learning fine-grained action concepts from webly supervised video data
• Towards Bidirectional Hierarchical Representations for Attention-Based Neural Machine Translation
• GCDT: A Global Context Enhanced Deep Transition Architecture for Sequence Labeling
• Convolutional Gated Recurrent Neural Network Incorporating Spatial Features for Audio Tagging
• Grammatical Error Correction with Neural Reinforcement Learning
• Simplified Minimal Gated Unit Variations for Recurrent Neural Networks
• Learning language through pictures
• Classification-based RNN machine translation using GRUs
• Sequence Modeling using Gated Recurrent Neural Networks
• Recurrent Neural Network based Part-of-Speech Tagger for Code-Mixed Social Media Text
• Byte-based Language Identification with Deep Convolutional Networks
• ACtuAL: Actor-Critic Under Adversarial Learning
• A Simple LSTM model for Transition-based Dependency Parsing
• SenGen: Sentence Generating Neural Variational Topic Model
• Attention and Localization based on a Deep Convolutional Recurrent Model for Weakly Supervised Audio Tagging
• End-to-End Video Classification with Knowledge Graphs
• A Bi-directional Transformer for Musical Chord Recognition
• Variational Bi-LSTMs
• Better Text Understanding Through Image-To-Text Transfer
• Syntactically Informed Text Compression with Recurrent Neural Networks
• CNN Is All You Need
• Recurrent Neural Networks for anomaly detection in the Post-Mortem time series of LHC superconducting magnets
• Social Scene Understanding: End-to-End Multi-Person Action Localization and Collective Activity Recognition
• Trajectory-based Radical Analysis Network for Online Handwritten Chinese Character Recognition
• A Sequential Matching Framework for Multi-turn Response Selection in Retrieval-based Chatbots
• Simplified Long Short-term Memory Recurrent Neural Networks: part III
• An Attention Mechanism for Answer Selection Using a Combined Global and Local View
• Trimming and Improving Skip-thought Vectors
• Tricks from Deep Learning
• Neural Response Generation with Meta-Words
• Deep Semi-Supervised Learning with Linguistically Motivated Sequence Labeling Task Hierarchies
• New Item Consumption Prediction Using Deep Learning
• Cross-lingual Word Segmentation and Morpheme Segmentation as Sequence Labelling
• Evaluating the Performance of Machine Learning Algorithms in Financial Market Forecasting: A Comprehensive Survey
• An Efficient Character-Level Neural Machine Translation
• Online Signature Verification using Recurrent Neural Network and Length-normalized Path Signature
• Recurrent Memory Array Structures
• Imitating Driver Behavior with Generative Adversarial Networks
• Text-guided Attention Model for Image Captioning
• Acoustic Feature Learning via Deep Variational Canonical Correlation Analysis
• Autoencoder Regularized Network For Driving Style Representation Learning
• Sequential Keystroke Behavioral Biometrics for Mobile User Identification via Multi-view Deep Learning
• Attention-based Memory Selection Recurrent Network for Language Modeling
• Simplified Gating in Long Short-term Memory (LSTM) Recurrent Neural Networks
• Exploring Different Dimensions of Attention for Uncertainty Detection
• Topic Sensitive Neural Headline Generation
• Empirical Evaluation of RNN Architectures on Sentence Classification Task
• Memory Visualization for Gated Recurrent Neural Networks in Speech Recognition
• A Neural Network Alternative to Non-Negative Audio Models
• Rethinking Skip-thought: A Neighborhood based Approach
• End-to-End Radio Traffic Sequence Recognition with Deep Recurrent Neural Networks
• A Language-independent and Compositional Model for Personality Trait Recognition from Short Texts
• Multi-Graph Decoding for Code-Switching ASR
• Syntax-Infused Variational Autoencoder for Text Generation
• Generating Nontrivial Melodies for Music as a Service
• A Stock Selection Method Based on Earning Yield Forecast Using Sequence Prediction Models
• Audio-replay attack detection countermeasures
• PoseConvGRU: A Monocular Approach for Visual Ego-motion Estimation by Learning
• Investigating how well contextual features are captured by bi-directional recurrent neural network models
• Recurrent Value Functions
• A Multifaceted Evaluation of Neural versus Phrase-Based Machine Translation for 9 Language Directions
• Ask Me Even More: Dynamic Memory Tensor Networks (Extended Model)
• Character-Based Text Classification using Top Down Semantic Model for Sentence Representation
• Balanced Quantization: An Effective and Efficient Approach to Quantized Neural Networks
• Cavs: A Vertex-centric Programming Interface for Dynamic Neural Networks
• Compositional Structure Learning for Sequential Video Data
• Self-view Grounding Given a Narrated 360° Video
• Bringing Semantic Structures to User Intent Detection in Online Medical Queries
• Variational Sequential Labelers for Semi-Supervised Learning
• Predicting Patient State-of-Health using Sliding Window and Recurrent Classifiers
• Empirical Evaluation of A New Approach to Simplifying Long Short-term Memory (LSTM)
• Memory Consolidation for Contextual Spoken Language Understanding with Dialogue Logistic Inference
• A Strong and Robust Baseline for Text-Image Matching
• A Semi-Supervised Approach for Low-Resourced Text Generation
• Deep Fisher Discriminant Learning for Mobile Hand Gesture Recognition
• NeuroTrainer: An Intelligent Memory Module for Deep Learning Training
• Recurrent Estimation of Distributions
• Sensor Transformation Attention Networks
• Content-Based Table Retrieval for Web Queries
• Dual Attention Network for Product Compatibility and Function Satisfiability Analysis
• SAM: Semantic Attribute Modulation for Language Modeling and Style Variation
• Topical Behavior Prediction from Massive Logs
• Click-Through Rate Prediction with the User Memory Network
• Recurrent Existence Determination Through Policy Optimization
• CRNN: A Joint Neural Network for Redundancy Detection
• Two-stream Spatiotemporal Feature for Video QA Task
• Network of Recurrent Neural Networks
• Conversational Response Re-ranking Based on Event Causality and Role Factored Tensor Event Embedding
• Learning Longer-term Dependencies via Grouped Distributor Unit
• Target-Guided Open-Domain Conversation
• Latent Multi-Criteria Ratings for Recommendations
• Conversational Help for Task Completion and Feature Discovery in Personal Assistants
• Predicting Annotation Difficulty to Improve Task Routing and Model Performance for Biomedical Information Extraction
• Towards the Improvement of Automated Scientific Document Categorization by Deep Learning
• Incremental Learning from Scratch for Task-Oriented Dialogue Systems
• Improving Minimal Gated Unit for Sequential Data
• Neural Image Captioning