Gated Feedback Recurrent Neural Networks
arXiv:1502.02367
Abstract
In this work, we propose a novel recurrent neural network (RNN) architecture. The proposed RNN, gated-feedback RNN (GF-RNN), extends the existing approach of stacking multiple recurrent layers by allowing and controlling signals flowing from upper recurrent layers to lower layers using a global gating unit for each pair of layers. The recurrent signals exchanged between layers are gated adaptively based on the previous hidden states and the current input. We evaluated the proposed GF-RNN with different types of recurrent units, such as tanh, long short-term memory and gated recurrent units, on the tasks of character-level language modeling and Python program evaluation. Our empirical evaluation of different RNN units, revealed that in both tasks, the GF-RNN outperforms the conventional approaches to build deep stacked RNNs. We suggest that the improvement arises because the GF-RNN can adaptively assign different layers to different timescales and layer-to-layer interactions (including the top-down ones which are not usually present in a stacked RNN) by learning to gate these interactions.
9 pages, removed appendix
References in corpus (5)
Cited by in corpus (63)
- Spatio-Temporal Backpropagation for Training High-performance Spiking Neural Networks
- A Decade Survey of Content Based Image Retrieval using Deep Learning
- ReNet: A Recurrent Neural Network Based Alternative to Convolutional Networks
- LXMERT: Learning Cross-Modality Encoder Representations from Transformers
- Flow: A Modular Learning Framework for Mixed Autonomy Traffic
- Recurrent Neural Networks (RNNs): A gentle Introduction and Overview
- Reinforcement Learning for Angle-Only Intercept Guidance of Maneuvering Targets
- GeoTrackNet-A Maritime Anomaly Detector using Probabilistic Neural Network Representation of AIS Tracks and A Contrario Detection
- Adaptive Guidance and Integrated Navigation with Reinforcement Meta-Learning
- A Survey of Deep Learning Techniques for Neural Machine Translation
- Representation Learning for Natural Language Processing
- Explicit Sparse Transformer: Concentrated Attention Through Explicit Selection
- Hitchhiker's Guide to Super-Resolution: Introduction and Recent Advances
- Six Degree-of-Freedom Body-Fixed Hovering over Unmapped Asteroids via LIDAR Altimetry and Reinforcement Meta-Learning
- 3G structure for image caption generation
- Deep Independently Recurrent Neural Network (IndRNN)
- DeepGS: Deep Representation Learning of Graphs and Sequences for Drug-Target Binding Affinity Prediction
- Addressing Some Limitations of Transformers with Feedback Memory
- Discrete Event, Continuous Time RNNs
- Gated Recurrent Neural Tensor Network
- Reinforcement Meta-Learning for Interception of Maneuvering Exoatmospheric Targets with Parasitic Attitude Loop
- PAM: Understanding Product Images in Cross Product Category Attribute Extraction
- LCSTS: A Large Scale Chinese Short Text Summarization Dataset
- The model of an anomaly detector for HiLumi LHC magnets based on Recurrent Neural Networks and adaptive quantization
- Deep Learning for Insider Threat Detection: Review, Challenges and Opportunities
- Language models and Automated Essay Scoring
- Gated Recurrent Neural Network Approach for Multilabel Emotion Detection in Microblogs
- SGCN:Sparse Graph Convolution Network for Pedestrian Trajectory Prediction
- To Tune or Not To Tune? How About the Best of Both Worlds?
- Interstellar: Searching Recurrent Architecture for Knowledge Graph Embedding
- Adaptive Guidance with Reinforcement Meta-Learning
- TextScanner: Reading Characters in Order for Robust Scene Text Recognition
- Sequential Dual Deep Learning with Shape and Texture Features for Sketch Recognition
- DBTagger: Multi-Task Learning for Keyword Mapping in NLIDBs Using Bi-Directional Recurrent Neural Networks
- From Theories on Styles to their Transfer in Text: Bridging the Gap with a Hierarchical Survey
- Recurrent Neural Networks with Stochastic Layers for Acoustic Novelty Detection
- PO-EMO: Conceptualization, Annotation, and Modeling of Aesthetic Emotions in German and English Poetry
- Counterfactual Credit Assignment in Model-Free Reinforcement Learning
- MEMO: A Deep Network for Flexible Combination of Episodic Memories
- Pelican: A Deep Residual Network for Network Intrusion Detection
- Unsupervised and Efficient Vocabulary Expansion for Recurrent Neural Network Language Models in ASR
- Empowering A* Search Algorithms with Neural Networks for Personalized Route Recommendation
- Learning to Denoise and Decode: A Novel Residual Neural Network Decoder for Polar Codes
- Revisiting Cross Modal Retrieval
- Multi-Scale Self-Attention for Text Classification
- Weakly Labeled Sound Event Detection Using Tri-training and Adversarial Learning
- MV-RNN: A Multi-View Recurrent Neural Network for Sequential Recommendation
- Piecewise Linear Units Improve Deep Neural Networks
- Finding a Needle in a Haystack: Tiny Flying Object Detection in 4K Videos using a Joint Detection-and-Tracking Approach
- RoadTagger: Robust Road Attribute Inference with Graph Neural Networks
- Time-Series Event Prediction with Evolutionary State Graph
- Formal models of Structure Building in Music, Language and Animal Songs
- Sequence-to-Set Semantic Tagging: End-to-End Multi-label Prediction using Neural Attention for Complex Query Reformulation and Automated Text Categorization
- Learned Interpretable Residual Extragradient ISTA for Sparse Coding
- SocialAI: Benchmarking Socio-Cognitive Abilities in Deep Reinforcement Learning Agents
- Exploiting Motion Information from Unlabeled Videos for Static Image Action Recognition
- A Deep Neural Framework for Contextual Affect Detection
- Cost-Quality Adaptive Active Learning for Chinese Clinical Named Entity Recognition
- Towards Privacy-Preserving Affect Recognition: A Two-Level Deep Learning Architecture
- Learning Preferences and Demands in Visual Recommendation
- Tree Search Network for Sparse Regression
- GWU NLP Lab at SemEval-2019 Task 3: EmoContext: Effective Contextual Information in Models for Emotion Detection in Sentence-level in a Multigenre Corpus
- Modeling Dyadic Conversations for Personality Inference