Hierarchical Multiscale Recurrent Neural Networks

Abstract

Learning both hierarchical and temporal representation has been among the long-standing challenges of recurrent neural networks. Multiscale recurrent neural networks have been considered as a promising approach to resolve this issue, yet there has been a lack of empirical evidence showing that this type of models can actually capture the temporal dependencies by discovering the latent hierarchical structure of the sequence. In this paper, we propose a novel multiscale approach, called the hierarchical multiscale recurrent neural networks, which can capture the latent hierarchical structure in the sequence by encoding the temporal dependencies with different timescales using a novel update mechanism. We show some evidence that our proposed multiscale architecture can discover underlying hierarchical structure in the sequences without using explicit boundary information. We evaluate our proposed model on character-level language modelling and handwriting sequence modelling.

References in corpus (18)

• Adam: A Method for Stochastic Optimization
• Deep Learning in Neural Networks: An Overview
• Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• BinaryConnect: Training Deep Neural Networks with binary weights during propagations
• Convolutional Neural Networks for Sentence Classification
• Grid Long Short-Term Memory
• Deep Networks with Stochastic Depth
• Character-based Neural Machine Translation
• Zoneout: Regularizing RNNs by Randomly Preserving Hidden Activations
• Architectural Complexity Measures of Recurrent Neural Networks
• Strategic Attentive Writer for Learning Macro-Actions
• One-Shot Adaptation of Supervised Deep Convolutional Models
• On Multiplicative Integration with Recurrent Neural Networks
• Regularization and nonlinearities for neural language models: when are they needed?
• Regularizing RNNs by Stabilizing Activations
• Surprisal-Driven Feedback in Recurrent Networks
• Recurrent Memory Array Structures

Cited by in corpus (101)

• An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling
• A Decade Survey of Content Based Image Retrieval using Deep Learning
• Neural Machine Translation in Linear Time
• Learning Multimodal Graph-to-Graph Translation for Molecular Optimization
• Unsupervised Learning of Disentangled and Interpretable Representations from Sequential Data
• CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representation
• An Analysis of Neural Language Modeling at Multiple Scales
• Lifelong Sequential Modeling with Personalized Memorization for User Response Prediction
• Independently Recurrent Neural Network (IndRNN): Building A Longer and Deeper RNN
• Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learning
• Review: Deep Learning in Electron Microscopy
• Deep Neural Decision Trees
• Multiplicative LSTM for sequence modelling
• Explicit Sparse Transformer: Concentrated Attention Through Explicit Selection
• Ordered Neurons: Integrating Tree Structures into Recurrent Neural Networks
• Unsupervised Automatic Speech Recognition: A Review
• Recurrent Independent Mechanisms
• Long-term Forecasting using Higher Order Tensor RNNs
• Fast differentiable DNA and protein sequence optimization for molecular design
• Encoding word order in complex embeddings
• Breaking the Softmax Bottleneck: A High-Rank RNN Language Model
• Dynamic Evaluation of Neural Sequence Models
• Gram-CTC: Automatic Unit Selection and Target Decomposition for Sequence Labelling
• Recurrent Batch Normalization
• Ithemal: Accurate, Portable and Fast Basic Block Throughput Estimation using Deep Neural Networks
• Dual-path RNN: efficient long sequence modeling for time-domain single-channel speech separation
• Compressive Transformers for Long-Range Sequence Modelling
• Dual Rectified Linear Units (DReLUs): A Replacement for Tanh Activation Functions in Quasi-Recurrent Neural Networks
• Maybe Deep Neural Networks are the Best Choice for Modeling Source Code
• Fast-Slow Recurrent Neural Networks
• SyntaxNet Models for the CoNLL 2017 Shared Task
• DVAE++: Discrete Variational Autoencoders with Overlapping Transformations
• Skip RNN: Learning to Skip State Updates in Recurrent Neural Networks
• 3G structure for image caption generation
• Deep-ESN: A Multiple Projection-encoding Hierarchical Reservoir Computing Framework
• Improving the Gating Mechanism of Recurrent Neural Networks
• Learning to Create and Reuse Words in Open-Vocabulary Neural Language Modeling
• Focused Hierarchical RNNs for Conditional Sequence Processing
• Evaluating Text GANs as Language Models
• Neural Language Modeling by Jointly Learning Syntax and Lexicon
• PonderNet: Learning to Ponder
• Deep reinforcement learning for time series: playing idealized trading games
• Improved Hierarchical Patient Classification with Language Model Pretraining over Clinical Notes
• Clockwork Variational Autoencoders
• Adversarial Domain Adaptation for Variational Neural Language Generation in Dialogue Systems
• Temporal Difference Variational Auto-Encoder
• Revisiting the Hierarchical Multiscale LSTM
• Time Perception Machine: Temporal Point Processes for the When, Where and What of Activity Prediction
• Deep Learning for Multi-Scale Changepoint Detection in Multivariate Time Series
• InfoCatVAE: Representation Learning with Categorical Variational Autoencoders
• Learning to Attend, Copy, and Generate for Session-Based Query Suggestion
• Unsupervised Video Decomposition using Spatio-temporal Iterative Inference
• MEMO: A Deep Network for Flexible Combination of Episodic Memories
• Multiscale sequence modeling with a learned dictionary
• Event Identification as a Decision Process with Non-linear Representation of Text
• Surprisal-Driven Zoneout
• Discourse-Wizard: Discovering Deep Discourse Structure in your Conversation with RNNs
• Do latent tree learning models identify meaningful structure in sentences?
• Hide-and-Seek: A Template for Explainable AI
• Exploring the Back Alleys: Analysing The Robustness of Alternative Neural Network Architectures against Adversarial Attacks
• EdgeNet: A novel approach for Arabic numeral classification
• Hierarchical Multi-scale Attention Networks for Action Recognition
• Benchmarking Deep Sequential Models on Volatility Predictions for Financial Time Series
• Cell-aware Stacked LSTMs for Modeling Sentences
• Multi-Scale Self-Attention for Text Classification
• Image Recognition Using Scale Recurrent Neural Networks
• Dynamic Automatic Differentiation of GPU Broadcast Kernels
• Multi-scale Transformer Language Models
• Encoding-based Memory Modules for Recurrent Neural Networks
• Learning Task Decomposition with Ordered Memory Policy Network
• One Size Does Not Fit All: Multi-Scale, Cascaded RNNs for Radar Classification
• Stochastic Sequential Neural Networks with Structured Inference
• Two-Timescale End-to-End Learning for Channel Acquisition and Hybrid Precoding
• Long Short-Term Memory with Dynamic Skip Connections
• VLSTM: Very Long Short-Term Memory Networks for High-Frequency Trading
• Bridging the gap between emotion and joint action
• Low-pass Recurrent Neural Networks - A memory architecture for longer-term correlation discovery
• Shortcut Sequence Tagging
• Rotational Unit of Memory
• Phonetic Temporal Neural Model for Language Identification
• Multi-Zone Unit for Recurrent Neural Networks
• Learning to Adaptively Scale Recurrent Neural Networks
• Options Discovery with Budgeted Reinforcement Learning
• Learning Simpler Language Models with the Differential State Framework
• Cut-Based Graph Learning Networks to Discover Compositional Structure of Sequential Video Data
• Recurrent Neural Networks with Mixed Hierarchical Structures for Natural Language Processing
• Recursive Top-Down Production for Sentence Generation with Latent Trees
• Perception-Prediction-Reaction Agents for Deep Reinforcement Learning
• Linear Memory Networks
• When in Doubt, Summon the Titans: Efficient Inference with Large Models
• ARMIN: Towards a More Efficient and Light-weight Recurrent Memory Network
• Toward Human-Level Artificial Intelligence
• Decoupling Hierarchical Recurrent Neural Networks With Locally Computable Losses
• Meta-Learning a Dynamical Language Model
• Training a Subsampling Mechanism in Expectation
• Solving Large-Scale 0-1 Knapsack Problems and its Application to Point Cloud Resampling
• A memory enhanced LSTM for modeling complex temporal dependencies
• Dual Latent Variable Model for Low-Resource Natural Language Generation in Dialogue Systems
• A Hierarchical Transformer for Unsupervised Parsing
• Analysis of memory in LSTM-RNNs for source separation
• DA-LSTM: A Long Short-Term Memory with Depth Adaptive to Non-uniform Information Flow in Sequential Data