Hierarchical Multiscale Recurrent Neural Networks
arXiv:1609.01704
Abstract
Learning both hierarchical and temporal representation has been among the long-standing challenges of recurrent neural networks. Multiscale recurrent neural networks have been considered as a promising approach to resolve this issue, yet there has been a lack of empirical evidence showing that this type of models can actually capture the temporal dependencies by discovering the latent hierarchical structure of the sequence. In this paper, we propose a novel multiscale approach, called the hierarchical multiscale recurrent neural networks, which can capture the latent hierarchical structure in the sequence by encoding the temporal dependencies with different timescales using a novel update mechanism. We show some evidence that our proposed multiscale architecture can discover underlying hierarchical structure in the sequences without using explicit boundary information. We evaluate our proposed model on character-level language modelling and handwriting sequence modelling.
References in corpus (4)
Cited by in corpus (16)
- Neural Machine Translation in Linear Time
- Unsupervised Learning of Disentangled and Interpretable Representations from Sequential Data
- Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learning
- Dynamic Evaluation of Neural Sequence Models
- Fast-Slow Recurrent Neural Networks
- SyntaxNet Models for the CoNLL 2017 Shared Task
- Deep-ESN: A Multiple Projection-encoding Hierarchical Reservoir Computing Framework
- Event Identification as a Decision Process with Non-linear Representation of Text
- Surprisal-Driven Zoneout
- Learning to Attend, Copy, and Generate for Session-Based Query Suggestion
- Multiscale sequence modeling with a learned dictionary
- Hierarchical Multi-scale Attention Networks for Action Recognition
- Stochastic Sequential Neural Networks with Structured Inference
- Shortcut Sequence Tagging
- Rotational Unit of Memory
- Options Discovery with Budgeted Reinforcement Learning