Revisiting the Hierarchical Multiscale LSTM
arXiv:1807.03595
Abstract
Hierarchical Multiscale LSTM (Chung et al., 2016a) is a state-of-the-art language model that learns interpretable structure from character-level input. Such models can provide fertile ground for (cognitive) computational linguistics studies. However, the high complexity of the architecture, training procedure and implementations might hinder its applicability. We provide a detailed reproduction and ablation study of the architecture, shedding light on some of the potential caveats of re-purposing complex deep-learning architectures. We further show that simplifying certain aspects of the architecture can in fact improve its performance. We also investigate the linguistic units (segments) learned by various levels of the model, and argue that their quality does not correlate with the overall performance of the model on language modeling.
arXiv commentTo appear in COLING 2018 (reproduction track)
References in corpus (6)
- Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
- Understanding Neural Networks through Representation Erasure
- Hierarchical Multiscale Recurrent Neural Networks
- Representations of language in a model of visually grounded speech signal
- Character-Level Question Answering with Attention
- What Do Recurrent Neural Network Grammars Learn About Syntax?
Cited by in corpus (4)
- Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLP
- Learning to Discover, Ground and Use Words with Segmental Neural Language Models
- VLSTM: Very Long Short-Term Memory Networks for High-Frequency Trading
- Indicatements that character language models learn English morpho-syntactic units and regularities