Character-Level Language Modeling with Hierarchical Recurrent Neural Networks
arXiv:1609.03777
Abstract
Recurrent neural network (RNN) based character-level language models (CLMs) are extremely useful for modeling out-of-vocabulary words by nature. However, their performance is generally much worse than the word-level language models (WLMs), since CLMs need to consider longer history of tokens to properly predict the next one. We address this problem by proposing hierarchical RNN architectures, which consist of multiple modules with different timescales. Despite the multi-timescale structures, the input and output layers operate with the character-level clock, which allows the existing RNN CLM training approaches to be directly applicable without any modifications. Our CLM models show better perplexity than Kneser-Ney (KN) 5-gram WLMs on the One Billion Word Benchmark with only 2% of parameters. Also, we present real-time character-level end-to-end speech recognition examples on the Wall Street Journal (WSJ) corpus, where replacing traditional mono-clock RNN CLMs with the proposed models results in better recognition accuracies even though the number of parameters are reduced to 30%.
arXiv commentSubmitted to NIPS 2016 on May 20, 2016 (v1), accepted to ICASSP 2017 (v2)
References in corpus (9)
- Improving neural networks by preventing co-adaptation of feature detectors
- ADADELTA: An Adaptive Learning Rate Method
- Character-Aware Neural Language Models
- Exploring the Limits of Language Modeling
- Character-based Neural Machine Translation
- Finding Function in Form: Compositional Character Models for Open Vocabulary Word Representation
- Character-Level Incremental Speech Recognition with Recurrent Neural Networks
- Single stream parallelization of generalized LSTM-like RNNs on a GPU
- LightRNN: Memory and Computation-Efficient Recurrent Neural Networks