Character n-gram Embeddings to Improve RNN Language Models
arXiv:1906.05506
Abstract
This paper proposes a novel Recurrent Neural Network (RNN) language model that takes advantage of character information. We focus on character n-grams based on research in the field of word embedding construction (Wieting et al. 2016). Our proposed method constructs word embeddings from character n-gram embeddings and combines them with ordinary word embeddings. We demonstrate that the proposed method achieves the best perplexities on the language modeling datasets: Penn Treebank, WikiText-2, and WikiText-103. Moreover, we conduct experiments on application tasks: machine translation and headline generation. The experimental results indicate that our proposed method also positively affects these tasks.
arXiv commentAAAI 2019 paper
References in corpus (12)
- Sequence to Sequence Learning with Neural Networks
- Recurrent Neural Network Regularization
- Language Modeling with Gated Convolutional Networks
- Character-Aware Neural Language Models
- Exploring the Limits of Language Modeling
- An Empirical Study of Smoothing Techniques for Language Modeling
- Pointer Sentinel Mixture Models
- Selective Encoding for Abstractive Sentence Summarization
- An Analysis of Neural Language Modeling at Multiple Scales
- Improving Neural Language Models with a Continuous Cache
- Dynamic Evaluation of Neural Sequence Models
- Source-side Prediction for Neural Headline Generation