Input-to-Output Gate to Improve RNN Language Models

Abstract

This paper proposes a reinforcing method that refines the output layers of existing Recurrent Neural Network (RNN) language models. We refer to our proposed method as Input-to-Output Gate (IOG). IOG has an extremely simple structure, and thus, can be easily combined with any RNN language models. Our experiments on the Penn Treebank and WikiText-2 datasets demonstrate that IOG consistently boosts the performance of several different types of current topline RNN language models.

Accepted as a conference paper in IJCNLP 2017

References in corpus (5)

• Sequence to Sequence Learning with Neural Networks
• Neural Architecture Search with Reinforcement Learning
• Recurrent Neural Network Regularization
• Pointer Sentinel Mixture Models
• Regularizing and Optimizing LSTM Language Models