Better Long-Range Dependency By Bootstrapping A Mutual Information Regularizer
arXiv:1905.11978
Abstract
In this work, we develop a novel regularizer to improve the learning of long-range dependency of sequence data. Applied on language modelling, our regularizer expresses the inductive bias that sequence variables should have high mutual information even though the model might not see abundant observations for complex long-range dependency. We show how the `next sentence prediction (classification)' heuristic can be derived in a principled way from our mutual information estimation framework, and be further extended to maximize the mutual information of sequence variables. The proposed approach not only is effective at increasing the mutual information of segments under the learned model but more importantly, leads to a higher likelihood on holdout data, and improved generation quality. Code is released at https://github.com/BorealisAI/BMI.
arXiv commentCamera-ready for AISTATS 2020
References in corpus (16)
- Adam: A Method for Stochastic Optimization
- Learning deep representations by mutual information estimation and maximization
- End-To-End Memory Networks
- A Simple Way to Initialize Recurrent Networks of Rectified Linear Units
- Pointer Sentinel Mixture Models
- Regularizing and Optimizing LSTM Language Models
- MINE: Mutual Information Neural Estimation
- A Clockwork RNN
- Learning Longer Memory in Recurrent Neural Networks
- Inferring Algorithmic Patterns with Stack-Augmented Recurrent Nets
- TopicRNN: A Recurrent Neural Network with Long-Range Semantic Dependency
- Learning Longer-term Dependencies in RNNs with Auxiliary Losses
- Breaking the Softmax Bottleneck: A High-Rank RNN Language Model
- Topic Compositional Neural Language Model
- Larger-Context Language Modelling
- What comes next? Extractive summarization by next-sentence prediction