Mutual Information Scaling and Expressive Power of Sequence Models
arXiv:1905.04271
Abstract
Sequence models assign probabilities to variable-length sequences such as natural language texts. The ability of sequence models to capture temporal dependence can be characterized by the temporal scaling of correlation and mutual information. In this paper, we study the mutual information of recurrent neural networks (RNNs) including long short-term memories and self-attention networks such as Transformers. Through a combination of theoretical study of linear RNNs and empirical study of nonlinear RNNs, we find their mutual information decays exponentially in temporal distance. On the other hand, Transformers can capture long-range mutual information more efficiently, making them preferable in modeling sequences with slow power-law mutual information, such as natural languages and stock prices. We discuss the connection of these results with statistical mechanics. We also point out the non-uniformity problem in many natural language datasets. We hope this work provides a new perspective in understanding the expressive power of sequence models and shed new light on improving the architecture of them.
arXiv comment12 + 15 pages. Comments are welcome
References in corpus (15)
- Adam: A Method for Stochastic Optimization
- Neural Machine Translation by Jointly Learning to Align and Translate
- Sequence to Sequence Learning with Neural Networks
- Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
- Visualizing and Understanding Recurrent Networks
- Opening the Black Box of Deep Neural Networks via Information
- End-To-End Memory Networks
- Inferring Algorithmic Patterns with Stack-Augmented Recurrent Nets
- Entropy and Long range correlations in literary English
- Entropy and Long range correlations in literary English
- Learning to Transduce with Unbounded Memory
- Improving Neural Language Models with a Continuous Cache
- Entropy Estimates from Insufficient Samplings
- Ising model description of long range correlations in DNA sequences
- Do Neural Nets Learn Statistical Laws behind Natural Language?