Do Neural Nets Learn Statistical Laws behind Natural Language?

Abstract

The performance of deep learning in natural language processing has been spectacular, but the reasons for this success remain unclear because of the inherent complexity of deep learning. This paper provides empirical evidence of its effectiveness and of a limitation of neural networks for language engineering. Precisely, we demonstrate that a neural language model based on long short-term memory (LSTM) effectively reproduces Zipf's law and Heaps' law, two representative statistical properties underlying natural language. We discuss the quality of reproducibility and the emergence of Zipf's law and Heaps' law as training progresses. We also point out that the neural language model has a limitation in reproducing long-range correlation, another statistical property of natural language. This understanding could provide a direction for improving the architectures of neural networks.

arXiv comment21 pages, 11 figures

References in corpus (21)

• Adam: A Method for Stochastic Optimization
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• WaveNet: A Generative Model for Raw Audio
• On the Number of Linear Regions of Deep Neural Networks
• Long-Term Clustering, Scaling, and Universality in the Temporal Occurrence of Earthquakes
• LSTM-based Deep Learning Models for Non-factoid Answer Selection
• Why does deep and cheap learning work so well?
• SampleRNN: An Unconditional End-to-End Neural Audio Generation Model
• Neural Machine Translation in Linear Time
• Exponential expressivity in deep neural networks through transient chaos
• Beyond word frequency: Bursts, lulls, and scaling in the temporal distributions of words
• Entropy and Long range correlations in literary English
• Entropy and Long range correlations in literary English
• Zipf's Law Leads to Heaps' Law: Analyzing Their Relation in Finite-Size Systems
• On the Expressive Power of Deep Learning: A Tensor Analysis
• On the origin of long-range correlations in texts
• Correlations and invariance of seismicity under renormalization-group transformations
• Language Time Series Analysis
• N-gram Language Modeling using Recurrent Neural Network Estimation
• Convolutional Rectifier Networks as Generalized Tensor Decompositions
• Comment on "Why does deep and cheap learning work so well?" [arXiv:1608.08225]

Cited by in corpus (5)

• Natural Language Statistical Features of LSTM-generated Texts
• Mutual Information Scaling and Expressive Power of Sequence Models
• Assessing Language Models with Scaling Properties
• On Extending NLP Techniques from the Categorical to the Latent Space: KL Divergence, Zipf's Law, and Similarity Search
• Evaluating Computational Language Models with Scaling Properties of Natural Language