From Boltzmann to Zipf through Shannon and Jaynes

Abstract

The word-frequency distribution provides the fundamental building blocks that generate discourse in language. It is well known, from empirical evidence, that the word-frequency distribution of almost any text is described by Zipf's law, at least approximately. Following Stephens and Bialek [Phys. Rev. E 81, 066119, 2010], we interpret the frequency of any word as arising from the interaction potential between its constituent letters. Indeed, Jaynes' maximum-entropy principle, with the constrains given by every empirical two-letter marginal distribution, leads to a Boltzmann distribution for word probabilities, with an energy-like function given by the sum of all pairwise (two-letter) potentials. The improved iterative-scaling algorithm allows us finding the potentials from the empirical two-letter marginals. Appling this formalism to words with up to six letters from the English subset of the recently created Standardized Project Gutenberg Corpus, we find that the model is able to reproduce Zipf's law, but with some limitations: the general Zipf's power-law regime is obtained, but the probability of individual words shows considerable scattering. In this way, a pure statistical-physics framework is used to describe the probabilities of words. As a by-product, we find that both the empirical two-letter marginal distributions and the interaction-potential distributions follow well-defined statistical laws.

arXiv commentSubmitted to special issue "Entropy, Nonlinear Dynamics and Complexity"

References in corpus (26)

• Power-law distributions in empirical data
• Power laws, Pareto distributions and Zipf's law
• Crackling Noise
• Collaborative Tagging and Semiotic Dynamics
• Zipf's Law in Gene Expression
• Universal Fluctuations in Correlated Systems
• 25 Years of Self-Organized Criticality: Concepts and Controversies
• Measuring the evolution of contemporary western popular music
• Gibrat's law for cities: uniformly most powerful unbiased test of the Pareto against the lognormal
• Scale-free Networks Well Done
• Large-scale analysis of Zipf's law in English texts
• When do generalized entropies apply? How phase space volume determines entropy
• Power-law size distributions in geoscience revisited
• Understanding scaling through history-dependent processes with collapsing sample space
• Compression and the origins of Zipf's law for word frequencies
• Toward a statistical mechanics of four letter words
• A maximum entropy framework for non-exponential distributions
• A scaling law beyond Zipf's law and its relation to Heaps' law
• Scaling in the Timing of Extreme Events
• The distinct flavors of Zipf's law in the rank-size and in the size-distribution representations, and its maximum-likelihood fitting
• The brevity law as a scaling law, and a possible origin of Zipf's law for word frequencies
• Non-characteristic Half-lives in Radioactive Decay
• Extreme robustness of scaling in sample space reducing processes explains Zipf's law in diffusion on directed networks
• Phase transition, scaling of moments, and order-parameter distributions in Brownian particles and branching processes with finite-size effects
• Analysis of an information-theoretic model for communication
• Percolation on trees as a Brownian excursion: from Gaussian to Kolmogorov-Smirnov to Exponential statistics

Cited by in corpus (2)

• The brevity law as a scaling law, and a possible origin of Zipf's law for word frequencies
• Lognormals, Power Laws and Double Power Laws in the Distribution of Frequencies of Harmonic Codewords from Classical Music