Scaling laws and fluctuations in the statistics of word frequencies

Abstract

In this paper we combine statistical analysis of large text databases and simple stochastic models to explain the appearance of scaling laws in the statistics of word frequencies. Besides the sublinear scaling of the vocabulary size with database size (Heaps' law), here we report a new scaling of the fluctuations around this average (fluctuation scaling analysis). We explain both scaling laws by modeling the usage of words by simple stochastic processes in which the overall distribution of word-frequencies is fat tailed (Zipf's law) and the frequency of a single word is subject to fluctuations across documents (as in topic models). In this framework, the mean and the variance of the vocabulary size can be expressed as quenched averages, implying that: i) the inhomogeneous dissemination of words cause a reduction of the average vocabulary size in comparison to the homogeneous case, and ii) correlations in the co-occurrence of words lead to an increase in the variance and the vocabulary size becomes a non-self-averaging quantity. We address the implications of these observations to the measurement of lexical richness. We test our results in three large text databases (Google-ngram, Enlgish Wikipedia, and a collection of scientific articles).

arXiv comment19 pages, 4 figures

References in corpus (14)

• Power-law distributions in empirical data
• Power laws, Pareto distributions and Zipf's law
• Fluctuation scaling in complex systems: Taylor's law and beyond
• Fluctuations in network dynamics
• Stochastic model for the vocabulary growth in natural languages
• Collective dynamics of social annotation
• Zipf's Law Leads to Heaps' Law: Analyzing Their Relation in Finite-Size Systems
• Notes on the occupancy problem with infinitely many boxes: general asymptotics and power laws
• A practical approach to language complexity: a Wikipedia case study
• Niche as a determinant of word fate in online groups
• A scaling law beyond Zipf's law and its relation to Heaps' law
• Towards the quantification of the semantic information encoded in written language
• The meta book and size-dependent properties of written language
• Innovation and Nested Preferential Growth in Chess Playing Behavior

Cited by in corpus (22)

• Statistical laws in linguistics
• Zipf's, Heaps' and Taylor's laws are determined by the expansion into the adjacent possible
• The exploration of the Adjacent Possible explains the emergence and evolution of social networks
• Similarity of symbol frequency distributions with heavy tails
• Taylor's law in innovation processes
• Log-log Convexity of Type-Token Growth in Zipf's Systems
• Zipf and Heaps laws from dependency structures in component systems
• Impact of lexical and sentiment factors on the popularity of scientific papers
• Generalized Entropies and the Similarity of Texts
• Spatial evolution of human dialects
• Maximal Diversity and Zipf's Law
• Heaps' law, statistics of shared components and temporal patterns from a sample-space-reducing process
• Scaling laws and dynamics of hashtags on Twitter
• Statistical properties of fluctuations of time series representing the appearance of words in nationwide blog data and their applications: An example of observations and the modelling of fluctuation scalings of nonstationary time series
• Information-theoretical analysis of the statistical dependencies among three variables: Applications to written language
• Statistical Physics of Language Maps in the USA
• Phase transitions in a decentralized graph-based approach to human language
• Did AI get more negative recently?
• A Comparison of Two Fluctuation Analyses for Natural Language Clustering Phenomena: Taylor and Ebeling & Neiman Methods
• Variation of word frequencies in Russian literary texts
• Relations between anomalous diffusion and fluctuation scaling: The case of ultraslow diffusion and time-scale-independent fluctuation scaling in language
• Repetition and recurrence times: Dual statements and summable mixing rates