A scaling law beyond Zipf's law and its relation to Heaps' law

Abstract

The dependence with text length of the statistical properties of word occurrences has long been considered a severe limitation quantitative linguistics. We propose a simple scaling form for the distribution of absolute word frequencies which uncovers the robustness of this distribution as text grows. In this way, the shape of the distribution is always the same and it is only a scale parameter which increases linearly with text length. By analyzing very long novels we show that this behavior holds both for raw, unlemmatized texts and for lemmatized texts. For the latter case, the word-frequency distribution is well fit by a double power law, maintaining the Zipf's exponent value $\gamma \simeq 2$ for large frequencies but yielding a smaller exponent in the low frequency regime. The growth of the distribution with text length allows us to estimate the size of the vocabulary at each step and to propose an alternative to Heaps' law, which turns out to be intimately connected to Zipf's law, thanks to the scaling behavior.

Final version, to appear in NJP

References in corpus (5)

• Power-law distributions in empirical data
• Languages cool as they expand: Allometric scaling and the decreasing need for new words
• Stochastic model for the vocabulary growth in natural languages
• Zipf's Law Leads to Heaps' Law: Analyzing Their Relation in Finite-Size Systems
• Analysis of an information-theoretic model for communication

Cited by in corpus (26)

• Large-scale analysis of Zipf's law in English texts
• A complex network approach to stylometry
• Zipf's law for word frequencies: word forms versus lemmas in long texts
• Statistical laws in linguistics
• Scaling laws and fluctuations in the statistics of word frequencies
• The origins of Zipf's meaning-frequency law
• Interacting discovery processes on complex networks
• Authorship recognition via fluctuation analysis of network topology and word intermittency
• Optimization models of natural communication
• Rank diversity of languages: Generic behavior in computational linguistics
• Polysemy and brevity versus frequency in language
• The distinct flavors of Zipf's law in the rank-size and in the size-distribution representations, and its maximum-likelihood fitting
• Log-log Convexity of Type-Token Growth in Zipf's Systems
• The brevity law as a scaling law, and a possible origin of Zipf's law for word frequencies
• Complexity measurement of natural and artificial languages
• Maximal Diversity and Zipf's Law
• Truncated lognormal distributions and scaling in the size of naturally defined population clusters
• Scaling laws and dynamics of hashtags on Twitter
• Lognormals, Power Laws and Double Power Laws in the Distribution of Frequencies of Harmonic Codewords from Classical Music
• Heaps' Law and Vocabulary Richness in the History of Classical Music Harmony
• Randomness versus specifics for word-frequency distributions
• Entropy and type-token ratio in gigaword corpora
• From Boltzmann to Zipf through Shannon and Jaynes
• The Dependence of Frequency Distributions on Multiple Meanings of Words, Codes and Signs
• Variation of word frequencies in Russian literary texts
• Recovering Zipf's law in intercontinental scientific collaboration