Zipf's law for word frequencies: word forms versus lemmas in long texts

Abstract

Zipf's law is a fundamental paradigm in the statistics of written and spoken natural language as well as in other communication systems. We raise the question of the elementary units for which Zipf's law should hold in the most natural way, studying its validity for plain word forms and for the corresponding lemma forms. In order to have as homogeneous sources as possible, we analyze some of the longest literary texts ever written, comprising four different languages, with different levels of morphological complexity. In all cases Zipf's law is fulfilled, in the sense that a power-law distribution of word or lemma frequencies is valid for several orders of magnitude. We investigate the extent to which the word-lemma transformation preserves two parameters of Zipf's law: the exponent and the low-frequency cut-off. We are not able to demonstrate a strict invariance of the tail, as for a few texts both exponents deviate significantly, but we conclude that the exponents are very similar, despite the remarkable transformation that going from words to lemmas represents, considerably affecting all ranges of frequencies. In contrast, the low-frequency cut-offs are less stable.

References in corpus (14)

• Power-law distributions in empirical data
• Measuring the evolution of contemporary western popular music
• Languages cool as they expand: Allometric scaling and the decreasing need for new words
• Gibrat's law for cities: uniformly most powerful unbiased test of the Pareto against the lognormal
• Stochastic model for the vocabulary growth in natural languages
• Universality of rain event size distributions
• Correlations and invariance of seismicity under renormalization-group transformations
• A scaling law beyond Zipf's law and its relation to Heaps' law
• Scaling in the Timing of Extreme Events
• The span of correlations in dolphin whistle sequences
• Non-characteristic Half-lives in Radioactive Decay
• When is Menzerath-Altmann law mathematically trivial? A new approach
• Statistical Patterns in Written Language
• A practical recipe to fit discrete power-law distributions

Cited by in corpus (17)

• Large-scale analysis of Zipf's law in English texts
• Power-law size distributions in geoscience revisited
• Complex systems approach to natural language
• Optimization models of natural communication
• Polysemy and brevity versus frequency in language
• Log-log Convexity of Type-Token Growth in Zipf's Systems
• The brevity law as a scaling law, and a possible origin of Zipf's law for word frequencies
• Random Language Model
• Breakdown of random matrix universality in Markov models
• Lognormals, Power Laws and Double Power Laws in the Distribution of Frequencies of Harmonic Codewords from Classical Music
• The Mechanism of Additive Composition
• Heaps' Law and Vocabulary Richness in the History of Classical Music Harmony
• Word Embedding based New Corpus for Low-resourced Language: Sindhi
• Ordinal analysis of lexical patterns
• Empirical Analysis of Zipf's Law, Power Law, and Lognormal Distributions in Medical Discharge Reports
• A decomposition of book structure through ousiometric fluctuations in cumulative word-time
• Zipf's laws of meaning in Catalan