The distinct flavors of Zipf's law in the rank-size and in the size-distribution representations, and its maximum-likelihood fitting

Abstract

In the last years, researchers have realized the difficulties of fitting power-law distributions properly. These difficulties are higher in Zipf's systems, due to the discreteness of the variables and to the existence of two representations for these systems, i.e., two versions about which is the random variable to fit. The discreteness implies that a power law in one of the representations is not a power law in the other, and vice versa. We generate synthetic power laws in both representations and apply a state-of-the-art fitting method (based on maximum-likelihood plus a goodness-of-fit test) for each of the two random variables. It is important to stress that the method does not fit the whole distribution, but the tail, understood as the part of a distribution above a cut-off that separates non-power-law behavior from power-law behavior. We find that, no matter which random variable is power-law distributed, the rank-size representation is not adequate for fitting, whereas the representation in terms of the distribution of sizes leads to the recovery of the simulated exponents, may be with some bias.

References in corpus (29)

• Power-law distributions in empirical data
• Power laws, Pareto distributions and Zipf's law
• Crackling Noise
• Problems with Fitting to the Power-Law Distribution
• A Poissonian explanation for heavy-tails in e-mail communication
• Zipf's Law in Gene Expression
• Measuring the evolution of contemporary western popular music
• Languages cool as they expand: Allometric scaling and the decreasing need for new words
• Gibrat's law for cities: uniformly most powerful unbiased test of the Pareto against the lognormal
• Parameter estimation for power-law distributions by maximum likelihood methods
• Scale-free Networks Well Done
• Stochastic model for the vocabulary growth in natural languages
• Zipf's Law Leads to Heaps' Law: Analyzing Their Relation in Finite-Size Systems
• Large-scale analysis of Zipf's law in English texts
• Power-law size distributions in geoscience revisited
• Universality of rain event size distributions
• Fitting Power-laws in empirical data with estimators that work for all exponents
• Statistical laws in linguistics
• Toward a statistical mechanics of four letter words
• A scaling law beyond Zipf's law and its relation to Heaps' law
• Testing statistical laws in complex systems
• Scaling in the Timing of Extreme Events
• Re-inventing Willis
• The brevity law as a scaling law, and a possible origin of Zipf's law for word frequencies
• Non-characteristic Half-lives in Radioactive Decay
• An efficient algorithm for accelerating the convergence of oscillatory series, useful for computing the polylogarithm and Hurwitz zeta functions
• Truncated lognormal distributions and scaling in the size of naturally defined population clusters
• Phase transition, scaling of moments, and order-parameter distributions in Brownian particles and branching processes with finite-size effects
• A practical recipe to fit discrete power-law distributions

Cited by in corpus (8)

• Linguistic laws in biology
• The brevity law as a scaling law, and a possible origin of Zipf's law for word frequencies
• Maximal Diversity and Zipf's Law
• On the emergence of Zipf's law in music
• Lognormals, Power Laws and Double Power Laws in the Distribution of Frequencies of Harmonic Codewords from Classical Music
• Heaps' Law and Vocabulary Richness in the History of Classical Music Harmony
• From Boltzmann to Zipf through Shannon and Jaynes
• Finite-size scaling of human-population distributions over fixed-size cells and its relation to fractal spatial structure