Probability-turbulence divergence: A tunable allotaxonometric instrument for comparing heavy-tailed categorical distributions

Abstract

Real-world complex systems often comprise many distinct types of elements as well as many more types of networked interactions between elements. When the relative abundances of types can be measured well, we often observe heavy-tailed categorical distributions for type frequencies. For the comparison of type frequency distributions of two systems or a system with itself at different time points in time -- a facet of allotaxonometry -- a great range of probability divergences are available. Here, we introduce and explore `probability-turbulence divergence', a tunable, straightforward, and interpretable instrument for comparing normalizable categorical frequency distributions. We model probability-turbulence divergence (PTD) after rank-turbulence divergence (RTD). While probability-turbulence divergence is more limited in application than rank-turbulence divergence, it is more sensitive to changes in type frequency. We build allotaxonographs to display probability turbulence, incorporating a way to visually accommodate zero probabilities for `exclusive types' which are types that appear in only one system. We explore comparisons of example distributions taken from literature, social media, and ecology. We show how probability-turbulence divergence either explicitly or functionally generalizes many existing kinds of distances and measures, including, as special cases, $L^{(p)}$ norms, the Sørensen-Dice coefficient (the $F_1$ statistic), and the Hellinger distance. We discuss similarities with the generalized entropies of R{é}nyi and Tsallis, and the diversity indices (or Hill numbers) from ecology. We close with thoughts on open problems concerning the optimization of the tuning of rank- and probability-turbulence divergence.

24 pages, 9 figures (8 in manuscript, 1 in frontispiece), 3 tables

References in corpus (9)

• Emergence of scaling in random networks
• Power-law distributions in empirical data
• Power laws, Pareto distributions and Zipf's law
• Robust estimation of microbial diversity in theory and in practice
• The growing amplification of social media: Measuring temporal and social contagion dynamics for over 150 languages on Twitter for 2009-2020
• Storywrangler: A massive exploratorium for sociolinguistic, cultural, socioeconomic, and political timelines using Twitter
• Text mixing shapes the anatomy of rank-frequency distributions: A modern Zipfian mechanics for natural language
• Computational timeline reconstruction of the stories surrounding Trump: Story turbulence, narrative control, and collective chronopathy
• Why scatter plots suggest causality, and what we can do about it

Cited by in corpus (1)

• Long-term word frequency dynamics derived from Twitter are corrupted: A bespoke approach to detecting and removing pathologies in ensembles of time series