Rank dynamics of word usage at multiple scales

Abstract

The recent dramatic increase in online data availability has allowed researchers to explore human culture with unprecedented detail, such as the growth and diversification of language. In particular, it provides statistical tools to explore whether word use is similar across languages, and if so, whether these generic features appear at different scales of language structure. Here we use the Google Books $N$-grams dataset to analyze the temporal evolution of word usage in several languages. We apply measures proposed recently to study rank dynamics, such as the diversity of $N$-grams in a given rank, the probability that an $N$-gram changes rank between successive time intervals, the rank entropy, and the rank complexity. Using different methods, results show that there are generic properties for different languages at different scales, such as a core of words necessary to minimally understand a language. We also propose a null model to explore the relevance of linguistic structure across multiple scales, concluding that $N$-gram statistics cannot be reduced to word statistics. We expect our results to be useful in improving text prediction algorithms, as well as in shedding light on the large-scale features of language use, beyond linguistic and cultural differences across human populations.

arXiv comment19 pages (main text) + 24 pages (supplementary information)

References in corpus (19)

• Power laws, Pareto distributions and Zipf's law
• Temporal patterns of happiness and information in a global social network: Hedonometrics and Twitter
• Sharp transition towards shared vocabularies in multi-agent systems
• Measuring the evolution of contemporary western popular music
• Languages cool as they expand: Allometric scaling and the decreasing need for new words
• Stochastic model for the vocabulary growth in natural languages
• Complexity and Information: Measuring Emergence, Self-organization, and Homeostasis at Multiple Scales
• Statistical Laws Governing Fluctuations in Word Use from Word Birth to Word Death
• Zipf's law unzipped
• Evolution of the most common English words and phrases over the centuries
• A complex network approach to stylometry
• Self-organization of progress across the century of physics
• Emergence of Zipf's Law in the Evolution of Communication
• Similarity of symbol frequency distributions with heavy tails
• Rank diversity of languages: Generic behavior in computational linguistics
• Extracting information from S-curves of language change
• Complexity measurement of natural and artificial languages
• Universal temporal features of rankings in competitive sports and games
• Measuring the Complexity of Continuous Distributions

Cited by in corpus (9)

• Dynamics of ranking
• Attention dynamics on the Chinese social media Sina Weibo during the COVID-19 pandemic
• Guiding the Self-organization of Cyber-Physical Systems
• Elites, communities and the limited benefits of mentorship in electronic music
• Emergence in artificial life
• Competition for popularity and interventions on a Chinese microblogging site
• Statistical Properties of Rankings in Sports and Games
• Language statistics at different spatial, temporal, and grammatical scales
• A Statistical Model of Word Rank Evolution