Generalized Entropies and the Similarity of Texts
arXiv:1611.03596 · doi:10.1088/1742-5468/aa53f5
Abstract
We show how generalized Gibbs-Shannon entropies can provide new insights on the statistical properties of texts. The universal distribution of word frequencies (Zipf's law) implies that the generalized entropies, computed at the word level, are dominated by words in a specific range of frequencies. Here we show that this is the case not only for the generalized entropies but also for the generalized (Jensen-Shannon) divergences, used to compute the similarity between different texts. This finding allows us to identify the contribution of specific words (and word frequencies) for the different generalized entropies and also to estimate the size of the databases needed to obtain a reliable estimation of the divergences. We test our results in large databases of books (from the Google n-gram database) and scientific papers (indexed by Web of Science).
arXiv comment13 pages, 6 figures; Results presented at the StatPhys-2016 meeting in Lyon
References in corpus (9)
- Temporal patterns of happiness and information in a global social network: Hedonometrics and Twitter
- Properties of Classical and Quantum Jensen-Shannon Divergence
- Languages cool as they expand: Allometric scaling and the decreasing need for new words
- Stochastic model for the vocabulary growth in natural languages
- Large-scale analysis of Zipf's law in English texts
- Statistical laws in linguistics
- Scaling laws and fluctuations in the statistics of word frequencies
- Wikipedia information flow analysis reveals the scale-free architecture of the Semantic Space
- Text mixing shapes the anatomy of rank-frequency distributions: A modern Zipfian mechanics for natural language
Cited by in corpus (8)
- Generalized Word Shift Graphs: A Method for Visualizing and Explaining Pairwise Comparisons Between Texts
- Using text analysis to quantify the similarity and evolution of scientific disciplines
- Multilayer Networks for Text Analysis with Multiple Data Types
- Pull out all the stops: Textual analysis via punctuation sequences
- Entropy and type-token ratio in gigaword corpora
- Quantifying the Dissimilarity of Texts
- Characterising heavy-tailed networks using q-generalised entropy and q-adjacency kernels
- Phase transitions in a decentralized graph-based approach to human language