Randomness versus specifics for word-frequency distributions
arXiv:1404.1461 · doi:10.1016/j.physa.2015.10.082
Abstract
The text-length-dependence of real word-frequency distributions can be connected to the general properties of a random book. It is pointed out that this finding has strong implications, when deciding between two conceptually different views on word-frequency distributions, i.e. the specific `Zipf's-view' and the non-specific `Randomness-view', as is discussed. It is also noticed that the text-length transformation of a random book does have an exact scaling property precisely for the power-law index , as opposed to the Zipf's exponent and the implication of this exact scaling property is discussed. However a real text has and as a consequence increases when shortening a real text. The connections to the predictions from the RGF(Random Group Formation) and to the infinite length-limit of a meta-book are also discussed. The difference between `curve-fitting' and `predicting' word-frequency distributions is stressed. It is pointed out that the question of randomness versus specifics for the distribution of outcomes in case of sufficiently complex systems has a much wider relevance than just the word-frequency example analyzed in the present work.
9 pages, 7 figures
References in corpus (8)
- Languages cool as they expand: Allometric scaling and the decreasing need for new words
- Diversity of individual mobility patterns and emergence of aggregated scaling laws
- Zipf's Law Leads to Heaps' Law: Analyzing Their Relation in Finite-Size Systems
- Network properties of written human language
- A Yule-Simon process with memory
- A scaling law beyond Zipf's law and its relation to Heaps' law
- Size dependent word frequencies and translational invariance of books
- Maximum Entropy, Word-Frequency, Chinese Characters, and Multiple Meanings