Size dependent word frequencies and translational invariance of books
arXiv:0906.0716 · doi:10.1016/j.physa.2009.09.022
Abstract
It is shown that a real novel shares many characteristic features with a null model in which the words are randomly distributed throughout the text. Such a common feature is a certain translational invariance of the text. Another is that the functional form of the word-frequency distribution of a novel depends on the length of the text in the same way as the null model. This means that an approximate power-law tail ascribed to the data will have an exponent which changes with the size of the text-section which is analyzed. A further consequence is that a novel cannot be described by text-evolution models like the Simon model. The size-transformation of a novel is found to be well described by a specific Random Book Transformation. This size transformation in addition enables a more precise determination of the functional form of the word-frequency distribution. The implications of the results are discussed.
arXiv comment10 pages, 2 appendices (6 pages), 5 figures
References in corpus (5)
Cited by in corpus (13)
- Languages cool as they expand: Allometric scaling and the decreasing need for new words
- Zipf's law unzipped
- Statistical laws in linguistics
- The meta book and size-dependent properties of written language
- Rank-frequency relation for Chinese characters
- Heaps' law, statistics of shared components and temporal patterns from a sample-space-reducing process
- The Ten Thousand Kims
- A Paradoxical Property of the Monkey Book
- Modular networks of word correlations on Twitter
- Maximum Entropy, Word-Frequency, Chinese Characters, and Multiple Meanings
- Randomness versus specifics for word-frequency distributions
- Two halves of a meaningful text are statistically different
- The Dependence of Frequency Distributions on Multiple Meanings of Words, Codes and Signs