Two halves of a meaningful text are statistically different
arXiv:2004.06474 · doi:10.1088/1742-5468/abe947
Abstract
Which statistical features distinguish a meaningful text (possibly written in an unknown system) from a meaningless set of symbols? Here we answer this question by comparing features of the first half of a text to its second half. This comparison can uncover hidden effects, because the halves have the same values of many parameters (style, genre {\it etc}). We found that the first half has more different words and more rare words than the second half. Also, words in the first half are distributed less homogeneously over the text in the sense of of the difference between the frequency and the inverse spatial period. These differences hold for the significant majority of several hundred relatively short texts we studied. The statistical significance is confirmed via the Wilcoxon test. Differences disappear after random permutation of words that destroys the linear structure of the text. The differences reveal a temporal asymmetry in meaningful texts, which is confirmed by showing that texts are much better compressible in their natural way (i.e. along the narrative) than in the word-inverted form. We conjecture that these results connect the semantic organization of a text (defined by the flow of its narrative) to its statistical features.
arXiv comment15 pages and 14 tables
References in corpus (18)
- The emotional arcs of stories are dominated by six basic shapes
- Languages cool as they expand: Allometric scaling and the decreasing need for new words
- On the origin of long-range correlations in texts
- Zipf's law unzipped
- Zipf's Law and Avoidance of Excessive Synonymy
- Statistical keyword detection in literary corpora
- Towards the quantification of the semantic information encoded in written language
- Statistical Criticality arises in Most Informative Representations
- Mapping Out Narrative Structures and Dynamics Using Networks and Textual Information
- A short account of a connection of Power Laws to the Information Entropy
- On the non-randomness of maximum Lempel Ziv complexity sequences of finite size
- Excess entropy in natural language: present state and perspectives
- Rank-frequency relation for Chinese characters
- Size dependent word frequencies and translational invariance of books
- Explaining Zipf's Law via Mental Lexicon
- Maximum entropy approach to power-law distributions in coupled dynamic-stochastic systems
- A Paradoxical Property of the Monkey Book
- Non-random structures in universal compression and the Fermi paradox