On Hilberg's Law and Its Links with Guiraud's Law

Abstract

Hilberg (1990) supposed that finite-order excess entropy of a random human text is proportional to the square root of the text length. Assuming that Hilberg's hypothesis is true, we derive Guiraud's law, which states that the number of word types in a text is greater than proportional to the square root of the text length. Our derivation is based on some mathematical conjecture in coding theory and on several experiments suggesting that words can be defined approximately as the nonterminals of the shortest context-free grammar for the text. Such operational definition of words can be applied even to texts deprived of spaces, which do not allow for Mandelbrot's ``intermittent silence'' explanation of Zipf's and Guiraud's laws. In contrast to Mandelbrot's, our model assumes some probabilistic long-memory effects in human narration and might be capable of explaining Menzerath's law.

To appear in Journal of Quantitative Linguistics

References in corpus (2)

• Entropy and Long range correlations in literary English
• Data Compression and Entropy Estimates by Non-sequential Recursive Pair Substitution

Cited by in corpus (6)

• Statistical laws in linguistics
• On the Vocabulary of Grammar-Based Codes and the Logical Consistency of Texts
• Is Natural Language a Perigraphic Process? The Theorem about Facts and Words Revisited
• On vocabulary size of grammar-based codes
• A Preadapted Universal Switch Distribution for Testing Hilberg's Conjecture
• Bounds for Algorithmic Mutual Information and a Unifilar Order Estimator