Compression and the origins of Zipf's law of abbreviation
arXiv:1504.04884
Abstract
Languages across the world exhibit Zipf's law of abbreviation, namely more frequent words tend to be shorter. The generalized version of the law - an inverse relationship between the frequency of a unit and its magnitude - holds also for the behaviours of other species and the genetic code. The apparent universality of this pattern in human language and its ubiquity in other domains calls for a theoretical understanding of its origins. To this end, we generalize the information theoretic concept of mean code length as a mean energetic cost function over the probability and the magnitude of the types of the repertoire. We show that the minimization of that cost function and a negative correlation between probability and the magnitude of types are intimately related.
arXiv commentNew results for optimal non-singular coding have been added; some sections have been reorganized
References in corpus (9)
- Community detection in graphs
- Power laws, Pareto distributions and Zipf's law
- Sharp transition towards shared vocabularies in multi-agent systems
- Stochastic model for the vocabulary growth in natural languages
- Statistical laws in linguistics
- The optimality of attaching unlinked labels to unlinked meanings
- The failure of the law of brevity in two New World primates. Statistical caveats
- Rank diversity of languages: Generic behavior in computational linguistics
- A stronger null hypothesis for crossing dependencies