Random Language Model

Abstract

Many complex generative systems use languages to create structured objects. We consider a model of random languages, defined by weighted context-free grammars. As the distribution of grammar weights broadens, a transition is found from a random phase, in which sentences are indistinguishable from noise, to an organized phase in which nontrivial information is carried. This marks the emergence of deep structure in the language, and can be understood by a competition between energy and entropy.

5 pages + 3 pages SI

References in corpus (1)

• Zipf's law for word frequencies: word forms versus lemmas in long texts

Cited by in corpus (17)

• Social physics
• Multi-scale structural complexity of natural patterns
• Is stochastic thermodynamics the key to understanding the energy costs of computation?
• How Deep Neural Networks Learn Compositional Data: The Random Hierarchy Model
• Estimating Patterns of Classical and Quantum Skyrmion States
• Breakdown of random matrix universality in Markov models
• Emergence of order in random languages
• Classical spin Hamiltonians are context-sensitive languages
• Language Modeling with Reduced Densities
• Tensor Networks for Probabilistic Sequence Modeling
• Absence of Phase Transition in Random Language Model
• Schottky's forgotten step to the Ising model
• Phase transitions in a decentralized graph-based approach to human language
• The grammar of the Ising model: A new complexity hierarchy
• Statistical properties of probabilistic context-sensitive grammars
• Robustness of the Random Language Model
• Entangled criticality and irreversibility in random Markov dynamics