Language Design as Information Renormalization

Abstract

Here we consider some well-known facts in syntax from a physics perspective, allowing us to establish equivalences between both fields with many consequences. Mainly, we observe that the operation MERGE, put forward by N. Chomsky in 1995, can be interpreted as a physical information coarse-graining. Thus, MERGE in linguistics entails information renormalization in physics, according to different time scales. We make this point mathematically formal in terms of language models. In this setting, MERGE amounts to a probability tensor implementing a coarse-graining, akin to a probabilistic context-free grammar. The probability vectors of meaningful sentences are given by stochastic tensor networks (TN) built from diagonal tensors and which are mostly loop-free, such as Tree Tensor Networks and Matrix Product States, thus being computationally very efficient to manipulate. We show that this implies the polynomially-decaying (long-range) correlations experimentally observed in language, and also provides arguments in favour of certain types of neural networks for language processing. Moreover, we show how to obtain such language models from quantum states that can be efficiently prepared on a quantum computer, and use this to find bounds on the perplexity of the probability distribution of words in a sentence. Implications of our results are discussed across several ambits.

25 pages, 21 figures, 1 table, Final Version

References in corpus (9)

• The density-matrix renormalization group in the age of matrix product states
• Matrix Product States, Projected Entangled Pair States, and variational renormalization group methods for quantum spin systems
• Area laws in quantum systems: mutual information and correlations
• Why does deep and cheap learning work so well?
• More is the Same; Phase Transitions and Mean Field Theories
• Simulating Strongly Correlated Quantum Systems with Tree Tensor Networks
• Advances on Tensor Network Theory: Symmetries, Fermions, Entanglement, and Holography
• On the origin of long-range correlations in texts
• Unconstrained Tree Tensor Network: An adaptive gauge picture for enhanced performance

Cited by in corpus (9)

• QNLP in Practice: Running Compositional Models of Meaning on a Quantum Computer
• Developments in the Tensor Network -- from Statistical Mechanics to Quantum Entanglement
• Hyper-optimized approximate contraction of tensor networks with arbitrary geometry
• lambeq: An Efficient High-Level Python Library for Quantum NLP
• Language Modeling with Reduced Densities
• Mathematical foundations of matrix syntax
• Tensor Networks for Probabilistic Sequence Modeling
• Tensor networks and efficient descriptions of classical data
• Representation Theorem for Matrix Product States