Language Design as Information Renormalization
arXiv:1708.01525 · doi:10.1007/s42979-021-01002-y
Abstract
Here we consider some well-known facts in syntax from a physics perspective, allowing us to establish equivalences between both fields with many consequences. Mainly, we observe that the operation MERGE, put forward by N. Chomsky in 1995, can be interpreted as a physical information coarse-graining. Thus, MERGE in linguistics entails information renormalization in physics, according to different time scales. We make this point mathematically formal in terms of language models. In this setting, MERGE amounts to a probability tensor implementing a coarse-graining, akin to a probabilistic context-free grammar. The probability vectors of meaningful sentences are given by stochastic tensor networks (TN) built from diagonal tensors and which are mostly loop-free, such as Tree Tensor Networks and Matrix Product States, thus being computationally very efficient to manipulate. We show that this implies the polynomially-decaying (long-range) correlations experimentally observed in language, and also provides arguments in favour of certain types of neural networks for language processing. Moreover, we show how to obtain such language models from quantum states that can be efficiently prepared on a quantum computer, and use this to find bounds on the perplexity of the probability distribution of words in a sentence. Implications of our results are discussed across several ambits.
arXiv comment25 pages, 21 figures, 1 table, Final Version
References in corpus (26)
- The density-matrix renormalization group in the age of matrix product states
- Entanglement in quantum critical phenomena
- A Practical Introduction to Tensor Networks: Matrix Product States and Projected Entangled Pair States
- Matrix Product States, Projected Entangled Pair States, and variational renormalization group methods for quantum spin systems
- Geometric and Renormalized Entropy in Conformal Field Theory
- Entanglement renormalization
- Area laws in quantum systems: mutual information and correlations
- Classical simulation of quantum many-body systems with a tree tensor network
- Why does deep and cheap learning work so well?
- Renormalization and tensor product states in spin chains and lattices
- The computational complexity of PEPS
- Matrix-product-groundstates for one-dimensional spin-1 quantum antiferromagnets
- More is the Same; Phase Transitions and Mean Field Theories
- Simulation of two-dimensional quantum systems using a tree tensor network that exploits the entropic area law
- Simulating Strongly Correlated Quantum Systems with Tree Tensor Networks
- Renormalization group transformations on quantum states
- Advances on Tensor Network Theory: Symmetries, Fermions, Entanglement, and Holography
- On the origin of long-range correlations in texts
- Fine-grained entanglement loss along renormalization group flows
- Density Matrix Renormalization Group and Reaction-Diffusion Processes
- Single-copy entanglement in critical spin chains
- Purifications of multipartite states: limitations and constructive methods
- Unconstrained Tree Tensor Network: An adaptive gauge picture for enhanced performance
- Half the entanglement in critical systems is distillable from a single specimen
- Entanglement and majorization in (1+1)-dimensional quantum systems
- Stochastic Matrix Product States
Cited by in corpus (9)
- QNLP in Practice: Running Compositional Models of Meaning on a Quantum Computer
- Developments in the Tensor Network -- from Statistical Mechanics to Quantum Entanglement
- Hyper-optimized approximate contraction of tensor networks with arbitrary geometry
- lambeq: An Efficient High-Level Python Library for Quantum NLP
- Language Modeling with Reduced Densities
- Mathematical foundations of matrix syntax
- Tensor Networks for Probabilistic Sequence Modeling
- Tensor networks and efficient descriptions of classical data
- Representation Theorem for Matrix Product States