Language Modeling with Reduced Densities
arXiv:2007.03834 · doi:10.32408/compositionality-3-4
Abstract
This work originates from the observation that today's state-of-the-art statistical language models are impressive not only for their performance, but also - and quite crucially - because they are built entirely from correlations in unstructured text data. The latter observation prompts a fundamental question that lies at the heart of this paper: What mathematical structure exists in unstructured text data? We put forth enriched category theory as a natural answer. We show that sequences of symbols from a finite alphabet, such as those found in a corpus of text, form a category enriched over probabilities. We then address a second fundamental question: How can this information be stored and modeled in a way that preserves the categorical structure? We answer this by constructing a functor from our enriched category of text to a particular enriched category of reduced density operators. The latter leverages the Loewner order on positive semidefinite operators, which can further be interpreted as a toy example of entailment.
arXiv comment21 pages; v2: added reference; v3: revised abstract and introduction for clarity; v4: Compositionality version
References in corpus (28)
- The density-matrix renormalization group in the age of matrix product states
- A Practical Introduction to Tensor Networks: Matrix Product States and Projected Entangled Pair States
- Scaling Laws for Neural Language Models
- Tensor networks for complex quantum systems
- Hand-waving and Interpretive Dance: An Introductory Course on Tensor Networks
- MADE: Masked Autoencoder for Distribution Estimation
- Tensor network states and geometry
- On the Expressive Power of Deep Learning: A Tensor Analysis
- Tensor Networks in a Nutshell
- Supervised Learning with Quantum-Inspired Tensor Networks
- Mathematical Foundations for a Compositional Distributional Model of Meaning
- Quantum Language Model with Entanglement Embedding for Question Answering
- Matrix Product Operators for Sequence to Sequence Learning
- Sentence Entailment in Compositional Distributional Semantics
- A Quantum Many-body Wave Function Inspired Language Modeling Approach
- Random Language Model
- Probabilistic Modeling with Matrix Product States
- Lectures on Quantum Tensor Networks
- Modeling Sequences with Quantum States: A Look Under the Hood
- Tight spans, Isbell completions and semi-tropical modules
- Anomaly Detection with Tensor Networks
- Language Design as Information Renormalization
- Tensor network language model
- Entanglement and Tensor Networks for Supervised Image Classification
- Language as a matrix product state
- A Multi-Scale Tensor Network Architecture for Classification and Regression
- Tensor Networks for Probabilistic Sequence Modeling
- At the Interface of Algebra and Statistics