Tensor Networks for Probabilistic Sequence Modeling
arXiv:2003.01039
Abstract
Tensor networks are a powerful modeling framework developed for computational many-body physics, which have only recently been applied within machine learning. In this work we utilize a uniform matrix product state (u-MPS) model for probabilistic modeling of sequence data. We first show that u-MPS enable sequence-level parallelism, with length-n sequences able to be evaluated in depth O(log n). We then introduce a novel generative algorithm giving trained u-MPS the ability to efficiently sample from a wide variety of conditional distributions, each one defined by a regular expression. Special cases of this algorithm correspond to autoregressive and fill-in-the-blank sampling, but more complex regular expressions permit the generation of richly structured data in a manner that has no direct analogue in neural generative models. Experiments on sequence modeling with synthetic and real text data show u-MPS outperforming a variety of baselines and effectively generalizing their predictions in the presence of limited data.
arXiv comment18 pages, 2 figures; v4 conference version; v3 link to code for experiments; v2 major revision with new main result on regular expression sampling. International Conference on Artificial Intelligence and Statistics. PMLR, 2021
References in corpus (15)
- Convolutional Sequence to Sequence Learning
- Tensorizing Neural Networks
- Unsupervised Generative Modeling Using Matrix Product States
- Differentiable Programming Tensor Networks
- Perfect Sampling with Unitary Tensor Networks
- Tree Tensor Networks for Generative Modeling
- Hidden Quantum Markov Models and non-adaptive read-out of many-body states
- Random Language Model
- Probabilistic Modeling with Matrix Product States
- Connecting Weighted Automata and Recurrent Neural Networks through Spectral Learning
- Parallelizing Linear Recurrent Neural Nets Over Sequence Length
- Language Design as Information Renormalization
- Tensor network language model
- Singular value automata and approximate minimization
- Language as a matrix product state