Explainable Natural Language Processing with Matrix Product States

Abstract

Despite empirical successes of recurrent neural networks (RNNs) in natural language processing (NLP), theoretical understanding of RNNs is still limited due to intrinsically complex non-linear computations. We systematically analyze RNNs' behaviors in a ubiquitous NLP task, the sentiment analysis of movie reviews, via the mapping between a class of RNNs called recurrent arithmetic circuits (RACs) and a matrix product state (MPS). Using the von-Neumann entanglement entropy (EE) as a proxy for information propagation, we show that single-layer RACs possess a maximum information propagation capacity, reflected by the saturation of the EE. Enlarging the bond dimension beyond the EE saturation threshold does not increase model prediction accuracies, so a minimal model that best estimates the data statistics can be inferred. Although the saturated EE is smaller than the maximum EE allowed by the area law, our minimal model still achieves ~99% training accuracies in realistic sentiment analysis data sets. Thus, low EE is not a warrant against the adoption of single-layer RACs for NLP. Contrary to a common belief that long-range information propagation is the main source of RNNs' successes, we show that single-layer RACs harness high expressiveness from the subtle interplay between the information propagation and the word vector embeddings. Our work sheds light on the phenomenology of learning in RACs, and more generally on the explainability of RNNs for NLP, using tools from many-body quantum physics.

arXiv comment25 pages, 7 figures. Accepted for publication in New Journal of Physics

References in corpus (20)

• Area laws for the entanglement entropy - a review
• Solving the Quantum Many-Body Problem with Artificial Neural Networks
• Machine learning phases of matter
• Simplifying Graph Convolutional Networks
• Entanglement renormalization
• A high-bias, low-variance introduction to Machine Learning for physicists
• Matrix product states represent ground states faithfully
• Area laws in quantum systems: mutual information and correlations
• Tensor networks for complex quantum systems
• Renormalization algorithms for Quantum-Many Body Systems in two and higher dimensions
• Quantum Entanglement in Deep Learning Architectures
• Entailment as Few-Shot Learner
• Machine learning quantum states in the NISQ era
• Machine Learning by Unitary Tensor Network of Hierarchical Tree Structure
• Learning Relevant Features of Data with Multi-scale Tensor Networks
• Revisiting LSTM Networks for Semi-Supervised Text Classification via Mixed Objective Function
• Matrix Product Operators for Sequence to Sequence Learning
• Graph Star Net for Generalized Multi-Task Learning
• Probabilistic Modeling with Matrix Product States
• Modeling Sequences with Quantum States: A Look Under the Hood

Cited by in corpus (3)

• TensorKrowch: Smooth integration of tensor networks in machine learning
• Tensorization of neural networks for improved privacy and interpretability
• Group-invariant tensor train networks for supervised learning