Tensorization of neural networks for improved privacy and interpretability

Abstract

We present a tensorization algorithm for constructing tensor train/matrix product state (MPS) representations of functions, drawing on sketching and cross interpolation ideas. The method only requires black-box access to the target function and a small set of sample points defining the domain of interest. Thus, it is particularly well-suited for machine learning models, where the domain of interest is naturally defined by the training dataset. We show that this approach can be used to enhance the privacy and interpretability of neural network models. Specifically, we apply our decomposition to (i) obfuscate neural networks whose parameters encode patterns tied to the training data distribution, and (ii) estimate topological phases of matter that are easily accessible from the MPS representation. Additionally, we show that this tensorization can serve as an efficient initialization method for optimizing MPS in general settings, and that, for model compression, our algorithm achieves a superior trade-off between memory and time complexity compared to conventional tensorization methods of neural networks.

44 pages, 9 figures, 3 tables. The code for the experiments is publicly available at https://github.com/joserapa98/tensorization-nns. V3: Published version

References in corpus (88)

• Deep Learning with Differential Privacy
• Deep Residual Learning for Image Recognition
• Training language models to follow instructions with human feedback
• LLaMA: Open and Efficient Foundation Language Models
• The density-matrix renormalization group
• LoRA: Low-Rank Adaptation of Large Language Models
• Efficient classical simulation of slightly entangled quantum computations
• Solving the Quantum Many-Body Problem with Artificial Neural Networks
• PaLM: Scaling Language Modeling with Pathways
• A Practical Introduction to Tensor Networks: Matrix Product States and Projected Entangled Pair States
• Machine learning phases of matter
• Entanglement spectrum of a topological phase in one dimension
• Symmetry protected topological orders and the group cohomology of their symmetry group
• Predicting Many Properties of a Quantum System from Very Few Measurements
• Zero-Shot Text-to-Image Generation
• Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
• Classification of Gapped Symmetric Phases in 1D Spin Systems
• Symmetry protection of topological order in one-dimensional quantum spin systems
• Matrix Product States and Projected Entangled Pair States: Concepts, Symmetries, and Theorems
• Classifying quantum phases using Matrix Product States and PEPS
• Learning phase transitions by confusion
• High-Resolution Image Synthesis with Latent Diffusion Models
• A Convergence Theory for Deep Learning via Over-Parameterization
• Tensorizing Neural Networks
• Classical simulation of quantum many-body systems with a tree tensor network
• This Looks Like That: Deep Learning for Interpretable Image Recognition
• Sketching as a Tool for Numerical Linear Algebra
• Hand-waving and Interpretive Dance: An Introductory Course on Tensor Networks
• Efficient Representation of Quantum Many-body States with Deep Neural Networks
• Restricted-Boltzmann-Machine Learning for Solving Strongly Correlated Quantum Systems
• Detection of Symmetry Protected Topological Phases in 1D
• Reconstructing quantum states with generative models
• The computational complexity of PEPS
• Simulation of two-dimensional quantum systems using a tree tensor network that exploits the entropic area law
• String order and symmetries in quantum spin lattices
• Recurrent Neural Network Wave Functions
• Simulating Strongly Correlated Quantum Systems with Tree Tensor Networks
• Local Differential Privacy for Deep Learning
• How to generate random matrices from the classical compact groups
• Solving the sampling problem of the Sycamore quantum circuits
• Efficient tensor network simulation of IBM's Eagle kicked Ising experiment
• Tree Tensor Networks for Generative Modeling
• Machine Learning by Unitary Tensor Network of Hierarchical Tree Structure
• Extracting Training Data from Diffusion Models
• Matrix Product States: Symmetries and Two-Body Hamiltonians
• Supervised Learning with Quantum-Inspired Tensor Networks
• An order parameter for symmetry-protected phases in one dimension
• Scalable Extraction of Training Data from (Production) Language Models
• Efficient quantum state tomography with convolutional neural networks
• Quasioptimality of maximum-volume cross interpolation of tensors
• Parallel cross interpolation for high-precision calculation of high-dimensional integrals
• Auditing Differentially Private Machine Learning: How Private is Private SGD?
• Quantum-inspired algorithms for multivariate analysis: from interpolation to partial differential equations
• Neural-network quantum state tomography
• Quantics Tensor Cross Interpolation for High-Resolution, Parsimonious Representations of Multivariate Functions in Physics and Beyond
• Contracting projected entangled pair states is average-case hard
• Classical algorithm for simulating experimental Gaussian boson sampling
• Learning tensor networks with tensor cross interpolation: new algorithms and libraries
• Investigating Topological Order using Recurrent Neural Networks
• A Survey on Neural Network Interpretability
• Compressing deep neural networks by matrix product operators
• Matrix Product State Representations
• Anomaly Detection with Tensor Networks
• Tensor Random Projection for Low Memory Dimension Reduction
• Defence against adversarial attacks using classical and quantum-enhanced Boltzmann machines
• tntorch: Tensor Network Learning with PyTorch
• Generative modeling with projected entangled-pair states
• Tensor Networks for Explainable Machine Learning in Cybersecurity
• Explainable Natural Language Processing with Matrix Product States
• Privacy-preserving Learning via Deep Net Pruning
• Privacy-preserving machine learning with tensor networks
• Classification of symmetry protected topological phases in quantum spin chains
• Towards Compact Neural Networks via End-to-End Training: A Bayesian Tensor Approach with Automatic Rank Determination
• Quantum Large Language Models via Tensor Network Disentanglers
• A Matrix Product State Model for Simultaneous Classification and Generation
• TensorKrowch: Smooth integration of tensor networks in machine learning
• High-dimensional density estimation with tensorizing flow
• RydbergGPT
• Tensor network compressibility of convolutional models
• Parallel algorithms for computing the tensor-train decomposition
• Compressing multivariate functions with tree tensor networks
• Improvements to Gradient Descent Methods for Quantum Tensor Network Machine Learning
• Differential Privacy Meets Neural Network Pruning
• Qubit seriation: Improving data-model alignment using spectral ordering
• Efficient Finite Initialization with Partial Norms for Tensorized Neural Networks and Tensor Networks Algorithms
• Generative modeling via tensor train sketching
• Efficient GPT Model Pre-training using Tensor Train Matrix Representation
• Generative Modeling via Tree Tensor Network States