Compressing deep neural networks by matrix product operators

Abstract

A deep neural network is a parametrization of a multilayer mapping of signals in terms of many alternatively arranged linear and nonlinear transformations. The linear transformations, which are generally used in the fully connected as well as convolutional layers, contain most of the variational parameters that are trained and stored. Compressing a deep neural network to reduce its number of variational parameters but not its prediction power is an important but challenging problem toward the establishment of an optimized scheme in training efficiently these parameters and in lowering the risk of overfitting. Here we show that this problem can be effectively solved by representing linear transformations with matrix product operators (MPOs), which is a tensor network originally proposed in physics to characterize the short-range entanglement in one-dimensional quantum states. We have tested this approach in five typical neural networks, including FC2, LeNet-5, VGG, ResNet, and DenseNet on two widely used data sets, namely, MNIST and CIFAR-10, and found that this MPO representation indeed sets up a faithful and efficient mapping between input and output signals, which can keep or even improve the prediction accuracy with a dramatically reduced number of parameters. Our method greatly simplifies the representations in deep learning, and opens a possible route toward establishing a framework of modern neural networks which might be simpler and cheaper, but more efficient.

arXiv comment8+9 pages, 3+7 figures, 2+11 tables

References in corpus (19)

• Deep Learning in Neural Networks: An Overview
• Area laws for the entanglement entropy - a review
• Machine learning and the physical sciences
• Efficient classical simulation of slightly entangled quantum computations
• Machine learning phases of matter
• Searching for Exotic Particles in High-Energy Physics with Deep Learning
• Matrix Product Density Operators: Simulation of finite-T and dissipative systems
• Entanglement renormalization
• Classical simulation of quantum many-body systems with a tree tensor network
• Machine learning modeling of superconducting critical temperature
• Matrix product operator representations
• Efficient Representation of Quantum Many-body States with Deep Neural Networks
• Quantum simulation of time-dependent Hamiltonians and the convenient illusion of Hilbert space
• Unsupervised Generative Modeling Using Matrix Product States
• Solving Statistical Mechanics Using Variational Autoregressive Networks
• Neural Network Renormalization Group
• Machine Learning by Unitary Tensor Network of Hierarchical Tree Structure
• Information Perspective to Probabilistic Modeling: Boltzmann Machines versus Born Machines
• Matrix Product Operators for Sequence to Sequence Learning

Cited by in corpus (11)

• Contracting Arbitrary Tensor Networks: General Approximate Algorithm and Applications in Graphical Models and Quantum Circuit Simulations
• Accurate simulation for finite projected entangled pair states in two dimensions
• Magnetization of the spin-1/2 Heisenberg antiferromagnet on the triangular lattice
• A Model Compression Method with Matrix Product Operators for Speech Enhancement
• Efficient calculation of three-dimensional tensor networks
• Residual Matrix Product State for Machine Learning
• Tensor networks and efficient descriptions of classical data
• Exploring explicit coarse-grained structure in artificial neural networks
• Long-range spin-orbital order in the spin-orbital SU(2)$\times$SU(2)$\times$U(1) model
• Permutation of Tensor-Train Cores for Computing Moments on Stochastic Differential Equations
• Tensorization of neural networks for improved privacy and interpretability