Compressing Neural Networks Using Tensor Networks with Exponentially Fewer Variational Parameters

Abstract

Neural network (NN) designed for challenging machine learning tasks is in general a highly nonlinear mapping that contains massive variational parameters. High complexity of NN, if unbounded or unconstrained, might unpredictably cause severe issues including \R{overfitting}, loss of generalization power, and unbearable cost of hardware. In this work, we propose a general compression scheme that significantly reduces the variational parameters of NN's, despite of their specific types (linear, convolutional, \textit{etc}), by encoding them to deep \R{automatically differentiable} tensor network (ADTN) that contains exponentially-fewer free parameters. Superior compression performance of our scheme is demonstrated on several widely-recognized NN's (FC-2, LeNet-5, AlextNet, ZFNet and VGG-16) and datasets (MNIST, CIFAR-10 and CIFAR-100). For instance, we compress two linear layers in VGG-16 with approximately $10^7$ parameters to two ADTN's with just 424 parameters, improving the testing accuracy on CIFAR-10 from $90.17\%$ to $91.74\%$. We argue that the deep structure of ADTN is an essential reason for the remarkable compression performance of ADTN, compared to existing compression schemes that are mainly based on tensor decompositions/factorization and shallow tensor networks. Our work suggests deep TN as an exceptionally efficient mathematical structure for representing the variational parameters of NN's, which exhibits superior compressibility over the commonly-used matrices and multi-way arrays.

9 pages, 5 figures, 2 tables for the main text; 6 pages for the appendices

References in corpus (20)

• Deep Learning in Neural Networks: An Overview
• Knowledge Distillation: A Survey
• LoRA: Low-Rank Adaptation of Large Language Models
• Machine learning and the physical sciences
• Solving the Quantum Many-Body Problem with Artificial Neural Networks
• Machine learning phases of matter
• Entropy scaling and simulability by Matrix Product States
• Matrix product operator representations
• Efficient Representation of Quantum Many-body States with Deep Neural Networks
• Solving Statistical Mechanics Using Variational Autoregressive Networks
• Lecture Notes of Tensor Network Contractions
• Generic Construction of Efficient Matrix Product Operators
• Neural Network Renormalization Group
• Supervised Learning with Projected Entangled Pair States
• The area law and real-space renormalization
• Automatically Differentiable Quantum Circuit for Many-qubit State Preparation
• Tensor networks for interpretable and efficient quantum-inspired machine learning
• A Geometric View of Optimal Transportation and Generative Model
• A Model Compression Method with Matrix Product Operators for Speech Enhancement
• Deep learning Local Reduced Density Matrices for Many-body Hamiltonian Estimation

Cited by in corpus (1)

• Neuralized Fermionic Tensor Networks for Quantum Many-Body Systems