Hybrid Tensor Decomposition in Neural Network Compression

Abstract

Deep neural networks (DNNs) have enabled impressive breakthroughs in various artificial intelligence (AI) applications recently due to its capability of learning high-level features from big data. However, the current demand of DNNs for computational resources especially the storage consumption is growing due to that the increasing sizes of models are being required for more and more complicated applications. To address this problem, several tensor decomposition methods including tensor-train (TT) and tensor-ring (TR) have been applied to compress DNNs and shown considerable compression effectiveness. In this work, we introduce the hierarchical Tucker (HT), a classical but rarely-used tensor decomposition method, to investigate its capability in neural network compression. We convert the weight matrices and convolutional kernels to both HT and TT formats for comparative study, since the latter is the most widely used decomposition method and the variant of HT. We further theoretically and experimentally discover that the HT format has better performance on compressing weight matrices, while the TT format is more suited for compressing convolutional kernels. Based on this phenomenon we propose a strategy of hybrid tensor decomposition by combining TT and HT together to compress convolutional and fully connected parts separately and attain better accuracy than only using the TT or HT format on convolutional neural networks (CNNs). Our work illuminates the prospects of hybrid tensor decomposition for neural network compression.

submitted to <<Neural Networks>> on Apr.18,2020; accepted on Sep.08,2020

References in corpus (17)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Distilling the Knowledge in a Neural Network
• Sequence to Sequence Learning with Neural Networks
• Neural Architecture Search with Reinforcement Learning
• Enhanced LSTM for Natural Language Inference
• Predicting Parameters in Deep Learning
• To prune, or not to prune: exploring the efficacy of pruning for model compression
• Training and Inference with Integers in Deep Neural Networks
• Memory-based control with recurrent neural networks
• On the Expressive Power of Deep Learning: A Tensor Analysis
• Compression of Deep Convolutional Neural Networks for Fast and Low Power Mobile Applications
• Ultimate tensorization: compressing convolutional and FC layers alike
• Tensor-Train Recurrent Neural Networks for Video Classification
• Compressing 3DCNNs Based on Tensor Train Decomposition
• Convolutional Rectifier Networks as Generalized Tensor Decompositions
• Regularized Computation of Approximate Pseudoinverse of Large Matrices Using Low-Rank Tensor Train Decompositions
• Learning Compact Recurrent Neural Networks with Block-Term Tensor Decomposition

Cited by in corpus (4)

• Kronecker CP Decomposition with Fast Multiplication for Compressing RNNs
• Efficient Visual Recognition with Deep Neural Networks: A Survey on Recent Advances and New Directions
• Semi-tensor Product-based TensorDecomposition for Neural Network Compression
• Joint Matrix Decomposition for Deep Convolutional Neural Networks Compression