Tensorial Neural Networks: Generalization of Neural Networks and Application to Model Compression

Abstract

We propose tensorial neural networks (TNNs), a generalization of existing neural networks by extending tensor operations on low order operands to those on high order ones. The problem of parameter learning is challenging, as it corresponds to hierarchical nonlinear tensor decomposition. We propose to solve the learning problem using stochastic gradient descent by deriving nontrivial backpropagation rules in generalized tensor algebra we introduce. Our proposed TNNs has three advantages over existing neural networks: (1) TNNs naturally apply to high order input object and thus preserve the multi-dimensional structure in the input, as there is no need to flatten the data. (2) TNNs interpret designs of existing neural network architectures. (3) Mapping a neural network to TNNs with the same expressive power results in a TNN of fewer parameters. TNN based compression of neural network improves existing low-rank approximation based compression methods as TNNs exploit two other types of invariant structures, periodicity and modulation, in addition to the low rankness. Experiments on LeNet-5 (MNIST), ResNet-32 (CIFAR10) and ResNet-50 (ImageNet) demonstrate that our TNN based compression outperforms (5% test accuracy improvement universally on CIFAR10) the state-of-the-art low-rank approximation based compression methods under the same compression rate, besides achieving orders of magnitude faster convergence rates due to the efficiency of TNNs.

References in corpus (20)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Distilling the Knowledge in a Neural Network
• Deep Residual Learning for Image Recognition
• FitNets: Hints for Thin Deep Nets
• BinaryConnect: Training Deep Neural Networks with binary weights during propagations
• Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
• A Survey of Model Compression and Acceleration for Deep Neural Networks
• Trained Ternary Quantization
• Tensorizing Neural Networks
• Speeding up Convolutional Neural Networks with Low Rank Expansions
• Low-Rank Tensor Networks for Dimensionality Reduction and Large-Scale Optimization Problems: Perspectives and Challenges PART 1
• Born Again Neural Networks
• Xception: Deep Learning with Depthwise Separable Convolutions
• Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decomposition
• Tensor Networks for Dimensionality Reduction and Large-Scale Optimizations. Part 2 Applications and Future Perspectives
• Ultimate tensorization: compressing convolutional and FC layers alike
• Guaranteed Non-Orthogonal Tensor Decomposition via Alternating Rank-$1$ Updates
• Learning Deep ResNet Blocks Sequentially using Boosting Theory
• Expressive power of recurrent neural networks

Cited by in corpus (9)

• Convolutional Tensor-Train LSTM for Spatio-temporal Learning
• Compressing 3DCNNs Based on Tensor Train Decomposition
• Efficient Visual Recognition with Deep Neural Networks: A Survey on Recent Advances and New Directions
• Taxonomy and Evaluation of Structured Compression of Convolutional Neural Networks
• Understanding Generalization in Deep Learning via Tensor Methods
• Structured Convolutions for Efficient Neural Network Design
• Scaling Up Exact Neural Network Compression by ReLU Stability
• Sampling-Free Learning of Bayesian Quantized Neural Networks
• Block-term Tensor Neural Networks