Wide Compression: Tensor Ring Nets

Abstract

Deep neural networks have demonstrated state-of-the-art performance in a variety of real-world applications. In order to obtain performance gains, these networks have grown larger and deeper, containing millions or even billions of parameters and over a thousand layers. The trade-off is that these large architectures require an enormous amount of memory, storage, and computation, thus limiting their usability. Inspired by the recent tensor ring factorization, we introduce Tensor Ring Networks (TR-Nets), which significantly compress both the fully connected layers and the convolutional layers of deep neural networks. Our results show that our TR-Nets approach {is able to compress LeNet-5 by $11\times$ without losing accuracy}, and can compress the state-of-the-art Wide ResNet by $243\times$ with only 2.3\% degradation in {Cifar10 image classification}. Overall, this compression scheme shows promise in scientific computing and deep learning, especially for emerging resource-constrained devices such as smartphones, wearables, and IoT devices.

arXiv commentAccepted to CVPR 2018

References in corpus (23)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
• Deep Residual Learning for Image Recognition
• Densely Connected Convolutional Networks
• Wide Residual Networks
• Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
• Going Deeper with Convolutions
• Compressing Deep Convolutional Networks using Vector Quantization
• Tensorizing Neural Networks
• Compressing Neural Networks with the Hashing Trick
• Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decomposition
• Tensor Ring Decomposition
• Tensor Networks for Dimensionality Reduction and Large-Scale Optimizations. Part 2 Applications and Future Perspectives
• On the Expressive Power of Deep Learning: A Tensor Analysis
• Ultimate tensorization: compressing convolutional and FC layers alike
• Tensor-Train Recurrent Neural Networks for Video Classification
• Topic Compositional Neural Language Model
• An exploration of parameter redundancy in deep networks with circulant projections
• Tensor Train Neighborhood Preserving Embedding
• Tensor Completion by Alternating Minimization under the Tensor Train (TT) Model
• Earliness-Aware Deep Convolutional Networks for Early Time Series Classification
• Efficient tensor completion: Low-rank tensor train

Cited by in corpus (12)

• Multi-Stage Hybrid Federated Learning over Large-Scale D2D-Enabled Fog Networks
• Tensorized Embedding Layers for Efficient Model Compression
• Tensor Train Neighborhood Preserving Embedding
• Bayesian Tensorized Neural Networks with Automatic Rank Selection
• Higher-dimension Tensor Completion via Low-rank Tensor Ring Decomposition
• Tensor Ring Decomposition with Rank Minimization on Latent Space: An Efficient Approach for Tensor Completion
• Lower and Upper Bounds on the VC-Dimension of Tensor Network Models
• Generalizing Tensor Decomposition for N-ary Relational Knowledge Bases
• MARS: Masked Automatic Ranks Selection in Tensor Decompositions
• Block-term Tensor Neural Networks
• Nonlinear Tensor Ring Network
• Spectral Tensor Train Parameterization of Deep Learning Layers