Ultimate tensorization: compressing convolutional and FC layers alike

Abstract

Convolutional neural networks excel in image recognition tasks, but this comes at the cost of high computational and memory complexity. To tackle this problem, [1] developed a tensor factorization framework to compress fully-connected layers. In this paper, we focus on compressing convolutional layers. We show that while the direct application of the tensor framework [1] to the 4-dimensional kernel of convolution does compress the layer, we can do better. We reshape the convolutional kernel into a tensor of higher order and factorize it. We combine the proposed approach with the previous work to compress both convolutional and fully-connected layers of a network and achieve 80x network compression rate with 1.1% accuracy drop on the CIFAR-10 dataset.

NIPS 2016 workshop: Learning with Tensors: Why Now and How?

References in corpus (4)

• Caffe: Convolutional Architecture for Fast Feature Embedding
• Predicting Parameters in Deep Learning
• Tensorizing Neural Networks
• PerforatedCNNs: Acceleration through Elimination of Redundant Convolutions

Cited by in corpus (49)

• Variational Dropout Sparsifies Deep Neural Networks
• Tensor Networks for Dimensionality Reduction and Large-Scale Optimizations. Part 2 Applications and Future Perspectives
• Tensor Methods in Computer Vision and Deep Learning
• Deep $k$-Means: Re-Training and Parameter Sharing with Harder Cluster Assignments for Compressing Deep Convolutions
• Tensor-Train Recurrent Neural Networks for Video Classification
• Tensorized Embedding Layers for Efficient Model Compression
• Hybrid Tensor Decomposition in Neural Network Compression
• Compression of Recurrent Neural Networks for Efficient Language Modeling
• Neural Networks Compression for Language Modeling
• Sharing Residual Units Through Collective Tensor Factorization in Deep Neural Networks
• Wide Compression: Tensor Ring Nets
• Compressing 3DCNNs Based on Tensor Train Decomposition
• Efficient Visual Recognition with Deep Neural Networks: A Survey on Recent Advances and New Directions
• Tensorial Neural Networks: Generalization of Neural Networks and Application to Model Compression
• Bayesian Tensorized Neural Networks with Automatic Rank Selection
• Taxonomy and Evaluation of Structured Compression of Convolutional Neural Networks
• BT-Nets: Simplifying Deep Neural Networks via Block Term Decomposition
• Understanding Generalization in Deep Learning via Tensor Methods
• QTN-VQC: An End-to-End Learning framework for Quantum Neural Networks
• Sparse Weight Activation Training
• SGAD: Soft-Guided Adaptively-Dropped Neural Network
• Adaptive Learning of Tensor Network Structures
• Tensorizing GAN with High-Order Pooling for Alzheimer's Disease Assessment
• CNN Acceleration by Low-rank Approximation with Quantized Factors
• A flexible, extensible software framework for model compression based on the LC algorithm
• Compressing Neural Networks: Towards Determining the Optimal Layer-wise Decomposition
• Dynamic Sparse Graph for Efficient Deep Learning
• Quantum Tensor Networks, Stochastic Processes, and Weighted Automata
• Heuristic Rank Selection with Progressively Searching Tensor Ring Network
• Active Subspace of Neural Networks: Structural Analysis and Universal Attacks
• Exploring Deep Hybrid Tensor-to-Vector Network Architectures for Regression Based Speech Enhancement
• Complexity for deep neural networks and other characteristics of deep feature representations
• DP-Net: Dynamic Programming Guided Deep Neural Network Compression
• 3U-EdgeAI: Ultra-Low Memory Training, Ultra-Low BitwidthQuantization, and Ultra-Low Latency Acceleration
• Joint Matrix Decomposition for Deep Convolutional Neural Networks Compression
• Enabling Lightweight Fine-tuning for Pre-trained Language Model Compression based on Matrix Product Operators
• Building Compact and Robust Deep Neural Networks with Toeplitz Matrices
• Semi-tensor Product-based TensorDecomposition for Neural Network Compression
• A Fully Tensorized Recurrent Neural Network
• Low-Rank Embedding of Kernels in Convolutional Neural Networks under Random Shuffling
• Penetrating the Fog: the Path to Efficient CNN Models
• A Sampling-Based Method for Tensor Ring Decomposition
• Low-Rank+Sparse Tensor Compression for Neural Networks
• Towards Efficient Tensor Decomposition-Based DNN Model Compression with Optimization Framework
• MARS: Masked Automatic Ranks Selection in Tensor Decompositions
• Tensor Yard: One-Shot Algorithm of Hardware-Friendly Tensor-Train Decomposition for Convolutional Neural Networks
• Spectral Tensor Train Parameterization of Deep Learning Layers
• Tensor Methods for Generating Compact Uncertainty Quantification and Deep Learning Models
• Nonlinear Tensor Ring Network