Ultimate tensorization: compressing convolutional and FC layers alike

Abstract

Convolutional neural networks excel in image recognition tasks, but this comes at the cost of high computational and memory complexity. To tackle this problem, [1] developed a tensor factorization framework to compress fully-connected layers. In this paper, we focus on compressing convolutional layers. We show that while the direct application of the tensor framework [1] to the 4-dimensional kernel of convolution does compress the layer, we can do better. We reshape the convolutional kernel into a tensor of higher order and factorize it. We combine the proposed approach with the previous work to compress both convolutional and fully-connected layers of a network and achieve 80x network compression rate with 1.1% accuracy drop on the CIFAR-10 dataset.

NIPS 2016 workshop: Learning with Tensors: Why Now and How?

References in corpus (1)

• Caffe: Convolutional Architecture for Fast Feature Embedding

Cited by in corpus (5)

• Tensor-Train Recurrent Neural Networks for Video Classification
• Compression of Recurrent Neural Networks for Efficient Language Modeling
• Sharing Residual Units Through Collective Tensor Factorization in Deep Neural Networks
• Bayesian Tensorized Neural Networks with Automatic Rank Selection
• BT-Nets: Simplifying Deep Neural Networks via Block Term Decomposition