Exploiting Local Structures with the Kronecker Layer in Convolutional Networks

Abstract

In this paper, we propose and study a technique to reduce the number of parameters and computation time in convolutional neural networks. We use Kronecker product to exploit the local structures within convolution and fully-connected layers, by replacing the large weight matrices by combinations of multiple Kronecker products of smaller matrices. Just as the Kronecker product is a generalization of the outer product from vectors to matrices, our method is a generalization of the low rank approximation method for convolution neural networks. We also introduce combinations of different shapes of Kronecker product to increase modeling capacity. Experiments on SVHN, scene text recognition and ImageNet dataset demonstrate that we can achieve $3.3\times$ speedup or $3.6\times$ parameter reduction with less than 1\% drop in accuracy, showing the effectiveness and efficiency of our method. Moreover, the computation efficiency of Kronecker layer makes using larger feature map possible, which in turn enables us to outperform the previous state-of-the-art on both SVHN(digit recognition) and CASIA-HWDB (handwritten Chinese character recognition) datasets.

References in corpus (9)

• Distilling the Knowledge in a Neural Network
• Compressing Deep Convolutional Networks using Vector Quantization
• Synthetic Data and Artificial Neural Networks for Natural Scene Text Recognition
• Multiple Object Recognition with Visual Attention
• Compressing Neural Networks with the Hashing Trick
• Speeding up Convolutional Neural Networks with Low Rank Expansions
• Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decomposition
• Theano-based Large-Scale Visual Recognition with Multiple GPUs
• Speeding Up Neural Networks for Large Scale Classification using WTA Hashing

Cited by in corpus (6)

• A Review on Deep Learning Techniques Applied to Semantic Segmentation
• Sketch2code: Generating a website from a paper mockup
• Deep Learning Techniques for Compressive Sensing-Based Reconstruction and Inference -- A Ubiquitous Systems Perspective
• Balanced Quantization: An Effective and Efficient Approach to Quantized Neural Networks
• Building Fast and Compact Convolutional Neural Networks for Offline Handwritten Chinese Character Recognition
• Doping: A technique for efficient compression of LSTM models using sparse structured additive matrices