Compressing Convolutional Neural Networks

Abstract

Convolutional neural networks (CNN) are increasingly used in many areas of computer vision. They are particularly attractive because of their ability to "absorb" great quantities of labeled data through millions of parameters. However, as model sizes increase, so do the storage and memory requirements of the classifiers. We present a novel network architecture, Frequency-Sensitive Hashed Nets (FreshNets), which exploits inherent redundancy in both convolutional layers and fully-connected layers of a deep learning model, leading to dramatic savings in memory and storage consumption. Based on the key observation that the weights of learned convolutional filters are typically smooth and low-frequency, we first convert filter weights to the frequency domain with a discrete cosine transform (DCT) and use a low-cost hash function to randomly group frequency parameters into hash buckets. All parameters assigned the same hash bucket share a single value learned with standard back-propagation. To further reduce model size we allocate fewer hash buckets to high-frequency components, which are generally less important. We evaluate FreshNets on eight data sets, and show that it leads to drastically better compressed performance than several relevant baselines.

References in corpus (12)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Do Deep Nets Really Need to be Deep?
• Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
• Going Deeper with Convolutions
• Deep Learning with Limited Numerical Precision
• Compressing Deep Convolutional Networks using Vector Quantization
• Predicting Parameters in Deep Learning
• Speeding up Convolutional Neural Networks with Low Rank Expansions
• Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decomposition
• Fast Training of Convolutional Networks through FFTs
• Fast Convolutional Nets With fbfft: A GPU Performance Evaluation
• Deep Fried Convnets

Cited by in corpus (12)

• Bayesian Compression for Deep Learning
• Improving Efficiency in Convolutional Neural Network with Multilinear Filters
• Quantized Convolutional Neural Networks for Mobile Devices
• In Teacher We Trust: Learning Compressed Models for Pedestrian Detection
• ASP Vision: Optically Computing the First Layer of Convolutional Neural Networks using Angle Sensitive Pixels
• BitNet: Bit-Regularized Deep Neural Networks
• Compression of Deep Neural Networks on the Fly
• Line-Circle-Square (LCS): A Multilayered Geometric Filter for Edge-Based Detection
• Sketching and Neural Networks
• Filter sharing: Efficient learning of parameters for volumetric convolutions
• Dense Pruning of Pointwise Convolutions in the Frequency Domain
• Learning Gradual Argumentation Frameworks using Genetic Algorithms