Training Skinny Deep Neural Networks with Iterative Hard Thresholding Methods

Abstract

Deep neural networks have achieved remarkable success in a wide range of practical problems. However, due to the inherent large parameter space, deep models are notoriously prone to overfitting and difficult to be deployed in portable devices with limited memory. In this paper, we propose an iterative hard thresholding (IHT) approach to train Skinny Deep Neural Networks (SDNNs). An SDNN has much fewer parameters yet can achieve competitive or even better performance than its full CNN counterpart. More concretely, the IHT approach trains an SDNN through following two alternative phases: (I) perform hard thresholding to drop connections with small activations and fine-tune the other significant filters; (II)~re-activate the frozen connections and train the entire network to improve its overall discriminative capability. We verify the superiority of SDNNs in terms of efficiency and classification performance on four benchmark object recognition datasets, including CIFAR-10, CIFAR-100, MNIST and ImageNet. Experimental results clearly demonstrate that IHT can be applied for training SDNN based on various CNN architectures such as NIN and AlexNet.

References in corpus (13)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• FitNets: Hints for Thin Deep Nets
• Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
• Going Deeper with Convolutions
• Deeply-Supervised Nets
• Compressing Deep Convolutional Networks using Vector Quantization
• Stochastic Pooling for Regularization of Deep Convolutional Neural Networks
• Learning Activation Functions to Improve Deep Neural Networks
• Memory Bounded Deep Convolutional Networks
• Deep Networks with Internal Selective Attention through Feedback Connections
• Improving Deep Neural Networks with Probabilistic Maxout Units
• Structured Pruning of Deep Convolutional Neural Networks
• Fast ConvNets Using Group-wise Brain Damage

Cited by in corpus (22)

• The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
• DSD: Dense-Sparse-Dense Training for Deep Neural Networks
• The Power of Sparsity in Convolutional Neural Networks
• Learning to Prune Deep Neural Networks via Layer-wise Optimal Brain Surgeon
• Task dependent Deep LDA pruning of neural networks
• Dynamic Hard Pruning of Neural Networks at the Edge of the Internet
• AC/DC: Alternating Compressed/DeCompressed Training of Deep Neural Networks
• A concatenating framework of shortcut convolutional neural networks
• BitNet: Bit-Regularized Deep Neural Networks
• Exploring Weight Importance and Hessian Bias in Model Pruning
• Generalized Bayesian Posterior Expectation Distillation for Deep Neural Networks
• WSNet: Compact and Efficient Networks Through Weight Sampling
• Meta-Learning with Network Pruning
• Localization-aware Channel Pruning for Object Detection
• Deep Double Sparsity Encoder: Learning to Sparsify Not Only Features But Also Parameters
• Unified Signal Compression Using a GAN with Iterative Latent Representation Optimization
• How Well Do Sparse Imagenet Models Transfer?
• Masked Training of Neural Networks with Partial Gradients
• Explore the Knowledge contained in Network Weights to Obtain Sparse Neural Networks
• Grow-Push-Prune: aligning deep discriminants for effective structural network compression
• Light Multi-segment Activation for Model Compression
• Generalization Bounds for High-dimensional M-estimation under Sparsity Constraint