Transformed ${\ell }_1$ Regularization for Learning Sparse Deep Neural Networks

Abstract

Deep neural networks (DNNs) have achieved extraordinary success in numerous areas. However, to attain this success, DNNs often carry a large number of weight parameters, leading to heavy costs of memory and computation resources. Overfitting is also likely to happen in such network when the training data are insufficient. These shortcomings severely hinder the application of DNNs in resource-constrained platforms. In fact, many network weights are known to be redundant and can be removed from the network without much loss of performance. To this end, we introduce a new non-convex integrated transformed ${\ell }_1$ regularizer to promote sparsity for DNNs, which removes both redundant connections and unnecessary neurons simultaneously. To be specific, we apply the transformed ${\ell }_1$ to the matrix space of network weights and utilize it to remove redundant connections. Besides, group sparsity is also employed as an auxiliary to remove unnecessary neurons. An efficient stochastic proximal gradient algorithm is presented to solve the new model at the same time. To the best of our knowledge, this is the first work to utilize a non-convex regularizer in sparse optimization based method to promote sparsity for DNNs. Experiments on several public datasets demonstrate the effectiveness of the proposed method.

References in corpus (11)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Deep Learning in Neural Networks: An Overview
• Improving neural networks by preventing co-adaptation of feature detectors
• Nearly unbiased variable selection under minimax concave penalty
• A Survey of Model Compression and Acceleration for Deep Neural Networks
• Group Sparse Regularization for Deep Neural Networks
• Learning the Number of Neurons in Deep Networks
• Exploring Sparsity in Recurrent Neural Networks
• Memory Bounded Deep Convolutional Networks
• An exploration of parameter redundancy in deep networks with circulant projections
• Efficient Sparse Group Feature Selection via Nonconvex Optimization

Cited by in corpus (4)

• Consistent Sparse Deep Learning: Theory and Computation
• A Semi-Smooth Newton Algorithm for High-Dimensional Nonconvex Sparse Learning
• A Support Detection and Root Finding Approach for Learning High-dimensional Generalized Linear Models
• Adaptive Low-Rank Regularization with Damping Sequences to Restrict Lazy Weights in Deep Networks