Group Sparse Regularization for Deep Neural Networks

Abstract

In this paper, we consider the joint task of simultaneously optimizing (i) the weights of a deep neural network, (ii) the number of neurons for each hidden layer, and (iii) the subset of active input features (i.e., feature selection). While these problems are generally dealt with separately, we present a simple regularized formulation allowing to solve all three of them in parallel, using standard optimization routines. Specifically, we extend the group Lasso penalty (originated in the linear regression literature) in order to impose group-level sparsity on the network's connections, where each group is defined as the set of outgoing weights from a unit. Depending on the specific case, the weights can be related to an input variable, to a hidden neuron, or to a bias unit, thus performing simultaneously all the aforementioned tasks in order to obtain a compact network. We perform an extensive experimental evaluation, by comparing with classical weight decay and Lasso penalties. We show that a sparse version of the group Lasso penalty is able to achieve competitive performances, while at the same time resulting in extremely compact networks with a smaller number of input features. We evaluate both on a toy dataset for handwritten digit recognition, and on multiple realistic large-scale classification problems.

References in corpus (7)

• Deep Learning in Neural Networks: An Overview
• Distilling the Knowledge in a Neural Network
• Deep Learning with Limited Numerical Precision
• Compressing Deep Convolutional Networks using Vector Quantization
• Consistency of the group Lasso and multiple kernel learning
• Compressing Neural Networks with the Hashing Trick
• Multi-Task Feature Learning Via Efficient l2,1-Norm Minimization

Cited by in corpus (7)

• Learning Efficient Convolutional Networks through Network Slimming
• Model Selection in Bayesian Neural Networks via Horseshoe Priors
• Transformed ${\ell }_1$ Regularization for Learning Sparse Deep Neural Networks
• SSN: Learning Sparse Switchable Normalization via SparsestMax
• Automatic Node Selection for Deep Neural Networks using Group Lasso Regularization
• Spatially-Coupled Neural Network Architectures
• Adam Induces Implicit Weight Sparsity in Rectifier Neural Networks