Group Sparse Regularization for Deep Neural Networks

Abstract

In this paper, we consider the joint task of simultaneously optimizing (i) the weights of a deep neural network, (ii) the number of neurons for each hidden layer, and (iii) the subset of active input features (i.e., feature selection). While these problems are generally dealt with separately, we present a simple regularized formulation allowing to solve all three of them in parallel, using standard optimization routines. Specifically, we extend the group Lasso penalty (originated in the linear regression literature) in order to impose group-level sparsity on the network's connections, where each group is defined as the set of outgoing weights from a unit. Depending on the specific case, the weights can be related to an input variable, to a hidden neuron, or to a bias unit, thus performing simultaneously all the aforementioned tasks in order to obtain a compact network. We perform an extensive experimental evaluation, by comparing with classical weight decay and Lasso penalties. We show that a sparse version of the group Lasso penalty is able to achieve competitive performances, while at the same time resulting in extremely compact networks with a smaller number of input features. We evaluate both on a toy dataset for handwritten digit recognition, and on multiple realistic large-scale classification problems.

References in corpus (7)

• Deep Learning in Neural Networks: An Overview
• Distilling the Knowledge in a Neural Network
• Deep Learning with Limited Numerical Precision
• Compressing Deep Convolutional Networks using Vector Quantization
• Consistency of the group Lasso and multiple kernel learning
• Compressing Neural Networks with the Hashing Trick
• Multi-Task Feature Learning Via Efficient l2,1-Norm Minimization

Cited by in corpus (14)

• Learning Efficient Convolutional Networks through Network Slimming
• Model Selection in Bayesian Neural Networks via Horseshoe Priors
• Discovering Nonlinear Relations with Minimum Predictive Information Regularization
• Transformed ${\ell }_1$ Regularization for Learning Sparse Deep Neural Networks
• SSN: Learning Sparse Switchable Normalization via SparsestMax
• Variable Selection with Rigorous Uncertainty Quantification using Deep Bayesian Neural Networks: Posterior Concentration and Bernstein-von Mises Phenomenon
• Intelligence, physics and information -- the tradeoff between accuracy and simplicity in machine learning
• Automatic Node Selection for Deep Neural Networks using Group Lasso Regularization
• Sparsely Grouped Input Variables for Neural Networks
• A Support Detection and Root Finding Approach for Learning High-dimensional Generalized Linear Models
• An Improving Framework of regularization for Network Compression
• Adam Induces Implicit Weight Sparsity in Rectifier Neural Networks
• Spatially-Coupled Neural Network Architectures
• Dynamic Regularizer with an Informative Prior