Group Sparse Regularization for Deep Neural Networks

Abstract

In this paper, we consider the joint task of simultaneously optimizing (i) the weights of a deep neural network, (ii) the number of neurons for each hidden layer, and (iii) the subset of active input features (i.e., feature selection). While these problems are generally dealt with separately, we present a simple regularized formulation allowing to solve all three of them in parallel, using standard optimization routines. Specifically, we extend the group Lasso penalty (originated in the linear regression literature) in order to impose group-level sparsity on the network's connections, where each group is defined as the set of outgoing weights from a unit. Depending on the specific case, the weights can be related to an input variable, to a hidden neuron, or to a bias unit, thus performing simultaneously all the aforementioned tasks in order to obtain a compact network. We perform an extensive experimental evaluation, by comparing with classical weight decay and Lasso penalties. We show that a sparse version of the group Lasso penalty is able to achieve competitive performances, while at the same time resulting in extremely compact networks with a smaller number of input features. We evaluate both on a toy dataset for handwritten digit recognition, and on multiple realistic large-scale classification problems.

References in corpus (16)

• Adam: A Method for Stochastic Optimization
• Scikit-learn: Machine Learning in Python
• Deep Learning in Neural Networks: An Overview
• Distilling the Knowledge in a Neural Network
• BinaryConnect: Training Deep Neural Networks with binary weights during propagations
• HOGWILD!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent
• Deep Learning with Limited Numerical Precision
• Compressing Deep Convolutional Networks using Vector Quantization
• Consistency of the group Lasso and multiple kernel learning
• Predicting Parameters in Deep Learning
• A note on the group lasso and a sparse group lasso
• Compressing Neural Networks with the Hashing Trick
• Multi-Task Feature Learning Via Efficient l2,1-Norm Minimization
• Structured Variable Selection with Sparsity-Inducing Norms
• Decomposition by Partial Linearization: Parallel Optimization of Multi-Agent Systems
• Deep Networks with Stochastic Depth

Cited by in corpus (74)

• Bayesian Compression for Deep Learning
• Variational Dropout Sparsifies Deep Neural Networks
• An overview and comparative analysis of Recurrent Neural Networks for Short Term Load Forecasting
• Learning Efficient Convolutional Networks through Network Slimming
• Sparse-Input Neural Networks for High-dimensional Nonparametric Regression and Classification
• Model Selection in Bayesian Neural Networks via Horseshoe Priors
• Neural Network Training Using ${\ell }_1$-Regularization and Bi-fidelity Data
• Regularization via Structural Label Smoothing
• A Framework for Parallel and Distributed Training of Neural Networks
• A Closer Look at Structured Pruning for Neural Network Compression
• A New Class of Efficient Adaptive Filters for Online Nonlinear Modeling
• Compressing deep quaternion neural networks with targeted regularization
• Online regularization of complex-valued neural networks for structure optimization in wireless-communication channel prediction
• On the Compression of Neural Networks Using ${\ell }_0$-Norm Regularization and Weight Pruning
• Discovering Nonlinear Relations with Minimum Predictive Information Regularization
• Differentiable Unsupervised Feature Selection based on a Gated Laplacian
• Layer Adaptive Node Selection in Bayesian Neural Networks: Statistical Guarantees and Implementation Details
• Deep learning insights into cosmological structure formation
• Continual BERT: Continual Learning for Adaptive Extractive Summarization of COVID-19 Literature
• Transformed ${\ell }_1$ Regularization for Learning Sparse Deep Neural Networks
• Regularizing by the Variance of the Activations' Sample-Variances
• SSN: Learning Sparse Switchable Normalization via SparsestMax
• Fiedler Regularization: Learning Neural Networks with Graph Sparsity
• Characterising Across-Stack Optimisations for Deep Convolutional Neural Networks
• Structured Bayesian Compression for Deep Neural Networks Based on The Turbo-VBI Approach
• Continual Graph Learning: A Survey
• Consistent Feature Selection for Analytic Deep Neural Networks
• Consistent Sparse Deep Learning: Theory and Computation
• VINNAS: Variational Inference-based Neural Network Architecture Search
• Momentum-based variance-reduced proximal stochastic gradient method for composite nonconvex stochastic optimization
• Schizophrenia-mimicking layers outperform conventional neural network layers
• Combined Sparse Regularization for Nonlinear Adaptive Filters
• Pruning deep neural networks generates a sparse, bio-inspired nonlinear controller for insect flight
• VACL: Variance-Aware Cross-Layer Regularization for Pruning Deep Residual Networks
• Variable Selection with Rigorous Uncertainty Quantification using Deep Bayesian Neural Networks: Posterior Concentration and Bernstein-von Mises Phenomenon
• Learning Neural Network Classifiers with Low Model Complexity
• Gradient-Coherent Strong Regularization for Deep Neural Networks
• Intelligence, physics and information -- the tradeoff between accuracy and simplicity in machine learning
• Variable selection with false discovery rate control in deep neural networks
• Active Subspace of Neural Networks: Structural Analysis and Universal Attacks
• Three Operator Splitting with Subgradients, Stochastic Gradients, and Adaptive Learning Rates
• Binary Stochastic Filtering: feature selection and beyond
• Automatic Node Selection for Deep Neural Networks using Group Lasso Regularization
• DECORE: Deep Compression with Reinforcement Learning
• A Bregman Learning Framework for Sparse Neural Networks
• On the convergence of group-sparse autoencoders
• Half-Space Proximal Stochastic Gradient Method for Group-Sparsity Regularized Problem
• Smaller Models, Better Generalization
• An ETF view of Dropout regularization
• Constant Step Stochastic Approximations Involving Differential Inclusions: Stability, Long-Run Convergence and Applications
• Improving Network Slimming with Nonconvex Regularization
• Non-linear, Sparse Dimensionality Reduction via Path Lasso Penalized Autoencoders
• BEAN: Interpretable Representation Learning with Biologically-Enhanced Artificial Neuronal Assembly Regularization
• Lockout: Sparse Regularization of Neural Networks
• CASTNet: Community-Attentive Spatio-Temporal Networks for Opioid Overdose Forecasting
• A Distributionally Robust Optimization Method for Adversarial Multiple Kernel Learning
• Efficient Proximal Mapping of the 1-path-norm of Shallow Networks
• Dataset Optimization Strategies for MalwareTraffic Detection
• A Semi-Smooth Newton Algorithm for High-Dimensional Nonconvex Sparse Learning
• Sparsely Grouped Input Variables for Neural Networks
• Neural Architecture Search as Sparse Supernet
• Hierarchical Group Sparse Regularization for Deep Convolutional Neural Networks
• An Improving Framework of regularization for Network Compression
• Spatially-Coupled Neural Network Architectures
• A Partial Regularization Method for Network Compression
• Neural Architecture Search via Bregman Iterations
• Unsupervised Deep Learning by Injecting Low-Rank and Sparse Priors
• RNN Architecture Learning with Sparse Regularization
• A Support Detection and Root Finding Approach for Learning High-dimensional Generalized Linear Models
• Adam Induces Implicit Weight Sparsity in Rectifier Neural Networks
• Sum-Rate-Distortion Function for Indirect Multiterminal Source Coding in Federated Learning
• Dynamic Regularizer with an Informative Prior
• Learning to Actively Reduce Memory Requirements for Robot Control Tasks
• Efficient Sparse Artificial Neural Networks