What needles do sparse neural networks find in nonlinear haystacks

Abstract

Using a sparsity inducing penalty in artificial neural networks (ANNs) avoids over-fitting, especially in situations where noise is high and the training set is small in comparison to the number of features. For linear models, such an approach provably also recovers the important features with high probability in regimes for a well-chosen penalty parameter. The typical way of setting the penalty parameter is by splitting the data set and performing the cross-validation, which is (1) computationally expensive and (2) not desirable when the data set is already small to be further split (for example, whole-genome sequence data). In this study, we establish the theoretical foundation to select the penalty parameter without cross-validation based on bounding with a high probability the infinite norm of the gradient of the loss function at zero under the zero-feature assumption. Our approach is a generalization of the universal threshold of Donoho and Johnstone (1994) to nonlinear ANN learning. We perform a set of comprehensive Monte Carlo simulations on a simple model, and the numerical results show the effectiveness of the proposed approach.

arXiv comment8 pages, 2 figures

References in corpus (6)

• Improving neural networks by preventing co-adaptation of feature detectors
• Square-Root Lasso: Pivotal Recovery of Sparse Signals via Conic Programming
• Needles and straw in haystacks: Empirical Bayes estimates of possibly sparse sequences
• The generalization error of random features regression: Precise asymptotics and double descent curve
• Scaling description of generalization with number of parameters in deep learning
• Memory Bounded Deep Convolutional Networks