Nonparametric regression using deep neural networks with ReLU activation function

Abstract

Consider the multivariate nonparametric regression model. It is shown that estimators based on sparsely connected deep neural networks with ReLU activation function and properly chosen network architecture achieve the minimax rates of convergence (up to $\log n$-factors) under a general composition assumption on the regression function. The framework includes many well-studied structural constraints such as (generalized) additive models. While there is a lot of flexibility in the network architecture, the tuning parameter is the sparsity of the network. Specifically, we consider large networks with number of potential network parameters exceeding the sample size. The analysis gives some insights into why multilayer feedforward neural networks perform well in practice. Interestingly, for ReLU activation function the depth (number of layers) of the neural network architectures plays an important role and our theory suggests that for nonparametric regression, scaling the network depth with the sample size is natural. It is also shown that under the composition assumption wavelet estimators can only achieve suboptimal rates.

article, rejoinder and supplementary material

References in corpus (4)

• Comparison of non-linear activation functions for deep neural networks on MNIST classification task
• Rate-optimal estimation for a general class of nonparametric regression models with unknown link functions
• Theory II: Landscape of the Empirical Risk in Deep Learning
• Nonparametric estimation of composite functions

Cited by in corpus (16)

• Significance tests of feature relevance for a black-box learner
• Near-Minimax Optimal Estimation With Shallow ReLU Neural Networks
• Deep Learning Meets Sparse Regularization: A Signal Processing Perspective
• How Deep Neural Networks Learn Compositional Data: The Random Hierarchy Model
• Audio Event-Relational Graph Representation Learning for Acoustic Scene Classification
• No More Mumbles: Enhancing Robot Intelligibility through Speech Adaptation
• Characterizing 4-string contact interaction using machine learning
• Data Augmentation for Bayesian Deep Learning
• Adaptive variational Bayes: Optimality, computation and applications
• On generalization bounds for deep networks based on loss surface implicit regularization
• Analytic function approximation by path norm regularized deep networks
• Minimax optimal high-dimensional classification using deep neural networks
• Fitted value iteration methods for bicausal optimal transport
• Deep learning for inverse problems with unknown operator
• Performance Guarantees for Quantum Neural Estimation of Entropies
• Mini-batch Estimation for Deep Cox Models: Statistical Foundations and Practical Guidance