Adaptive Approximation and Generalization of Deep Neural Network with Intrinsic Dimensionality

Abstract

In this study, we prove that an intrinsic low dimensionality of covariates is the main factor that determines the performance of deep neural networks (DNNs). DNNs generally provide outstanding empirical performance. Hence, numerous studies have actively investigated the theoretical properties of DNNs to understand their underlying mechanisms. In particular, the behavior of DNNs in terms of high-dimensional data is one of the most critical questions. However, this issue has not been sufficiently investigated from the aspect of covariates, although high-dimensional data have practically low intrinsic dimensionality. In this study, we derive bounds for an approximation error and a generalization error regarding DNNs with intrinsically low dimensional covariates. We apply the notion of the Minkowski dimension and develop a novel proof technique. Consequently, we show that convergence rates of the errors by DNNs do not depend on the nominal high dimensionality of data, but on its lower intrinsic dimension. We further prove that the rate is optimal in the minimax sense. We identify an advantage of DNNs by showing that DNNs can handle a broader class of intrinsic low dimensional data than other adaptive estimators. Finally, we conduct a numerical simulation to validate the theoretical results.

arXiv comment38 pages

Cited by in corpus (10)

• Deep Network Approximation for Smooth Functions
• Machine Learning for Prediction with Missing Dynamics
• Deep learning is adaptive to intrinsic dimensionality of model smoothness in anisotropic Besov space
• An error analysis of generative adversarial networks for learning distributions
• Finite Sample Analysis of Minimax Offline Reinforcement Learning: Completeness, Fast Rates and First-Order Efficiency
• A likelihood approach to nonparametric estimation of a singular distribution using deep generative models
• Quantile regression with deep ReLU Networks: Estimators and minimax rates
• InfoNEAT: Information Theory-based NeuroEvolution of Augmenting Topologies for Side-channel Analysis
• Stationary Density Estimation of Itô Diffusions Using Deep Learning
• Nearly Optimal Learning using Sparse Deep ReLU Networks in Regularized Empirical Risk Minimization with Lipschitz Loss