Analyzing Upper Bounds on Mean Absolute Errors for Deep Neural Network Based Vector-to-Vector Regression
arXiv:2008.05459 · doi:10.1109/TSP.2020.2993164
Abstract
In this paper, we show that, in vector-to-vector regression utilizing deep neural networks (DNNs), a generalized loss of mean absolute error (MAE) between the predicted and expected feature vectors is upper bounded by the sum of an approximation error, an estimation error, and an optimization error. Leveraging upon error decomposition techniques in statistical learning theory and non-convex optimization theory, we derive upper bounds for each of the three aforementioned errors and impose necessary constraints on DNN models. Moreover, we assess our theoretical results through a set of image de-noising and speech enhancement experiments. Our proposed upper bounds of MAE for DNN based vector-to-vector regression are corroborated by the experimental results and the upper bounds are valid with and without the "over-parametrization" technique.
References in corpus (14)
- Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising
- A Convergence Theory for Deep Learning via Over-Parameterization
- A Mean Field View of the Landscape of Two-Layers Neural Networks
- Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Data
- The Power of Depth for Feedforward Neural Networks
- Universal Function Approximation by Deep Neural Nets with Bounded Width and ReLU Activations
- Towards Understanding the Role of Over-Parametrization in Generalization of Neural Networks
- Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layers
- Why Deep Neural Networks for Function Approximation?
- The power of deeper networks for expressing natural functions
- On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transport
- Norm-Based Capacity Control in Neural Networks
- Characterizing Speech Adversarial Examples Using Self-Attention U-Net Enhancement
- On exponential convergence of SGD in non-convex over-parametrized learning