Generalization Error Analysis of Neural networks with Gradient Based Regularization

Abstract

We study gradient-based regularization methods for neural networks. We mainly focus on two regularization methods: the total variation and the Tikhonov regularization. Applying these methods is equivalent to using neural networks to solve some partial differential equations, mostly in high dimensions in practical applications. In this work, we introduce a general framework to analyze the generalization error of regularized networks. The error estimate relies on two assumptions on the approximation error and the quadrature error. Moreover, we conduct some experiments on the image classification tasks to show that gradient-based methods can significantly improve the generalization ability and adversarial robustness of neural networks. A graphical extension of the gradient-based methods are also considered in the experiments.

References in corpus (13)

• Adam: A Method for Stochastic Optimization
• Decoupled Weight Decay Regularization
• Explaining and Harnessing Adversarial Examples
• Intriguing properties of neural networks
• DGM: A deep learning algorithm for solving partial differential equations
• Solving high-dimensional partial differential equations using deep learning
• One pixel attack for fooling deep neural networks
• Hidden Physics Models: Machine Learning of Nonlinear Partial Differential Equations
• Adaptive activation functions accelerate convergence in deep and physics-informed neural networks
• Multi-scale Deep Neural Network (MscaleDNN) for Solving Poisson-Boltzmann Equation in Complex Domains
• Deep Nitsche Method: Deep Ritz Method with Essential Boundary Conditions
• Machine Learning and Computational Mathematics
• Multi-scale Deep Neural Network (MscaleDNN) Methods for Oscillatory Stokes Flows in Complex Domains