Diverse Neural Network Learns True Target Functions

Abstract

Neural networks are a powerful class of functions that can be trained with simple gradient descent to achieve state-of-the-art performance on a variety of applications. Despite their practical success, there is a paucity of results that provide theoretical guarantees on why they are so effective. Lying in the center of the problem is the difficulty of analyzing the non-convex loss function with potentially numerous local minima and saddle points. Can neural networks corresponding to the stationary points of the loss function learn the true target function? If yes, what are the key factors contributing to such nice optimization properties? In this paper, we answer these questions by analyzing one-hidden-layer neural networks with ReLU activation, and show that despite the non-convexity, neural networks with diverse units have no spurious local minima. We bypass the non-convexity issue by directly analyzing the first order optimality condition, and show that the loss can be made arbitrarily small if the minimum singular value of the "extended feature matrix" is large enough. We make novel use of techniques from kernel methods and geometric discrepancy, and identify a new relation linking the smallest singular value to the spectrum of a kernel function associated with the activation function and to the diversity of the units. Our results also suggest a novel regularization function to promote unit diversity for potentially better generalization.

Cited by in corpus (49)

• Visualizing the Loss Landscape of Neural Nets
• Benign Overfitting in Linear Regression
• Gradient Descent Provably Optimizes Over-parameterized Neural Networks
• Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks
• Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layers
• Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learning
• The Convergence Rate of Neural Networks for Learned Functions of Different Frequencies
• A Comparative Analysis of the Optimization and Generalization Property of Two-layer Neural Network and Random Feature Models Under Gradient Descent Dynamics
• What Can ResNet Learn Efficiently, Going Beyond Kernels?
• Fast Convergence of Natural Gradient Descent for Overparameterized Neural Networks
• Generalization Error Bounds of Gradient Descent for Learning Over-parameterized Deep ReLU Networks
• Global optimality conditions for deep neural networks
• Effect of Depth and Width on Local Minima in Deep Learning
• Frequency Bias in Neural Networks for Input of Non-Uniform Density
• How Much Over-parameterization Is Sufficient to Learn Deep ReLU Networks?
• On Learning Over-parameterized Neural Networks: A Functional Approximation Perspective
• Forward and Backward Information Retention for Accurate Binary Neural Networks
• Stability and Generalization of Learning Algorithms that Converge to Global Optima
• Analysis of the Gradient Descent Algorithm for a Deep Neural Network Model with Skip-connections
• WeMix: How to Better Utilize Data Augmentation
• Fix your classifier: the marginal value of training the last weight layer
• Effect of Activation Functions on the Training of Overparametrized Neural Nets
• A Second look at Exponential and Cosine Step Sizes: Simplicity, Adaptivity, and Performance
• Rethinking Influence Functions of Neural Networks in the Over-parameterized Regime
• When Does Preconditioning Help or Hurt Generalization?
• Learning Parities with Neural Networks
• BiPointNet: Binary Neural Network for Point Clouds
• Deep Neural Networks with Multi-Branch Architectures Are Less Non-Convex
• Deep Equals Shallow for ReLU Networks in Kernel Regimes
• Symmetry & critical points for a model shallow neural network
• Memorizing Gaussians with no over-parameterizaion via gradient decent on neural networks
• On Connected Sublevel Sets in Deep Learning
• Hardness of Learning Neural Networks with Natural Weights
• The Quenching-Activation Behavior of the Gradient Descent Dynamics for Two-layer Neural Network Models
• The Slow Deterioration of the Generalization Error of the Random Feature Model
• A Theory-Driven Self-Labeling Refinement Method for Contrastive Representation Learning
• On the Principle of Least Symmetry Breaking in Shallow ReLU Models
• Are deep ResNets provably better than linear predictors?
• Learning towards Minimum Hyperspherical Energy
• Orthogonal Over-Parameterized Training
• Distribution-sensitive Information Retention for Accurate Binary Neural Network
• Forward Super-Resolution: How Can GANs Learn Hierarchical Generative Models for Real-World Distributions
• Deep Semi-Random Features for Nonlinear Function Approximation
• Proxy Convexity: A Unified Framework for the Analysis of Neural Networks Trained by Gradient Descent
• Fuzzy Logic Interpretation of Quadratic Networks
• When Are Solutions Connected in Deep Networks?
• The Restricted Isometry of ReLU Networks: Generalization through Norm Concentration
• Fine-grained Optimization of Deep Neural Networks
• Learning with Hyperspherical Uniformity