Directional Convergence Analysis under Spherically Symmetric Distribution

Abstract

We consider the fundamental problem of learning linear predictors (i.e., separable datasets with zero margin) using neural networks with gradient flow or gradient descent. Under the assumption of spherically symmetric data distribution, we show directional convergence guarantees with exact convergence rate for two-layer non-linear networks with only two hidden nodes, and (deep) linear networks. Moreover, our discovery is built on dynamic from the initialization without both initial loss and perfect classification constraint in contrast to previous works. We also point out and study the challenges in further strengthening and generalizing our results.

References in corpus (14)

• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• A Convergence Theory for Deep Learning via Over-Parameterization
• Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Data
• Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks
• Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networks
• On the Optimization of Deep Networks: Implicit Acceleration by Overparameterization
• Gradient descent aligns the layers of deep linear networks
• Gradient Descent Maximizes the Margin of Homogeneous Neural Networks
• Risk and parameter convergence of logistic regression
• Generalization Error Bounds of Gradient Descent for Learning Over-parameterized Deep ReLU Networks
• Stochastic Gradient Descent on Separable Data: Exact Convergence with a Fixed Learning Rate
• Luck Matters: Understanding Training Dynamics of Deep ReLU Networks
• Learning a Single Neuron with Gradient Methods
• Gradient descent follows the regularization path for general losses