Learning a Single Neuron with Gradient Methods

Abstract

We consider the fundamental problem of learning a single neuron $x\mapsto \sigma (w^{\top }x)$ using standard gradient methods. As opposed to previous works, which considered specific (and not always realistic) input distributions and activation functions $\sigma (\cdot )$, we ask whether a more general result is attainable, under milder assumptions. On the one hand, we show that some assumptions on the distribution and the activation function are necessary. On the other hand, we prove positive guarantees under mild assumptions, which go beyond those studied in the literature so far. We also point out and study the challenges in further strengthening and generalizing our results.

arXiv commentFixed a small bug in the proof of Theorem 4.2

References in corpus (12)

• Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks
• Beating the Perils of Non-Convexity: Guaranteed Training of Neural Networks using Tensor Methods
• When Are Nonconvex Problems Not Scary?
• Learning ReLUs via Gradient Descent
• An Analytical Formula of Population Gradient for two-layered ReLU network and its Applications in Convergence and Critical Point Analysis
• The Landscape of Empirical Risk for Non-convex Losses
• Overparameterized Nonlinear Learning: Gradient Descent Takes the Shortest Path?
• Generalization Error Bounds of Gradient Descent for Learning Over-parameterized Deep ReLU Networks
• Reliably Learning the ReLU in Polynomial Time
• On the Power and Limitations of Random Features for Understanding Neural Networks
• A Variant of Azuma's Inequality for Martingales with Subgaussian Tails
• Online Stochastic Gradient Descent with Arbitrary Initialization Solves Non-smooth, Non-convex Phase Retrieval

Cited by in corpus (8)

• Statistical-Query Lower Bounds via Functional Gradients
• Early-stopped neural networks are consistent
• Implicit Regularization in ReLU Networks with the Square Loss
• The Connection Between Approximation, Depth Separation and Learnability in Neural Networks
• The Effects of Mild Over-parameterization on the Optimization Landscape of Shallow ReLU Neural Networks
• Understanding How Over-Parametrization Leads to Acceleration: A case of learning a single teacher neuron
• A Modular Analysis of Provable Acceleration via Polyak's Momentum: Training a Wide ReLU Network and a Deep Linear Network
• Directional Convergence Analysis under Spherically Symmetric Distribution