How neural networks find generalizable solutions: Self-tuned annealing in deep learning

Abstract

Despite the tremendous success of Stochastic Gradient Descent (SGD) algorithm in deep learning, little is known about how SGD finds generalizable solutions in the high-dimensional weight space. By analyzing the learning dynamics and loss function landscape, we discover a robust inverse relation between the weight variance and the landscape flatness (inverse of curvature) for all SGD-based learning algorithms. To explain the inverse variance-flatness relation, we develop a random landscape theory, which shows that the SGD noise strength (effective temperature) depends inversely on the landscape flatness. Our study indicates that SGD attains a self-tuned landscape-dependent annealing strategy to find generalizable solutions at the flat minima of the landscape. Finally, we demonstrate how these new theoretical insights lead to more efficient algorithms, e.g., for avoiding catastrophic forgetting.

References in corpus (7)

• Overcoming catastrophic forgetting in neural networks
• A Mean Field View of the Landscape of Two-Layers Neural Networks
• Gradient Episodic Memory for Continual Learning
• Unreasonable Effectiveness of Learning Neural Networks: From Accessible States and Robust Ensembles to Basic Algorithmic Schemes
• Gradient Descent Happens in a Tiny Subspace
• Entropy-SGD: Biasing Gradient Descent Into Wide Valleys
• Energy-entropy competition and the effectiveness of stochastic gradient descent in machine learning

Cited by in corpus (3)

• Phases of learning dynamics in artificial neural networks: with or without mislabeled data
• Privately Learning Subspaces
• SGB: Stochastic Gradient Bound Method for Optimizing Partition Functions