How neural networks find generalizable solutions: Self-tuned annealing in deep learning

Yu Feng, Yuhai Tu

Despite the tremendous success of Stochastic Gradient Descent (SGD) algorithm in deep learning, little is known about how SGD finds generalizable solutions in the high-dimensional…