Stochastic Training of Residual Networks: a Differential Equation Viewpoint

Abstract

During the last few years, significant attention has been paid to the stochastic training of artificial neural networks, which is known as an effective regularization approach that helps improve the generalization capability of trained models. In this work, the method of modified equations is applied to show that the residual network and its variants with noise injection can be regarded as weak approximations of stochastic differential equations. Such observations enable us to bridge the stochastic training processes with the optimal control of backward Kolmogorov's equations. This not only offers a novel perspective on the effects of regularization from the loss landscape viewpoint but also sheds light on the design of more reliable and efficient stochastic training strategies. As an example, we propose a new way to utilize Bernoulli dropout within the plain residual network architecture and conduct experiments on a real-world image classification task to substantiate our theoretical findings.

arXiv comment20 pages, 8 figures, and 1 table

References in corpus (14)

• Deep Learning in Neural Networks: An Overview
• Wide Residual Networks
• Residual Networks Behave Like Ensembles of Relatively Shallow Networks
• Don't Decay the Learning Rate, Increase the Batch Size
• On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
• Train longer, generalize better: closing the generalization gap in large batch training of neural networks
• Deep Networks with Stochastic Depth
• Beyond Finite Layer Neural Networks: Bridging Deep Architectures and Numerical Differential Equations
• Entropy-SGD: Biasing Gradient Descent Into Wide Valleys
• Multi-level Residual Networks from Dynamical Systems View
• Deep Residual Learning and PDEs on Manifold
• Laplacian Smoothing Gradient Descent
• On the diffusion approximation of nonconvex stochastic gradient descent
• Learning across scales - A multiscale method for Convolution Neural Networks

Cited by in corpus (10)

• A Review on Deep Learning in Medical Image Reconstruction
• Neural SDE: Stabilizing Neural ODE Networks with Stochastic Noise
• ResNets Ensemble via the Feynman-Kac Formalism to Improve Natural and Robust Accuracies
• Sinkformers: Transformers with Doubly Stochastic Attention
• Momentum Residual Neural Networks
• Discovery of Dynamics Using Linear Multistep Methods
• Understanding Recurrent Neural Networks Using Nonequilibrium Response Theory
• Noisy Recurrent Neural Networks
• A Differential Game Theoretic Neural Optimizer for Training Residual Networks
• Neural Delay Differential Equations