A priori guarantees of finite-time convergence for Deep Neural Networks

Abstract

In this paper, we perform Lyapunov based analysis of the loss function to derive an a priori upper bound on the settling time of deep neural networks. While previous studies have attempted to understand deep learning using control theory framework, there is limited work on a priori finite time convergence analysis. Drawing from the advances in analysis of finite-time control of non-linear systems, we provide a priori guarantees of finite-time convergence in a deterministic control theoretic setting. We formulate the supervised learning framework as a control problem where weights of the network are control inputs and learning translates into a tracking problem. An analytical formula for finite-time upper bound on settling time is computed a priori under the assumptions of boundedness of input. Finally, we prove the robustness and sensitivity of the loss function against input perturbations.

References in corpus (6)

• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
• A Convergence Analysis of Gradient Descent for Deep Linear Neural Networks
• Deep Learning Theory Review: An Optimal Control and Dynamical Systems Perspective
• Finite Time Analysis of Linear Two-timescale Stochastic Approximation with Markovian Noise
• A Multistep Lyapunov Approach for Finite-Time Analysis of Biased Stochastic Approximation