Global Convergence of Block Coordinate Descent in Deep Learning

Abstract

Deep learning has aroused extensive attention due to its great empirical success. The efficiency of the block coordinate descent (BCD) methods has been recently demonstrated in deep neural network (DNN) training. However, theoretical studies on their convergence properties are limited due to the highly nonconvex nature of DNN training. In this paper, we aim at providing a general methodology for provable convergence guarantees for this type of methods. In particular, for most of the commonly used DNN training models involving both two- and three-splitting schemes, we establish the global convergence to a critical point at a rate of $\mathcal{O}(1/k)$, where $k$ is the number of iterations. The results extend to general loss functions which have Lipschitz continuous gradients and deep residual networks (ResNets). Our key development adds several new elements to the Kurdyka-Łojasiewicz inequality framework that enables us to carry out the global convergence analysis of BCD in the general scenario of deep learning.

arXiv comment27 pages, 2 figures

Cited by in corpus (11)

• Optimization for deep learning: theory and algorithms
• Communication-Efficient Distributed Deep Learning: A Comprehensive Survey
• Predict Globally, Correct Locally: Parallel-in-Time Optimal Control of Neural Networks
• Hybrid Federated Learning: Algorithms and Implementation
• Convergence Analysis of a Momentum Algorithm with Adaptive Step Size for Non Convex Optimization
• Task-Feature Collaborative Learning with Application to Personalized Attribute Prediction
• LocoProp: Enhancing BackProp via Local Loss Optimization
• Learning DNN networks using un-rectifying ReLU with compressed sensing application
• Inertial Proximal Deep Learning Alternating Minimization for Efficient Neutral Network Training
• DessiLBI: Exploring Structural Sparsity of Deep Networks via Differential Inclusion Paths
• Communication Efficient Generalized Tensor Factorization for Decentralized Healthcare Networks