A Proximal Block Coordinate Descent Algorithm for Deep Neural Network Training
arXiv:1803.09082
Abstract
Training deep neural networks (DNNs) efficiently is a challenge due to the associated highly nonconvex optimization. The backpropagation (backprop) algorithm has long been the most widely used algorithm for gradient computation of parameters of DNNs and is used along with gradient descent-type algorithms for this optimization task. Recent work have shown the efficiency of block coordinate descent (BCD) type methods empirically for training DNNs. In view of this, we propose a novel algorithm based on the BCD method for training DNNs and provide its global convergence results built upon the powerful framework of the Kurdyka-Lojasiewicz (KL) property. Numerical experiments on standard datasets demonstrate its competitive efficiency against standard optimizers with backprop.
arXiv commentThe 6th International Conference on Learning Representations (ICLR 2018), Workshop Track
Cited by in corpus (5)
- Communication-Efficient Distributed Deep Learning: A Comprehensive Survey
- Fenchel Lifted Networks: A Lagrange Relaxation of Neural Network Training
- Predict Globally, Correct Locally: Parallel-in-Time Optimal Control of Neural Networks
- Task-Feature Collaborative Learning with Application to Personalized Attribute Prediction
- Learning DNN networks using un-rectifying ReLU with compressed sensing application