Convergent Block Coordinate Descent for Training Tikhonov Regularized Deep Neural Networks
arXiv:1711.07354
Abstract
By lifting the ReLU function into a higher dimensional space, we develop a smooth multi-convex formulation for training feed-forward deep neural networks (DNNs). This allows us to develop a block coordinate descent (BCD) training algorithm consisting of a sequence of numerically well-behaved convex optimizations. Using ideas from proximal point methods in convex analysis, we prove that this BCD algorithm will converge globally to a stationary point with R-linear convergence rate of order one. In experiments with the MNIST database, DNNs trained with this BCD algorithm consistently yielded better test-set error rates than identical DNN architectures trained via all the stochastic gradient descent (SGD) variants in the Caffe toolbox.
arXiv commentNIPS 2017
References in corpus (10)
- Adam: A Method for Stochastic Optimization
- ADADELTA: An Adaptive Learning Rate Method
- Caffe: Convolutional Architecture for Fast Feature Embedding
- The Loss Surfaces of Multilayer Networks
- Identifying and attacking the saddle point problem in high-dimensional non-convex optimization
- Global Convergence of ADMM in Nonconvex Nonsmooth Optimization
- A General Analysis of the Convergence of ADMM
- Subdominant Dense Clusters Allow for Simple Learning and High Computational Performance in Neural Networks with Discrete Synapses
- How Auto-Encoders Could Provide Credit Assignment in Deep Networks via Target Propagation
- Entropy-SGD: Biasing Gradient Descent Into Wide Valleys
Cited by in corpus (5)
- Co-Designing Statistical MIMO Radar and In-band Full-Duplex Multi-User MIMO Communications -- Part I: Signal Processing
- Neural Network Training as an Optimal Control Problem: An Augmented Lagrangian Approach
- LocoProp: Enhancing BackProp via Local Loss Optimization
- Lifted Regression/Reconstruction Networks
- BPGrad: Towards Global Optimality in Deep Learning via Branch and Pruning