Convergent Block Coordinate Descent for Training Tikhonov Regularized Deep Neural Networks
arXiv:1711.07354
Abstract
By lifting the ReLU function into a higher dimensional space, we develop a smooth multi-convex formulation for training feed-forward deep neural networks (DNNs). This allows us to develop a block coordinate descent (BCD) training algorithm consisting of a sequence of numerically well-behaved convex optimizations. Using ideas from proximal point methods in convex analysis, we prove that this BCD algorithm will converge globally to a stationary point with R-linear convergence rate of order one. In experiments with the MNIST database, DNNs trained with this BCD algorithm consistently yielded better test-set error rates than identical DNN architectures trained via all the stochastic gradient descent (SGD) variants in the Caffe toolbox.
NIPS 2017
References in corpus (5)
- ADADELTA: An Adaptive Learning Rate Method
- Caffe: Convolutional Architecture for Fast Feature Embedding
- The Loss Surfaces of Multilayer Networks
- Identifying and attacking the saddle point problem in high-dimensional non-convex optimization
- How Auto-Encoders Could Provide Credit Assignment in Deep Networks via Target Propagation