Block-Cyclic Stochastic Coordinate Descent for Deep Neural Networks
arXiv:1711.07190
Abstract
We present a stochastic first-order optimization algorithm, named BCSC, that adds a cyclic constraint to stochastic block-coordinate descent. It uses different subsets of the data to update different subsets of the parameters, thus limiting the detrimental effect of outliers in the training set. Empirical tests in benchmark datasets show that our algorithm outperforms state-of-the-art optimization methods in both accuracy as well as convergence speed. The improvements are consistent across different architectures, and can be combined with other training techniques and regularization methods.
10 pages
References in corpus (6)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Improving neural networks by preventing co-adaptation of feature detectors
- ADADELTA: An Adaptive Learning Rate Method
- A Stochastic Gradient Method with an Exponential Convergence Rate for Finite Training Sets
- Adding noise to the input of a model trained with a regularized objective
- Randomized Block Coordinate Descent for Online and Stochastic Optimization