Block-Cyclic Stochastic Coordinate Descent for Deep Neural Networks
arXiv:1711.07190
Abstract
We present a stochastic first-order optimization algorithm, named BCSC, that adds a cyclic constraint to stochastic block-coordinate descent. It uses different subsets of the data to update different subsets of the parameters, thus limiting the detrimental effect of outliers in the training set. Empirical tests in benchmark datasets show that our algorithm outperforms state-of-the-art optimization methods in both accuracy as well as convergence speed. The improvements are consistent across different architectures, and can be combined with other training techniques and regularization methods.
arXiv comment10 pages
References in corpus (12)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
- Improving neural networks by preventing co-adaptation of feature detectors
- ADADELTA: An Adaptive Learning Rate Method
- Deep Residual Learning for Image Recognition
- Fully Convolutional Networks for Semantic Segmentation
- A Stochastic Gradient Method with an Exponential Convergence Rate for Finite Training Sets
- Train longer, generalize better: closing the generalization gap in large batch training of neural networks
- Aggregated Residual Transformations for Deep Neural Networks
- Adding noise to the input of a model trained with a regularized objective
- Randomized Block Coordinate Descent for Online and Stochastic Optimization
- On the Finite Time Convergence of Cyclic Coordinate Descent Methods