DANTE: Deep AlterNations for Training nEural networks

Abstract

We present DANTE, a novel method for training neural networks using the alternating minimization principle. DANTE provides an alternate perspective to traditional gradient-based backpropagation techniques commonly used to train deep networks. It utilizes an adaptation of quasi-convexity to cast training a neural network as a bi-quasi-convex optimization problem. We show that for neural network configurations with both differentiable (e.g. sigmoid) and non-differentiable (e.g. ReLU) activation functions, we can perform the alternations effectively in this formulation. DANTE can also be extended to networks with multiple hidden layers. In experiments on standard datasets, neural networks trained using the proposed method were found to be promising and competitive to traditional backpropagation techniques, both in terms of quality of the solution, as well as training speed.

arXiv comment19 pages

References in corpus (14)

• Identifying and attacking the saddle point problem in high-dimensional non-convex optimization
• Towards Biologically Plausible Deep Learning
• Direct Feedback Alignment Provides Learning in Deep Neural Networks
• Training Neural Networks Without Gradients: A Scalable ADMM Approach
• Beating the Perils of Non-Convexity: Guaranteed Training of Neural Networks using Tensor Methods
• Recovery Guarantees for One-hidden-layer Neural Networks
• How Auto-Encoders Could Provide Credit Assignment in Deep Networks via Target Propagation
• Efficient approaches for escaping higher order saddle points in non-convex optimization
• Random feedback weights support learning in deep neural networks
• Decoupled Neural Interfaces using Synthetic Gradients
• Beyond Convexity: Stochastic Quasi-Convex Optimization
• Beyond Backprop: Online Alternating Minimization with Auxiliary Variables
• A Provably Correct Algorithm for Deep Learning that Actually Works
• Learning ReLU Networks via Alternating Minimization