Training Neural Networks Without Gradients: A Scalable ADMM Approach

Abstract

With the growing importance of large network models and enormous training datasets, GPUs have become increasingly necessary to train neural networks. This is largely because conventional optimization algorithms rely on stochastic gradient methods that don't scale well to large numbers of cores in a cluster setting. Furthermore, the convergence of all gradient methods, including batch methods, suffers from common problems like saturation effects, poor conditioning, and saddle points. This paper explores an unconventional training method that uses alternating direction methods and Bregman iteration to train networks without gradient descent steps. The proposed method reduces the network training problem to a sequence of minimization sub-steps that can each be solved globally in closed form. The proposed method is advantageous because it avoids many of the caveats that make gradient methods slow on highly non-convex problems. The method exhibits strong scaling in the distributed setting, yielding linear speedups even when split over thousands of cores.

References in corpus (5)

• Searching for Exotic Particles in High-Energy Physics with Deep Learning
• The Loss Surfaces of Multilayer Networks
• Identifying and attacking the saddle point problem in high-dimensional non-convex optimization
• A Probabilistic Theory of Deep Learning
• Distributed optimization of deeply nested systems

Cited by in corpus (54)

• Ensemble Kalman Inversion: A Derivative-Free Technique For Machine Learning Tasks
• You Only Propagate Once: Accelerating Adversarial Training via Maximal Principle
• Decoupled Neural Interfaces using Synthetic Gradients
• Training Neural Networks Using Features Replay
• ADMM for Efficient Deep Learning with Global Convergence
• Adaptive Consensus ADMM for Distributed Optimization
• Revisiting Locally Supervised Learning: an Alternative to End-to-end Training
• Deep Frank-Wolfe For Neural Network Optimization
• Fenchel Lifted Networks: A Lagrange Relaxation of Neural Network Training
• Local Critic Training for Model-Parallel Learning of Deep Neural Networks
• Deriving Differential Target Propagation from Iterating Approximate Inverses
• Generalized gradient optimization over lossy networks for partition-based estimation
• A Probabilistic Framework for Deep Learning
• Memorized Sparse Backpropagation
• On Training Implicit Models
• Alternating Direction Method of Multipliers-Based Parallel Optimization for Multi-Agent Collision-Free Model Predictive Control
• Proximal Backpropagation
• Fast Polynomial Kernel Classification for Massive Data
• A Lagrangian Approach to Information Propagation in Graph Neural Networks
• DANTE: Deep AlterNations for Training nEural networks
• A Block-wise, Asynchronous and Distributed ADMM Algorithm for General Form Consensus Optimization
• Differentially Private ADMM for Distributed Medical Machine Learning
• Local Propagation in Constraint-based Neural Network
• Large-Scale Gradient-Free Deep Learning with Recursive Local Representation Alignment
• Nonconvex Zeroth-Order Stochastic ADMM Methods with Lower Function Query Complexity
• Convergence of a Relaxed Variable Splitting Method for Learning Sparse Neural Networks via ${\ell }_1,{\ell }_0$, and transformed-${\ell }_1$ Penalties
• ADMM for Multiaffine Constrained Optimization
• RSO: A Gradient Free Sampling Based Approach For Training Deep Neural Networks
• Asynchronous Decentralized Learning of a Neural Network
• Social Discrete Choice Models
• LocoProp: Enhancing BackProp via Local Loss Optimization
• Toward Model Parallelism for Deep Neural Network based on Gradient-free ADMM Framework
• Meta-Learning Bidirectional Update Rules
• A Practical Layer-Parallel Training Algorithm for Residual Networks
• A Low Complexity Decentralized Neural Net with Centralized Equivalence using Layer-wise Learning
• Stochastic Block-ADMM for Training Deep Networks
• Deep Constraint-based Propagation in Graph Neural Networks
• Learning DNN networks using un-rectifying ReLU with compressed sensing application
• Contrastive Learning for Lifted Networks
• Robust Trust Region for Weakly Supervised Segmentation
• Developing Constrained Neural Units Over Time
• Beyond Gradient Descent for Regularized Segmentation Losses
• Inertial Proximal Deep Learning Alternating Minimization for Efficient Neutral Network Training
• Deep Convolutional Transform Learning -- Extended version
• Network Learning with Local Propagation
• Towards Quantized Model Parallelism for Graph-Augmented MLPs Based on Gradient-Free ADMM Framework
• Convergence of a Relaxed Variable Splitting Coarse Gradient Descent Method for Learning Sparse Weight Binarized Activation Neural Networks
• Semi-Implicit Back Propagation
• Target Propagation via Regularized Inversion
• Zeroth-Order Stochastic Alternating Direction Method of Multipliers for Nonconvex Nonsmooth Optimization
• Learning and Optimization of Blackbox Combinatorial Solvers in Neural Networks
• Managing Randomization in the Multi-Block Alternating Direction Method of Multipliers for Quadratic Optimization
• Associated Learning: Decomposing End-to-end Backpropagation based on Auto-encoders and Target Propagation
• Deep Learning in Target Space