Beyond Backprop: Online Alternating Minimization with Auxiliary Variables

Abstract

Despite significant recent advances in deep neural networks, training them remains a challenge due to the highly non-convex nature of the objective function. State-of-the-art methods rely on error backpropagation, which suffers from several well-known issues, such as vanishing and exploding gradients, inability to handle non-differentiable nonlinearities and to parallelize weight-updates across layers, and biological implausibility. These limitations continue to motivate exploration of alternative training algorithms, including several recently proposed auxiliary-variable methods which break the complex nested objective function into local subproblems. However, those techniques are mainly offline (batch), which limits their applicability to extremely large datasets, as well as to online, continual or reinforcement learning. The main contribution of our work is a novel online (stochastic/mini-batch) alternating minimization (AM) approach for training deep neural networks, together with the first theoretical convergence guarantees for AM in stochastic settings and promising empirical results on a variety of architectures and datasets.

arXiv commentFirst six authors contributed equally to this work: A.C. - theory, manuscript, B.C. - code, experiments, S.K. - code, experiments, R.L. - algorithm, experiments, M.R. - code, experiments, I.R. - algorithm, manuscript

Cited by in corpus (16)

• Drawing Inspiration from Biological Dendrites to Empower Artificial Neural Networks
• Deriving Differential Target Propagation from Iterating Approximate Inverses
• Online learning with Corrupted context: Corrupted Contextual Bandits
• Predict Globally, Correct Locally: Parallel-in-Time Optimal Control of Neural Networks
• DANTE: Deep AlterNations for Training nEural networks
• How can AI Automate End-to-End Data Science?
• Contextual Bandit with Missing Rewards
• A Practical Layer-Parallel Training Algorithm for Residual Networks
• Adaptive Subcarrier, Parameter, and Power Allocation for Partitioned Edge Learning Over Broadband Channels
• Computing the Dirichlet-Multinomial Log-Likelihood Function
• Spectral Clustering using Eigenspectrum Shape Based Nystrom Sampling
• Learning DNN networks using un-rectifying ReLU with compressed sensing application
• Edge Artificial Intelligence for 6G: Vision, Enabling Technologies, and Applications
• SGB: Stochastic Gradient Bound Method for Optimizing Partition Functions
• Front Contribution instead of Back Propagation
• Etat de l'art sur l'application des bandits multi-bras