Distributed optimization of deeply nested systems

Abstract

In science and engineering, intelligent processing of complex signals such as images, sound or language is often performed by a parameterized hierarchy of nonlinear processing layers, sometimes biologically inspired. Hierarchical systems (or, more generally, nested systems) offer a way to generate complex mappings using simple stages. Each layer performs a different operation and achieves an ever more sophisticated representation of the input, as, for example, in an deep artificial neural network, an object recognition cascade in computer vision or a speech front-end processing. Joint estimation of the parameters of all the layers and selection of an optimal architecture is widely considered to be a difficult numerical nonconvex optimization problem, difficult to parallelize for execution in a distributed computation environment, and requiring significant human expert effort, which leads to suboptimal systems in practice. We describe a general mathematical strategy to learn the parameters and, to some extent, the architecture of nested systems, called the method of auxiliary coordinates (MAC). This replaces the original problem involving a deeply nested function with a constrained problem involving a different function in an augmented space without nesting. The constrained problem may be solved with penalty-based methods using alternating optimization over the parameters and the auxiliary coordinates. MAC has provable convergence, is easy to implement reusing existing algorithms for single layers, can be parallelized trivially and massively, applies even when parameter derivatives are not available or not desirable, and is competitive with state-of-the-art nonlinear optimizers even in the serial computation setting, often providing reasonable models within a few iterations.

21 pages, 7 figures

References in corpus (1)

• Fast Inference in Sparse Coding Algorithms with Applications to Object Recognition

Cited by in corpus (29)

• A practical tutorial on autoencoders for nonlinear feature fusion: Taxonomy, models, software and guidelines
• Training Neural Networks Without Gradients: A Scalable ADMM Approach
• Ensemble Kalman Inversion: A Derivative-Free Technique For Machine Learning Tasks
• Deep Learning: A Bayesian Perspective
• Decoupled Neural Interfaces using Synthetic Gradients
• Training Neural Networks Using Features Replay
• Training Deep Networks with Structured Layers by Matrix Backpropagation
• Hashing with binary autoencoders
• Model compression as constrained optimization, with application to neural nets. Part I: general framework
• Fenchel Lifted Networks: A Lagrange Relaxation of Neural Network Training
• SNODE: Spectral Discretization of Neural ODEs for System Identification
• Optimizing affinity-based binary hashing using auxiliary coordinates
• Lifted Neural Networks
• Predict Globally, Correct Locally: Parallel-in-Time Optimal Control of Neural Networks
• Proximal Backpropagation
• Efficient Training of Very Deep Neural Networks for Supervised Hashing
• A Lagrangian Approach to Information Propagation in Graph Neural Networks
• Neural Network Training as an Optimal Control Problem: An Augmented Lagrangian Approach
• Local Propagation in Constraint-based Neural Network
• Large-Scale Gradient-Free Deep Learning with Recursive Local Representation Alignment
• Recurrent Neural Network Postfilters for Statistical Parametric Speech Synthesis
• ParMAC: distributed optimisation of nested functions, with application to learning binary autoencoders
• Layer-wise Conditioning Analysis in Exploring the Learning Dynamics of DNNs
• Joint Parameter-and-Bandwidth Allocation for Improving the Efficiency of Partitioned Edge Learning
• Contrastive Learning for Lifted Networks
• Deep Constraint-based Propagation in Graph Neural Networks
• Training Deep Neural Networks via Optimization Over Graphs
• Deep Learning in Target Space
• Distributed Dual Coordinate Ascent in General Tree Networks and Communication Network Effect on Synchronous Machine Learning