Local Propagation in Constraint-based Neural Network

Abstract

In this paper we study a constraint-based representation of neural network architectures. We cast the learning problem in the Lagrangian framework and we investigate a simple optimization procedure that is well suited to fulfil the so-called architectural constraints, learning from the available supervisions. The computational structure of the proposed Local Propagation (LP) algorithm is based on the search for saddle points in the adjoint space composed of weights, neural outputs, and Lagrange multipliers. All the updates of the model variables are locally performed, so that LP is fully parallelizable over the neural units, circumventing the classic problem of gradient vanishing in deep networks. The implementation of popular neural models is described in the context of LP, together with those conditions that trace a natural connection with Backpropagation. We also investigate the setting in which we tolerate bounded violations of the architectural constraints, and we provide experimental evidence that LP is a feasible approach to train shallow and deep networks, opening the road to further investigations on more complex architectures, easily describable by constraints.

References in corpus (6)

• Deep Learning in Neural Networks: An Overview
• Deep Convolutional Networks on Graph-Structured Data
• Learning Deep Generative Models of Graphs
• Adding Gradient Noise Improves Learning for Very Deep Networks
• Training Neural Networks Without Gradients: A Scalable ADMM Approach
• Distributed optimization of deeply nested systems

Cited by in corpus (3)

• Memorized Sparse Backpropagation
• A Practical Layer-Parallel Training Algorithm for Residual Networks
• Constrained Machine Learning: The Bagel Framework