Conducting Credit Assignment by Aligning Local Representations

Abstract

Using back-propagation and its variants to train deep networks is often problematic for new users. Issues such as exploding gradients, vanishing gradients, and high sensitivity to weight initialization strategies often make networks difficult to train, especially when users are experimenting with new architectures. Here, we present Local Representation Alignment (LRA), a training procedure that is much less sensitive to bad initializations, does not require modifications to the network architecture, and can be adapted to networks with highly nonlinear and discrete-valued activation functions. Furthermore, we show that one variation of LRA can start with a null initialization of network weights and still successfully train networks with a wide variety of nonlinearities, including tanh, ReLU-6, softplus, signum and others that may draw their inspiration from biology. A comprehensive set of experiments on MNIST and the much harder Fashion MNIST data sets show that LRA can be used to train networks robustly and effectively, succeeding even when back-propagation fails and outperforming other alternative learning algorithms, such as target propagation and feedback alignment.

Full document revision/overhaul, new results/analyses, new diagrams, addition of appendices

References in corpus (13)

• Adam: A Method for Stochastic Optimization
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• On the difficulty of training Recurrent Neural Networks
• FitNets: Hints for Thin Deep Nets
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• Deeply-Supervised Nets
• All you need is a good init
• Direct Feedback Alignment Provides Learning in Deep Neural Networks
• How Auto-Encoders Could Provide Credit Assignment in Deep Networks via Target Propagation
• Decoupled Neural Interfaces using Synthetic Gradients
• How Important is Weight Symmetry in Backpropagation?
• Online Semi-Supervised Learning with Deep Hybrid Boltzmann Machines and Denoising Autoencoders
• Learning to Adapt by Minimizing Discrepancy

Cited by in corpus (13)

• Unsupervised Learning by Competing Hidden Units
• Putting An End to End-to-End: Gradient-Isolated Learning of Representations
• Learning to solve the credit assignment problem
• Activation Relaxation: A Local Dynamical Approximation to Backpropagation in the Brain
• Spiking Neural Predictive Coding for Continual Learning from Data Streams
• Lifelong Neural Predictive Coding: Learning Cumulatively Online without Forgetting
• Gradient target propagation
• Large-Scale Gradient-Free Deep Learning with Recursive Local Representation Alignment
• Ghost Units Yield Biologically Plausible Backprop in Deep Neural Networks
• Contrastive Similarity Matching for Supervised Learning
• InferNet for Delayed Reinforcement Tasks: Addressing the Temporal Credit Assignment Problem
• Associated Learning: Decomposing End-to-end Backpropagation based on Auto-encoders and Target Propagation
• Front Contribution instead of Back Propagation