Understanding Synthetic Gradients and Decoupled Neural Interfaces

Abstract

When training neural networks, the use of Synthetic Gradients (SG) allows layers or modules to be trained without update locking - without waiting for a true error gradient to be backpropagated - resulting in Decoupled Neural Interfaces (DNIs). This unlocked ability of being able to update parts of a neural network asynchronously and with only local information was demonstrated to work empirically in Jaderberg et al (2016). However, there has been very little demonstration of what changes DNIs and SGs impose from a functional, representational, and learning dynamics point of view. In this paper, we study DNIs through the use of synthetic gradients on feed-forward networks to better understand their behaviour and elucidate their effect on optimisation. We show that the incorporation of SGs does not affect the representational strength of the learning system for a neural network, and prove the convergence of the learning system for linear and deep linear models. On practical problems we investigate the mechanism by which synthetic gradient estimators approximate the true loss, and, surprisingly, how that leads to drastically different layer-wise representations. Finally, we also expose the relationship of using synthetic gradients to other error approximation techniques and find a unifying language for discussion and comparison.

References in corpus (6)

• Adam: A Method for Stochastic Optimization
• TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
• Towards Biologically Plausible Deep Learning
• Direct Feedback Alignment Provides Learning in Deep Neural Networks
• Decoupled Neural Interfaces using Synthetic Gradients
• Learning in the Machine: Random Backpropagation and the Deep Learning Channel

Cited by in corpus (13)

• Neuromorphic Deep Learning Machines
• Adaptive Extreme Edge Computing for Wearable Devices
• Bottom-up and top-down approaches for the design of neuromorphic processing systems: Tradeoffs and synergies between natural and artificial intelligence
• Learning without feedback: Fixed random learning signals allow for feedforward training of deep neural networks
• Biologically inspired alternatives to backpropagation through time for learning in recurrent neural nets
• Learning to solve the credit assignment problem
• SoftHebb: Bayesian Inference in Unsupervised Hebbian Soft Winner-Take-All Networks
• ETLP: Event-based Three-factor Local Plasticity for online learning with neuromorphic hardware
• On Training Implicit Models
• Low-pass Recurrent Neural Networks - A memory architecture for longer-term correlation discovery
• Perceptual Gradient Networks
• Alternating Synthetic and Real Gradients for Neural Language Modeling
• Network Learning with Local Propagation