Putting An End to End-to-End: Gradient-Isolated Learning of Representations

Abstract

We propose a novel deep learning method for local self-supervised representation learning that does not require labels nor end-to-end backpropagation but exploits the natural order in data instead. Inspired by the observation that biological neural networks appear to learn without backpropagating a global error signal, we split a deep neural network into a stack of gradient-isolated modules. Each module is trained to maximally preserve the information of its inputs using the InfoNCE bound from Oord et al. [2018]. Despite this greedy training, we demonstrate that each module improves upon the output of its predecessor, and that the representations created by the top module yield highly competitive results on downstream classification tasks in the audio and visual domain. The proposal enables optimizing modules asynchronously, allowing large-scale distributed training of very deep neural networks on unlabelled datasets.

Honorable Mention for Outstanding New Directions Paper Award at NeurIPS 2019

References in corpus (21)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Distributed Representations of Words and Phrases and their Compositionality
• Representation Learning with Contrastive Predictive Coding
• Deep Residual Learning for Image Recognition
• Improved Regularization of Convolutional Neural Networks with Cutout
• Learning deep representations by mutual information estimation and maximization
• A Review of Relational Machine Learning for Knowledge Graphs
• Data-Efficient Image Recognition with Contrastive Predictive Coding
• Variational Graph Auto-Encoders
• Towards Biologically Plausible Deep Learning
• Learning Discrete Representations via Information Maximizing Self-Augmented Training
• Unsupervised Feature Extraction by Time-Contrastive Learning and Nonlinear ICA
• Training Neural Networks with Local Error Signals
• i-RevNet: Deep Invertible Networks
• Nonlinear ICA Using Auxiliary Variables and Generalized Contrastive Learning
• Biologically-plausible learning algorithms can scale to large datasets
• Auto-Encoding Total Correlation Explanation
• Conducting Credit Assignment by Aligning Local Representations
• Error Forward-Propagation: Reusing Feedforward Connections to Propagate Errors in Deep Learning
• Information Theoretic Co-Training

Cited by in corpus (20)

• Contrastive Representation Learning: A Framework and Review
• Hard Negative Mixing for Contrastive Learning
• Data Augmenting Contrastive Learning of Speech Representations in the Time Domain
• A Good Image Generator Is What You Need for High-Resolution Video Synthesis
• LoCo: Local Contrastive Representation Learning
• Kernelized information bottleneck leads to biologically plausible 3-factor Hebbian learning in deep networks
• Local plasticity rules can learn deep representations using self-supervised contrastive predictions
• A General Framework for Interpretable Neural Learning based on Local Information-Theoretic Goal Functions
• Parallel Training of Deep Networks with Local Updates
• Language-Driven Image Style Transfer
• Biologically Plausible Training Mechanisms for Self-Supervised Learning in Deep Networks
• Greedy Hierarchical Variational Autoencoders for Large-Scale Video Prediction
• Gaussian Gated Linear Networks
• Direct Feedback Alignment Scales to Modern Deep Learning Tasks and Architectures
• Interlocking Backpropagation: Improving depthwise model-parallelism
• Meta-Learning Bidirectional Update Rules
• Discriminative Clustering with Representation Learning with any Ratio of Labeled to Unlabeled Data
• ConFuse: Convolutional Transform Learning Fusion Framework For Multi-Channel Data Analysis
• Self Normalizing Flows
• Think Global, Act Local: Relating DNN generalisation and node-level SNR