Revisiting Locally Supervised Learning: an Alternative to End-to-end Training

Abstract

Due to the need to store the intermediate activations for back-propagation, end-to-end (E2E) training of deep networks usually suffers from high GPUs memory footprint. This paper aims to address this problem by revisiting the locally supervised learning, where a network is split into gradient-isolated modules and trained with local supervision. We experimentally show that simply training local modules with E2E loss tends to collapse task-relevant information at early layers, and hence hurts the performance of the full model. To avoid this issue, we propose an information propagation (InfoPro) loss, which encourages local modules to preserve as much useful information as possible, while progressively discard task-irrelevant information. As InfoPro loss is difficult to compute in its original form, we derive a feasible upper bound as a surrogate optimization objective, yielding a simple but effective algorithm. In fact, we show that the proposed method boils down to minimizing the combination of a reconstruction loss and a normal cross-entropy/contrastive term. Extensive empirical results on five datasets (i.e., CIFAR, SVHN, STL-10, ImageNet and Cityscapes) validate that InfoPro is capable of achieving competitive performance with less than 40% memory footprint compared to E2E training, while allowing using training data with higher-resolution or larger batch sizes under the same GPU memory constraint. Our method also enables training local modules asynchronously for potential training acceleration. Code is available at: https://github.com/blackfeather-wang/InfoPro-Pytorch.

arXiv commentAccepted by ICLR 2021

References in corpus (23)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Rethinking Atrous Convolution for Semantic Image Segmentation
• The information bottleneck method
• Learning deep representations by mutual information estimation and maximization
• Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results
• MMDetection: Open MMLab Detection Toolbox and Benchmark
• Opening the Black Box of Deep Neural Networks via Information
• What Makes for Good Views for Contrastive Learning?
• Training Deep Nets with Sublinear Memory Cost
• Deep Variational Information Bottleneck
• Towards Biologically Plausible Deep Learning
• Multi-Scale Dense Networks for Resource Efficient Image Classification
• Training Neural Networks Without Gradients: A Scalable ADMM Approach
• Training Neural Networks with Local Error Signals
• Supervised Contrastive Learning
• Assessing the Scalability of Biologically-Motivated Deep Learning Algorithms and Architectures
• Random feedback weights support learning in deep neural networks
• Implicit Semantic Data Augmentation for Deep Networks
• A Provably Correct Algorithm for Deep Learning that Actually Works
• Layer-wise training of deep networks using kernel similarity
• Decoupled Greedy Learning of CNNs

Cited by in corpus (4)

• Regularizing Deep Networks with Semantic Data Augmentation
• Multi-Scale and Multi-Layer Contrastive Learning for Domain Generalization
• GRAM: Fast Fine-tuning of Pre-trained Language Models for Content-based Collaborative Filtering
• Adaptive Focus for Efficient Video Recognition