A Survey on Green Deep Learning

Abstract

In recent years, larger and deeper models are springing up and continuously pushing state-of-the-art (SOTA) results across various fields like natural language processing (NLP) and computer vision (CV). However, despite promising results, it needs to be noted that the computations required by SOTA models have been increased at an exponential rate. Massive computations not only have a surprisingly large carbon footprint but also have negative effects on research inclusiveness and deployment on real-world applications. Green deep learning is an increasingly hot research field that appeals to researchers to pay attention to energy usage and carbon emission during model training and inference. The target is to yield novel results with lightweight and efficient technologies. Many technologies can be used to achieve this goal, like model compression and knowledge distillation. This paper focuses on presenting a systematic review of the development of Green deep learning technologies. We classify these approaches into four categories: (1) compact networks, (2) energy-efficient training strategies, (3) energy-efficient inference approaches, and (4) efficient data usage. For each category, we discuss the progress that has been achieved and the unresolved challenges.

References in corpus (120)

• Efficient Estimation of Word Representations in Vector Space
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Distributed Representations of Words and Phrases and their Compositionality
• Distilling the Knowledge in a Neural Network
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Natural Language Processing (almost) from Scratch
• How transferable are features in deep neural networks?
• Bootstrap your own latent: A new approach to self-supervised Learning
• Conditional Random Fields as Recurrent Neural Networks
• Improved Baselines with Momentum Contrastive Learning
• Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• Neural Architecture Search: A Survey
• Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results
• Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer
• Pre-trained Models for Natural Language Processing: A Survey
• Do Deep Nets Really Need to be Deep?
• Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
• Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
• DARTS: Differentiable Architecture Search
• Pruning Convolutional Neural Networks for Resource Efficient Inference
• AMC: AutoML for Model Compression and Acceleration on Mobile Devices
• Unsupervised Domain Adaptation with Residual Transfer Networks
• Compressing Deep Convolutional Networks using Vector Quantization
• Character-Aware Neural Language Models
• Rethinking the Value of Network Pruning
• Exploring the Limits of Language Modeling
• ERNIE: Enhanced Representation through Knowledge Integration
• DenseNet: Implementing Efficient ConvNet Descriptor Pyramids
• Multilingual Denoising Pre-training for Neural Machine Translation
• TernausNet: U-Net with VGG11 Encoder Pre-Trained on ImageNet for Image Segmentation
• SNIP: Single-shot Network Pruning based on Connection Sensitivity
• Training Deep Nets with Sublinear Memory Cost
• Layer Normalization
• Generating Long Sequences with Sparse Transformers
• Neural Architecture Optimization
• Beyond English-Centric Multilingual Machine Translation
• Hyper-Parameter Optimization: A Review of Algorithms and Applications
• From Softmax to Sparsemax: A Sparse Model of Attention and Multi-Label Classification
• Massively Multitask Networks for Drug Discovery
• Universal Transformers
• Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
• Knowledge Distillation by On-the-Fly Native Ensemble
• Adaptive Computation Time for Recurrent Neural Networks
• Like What You Like: Knowledge Distill via Neuron Selectivity Transfer
• Training and Inference with Integers in Deep Neural Networks
• Reformer: The Efficient Transformer
• ReasoNet: Learning to Stop Reading in Machine Comprehension
• Reducing Transformer Depth on Demand with Structured Dropout
• Toward Multilingual Neural Machine Translation with Universal Encoder and Decoder
• Understanding and Robustifying Differentiable Architecture Search
• Deep Equilibrium Models
• Multi-Scale Dense Networks for Resource Efficient Image Classification
• CLEAR: Contrastive Learning for Sentence Representation
• Transfer Learning for Sequence Tagging with Hierarchical Recurrent Networks
• Flattened Convolutional Neural Networks for Feedforward Acceleration
• A Survey on Methods and Theories of Quantized Neural Networks
• The Evolved Transformer
• The Reversible Residual Network: Backpropagation Without Storing Activations
• Understanding and Improving Layer Normalization
• Adversarial Active Learning for Deep Networks: a Margin Based Approach
• On weight initialization in deep neural networks
• Empower Sequence Labeling with Task-Aware Neural Language Model
• Exploring Sparsity in Recurrent Neural Networks
• ImageNet pre-trained models with batch normalization
• Quantifying the Carbon Emissions of Machine Learning
• Multilingual Neural Machine Translation with Knowledge Distillation
• Pre-Trained Image Processing Transformer
• Rethinking Attention with Performers
• StructBERT: Incorporating Language Structures into Pre-training for Deep Language Understanding
• A Tensorized Transformer for Language Modeling
• Block-Sparse Recurrent Neural Networks
• Compositional Embeddings Using Complementary Partitions for Memory-Efficient Recommendation Systems
• Memory-Efficient Implementation of DenseNets
• Knowledge Transfer with Jacobian Matching
• Feature-map-level Online Adversarial Knowledge Distillation
• A Regularized Framework for Sparse and Structured Neural Attention
• Multilingual Neural Machine Translation with Task-Specific Attention
• Structured Pruning of a BERT-based Question Answering Model
• Finetuned Language Models Are Zero-Shot Learners
• Zero-Cost Proxies for Lightweight NAS
• Effective Quantization Methods for Recurrent Neural Networks
• Efficient 8-Bit Quantization of Transformer Neural Machine Language Translation Model
• Pruning and Quantization for Deep Neural Network Acceleration: A Survey
• Recurrent Neural Networks With Limited Numerical Precision
• M6: A Chinese Multimodal Pretrainer
• Learning Student-Friendly Teacher Networks for Knowledge Distillation
• Pre-training Multilingual Neural Machine Translation by Leveraging Alignment Information
• BinaryBERT: Pushing the Limit of BERT Quantization
• Neural Architecture Search without Training
• Knowledge Adaptation: Teaching to Adapt
• Dynamic Neural Networks: A Survey
• Noisy Differentiable Architecture Search
• Neural Networks Compression for Language Modeling
• Variable Computation in Recurrent Neural Networks
• Strategies for Training Large Vocabulary Neural Language Models
• Learning Accurate Low-Bit Deep Neural Networks with Stochastic Quantization
• On the Effectiveness of Low-Rank Matrix Factorization for LSTM Model Compression
• Learning Implicitly Recurrent CNNs Through Parameter Sharing
• Word2Bits - Quantized Word Vectors
• Lessons on Parameter Sharing across Layers in Transformers
• Parameter-Efficient Transfer Learning with Diff Pruning
• FEED: Feature-level Ensemble for Knowledge Distillation
• Reversible Recurrent Neural Networks
• Knowledge Flow: Improve Upon Your Teachers
• Learned Low Precision Graph Neural Networks
• Knowledge Projection for Deep Neural Networks
• Layer-wise training of deep networks using kernel similarity
• CAPT: Contrastive Pre-Training for Learning Denoised Sequence Representations
• Early Exiting with Ensemble Internal Classifiers
• Does Knowledge Distillation Really Work?
• Compressing Neural Language Models by Sparse Word Representations
• TernaryBERT: Distillation-aware Ultra-low Bit BERT
• Progressively Stacking 2.0: A Multi-stage Layerwise Training Method for BERT Training Speedup
• RomeBERT: Robust Training of Multi-Exit BERT
• The OoO VLIW JIT Compiler for GPU Inference
• PydMobileNet: Improved Version of MobileNets with Pyramid Depthwise Separable Convolution
• KNAS: Green Neural Architecture Search
• Neural Parameter Allocation Search