One weird trick for parallelizing convolutional neural networks

Abstract

I present a new way to parallelize the training of convolutional neural networks across multiple GPUs. The method scales significantly better than all alternatives when applied to modern convolutional neural networks.

References in corpus (1)

• GPU Asynchronous Stochastic Gradient Descent to Speed Up Neural Network Training

Cited by in corpus (244)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
• TensorFlow: A system for large-scale machine learning
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
• Toward Multimodal Image-to-Image Translation
• The Unreasonable Effectiveness of Deep Features as a Perceptual Metric
• Improving Computer-aided Detection using Convolutional Neural Networks and Random View Aggregation
• Don't Decay the Learning Rate, Increase the Batch Size
• Communication-Efficient Federated Deep Learning with Asynchronous Model Update and Temporally Weighted Aggregation
• Large Batch Training of Convolutional Networks
• Distributed Prioritized Experience Replay
• Deep Learning is Robust to Massive Label Noise
• Train longer, generalize better: closing the generalization gap in large batch training of neural networks
• Revisiting Small Batch Training for Deep Neural Networks
• Deep Image: Scaling up Image Recognition
• Systematic evaluation of CNN advances on the ImageNet
• Deep Roots: Improving CNN Efficiency with Hierarchical Filter Groups
• Training Tips for the Transformer Model
• GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism
• ReNet: A Recurrent Neural Network Based Alternative to Convolutional Networks
• Three Factors Influencing Minima in SGD
• Deep Residual Learning in Spiking Neural Networks
• Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis
• Scalable Methods for 8-bit Training of Neural Networks
• Anatomy-specific classification of medical images using deep convolutional nets
• Deep convolutional networks for pancreas segmentation in CT imaging
• MLPerf Training Benchmark
• Large Batch Optimization for Deep Learning: Training BERT in 76 minutes
• Deep Learning on FPGAs: Past, Present, and Future
• Supporting Very Large Models using Automatic Dataflow Graph Partitioning
• Beyond Data and Model Parallelism for Deep Neural Networks
• A CNN Approach to Simultaneously Count Plants and Detect Plantation-Rows from UAV Imagery
• Deep Semantic Ranking Based Hashing for Multi-Label Image Retrieval
• Adaptive Federated Optimization
• Tropical Cyclone Track Forecasting using Fused Deep Learning from Aligned Reanalysis Data
• The Origins and Prevalence of Texture Bias in Convolutional Neural Networks
• Chimera: Efficiently Training Large-Scale Neural Networks with Bidirectional Pipelines
• Training Confidence-calibrated Classifiers for Detecting Out-of-Distribution Samples
• Recent Advances in Object Detection in the Age of Deep Convolutional Neural Networks
• Hardware Acceleration of Sparse and Irregular Tensor Computations of ML Models: A Survey and Insights
• 8-Bit Approximations for Parallelism in Deep Learning
• Facilitated machine learning for image-based fruit quality assessment
• Self-labelling via simultaneous clustering and representation learning
• Transfer Learning using CNN for Handwritten Devanagari Character Recognition
• Communication-Efficient Distributed Deep Learning: A Comprehensive Survey
• Partial Connection Based on Channel Attention for Differentiable Neural Architecture Search
• GossipGraD: Scalable Deep Learning using Gossip Communication based Asynchronous Gradient Descent
• NeuralPower: Predict and Deploy Energy-Efficient Convolutional Neural Networks
• Efficient and Robust Parallel DNN Training through Model Parallelism on Multi-GPU Platform
• Understanding Data Storage and Ingestion for Large-Scale Deep Recommendation Model Training
• Fast Algorithms for Convolutional Neural Networks
• Generic 3D Representation via Pose Estimation and Matching
• Trustworthy Privacy-preserving Hierarchical Ensemble and Federated Learning in Healthcare 4.0 with Blockchain
• Revisiting the Calibration of Modern Neural Networks
• Communication optimization strategies for distributed deep neural network training: A survey
• Robust Convolutional Neural Networks under Adversarial Noise
• A System for Massively Parallel Hyperparameter Tuning
• NeST: A Neural Network Synthesis Tool Based on a Grow-and-Prune Paradigm
• Exploring Hidden Dimensions in Parallelizing Convolutional Neural Networks
• Scalable Graph Neural Network Training: The Case for Sampling
• Detecting Dementia from Speech and Transcripts using Transformers
• WasteNet: Waste Classification at the Edge for Smart Bins
• TensorOpt: Exploring the Tradeoffs in Distributed DNN Training with Auto-Parallelism
• ImageNet Training in Minutes
• Deep Temporal Appearance-Geometry Network for Facial Expression Recognition
• Scalable Deep Learning on Distributed Infrastructures: Challenges, Techniques and Tools
• On the Computational Inefficiency of Large Batch Sizes for Stochastic Gradient Descent
• Poseidon: A System Architecture for Efficient GPU-based Deep Learning on Multiple Machines
• Approximate Data Deletion from Machine Learning Models
• Multi-Residual Networks: Improving the Speed and Accuracy of Residual Networks
• Theano-based Large-Scale Visual Recognition with Multiple GPUs
• Biologically-plausible learning algorithms can scale to large datasets
• GPU-accelerated Gibbs sampling: a case study of the Horseshoe Probit model
• The Implicit Regularization of Stochastic Gradient Flow for Least Squares
• DecomposeMe: Simplifying ConvNets for End-to-End Learning
• Online Normalization for Training Neural Networks
• Declarative Recursive Computation on an RDBMS, or, Why You Should Use a Database For Distributed Machine Learning
• E-LPIPS: Robust Perceptual Image Similarity via Random Transformation Ensembles
• TeraPipe: Token-Level Pipeline Parallelism for Training Large-Scale Language Models
• torchgpipe: On-the-fly Pipeline Parallelism for Training Giant Models
• Learning Adaptive Sampling and Reconstruction for Volume Visualization
• Bit Fusion: Bit-Level Dynamically Composable Architecture for Accelerating Deep Neural Networks
• DAPPLE: A Pipelined Data Parallel Approach for Training Large Models
• Convolutional Neural Networks at Constrained Time Cost
• MegDet: A Large Mini-Batch Object Detector
• Multi-objective optimization determines when, which and how to fuse deep networks: an application to predict COVID-19 outcomes
• A Diffusion Theory For Deep Learning Dynamics: Stochastic Gradient Descent Exponentially Favors Flat Minima
• CHAOS: A Parallelization Scheme for Training Convolutional Neural Networks on Intel Xeon Phi
• Pollux: Co-adaptive Cluster Scheduling for Goodput-Optimized Deep Learning
• Training Large Neural Networks with Constant Memory using a New Execution Algorithm
• Dataset Inference: Ownership Resolution in Machine Learning
• Dataset Distillation
• LM-CPPF: Paraphrasing-Guided Data Augmentation for Contrastive Prompt-Based Few-Shot Fine-Tuning
• On the Outsized Importance of Learning Rates in Local Update Methods
• The Evolution of Out-of-Distribution Robustness Throughout Fine-Tuning
• End-to-End Learning of Geometric Deformations of Feature Maps for Virtual Try-On
• LoCo: Local Contrastive Representation Learning
• Multi-path Convolutional Neural Networks for Complex Image Classification
• Automatic Cross-Replica Sharding of Weight Update in Data-Parallel Training
• Dorylus: Affordable, Scalable, and Accurate GNN Training with Distributed CPU Servers and Serverless Threads
• Parle: parallelizing stochastic gradient descent
• Flip-Rotate-Pooling Convolution and Split Dropout on Convolution Neural Networks for Image Classification
• On the Validity of Modeling SGD with Stochastic Differential Equations (SDEs)
• ReCycle: Resilient Training of Large DNNs using Pipeline Adaptation
• Deep Fried Convnets
• Quantization of Deep Neural Networks for Accumulator-constrained Processors
• Decoding Dark Matter Substructure without Supervision
• Fairness Through Robustness: Investigating Robustness Disparity in Deep Learning
• ImAiR: Airwriting Recognition framework using Image Representation of IMU Signals
• Efficient Algorithms for Device Placement of DNN Graph Operators
• ThUnderVolt: Enabling Aggressive Voltage Underscaling and Timing Error Resilience for Energy Efficient Deep Neural Network Accelerators
• Towards Practical Adam: Non-Convexity, Convergence Theory, and Mini-Batch Acceleration
• Exploring the Design Space of Deep Convolutional Neural Networks at Large Scale
• KaoKore: A Pre-modern Japanese Art Facial Expression Dataset
• Enabling Large Batch Size Training for DNN Models Beyond the Memory Limit While Maintaining Performance
• Distributed Learning of Deep Neural Networks using Independent Subnet Training
• HPC AI500: The Methodology, Tools, Roofline Performance Models, and Metrics for Benchmarking HPC AI Systems
• Orchestrating the Development Lifecycle of Machine Learning-Based IoT Applications: A Taxonomy and Survey
• Real-time Universal Style Transfer on High-resolution Images via Zero-channel Pruning
• Continual Learning in Neural Networks
• Unifying Data, Model and Hybrid Parallelism in Deep Learning via Tensor Tiling
• Web-Scale Training for Face Identification
• Large-Batch Training for LSTM and Beyond
• CROSSBOW: Scaling Deep Learning with Small Batch Sizes on Multi-GPU Servers
• Domain-specific Communication Optimization for Distributed DNN Training
• Near-Precise Parameter Approximation for Multiple Multiplications on A Single DSP Block
• HyPar: Towards Hybrid Parallelism for Deep Learning Accelerator Array
• AutoDDL: Automatic Distributed Deep Learning with Near-Optimal Bandwidth Cost
• An Oracle for Guiding Large-Scale Model/Hybrid Parallel Training of Convolutional Neural Networks
• ACE: Ally Complementary Experts for Solving Long-Tailed Recognition in One-Shot
• PaSE: Parallelization Strategies for Efficient DNN Training
• A detailed comparative study of open source deep learning frameworks
• Revisiting Spatial Invariance with Low-Rank Local Connectivity
• Compiling Deep Learning Models for Custom Hardware Accelerators
• Vid2Game: Controllable Characters Extracted from Real-World Videos
• Accelerating DNN Training in Wireless Federated Edge Learning Systems
• Convolution-Weight-Distribution Assumption: Rethinking the Criteria of Channel Pruning
• A Distributed Synchronous SGD Algorithm with Global Top-$k$ Sparsification for Low Bandwidth Networks
• Learning Rates as a Function of Batch Size: A Random Matrix Theory Approach to Neural Network Training
• Distributed Training Large-Scale Deep Architectures
• Diversified Arbitrary Style Transfer via Deep Feature Perturbation
• Large-Scale Deep Learning Optimizations: A Comprehensive Survey
• The Case for Strong Scaling in Deep Learning: Training Large 3D CNNs with Hybrid Parallelism
• Learning Rate Annealing Can Provably Help Generalization, Even for Convex Problems
• Automated Learning Rate Scheduler for Large-batch Training
• Deep CNNs for Peripheral Blood Cell Classification
• Campfire: Compressible, Regularization-Free, Structured Sparse Training for Hardware Accelerators
• Pushing the Limits of Capsule Networks
• Disentangling Adaptive Gradient Methods from Learning Rates
• Large-scale Pretraining for Neural Machine Translation with Tens of Billions of Sentence Pairs
• Towards Testing of Deep Learning Systems with Training Set Reduction
• SpecNet: Spectral Domain Convolutional Neural Network
• Verification of Size Invariance in DNN Activations using Concept Embeddings
• Transfer Learning Study of Motion Transformer-based Trajectory Predictions
• Beyond Human-Level Accuracy: Computational Challenges in Deep Learning
• Multi-node Bert-pretraining: Cost-efficient Approach
• Pruning at a Glance: Global Neural Pruning for Model Compression
• Accelerated Training for CNN Distributed Deep Learning through Automatic Resource-Aware Layer Placement
• Efficient batchwise dropout training using submatrices
• Neural networks for semantic segmentation of historical city maps: Cross-cultural performance and the impact of figurative diversity
• dMath: A Scalable Linear Algebra and Math Library for Heterogeneous GP-GPU Architectures
• Shift Invariance Can Reduce Adversarial Robustness
• Reducing the Training Time of Neural Networks by Partitioning
• Activations and Gradients Compression for Model-Parallel Training
• Discriminator Feature-based Inference by Recycling the Discriminator of GANs
• Optimizing Multi-GPU Parallelization Strategies for Deep Learning Training
• HyPar-Flow: Exploiting MPI and Keras for Scalable Hybrid-Parallel DNN Training using TensorFlow
• Auto-MAP: A DQN Framework for Exploring Distributed Execution Plans for DNN Workloads
• Diversity/Parallelism Trade-off in Distributed Systems with Redundancy
• SparCE: Sparsity aware General Purpose Core Extensions to Accelerate Deep Neural Networks
• Nested Dithered Quantization for Communication Reduction in Distributed Training
• Parameter Re-Initialization through Cyclical Batch Size Schedules
• Data Streaming and Traffic Gathering in Mesh-based NoC for Deep Neural Network Acceleration
• AI Matrix - Synthetic Benchmarks for DNN
• Theano-MPI: a Theano-based Distributed Training Framework
• AutoLRS: Automatic Learning-Rate Schedule by Bayesian Optimization on the Fly
• Hybrid Data-Model Parallel Training for Sequence-to-Sequence Recurrent Neural Network Machine Translation
• Optimal Mini-Batch Size Selection for Fast Gradient Descent
• Gradient Energy Matching for Distributed Asynchronous Gradient Descent
• The Limiting Dynamics of SGD: Modified Loss, Phase Space Oscillations, and Anomalous Diffusion
• EqCo: Equivalent Rules for Self-supervised Contrastive Learning
• Purine: A bi-graph based deep learning framework
• Dynamic Stale Synchronous Parallel Distributed Training for Deep Learning
• SeesawFaceNets: sparse and robust face verification model for mobile platform
• Improving Disentangled Representation Learning with the Beta Bernoulli Process
• LCP: A Low-Communication Parallelization Method for Fast Neural Network Inference in Image Recognition
• A 7T fMRI dataset of synthetic images for out-of-distribution modeling of vision
• A Study of Checkpointing in Large Scale Training of Deep Neural Networks
• MOSQUITO-NET: A deep learning based CADx system for malaria diagnosis along with model interpretation using GradCam and class activation maps
• Improved generalization by noise enhancement
• What Happens after SGD Reaches Zero Loss? --A Mathematical Framework
• Going Deeper in Frequency Convolutional Neural Network: A Theoretical Perspective
• NAS-HPO-Bench-II: A Benchmark Dataset on Joint Optimization of Convolutional Neural Network Architecture and Training Hyperparameters
• Federated Learning is Better with Non-Homomorphic Encryption
• One-Vote Veto: Semi-Supervised Learning for Low-Shot Glaucoma Diagnosis
• Sequential Place Learning: Heuristic-Free High-Performance Long-Term Place Recognition
• Understanding the Effects of Data Parallelism and Sparsity on Neural Network Training
• On Large-Cohort Training for Federated Learning
• Pretraining Image Encoders without Reconstruction via Feature Prediction Loss
• Adaptive Elastic Training for Sparse Deep Learning on Heterogeneous Multi-GPU Servers
• Derivation and Analysis of Fast Bilinear Algorithms for Convolution
• Non-Blocking Simultaneous Multithreading: Embracing the Resiliency of Deep Neural Networks
• Scraping Social Media Photos Posted in Kenya and Elsewhere to Detect and Analyze Food Types
• Distributed Deep Reinforcement Learning: An Overview
• Adaptive Periodic Averaging: A Practical Approach to Reducing Communication in Distributed Learning
• Restore from Restored: Single-image Inpainting
• Efficient Deep Learning Pipelines for Accurate Cost Estimations Over Large Scale Query Workload
• On the Empirical Neural Tangent Kernel of Standard Finite-Width Convolutional Neural Network Architectures
• HMCNAS: Neural Architecture Search using Hidden Markov Chains and Bayesian Optimization
• Large Batch Training Does Not Need Warmup
• Enhancing Sound Texture in CNN-Based Acoustic Scene Classification
• What Do Adversarially Robust Models Look At?
• Apache Submarine: A Unified Machine Learning Platform Made Simple
• Resource-Adaptive Successive Doubling for Hyperparameter Optimization with Large Datasets on High-Performance Computing Systems
• DistIR: An Intermediate Representation and Simulator for Efficient Neural Network Distribution
• Beyond the Memory Wall: A Case for Memory-centric HPC System for Deep Learning
• Compressed Hierarchical Representations for Multi-Task Learning and Task Clustering
• Scaling Distributed Training of Flood-Filling Networks on HPC Infrastructure for Brain Mapping
• Conditional Adversarial Camera Model Anonymization
• Contrastive Weight Regularization for Large Minibatch SGD
• Real-Time Lane ID Estimation Using Recurrent Neural Networks With Dual Convention
• Classifying Textual Data with Pre-trained Vision Models through Transfer Learning and Data Transformations
• BPPSA: Scaling Back-propagation by Parallel Scan Algorithm
• Analytical bounds on the local Lipschitz constants of affine-ReLU functions
• False Negative Distillation and Contrastive Learning for Personalized Outfit Recommendation
• Gradient Sparification for Asynchronous Distributed Training
• SSGD: A safe and efficient method of gradient descent
• Training CNNs faster with Dynamic Input and Kernel Downsampling
• Backpropagation for long sequences: beyond memory constraints with constant overheads
• Alleviating Bottlenecks for DNN Execution on GPUs via Opportunistic Computing
• Caramel: Accelerating Decentralized Distributed Deep Learning with Computation Scheduling
• Reducing Data Motion to Accelerate the Training of Deep Neural Networks
• Tuning the Scheduling of Distributed Stochastic Gradient Descent with Bayesian Optimization
• Graph Convolution Neural Network For Weakly Supervised Abnormality Localization In Long Capsule Endoscopy Videos
• Simulating Performance of ML Systems with Offline Profiling
• Backtracking gradient descent method for general $C^1$ functions, with applications to Deep Learning
• Workload-aware Automatic Parallelization for Multi-GPU DNN Training
• Improving Layer-wise Adaptive Rate Methods using Trust Ratio Clipping
• PR Product: A Substitute for Inner Product in Neural Networks
• (Pen-) Ultimate DNN Pruning
• SegCodeNet: Color-Coded Segmentation Masks for Activity Detection from Wearable Cameras
• Measuring Information Transfer in Neural Networks