Pufferfish: Communication-efficient Models At No Extra Cost
arXiv:2103.03936
Abstract
To mitigate communication overheads in distributed model training, several studies propose the use of compressed stochastic gradients, usually achieved by sparsification or quantization. Such techniques achieve high compression ratios, but in many cases incur either significant computational overheads or some accuracy loss. In this work, we present Pufferfish, a communication and computation efficient distributed training framework that incorporates the gradient compression into the model training process via training low-rank, pre-factorized deep networks. Pufferfish not only reduces communication, but also completely bypasses any computation overheads related to compression, and achieves the same accuracy as state-of-the-art, off-the-shelf deep models. Pufferfish can be directly integrated into current deep learning frameworks with minimum implementation modification. Our extensive experiments over real distributed setups, across a variety of large-scale machine learning tasks, indicate that Pufferfish achieves up to 1.64x end-to-end speedup over the latest distributed training API in PyTorch without accuracy loss. Compared to the Lottery Ticket Hypothesis models, Pufferfish leads to equally accurate, small-parameter models while avoiding the burden of "winning the lottery". Pufferfish also leads to more accurate and smaller models than SOTA structured model pruning methods.
Accepted by MLSys 2021
References in corpus (37)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
- EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
- Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding
- Language Models are Few-Shot Learners
- Federated Learning: Strategies for Improving Communication Efficiency
- Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
- Wide Residual Networks
- DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
- Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
- The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
- QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding
- Binarized Neural Networks
- Trained Ternary Quantization
- Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architectures
- Pruning Filters for Efficient ConvNets
- Revisiting Distributed Synchronous SGD
- On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
- Speeding up Convolutional Neural Networks with Low Rank Expansions
- Incremental Network Quantization: Towards Lossless CNNs with Low-Precision Weights
- Pointer Sentinel Mixture Models
- Slimmable Neural Networks
- Sparsified SGD with Memory
- Error Feedback Fixes SignSGD and other Gradient Compression Schemes
- signSGD with Majority Vote is Communication Efficient And Fault Tolerant
- DoubleSqueeze: Parallel Stochastic Gradient Descent with Double-Pass Error-Compensated Compression
- Taming the Wild: A Unified Analysis of Hogwild!-Style Algorithms
- High-Accuracy Low-Precision Training
- Fixing the train-test resolution discrepancy: FixEfficientNet
- Variance-based Gradient Compression for Efficient Distributed Deep Learning
- Drawing Early-Bird Tickets: Towards More Efficient Training of Deep Networks
- Accordion: Adaptive Gradient Communication via Critical Learning Regime Identification
- The Two Regimes of Deep Network Training
- ASAGA: Asynchronous Parallel SAGA
- Randomized Distributed Mean Estimation: Accuracy vs Communication
- Principal Component Networks: Parameter Reduction Early in Training