ATOMO: Communication-efficient Learning via Atomic Sparsification

Abstract

Distributed model training suffers from communication overheads due to frequent gradient updates transmitted between compute nodes. To mitigate these overheads, several studies propose the use of sparsified stochastic gradients. We argue that these are facets of a general sparsification method that can operate on any possible atomic decomposition. Notable examples include element-wise, singular value, and Fourier decompositions. We present ATOMO, a general framework for atomic sparsification of stochastic gradients. Given a gradient, an atomic decomposition, and a sparsity budget, ATOMO gives a random unbiased sparsification of the atoms minimizing variance. We show that recent methods such as QSGD and TernGrad are special cases of ATOMO and that sparsifiying the singular value decomposition of neural networks gradients, rather than their coordinates, can lead to significantly faster distributed training.

References in corpus (3)

• Federated Learning: Strategies for Improving Communication Efficiency
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• High-Accuracy Low-Precision Training

Cited by in corpus (48)

• Machine Learning at the Wireless Edge: Distributed Stochastic Gradient Descent Over-the-Air
• Group Knowledge Transfer: Federated Learning of Large CNNs at the Edge
• Expanding the Reach of Federated Learning by Reducing Client Resource Requirements
• Federated Learning in Mobile Edge Networks: A Comprehensive Survey
• PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimization
• A Survey on Approximate Edge AI for Energy Efficient Autonomous Driving Services
• Communication-Efficient Distributed Deep Learning: A Comprehensive Survey
• No One Left Behind: Inclusive Federated Learning over Heterogeneous Devices
• Fast Federated Learning by Balancing Communication Trade-Offs
• Communication optimization strategies for distributed deep neural network training: A survey
• FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropout
• Computation Scheduling for Distributed Machine Learning with Straggling Workers
• Towards Communication-efficient and Attack-Resistant Federated Edge Learning for Industrial Internet of Things
• ErasureHead: Distributed Gradient Descent without Delays Using Approximate Gradient Coding
• RedSync : Reducing Synchronization Traffic for Distributed Deep Learning
• Optimal Gradient Compression for Distributed and Federated Learning
• Breaking the Communication-Privacy-Accuracy Trilemma
• Moniqua: Modulo Quantized Communication in Decentralized SGD
• Hierarchical Federated Learning through LAN-WAN Orchestration
• Communication-Efficient Local Decentralized SGD Methods
• Election Coding for Distributed Learning: Protecting SignSGD against Byzantine Attacks
• FEDZIP: A Compression Framework for Communication-Efficient Federated Learning
• Reducing Communication for Split Learning by Randomized Top-k Sparsification
• $\text{DeepSqueeze}$: Decentralization Meets Error-Compensated Compression
• On the Utility of Gradient Compression in Distributed Training Systems
• rTop-k: A Statistical Estimation Approach to Distributed SGD
• Stochastic Sign Descent Methods: New Algorithms and Better Theory
• DRIVE: One-bit Distributed Mean Estimation
• Wyner-Ziv Gradient Compression for Federated Learning
• PowerGossip: Practical Low-Rank Communication Compression in Decentralized Deep Learning
• Activations and Gradients Compression for Model-Parallel Training
• NUQSGD: Provably Communication-efficient Data-parallel SGD via Nonuniform Quantization
• Is Network the Bottleneck of Distributed Training?
• Federated Learning in Adversarial Settings
• Distributed Newton Can Communicate Less and Resist Byzantine Workers
• Communication Efficient Federated Learning with Energy Awareness over Wireless Networks
• New Bounds For Distributed Mean Estimation and Variance Reduction
• Trends and Advancements in Deep Neural Network Communication
• Compressing gradients by exploiting temporal correlation in momentum-SGD
• A flexible framework for communication-efficient machine learning: from HPC to IoT
• CGX: Adaptive System Support for Communication-Efficient Deep Learning
• Communication-Efficient Distributed Learning via Sparse and Adaptive Stochastic Gradient
• Towards Tight Communication Lower Bounds for Distributed Optimisation
• The Gradient Convergence Bound of Federated Multi-Agent Reinforcement Learning with Efficient Communication
• Edge Artificial Intelligence for 6G: Vision, Enabling Technologies, and Applications
• Doing More by Doing Less: How Structured Partial Backpropagation Improves Deep Learning Clusters
• Optimal Compression of Locally Differentially Private Mechanisms
• Associative Convolutional Layers