Trends and Advancements in Deep Neural Network Communication

Abstract

Due to their great performance and scalability properties neural networks have become ubiquitous building blocks of many applications. With the rise of mobile and IoT, these models now are also being increasingly applied in distributed settings, where the owners of the data are separated by limited communication channels and privacy constraints. To address the challenges of these distributed environments, a wide range of training and evaluation schemes have been developed, which require the communication of neural network parametrizations. These novel approaches, which bring the "intelligence to the data" have many advantages over traditional cloud solutions such as privacy-preservation, increased security and device autonomy, communication efficiency and high training speed. This paper gives an overview over the recent advancements and challenges in this new field of research at the intersection of machine learning and communications.

References in corpus (41)

• Distilling the Knowledge in a Neural Network
• Sequence to Sequence Learning with Neural Networks
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Communication-Efficient Learning of Deep Networks from Decentralized Data
• Federated Learning: Strategies for Improving Communication Efficiency
• Federated Learning with Non-IID Data
• HOGWILD!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent
• Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training
• Character-Aware Neural Language Models
• QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding
• Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
• How To Backdoor Federated Learning
• Deep Learning for Identifying Metastatic Breast Cancer
• TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning
• Revisiting Distributed Synchronous SGD
• Differentially Private Federated Learning: A Client Level Perspective
• Federated Optimization in Heterogeneous Networks
• Improving Federated Learning Personalization via Model Agnostic Meta Learning
• Variational Dropout Sparsifies Deep Neural Networks
• Pruning by Explaining: A Novel Criterion for Deep Neural Network Pruning
• Sparsified SGD with Memory
• Local SGD Converges Fast and Communicates Little
• Training Deep Neural Networks with 8-bit Floating Point Numbers
• ATOMO: Communication-efficient Learning via Atomic Sparsification
• Practical Secure Aggregation for Federated Learning on User-Held Data
• Decentralized Stochastic Optimization and Gossip Algorithms with Compressed Communication
• D${}^2$: Decentralized Training over Decentralized Data
• Cooperative SGD: A unified Framework for the Design and Analysis of Communication-Efficient SGD Algorithms
• Expanding the Reach of Federated Learning by Reducing Client Resource Requirements
• PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimization
• Peer-to-peer Federated Learning on Graphs
• DeepCABAC: A Universal Compression Algorithm for Deep Neural Networks
• DoubleSqueeze: Parallel Stochastic Gradient Descent with Double-Pass Error-Compensated Compression
• Taming the Wild: A Unified Analysis of Hogwild!-Style Algorithms
• Towards the Limit of Network Quantization
• Personalized and Private Peer-to-Peer Machine Learning
• Privacy for Free: Communication-Efficient Learning with Differential Privacy Using Sketches
• Decentralized Deep Learning with Arbitrary Communication Compression
• Client-Edge-Cloud Hierarchical Federated Learning
• $\text{DeepSqueeze}$: Decentralization Meets Error-Compensated Compression
• Communication-efficient distributed SGD with Sketching

Cited by in corpus (2)

• Communication-Efficient Federated Distillation
• Bristle: Decentralized Federated Learning in Byzantine, Non-i.i.d. Environments