The Non-IID Data Quagmire of Decentralized Machine Learning

Abstract

Many large-scale machine learning (ML) applications need to perform decentralized learning over datasets generated at different devices and locations. Such datasets pose a significant challenge to decentralized learning because their different contexts result in significant data distribution skew across devices/locations. In this paper, we take a step toward better understanding this challenge by presenting a detailed experimental study of decentralized DNN training on a common type of data skew: skewed distribution of data labels across devices/locations. Our study shows that: (i) skewed data labels are a fundamental and pervasive problem for decentralized learning, causing significant accuracy loss across many ML applications, DNN models, training datasets, and decentralized learning algorithms; (ii) the problem is particularly challenging for DNN models with batch normalization; and (iii) the degree of data skew is a key determinant of the difficulty of the problem. Based on these findings, we present SkewScout, a system-level approach that adapts the communication frequency of decentralized learning algorithms to the (skew-induced) accuracy loss between data partitions. We also show that group normalization can recover much of the accuracy loss of batch normalization.

References in corpus (22)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Communication-Efficient Learning of Deep Networks from Decentralized Data
• Caffe: Convolutional Architecture for Fast Feature Embedding
• On the difficulty of training Recurrent Neural Networks
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Federated Learning with Non-IID Data
• Learning Face Representation from Scratch
• HOGWILD!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent
• Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training
• On the Convergence of FedAvg on Non-IID Data
• Making Gradient Descent Optimal for Strongly Convex Stochastic Optimization
• Communication Efficient Distributed Optimization using an Approximate Newton-type Method
• Three Approaches for Personalization with Applications to Federated Learning
• LEAF: A Benchmark for Federated Settings
• Batch Renormalization: Towards Reducing Minibatch Dependence in Batch-Normalized Models
• D${}^2$: Decentralized Training over Decentralized Data
• Overcoming Forgetting in Federated Learning on Non-IID Data
• Federated Learning with Matched Averaging
• Variance Reduced Local SGD with Lower Communication Complexity
• Mini-Batch Primal and Dual Methods for SVMs
• Salvaging Federated Learning by Local Adaptation
• Real-World Image Datasets for Federated Learning

Cited by in corpus (27)

• Ensemble Distillation for Robust Model Fusion in Federated Learning
• FedML: A Research Library and Benchmark for Federated Machine Learning
• FedBN: Federated Learning on Non-IID Features via Local Batch Normalization
• Group Knowledge Transfer: Federated Learning of Large CNNs at the Edge
• Adaptive Federated Optimization
• Attack of the Tails: Yes, You Really Can Backdoor Federated Learning
• Training Speech Recognition Models with Federated Learning: A Quality/Cost Framework
• A first look into the carbon footprint of federated learning
• No Fear of Heterogeneity: Classifier Calibration for Federated Learning with Non-IID Data
• Federated Learning via Posterior Averaging: A New Perspective and Practical Algorithms
• Towards Flexible Device Participation in Federated Learning
• Federated Learning With Quantized Global Model Updates
• Personalized Federated Learning with First Order Model Optimization
• On the Outsized Importance of Learning Rates in Local Update Methods
• What Do We Mean by Generalization in Federated Learning?
• Improving Accuracy of Federated Learning in Non-IID Settings
• Distributed Optimization over Block-Cyclic Data
• Aggregate or Not? Exploring Where to Privatize in DNN Based Federated Learning Under Different Non-IID Scenes
• CDMPP: A Device-Model Agnostic Framework for Latency Prediction of Tensor Programs
• Convergence and Accuracy Trade-Offs in Federated Learning and Meta-Learning
• Federated Learning of User Authentication Models
• SPATL: Salient Parameter Aggregation and Transfer Learning for Heterogeneous Clients in Federated Learning
• Cross-Gradient Aggregation for Decentralized Learning from Non-IID data
• Accelerating MoE Model Inference with Expert Sharding
• Practical Federated Learning without a Server
• Federated Learning of User Verification Models Without Sharing Embeddings
• Marketplace for AI Models