Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification

Abstract

Federated Learning enables visual models to be trained in a privacy-preserving way using real-world data from mobile devices. Given their distributed nature, the statistics of the data across these devices is likely to differ significantly. In this work, we look at the effect such non-identical data distributions has on visual classification via Federated Learning. We propose a way to synthesize datasets with a continuous range of identicalness and provide performance measures for the Federated Averaging algorithm. We show that performance degrades as distributions differ more, and propose a mitigation strategy via server momentum. Experiments on CIFAR-10 demonstrate improved classification performance over a range of non-identicalness, with classification accuracy improved from 30.1% to 76.9% in the most skewed settings.

References in corpus (8)

• Communication-Efficient Learning of Deep Networks from Decentralized Data
• Federated Learning with Non-IID Data
• On the Convergence of FedAvg on Non-IID Data
• EMNIST: an extension of MNIST to handwritten letters
• Federated Optimization in Heterogeneous Networks
• LEAF: A Benchmark for Federated Settings
• Measuring the Effects of Data Parallelism on Neural Network Training
• Bayesian Nonparametric Federated Learning of Neural Networks

Cited by in corpus (116)

• Towards Personalized Federated Learning
• SCAFFOLD: Stochastic Controlled Averaging for Federated Learning
• Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization
• Ensemble Distillation for Robust Model Fusion in Federated Learning
• FedBN: Federated Learning on Non-IID Features via Local Batch Normalization
• Label-Efficient Self-Supervised Federated Learning for Tackling Data Heterogeneity in Medical Imaging
• A Field Guide to Federated Optimization
• Analysis and Optimal Edge Assignment For Hierarchical Federated Learning on Non-IID Data
• Adaptive Federated Optimization
• Federated Learning Based on Dynamic Regularization
• Mime: Mimicking Centralized Stochastic Algorithms in Federated Learning
• Differentially Private Learning with Adaptive Clipping
• Federated Learning on Non-IID Data Silos: An Experimental Study
• A review of federated learning in renewable energy applications: Potential, challenges, and future directions
• Training Speech Recognition Models with Federated Learning: A Quality/Cost Framework
• Federated Learning for 5G Base Station Traffic Forecasting
• Continual Local Training for Better Initialization of Federated Models
• Salvaging Federated Learning by Local Adaptation
• Federated Unlearning: A Survey on Methods, Design Guidelines, and Evaluation Metrics
• A first look into the carbon footprint of federated learning
• Semi-Synchronous Federated Learning for Energy-Efficient Training and Accelerated Convergence in Cross-Silo Settings
• FedCV: A Federated Learning Framework for Diverse Computer Vision Tasks
• No Fear of Heterogeneity: Classifier Calibration for Federated Learning with Non-IID Data
• SemiFL: Semi-Supervised Federated Learning for Unlabeled Clients with Alternate Training
• Federated Learning with Buffered Asynchronous Aggregation
• Federated Multi-organ Segmentation with Inconsistent Labels
• Local-Global Knowledge Distillation in Heterogeneous Federated Learning with Non-IID Data
• Auto-weighted Robust Federated Learning with Corrupted Data Sources
• Towards Efficient Synchronous Federated Training: A Survey on System Optimization Strategies
• Decentralized and Robust Privacy-Preserving Model Using Blockchain-Enabled Federated Deep Learning in Intelligent Enterprises
• Quasi-Global Momentum: Accelerating Decentralized Deep Learning on Heterogeneous Data
• A Lightweight and Secure Deep Learning Model for Privacy-Preserving Federated Learning in Intelligent Enterprises
• FedZero: Leveraging Renewable Excess Energy in Federated Learning
• FedCross: Towards Accurate Federated Learning via Multi-Model Cross-Aggregation
• Personalized Federated Learning with First Order Model Optimization
• Social-Aware Clustered Federated Learning with Customized Privacy Preservation
• Prototype Guided Federated Learning of Visual Feature Representations
• FedGEMS: Federated Learning of Larger Server Models via Selective Knowledge Fusion
• Federated Motor Imagery Classification for Privacy-Preserving Brain-Computer Interfaces
• FedCM: Federated Learning with Client-level Momentum
• Improving Performance of Federated Learning based Medical Image Analysis in Non-IID Settings using Image Augmentation
• Local Adaptivity in Federated Learning: Convergence and Consistency
• Connecting Low-Loss Subspace for Personalized Federated Learning
• On the Outsized Importance of Learning Rates in Local Update Methods
• Speeding up Heterogeneous Federated Learning with Sequentially Trained Superclients
• Model-Contrastive Federated Learning
• Stochastic-Sign SGD for Federated Learning with Theoretical Guarantees
• Resisting Backdoor Attacks in Federated Learning via Bidirectional Elections and Individual Perspective
• Federated Learning for Clinical Structured Data: A Benchmark Comparison of Engineering and Statistical Approaches
• Scaling Survival Analysis in Healthcare with Federated Survival Forests: A Comparative Study on Heart Failure and Breast Cancer Genomics
• Improving Semi-supervised Federated Learning by Reducing the Gradient Diversity of Models
• Secure Aggregation with Heterogeneous Quantization in Federated Learning
• Federated Survival Forests
• Specialized federated learning using a mixture of experts
• Achieving Linear Speedup with Partial Worker Participation in Non-IID Federated Learning
• OnDev-LCT: On-Device Lightweight Convolutional Transformers towards federated learning
• An Experimental Study of Data Heterogeneity in Federated Learning Methods for Medical Imaging
• Effective Federated Adaptive Gradient Methods with Non-IID Decentralized Data
• FedCos: A Scene-adaptive Federated Optimization Enhancement for Performance Improvement
• Federated Stain Normalization for Computational Pathology
• Communication-Efficient Federated Distillation
• SSFL: Tackling Label Deficiency in Federated Learning via Personalized Self-Supervision
• What Do We Mean by Generalization in Federated Learning?
• Rethinking Architecture Design for Tackling Data Heterogeneity in Federated Learning
• FedCL: Federated Multi-Phase Curriculum Learning to Synchronously Correlate User Heterogeneity
• Efficient and Private Federated Learning with Partially Trainable Networks
• A Clustering Strategy for Enhanced FL-Based Intrusion Detection in IoT Networks
• Personalized Federated Learning with Gaussian Processes
• Joint Local Relational Augmentation and Global Nash Equilibrium for Federated Learning with Non-IID Data
• Federated Learning for Commercial Image Sources
• Sharp Bounds for Federated Averaging (Local SGD) and Continuous Perspective
• Interpretable Data Fusion for Distributed Learning: A Representative Approach via Gradient Matching
• Federated Mixture of Experts
• Federated $f$-Differential Privacy
• RingFed: Reducing Communication Costs in Federated Learning on Non-IID Data
• Federated Learning on Non-iid Data via Local and Global Distillation
• Federated Learning with Intermediate Representation Regularization
• A Robust Federated Learning Approach for Combating Attacks Against IoT Systems Under non-IID Challenges
• Convergence and Accuracy Trade-Offs in Federated Learning and Meta-Learning
• Multi-VFL: A Vertical Federated Learning System for Multiple Data and Label Owners
• A Survey and Future Outlook on Indoor Location Fingerprinting Privacy Preservation
• Near-Optimal Resilient Aggregation Rules for Distributed Learning Using 1-Center and 1-Mean Clustering with Outliers
• Noiseless Privacy-Preserving Decentralized Learning
• Quantized and Asynchronous Federated Learning
• SPATL: Salient Parameter Aggregation and Transfer Learning for Heterogeneous Clients in Federated Learning
• IFedAvg: Interpretable Data-Interoperability for Federated Learning
• A Framework for testing Federated Learning algorithms using an edge-like environment
• Accurate and Fast Federated Learning via Combinatorial Multi-Armed Bandits
• Federated Nonconvex Sparse Learning
• Multi-Level Analyzation of Imbalance to Resolve Non-IID-Ness in Federated Learning
• Towards Fast and Stable Federated Learning: Confronting Heterogeneity via Knowledge Anchor
• Federated Multi-Target Domain Adaptation
• Robust Federated Learning with Attack-Adaptive Aggregation
• Federated Loss Exploration for Improved Convergence on Non-IID Data
• LIA: Privacy-Preserving Data Quality Evaluation in Federated Learning Using a Lazy Influence Approximation
• DP-REC: Private & Communication-Efficient Federated Learning
• Compare Where It Matters: Using Layer-Wise Regularization To Improve Federated Learning on Heterogeneous Data
• On Large-Cohort Training for Federated Learning
• Federated Bayesian Deep Learning: The Application of Statistical Aggregation Methods to Bayesian Models
• Behavior Mimics Distribution: Combining Individual and Group Behaviors for Federated Learning
• Privacy and Accuracy Implications of Model Complexity and Integration in Heterogeneous Federated Learning
• Accelerating Federated Learning in Heterogeneous Data and Computational Environments
• New Metrics to Evaluate the Performance and Fairness of Personalized Federated Learning
• Federated Learning via Plurality Vote
• An Expectation-Maximization Perspective on Federated Learning
• Federated Few-Shot Learning with Adversarial Learning
• On Provable Backdoor Defense in Collaborative Learning
• Fairness-aware Federated Minimax Optimization with Convergence Guarantee
• Neural Tangent Kernel Empowered Federated Learning
• Demystifying the Effects of Non-Independence in Federated Learning
• Learn More by Using Less: Distributed Learning with Energy-Constrained Devices
• Iterated Vector Fields and Conservatism, with Applications to Federated Learning
• Towards More Efficient Federated Learning with Better Optimization Objects
• Prior-Independent Auctions for the Demand Side of Federated Learning
• Efficient Federated Learning for AIoT Applications Using Knowledge Distillation
• Compositional federated learning: Applications in distributionally robust averaging and meta learning