FedEval: A Holistic Evaluation Framework for Federated Learning

Abstract

Federated Learning (FL) has been widely accepted as the solution for privacy-preserving machine learning without collecting raw data. While new technologies proposed in the past few years do evolve the FL area, unfortunately, the evaluation results presented in these works fall short in integrity and are hardly comparable because of the inconsistent evaluation metrics and experimental settings. In this paper, we propose a holistic evaluation framework for FL called FedEval, and present a benchmarking study on seven state-of-the-art FL algorithms. Specifically, we first introduce the core evaluation taxonomy model, called FedEval-Core, which covers four essential evaluation aspects for FL: Privacy, Robustness, Effectiveness, and Efficiency, with various well-defined metrics and experimental settings. Based on the FedEval-Core, we further develop an FL evaluation platform with standardized evaluation settings and easy-to-use interfaces. We then provide an in-depth benchmarking study between the seven well-known FL algorithms, including FedSGD, FedAvg, FedProx, FedOpt, FedSTC, SecAgg, and HEAgg. We comprehensively analyze the advantages and disadvantages of these algorithms and further identify the suitable practical scenarios for different algorithms, which is rarely done by prior work. Lastly, we excavate a set of take-away insights and future research directions, which are very helpful for researchers in the FL area.

References in corpus (26)

• Adam: A Method for Stochastic Optimization
• Communication-Efficient Learning of Deep Networks from Decentralized Data
• Federated Learning with Non-IID Data
• Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training
• On the Convergence of FedAvg on Non-IID Data
• How To Backdoor Federated Learning
• SCAFFOLD: Stochastic Controlled Averaging for Federated Learning
• Revisiting Distributed Synchronous SGD
• Communication-Efficient On-Device Machine Learning: Federated Distillation and Augmentation under Non-IID Private Data
• Improving Federated Learning Personalization via Model Agnostic Meta Learning
• Mitigating Sybils in Federated Learning Poisoning
• FedML: A Research Library and Benchmark for Federated Machine Learning
• Fair Resource Allocation in Federated Learning
• A generic framework for privacy preserving deep learning
• LEAF: A Benchmark for Federated Settings
• Federated Meta-Learning with Fast Convergence and Efficient Communication
• Agnostic Federated Learning
• Adaptive Federated Optimization
• Real-World Image Datasets for Federated Learning
• Acceleration for Compressed Gradient Descent in Distributed and Federated Optimization
• FetchSGD: Communication-Efficient Federated Learning with Sketching
• A Communication Efficient Collaborative Learning Framework for Distributed Features
• Variance-based Gradient Compression for Efficient Distributed Deep Learning
• The OARF Benchmark Suite: Characterization and Implications for Federated Learning Systems
• Federated Learning with Only Positive Labels
• From Local SGD to Local Fixed-Point Methods for Federated Learning

Cited by in corpus (7)

• A Survey on Federated Learning Systems: Vision, Hype and Reality for Data Privacy and Protection
• Federated Learning for Healthcare Domain - Pipeline, Applications and Challenges
• GRNN: Generative Regression Neural Network -- A Data Leakage Attack for Federated Learning
• The OARF Benchmark Suite: Characterization and Implications for Federated Learning Systems
• FedScale: Benchmarking Model and System Performance of Federated Learning at Scale
• A Privacy Preserving System for Movie Recommendations Using Federated Learning
• Federated Learning for Clinical Structured Data: A Benchmark Comparison of Engineering and Statistical Approaches