The OARF Benchmark Suite: Characterization and Implications for Federated Learning Systems

Abstract

This paper presents and characterizes an Open Application Repository for Federated Learning (OARF), a benchmark suite for federated machine learning systems. Previously available benchmarks for federated learning have focused mainly on synthetic datasets and use a limited number of applications. OARF mimics more realistic application scenarios with publicly available data sets as different data silos in image, text and structured data. Our characterization shows that the benchmark suite is diverse in data size, distribution, feature distribution and learning task complexity. The extensive evaluations with reference implementations show the future research opportunities for important aspects of federated learning systems. We have developed reference implementations, and evaluated the important aspects of federated learning, including model accuracy, communication cost, throughput and convergence time. Through these evaluations, we discovered some interesting findings such as federated learning can effectively increase end-to-end throughput.

ACM Transactions on Intelligent Systems and Technology, Vol. 13, No. 4, Article 63

References in corpus (22)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Federated Optimization: Distributed Machine Learning for On-Device Intelligence
• Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization
• Private federated learning on vertically partitioned data via entity resolution and additively homomorphic encryption
• Deep Learning Recommendation Model for Personalization and Recommendation Systems
• FedML: A Research Library and Benchmark for Federated Machine Learning
• HybridAlpha: An Efficient Approach for Privacy-Preserving Federated Learning
• Neural Collaborative Filtering
• Practical Secure Aggregation for Federated Learning on User-Held Data
• Decentralized Federated Learning: A Segmented Gossip Approach
• Performance Optimization for Federated Person Re-identification via Benchmark Analysis
• Federated Learning on Non-IID Data Silos: An Experimental Study
• Real-World Image Datasets for Federated Learning
• Central Server Free Federated Learning over Single-sided Trust Social Networks
• Mitigate Bias in Face Recognition using Skewness-Aware Reinforcement Learning
• Practical Federated Gradient Boosting Decision Trees
• FedEval: A Holistic Evaluation Framework for Federated Learning
• Evaluation Framework For Large-scale Federated Learning
• Revocable Federated Learning: A Benchmark of Federated Forest
• Edge AIBench: Towards Comprehensive End-to-end Edge Computing Benchmarking
• An Empirical Study on the Intrinsic Privacy of SGD
• Communication-Efficient Federated Learning via Optimal Client Sampling

Cited by in corpus (12)

• Towards Personalized Federated Learning
• Federated Learning on Non-IID Data Silos: An Experimental Study
• FedScale: Benchmarking Model and System Performance of Federated Learning at Scale
• Towards Efficient Synchronous Federated Training: A Survey on System Optimization Strategies
• Decentralized and Robust Privacy-Preserving Model Using Blockchain-Enabled Federated Deep Learning in Intelligent Enterprises
• Optimizing Performance of Federated Person Re-identification: Benchmarking and Analysis
• FedEval: A Holistic Evaluation Framework for Federated Learning
• Model-Contrastive Federated Learning
• Federated Learning for Clinical Structured Data: A Benchmark Comparison of Engineering and Statistical Approaches
• Towards Unsupervised Domain Adaptation for Deep Face Recognition under Privacy Constraints via Federated Learning
• A Robust Federated Learning Approach for Combating Attacks Against IoT Systems Under non-IID Challenges
• FLBench: A Benchmark Suite for Federated Learning