The OARF Benchmark Suite: Characterization and Implications for Federated Learning Systems
arXiv:2006.07856 · doi:10.1145/3510540
Abstract
This paper presents and characterizes an Open Application Repository for Federated Learning (OARF), a benchmark suite for federated machine learning systems. Previously available benchmarks for federated learning have focused mainly on synthetic datasets and use a limited number of applications. OARF mimics more realistic application scenarios with publicly available data sets as different data silos in image, text and structured data. Our characterization shows that the benchmark suite is diverse in data size, distribution, feature distribution and learning task complexity. The extensive evaluations with reference implementations show the future research opportunities for important aspects of federated learning systems. We have developed reference implementations, and evaluated the important aspects of federated learning, including model accuracy, communication cost, throughput and convergence time. Through these evaluations, we discovered some interesting findings such as federated learning can effectively increase end-to-end throughput.
arXiv commentACM Transactions on Intelligent Systems and Technology, Vol. 13, No. 4, Article 63
References in corpus (42)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Deep Learning with Differential Privacy
- Communication-Efficient Learning of Deep Networks from Decentralized Data
- Federated Learning with Non-IID Data
- Federated Optimization: Distributed Machine Learning for On-Device Intelligence
- A Survey on Federated Learning Systems: Vision, Hype and Reality for Data Privacy and Protection
- Renyi Differential Privacy
- Deep Face Recognition: A Survey
- Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training
- QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding
- Learning Differentially Private Recurrent Language Models
- Secure Federated Transfer Learning
- Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization
- Insecurity of Quantum Secure Computations
- Applied Federated Learning: Improving Google Keyboard Query Suggestions
- Private federated learning on vertically partitioned data via entity resolution and additively homomorphic encryption
- Deep Learning Recommendation Model for Personalization and Recommendation Systems
- FedML: A Research Library and Benchmark for Federated Machine Learning
- HybridAlpha: An Efficient Approach for Privacy-Preserving Federated Learning
- LEAF: A Benchmark for Federated Settings
- Neural Collaborative Filtering
- Practical Secure Aggregation for Federated Learning on User-Held Data
- SecureBoost: A Lossless Federated Learning Framework
- D: Decentralized Training over Decentralized Data
- Split learning for health: Distributed deep learning without sharing raw patient data
- FMA: A Dataset For Music Analysis
- PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimization
- Federated Forest
- Decentralized Federated Learning: A Segmented Gossip Approach
- Performance Optimization for Federated Person Re-identification via Benchmark Analysis
- Federated Learning on Non-IID Data Silos: An Experimental Study
- Real-World Image Datasets for Federated Learning
- Central Server Free Federated Learning over Single-sided Trust Social Networks
- Personalized and Private Peer-to-Peer Machine Learning
- Mitigate Bias in Face Recognition using Skewness-Aware Reinforcement Learning
- Practical Federated Gradient Boosting Decision Trees
- FedEval: A Holistic Evaluation Framework for Federated Learning
- Evaluation Framework For Large-scale Federated Learning
- Revocable Federated Learning: A Benchmark of Federated Forest
- Edge AIBench: Towards Comprehensive End-to-end Edge Computing Benchmarking
- An Empirical Study on the Intrinsic Privacy of SGD
- Communication-Efficient Federated Learning via Optimal Client Sampling
Cited by in corpus (13)
- A Survey on Federated Learning Systems: Vision, Hype and Reality for Data Privacy and Protection
- Towards Personalized Federated Learning
- Federated Learning on Non-IID Data Silos: An Experimental Study
- FedScale: Benchmarking Model and System Performance of Federated Learning at Scale
- Towards Efficient Synchronous Federated Training: A Survey on System Optimization Strategies
- Decentralized and Robust Privacy-Preserving Model Using Blockchain-Enabled Federated Deep Learning in Intelligent Enterprises
- Optimizing Performance of Federated Person Re-identification: Benchmarking and Analysis
- FedEval: A Holistic Evaluation Framework for Federated Learning
- Model-Contrastive Federated Learning
- Federated Learning for Clinical Structured Data: A Benchmark Comparison of Engineering and Statistical Approaches
- Towards Unsupervised Domain Adaptation for Deep Face Recognition under Privacy Constraints via Federated Learning
- A Robust Federated Learning Approach for Combating Attacks Against IoT Systems Under non-IID Challenges
- FLBench: A Benchmark Suite for Federated Learning