LEAF: A Benchmark for Federated Settings

Abstract

Modern federated networks, such as those comprised of wearable devices, mobile phones, or autonomous vehicles, generate massive amounts of data each day. This wealth of data can help to learn models that can improve the user experience on each device. However, the scale and heterogeneity of federated data presents new challenges in research areas such as federated learning, meta-learning, and multi-task learning. As the machine learning community begins to tackle these challenges, we are at a critical time to ensure that developments made in these areas are grounded with realistic benchmarks. To this end, we propose LEAF, a modular benchmarking framework for learning in federated settings. LEAF includes a suite of open-source federated datasets, a rigorous evaluation framework, and a set of reference implementations, all geared towards capturing the obstacles and intricacies of practical federated environments.

References in corpus (13)

• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Prototypical Networks for Few-shot Learning
• Communication-Efficient Learning of Deep Networks from Decentralized Data
• Federated Learning: Strategies for Improving Communication Efficiency
• How To Backdoor Federated Learning
• Learning Differentially Private Recurrent Language Models
• On First-Order Meta-Learning Algorithms
• Differentially Private Federated Learning: A Client Level Perspective
• EMNIST: an extension of MNIST to handwritten letters
• Improving Federated Learning Personalization via Model Agnostic Meta Learning
• Fair Resource Allocation in Federated Learning
• Federated Meta-Learning with Fast Convergence and Efficient Communication
• Differentially-Private "Draw and Discard" Machine Learning

Cited by in corpus (167)

• A Survey on Federated Learning Systems: Vision, Hype and Reality for Data Privacy and Protection
• Robust Aggregation for Federated Learning
• A Blockchain-based Decentralized Federated Learning Framework with Committee Consensus
• Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification
• Ensemble Distillation for Robust Model Fusion in Federated Learning
• FedMD: Heterogenous Federated Learning via Model Distillation
• Federated Optimization in Heterogeneous Networks
• Can You Really Backdoor Federated Learning?
• Improving Federated Learning Personalization via Model Agnostic Meta Learning
• From Distributed Machine Learning to Federated Learning: A Survey
• FedML: A Research Library and Benchmark for Federated Machine Learning
• Distillation-Based Semi-Supervised Federated Learning for Communication-Efficient Collaborative Training with Non-IID Private Data
• Fair Resource Allocation in Federated Learning
• Adaptive Personalized Federated Learning
• Three Approaches for Personalization with Applications to Federated Learning
• WILDS: A Benchmark of in-the-Wild Distribution Shifts
• Federated Meta-Learning with Fast Convergence and Efficient Communication
• Ditto: Fair and Robust Federated Learning Through Personalization
• Group Knowledge Transfer: Federated Learning of Large CNNs at the Edge
• FedPD: A Federated Learning Framework with Optimal Rates and Adaptivity to Non-IID Data
• On the Convergence of Local Descent Methods in Federated Learning
• A Field Guide to Federated Optimization
• Federated Learning in Mobile Edge Networks: A Comprehensive Survey
• One-Shot Federated Learning
• Think Locally, Act Globally: Federated Learning with Local and Global Representations
• Federated Forest
• A Decentralized Federated Learning Framework via Committee Mechanism with Convergence Guarantee
• Adaptive Federated Optimization
• Variational Federated Multi-Task Learning
• Federated Learning Based on Dynamic Regularization
• Abnormal Client Behavior Detection in Federated Learning
• Review: Deep Learning in Electron Microscopy
• Flower: A Friendly Federated Learning Research Framework
• FedGraphNN: A Federated Learning System and Benchmark for Graph Neural Networks
• Federated Learning with Matched Averaging
• Optimal Client Sampling for Federated Learning
• Mime: Mimicking Centralized Stochastic Algorithms in Federated Learning
• Federated Multi-Task Learning under a Mixture of Distributions
• The Non-IID Data Quagmire of Decentralized Machine Learning
• Federated Learning on Non-IID Data Silos: An Experimental Study
• Concept drift detection and adaptation for federated and continual learning
• Privacy-Preserving Machine Learning: Methods, Challenges and Directions
• Exploiting Shared Representations for Personalized Federated Learning
• Real-World Image Datasets for Federated Learning
• FedMix: Approximation of Mixup under Mean Augmented Federated Learning
• Federated Learning for Healthcare Informatics
• FedSAE: A Novel Self-Adaptive Federated Learning Framework in Heterogeneous Systems
• FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropout
• FedLab: A Flexible Federated Learning Framework
• Continual Learning of a Mixed Sequence of Similar and Dissimilar Tasks
• Throughput-Optimal Topology Design for Cross-Silo Federated Learning
• Federated Learning with Unbiased Gradient Aggregation and Controllable Meta Updating
• CSAFL: A Clustered Semi-Asynchronous Federated Learning Framework
• The OARF Benchmark Suite: Characterization and Implications for Federated Learning Systems
• Generative Models for Effective ML on Private, Decentralized Datasets
• Federated Learning for Big Data: A Survey on Opportunities, Applications, and Future Directions
• FLASHE: Additively Symmetric Homomorphic Encryption for Cross-Silo Federated Learning
• Privacy for Free: Communication-Efficient Learning with Differential Privacy Using Sketches
• FedScale: Benchmarking Model and System Performance of Federated Learning at Scale
• Federated Learning via Posterior Averaging: A New Perspective and Practical Algorithms
• Towards Probabilistic Verification of Machine Unlearning
• The Skellam Mechanism for Differentially Private Federated Learning
• Federated Learning with Buffered Asynchronous Aggregation
• Faster On-Device Training Using New Federated Momentum Algorithm
• Auto-weighted Robust Federated Learning with Corrupted Data Sources
• Edge-assisted Democratized Learning Towards Federated Analytics
• Federated Learning with Additional Mechanisms on Clients to Reduce Communication Costs
• GS-WGAN: A Gradient-Sanitized Approach for Learning Differentially Private Generators
• Papaya: Practical, Private, and Scalable Federated Learning
• FedAT: A High-Performance and Communication-Efficient Federated Learning System with Asynchronous Tiers
• Secure Federated Submodel Learning
• Federated Learning with Compression: Unified Analysis and Sharp Guarantees
• The Distributed Discrete Gaussian Mechanism for Federated Learning with Secure Aggregation
• Prototype Guided Federated Learning of Visual Feature Representations
• Federated Model Distillation with Noise-Free Differential Privacy
• FedGEMS: Federated Learning of Larger Server Models via Selective Knowledge Fusion
• FEDZIP: A Compression Framework for Communication-Efficient Federated Learning
• SemiFed: Semi-supervised Federated Learning with Consistency and Pseudo-Labeling
• Enhancing the Privacy of Federated Learning with Sketching
• Federated Neural Architecture Search
• Adaptive Gradient-Based Meta-Learning Methods
• GFL: A Decentralized Federated Learning Framework Based On Blockchain
• An Experimental Study of Class Imbalance in Federated Learning
• Personalized Federated Learning through Local Memorization
• Heterogeneity for the Win: One-Shot Federated Clustering
• FedEval: A Holistic Evaluation Framework for Federated Learning
• On the Outsized Importance of Learning Rates in Local Update Methods
• FedJAX: Federated learning simulation with JAX
• Federated Split Vision Transformer for COVID-19 CXR Diagnosis using Task-Agnostic Training
• BREEDS: Benchmarks for Subpopulation Shift
• Model-Contrastive Federated Learning
• Asynchronous Online Federated Learning for Edge Devices with Non-IID Data
• Blockchain-based Federated Learning for Device Failure Detection in Industrial IoT
• Implicit Model Specialization through DAG-based Decentralized Federated Learning
• A Federated Multi-View Deep Learning Framework for Privacy-Preserving Recommendations
• FedPara: Low-Rank Hadamard Product for Communication-Efficient Federated Learning
• Accurate and Fast Federated Learning via IID and Communication-Aware Grouping
• Software Engineering for Responsible AI: An Empirical Study and Operationalised Patterns
• A Theoretical Perspective on Differentially Private Federated Multi-task Learning
• HFEL: Joint Edge Association and Resource Allocation for Cost-Efficient Hierarchical Federated Edge Learning
• What Do We Mean by Generalization in Federated Learning?
• Rethinking Architecture Design for Tackling Data Heterogeneity in Federated Learning
• On the Impact of Device and Behavioral Heterogeneity in Federated Learning
• FedDR -- Randomized Douglas-Rachford Splitting Algorithms for Nonconvex Federated Composite Optimization
• Federated Hyperparameter Tuning: Challenges, Baselines, and Connections to Weight-Sharing
• Efficient and Private Federated Learning with Partially Trainable Networks
• Federated Composite Optimization
• FedSkel: Efficient Federated Learning on Heterogeneous Systems with Skeleton Gradients Update
• Characterizing Impacts of Heterogeneity in Federated Learning upon Large-Scale Smartphone Data
• Towards Unsupervised Domain Adaptation for Deep Face Recognition under Privacy Constraints via Federated Learning
• On the Practicality of Differential Privacy in Federated Learning by Tuning Iteration Times
• DRIVE: One-bit Distributed Mean Estimation
• FedMAX: Mitigating Activation Divergence for Accurate and Communication-Efficient Federated Learning
• Adaptive Federated Dropout: Improving Communication Efficiency and Generalization for Federated Learning
• Heterogeneous Data-Aware Federated Learning
• DistFL: Distribution-aware Federated Learning for Mobile Scenarios
• Federated Mixture of Experts
• Evaluation Framework For Large-scale Federated Learning
• Differentially Private Federated Learning with Laplacian Smoothing
• Adaptive Gradient Sparsification for Efficient Federated Learning: An Online Learning Approach
• Aggregate or Not? Exploring Where to Privatize in DNN Based Federated Learning Under Different Non-IID Scenes
• FedDANE: A Federated Newton-Type Method
• Convergence and Accuracy Trade-Offs in Federated Learning and Meta-Learning
• Privacy-preserving Decentralized Aggregation for Federated Learning
• Secure Byzantine-Robust Distributed Learning via Clustering
• Distributed Additive Encryption and Quantization for Privacy Preserving Federated Deep Learning
• QuPeD: Quantized Personalization via Distillation with Applications to Federated Learning
• Efficient Ring-topology Decentralized Federated Learning with Deep Generative Models for Industrial Artificial Intelligent
• Jointly Learning from Decentralized (Federated) and Centralized Data to Mitigate Distribution Shift
• Understanding the Tradeoffs in Client-side Privacy for Downstream Speech Tasks
• Addressing Class Imbalance in Federated Learning
• OpenFed: A Comprehensive and Versatile Open-Source Federated Learning Framework
• Private Multi-Task Learning: Formulation and Applications to Federated Learning
• Efficient and Less Centralized Federated Learning
• QLSD: Quantised Langevin stochastic dynamics for Bayesian federated learning
• FAT: Federated Adversarial Training
• FedPrune: Towards Inclusive Federated Learning
• Curse or Redemption? How Data Heterogeneity Affects the Robustness of Federated Learning
• DP-REC: Private & Communication-Efficient Federated Learning
• Communication-Efficient Agnostic Federated Averaging
• Dubhe: Towards Data Unbiasedness with Homomorphic Encryption in Federated Learning Client Selection
• On Large-Cohort Training for Federated Learning
• Differentially Private Federated Learning via Inexact ADMM
• Federated Learning of User Verification Models Without Sharing Embeddings
• A Survey on Edge Performance Benchmarking
• An Operator Splitting View of Federated Learning
• TiFL: A Tier-based Federated Learning System
• More Industry-friendly: Federated Learning with High Efficient Design
• EasyFL: A Low-code Federated Learning Platform For Dummies
• Accelerating Federated Learning in Heterogeneous Data and Computational Environments
• New Directions in Distributed Deep Learning: Bringing the Network at Forefront of IoT Design
• Federated Few-Shot Learning with Adversarial Learning
• On-device Federated Learning with Flower
• Two Sides of Meta-Learning Evaluation: In vs. Out of Distribution
• Faster Secure Data Mining via Distributed Homomorphic Encryption
• ADDS: Adaptive Differentiable Sampling for Robust Multi-Party Learning
• Scalable Federated Learning over Passive Optical Networks
• Syft 0.5: A Platform for Universally Deployable Structured Transparency
• Bandwidth Slicing to Boost Federated Learning in Edge Computing
• Optimizing the Numbers of Queries and Replies in Federated Learning with Differential Privacy
• Slashing Communication Traffic in Federated Learning by Transmitting Clustered Model Updates
• Design and Analysis of Uplink and Downlink Communications for Federated Learning
• Towards Explainable Multi-Party Learning: A Contrastive Knowledge Sharing Framework
• Blockchain and Federated Edge Learning for Privacy-Preserving Mobile Crowdsensing
• Adversarially Robust and Explainable Model Compression with On-Device Personalization for Text Classification
• On Data Efficiency of Meta-learning
• TESSERACT: Gradient Flip Score to Secure Federated Learning Against Model Poisoning Attacks