Towards Federated Learning at Scale: System Design

Abstract

Federated Learning is a distributed machine learning approach which enables model training on a large corpus of decentralized data. We have built a scalable production system for Federated Learning in the domain of mobile devices, based on TensorFlow. In this paper, we describe the resulting high-level design, sketch some of the challenges and their solutions, and touch upon the open problems and future directions.

References in corpus (10)

• Communication-Efficient Learning of Deep Networks from Decentralized Data
• Federated Learning: Strategies for Improving Communication Efficiency
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Federated Optimization: Distributed Machine Learning for On-Device Intelligence
• Federated Learning for Mobile Keyboard Prediction
• How To Backdoor Federated Learning
• Learning Differentially Private Recurrent Language Models
• Applied Federated Learning: Improving Google Keyboard Query Suggestions
• Expanding the Reach of Federated Learning by Reducing Client Resource Requirements
• Differentially-Private "Draw and Discard" Machine Learning

Cited by in corpus (233)

• Convergence of Edge Computing and Deep Learning: A Comprehensive Survey
• A Survey on Federated Learning Systems: Vision, Hype and Reality for Data Privacy and Protection
• All One Needs to Know about Metaverse: A Complete Survey on Technological Singularity, Virtual Ecosystem, and Research Agenda
• Robust Aggregation for Federated Learning
• How To Backdoor Federated Learning
• Privacy-preserving Traffic Flow Prediction: A Federated Learning Approach
• Ensemble Distillation for Robust Model Fusion in Federated Learning
• Federated Optimization in Heterogeneous Networks
• Asynchronous Federated Optimization
• Precision Health Data: Requirements, Challenges and Existing Techniques for Data Security and Privacy
• Sustainable AI: Environmental Implications, Challenges and Opportunities
• Multi-Armed Bandit Based Client Scheduling for Federated Learning
• UVeQFed: Universal Vector Quantization for Federated Learning
• Federated Meta-Learning with Fast Convergence and Efficient Communication
• Inverting Gradients -- How easy is it to break privacy in federated learning?
• BrainTorrent: A Peer-to-Peer Environment for Decentralized Federated Learning
• Federated Learning with Bayesian Differential Privacy
• Federated Neural Collaborative Filtering
• HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clients
• Group Knowledge Transfer: Federated Learning of Large CNNs at the Edge
• Federated Learning: A Signal Processing Perspective
• Federated Learning and Differential Privacy: Software tools analysis, the Sherpa.ai FL framework and methodological guidelines for preserving data privacy
• Privacy Preserving Distributed Machine Learning with Federated Learning
• Federated Learning: Opportunities and Challenges
• Federated Learning in Mobile Edge Networks: A Comprehensive Survey
• Do We Really Need to Access the Source Data? Source Hypothesis Transfer for Unsupervised Domain Adaptation
• Think Locally, Act Globally: Federated Learning with Local and Global Representations
• Decentralized Federated Learning: A Segmented Gossip Approach
• Backdoor Attacks and Countermeasures on Deep Learning: A Comprehensive Review
• Dual Attention-Based Federated Learning for Wireless Traffic Prediction
• Adaptive Federated Optimization
• Battery-constrained Federated Edge Learning in UAV-enabled IoT for B5G/6G Networks
• Edge Intelligence: Architectures, Challenges, and Applications
• IBM Federated Learning: an Enterprise Framework White Paper V0.1
• Topology-aware Federated Learning in Edge Computing: A Comprehensive Survey
• Abnormal Client Behavior Detection in Federated Learning
• Attack of the Tails: Yes, You Really Can Backdoor Federated Learning
• 6G White paper: Research challenges for Trust, Security and Privacy
• Flower: A Friendly Federated Learning Research Framework
• Federated Learning with Matched Averaging
• Accelerating Federated Learning over Reliability-Agnostic Clients in Mobile Edge Computing Systems
• Mime: Mimicking Centralized Stochastic Algorithms in Federated Learning
• No One Left Behind: Inclusive Federated Learning over Heterogeneous Devices
• FedSplit: An algorithmic framework for fast federated optimization
• Dynamic Sampling and Selective Masking for Communication-Efficient Federated Learning
• Advancing COVID-19 Diagnosis with Privacy-Preserving Collaboration in Artificial Intelligence
• Client Selection in Federated Learning: Convergence Analysis and Power-of-Choice Selection Strategies
• An Overview of Federated Learning at the Edge and Distributed Ledger Technologies for Robotic and Autonomous Systems
• DQRE-SCnet: A novel hybrid approach for selecting users in Federated Learning with Deep-Q-Reinforcement Learning based on Spectral Clustering
• Real-World Image Datasets for Federated Learning
• FedSAE: A Novel Self-Adaptive Federated Learning Framework in Heterogeneous Systems
• Federated Learning for Healthcare Informatics
• Federated Multi-view Matrix Factorization for Personalized Recommendations
• A Systematic Survey of Blockchained Federated Learning
• PyVertical: A Vertical Federated Learning Framework for Multi-headed SplitNN
• Stratified cross-validation for unbiased and privacy-preserving federated learning
• Towards Utilizing Unlabeled Data in Federated Learning: A Survey and Prospective
• Private Federated Learning with Domain Adaptation
• Privacy-Preserving Blockchain-Based Federated Learning for IoT Devices
• Energy Efficient Federated Learning Over Wireless Communication Networks
• Federated Learning via Over-the-Air Computation
• Federated Learning for Big Data: A Survey on Opportunities, Applications, and Future Directions
• OnSlicing: Online End-to-End Network Slicing with Reinforcement Learning
• Linear Convergence in Federated Learning: Tackling Client Heterogeneity and Sparse Gradients
• Orthogonal Gradient Descent for Continual Learning
• Federated Learning via Posterior Averaging: A New Perspective and Practical Algorithms
• Triggerflow: Trigger-based Orchestration of Serverless Workflows
• Robust Blockchained Federated Learning with Model Validation and Proof-of-Stake Inspired Consensus
• SemiFL: Semi-Supervised Federated Learning for Unlabeled Clients with Alternate Training
• Federated Learning with Buffered Asynchronous Aggregation
• Faster On-Device Training Using New Federated Momentum Algorithm
• Towards Ubiquitous AI in 6G with Federated Learning
• Data-Quality Based Scheduling for Federated Edge Learning
• Adversarial Examples Make Strong Poisons
• Improving Fairness via Federated Learning
• Towards Flexible Device Participation in Federated Learning
• Distributed Training with Heterogeneous Data: Bridging Median- and Mean-Based Algorithms
• Efficient Adaptive Ensembling for Image Classification
• Hierarchical Federated Learning through LAN-WAN Orchestration
• ElasticTrainer: Speeding Up On-Device Training with Runtime Elastic Tensor Selection
• Towards Non-I.I.D. and Invisible Data with FedNAS: Federated Deep Learning via Neural Architecture Search
• FedVision: An Online Visual Object Detection Platform Powered by Federated Learning
• Papaya: Practical, Private, and Scalable Federated Learning
• FedBE: Making Bayesian Model Ensemble Applicable to Federated Learning
• A Vertical Federated Learning Framework for Graph Convolutional Network
• FedAT: A High-Performance and Communication-Efficient Federated Learning System with Asynchronous Tiers
• Federated Few-Shot Learning for Mobile NLP
• Co-clustering for Federated Recommender System
• How to Put Users in Control of their Data in Federated Top-N Recommendation with Learning to Rank
• Personalized Federated Learning with First Order Model Optimization
• Privacy Preservation in Federated Learning: An insightful survey from the GDPR Perspective
• TornadoAggregate: Accurate and Scalable Federated Learning via the Ring-Based Architecture
• Practical Defences Against Model Inversion Attacks for Split Neural Networks
• The Distributed Discrete Gaussian Mechanism for Federated Learning with Secure Aggregation
• Deep Leakage from Gradients
• FedGEMS: Federated Learning of Larger Server Models via Selective Knowledge Fusion
• LDP-Fed: Federated Learning with Local Differential Privacy
• A Federated Learning Approach for Mobile Packet Classification
• Federated Dynamic GNN with Secure Aggregation
• FEDZIP: A Compression Framework for Communication-Efficient Federated Learning
• Federated Reconstruction: Partially Local Federated Learning
• Practical and Private (Deep) Learning without Sampling or Shuffling
• Edge-cloud Collaborative Learning with Federated and Centralized Features
• Efficient, high-performance pancreatic segmentation using multi-scale feature extraction
• Learning, Computing, and Trustworthiness in Intelligent IoT Environments: Performance-Energy Tradeoffs
• Helios: Heterogeneity-Aware Federated Learning with Dynamically Balanced Collaboration
• Preserving Patient Privacy while Training a Predictive Model of In-hospital Mortality
• On the Utility of Gradient Compression in Distributed Training Systems
• Understanding Unintended Memorization in Federated Learning
• Personalized Federated Deep Learning for Pain Estimation From Face Images
• Asynchronous Federated Learning with Reduced Number of Rounds and with Differential Privacy from Less Aggregated Gaussian Noise
• Federated Learning for Smart Healthcare: A Survey
• Natural Backdoor Attacks on Speech Recognition Models
• Federated Over-Air Subspace Tracking from Incomplete and Corrupted Data
• FedV: Privacy-Preserving Federated Learning over Vertically Partitioned Data
• Training Production Language Models without Memorizing User Data
• Bayesian Federated Learning over Wireless Networks
• Device Heterogeneity in Federated Learning: A Superquantile Approach
• Personalized Federated Learning for Heterogeneous Clients with Clustered Knowledge Transfer
• Federated Learning with Superquantile Aggregation for Heterogeneous Data
• Fast-adapting and Privacy-preserving Federated Recommender System
• Gradient Statistics Aware Power Control for Over-the-Air Federated Learning
• FedLoc: Federated Learning Framework for Data-Driven Cooperative Localization and Location Data Processing
• Proof of Federated Learning: A Novel Energy-recycling Consensus Algorithm
• Joint Device Scheduling and Resource Allocation for Latency Constrained Wireless Federated Learning
• Orchestrating the Development Lifecycle of Machine Learning-Based IoT Applications: A Taxonomy and Survey
• Ternary Compression for Communication-Efficient Federated Learning
• Dynamic backdoor attacks against federated learning
• JSDoop and TensorFlow.js: Volunteer Distributed Web Browser-Based Neural Network Training
• Applications of Federated Learning in Smart Cities: Recent Advances, Taxonomy, and Open Challenges
• Efficient and Private Federated Learning with Partially Trainable Networks
• A Vertical Federated Learning Framework for Horizontally Partitioned Labels
• Substra: a framework for privacy-preserving, traceable and collaborative Machine Learning
• Fed-BioMed: Open, Transparent and Trusted Federated Learning for Real-world Healthcare Applications
• Privacy-Preserving Blockchain Based Federated Learning with Differential Data Sharing
• Neurosciences and 6G: Lessons from and Needs of Communicative Brains
• Federated Mixture of Experts
• Privacy-Preserving Multi-Party Contextual Bandits
• Stochastic Channel-Based Federated Learning for Medical Data Privacy Preserving
• Loss Tolerant Federated Learning
• ByGARS: Byzantine SGD with Arbitrary Number of Attackers
• FedSel: Federated SGD under Local Differential Privacy with Top-k Dimension Selection
• Federated Learning on Non-IID Data: A Survey
• Revocable Federated Learning: A Benchmark of Federated Forest
• FedFm: Towards a Robust Federated Learning Approach For Fault Mitigation at the Edge Nodes
• RPT: Relational Pre-trained Transformer Is Almost All You Need towards Democratizing Data Preparation
• Federated Noisy Client Learning
• Towards On-Device Federated Learning: A Direct Acyclic Graph-based Blockchain Approach
• SAPAG: A Self-Adaptive Privacy Attack From Gradients
• Federated Echo State Learning for Minimizing Breaks in Presence in Wireless Virtual Reality Networks
• Secure Byzantine-Robust Distributed Learning via Clustering
• Two-stage Federated Phenotyping and Patient Representation Learning
• SECure: A Social and Environmental Certificate for AI Systems
• EigenGame: PCA as a Nash Equilibrium
• OCTOPUS: Overcoming Performance andPrivatization Bottlenecks in Distributed Learning
• Federated Block Coordinate Descent Scheme for Learning Global and Personalized Models
• ESMFL: Efficient and Secure Models for Federated Learning
• Privacy Preserving Stochastic Channel-Based Federated Learning with Neural Network Pruning
• Good AI for the Present of Humanity Democratizing AI Governance
• Byzantine-resilient Decentralized Stochastic Gradient Descent
• FedSEAL: Semi-Supervised Federated Learning with Self-Ensemble Learning and Negative Learning
• Federated Action Recognition on Heterogeneous Embedded Devices
• OpenFed: A Comprehensive and Versatile Open-Source Federated Learning Framework
• An Exploratory Analysis on Users' Contributions in Federated Learning
• Prophet: Proactive Candidate-Selection for Federated Learning by Predicting the Qualities of Training and Reporting Phases
• TIFeD: a Tiny Integer-based Federated learning algorithm with Direct feedback alignment
• WAFFLE: Watermarking in Federated Learning
• Scheduling for Cellular Federated Edge Learning with Importance and Channel Awareness
• Communication Efficient Federated Learning with Energy Awareness over Wireless Networks
• MAB-based Client Selection for Federated Learning with Uncertain Resources in Mobile Networks
• On the Convergence of Nested Decentralized Gradient Methods with Multiple Consensus and Gradient Steps
• Performance Analysis and Characterization of Training Deep Learning Models on Mobile Devices
• FedPrune: Towards Inclusive Federated Learning
• Linear Regression over Networks with Communication Guarantees
• Enabling On-Device Self-Supervised Contrastive Learning With Selective Data Contrast
• Coded Computing for Master-Aided Distributed Computing Systems
• Exploiting Heterogeneity in Robust Federated Best-Arm Identification
• Data Selection for Efficient Model Update in Federated Learning
• Curse or Redemption? How Data Heterogeneity Affects the Robustness of Federated Learning
• FAT: Federated Adversarial Training
• Sisyphus: A Cautionary Tale of Using Low-Degree Polynomial Activations in Privacy-Preserving Deep Learning
• Ethics of Artificial Intelligence in Surgery
• Multi-Task Federated Reinforcement Learning with Adversaries
• FedDropoutAvg: Generalizable federated learning for histopathology image classification
• FLRA: A Reference Architecture for Federated Learning Systems
• Separation of Powers in Federated Learning
• Accelerating MoE Model Inference with Expert Sharding
• Practical Federated Learning without a Server
• Federated Extra-Trees with Privacy Preserving
• ABC-FL: Anomalous and Benign client Classification in Federated Learning
• Federated Edge Learning with Misaligned Over-The-Air Computation
• Hogwild! over Distributed Local Data Sets with Linearly Increasing Mini-Batch Sizes
• Optimal Complexity in Decentralized Training
• A Survey on Large-scale Machine Learning
• Adaptive Distillation for Decentralized Learning from Heterogeneous Clients
• Accelerating Federated Learning in Heterogeneous Data and Computational Environments
• ResiliNet: Failure-Resilient Inference in Distributed Neural Networks
• TiFL: A Tier-based Federated Learning System
• Federated Learning using Smart Contracts on Blockchains, based on Reward Driven Approach
• Optimizing Resource-Efficiency for Federated Edge Intelligence in IoT Networks
• Slashing Communication Traffic in Federated Learning by Transmitting Clustered Model Updates
• Data Poisoning and Leakage Analysis in Federated Learning
• MDLdroid: a ChainSGD-reduce Approach to Mobile Deep Learning for Personal Mobile Sensing
• Dropout against Deep Leakage from Gradients
• Lightweight Transformer in Federated Setting for Human Activity Recognition
• FGLP: A Federated Fine-Grained Location Prediction System for Mobile Users
• Probabilistic Federated Learning of Neural Networks Incorporated with Global Posterior Information
• Optimizing Over-the-Air Computation in IRS-Aided C-RAN Systems
• A Comparative Analysis of Machine Learning Algorithms for Intrusion Detection in Edge-Enabled IoT Networks
• Towards Scheduling Federated Deep Learning using Meta-Gradients for Inter-Hospital Learning
• Corella: A Private Multi Server Learning Approach based on Correlated Queries
• Small-Group Learning, with Application to Neural Architecture Search
• Edge Artificial Intelligence for 6G: Vision, Enabling Technologies, and Applications
• HADFL: Heterogeneity-aware Decentralized Federated Learning Framework
• Acceleration of Cooperative Least Mean Square via Chebyshev Periodical Successive Over-Relaxation
• Mixing Deep Learning and Multiple Criteria Optimization: An Application to Distributed Learning with Multiple Datasets
• Budgeted Online Selection of Candidate IoT Clients to Participate in Federated Learning
• Distributed Learning and its Application for Time-Series Prediction
• Consensus Driven Learning
• OL4EL: Online Learning for Edge-cloud Collaborative Learning on Heterogeneous Edges with Resource Constraints
• A Joint Energy and Latency Framework for Transfer Learning over 5G Industrial Edge Networks
• Preventing Adversarial Use of Datasets through Fair Core-Set Construction
• Prior-Independent Auctions for the Demand Side of Federated Learning
• AutoFL: Enabling Heterogeneity-Aware Energy Efficient Federated Learning
• Enabling Design Methodologies and Future Trends for Edge AI: Specialization and Co-design
• Sum-Rate-Distortion Function for Indirect Multiterminal Source Coding in Federated Learning
• Canoe : A System for Collaborative Learning for Neural Nets
• Federated Classification using Parsimonious Functions in Reproducing Kernel Hilbert Spaces
• DEAL: Decremental Energy-Aware Learning in a Federated System
• Pronto: Federated Task Scheduling
• $C^{3}DRec$: Cloud-Client Cooperative Deep Learning for Temporal Recommendation in the Post-GDPR Era
• Management of Resource at the Network Edge for Federated Learning
• Multi-task Federated Edge Learning (MtFEEL) in Wireless Networks