Think Locally, Act Globally: Federated Learning with Local and Global Representations

Abstract

Federated learning is a method of training models on private data distributed over multiple devices. To keep device data private, the global model is trained by only communicating parameters and updates which poses scalability challenges for large models. To this end, we propose a new federated learning algorithm that jointly learns compact local representations on each device and a global model across all devices. As a result, the global model can be smaller since it only operates on local representations, reducing the number of communicated parameters. Theoretically, we provide a generalization analysis which shows that a combination of local and global models reduces both variance in the data as well as variance across device distributions. Empirically, we demonstrate that local models enable communication-efficient training while retaining performance. We also evaluate on the task of personalized mood prediction from real-world mobile data where privacy is key. Finally, local models handle heterogeneous data from new devices, and learn fair representations that obfuscate protected attributes such as race, age, and gender.

NeurIPS 2019 Workshop on Federated Learning distinguished student paper award. Code: https://github.com/pliang279/LG-FedAvg

References in corpus (14)

• Towards Federated Learning at Scale: System Design
• Compressing Neural Networks with the Hashing Trick
• Agnostic Federated Learning
• The generalization error of random features regression: Precise asymptotics and double descent curve
• Threats to Federated Learning: A Survey
• Expanding the Reach of Federated Learning by Reducing Client Resource Requirements
• One-Shot Federated Learning
• Joint Multimodal Learning with Deep Generative Models
• Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limit
• Gender Bias in Contextualized Word Embeddings
• Improved Adversarial Learning for Fair Classification
• Learning Not to Learn in the Presence of Noisy Labels
• Hardware/Software Security Patches for Internet of Trillions of Things
• Quantized Epoch-SGD for Communication-Efficient Distributed Learning

Cited by in corpus (47)

• Towards Personalized Federated Learning
• Adaptive Personalized Federated Learning
• Three Approaches for Personalization with Applications to Federated Learning
• Ditto: Fair and Robust Federated Learning Through Personalization
• HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clients
• Label-Efficient Self-Supervised Federated Learning for Tackling Data Heterogeneity in Medical Imaging
• A Field Guide to Federated Optimization
• VAFL: a Method of Vertical Asynchronous Federated Learning
• LotteryFL: Personalized and Communication-Efficient Federated Learning with Lottery Ticket Hypothesis on Non-IID Datasets
• Federated Mutual Learning
• Exploiting Shared Representations for Personalized Federated Learning
• FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropout
• The Internet of Federated Things (IoFT): A Vision for the Future and In-depth Survey of Data-driven Approaches for Federated Learning
• FedBABU: Towards Enhanced Representation for Federated Image Classification
• Mitigating Bias in Federated Learning
• No Fear of Heterogeneity: Classifier Calibration for Federated Learning with Non-IID Data
• Hypernetwork-based Personalized Federated Learning for Multi-Institutional CT Imaging
• SemiFL: Semi-Supervised Federated Learning for Unlabeled Clients with Alternate Training
• Personalized Federated Learning with First Order Model Optimization
• Federated Reconstruction: Partially Local Federated Learning
• MultiBench: Multiscale Benchmarks for Multimodal Representation Learning
• Personalized Federated Learning through Local Memorization
• Rethinking Architecture Design for Tackling Data Heterogeneity in Federated Learning
• Personalized Federated Learning with Gaussian Processes
• FedSkel: Efficient Federated Learning on Heterogeneous Systems with Skeleton Gradients Update
• Towards Open-World Feature Extrapolation: An Inductive Graph Learning Approach
• DistFL: Distribution-aware Federated Learning for Mobile Scenarios
• Biases in Data Science Lifecycle
• Federated Mixture of Experts
• Heterogeneous Data-Aware Federated Learning
• Aggregate or Not? Exploring Where to Privatize in DNN Based Federated Learning Under Different Non-IID Scenes
• Federated Learning on Non-IID Data: A Survey
• SoK: Training Machine Learning Models over Multiple Sources with Privacy Preservation
• Personalized Federated Learning: A Unified Framework and Universal Optimization Techniques
• Personalized Federated Learning by Structured and Unstructured Pruning under Data Heterogeneity
• Understanding the Tradeoffs in Client-side Privacy for Downstream Speech Tasks
• Multimodal Privacy-preserving Mood Prediction from Mobile Data: A Preliminary Study
• Decentralised Person Re-Identification with Selective Knowledge Aggregation
• Federated Learning for Open Banking
• Data Selection for Efficient Model Update in Federated Learning
• New Metrics to Evaluate the Performance and Fairness of Personalized Federated Learning
• More Industry-friendly: Federated Learning with High Efficient Design
• Cross-Node Federated Graph Neural Network for Spatio-Temporal Data Modeling
• Preliminary Steps Towards Federated Sentiment Classification
• Local Learning at the Network Edge for Efficient & Secure Real-Time Predictive Analytics
• Volumization as a Natural Generalization of Weight Decay
• Multi-task Federated Edge Learning (MtFEEL) in Wireless Networks