Exploiting Shared Representations for Personalized Federated Learning

Abstract

Deep neural networks have shown the ability to extract universal feature representations from data such as images and text that have been useful for a variety of learning tasks. However, the fruits of representation learning have yet to be fully-realized in federated settings. Although data in federated settings is often non-i.i.d. across clients, the success of centralized deep learning suggests that data often shares a global feature representation, while the statistical heterogeneity across clients or tasks is concentrated in the labels. Based on this intuition, we propose a novel federated learning framework and algorithm for learning a shared data representation across clients and unique local heads for each client. Our algorithm harnesses the distributed computational power across clients to perform many local-updates with respect to the low-dimensional local parameters for every update of the representation. We prove that this method obtains linear convergence to the ground-truth representation with near-optimal sample complexity in a linear setting, demonstrating that it can efficiently reduce the problem dimension for each client. This result is of interest beyond federated learning to a broad class of problems in which we aim to learn a shared low-dimensional representation among data distributions, for example in meta-learning and multi-task learning. Further, extensive experimental results show the empirical improvement of our method over alternative personalized federated learning approaches in federated environments with heterogeneous data.

References in corpus (23)

• Communication-Efficient Learning of Deep Networks from Decentralized Data
• Federated Learning with Personalization Layers
• Federated Optimization in Heterogeneous Networks
• Nonconvex Optimization Meets Low-Rank Matrix Factorization: An Overview
• Improving Federated Learning Personalization via Model Agnostic Meta Learning
• Adaptive Personalized Federated Learning
• Three Approaches for Personalization with Applications to Federated Learning
• LEAF: A Benchmark for Federated Settings
• Federated Meta-Learning with Fast Convergence and Efficient Communication
• Federated Learning of a Mixture of Global and Local Models
• Ditto: Fair and Robust Federated Learning Through Personalization
• Think Locally, Act Globally: Federated Learning with Local and Global Representations
• Adaptive Federated Optimization
• Convergence Analysis for Rectangular Matrix Completion Using Burer-Monteiro Factorization and Gradient Descent
• The Benefit of Multitask Representation Learning
• Rapid Learning or Feature Reuse? Towards Understanding the Effectiveness of MAML
• Random design analysis of ridge regression
• FedSplit: An algorithmic framework for fast federated optimization
• Salvaging Federated Learning by Local Adaptation
• On the Theory of Transfer Learning: The Importance of Task Diversity
• Few-Shot Learning via Learning the Representation, Provably
• Provable Meta-Learning of Linear Representations
• Federated Learning with Compression: Unified Analysis and Sharp Guarantees

Cited by in corpus (10)

• FedALA: Adaptive Local Aggregation for Personalized Federated Learning
• FedBABU: Towards Enhanced Representation for Federated Image Classification
• Behavioral Authentication for Security and Safety
• CDKT-FL: Cross-Device Knowledge Transfer using Proxy Dataset in Federated Learning
• Device Heterogeneity in Federated Learning: A Superquantile Approach
• Federated Learning with Superquantile Aggregation for Heterogeneous Data
• Exploiting Heterogeneity in Robust Federated Best-Arm Identification
• Optimal Multitask Linear Regression and Contextual Bandits under Sparse Heterogeneity
• WAFFLE: Weighted Averaging for Personalized Federated Learning
• Sample Efficient Linear Meta-Learning by Alternating Minimization