Personalized Federated Learning by Structured and Unstructured Pruning under Data Heterogeneity

Abstract

The traditional approach in FL tries to learn a single global model collaboratively with the help of many clients under the orchestration of a central server. However, learning a single global model might not work well for all clients participating in the FL under data heterogeneity. Therefore, the personalization of the global model becomes crucial in handling the challenges that arise with statistical heterogeneity and the non-IID distribution of data. Unlike prior works, in this work we propose a new approach for obtaining a personalized model from a client-level objective. This further motivates all clients to participate in federation even under statistical heterogeneity in order to improve their performance, instead of merely being a source of data and model training for the central server. To realize this personalization, we leverage finding a small subnetwork for each client by applying hybrid pruning (combination of structured and unstructured pruning), and unstructured pruning. Through a range of experiments on different benchmarks, we observed that the clients with similar data (labels) share similar personal parameters. By finding a subnetwork for each client ...

References in corpus (14)

• Communication-Efficient Learning of Deep Networks from Decentralized Data
• Federated Learning: Strategies for Improving Communication Efficiency
• Federated Learning with Non-IID Data
• The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
• Federated Learning for Mobile Keyboard Prediction
• Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training
• Pruning Filters for Efficient ConvNets
• Federated Optimization in Heterogeneous Networks
• Improving Federated Learning Personalization via Model Agnostic Meta Learning
• Three Approaches for Personalization with Applications to Federated Learning
• Federated Meta-Learning with Fast Convergence and Efficient Communication
• Federated Learning of a Mixture of Global and Local Models
• Think Locally, Act Globally: Federated Learning with Local and Global Representations
• Salvaging Federated Learning by Local Adaptation