A Theoretical Perspective on Differentially Private Federated Multi-task Learning

Abstract

In the era of big data, the need to expand the amount of data through data sharing to improve model performance has become increasingly compelling. As a result, effective collaborative learning models need to be developed with respect to both privacy and utility concerns. In this work, we propose a new federated multi-task learning method for effective parameter transfer with differential privacy to protect gradients at the client level. Specifically, the lower layers of the networks are shared across all clients to capture transferable feature representation, while top layers of the network are task-specific for on-client personalization. Our proposed algorithm naturally resolves the statistical heterogeneity problem in federated networks. We are, to the best of knowledge, the first to provide both privacy and utility guarantees for such a proposed federated algorithm. The convergences are proved for the cases with Lipschitz smooth objective functions under the non-convex, convex, and strongly convex settings. Empirical experiment results on different datasets have been conducted to demonstrate the effectiveness of the proposed algorithm and verify the implications of the theoretical findings.

References in corpus (17)

• Deep Learning with Differential Privacy
• On the Convergence of FedAvg on Non-IID Data
• A Survey on Multi-Task Learning
• Differentially Private Federated Learning: A Client Level Perspective
• FedMD: Heterogenous Federated Learning via Model Distillation
• Federated Optimization in Heterogeneous Networks
• Three Approaches for Personalization with Applications to Federated Learning
• LEAF: A Benchmark for Federated Settings
• Practical Secure Aggregation for Federated Learning on User-Held Data
• On the Convergence of Local Descent Methods in Federated Learning
• Gaussian Differential Privacy
• First Analysis of Local GD on Heterogeneous Data
• Differentially Private Bayesian Learning on Distributed Data
• Wireless Federated Learning with Local Differential Privacy
• Deep Learning with Gaussian Differential Privacy
• Differentially Private Federated Learning with Laplacian Smoothing
• Randomized Fast Subspace Descent Methods

Cited by in corpus (4)

• Privacy-Preserving Federated Learning with Differentially Private Hyperdimensional Computing
• A New Dimensionality Reduction Method Based on Hensel's Compression for Privacy Protection in Federated Learning
• Private Multi-Task Learning: Formulation and Applications to Federated Learning
• An Energy Consumption Model for Electrical Vehicle Networks via Extended Federated-learning