Differentially Private Meta-Learning

Abstract

Parameter-transfer is a well-known and versatile approach for meta-learning, with applications including few-shot learning, federated learning, and reinforcement learning. However, parameter-transfer algorithms often require sharing models that have been trained on the samples from specific tasks, thus leaving the task-owners susceptible to breaches of privacy. We conduct the first formal study of privacy in this setting and formalize the notion of task-global differential privacy as a practical relaxation of more commonly studied threat models. We then propose a new differentially private algorithm for gradient-based parameter transfer that not only satisfies this privacy requirement but also retains provable transfer learning guarantees in convex settings. Empirically, we apply our analysis to the problems of federated learning with personalization and few-shot classification, showing that allowing the relaxation to task-global privacy from the more commonly studied notion of local privacy leads to dramatically increased performance in recurrent neural language modeling and image classification.

References in corpus (7)

• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Communication-Efficient Learning of Deep Networks from Decentralized Data
• On First-Order Meta-Learning Algorithms
• Differentially Private Federated Learning: A Client Level Perspective
• Protection Against Reconstruction and Its Applications in Private Federated Learning
• Online Meta-Learning
• Learning-to-Learn Stochastic Gradient Descent with Biased Regularization

Cited by in corpus (9)

• Personalized Federated Learning: A Meta-Learning Approach
• A Comprehensive Survey on Local Differential Privacy Toward Data Statistics and Analysis
• Federated Learning with Differential Privacy: Algorithms and Performance Analysis
• Privacy-Preserving Machine Learning: Methods, Challenges and Directions
• Robust Federated Learning: The Case of Affine Distribution Shifts
• Privacy for Free: Communication-Efficient Learning with Differential Privacy Using Sketches
• A Comprehensive Overview and Survey of Recent Advances in Meta-Learning
• Federated $f$-Differential Privacy
• SoK: Training Machine Learning Models over Multiple Sources with Privacy Preservation