Public Data-Assisted Mirror Descent for Private Model Training

Abstract

In this paper, we revisit the problem of using in-distribution public data to improve the privacy/utility trade-offs for differentially private (DP) model training. (Here, public data refers to auxiliary data sets that have no privacy concerns.) We design a natural variant of DP mirror descent, where the DP gradients of the private/sensitive data act as the linear term, and the loss generated by the public data as the mirror map. We show that, for linear regression with feature vectors drawn from a non-isotropic sub-Gaussian distribution, our algorithm, PDA-DPMD (a variant of mirror descent), provides population risk guarantees that are asymptotically better than the best known guarantees under DP (without having access to public data), when the number of public data samples ($n_{\mathsf{pub}}$) is sufficiently large. We further show that our algorithm has natural "noise stability" properties that control the variance due to noise added to ensure DP. We demonstrate the efficacy of our algorithm by showing privacy/utility trade-offs on four benchmark datasets (StackOverflow, WikiText-2, CIFAR-10, and EMNIST). We show that our algorithm not only significantly improves over traditional DP-SGD, which does not have access to public data, but to our knowledge is the first to improve over DP-SGD on models that have been pre-trained with public data.

20 pages, 8 figures, 3 tables

References in corpus (7)

• Gradient Descent Happens in a Tiny Subspace
• Large Language Models Can Be Strong Differentially Private Learners
• Differentially Private Fine-tuning of Language Models
• Tempered Sigmoid Activations for Deep Learning with Differential Privacy
• Limits of Private Learning with Access to Public Data
• Private Adaptive Gradient Methods for Convex Optimization
• Learning from Mixtures of Private and Public Populations

Cited by in corpus (1)

• How to DP-fy ML: A Practical Guide to Machine Learning with Differential Privacy