A Knowledge Transfer Framework for Differentially Private Sparse Learning

Abstract

We study the problem of estimating high dimensional models with underlying sparse structures while preserving the privacy of each training example. We develop a differentially private high-dimensional sparse learning framework using the idea of knowledge transfer. More specifically, we propose to distill the knowledge from a "teacher" estimator trained on a private dataset, by creating a new dataset from auxiliary features, and then train a differentially private "student" estimator using this new dataset. In addition, we establish the linear convergence rate as well as the utility guarantee for our proposed method. For sparse linear regression and sparse logistic regression, our method achieves improved utility guarantees compared with the best known results (Kifer et al., 2012; Wang and Gu, 2019). We further demonstrate the superiority of our framework through both synthetic and real-world data experiments.

arXiv comment24 pages, 2 figures, 3 tables

References in corpus (9)

• Distilling the Knowledge in a Neural Network
• Simultaneous analysis of Lasso and Dantzig selector
• A Unified Framework for High-Dimensional Analysis of M-Estimators with Decomposable Regularizers
• Regularized M-estimators with nonconvexity: Statistical and algorithmic theory for local optima
• Gradient Hard Thresholding Pursuit for Sparsity-Constrained Optimization
• Differentially Private Empirical Risk Minimization: Efficient Algorithms and Tight Error Bounds
• Nonconvex Sparse Learning via Stochastic Optimization with Progressive Variance Reduction
• Reconstruction from anisotropic random measurements
• Model-Agnostic Private Learning via Stability