Differential Privacy Has Disparate Impact on Model Accuracy

Abstract

Differential privacy (DP) is a popular mechanism for training machine learning models with bounded leakage about the presence of specific points in the training data. The cost of differential privacy is a reduction in the model's accuracy. We demonstrate that in the neural networks trained using differentially private stochastic gradient descent (DP-SGD), this cost is not borne equally: accuracy of DP models drops much more for the underrepresented classes and subgroups. For example, a gender classification model trained using DP-SGD exhibits much lower accuracy for black faces than for white faces. Critically, this gap is bigger in the DP model than in the non-DP model, i.e., if the original model is unfair, the unfairness becomes worse once DP is applied. We demonstrate this effect for a variety of tasks and models, including sentiment analysis of text and image classification. We then explain why DP training mechanisms such as gradient clipping and noise addition have disproportionate effect on the underrepresented and more complex subgroups, resulting in a disparate reduction of model accuracy.

References in corpus (15)

• SMOTE: Synthetic Minority Over-sampling Technique
• Deep Learning with Differential Privacy
• On the difficulty of training Recurrent Neural Networks
• A systematic study of the class imbalance problem in convolutional neural networks
• Equality of Opportunity in Supervised Learning
• Renyi Differential Privacy
• Learning Differentially Private Recurrent Language Models
• Differentially Private Federated Learning: A Client Level Perspective
• Preventing Fairness Gerrymandering: Auditing and Learning for Subgroup Fairness
• Data Decisions and Theoretical Implications when Adversarially Learning Fair Representations
• Agnostic Federated Learning
• Adding Gradient Noise Improves Learning for Very Deep Networks
• A General Approach to Adding Differential Privacy to Iterative Training Procedures
• Federated Heavy Hitters Discovery with Differential Privacy
• Fair Decision Making using Privacy-Protected Data

Cited by in corpus (15)

• How Different Groups Prioritize Ethical Values for Responsible AI
• A Framework for Understanding Sources of Harm throughout the Machine Learning Life Cycle
• Robust Anomaly Detection and Backdoor Attack Detection Via Differential Privacy
• Equity and Privacy: More Than Just a Tradeoff
• Uncovering Bias in Personal Informatics
• Fair Decision Making using Privacy-Protected Data
• Open Government Data Programs and Information Privacy Concerns: A Literature Review
• Fairness-aware Differentially Private Collaborative Filtering
• Accuracy, Interpretability, and Differential Privacy via Explainable Boosting
• Defending Medical Image Diagnostics against Privacy Attacks using Generative Methods
• Transitioning from Real to Synthetic data: Quantifying the bias in model
• Integral Privacy for Sampling
• Partial sensitivity analysis in differential privacy
• Identifying and Exploiting Structures for Reliable Deep Learning
• Reducing Risk of Model Inversion Using Privacy-Guided Training