Comprehensive Privacy Analysis of Deep Learning: Passive and Active White-box Inference Attacks against Centralized and Federated Learning

Abstract

Deep neural networks are susceptible to various inference attacks as they remember information about their training data. We design white-box inference attacks to perform a comprehensive privacy analysis of deep learning models. We measure the privacy leakage through parameters of fully trained models as well as the parameter updates of models during training. We design inference algorithms for both centralized and federated learning, with respect to passive and active inference attackers, and assuming different adversary prior knowledge. We evaluate our novel white-box membership inference attacks against deep learning algorithms to trace their training data records. We show that a straightforward extension of the known black-box attacks to the white-box setting (through analyzing the outputs of activation functions) is ineffective. We therefore design new algorithms tailored to the white-box setting by exploiting the privacy vulnerabilities of the stochastic gradient descent algorithm, which is the algorithm used to train deep neural networks. We investigate the reasons why deep learning models may leak information about their training data. We then show that even well-generalized models are significantly susceptible to white-box membership inference attacks, by analyzing state-of-the-art pre-trained and publicly available models for the CIFAR dataset. We also show how adversarial participants, in the federated learning setting, can successfully run active membership inference attacks against other participants, even when the global model achieves high prediction accuracies.

2019 IEEE Symposium on Security and Privacy (SP)

References in corpus (13)

• Deep Learning with Differential Privacy
• Communication-Efficient Learning of Deep Networks from Decentralized Data
• Federated Learning: Strategies for Improving Communication Efficiency
• Understanding deep learning requires rethinking generalization
• Differentially Private Empirical Risk Minimization
• The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks
• Scalable Private Learning with PATE
• Deep Models Under the GAN: Information Leakage from Collaborative Deep Learning
• Understanding Membership Inferences on Well-Generalized Learning Models
• Stealing Hyperparameters in Machine Learning
• Minimax Filter: Learning to Preserve Privacy from Inference Attacks
• Generative Adversarial Privacy
• LOGAN: Membership Inference Attacks Against Generative Models

Cited by in corpus (199)

• A Survey on Federated Learning Systems: Vision, Hype and Reality for Data Privacy and Protection
• Precision Health Data: Requirements, Challenges and Existing Techniques for Data Security and Privacy
• Federated Learning Enables Big Data for Rare Cancer Boundary Detection
• FedML: A Research Library and Benchmark for Federated Machine Learning
• HybridAlpha: An Efficient Approach for Privacy-Preserving Federated Learning
• GAN-Leaks: A Taxonomy of Membership Inference Attacks against Generative Models
• Extracting Training Data from Large Language Models
• Machine Learning for Microcontroller-Class Hardware: A Review
• Threats to Federated Learning: A Survey
• A Survey of Privacy Attacks in Machine Learning
• DeepSight: Mitigating Backdoor Attacks in Federated Learning Through Deep Model Inspection
• More Than Privacy: Applying Differential Privacy in Key Areas of Artificial Intelligence
• Privacy Risks of Securing Machine Learning Models against Adversarial Examples
• When Machine Unlearning Jeopardizes Privacy
• Zero-Shot Machine Unlearning
• Federated Continual Learning via Knowledge Fusion: A Survey
• FastSecAgg: Scalable Secure Aggregation for Privacy-Preserving Federated Learning
• Blockchain-based Federated Learning with SMPC Model Verification Against Poisoning Attack for Healthcare Systems
• From Distributed Machine Learning To Federated Learning: In The View Of Data Privacy And Security
• When Machine Learning Meets Privacy: A Survey and Outlook
• Quantifying Privacy Leakage in Graph Embedding
• Practical Blind Membership Inference Attack via Differential Comparisons
• Systematic Evaluation of Privacy Risks of Machine Learning Models
• Resource-Efficient Federated Learning
• Privacy-Preserving Machine Learning: Methods, Challenges and Directions
• Federated Learning for Healthcare Informatics
• A Study of Face Obfuscation in ImageNet
• PrivacyFL: A simulator for privacy-preserving and secure federated learning
• Federated Unlearning: A Survey on Methods, Design Guidelines, and Evaluation Metrics
• Membership Inference Attacks and Defenses in Classification Models
• Decentralized Deep Learning for Multi-Access Edge Computing: A Survey on Communication Efficiency and Trustworthiness
• Central Server Free Federated Learning over Single-sided Trust Social Networks
• TAPFed: Threshold Secure Aggregation for Privacy-Preserving Federated Learning
• Security and Privacy Issues in Deep Learning
• Privacy-Preserving Blockchain-Based Federated Learning for IoT Devices
• The Federation Strikes Back: A Survey of Federated Learning Privacy Attacks, Defenses, Applications, and Policy Landscape
• Dynamic Defense Against Byzantine Poisoning Attacks in Federated Learning
• Advances in Robust Federated Learning: A Survey with Heterogeneity Considerations
• Data Poisoning Attacks Against Federated Learning Systems
• Defending Model Inversion and Membership Inference Attacks via Prediction Purification
• Node-Level Membership Inference Attacks Against Graph Neural Networks
• Enhanced Security and Privacy via Fragmented Federated Learning
• Federated Learning Survey: A Multi-Level Taxonomy of Aggregation Techniques, Experimental Insights, and Future Frontiers
• ML-Doctor: Holistic Risk Assessment of Inference Attacks Against Machine Learning Models
• Towards Communication-efficient and Attack-Resistant Federated Edge Learning for Industrial Internet of Things
• Hierarchically Fair Federated Learning
• PPFL: Privacy-preserving Federated Learning with Trusted Execution Environments
• Social Metaverse: Challenges and Solutions
• Stealing Links from Graph Neural Networks
• ML Privacy Meter: Aiding Regulatory Compliance by Quantifying the Privacy Risks of Machine Learning
• Data Privacy and Trustworthy Machine Learning
• An Efficient Federated Learning Framework for Training Semantic Communication System
• Towards Energy-Aware Federated Learning on Battery-Powered Clients
• Eavesdrop the Composition Proportion of Training Labels in Federated Learning
• Threats and Defenses in Federated Learning Life Cycle: A Comprehensive Survey and Challenges
• Privacy-preserving Voice Analysis via Disentangled Representations
• PEPPER: Empowering User-Centric Recommender Systems over Gossip Learning
• A Survey on Federated Learning and its Applications for Accelerating Industrial Internet of Things
• Robust Aggregation for Adaptive Privacy Preserving Federated Learning in Healthcare
• Not one but many Tradeoffs: Privacy Vs. Utility in Differentially Private Machine Learning
• Practical Vertical Federated Learning with Unsupervised Representation Learning
• Debiasing Learning for Membership Inference Attacks Against Recommender Systems
• Privacy Preservation in Federated Learning: An insightful survey from the GDPR Perspective
• Sampling Attacks: Amplification of Membership Inference Attacks by Repeated Queries
• Social-Aware Clustered Federated Learning with Customized Privacy Preservation
• A Federated Learning Approach for Mobile Packet Classification
• Towards General Deep Leakage in Federated Learning
• Training Data Leakage Analysis in Language Models
• Mitigating Membership Inference Attacks by Self-Distillation Through a Novel Ensemble Architecture
• Membership Inference Attack Susceptibility of Clinical Language Models
• Feature Inference Attack on Shapley Values
• D2P-Fed: Differentially Private Federated Learning With Efficient Communication
• Helios: Heterogeneity-Aware Federated Learning with Dynamically Balanced Collaboration
• Securing Secure Aggregation: Mitigating Multi-Round Privacy Leakage in Federated Learning
• Turning Federated Learning Systems Into Covert Channels
• Shielding Federated Learning Systems against Inference Attacks with ARM TrustZone
• Diffusion Models for Tabular Data Imputation and Synthetic Data Generation
• DPIS: An Enhanced Mechanism for Differentially Private SGD with Importance Sampling
• A Differentially Private Framework for Deep Learning with Convexified Loss Functions
• Membership Inference Attacks and Generalization: A Causal Perspective
• Federated Learning for Smart Healthcare: A Survey
• Knowledge Cross-Distillation for Membership Privacy
• Anticipating Safety Issues in E2E Conversational AI: Framework and Tooling
• Secure Aggregation with Heterogeneous Quantization in Federated Learning
• Privacy Analysis of Deep Learning in the Wild: Membership Inference Attacks against Transfer Learning
• Fast-adapting and Privacy-preserving Federated Recommender System
• An Overview of Privacy in Machine Learning
• Evaluations of Machine Learning Privacy Defenses are Misleading
• Perfectly Accurate Membership Inference by a Dishonest Central Server in Federated Learning
• Membership Leakage in Label-Only Exposures
• Quantifying and Defending against Privacy Threats on Federated Knowledge Graph Embedding
• TransMIA: Membership Inference Attacks Using Transfer Shadow Training
• When Crowdsensing Meets Federated Learning: Privacy-Preserving Mobile Crowdsensing System
• Secure Aggregation for Buffered Asynchronous Federated Learning
• Achilles' Heels: Vulnerable Record Identification in Synthetic Data Publishing
• Information-theoretic generalization bounds for black-box learning algorithms
• Membership Inference on Word Embedding and Beyond
• Dordis: Efficient Federated Learning with Dropout-Resilient Differential Privacy
• A Privacy-Preserving Federated Framework with Hybrid Quantum-Enhanced Learning for Financial Fraud Detection
• A Novel Attribute Reconstruction Attack in Federated Learning
• Enhanced Membership Inference Attacks against Machine Learning Models
• A Survey of AI-Related Cyber Security Risks and Countermeasures in Mobility-as-a-Service
• Bounding Information Leakage in Machine Learning
• Security and Privacy for Artificial Intelligence: Opportunities and Challenges
• On the Impact of Device and Behavioral Heterogeneity in Federated Learning
• Layer-wise Characterization of Latent Information Leakage in Federated Learning
• Optimal query complexity for private sequential learning against eavesdropping
• A Comprehensive Analysis of Information Leakage in Deep Transfer Learning
• Privacy for All: Demystify Vulnerability Disparity of Differential Privacy against Membership Inference Attack
• Group privacy for personalized federated learning
• PriMask: Cascadable and Collusion-Resilient Data Masking for Mobile Cloud Inference
• Advancing Personalized Federated Learning: Group Privacy, Fairness, and Beyond
• SoK: Machine Learning Governance
• MixNN: Protection of Federated Learning Against Inference Attacks by Mixing Neural Network Layers
• Towards Security Threats of Deep Learning Systems: A Survey
• BlockFLow: An Accountable and Privacy-Preserving Solution for Federated Learning
• Characterizing Impacts of Heterogeneity in Federated Learning upon Large-Scale Smartphone Data
• ScionFL: Efficient and Robust Secure Quantized Aggregation
• The More, the Better? A Study on Collaborative Machine Learning for DGA Detection
• Sharing Models or Coresets: A Study based on Membership Inference Attack
• DistFL: Distribution-aware Federated Learning for Mobile Scenarios
• Anonymizing Data for Privacy-Preserving Federated Learning
• Improving LIME Robustness with Smarter Locality Sampling
• Towards Characterizing and Limiting Information Exposure in DNN Layers
• FaceLeaks: Inference Attacks against Transfer Learning Models via Black-box Queries
• DAMIA: Leveraging Domain Adaptation as a Defense against Membership Inference Attacks
• Exploring Privacy and Fairness Risks in Sharing Diffusion Models: An Adversarial Perspective
• Membership Inference Attacks Against Recommender Systems
• Privacy-preserving Federated Bayesian Learning of a Generative Model for Imbalanced Classification of Clinical Data
• NeuGuard: Lightweight Neuron-Guided Defense against Membership Inference Attacks
• Safety and Performance, Why not Both? Bi-Objective Optimized Model Compression toward AI Software Deployment
• Privacy-preserving Decentralized Aggregation for Federated Learning
• Quantifying and Mitigating Privacy Risks of Contrastive Learning
• On the Importance of Difficulty Calibration in Membership Inference Attacks
• On the Privacy Risks of Cell-Based NAS Architectures
• On the Privacy Risks of Algorithmic Fairness
• Classifying the classifier: dissecting the weight space of neural networks
• ADI: Adversarial Dominating Inputs in Vertical Federated Learning Systems
• Convergence-Privacy-Fairness Trade-Off in Personalized Federated Learning
• Is My Data in Your AI? Membership Inference Test (MINT) applied to Face Biometrics
• Federated Learning in Adversarial Settings
• High Dimensional Model Explanations: an Axiomatic Approach
• Privacy Assessment of Federated Learning using Private Personalized Layers
• Exploiting Defenses against GAN-Based Feature Inference Attacks in Federated Learning
• Deep Learning and Traffic Classification: Lessons learned from a commercial-grade dataset with hundreds of encrypted and zero-day applications
• WAFFLE: Watermarking in Federated Learning
• Enhancing Convergence, Privacy and Fairness for Wireless Personalized Federated Learning: Quantization-Assisted Min-Max Fair Scheduling
• Privacy-preserving Collaborative Learning with Automatic Transformation Search
• Combining Image Space and q-Space PDEs for Lossless Compression of Diffusion MR Images
• Source Inference Attacks in Federated Learning
• Privacy Inference Attacks and Defenses in Cloud-based Deep Neural Network: A Survey
• LazyDP: Co-Designing Algorithm-Software for Scalable Training of Differentially Private Recommendation Models
• Correlation inference attacks against machine learning models
• Automatic Fairness Testing of Neural Classifiers through Adversarial Sampling
• On the Privacy Risks of Deploying Recurrent Neural Networks in Machine Learning Models
• EncoderMI: Membership Inference against Pre-trained Encoders in Contrastive Learning
• Teacher Model Fingerprinting Attacks Against Transfer Learning
• Privacy-Preserving Federated Learning on Partitioned Attributes
• Confined Gradient Descent: Privacy-preserving Optimization for Federated Learning
• Differential Privacy Dynamics of Langevin Diffusion and Noisy Gradient Descent
• Federated Unlearning
• DPD-InfoGAN: Differentially Private Distributed InfoGAN
• Towards Causal Federated Learning For Enhanced Robustness and Privacy
• Constrained Differentially Private Federated Learning for Low-bandwidth Devices
• Marich: A Query-efficient Distributionally Equivalent Model Extraction Attack using Public Data
• Towards Bidirectional Protection in Federated Learning
• TiFL: A Tier-based Federated Learning System
• An Operator Splitting View of Federated Learning
• Differential Privacy in Cognitive Radio Networks: A Comprehensive Survey
• Robust Membership Encoding: Inference Attacks and Copyright Protection for Deep Learning
• Defending Against Diverse Attacks in Federated Learning Through Consensus-Based Bi-Level Optimization
• Quantifying the Privacy Risks of Learning High-Dimensional Graphical Models
• Alleviating Privacy Attacks via Causal Learning
• Privacy Threats Analysis to Secure Federated Learning
• 10 Security and Privacy Problems in Large Foundation Models
• MedLeak: Multimodal Medical Data Leakage in Secure Federated Learning with Crafted Models
• GRAFFL: Gradient-free Federated Learning of a Bayesian Generative Model
• Sample-Independent Federated Learning Backdoor Attack in Speaker Recognition
• An Extension of Fano's Inequality for Characterizing Model Susceptibility to Membership Inference Attacks
• Theoretical Guarantees for Model Auditing with Finite Adversaries
• Data Poisoning and Leakage Analysis in Federated Learning
• TableGAN-MCA: Evaluating Membership Collisions of GAN-Synthesized Tabular Data Releasing
• A Critical Look into Threshold Homomorphic Encryption for Private Average Aggregation
• A Tandem Framework Balancing Privacy and Security for Voice User Interfaces
• Improving the Algorithm of Deep Learning with Differential Privacy
• Privacy-Preserving Eye-tracking Using Deep Learning
• Marketplace for AI Models
• A Generative Framework for Low-Cost Result Validation of Machine Learning-as-a-Service Inference
• The Ephemeral Threat: Assessing the Security of Algorithmic Trading Systems powered by Deep Learning
• Security and Privacy Issues and Solutions in Federated Learning for Digital Healthcare
• Over-the-Air Membership Inference Attacks as Privacy Threats for Deep Learning-based Wireless Signal Classifiers
• BGTplanner: Maximizing Training Accuracy for Differentially Private Federated Recommenders via Strategic Privacy Budget Allocation
• Blockchain-based Federated Recommendation with Incentive Mechanism
• Backdoor attacks on DNN and GBDT -- A Case Study from the insurance domain
• Learning to Prevent Leakage: Privacy-Preserving Inference in the Mobile Cloud
• Evaluating Membership Inference Attacks in heterogeneous-data setups
• When Better Features Mean Greater Risks: The Performance-Privacy Trade-Off in Contrastive Learning
• Building Legal Datasets
• Privacy-Aware Document Visual Question Answering