Inverting Gradients -- How easy is it to break privacy in federated learning?

Abstract

The idea of federated learning is to collaboratively train a neural network on a server. Each user receives the current weights of the network and in turns sends parameter updates (gradients) based on local data. This protocol has been designed not only to train neural networks data-efficiently, but also to provide privacy benefits for users, as their input data remains on device and only parameter gradients are shared. But how secure is sharing parameter gradients? Previous attacks have provided a false sense of security, by succeeding only in contrived settings - even for a single image. However, by exploiting a magnitude-invariant loss along with optimization strategies based on adversarial attacks, we show that is is actually possible to faithfully reconstruct images at high resolution from the knowledge of their parameter gradients, and demonstrate that such a break of privacy is possible even for trained deep networks. We analyze the effects of architecture as well as parameters on the difficulty of reconstructing an input image and prove that any input to a fully connected layer can be reconstructed analytically independent of the remaining architecture. Finally we discuss settings encountered in practice and show that even averaging gradients over several iterations or several images does not protect the user's privacy in federated learning applications in computer vision.

arXiv comment23 pages, 20 figures. The first three authors contributed equally

References in corpus (19)

• Communication-Efficient Learning of Deep Networks from Decentralized Data
• Federated Learning with Non-IID Data
• Towards Deep Learning Models Resistant to Adversarial Attacks
• Understanding Black-box Predictions via Influence Functions
• Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples
• Towards Federated Learning at Scale: System Design
• Learning Differentially Private Recurrent Language Models
• Federated Machine Learning: Concept and Applications
• Federated Optimization:Distributed Optimization Beyond the Datacenter
• Visualizing Deep Convolutional Neural Networks Using Natural Pre-Images
• iDLG: Improved Deep Leakage from Gradients
• Algorithms that Remember: Model Inversion Attacks and Data Protection Law
• Adaptive Federated Optimization
• Evaluating Differentially Private Machine Learning in Practice
• Reversible Architectures for Arbitrarily Deep Residual Neural Networks
• Beyond Inferring Class Representatives: User-Level Privacy Leakage From Federated Learning
• Logit Pairing Methods Can Fool Gradient-Based Attacks
• The Secret Revealer: Generative Model-Inversion Attacks Against Deep Neural Networks
• Input Similarity from the Neural Network Perspective

Cited by in corpus (49)

• A Survey on Federated Learning Systems: Vision, Hype and Reality for Data Privacy and Protection
• From Distributed Machine Learning to Federated Learning: A Survey
• GRNN: Generative Regression Neural Network -- A Data Leakage Attack for Federated Learning
• Introducing the VoicePrivacy Initiative
• A Framework for Evaluating Gradient Leakage Attacks in Federated Learning
• Privacy-Preserving Machine Learning: Methods, Challenges and Directions
• The Federation Strikes Back: A Survey of Federated Learning Privacy Attacks, Defenses, Applications, and Policy Landscape
• Federated Learning Survey: A Multi-Level Taxonomy of Aggregation Techniques, Experimental Insights, and Future Frontiers
• PPFL: Privacy-preserving Federated Learning with Trusted Execution Environments
• Witches' Brew: Industrial Scale Data Poisoning via Gradient Matching
• Mind the Pad -- CNNs can Develop Blind Spots
• Secure Shapley Value for Cross-Silo Federated Learning (Technical Report)
• Turbo-Aggregate: Breaking the Quadratic Aggregation Barrier in Secure Federated Learning
• Machine Learning with Confidential Computing: A Systematization of Knowledge
• Privacy Preservation in Federated Learning: An insightful survey from the GDPR Perspective
• Differentially Private Federated Learning on Heterogeneous Data
• Federated Reconstruction: Partially Local Federated Learning
• Towards General Deep Leakage in Federated Learning
• A Systematic Literature Review on Federated Learning: From A Model Quality Perspective
• InstaHide: Instance-hiding Schemes for Private Distributed Learning
• Securing Secure Aggregation: Mitigating Multi-Round Privacy Leakage in Federated Learning
• Privacy attacks for automatic speech recognition acoustic models in a federated learning framework
• Perfectly Accurate Membership Inference by a Dishonest Central Server in Federated Learning
• Secure Aggregation for Buffered Asynchronous Federated Learning
• Byzantine-Robust and Privacy-Preserving Framework for FedML
• Rethinking Architecture Design for Tackling Data Heterogeneity in Federated Learning
• Confidential Machine Learning Computation in Untrusted Environments: A Systems Security Perspective
• SoK: Machine Learning Governance
• Collaborative Distributed Machine Learning
• From Federated Learning to Federated Neural Architecture Search: A Survey
• Byzantine-Resilient Secure Federated Learning
• Bayesian Framework for Gradient Leakage
• Sharing Models or Coresets: A Study based on Membership Inference Attack
• Revealing and Protecting Labels in Distributed Training
• Privacy-preserving medical image analysis
• Complex-valued Federated Learning with Differential Privacy and MRI Applications
• Rethinking Privacy Preserving Deep Learning: How to Evaluate and Thwart Privacy Attacks
• Distributed Additive Encryption and Quantization for Privacy Preserving Federated Deep Learning
• Privacy-Preserving Federated Learning on Partitioned Attributes
• Separation of Powers in Federated Learning
• Constrained Differentially Private Federated Learning for Low-bandwidth Devices
• Defending Against Diverse Attacks in Federated Learning Through Consensus-Based Bi-Level Optimization
• SSGD: A safe and efficient method of gradient descent
• A Quantitative Metric for Privacy Leakage in Federated Learning
• Two-Bit Aggregation for Communication Efficient and Differentially Private Federated Learning
• Learning to Prevent Leakage: Privacy-Preserving Inference in the Mobile Cloud
• A Privacy-Preserving and Trustable Multi-agent Learning Framework
• NeuralDP Differentially private neural networks by design
• Prior-Independent Auctions for the Demand Side of Federated Learning