SPEED: Secure, PrivatE, and Efficient Deep learning

Abstract

We introduce a deep learning framework able to deal with strong privacy constraints. Based on collaborative learning, differential privacy and homomorphic encryption, the proposed approach advances state-of-the-art of private deep learning against a wider range of threats, in particular the honest-but-curious server assumption. We address threats from both the aggregation server, the global model and potentially colluding data holders. Building upon distributed differential privacy and a homomorphic argmax operator, our method is specifically designed to maintain low communication loads and efficiency. The proposed method is supported by carefully crafted theoretical results. We provide differential privacy guarantees from the point of view of any entity having access to the final model, including colluding data holders, as a function of the ratio of data holders who kept their noise secret. This makes our method practical to real-life scenarios where data holders do not trust any third party to process their datasets nor the other data holders. Crucially the computational burden of the approach is maintained reasonable, and, to the best of our knowledge, our framework is the first one to be efficient enough to investigate deep learning applications while addressing such a large scope of threats. To assess the practical usability of our framework, experiments have been carried out on image datasets in a classification context. We present numerical results that show that the learning procedure is both accurate and private.

arXiv comment32 pages, 3 figures. Mach Learn (2021)

References in corpus (17)

• Deep Learning with Differential Privacy
• Improved Techniques for Training GANs
• Stealing Machine Learning Models via Prediction APIs
• Differentially Private Federated Learning: A Client Level Perspective
• A generic framework for privacy preserving deep learning
• Scalable Private Learning with PATE
• CryptoDL: Deep Neural Networks over Encrypted Data
• Extremal Mechanisms for Local Differential Privacy
• Protection Against Reconstruction and Its Applications in Private Federated Learning
• Differential Privacy and Machine Learning: a Survey and Review
• Practical Secure Aggregation for Federated Learning on User-Held Data
• Semi-supervised Knowledge Transfer for Deep Learning from Private Training Data
• Gazelle: A Low Latency Framework for Secure Neural Network Inference
• Cache Telepathy: Leveraging Shared Resource Attacks to Learn DNN Architectures
• Privacy-Preserving Distributed Deep Learning for Clinical Data
• Tight Lower Bounds for Locally Differentially Private Selection
• Glyph: Fast and Accurately Training Deep Neural Networks on Encrypted Data