Secure Federated Submodel Learning

Abstract

Federated learning was proposed with an intriguing vision of achieving collaborative machine learning among numerous clients without uploading their private data to a cloud server. However, the conventional framework requires each client to leverage the full model for learning, which can be prohibitively inefficient for resource-constrained clients and large-scale deep learning tasks. We thus propose a new framework, called federated submodel learning, where clients download only the needed parts of the full model, namely submodels, and then upload the submodel updates. Nevertheless, the "position" of a client's truly required submodel corresponds to her private data, and its disclosure to the cloud server during interactions inevitably breaks the tenet of federated learning. To integrate efficiency and privacy, we have designed a secure federated submodel learning scheme coupled with a private set union protocol as a cornerstone. Our secure scheme features the properties of randomized response, secure aggregation, and Bloom filter, and endows each client with a customized plausible deniability, in terms of local differential privacy, against the position of her desired submodel, thus protecting her private data. We further instantiated our scheme with the e-commerce recommendation scenario in Alibaba, implemented a prototype system, and extensively evaluated its performance over 30-day Taobao user data. The analysis and evaluation results demonstrate the feasibility and scalability of our scheme from model accuracy and convergency, practical communication, computation, and storage overheads, as well as manifest its remarkable advantages over the conventional federated learning framework.

References in corpus (15)

• Deep Learning with Differential Privacy
• Communication-Efficient Learning of Deep Networks from Decentralized Data
• Federated Learning: Strategies for Improving Communication Efficiency
• Federated Learning for Mobile Keyboard Prediction
• How To Backdoor Federated Learning
• Learning Differentially Private Recurrent Language Models
• Applied Federated Learning: Improving Google Keyboard Query Suggestions
• LEAF: A Benchmark for Federated Settings
• Expanding the Reach of Federated Learning by Reducing Client Resource Requirements
• Prio: Private, Robust, and Scalable Computation of Aggregate Statistics
• On the Linear Speedup Analysis of Communication Efficient Momentum SGD for Distributed Non-Convex Optimization
• Distributed Mean Estimation with Limited Communication
• Privacy Amplification by Iteration
• Federated Learning Of Out-Of-Vocabulary Words
• Locally Differentially Private Frequency Estimation with Consistency

Cited by in corpus (5)

• VAFL: a Method of Vertical Asynchronous Federated Learning
• Characterizing Impacts of Heterogeneity in Federated Learning upon Large-Scale Smartphone Data
• Federated learning with differential privacy and an untrusted aggregator
• Federated Submodel Optimization for Hot and Cold Data Features
• Design and Analysis of Uplink and Downlink Communications for Federated Learning