Matrix Sketching for Secure Collaborative Machine Learning

Abstract

Collaborative learning allows participants to jointly train a model without data sharing. To update the model parameters, the central server broadcasts model parameters to the clients, and the clients send updating directions such as gradients to the server. While data do not leave a client device, the communicated gradients and parameters will leak a client's privacy. Attacks that infer clients' privacy from gradients and parameters have been developed by prior work. Simple defenses such as dropout and differential privacy either fail to defend the attacks or seriously hurt test accuracy. We propose a practical defense which we call Double-Blind Collaborative Learning (DBCL). The high-level idea is to apply random matrix sketching to the parameters (aka weights) and re-generate random sketching after each iteration. DBCL prevents clients from conducting gradient-based privacy inferences which are the most effective attacks. DBCL works because from the attacker's perspective, sketching is effectively random noise that outweighs the signal. Notably, DBCL does not much increase computation and communication costs and does not hurt test accuracy at all.

In International Conference on Machine Learning (ICML), 2021

References in corpus (16)

• Communication-Efficient Learning of Deep Networks from Decentralized Data
• On the Convergence of FedAvg on Non-IID Data
• Secure Federated Transfer Learning
• Federated Optimization in Heterogeneous Networks
• Local SGD Converges Fast and Communicates Little
• Decentralized Stochastic Optimization and Gossip Algorithms with Compressed Communication
• D${}^2$: Decentralized Training over Decentralized Data
• Cooperative SGD: A unified Framework for the Design and Analysis of Communication-Efficient SGD Algorithms
• Privacy for Free: Communication-Efficient Learning with Differential Privacy Using Sketches
• MATCHA: Speeding Up Decentralized SGD via Matching Decomposition Sampling
• SEGA: Variance Reduction via Gradient Sketching
• Differentially Private Data Generative Models
• Gossip Dual Averaging for Decentralized Optimization of Pairwise Functions
• Randomized Numerical Linear Algebra: Foundations & Algorithms
• Gradient Descent with Compressed Iterates
• Heterogeneity-Aware Asynchronous Decentralized Training