LazyDP: Co-Designing Algorithm-Software for Scalable Training of Differentially Private Recommendation Models

Abstract

Differential privacy (DP) is widely being employed in the industry as a practical standard for privacy protection. While private training of computer vision or natural language processing applications has been studied extensively, the computational challenges of training of recommender systems (RecSys) with DP have not been explored. In this work, we first present our detailed characterization of private RecSys training using DP-SGD, root-causing its several performance bottlenecks. Specifically, we identify DP-SGD's noise sampling and noisy gradient update stage to suffer from a severe compute and memory bandwidth limitation, respectively, causing significant performance overhead in training private RecSys. Based on these findings, we propose LazyDP, an algorithm-software co-design that addresses the compute and memory challenges of training RecSys with DP-SGD. Compared to a state-of-the-art DP-SGD training system, we demonstrate that LazyDP provides an average 119x training throughput improvement while also ensuring mathematically equivalent, differentially private RecSys models to be trained.

References in corpus (23)

• Communication-Efficient Learning of Deep Networks from Decentralized Data
• Comprehensive Privacy Analysis of Deep Learning: Passive and Active White-box Inference Attacks against Centralized and Federated Learning
• Deep Learning Recommendation Model for Personalization and Recommendation Systems
• Extracting Training Data from Large Language Models
• Scaling Distributed Machine Learning with In-Network Aggregation
• Privacy Amplification by Iteration
• Parameter Hub: a Rack-Scale Parameter Server for Distributed Deep Neural Network Training
• Label-Only Membership Inference Attacks
• Scalable Extraction of Training Data from (Production) Language Models
• Large Language Models Can Be Strong Differentially Private Learners
• DeepRecSys: A System for Optimizing End-To-End At-scale Neural Recommendation Inference
• Unlocking High-Accuracy Differentially Private Image Classification through Scale
• Reconstructing Training Data from Trained Neural Networks
• Opacus: User-Friendly Differential Privacy Library in PyTorch
• Toward Training at ImageNet Scale with Differential Privacy
• TT-Rec: Tensor Train Compression for Deep Learning Recommendation Models
• RecNMP: Accelerating Personalized Recommendation with Near-Memory Processing
• On the Importance of Difficulty Calibration in Membership Inference Attacks
• Applied Federated Learning: Architectural Design for Robust and Efficient Learning in Privacy Aware Settings
• Plug-In Inversion: Model-Agnostic Inversion for Vision with Data Augmentations
• Augmenting Netflix Search with In-Session Adapted Recommendations
• LAORAM: A Look Ahead ORAM Architecture for Training Large Embedding Tables
• DiVa: An Accelerator for Differentially Private Machine Learning