6 papers
Gradient Compression May Hurt Generalization: A Remedy by Synthetic Data Guided Sharpness Aware Minimization
Yujie Gu, Richeng Jin, Zhaoyang Zhang +1
It is commonly believed that gradient compression in federated learning (FL) enjoys significant improvement in communication efficiency with negligible performance degradation. In…
Differentially Private and Communication Efficient Large Language Model Split Inference via Stochastic Quantization and Soft Prompt
Yujie Gu, Richeng Jin, Xiaoyu Ji +2
Large Language Models (LLMs) have achieved remarkable performance and received significant research interest. The enormous computational demands, however, hinder the local deployme…
BicKD: Bilateral Contrastive Knowledge Distillation
Jiangnan Zhu, Yukai Xu, Li Xiong +4
Knowledge distillation (KD) is a machine learning framework that transfers knowledge from a teacher model to a student model. The vanilla KD proposed by Hinton et al. has been the…
Enhanced Privacy Bound for Shuffle Model with Personalized Privacy
Yixuan Liu, Yuhan Liu, Li Xiong +2
The shuffle model of Differential Privacy (DP) is an enhanced privacy protocol which introduces an intermediate trusted server between local users and a central data curator. It si…
DPDR: Gradient Decomposition and Reconstruction for Differentially Private Deep Learning
Yixuan Liu, Li Xiong, Yuhan Liu +3
Differentially Private Stochastic Gradients Descent (DP-SGD) is a prominent paradigm for preserving privacy in deep learning. It ensures privacy by perturbing gradients with random…
TernaryVote: Differentially Private, Communication Efficient, and Byzantine Resilient Distributed Optimization on Heterogeneous Data
Richeng Jin, Yujie Gu, Kai Yue +3
Distributed training of deep neural networks faces three critical challenges: privacy preservation, communication efficiency, and robustness to fault and adversarial behaviors. Alt…