4 papers
Divergence-Routed On-Policy Distillation for Language Models
Tiantian Peng, Guannan Lv, Hongyang Li +5
On-policy distillation (OPD) transfers knowledge from a teacher model to a student along student-generated trajectories, typically by minimizing the reverse Kullback-Leibler (RKL)…
Trustworthy Efficient Communication for Distributed Learning using LQ-SGD Algorithm
Hongyang Li, Lincen Bai, Caesar Wu +3
We propose LQ-SGD (Low-Rank Quantized Stochastic Gradient Descent), an efficient communication gradient compression algorithm designed for distributed training. LQ-SGD further deve…
Lightweight Trustworthy Distributed Clustering
Hongyang Li, Caesar Wu, Mohammed Chadli +2
Ensuring data trustworthiness within individual edge nodes while facilitating collaborative data processing poses a critical challenge in edge computing systems (ECS), particularly…
Trustworthiness of Stochastic Gradient Descent in Distributed Learning
Hongyang Li, Caesar Wu, Mohammed Chadli +2
Distributed learning (DL) uses multiple nodes to accelerate training, enabling efficient optimization of large-scale models. Stochastic Gradient Descent (SGD), a key optimization a…