5 papers
FedEve: On Bridging the Client Drift and Period Drift for Cross-device Federated Learning
Tao Shen, Zexi Li, Didi Zhu +3
Federated learning (FL) is a machine learning paradigm that allows multiple clients to collaboratively train a shared model without exposing their private data. Data heterogeneity…
Will LLMs Scaling Hit the Wall? Breaking Barriers via Distributed Resources on Massive Edge Devices
Tao Shen, Didi Zhu, Ziyu Zhao +3
The remarkable success of foundation models has been driven by scaling laws, demonstrating that model performance improves predictably with increased training data and model size.…
Each Rank Could be an Expert: Single-Ranked Mixture of Experts LoRA for Multi-Task Learning
Ziyu Zhao, Yixiao Zhou, Zhi Zhang +10
Low-Rank Adaptation (LoRA) is widely used for adapting large language models (LLMs) to specific domains due to its efficiency and modularity. Meanwhile, vanilla LoRA struggles with…
FedCFA: Alleviating Simpson's Paradox in Model Aggregation with Counterfactual Federated Learning
Zhonghua Jiang, Jimin Xu, Shengyu Zhang +5
Federated learning (FL) is a promising technology for data privacy and distributed optimization, but it suffers from data imbalance and heterogeneity among clients. Existing FL met…
Merging LoRAs like Playing LEGO: Pushing the Modularity of LoRA to Extremes Through Rank-Wise Clustering
Ziyu Zhao, Tao Shen, Didi Zhu +5
Low-Rank Adaptation (LoRA) has emerged as a popular technique for fine-tuning large language models (LLMs) to various domains due to its modular design and widespread availability…