7 citations · 7 across the 12 of their papers we have counts for
5 papers · 1 filter
BoRA: Towards More Expressive Low-Rank Adaptation with Block Diversity
Shiwei Li, Xiandi Luo, Haozhao Wang +6
Low-rank adaptation (LoRA) is a parameter-efficient fine-tuning (PEFT) method widely used in large language models (LLMs). It approximates the update of a pretrained weight matrix…
Beyond Zero Initialization: Investigating the Impact of Non-Zero Initialization on LoRA Fine-Tuning Dynamics
Shiwei Li, Xiandi Luo, Xing Tang +6
Low-rank adaptation (LoRA) is a widely used parameter-efficient fine-tuning method. In standard LoRA layers, one of the matrices, or , is initialized to zero, ensuring that…
The Panaceas for Improving Low-Rank Decomposition in Communication-Efficient Federated Learning
Shiwei Li, Xiandi Luo, Haozhao Wang +6
To improve the training efficiency of federated learning (FL), previous research has employed low-rank decomposition techniques to reduce communication overhead. In this paper, we…
Masked Random Noise for Communication Efficient Federated Learning
Shiwei Li, Yingyi Cheng, Haozhao Wang +7
Federated learning is a promising distributed training paradigm that effectively safeguards data privacy. However, it may involve significant communication costs, which hinders tra…
FedBAT: Communication-Efficient Federated Learning via Learnable Binarization
Shiwei Li, Wenchao Xu, Haozhao Wang +7
Federated learning is a promising distributed machine learning paradigm that can effectively exploit large-scale data without exposing users' privacy. However, it may incur signifi…