7 papers · 1 filter
Ignition Phase : Standard Training for Fast Adversarial Robustness
Wang Yu-Hang, Liu ying, Fang liang +6
Adversarial Training (AT) is a cornerstone defense, but many variants overlook foundational feature representations by primarily focusing on stronger attack generation. We introduc…
FedBiF: Communication-Efficient Federated Learning via Bits Freezing
Shiwei Li, Qunwei Li, Haozhao Wang +3
Federated learning (FL) is an emerging distributed machine learning paradigm that enables collaborative model training without sharing local data. Despite its advantages, FL suffer…
Beyond Zero Initialization: Investigating the Impact of Non-Zero Initialization on LoRA Fine-Tuning Dynamics
Shiwei Li, Xiandi Luo, Xing Tang +6
Low-rank adaptation (LoRA) is a widely used parameter-efficient fine-tuning method. In standard LoRA layers, one of the matrices, or , is initialized to zero, ensuring that…
The Panaceas for Improving Low-Rank Decomposition in Communication-Efficient Federated Learning
Shiwei Li, Xiandi Luo, Haozhao Wang +6
To improve the training efficiency of federated learning (FL), previous research has employed low-rank decomposition techniques to reduce communication overhead. In this paper, we…
BoRA: Towards More Expressive Low-Rank Adaptation with Block Diversity
Shiwei Li, Xiandi Luo, Haozhao Wang +6
Low-rank adaptation (LoRA) is a parameter-efficient fine-tuning (PEFT) method widely used in large language models (LLMs). It approximates the update of a pretrained weight matrix…
Masked Random Noise for Communication Efficient Federated Learning
Shiwei Li, Yingyi Cheng, Haozhao Wang +7
Federated learning is a promising distributed training paradigm that effectively safeguards data privacy. However, it may involve significant communication costs, which hinders tra…