RW-LoRA: Communication-Efficient Decentralized LoRA Fine-Tuning via Random Walks

Xingran Chen, Rohit Bhagat, Ghadir Ayache +3

Parameter-efficient fine-tuning methods such as LoRA have become a standard approach for adapting large foundation models. Adopting fine-tuning to distributed settings faces severa…