You Don't Need All Attentions: Distributed Dynamic Fine-Tuning for Foundation Models

Shiwei Ding, Lan Zhang, Zhenlin Wang +2

Fine-tuning plays a crucial role in adapting models to downstream tasks with minimal training efforts. However, the rapidly increasing size of foundation models poses a daunting ch…