Memory-Efficient Transfer Learning with Fading Side Networks via Masked Dual Path Distillation

Yutong Zhang, Jiaxin Chen, Honglin Chen +5

Memory-efficient transfer learning (METL) approaches have recently achieved promising performance in adapting pre-trained models to downstream tasks. They avoid applying gradient b…