Pretraining Induces a Reusable Spectral Basis for Downstream Task Adaptation

Junjie Yu, Yue Wang, Zihan Deng +3

Finetuning pretrained models occurs in a low-dimensional subspace of the full parameter space. Prior work has focused on characterizing this optimization subspace, but largely igno…