When pre-training hurts LoRA fine-tuning: a dynamical analysis via single-index models

Gibbs Nwemadji, Bruno Loureiro, Jean Barbier

Pre-training on a source task is usually expected to facilitate fine-tuning on similar downstream problems. In this work, we mathematically show that this naive intuition is not al…