Knowledge That Is Linear Before Fine-Tuning Generalizes After It

Faridoun Mehri, Hinrich Schütze

Not all facts a language model can recall support further learning equally well. We distinguish active facts, which support generalization beyond their own recall, from inert facts…