Understanding LLM Parameter Update Sparsity through the Lens of Fisher

Yufan Zhang, Sagnik Mukherjee, Hao Peng

Recent studies have observed that parameter changes during language-model post-training can be concentrated in a small subset of coordinates. This phenomenon has been reported in r…