Super Weights in LLMs and the Failure of Selective Training

Shreyas Subramanian, Adewale Akinfaderin, Akarsha Sehwag

Recent work identified Super Weights, individual parameters whose removal degrades model performance by orders of magnitude. We show that this degradation due to pruning Super Weig…