Principal Component Networks: Parameter Reduction Early in Training

Roger Waleffe, Theodoros Rekatsinas

Recent works show that overparameterized networks contain small subnetworks that exhibit comparable accuracy to the full model when trained in isolation. These results highlight th…