SimReg: Regression as a Simple Yet Effective Tool for Self-supervised Knowledge Distillation

K L Navaneet, Soroush Abbasi Koohpayegani, Ajinkya Tejankar +1

Feature regression is a simple way to distill large neural network models to smaller ones. We show that with simple changes to the network architecture, regression can outperform m…