Self-supervised Knowledge Distillation Using Singular Value Decomposition

Seung Hyun Lee, Dae Ha Kim, Byung Cheol Song

To solve deep neural network (DNN)'s huge training dataset and its high computation issue, so-called teacher-student (T-S) DNN which transfers the knowledge of T-DNN to S-DNN has b…