Distill on the Go: Online knowledge distillation in self-supervised learning

Prashant Bhat, Elahe Arani, Bahram Zonooz

Self-supervised learning solves pretext prediction tasks that do not require annotations to learn feature representations. For vision tasks, pretext tasks such as predicting rotati…