Rényi Divergence Deep Mutual Learning

Weipeng Huang, Junjie Tao, Changbo Deng +4

This paper revisits Deep Mutual Learning (DML), a simple yet effective computing paradigm. We propose using Rényi divergence instead of the KL divergence, which is more flexible an…