Balancing Knowledge Distillation for Imbalance Learning with Bilevel Optimization

Anh B. H. Nguyen, Ba Tho Phan, Viet Cuong Ta

Knowledge distillation transfers knowledge from a high capacity teacher to a compact student using a mixture of hard and soft losses. On imbalanced data, a fixed weighting between…