REDistill: Robust Estimator Distillation for Balancing Robustness and Efficiency

Ondrej Tybl, Lukas Neumann

Knowledge Distillation (KD) transfers knowledge from a large teacher model to a smaller student by aligning their predictive distributions. However, conventional KD formulations -…