Self-Distillation is Optimal Among Spectral Shrinkage Estimators in Spiked Covariance Models

Radu Lecoiu, Debarghya Mukherjee, Pragya Sur

Self-distillation has emerged as a promising technique for improving model performance in modern machine learning systems. We develop the statistical foundations of self-distillati…