Progressive Ensemble Distillation: Building Ensembles for Efficient Inference

Don Kurian Dennis, Abhishek Shetty, Anish Sevekari +2

We study the problem of progressive ensemble distillation: Given a large, pretrained teacher model $g$, we seek to decompose the model into smaller, low-inference cost student mode…