Extracurricular Learning: Knowledge Transfer Beyond Empirical Distribution

Hadi Pouransari, Mojan Javaheripi, Vinay Sharma +1

Knowledge distillation has been used to transfer knowledge learned by a sophisticated model (teacher) to a simpler model (student). This technique is widely used to compress model…