LEAD: Liberal Feature-based Distillation for Dense Retrieval

Hao Sun, Xiao Liu, Yeyun Gong +6

Knowledge distillation is often used to transfer knowledge from a strong teacher model to a relatively weak student model. Traditional methods include response-based methods and fe…