COFFEQA: Counterfactual-Guided Few-Shot Distillation for LLM Question Answering

Kavinda Kehelella, Sanghamitra Dutta

Knowledge distillation aims to train a smaller student model to mimic a larger teacher model for resource-constrained environments. However, the process of distillation is typicall…