What Should Feature Distillation Transfer in LLMs? A Task-Tangent Geometry View

Khouloud Saadi, Di Wang

Feature-based knowledge distillation aims to transfer intermediate representations from a teacher LLM model to a student. Existing approaches typically rely on direct feature match…