Distilled Circuits: A Mechanistic Study of Internal Restructuring in Knowledge Distillation

Reilly Haskins, Benjamin Adams

Knowledge distillation compresses a larger neural model (teacher) into smaller, faster student models by training them to match teacher outputs. However, the internal computational…