AMiD: Knowledge Distillation for LLMs with $\alpha$-mixture Assistant Distribution

Donghyeok Shin, Yeongmin Kim, Suhyeon Jo +2

Autoregressive large language models (LLMs) have achieved remarkable improvement across many tasks but incur high computational and memory costs. Knowledge distillation (KD) mitiga…