Diversity-Aware Reverse Kullback-Leibler Divergence for Large Language Model Distillation

Hoang-Chau Luong, Dat Ba Tran, Lingwei Chen

Reverse Kullback-Leibler (RKL) divergence has recently emerged as the preferred objective for large language model (LLM) distillation, consistently outperforming forward KL (FKL),…