13 citations · 14 across the 12 of their papers we have counts for
3 papers · 1 filter
AMiD: Knowledge Distillation for LLMs with -mixture Assistant Distribution
Donghyeok Shin, Yeongmin Kim, Suhyeon Jo +2
Autoregressive large language models (LLMs) have achieved remarkable improvement across many tasks but incur high computational and memory costs. Knowledge distillation (KD) mitiga…
Distillation of Large Language Models via Concrete Score Matching
Yeongmin Kim, Donghyeok Shin, Mina Kang +2
Large language models (LLMs) deliver remarkable performance but are costly to deploy, motivating knowledge distillation (KD) for efficient inference. Existing KD objectives typical…
Preference Optimization by Estimating the Ratio of the Data Distribution
Yeongmin Kim, Heesun Bae, Byeonghu Na +1
Direct preference optimization (DPO) is widely used as a simple and stable method for aligning large language models (LLMs) with human preferences. This paper investigates a genera…