Knowledge Distillation Using Frontier Open-source LLMs: Generalizability and the Role of Synthetic Data

Anup Shirgaonkar, Nikhil Pandey, Nazmiye Ceren Abay +2

Leading open-source large language models (LLMs) such as Llama-3.1-Instruct-405B are extremely capable at generating text, answering questions, and solving a variety of natural lan…