Brewing Knowledge in Context: Distillation Perspectives on In-Context Learning

Chengye Li, Haiyun Liu, Yuanxi Li

In-context learning (ICL) allows large language models (LLMs) to solve novel tasks without weight updates. Despite its empirical success, the mechanism behind ICL remains poorly un…