In-context Learning Distillation: Transferring Few-shot Learning Ability of Pre-trained Language Models

Yukun Huang, Yanda Chen, Zhou Yu +1

Given the success with in-context learning of large pre-trained language models, we introduce in-context learning distillation to transfer in-context few-shot learning ability from…