In-Context Learning Distillation for Efficient Few-Shot Fine-Tuning

Yifei Duan, Liu Li, Zirui Zhai +1

We applied few-shot in-context learning on the OPT-1.3B model for the natural language inference task and employed knowledge distillation to internalize the context information, re…