The Initialization Determines Whether In-Context Learning Is Gradient Descent

Shifeng Xie, Rui Yuan, Simone Rossi +1

In-context learning (ICL) in large language models (LLMs) is a striking phenomenon, yet its underlying mechanisms remain only partially understood. Previous work connects linear se…