Differential learning kinetics govern the transition from memorization to generalization during in-context learning

Alex Nguyen, Gautam Reddy

Transformers exhibit in-context learning (ICL): the ability to use novel information presented in the context without additional weight updates. Recent work shows that ICL emerges…