Transformers generalize differently from information stored in context vs in weights

Stephanie C. Y. Chan, Ishita Dasgupta, Junkyung Kim +3

Transformer models can use two fundamentally different kinds of information: information stored in weights during training, and information provided ``in-context'' at inference tim…