When can transformers compositionally generalize in-context?

Seijin Kobayashi, Simon Schug, Yassir Akram +5

Many tasks can be composed from a few independent components. This gives rise to a combinatorial explosion of possible tasks, only some of which might be encountered during trainin…