Uncovering mesa-optimization algorithms in Transformers

Johannes von Oswald, Maximilian Schlegel, Alexander Meulemans +10

Some autoregressive models exhibit in-context learning capabilities: being able to learn as an input sequence is processed, without undergoing any parameter changes, and without be…