Lifelong In-Context Learning with Transformers Requires Parametric Forms of Attention

Luke McDermott, Robert W. Heath, Rahul Parhi

Lifelong continual learning remains an obstacle on the path to human-like intelligence. Modern transformers show sparks of intelligence with in-context learning. The quadratic natu…