Learning in the Recurrent State: Gradient Descent with Linear Recurrent Networks

Yudou Tian, Neeraj Mohan Sushma, Harshvardhan Mestha +3

In-context learning lets a sequence model adapt to a new task from examples in its input. A prominent line of work shows how self-attention can be constructed to implement gradient…