State Soup: In-Context Skill Learning, Retrieval and Mixing

Maciej Pióro, Maciej Wołczyk, Razvan Pascanu +2

A new breed of gated-linear recurrent neural networks has reached state-of-the-art performance on a range of sequence modeling problems. Such models naturally handle long sequences…