Linearizing Large Language Models

Jean Mercat, Igor Vasiljevic, Sedrick Keh +4

Linear transformers have emerged as a subquadratic-time alternative to softmax attention and have garnered significant interest due to their fixed-size recurrent state that lowers…