Understanding the differences in Foundation Models: Attention, State Space Models, and Recurrent Neural Networks

Jerome Sieber, Carmen Amo Alonso, Alexandre Didier +2

Softmax attention is the principle backbone of foundation models for various artificial intelligence applications, yet its quadratic complexity in sequence length can limit its inf…