Showing 2024Show all
2 papers · 1 filter
cs.LG2024
Understanding the differences in Foundation Models: Attention, State Space Models, and Recurrent Neural Networks
Jerome Sieber, Carmen Amo Alonso, Alexandre Didier +2
Softmax attention is the principle backbone of foundation models for various artificial intelligence applications, yet its quadratic complexity in sequence length can limit its inf…
eess.SY2024
State Space Models as Foundation Models: A Control Theoretic Overview
Carmen Amo Alonso, Jerome Sieber, Melanie N. Zeilinger
In recent years, there has been a growing interest in integrating linear state-space models (SSM) in deep neural network architectures of foundation models. This is exemplified by…