37 citations · 64 across the 15 of their papers we have counts for
4 papers · 1 filter
Lambda-Skip Connections: the architectural component that prevents Rank Collapse
Federico Arangath Joseph, Jerome Sieber, Melanie N. Zeilinger +1
Rank collapse, a phenomenon where embedding vectors in sequence models rapidly converge to a uniform token or equilibrium state, has recently gained attention in the deep learning…
Computationally Efficient System Level Tube-MPC for Uncertain Systems
Jerome Sieber, Alexandre Didier, Melanie N. Zeilinger
Tube-based model predictive control (MPC) is one of the principal robust control techniques for constrained linear systems affected by additive disturbances. While tube-based metho…
Understanding the differences in Foundation Models: Attention, State Space Models, and Recurrent Neural Networks
Jerome Sieber, Carmen Amo Alonso, Alexandre Didier +2
Softmax attention is the principle backbone of foundation models for various artificial intelligence applications, yet its quadratic complexity in sequence length can limit its inf…
State Space Models as Foundation Models: A Control Theoretic Overview
Carmen Amo Alonso, Jerome Sieber, Melanie N. Zeilinger
In recent years, there has been a growing interest in integrating linear state-space models (SSM) in deep neural network architectures of foundation models. This is exemplified by…