5 citations · 9 across the 7 of their papers we have counts for
9 papers
Efficiently Approximating Attention Is Hard
Lukas Haverbeck, Carmen Amo Alonso, Andres Felipe Posada-Moreno +2
Softmax attention is ubiquitous in modern machine learning, but its quadratic scaling with sequence length makes it costly. To reduce this cost, attention is often approximated wit…
The risk of KV cache compression
Lukas Haverbeck, Carmen Amo Alonso, Andres Felipe Posada-Moreno +2
Transformer inference on long sequences is expensive because softmax attention repeatedly reads from a large KV cache. The prevalent approach to this bottleneck is KV cache compres…
Gated Normalization Removal and Scale Anchoring in Pre-Norm Transformers
Andrei Kanavalau, Carmen Amo Alonso, Sanjay Lall
Normalization layers are standard in transformers, but it is not clear whether their sample-dependent computations are necessary throughout both training and inference. This work d…
Design Principles for Sequence Models via Coefficient Dynamics
Jerome Sieber, Antonio Orvieto, Melanie N. Zeilinger +1
Deep sequence models, ranging from Transformers and State Space Models (SSMs) to more recent approaches such as gated linear RNNs, fundamentally compute outputs as linear combinati…
Bridging Expressivity and Scalability with Adaptive Unitary SSMs
Arjun Karuvally, Franz Nowak, Anderson T. Keller +3
Recent work has revealed that state space models (SSMs), while efficient for long-sequence processing, are fundamentally limited in their ability to represent formal languages-part…
Lambda-Skip Connections: the architectural component that prevents Rank Collapse
Federico Arangath Joseph, Jerome Sieber, Melanie N. Zeilinger +1
Rank collapse, a phenomenon where embedding vectors in sequence models rapidly converge to a uniform token or equilibrium state, has recently gained attention in the deep learning…
Understanding the differences in Foundation Models: Attention, State Space Models, and Recurrent Neural Networks
Jerome Sieber, Carmen Amo Alonso, Alexandre Didier +2
Softmax attention is the principle backbone of foundation models for various artificial intelligence applications, yet its quadratic complexity in sequence length can limit its inf…
State Space Models as Foundation Models: A Control Theoretic Overview
Carmen Amo Alonso, Jerome Sieber, Melanie N. Zeilinger
In recent years, there has been a growing interest in integrating linear state-space models (SSM) in deep neural network architectures of foundation models. This is exemplified by…
Global Performance Guarantees for Localized Model Predictive Control
Jing Shuang Li, Carmen Amo Alonso
Recent advances in model predictive control (MPC) leverage local communication constraints to produce localized MPC algorithms whose complexities scale independently of total netwo…