37 citations · 64 across the 15 of their papers we have counts for
Showing 2024 · cs.LGShow all
2 papers · 2 filters
cs.LG2024
Lambda-Skip Connections: the architectural component that prevents Rank Collapse
Federico Arangath Joseph, Jerome Sieber, Melanie N. Zeilinger +1
Rank collapse, a phenomenon where embedding vectors in sequence models rapidly converge to a uniform token or equilibrium state, has recently gained attention in the deep learning…
cs.LG2024★ 3 cited
Understanding the differences in Foundation Models: Attention, State Space Models, and Recurrent Neural Networks
Jerome Sieber, Carmen Amo Alonso, Alexandre Didier +2
Softmax attention is the principle backbone of foundation models for various artificial intelligence applications, yet its quadratic complexity in sequence length can limit its inf…