3 papers
cs.LG2024
Lambda-Skip Connections: the architectural component that prevents Rank Collapse
Federico Arangath Joseph, Jerome Sieber, Melanie N. Zeilinger +1
Rank collapse, a phenomenon where embedding vectors in sequence models rapidly converge to a uniform token or equilibrium state, has recently gained attention in the deep learning…
cs.LG2024
Geometric Active Exploration in Markov Decision Processes: the Benefit of Abstraction
Riccardo De Santi, Federico Arangath Joseph, Noah Liniger +2
How can a scientist use a Reinforcement Learning (RL) algorithm to design experiments over a dynamical system's state space? In the case of finite and Markovian systems, an area ca…
cs.LG2024
HiPPO-Prophecy: State-Space Models can Provably Learn Dynamical Systems in Context
Federico Arangath Joseph, Kilian Konstantin Haefeli, Noah Liniger +1
This work explores the in-context learning capabilities of State Space Models (SSMs) and presents, to the best of our knowledge, the first theoretical explanation of a possible und…