6 papers · 1 filter
Provable Benefits of Complex Parameterizations for Structured State Space Models
Yuval Ran-Milo, Eden Lumbroso, Edo Cohen-Karlik +3
Structured state space models (SSMs), the core engine behind prominent neural networks such as S4 and Mamba, are linear dynamical systems adhering to a specified structure, most no…
Data-driven Coreference-based Ontology Building
Shir Ashury-Tahan, Amir David Nissan Cohen, Nadav Cohen +2
While coreference resolution is traditionally used as a component in individual document understanding, in this work we take a more global view and explore what can we learn about…
The Implicit Bias of Structured State Space Models Can Be Poisoned With Clean Labels
Yonatan Slutzky, Yotam Alexander, Noam Razin +1
Neural networks are powered by an implicit bias: a tendency of gradient descent to fit training data in a way that generalizes to unseen data. A recent class of neural network mode…
Lecture Notes on Linear Neural Networks: A Tale of Optimization and Generalization in Deep Learning
Nadav Cohen, Noam Razin
These notes are based on a lecture delivered by NC on March 2021, as part of an advanced course in Princeton University on the mathematical understanding of deep learning. They pre…
DeciMamba: Exploring the Length Extrapolation Potential of Mamba
Assaf Ben-Kish, Itamar Zimerman, Shady Abu-Hussein +4
Long-range sequence processing poses a significant challenge for Transformers due to their quadratic complexity in input length. A promising alternative is Mamba, which demonstrate…
Implicit Bias of Policy Gradient in Linear Quadratic Control: Extrapolation to Unseen Initial States
Noam Razin, Yotam Alexander, Edo Cohen-Karlik +3
In modern machine learning, models can often fit training data in numerous ways, some of which perform well on unseen (test) data, while others do not. Remarkably, in such cases gr…