2 citations · 2 across the 12 of their papers we have counts for
7 papers · 1 filter
Local Support Learning
Assaf Ben-Kish, Akarsh Kumar, James Glass +1
We explore catastrophic forgetting in the context of large pre-trained models. By considering forgetting as a geometric problem in the input space of each weight matrix, we uncover…
On the Recall Scaling Laws in Mamba: A Theoretical and Mechanistic Study via Hashing
Yuval Koren, Assaf Ben-Kish, Raja Giryes +2
Associative Recall (AR) is the cognitive ability to learn and retrieve links between items in memory. In NLP, AR is used as a benchmark for evaluating the in-context memory capacit…
Entropy-Preserving Reinforcement Learning
Aleksei Petrenko, Ben Lipkin, Kevin Chen +4
Policy gradient algorithms have driven many recent advancements in language model reasoning. An appealing property is their ability to learn from exploration on their own trajector…
Provable Benefits of Complex Parameterizations for Structured State Space Models
Yuval Ran-Milo, Eden Lumbroso, Edo Cohen-Karlik +3
Structured state space models (SSMs), the core engine behind prominent neural networks such as S4 and Mamba, are linear dynamical systems adhering to a specified structure, most no…
DeciMamba: Exploring the Length Extrapolation Potential of Mamba
Assaf Ben-Kish, Itamar Zimerman, Shady Abu-Hussein +4
Long-range sequence processing poses a significant challenge for Transformers due to their quadratic complexity in input length. A promising alternative is Mamba, which demonstrate…
ICC: Quantifying Image Caption Concreteness for Multimodal Dataset Curation
Moran Yanuka, Morris Alper, Hadar Averbuch-Elor +1
Web-scale training on paired text-image data is becoming increasingly central to multimodal learning, but is challenged by the highly noisy nature of datasets in the wild. Standard…
Implicit Bias of Policy Gradient in Linear Quadratic Control: Extrapolation to Unseen Initial States
Noam Razin, Yotam Alexander, Edo Cohen-Karlik +3
In modern machine learning, models can often fit training data in numerous ways, some of which perform well on unseen (test) data, while others do not. Remarkably, in such cases gr…