8 citations · 17 across the 16 of their papers we have counts for
13 papers · 1 filter
On the Importance of Gating: Memorization vs. In-Context Learning in State Space Models
William L. Tong, Aryo Lotfi, Emmanuel Abbe +6
State Space Models (SSMs) have emerged as a compelling alternative to Transformers, enabling sequence modeling with constant memory and linear compute. Although SSMs exhibit reason…
To Infinity and Beyond: Tool-Use Unlocks Length Generalization in State Space Models
Eran Malach, Omid Saremi, Sinead Williamson +5
State Space Models (SSMs) have become the leading alternative to Transformers for sequence modeling. Their primary advantage is efficiency in long-context and long-form generation,…
Rethinking JEPA: Compute-Efficient Video SSL with Frozen Teachers
Xianhang Li, Chen Huang, Chun-Liang Li +4
Video Joint Embedding Predictive Architectures (V-JEPA) learn generalizable off-the-shelf video representation by predicting masked regions in latent space with an exponential movi…
Proxy-FDA: Proxy-based Feature Distribution Alignment for Fine-tuning Vision Foundation Models without Forgetting
Chen Huang, Skyler Seto, Hadi Pouransari +6
Vision foundation models pre-trained on massive data encode rich representations of real-world concepts, which can be adapted to downstream tasks by fine-tuning. However, fine-tuni…
Mechanisms of Projective Composition of Diffusion Models
Arwen Bradley, Preetum Nakkiran, David Berthelot +2
We study the theoretical foundations of composition in diffusion models, with a particular focus on out-of-distribution extrapolation and length-generalization. Prior work has show…
Parameters vs FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of-Experts Language Models
Samira Abnar, Harshay Shah, Dan Busbridge +3
Scaling the capacity of language models has consistently proven to be a reliable approach for improving performance and unlocking new capabilities. Capacity can be primarily define…
Aggregate-and-Adapt Natural Language Prompts for Downstream Generalization of CLIP
Chen Huang, Skyler Seto, Samira Abnar +3
Large pretrained vision-language models like CLIP have shown promising generalization capability, but may struggle in specialized domains (e.g., satellite imagery) or fine-grained…
How JEPA Avoids Noisy Features: The Implicit Bias of Deep Linear Self Distillation Networks
Etai Littwin, Omid Saremi, Madhu Advani +4
Two competing paradigms exist for self-supervised learning of data representations. Joint Embedding Predictive Architecture (JEPA) is a class of architectures in which semantically…
LiDAR: Sensing Linear Probing Performance in Joint Embedding SSL Architectures
Vimal Thilak, Chen Huang, Omid Saremi +5
Joint embedding (JE) architectures have emerged as a promising avenue for acquiring transferable data representations. A key obstacle to using JE methods, however, is the inherent…
What Algorithms can Transformers Learn? A Study in Length Generalization
Hattie Zhou, Arwen Bradley, Etai Littwin +5
Large language models exhibit surprising emergent generalization properties, yet also struggle on many simple reasoning tasks such as arithmetic and parity. This raises the questio…
Adaptivity and Modularity for Efficient Generalization Over Task Complexity
Samira Abnar, Omid Saremi, Laurent Dinh +8
Can transformers generalize efficiently on problems that require dealing with examples with different levels of difficulty? We introduce a new task tailored to assess generalizatio…
Vanishing Gradients in Reinforcement Finetuning of Language Models
Noam Razin, Hattie Zhou, Omid Saremi +5
Pretrained language models are commonly aligned with human preferences and downstream tasks via reinforcement finetuning (RFT), which refers to maximizing a (possibly learned) rewa…
Boolformer: Symbolic Regression of Logic Functions with Transformers
Stéphane d'Ascoli, Arthur Renard, Vassilis Papadopoulos +3
We introduce Boolformer, a Transformer-based model trained to perform end-to-end symbolic regression of Boolean functions. First, we show that it can predict compact formulas for c…