8 citations · 17 across the 16 of their papers we have counts for
18 papers
A Four-by-Four Obstruction to Polynomial Removal for Ordered Binary Matrices
Zinan Wang
We exhibit a fixed binary matrix with no polynomial induced ordered removal bound. At depth , let . Our explicit host has order , exactly $2m^…
On the Importance of Gating: Memorization vs. In-Context Learning in State Space Models
William L. Tong, Aryo Lotfi, Emmanuel Abbe +6
State Space Models (SSMs) have emerged as a compelling alternative to Transformers, enabling sequence modeling with constant memory and linear compute. Although SSMs exhibit reason…
To Infinity and Beyond: Tool-Use Unlocks Length Generalization in State Space Models
Eran Malach, Omid Saremi, Sinead Williamson +5
State Space Models (SSMs) have become the leading alternative to Transformers for sequence modeling. Their primary advantage is efficiency in long-context and long-form generation,…
Rethinking JEPA: Compute-Efficient Video SSL with Frozen Teachers
Xianhang Li, Chen Huang, Chun-Liang Li +4
Video Joint Embedding Predictive Architectures (V-JEPA) learn generalizable off-the-shelf video representation by predicting masked regions in latent space with an exponential movi…
How PARTs assemble into wholes: Learning the relative composition of images
Melika Ayoughi, Samira Abnar, Chen Huang +10
The composition of objects and their parts, along with object-object positional relationships, provides a rich source of information for representation learning. Hence, spatial-awa…
Proxy-FDA: Proxy-based Feature Distribution Alignment for Fine-tuning Vision Foundation Models without Forgetting
Chen Huang, Skyler Seto, Hadi Pouransari +6
Vision foundation models pre-trained on massive data encode rich representations of real-world concepts, which can be adapted to downstream tasks by fine-tuning. However, fine-tuni…
Mechanisms of Projective Composition of Diffusion Models
Arwen Bradley, Preetum Nakkiran, David Berthelot +2
We study the theoretical foundations of composition in diffusion models, with a particular focus on out-of-distribution extrapolation and length-generalization. Prior work has show…
Parameters vs FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of-Experts Language Models
Samira Abnar, Harshay Shah, Dan Busbridge +3
Scaling the capacity of language models has consistently proven to be a reliable approach for improving performance and unlocking new capabilities. Capacity can be primarily define…
Aggregate-and-Adapt Natural Language Prompts for Downstream Generalization of CLIP
Chen Huang, Skyler Seto, Samira Abnar +3
Large pretrained vision-language models like CLIP have shown promising generalization capability, but may struggle in specialized domains (e.g., satellite imagery) or fine-grained…
How JEPA Avoids Noisy Features: The Implicit Bias of Deep Linear Self Distillation Networks
Etai Littwin, Omid Saremi, Madhu Advani +4
Two competing paradigms exist for self-supervised learning of data representations. Joint Embedding Predictive Architecture (JEPA) is a class of architectures in which semantically…
Overcoming the Pitfalls of Vision-Language Model Finetuning for OOD Generalization
Yuhang Zang, Hanlin Goh, Josh Susskind +1
Existing vision-language models exhibit strong generalization on a variety of visual domains and tasks. However, such models mainly perform zero-shot recognition in a closed-set ma…
LiDAR: Sensing Linear Probing Performance in Joint Embedding SSL Architectures
Vimal Thilak, Chen Huang, Omid Saremi +5
Joint embedding (JE) architectures have emerged as a promising avenue for acquiring transferable data representations. A key obstacle to using JE methods, however, is the inherent…
What Algorithms can Transformers Learn? A Study in Length Generalization
Hattie Zhou, Arwen Bradley, Etai Littwin +5
Large language models exhibit surprising emergent generalization properties, yet also struggle on many simple reasoning tasks such as arithmetic and parity. This raises the questio…
Adaptivity and Modularity for Efficient Generalization Over Task Complexity
Samira Abnar, Omid Saremi, Laurent Dinh +8
Can transformers generalize efficiently on problems that require dealing with examples with different levels of difficulty? We introduce a new task tailored to assess generalizatio…
Vanishing Gradients in Reinforcement Finetuning of Language Models
Noam Razin, Hattie Zhou, Omid Saremi +5
Pretrained language models are commonly aligned with human preferences and downstream tasks via reinforcement finetuning (RFT), which refers to maximizing a (possibly learned) rewa…
When can transformers reason with abstract symbols?
Enric Boix-Adsera, Omid Saremi, Emmanuel Abbe +3
We investigate the capabilities of transformer models on relational reasoning tasks. In these tasks, models are trained on a set of strings encoding abstract relations, and are the…
Construction of Paired Knowledge Graph-Text Datasets Informed by Cyclic Evaluation
Ali Mousavi, Xin Zhan, He Bai +9
Datasets that pair Knowledge Graphs (KG) and text together (KG-T) can be used to train forward and reverse neural models that generate text from KG and vice versa. However models t…
Boolformer: Symbolic Regression of Logic Functions with Transformers
Stéphane d'Ascoli, Arthur Renard, Vassilis Papadopoulos +3
We introduce Boolformer, a Transformer-based model trained to perform end-to-end symbolic regression of Boolean functions. First, we show that it can predict compact formulas for c…