24 citations · 26 across the 5 of their papers we have counts for
6 papers
On the Importance of Gating: Memorization vs. In-Context Learning in State Space Models
William L. Tong, Aryo Lotfi, Emmanuel Abbe +6
State Space Models (SSMs) have emerged as a compelling alternative to Transformers, enabling sequence modeling with constant memory and linear compute. Although SSMs exhibit reason…
Boule or Baguette? A Study on Task Topology, Length Generalization, and the Benefit of Reasoning Traces
William L. Tong, Ege Cakar, Cengiz Pehlevan
Recent years have witnessed meteoric progress in reasoning models: neural networks that generate intermediate reasoning traces (RTs) before producing a final output. Despite the ra…
Learning richness modulates equality reasoning in neural networks
William L. Tong, Cengiz Pehlevan
Equality reasoning is ubiquitous and purely abstract: sameness or difference may be evaluated no matter the nature of the underlying objects. As a result, same-different (SD) tasks…
No Free Lunch From Random Feature Ensembles: Scaling Laws and Near-Optimality Conditions
Benjamin S. Ruben, William L. Tong, Hamza Tahir Chaudhry +1
Given a fixed budget for total model size, one must choose between training a single large model or combining the predictions of multiple smaller models. We investigate this trade-…
MLPs Learn In-Context on Regression and Classification Tasks
William L. Tong, Cengiz Pehlevan
In-context learning (ICL), the remarkable ability to solve a task from only input exemplars, is often assumed to be a unique hallmark of Transformer models. By examining commonly e…
Contrasting random and learned features in deep Bayesian linear regression
Jacob A. Zavatone-Veth, William L. Tong, Cengiz Pehlevan
Understanding how feature learning affects generalization is among the foremost goals of modern deep learning theory. Here, we study how the ability to learn representations affect…