13 papers
Tight Sample Complexity of Transformers
Chenxiao Yang, Nathan Srebro, Zhiyuan Li
We tightly characterize the VC dimension of depth- Transformers with a total of parameters, mapping an input sequence of length to a single output, establishing an upper…
On Incentivized Exploration beyond Bayesianism and Full-Information
Dimitar Chakarov, Lee Cohen, Nathan Srebro
We extend Incentive Compatible Exploration beyond the Bayesian full-information setting of Kremer et al. [2014]. We consider agents that may possess external information unknown to…
Learning through Internalization
Nikolaos Tsilivis, Nirmit Joshi, Marko Medvedev +2
We study internalization processes, by which neural-network-based systems absorb an explicit computational procedure into their own weights, and how they facilitate learning. We in…
Learning to Think from Multiple Thinkers
Nirmit Joshi, Roey Magen, Nathan Srebro +2
We study learning with Chain-of-Thought (CoT) supervision from multiple thinkers, all of whom provide correct but possibly systematically different solutions, e.g., step-by-step so…
Recursive Models for Long-Horizon Reasoning
Chenxiao Yang, Nathan Srebro, Zhiyuan Li
Modern language models reason within bounded context, an inherent constraint that poses a fundamental barrier to long-horizon reasoning. We identify recursion as a core principle f…
Positive Distribution Shift as a Framework for Understanding Tractable Learning
Marko Medvedev, Idan Attias, Elisabetta Cornacchia +3
We study a setting where the goal is to learn a target function f(x) with respect to a target distribution D(x), but training is done on i.i.d. samples from a different training di…
On the Hardness of Learning Regular Expressions
Idan Attias, Lev Reyzin, Nathan Srebro +1
Despite the theoretical significance and wide practical use of regular expressions, the computational complexity of learning them has been largely unexplored. We study the computat…
Learning to Answer from Correct Demonstrations
Nirmit Joshi, Gene Li, Siddharth Bhandari +3
We study the problem of learning to generate an answer (or completion) to a question (or prompt), where there could be multiple correct answers, any one of which is acceptable at t…
Learning single-index models via harmonic decomposition
Nirmit Joshi, Hugo Koubbi, Theodor Misiakiewicz +1
We study the problem of learning single-index models, where the label depends on the input only through an unknown one-dimensio…
Temperature is All You Need for Generalization in Langevin Dynamics and other Markov Processes
Itamar Harel, Yonathan Wolanowsky, Gal Vardi +2
We analyze the generalization gap (gap between the training and test errors) when training a potentially over-parametrized model using a Markovian stochastic training algorithm, in…
A Theory of Learning with Autoregressive Chain of Thought
Nirmit Joshi, Gal Vardi, Adam Block +4
For a given base class of sequence-to-next-token generators, we consider learning prompt-to-answer mappings obtained by iterating a fixed, time-invariant generator for multiple ste…
Score Design for Multi-Criteria Incentivization
Anmol Kabra, Mina Karzand, Tosca Lechner +2
We present a framework for designing scores to summarize performance metrics. Our design has two multi-criteria objectives: (1) improving on scores should improve all performance m…
On the Complexity of Learning Sparse Functions with Statistical and Gradient Queries
Nirmit Joshi, Theodor Misiakiewicz, Nathan Srebro
The goal of this paper is to investigate the complexity of gradient algorithms when learning sparse functions (juntas). We introduce a type of Statistical Queries (),…