1.6k citations · 2k across the 26 of their papers we have counts for
11 papers · 1 filter
TinyGSM: achieving >80% on GSM8k with small language models
Bingbin Liu, Sebastien Bubeck, Ronen Eldan +5
Small-scale models offer various computational advantages, and yet to which extent size is critical for problem-solving abilities remains an open question. Specifically for solving…
Unveiling Transformers with LEGO: a synthetic reasoning task
Yi Zhang, Arturs Backurs, Sébastien Bubeck +3
We propose a synthetic reasoning task, LEGO (Learning Equality and Group Operations), that encapsulates the problem of following a chain of reasoning, and we study how the Transfor…
LiteTransformerSearch: Training-free Neural Architecture Search for Efficient Language Models
Mojan Javaheripi, Gustavo H. de Rosa, Subhabrata Mukherjee +6
The Transformer architecture is ubiquitously used as the building block of large-scale autoregressive language models. However, finding architectures with the optimal trade-off bet…
Data Augmentation as Feature Manipulation
Ruoqi Shen, Sébastien Bubeck, Suriya Gunasekar
Data augmentation is a cornerstone of the machine learning pipeline, yet its theoretical underpinnings remain unclear. Is it merely a way to artificially augment the data set size?…
Adversarial Examples in Multi-Layer Random ReLU Networks
Peter L. Bartlett, Sébastien Bubeck, Yeshwanth Cherapanamjeri
We consider the phenomenon of adversarial examples in ReLU networks with independent gaussian parameters. For networks of constant depth and with a large range of widths (for insta…
FEAR: A Simple Lightweight Method to Rank Architectures
Debadeepta Dey, Shital Shah, Sebastien Bubeck
The fundamental problem in Neural Architecture Search (NAS) is to efficiently find high-performing architectures from a given search space. We propose a simple but powerful method…
A Universal Law of Robustness via Isoperimetry
Sébastien Bubeck, Mark Sellke
Classically, data interpolation with a parametrized model class is possible as long as the number of parameters is larger than the number of equations to be satisfied. A puzzling p…
A single gradient step finds adversarial examples on random two-layers neural networks
Sébastien Bubeck, Yeshwanth Cherapanamjeri, Gauthier Gidel +1
Daniely and Schacham recently showed that gradient descent finds adversarial examples on random undercomplete two-layers ReLU neural networks. The term "undercomplete" refers to th…
Cooperative and Stochastic Multi-Player Multi-Armed Bandit: Optimal Regret With Neither Communication Nor Collisions
Sébastien Bubeck, Thomas Budzinski, Mark Sellke
We consider the cooperative multi-player version of the stochastic multi-armed bandit problem. We study the regime where the players cannot communicate but have access to shared ra…
A law of robustness for two-layers neural networks
Sébastien Bubeck, Yuanzhi Li, Dheeraj Nagaraj
We initiate the study of the inherent tradeoffs between the size of a neural network and its robustness, as measured by its Lipschitz constant. We make a precise conjecture that, f…
Network size and weights size for memorization with two-layers neural networks
Sébastien Bubeck, Ronen Eldan, Yin Tat Lee +1
In 1988, Eric B. Baum showed that two-layers neural networks with threshold activation function can perfectly memorize the binary labels of points in general position in $\math…