1.6k citations · 2k across the 26 of their papers we have counts for
29 papers
A Cognitive Paradigm Approach to Probe the Perception-Reasoning Interface in VLMs
Mohit Vaishnav, Tanel Tammet
A fundamental challenge in artificial intelligence involves understanding the cognitive mechanisms underlying visual reasoning in sophisticated models like Vision-Language Models (…
Phi-4 Technical Report
Marah Abdin, Jyoti Aneja, Harkirat Behl +24
We present phi-4, a 14-billion parameter language model developed with a training recipe that is centrally focused on data quality. Unlike most language models, where pre-training…
Small Language Models for Application Interactions: A Case Study
Beibin Li, Yi Zhang, Sébastien Bubeck +2
We study the efficacy of Small Language Models (SLMs) in facilitating application usage through natural language interactions. Our focus here is on a particular internal applicatio…
Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
Marah Abdin, Jyoti Aneja, Hany Awadalla +126
We introduce phi-3-mini, a 3.8 billion parameter language model trained on 3.3 trillion tokens, whose overall performance, as measured by both academic benchmarks and internal test…
TinyGSM: achieving >80% on GSM8k with small language models
Bingbin Liu, Sebastien Bubeck, Ronen Eldan +5
Small-scale models offer various computational advantages, and yet to which extent size is critical for problem-solving abilities remains an open question. Specifically for solving…
Positional Description Matters for Transformers Arithmetic
Ruoqi Shen, Sébastien Bubeck, Ronen Eldan +3
Transformers, central to the successes in modern Natural Language Processing, often falter on arithmetic tasks despite their vast capabilities --which paradoxically include remarka…
Textbooks Are All You Need
Suriya Gunasekar, Yi Zhang, Jyoti Aneja +16
We introduce phi-1, a new large language model for code, with significantly smaller size than competing models: phi-1 is a Transformer-based model with 1.3B parameters, trained for…
Sparks of Artificial General Intelligence: Early experiments with GPT-4
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan +11
Artificial intelligence (AI) researchers have been developing and refining large language models (LLMs) that exhibit remarkable capabilities across a variety of domains and tasks,…
How to Fine-Tune Vision Models with SGD
Ananya Kumar, Ruoqi Shen, Sebastien Bubeck +1
SGD and AdamW are the two most used optimizers for fine-tuning large neural networks in computer vision. When the two methods perform the same, SGD is preferable because it uses le…
AutoMoE: Heterogeneous Mixture-of-Experts with Adaptive Computation for Efficient Neural Machine Translation
Ganesh Jawahar, Subhabrata Mukherjee, Xiaodong Liu +6
Mixture-of-Expert (MoE) models have obtained state-of-the-art performance in Neural Machine Translation (NMT) tasks. Existing works in MoE mostly consider a homogeneous design wher…
Unveiling Transformers with LEGO: a synthetic reasoning task
Yi Zhang, Arturs Backurs, Sébastien Bubeck +3
We propose a synthetic reasoning task, LEGO (Learning Equality and Group Operations), that encapsulates the problem of following a chain of reasoning, and we study how the Transfor…
LiteTransformerSearch: Training-free Neural Architecture Search for Efficient Language Models
Mojan Javaheripi, Gustavo H. de Rosa, Subhabrata Mukherjee +6
The Transformer architecture is ubiquitously used as the building block of large-scale autoregressive language models. However, finding architectures with the optimal trade-off bet…
Data Augmentation as Feature Manipulation
Ruoqi Shen, Sébastien Bubeck, Suriya Gunasekar
Data augmentation is a cornerstone of the machine learning pipeline, yet its theoretical underpinnings remain unclear. Is it merely a way to artificially augment the data set size?…
Adversarial Examples in Multi-Layer Random ReLU Networks
Peter L. Bartlett, Sébastien Bubeck, Yeshwanth Cherapanamjeri
We consider the phenomenon of adversarial examples in ReLU networks with independent gaussian parameters. For networks of constant depth and with a large range of widths (for insta…
FEAR: A Simple Lightweight Method to Rank Architectures
Debadeepta Dey, Shital Shah, Sebastien Bubeck
The fundamental problem in Neural Architecture Search (NAS) is to efficiently find high-performing architectures from a given search space. We propose a simple but powerful method…
A Universal Law of Robustness via Isoperimetry
Sébastien Bubeck, Mark Sellke
Classically, data interpolation with a parametrized model class is possible as long as the number of parameters is larger than the number of equations to be satisfied. A puzzling p…
A single gradient step finds adversarial examples on random two-layers neural networks
Sébastien Bubeck, Yeshwanth Cherapanamjeri, Gauthier Gidel +1
Daniely and Schacham recently showed that gradient descent finds adversarial examples on random undercomplete two-layers ReLU neural networks. The term "undercomplete" refers to th…
Cooperative and Stochastic Multi-Player Multi-Armed Bandit: Optimal Regret With Neither Communication Nor Collisions
Sébastien Bubeck, Thomas Budzinski, Mark Sellke
We consider the cooperative multi-player version of the stochastic multi-armed bandit problem. We study the regime where the players cannot communicate but have access to shared ra…
Metrical Service Systems with Transformations
Sébastien Bubeck, Niv Buchbinder, Christian Coester +1
We consider a generalization of the fundamental online metrical service systems (MSS) problem where the feasible region can be transformed between requests. In this problem, which…
A law of robustness for two-layers neural networks
Sébastien Bubeck, Yuanzhi Li, Dheeraj Nagaraj
We initiate the study of the inherent tradeoffs between the size of a neural network and its robustness, as measured by its Lipschitz constant. We make a precise conjecture that, f…
Network size and weights size for memorization with two-layers neural networks
Sébastien Bubeck, Ronen Eldan, Yin Tat Lee +1
In 1988, Eric B. Baum showed that two-layers neural networks with threshold activation function can perfectly memorize the binary labels of points in general position in $\math…
How to trap a gradient flow
Sébastien Bubeck, Dan Mikulincer
We consider the problem of finding an -approximate stationary point of a smooth function on a compact domain of . In contrast with dimension-free approac…
Complexity of Highly Parallel Non-Smooth Convex Optimization
Sébastien Bubeck, Qijia Jiang, Yin Tat Lee +2
A landmark result of non-smooth convex optimization is that gradient descent is an optimal algorithm whenever the number of computed gradients is smaller than the dimension . In…
Near-optimal method for highly smooth convex optimization
Sébastien Bubeck, Qijia Jiang, Yin Tat Lee +2
We propose a near-optimal method for highly smooth convex optimization. More precisely, in the oracle model where one obtains the order Taylor expansion of a function at t…