39 citations · 158 across the 36 of their papers we have counts for
41 papers
Hard Way, The Only Way: Provable Merits of Training on Hardest Examples
Elvis Dohmatob
Despite a surge of empirical interest in the ``unreasonable effectiveness'' of drastic data pruning for AI systems, a unified theoretical understanding of this phenomenon remains e…
Exact Scaling Laws for a Random-Features Bigram Model
Julian Enoch, Elvis Dohmatob
Understanding neural scaling laws is increasingly important for theory and practice, yet first-principles explanations remain limited even in simple settings. In this work, we stud…
Statistical Vignettes on the Grokking Phenomenon
Baye Malick Gning, Elvis Dohmatob
In this work, we study grokking in a simple, analytically tractable setting whose rich phenomenology allows us to disentangle the effects of initialization, signal strength, weight…
Efficient Refusal Ablation in LLM through Optimal Transport
Geraldin Nanfack, Eugene Belilovsky, Elvis Dohmatob
Safety-aligned language models refuse harmful requests through learned refusal behaviors encoded in their internal representations. Recent activation-based jailbreaking methods cir…
Why Less is More (Sometimes): A Theory of Data Curation
Elvis Dohmatob, Mohammad Pezeshki, Reyhane Askari-Hemmat
This paper introduces a theoretical framework to resolve a central paradox in modern machine learning: When is it better to use less data? This question has become critical as clas…
Egalitarian Gradient Descent: A Simple Approach to Accelerated Grokking
Ali Saheb Pasand, Elvis Dohmatob
Grokking is the phenomenon whereby, unlike the training performance, which peaks early in the training process, the test/generalization performance of a model stagnates over arbitr…
auto-fpt: Automating Free Probability Theory Calculations for Machine Learning Theory
Arjun Subramonian, Elvis Dohmatob
A large part of modern machine learning theory often involves computing the high-dimensional expected trace of a rational expression of large rectangular random matrices. To symbol…
Improving the Scaling Laws of Synthetic Data with Deliberate Practice
Reyhane Askari-Hemmat, Mohammad Pezeshki, Elvis Dohmatob +6
Inspired by the principle of deliberate practice in human learning, we propose Deliberate Practice for Synthetic Data Generation (DP), a novel framework that improves sample effici…
The Pitfalls of Memorization: When Memorization Hurts Generalization
Reza Bayat, Mohammad Pezeshki, Elvis Dohmatob +2
Neural networks often learn simple explanations that fit the majority of the data while memorizing exceptions that deviate from these explanations.This behavior leads to poor gener…
Strong Model Collapse
Elvis Dohmatob, Yunzhen Feng, Arjun Subramonian +1
Within the scaling laws paradigm, which underpins the training of large neural networks like ChatGPT and Llama, we consider a supervised regression setting and establish the exista…
An Effective Theory of Bias Amplification
Arjun Subramonian, Samuel J. Bell, Levent Sagun +1
Machine learning models can capture and amplify biases present in data, leading to disparate test performance across social groups. To better understand, evaluate, and mitigate the…
Beyond Model Collapse: Scaling Up with Synthesized Data Requires Verification
Yunzhen Feng, Elvis Dohmatob, Pu Yang +2
Large Language Models (LLM) are increasingly trained on data generated by other LLM, either because generated text and images become part of the pre-training corpus, or because syn…
Model Collapse Demystified: The Case of Regression
Elvis Dohmatob, Yunzhen Feng, Julia Kempe
In the era of proliferation of large language and image generation models, the phenomenon of "model collapse" refers to the situation whereby as a model is trained recursively on d…
A Tale of Tails: Model Collapse as a Change of Scaling Laws
Elvis Dohmatob, Yunzhen Feng, Pu Yang +2
As AI model size grows, neural scaling laws have become a crucial tool to predict the improvements of large models when increasing capacity and the size of original (human or natur…
Scaling Laws for Associative Memories
Vivien Cabannes, Elvis Dohmatob, Alberto Bietti
Learning arguably involves the discovery and memorization of abstract rules. The aim of this paper is to study associative memory mechanisms. Our model is based on high-dimensional…
Robust Linear Regression: Phase-Transitions and Precise Tradeoffs for General Norms
Elvis Dohmatob, Meyer Scetbon
In this paper, we investigate the impact of test-time adversarial attacks on linear regression models and determine the optimal level of robustness that any model can reach while m…
Robust Linear Regression: Gradient-descent, Early-stopping, and Beyond
Meyer Scetbon, Elvis Dohmatob
In this work we study the robustness to adversarial attacks, of early-stopping strategies on gradient-descent (GD) methods for linear regression. More precisely, we show that early…
An Adversarial Robustness Perspective on the Topology of Neural Networks
Morgane Goibert, Thomas Ricatte, Elvis Dohmatob
In this paper, we investigate the impact of neural networks (NNs) topology on adversarial robustness. Specifically, we study the graph produced when an input traverses all the laye…
Contextual bandits with concave rewards, and an application to fair ranking
Virginie Do, Elvis Dohmatob, Matteo Pirotta +2
We consider Contextual Bandits with Concave Rewards (CBCR), a multi-objective bandit problem where the desired trade-off between the rewards is defined by a known concave objective…
Fast online ranking with fairness of exposure
Nicolas Usunier, Virginie Do, Elvis Dohmatob
As recommender systems become increasingly central for sorting and prioritizing the content available online, they have a growing impact on the opportunities or revenue of their it…
Scalable MCMC Sampling for Nonsymmetric Determinantal Point Processes
Insu Han, Mike Gartrell, Elvis Dohmatob +1
A determinantal point process (DPP) is an elegant model that assigns a probability to every subset of a collection of items. While conventionally a DPP is parameterized by a sy…
Origins of Low-dimensional Adversarial Perturbations
Elvis Dohmatob, Chuan Guo, Morgane Goibert
In this paper, we initiate a rigorous study of the phenomenon of low-dimensional adversarial perturbations (LDAPs) in classification. Unlike the classical setting, these perturbati…
On the (Non-)Robustness of Two-Layer Neural Networks in Different Learning Regimes
Elvis Dohmatob, Alberto Bietti
Neural networks are known to be highly sensitive to adversarial examples. These may arise due to different factors, such as random initialization, or spurious correlations in the l…
Scalable Sampling for Nonsymmetric Determinantal Point Processes
Insu Han, Mike Gartrell, Jennifer Gillenwater +2
A determinantal point process (DPP) on a collection of items is a model, parameterized by a symmetric kernel matrix, that assigns a probability to every subset of those items.…