39 citations · 158 across the 36 of their papers we have counts for
21 papers · 1 filter
Hard Way, The Only Way: Provable Merits of Training on Hardest Examples
Elvis Dohmatob
Despite a surge of empirical interest in the ``unreasonable effectiveness'' of drastic data pruning for AI systems, a unified theoretical understanding of this phenomenon remains e…
Exact Scaling Laws for a Random-Features Bigram Model
Julian Enoch, Elvis Dohmatob
Understanding neural scaling laws is increasingly important for theory and practice, yet first-principles explanations remain limited even in simple settings. In this work, we stud…
Statistical Vignettes on the Grokking Phenomenon
Baye Malick Gning, Elvis Dohmatob
In this work, we study grokking in a simple, analytically tractable setting whose rich phenomenology allows us to disentangle the effects of initialization, signal strength, weight…
Efficient Refusal Ablation in LLM through Optimal Transport
Geraldin Nanfack, Eugene Belilovsky, Elvis Dohmatob
Safety-aligned language models refuse harmful requests through learned refusal behaviors encoded in their internal representations. Recent activation-based jailbreaking methods cir…
Why Less is More (Sometimes): A Theory of Data Curation
Elvis Dohmatob, Mohammad Pezeshki, Reyhane Askari-Hemmat
This paper introduces a theoretical framework to resolve a central paradox in modern machine learning: When is it better to use less data? This question has become critical as clas…
Egalitarian Gradient Descent: A Simple Approach to Accelerated Grokking
Ali Saheb Pasand, Elvis Dohmatob
Grokking is the phenomenon whereby, unlike the training performance, which peaks early in the training process, the test/generalization performance of a model stagnates over arbitr…
auto-fpt: Automating Free Probability Theory Calculations for Machine Learning Theory
Arjun Subramonian, Elvis Dohmatob
A large part of modern machine learning theory often involves computing the high-dimensional expected trace of a rational expression of large rectangular random matrices. To symbol…
Improving the Scaling Laws of Synthetic Data with Deliberate Practice
Reyhane Askari-Hemmat, Mohammad Pezeshki, Elvis Dohmatob +6
Inspired by the principle of deliberate practice in human learning, we propose Deliberate Practice for Synthetic Data Generation (DP), a novel framework that improves sample effici…
The Pitfalls of Memorization: When Memorization Hurts Generalization
Reza Bayat, Mohammad Pezeshki, Elvis Dohmatob +2
Neural networks often learn simple explanations that fit the majority of the data while memorizing exceptions that deviate from these explanations.This behavior leads to poor gener…
Strong Model Collapse
Elvis Dohmatob, Yunzhen Feng, Arjun Subramonian +1
Within the scaling laws paradigm, which underpins the training of large neural networks like ChatGPT and Llama, we consider a supervised regression setting and establish the exista…
An Effective Theory of Bias Amplification
Arjun Subramonian, Samuel J. Bell, Levent Sagun +1
Machine learning models can capture and amplify biases present in data, leading to disparate test performance across social groups. To better understand, evaluate, and mitigate the…
Beyond Model Collapse: Scaling Up with Synthesized Data Requires Verification
Yunzhen Feng, Elvis Dohmatob, Pu Yang +2
Large Language Models (LLM) are increasingly trained on data generated by other LLM, either because generated text and images become part of the pre-training corpus, or because syn…
Model Collapse Demystified: The Case of Regression
Elvis Dohmatob, Yunzhen Feng, Julia Kempe
In the era of proliferation of large language and image generation models, the phenomenon of "model collapse" refers to the situation whereby as a model is trained recursively on d…
A Tale of Tails: Model Collapse as a Change of Scaling Laws
Elvis Dohmatob, Yunzhen Feng, Pu Yang +2
As AI model size grows, neural scaling laws have become a crucial tool to predict the improvements of large models when increasing capacity and the size of original (human or natur…
An Adversarial Robustness Perspective on the Topology of Neural Networks
Morgane Goibert, Thomas Ricatte, Elvis Dohmatob
In this paper, we investigate the impact of neural networks (NNs) topology on adversarial robustness. Specifically, we study the graph produced when an input traverses all the laye…
Contextual bandits with concave rewards, and an application to fair ranking
Virginie Do, Elvis Dohmatob, Matteo Pirotta +2
We consider Contextual Bandits with Concave Rewards (CBCR), a multi-objective bandit problem where the desired trade-off between the rewards is defined by a known concave objective…
Scalable MCMC Sampling for Nonsymmetric Determinantal Point Processes
Insu Han, Mike Gartrell, Elvis Dohmatob +1
A determinantal point process (DPP) is an elegant model that assigns a probability to every subset of a collection of items. While conventionally a DPP is parameterized by a sy…
Scalable Sampling for Nonsymmetric Determinantal Point Processes
Insu Han, Mike Gartrell, Jennifer Gillenwater +2
A determinantal point process (DPP) on a collection of items is a model, parameterized by a symmetric kernel matrix, that assigns a probability to every subset of those items.…
Scalable Learning and MAP Inference for Nonsymmetric Determinantal Point Processes
Mike Gartrell, Insu Han, Elvis Dohmatob +2
Determinantal point processes (DPPs) have attracted significant attention in machine learning for their ability to model subsets drawn from a large item collection. Recent work sho…
Adversarial Robustness via Label-Smoothing
Morgane Goibert, Elvis Dohmatob
We study Label-Smoothing as a means for improving adversarial robustness of supervised deep-learning models. After establishing a thorough and unified framework, we propose several…
Learning Nonsymmetric Determinantal Point Processes
Mike Gartrell, Victor-Emmanuel Brunel, Elvis Dohmatob +1
Determinantal point processes (DPPs) have attracted substantial attention as an elegant probabilistic model that captures the balance between quality and diversity within sets. DPP…