Publications (28)
DataMIL: Selecting Data for Robot Imitation Learning with Datamodels
Shivin Dass, Alaa Khaddaj, Logan Engstrom +3
Recently, the robotics community has amassed ever larger and more diverse datasets to train generalist policies. However, while these policies achieve strong mean performance acros…
When does Bias Transfer in Transfer Learning?
Hadi Salman, Saachi Jain, Andrew Ilyas +3
Using transfer learning to adapt a pre-trained "source model" to a downstream "target task" can dramatically increase performance with seemingly no downside. In this work, we demon…
Adversarial Robustness as a Prior for Learned Representations
Logan Engstrom, Andrew Ilyas, Shibani Santurkar +3
An important goal in deep learning is to learn versatile, high-level feature representations of input data. However, standard networks' representations seem to possess shortcomings…
Small-to-Large Generalization: Data Influences Models Consistently Across Scale
Alaa Khaddaj, Logan Engstrom, Aleksander Madry
Choice of training data distribution greatly influences model behavior. Yet, in large-scale settings, precisely characterizing how changes in training data affects predictions is o…
Do Adversarially Robust ImageNet Models Transfer Better?
Hadi Salman, Andrew Ilyas, Logan Engstrom +2
Transfer learning is a widely-used paradigm in deep learning, where models pre-trained on standard datasets can be efficiently adapted to downstream tasks. Typically, better pre-tr…
FFCV: Accelerating Training by Removing Data Bottlenecks
Guillaume Leclerc, Andrew Ilyas, Logan Engstrom +3
We present FFCV, a library for easy and fast machine learning model training. FFCV speeds up model training by eliminating (often subtle) data bottlenecks from the training process…
Prior Convictions: Black-Box Adversarial Attacks with Bandits and Priors
Andrew Ilyas, Logan Engstrom, Aleksander Madry
We study the problem of generating adversarial examples in a black-box setting in which only loss-oracle access to a model is available. We introduce a framework that conceptually…
MAGIC: Near-Optimal Data Attribution for Deep Learning
Andrew Ilyas, Logan Engstrom
The goal of predictive data attribution is to estimate how adding or removing a given set of training datapoints will affect model predictions. In convex settings, this goal is str…
Datamodels: Predicting Predictions from Training Data
Andrew Ilyas, Sung Min Park, Logan Engstrom +2
We present a conceptual framework, datamodeling, for analyzing the behavior of a model class in terms of the training data. For any fixed "target" example , training set , an…
Dataset Interfaces: Diagnosing Model Failures Using Controllable Counterfactual Generation
Joshua Vendrow, Saachi Jain, Logan Engstrom +1
Distribution shift is a major source of failure for machine learning models. However, evaluating model reliability under distribution shift can be challenging, especially since it…
A Closer Look at Deep Policy Gradients
Andrew Ilyas, Logan Engstrom, Shibani Santurkar +4
We study how the behavior of deep policy gradient algorithms reflects the conceptual framework motivating their development. To this end, we propose a fine-grained analysis of stat…
Synthesizing Robust Adversarial Examples
Anish Athalye, Logan Engstrom, Andrew Ilyas +1
Standard methods for generating adversarial examples for neural networks do not consistently fool neural network classifiers in the physical world due to a combination of viewpoint…
Optimizing ML Training with Metagradient Descent
Logan Engstrom, Andrew Ilyas, Benjamin Chen +3
A major challenge in training large-scale machine learning models is configuring the training process to maximize model performance, i.e., finding the best training setup from a va…
3DB: A Framework for Debugging Computer Vision Models
Guillaume Leclerc, Hadi Salman, Andrew Ilyas +9
We introduce 3DB: an extendable, unified framework for testing and debugging vision models using photorealistic simulation. We demonstrate, through a wide range of use cases, that…
Image Synthesis with a Single (Robust) Classifier
Shibani Santurkar, Dimitris Tsipras, Brandon Tran +3
We show that the basic classification framework alone can be used to tackle some of the most challenging tasks in image synthesis. In contrast to other state-of-the-art approaches,…
Black-box Adversarial Attacks with Limited Queries and Information
Andrew Ilyas, Logan Engstrom, Anish Athalye +1
Current neural network-based classifiers are susceptible to adversarial examples even in the black-box setting, where the attacker only has query access to the model. In practice,…
DsDm: Model-Aware Dataset Selection with Datamodels
Logan Engstrom, Axel Feldmann, Aleksander Madry
When selecting data for training large-scale models, standard practice is to filter for examples that match human notions of data quality. Such filtering yields qualitatively clean…
Robustness May Be at Odds with Accuracy
Dimitris Tsipras, Shibani Santurkar, Logan Engstrom +2
We show that there may exist an inherent tension between the goal of adversarial robustness and that of standard generalization. Specifically, training robust models may not only b…
Identifying Statistical Bias in Dataset Replication
Logan Engstrom, Andrew Ilyas, Shibani Santurkar +3
Dataset replication is a useful tool for assessing whether improvements in test accuracy on a specific benchmark correspond to improvements in models' ability to generalize reliabl…
Noise or Signal: The Role of Image Backgrounds in Object Recognition
Kai Xiao, Logan Engstrom, Andrew Ilyas +1
We assess the tendency of state-of-the-art object recognition models to depend on signals from image backgrounds. We create a toolkit for disentangling foreground and background si…
Adversarial Examples Are Not Bugs, They Are Features
Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras +3
Adversarial examples have attracted significant attention in machine learning, but the reasons for their existence and pervasiveness remain unclear. We demonstrate that adversarial…
Evaluating and Understanding the Robustness of Adversarial Logit Pairing
Logan Engstrom, Andrew Ilyas, Anish Athalye
We evaluate the robustness of Adversarial Logit Pairing, a recently proposed defense against adversarial examples. We find that a network trained with Adversarial Logit Pairing ach…
Unadversarial Examples: Designing Objects for Robust Vision
Hadi Salman, Andrew Ilyas, Logan Engstrom +3
We study a class of realistic computer vision settings wherein one can influence the design of the objects being recognized. We develop a framework that leverages this capability t…
Exploring the Landscape of Spatial Robustness
Logan Engstrom, Brandon Tran, Dimitris Tsipras +2
The study of adversarial robustness has so far largely focused on perturbations bound in p-norms. However, state-of-the-art models turn out to be also vulnerable to other, more nat…
Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPO
Logan Engstrom, Andrew Ilyas, Shibani Santurkar +4
We study the roots of algorithmic progress in deep policy gradient algorithms through a case study on two popular algorithms: Proximal Policy Optimization (PPO) and Trust Region Po…
From ImageNet to Image Classification: Contextualizing Progress on Benchmarks
Dimitris Tsipras, Shibani Santurkar, Logan Engstrom +2
Building rich machine learning datasets in a scalable manner often necessitates a crowd-sourced data collection pipeline. In this work, we use human studies to investigate the cons…
Query-Efficient Black-box Adversarial Examples (superceded)
Andrew Ilyas, Logan Engstrom, Anish Athalye +1
Note that this paper is superceded by "Black-Box Adversarial Attacks with Limited Queries and Information." Current neural network-based image classifiers are susceptible to advers…
OpenAI GPT-5 System Card
Aaditya Singh, Adam Fry, Adam Perelman +483
This is the system card published alongside the OpenAI GPT-5 launch, August 2025. GPT-5 is a unified system with a smart and fast model that answers most questions, a deeper reason…