Publications (76)
Semi-Supervised Learning with Context-Conditional Generative Adversarial Networks
Remi Denton, Sam Gross, Rob Fergus
We introduce a simple semi-supervised learning approach for images based on in-painting using an adversarial loss. Images with random patches removed are presented to a generator w…
OverFeat: Integrated Recognition, Localization and Detection using Convolutional Networks
Pierre Sermanet, David Eigen, Xiang Zhang +3
We present an integrated framework for using Convolutional Networks for classification, localization and detection. We show how a multiscale and sliding window approach can be effi…
Training Language Models on Synthetic Edit Sequences Improves Code Synthesis
Ulyana Piterbarg, Lerrel Pinto, Rob Fergus
Software engineers mainly write code by editing existing programs. In contrast, language models (LMs) autoregressively synthesize programs in a single pass. One explanation for thi…
Understanding Deep Architectures using a Recursive Convolutional Network
David Eigen, Jason Rolfe, Rob Fergus +1
A key challenge in designing convolutional network models is sizing them appropriately. Many factors are involved in these decisions, including number of layers, feature maps, kern…
Fast Adaptation via Policy-Dynamics Value Functions
Roberta Raileanu, Max Goldstein, Arthur Szlam +1
Standard RL algorithms assume fixed environment dynamics and require a significant amount of interaction to adapt to new environments. We introduce Policy-Dynamics Value Functions…
Automatic Data Augmentation for Generalization in Deep Reinforcement Learning
Roberta Raileanu, Max Goldstein, Denis Yarats +2
Deep reinforcement learning (RL) agents often fail to generalize to unseen scenarios, even when they are trained on many instances of semantically similar environments. Data augmen…
Empirically Verifying Hypotheses Using Reinforcement Learning
Kenneth Marino, Rob Fergus, Arthur Szlam +1
This paper formulates hypothesis verification as an RL problem. Specifically, we aim to build an agent that, given a hypothesis about the dynamics of the world, can take actions to…
Improving Sample Efficiency in Model-Free Reinforcement Learning from Images
Denis Yarats, Amy Zhang, Ilya Kostrikov +3
Training an agent to solve control tasks directly from high-dimensional images with model-free reinforcement learning (RL) has proven difficult. A promising approach is to learn a…
Maximizing Kepler science return per telemetered pixel: Searching the habitable zones of the brightest stars
Benjamin T. Montet, Ruth Angus, Tom Barclay +9
In today's mailing, Hogg et al. propose image modeling techniques to maintain 10-ppm-level precision photometry in Kepler data with only two working reaction wheels. While these re…
PaintBench: Deterministic Evaluation of Precise Visual Editing
Kai Xu, Ellis Brown, Shrikar Madhu +3
While current multimodal models are proficient at open-ended visual editing, executing precise single-answer edits remains an important obstacle. To probe this challenge, we introd…
De novo design of high-affinity protein binders with AlphaProteo
Vinicius Zambaldi, David La, Alexander E. Chu +29
Computational design of protein-binding proteins is a fundamental capability with broad utility in biomedical research and biotechnology. Recent methods have made strides against s…
Finding Generalizable Evidence by Learning to Convince Q&A Models
Ethan Perez, Siddharth Karamcheti, Rob Fergus +3
We propose a system that finds the strongest supporting evidence for a given answer to a question, using passage-based question-answering (QA) as a testbed. We train evidence agent…
Stochastic Pooling for Regularization of Deep Convolutional Neural Networks
Matthew D. Zeiler, Rob Fergus
We introduce a simple and effective method for regularizing large convolutional neural networks. We replace the conventional deterministic pooling operations with a stochastic proc…
Blind Deconvolution with Non-local Sparsity Reweighting
Dilip Krishnan, Joan Bruna, Rob Fergus
Blind deconvolution has made significant progress in the past decade. Most successful algorithms are classified either as Variational or Maximum a-Posteriori (). In spite of t…
Scaling Text-to-Image Diffusion Transformers with Representation Autoencoders
Shengbang Tong, Boyang Zheng, Ziteng Wang +7
Representation Autoencoders (RAEs) have shown distinct advantages in diffusion modeling on ImageNet by training in high-dimensional semantic latent spaces. In this work, we investi…
Imitation by Predicting Observations
Andrew Jaegle, Yury Sulsky, Arun Ahuja +3
Imitation learning enables agents to reuse and adapt the hard-won expertise of others, offering a solution to several key challenges in learning behavior. Although it is easy to ob…
Learning Goal Embeddings via Self-Play for Hierarchical Reinforcement Learning
Sainbayar Sukhbaatar, Emily Denton, Arthur Szlam +1
In hierarchical reinforcement learning a major challenge is determining appropriate low-level policies. We propose an unsupervised learning scheme, based on asymmetric self-play fr…
Hierarchical reinforcement learning with natural language subgoals
Arun Ahuja, Kavya Kopparapu, Rob Fergus +1
Hierarchical reinforcement learning has been a compelling approach for achieving goal directed behavior over long sequences of actions. However, it has been challenging to implemen…
Adaptive Retrieval and Scalable Indexing for k-NN Search with Cross-Encoders
Nishant Yadav, Nicholas Monath, Manzil Zaheer +2
Cross-encoder (CE) models which compute similarity by jointly encoding a query-item pair perform better than embedding-based models (dual-encoders) at estimating query-item relevan…
Stochastic Video Generation with a Learned Prior
Remi Denton, Rob Fergus
Generating video frames that accurately predict future world states is challenging. Existing approaches either fail to capture the full distribution of outcomes, or yield blurry ge…
Accelerating exploration and representation learning with offline pre-training
Bogdan Mazoure, Jake Bruce, Doina Precup +2
Sequential decision-making agents struggle with long horizon tasks, since solving them requires multi-step reasoning. Most reinforcement learning (RL) algorithms address this chall…
Distilling Internet-Scale Vision-Language Models into Embodied Agents
Theodore Sumers, Kenneth Marino, Arun Ahuja +2
Instruction-following agents must ground language into their observation and action spaces. Learning to ground language is challenging, typically requiring domain-specific engineer…
Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels
Ilya Kostrikov, Denis Yarats, Rob Fergus
We propose a simple data augmentation technique that can be applied to standard model-free reinforcement learning algorithms, enabling robust learning directly from pixels without…
Reinforcement Learning with Prototypical Representations
Denis Yarats, Rob Fergus, Alessandro Lazaric +1
Learning effective representations in image-based environments is crucial for sample efficient Reinforcement Learning (RL). Unfortunately, in RL, representation learning is confoun…
Learning Spatiotemporal Features with 3D Convolutional Networks
Du Tran, Lubomir Bourdev, Rob Fergus +2
We propose a simple, yet effective approach for spatiotemporal feature learning using deep 3-dimensional convolutional networks (3D ConvNets) trained on a large scale supervised vi…
Learning Multiagent Communication with Backpropagation
Sainbayar Sukhbaatar, Arthur Szlam, Rob Fergus
Many tasks in AI require the collaboration of multiple agents. Typically, the communication protocol between agents is manually specified and not altered during training. In this p…
Collaborating with language models for embodied reasoning
Ishita Dasgupta, Christine Kaeser-Chen, Kenneth Marino +4
Reasoning in a complex and ambiguous environment is a key goal for Reinforcement Learning (RL) agents. While some sophisticated RL agents can successfully solve difficult tasks, th…
Training Convolutional Networks with Noisy Labels
Sainbayar Sukhbaatar, Joan Bruna, Manohar Paluri +2
The availability of large labeled datasets has allowed Convolutional Network models to achieve impressive recognition results. However, in many settings manual annotation of the da…
Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs
Shengbang Tong, Ellis Brown, Penghao Wu +11
We introduce Cambrian-1, a family of multimodal LLMs (MLLMs) designed with a vision-centric approach. While stronger language models can enhance multimodal capabilities, the design…
Decoupling Value and Policy for Generalization in Reinforcement Learning
Roberta Raileanu, Rob Fergus
Standard deep reinforcement learning algorithms use a shared representation for the policy and value function, especially when training directly from images. However, we argue that…
Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks
Emily Denton, Soumith Chintala, Arthur Szlam +1
In this paper we introduce a generative parametric model capable of producing high quality samples of natural images. Our approach uses a cascade of convolutional networks within a…
SIMS-V: Simulated Instruction-Tuning for Spatial Video Understanding
Ellis Brown, Arijit Ray, Ranjay Krishna +3
Despite impressive high-level video comprehension, multimodal language models struggle with spatial reasoning across time and space. While current spatial training approaches rely…
Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learning
Denis Yarats, Rob Fergus, Alessandro Lazaric +1
We present DrQ-v2, a model-free reinforcement learning (RL) algorithm for visual continuous control. DrQ-v2 builds on DrQ, an off-policy actor-critic approach that uses data augmen…
Learning Physical Intuition of Block Towers by Example
Adam Lerer, Sam Gross, Rob Fergus
Wooden blocks are a common toy for infants, allowing them to develop motor skills and gain intuition about the physical behavior of the world. In this paper, we explore the ability…
Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
Remi Denton, Wojciech Zaremba, Joan Bruna +2
We present techniques for speeding up the test-time evaluation of large convolutional networks, designed for object recognition tasks. These models deliver impressive accuracy but…
Language Agents Mirror Human Causal Reasoning Biases. How Can We Help Them Think Like Scientists?
Anthony GX-Chen, Dongyan Lin, Mandana Samiei +4
Language model (LM) agents are increasingly used as autonomous decision-makers which need to actively gather information to guide their decisions. A crucial cognitive skill for suc…
Reduce, Reuse, Recycle: Compositional Generation with Energy-Based Diffusion Models and MCMC
Yilun Du, Conor Durkan, Robin Strudel +6
Since their introduction, diffusion models have quickly become the prevailing approach to generative modeling in many domains. They can be interpreted as learning the gradients of…
Visualizing and Understanding Convolutional Networks
Matthew D Zeiler, Rob Fergus
Large Convolutional Network models have recently demonstrated impressive classification performance on the ImageNet benchmark. However there is no clear understanding of why they p…
End-To-End Memory Networks
Sainbayar Sukhbaatar, Arthur Szlam, Jason Weston +1
We introduce a neural network with a recurrent attention model over a possibly large external memory. The architecture is a form of Memory Network (Weston et al., 2015) but unlike…
S4: A Spatial-Spectral model for Speckle Suppression
Rob Fergus, David W. Hogg, Rebecca Oppenheimer +2
High dynamic-range imagers aim to block out or null light from a very bright primary star to make it possible to detect and measure far fainter companions; in real systems a small…
Learning by Asking Questions
Ishan Misra, Ross Girshick, Rob Fergus +3
We introduce an interactive learning framework for the development and testing of intelligent visual systems, called learning-by-asking (LBA). We explore LBA in context of the Visu…
A Fresh Take on Stale Embeddings: Improving Dense Retriever Training with Corrector Networks
Nicholas Monath, Will Grathwohl, Michael Boratko +3
In dense retrieval, deep encoders provide embeddings for both inputs and targets, and the softmax function is used to parameterize a distribution over a large number of candidate t…
diff History for Neural Language Agents
Ulyana Piterbarg, Lerrel Pinto, Rob Fergus
Neural Language Models (LMs) offer an exciting solution for general-purpose embodied control. However, a key technical issue arises when using an LM-based controller: environment o…
Beyond Frontal Faces: Improving Person Recognition Using Multiple Cues
Ning Zhang, Manohar Paluri, Yaniv Taigman +2
We explore the task of recognizing peoples' identities in photo albums in an unconstrained setting. To facilitate this, we introduce the new People In Photo Albums (PIPA) dataset,…
Experience Graphs: The Data Foundation for Self-Improving Agents
Gang Liao, Yujia He, Abdullah Ozturk +22
The database community has repeatedly advanced the state of the art by recognizing that new workloads demand new system architectures. We argue that long-horizon agentic tasks -- c…
Cambrian-S: Towards Spatial Supersensing in Video
Shusheng Yang, Jihan Yang, Pinzhi Huang +12
We argue that progress in true multimodal intelligence calls for a shift from reactive, task-driven systems and brute-force long context towards a broader paradigm of supersensing.…
Simple Baseline for Visual Question Answering
Bolei Zhou, Yuandong Tian, Sainbayar Sukhbaatar +2
We describe a very simple bag-of-words baseline for visual question answering. This baseline concatenates the word features from the question and CNN features from the image to pre…
Improving Image Classification with Location Context
Kevin Tang, Manohar Paluri, Li Fei-Fei +2
With the widespread availability of cellphones and cameras that have GPS capabilities, it is common for images being uploaded to the Internet today to have GPS coordinates associat…
Maximizing Kepler science return per telemetered pixel: Detailed models of the focal plane in the two-wheel era
David W. Hogg, Ruth Angus, Tom Barclay +9
Kepler's immense photometric precision to date was maintained through satellite stability and precise pointing. In this white paper, we argue that image modeling--fitting the Keple…
EmbedDistill: A Geometric Knowledge Distillation for Information Retrieval
Seungyeon Kim, Ankit Singh Rawat, Manzil Zaheer +6
Large neural models (such as Transformers) achieve state-of-the-art performance for information retrieval (IR). In this paper, we aim to improve distillation methods that pave the…
End-to-End Integration of a Convolutional Network, Deformable Parts Model and Non-Maximum Suppression
Li Wan, David Eigen, Rob Fergus
Deformable Parts Models and Convolutional Networks each have achieved notable performance in object detection. Yet these two approaches find their strengths in complementary areas:…
BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games
Davide Paglieri, BartÅomiej CupiaÅ, Samuel Coward +10
Large Language Models (LLMs) and Vision Language Models (VLMs) possess extensive knowledge and exhibit promising reasoning abilities, however, they still struggle to perform well i…
Learning to Navigate Wikipedia by Taking Random Walks
Manzil Zaheer, Kenneth Marino, Will Grathwohl +7
A fundamental ability of an intelligent web-based agent is seeking out and acquiring new information. Internet search engines reliably find the correct vicinity but the top results…
Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-Scale Convolutional Architecture
David Eigen, Rob Fergus
In this paper we address three different computer vision tasks using a single basic architecture: depth prediction, surface normal estimation, and semantic labeling. We use a multi…
Efficient Exploration and Discriminative World Model Learning with an Object-Centric Abstraction
Anthony GX-Chen, Kenneth Marino, Rob Fergus
In the face of difficult exploration problems in reinforcement learning, we study whether giving an agent an object-centric mapping (describing a set of items and their attributes)…
Modeling Others using Oneself in Multi-Agent Reinforcement Learning
Roberta Raileanu, Emily Denton, Arthur Szlam +1
We consider the multi-agent reinforcement learning setting with imperfect information in which each agent is trying to maximize its own utility. The reward function depends on the…
MazeBase: A Sandbox for Learning from Games
Sainbayar Sukhbaatar, Arthur Szlam, Gabriel Synnaeve +2
This paper introduces MazeBase: an environment for simple 2D games, designed as a sandbox for machine learning approaches to reasoning and planning. Within it, we create 10 simple…
Understanding Outer Optimizers in Local SGD: Learning Rates, Momentum, and Acceleration
Ahmed Khaled, Satyen Kale, Arthur Douillard +3
Modern machine learning often requires training with large batch size, distributed data, and massively parallel compute hardware (like mobile and other edge devices or distributed…
Intrinsic Motivation and Automatic Curricula via Asymmetric Self-Play
Sainbayar Sukhbaatar, Zeming Lin, Ilya Kostrikov +3
We describe a simple scheme that allows an agent to learn about its environment in an unsupervised manner. Our scheme pits two versions of the same agent, Alice and Bob, against on…
Disentangling Video with Independent Prediction
William F. Whitney, Rob Fergus
We propose an unsupervised variational model for disentangling video into independent factors, i.e. each factor's future can be predicted from its past without considering the othe…
Intriguing properties of neural networks
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever +4
Deep neural networks are highly expressive models that have recently achieved state of the art performance on speech and visual recognition tasks. While their expressiveness is the…
Differentiable Pooling for Hierarchical Feature Learning
Matthew D. Zeiler, Rob Fergus
We introduce a parametric form of pooling, based on a Gaussian, which can be optimized alongside the features in a single global objective function. By contrast, existing pooling s…
Deep Poselets for Human Detection
Lubomir Bourdev, Fei Yang, Rob Fergus
We address the problem of detecting people in natural scenes using a part approach based on poselets. We propose a bootstrapping method that allows us to collect millions of weakly…
Offline Reinforcement Learning with Fisher Divergence Critic Regularization
Ilya Kostrikov, Jonathan Tompson, Rob Fergus +1
Many modern approaches to offline Reinforcement Learning (RL) utilize behavior regularization, typically augmenting a model-free actor critic algorithm with a penalty measuring div…
IntPhys: A Framework and Benchmark for Visual Intuitive Physics Reasoning
Ronan Riochet, Mario Ynocente Castro, Mathieu Bernard +4
In order to reach human performance on complexvisual tasks, artificial systems need to incorporate a sig-nificant amount of understanding of the world in termsof macroscopic object…
Depth Map Prediction from a Single Image using a Multi-Scale Deep Network
David Eigen, Christian Puhrsch, Rob Fergus
Predicting depth is an essential component in understanding the 3D geometry of a scene. While for stereo images local correspondence suffices for estimation, finding depth relation…
Learning Simple Algorithms from Examples
Wojciech Zaremba, Tomas Mikolov, Armand Joulin +1
We present an approach for learning simple algorithms such as copying, multi-digit addition and single digit multiplication directly from examples. Our framework consists of a set…
Composable Planning with Attributes
Amy Zhang, Adam Lerer, Sainbayar Sukhbaatar +2
The tasks that an agent will need to solve often are not known during training. However, if the agent knows which properties of the environment are important then, after learning h…
Deep End2End Voxel2Voxel Prediction
Du Tran, Lubomir Bourdev, Rob Fergus +2
Over the last few years deep learning methods have emerged as one of the most prominent approaches for video analysis. However, so far their most successful applications have been…
KL-Regularized Reinforcement Learning is Designed to Mode Collapse
Anthony GX-Chen, Jatin Prakash, Jeff Guo +2
It is commonly believed that optimizing the reverse KL divergence results in "mode seeking", while optimizing forward KL results in "mass covering", with the latter being preferred…
Beyond Language Modeling: An Exploration of Multimodal Pretraining
Shengbang Tong, David Fan, John Nguyen +18
The visual world offers a critical axis for advancing foundation models beyond language. Despite growing interest in this direction, the design space for native multimodal models r…
NetHack is Hard to Hack
Ulyana Piterbarg, Lerrel Pinto, Rob Fergus
Neural policy learning methods have achieved remarkable results in various control problems, ranging from Atari games to simulated locomotion. However, these methods struggle in lo…
Energy-based models for atomic-resolution protein conformations
Yilun Du, Joshua Meier, Jerry Ma +2
We propose an energy-based model (EBM) of protein conformations that operates at atomic scale. The model is trained solely on crystallized protein data. By contrast, existing appro…
Benchmark Designers Should "Train on the Test Set" to Expose Exploitable Non-Visual Shortcuts
Ellis Brown, Jihan Yang, Shusheng Yang +2
Robust benchmarks are crucial for evaluating Multimodal Large Language Models (MLLMs). Yet we find that models can ace many multimodal benchmarks without strong visual understandin…
Learning to Discover Efficient Mathematical Identities
Wojciech Zaremba, Karol Kurach, Rob Fergus
In this paper we explore how machine learning techniques can be applied to the discovery of efficient mathematical identities. We introduce an attribute grammar framework for repre…
Teacher Guided Training: An Efficient Framework for Knowledge Transfer
Manzil Zaheer, Ankit Singh Rawat, Seungyeon Kim +5
The remarkable performance gains realized by large pretrained models, e.g., GPT-3, hinge on the massive amounts of data they are exposed to during training. Analogously, distilling…