Publications (36)
Shaking the foundations: delusions in sequence models for interaction and control
Pedro A. Ortega, Markus Kunesch, Grégoire Delétang +16
The recent phenomenal success of language models has reinvigorated machine learning research, and large sequence models such as transformers are being applied to a variety of domai…
Learning Compositional Neural Programs with Recursive Tree Search and Planning
Thomas Pierrot, Guillaume Ligner, Scott Reed +6
We propose a novel reinforcement learning algorithm, AlphaNPI, that incorporates the strengths of Neural Programmer-Interpreters (NPI) and AlphaZero. NPI contributes structural bia…
Scalable, High-Quality Object Detection
Christian Szegedy, Scott Reed, Dumitru Erhan +2
Current high-quality object detection approaches use the scheme of salience-based object proposal methods followed by post-classification using deep convolutional features. This sp…
Neural Programmer-Interpreters
Scott Reed, Nando de Freitas
We propose the neural programmer-interpreter (NPI): a recurrent and compositional neural network that learns to represent and execute programs. NPI has three learnable components:…
Neural Arithmetic Logic Units
Andrew Trask, Felix Hill, Scott Reed +3
Neural networks can learn to represent and manipulate numerical information, but they seldom generalize well outside of the range of numerical values encountered during training. T…
Scaling data-driven robotics with reward sketching and batch reinforcement learning
Serkan Cabi, Sergio Gómez Colmenarejo, Alexander Novikov +13
We present a framework for data-driven robotics that makes use of a large dataset of recorded robot experience and scales to several tasks using learned reward functions. We show h…
World Action Models are Zero-shot Policies
Seonghyeon Ye, Yunhao Ge, Kaiyuan Zheng +33
State-of-the-art Vision-Language-Action (VLA) models excel at semantic generalization but struggle to generalize to unseen physical motions in novel environments. We introduce Drea…
One-Shot High-Fidelity Imitation: Training Large-Scale Deep Nets with RL
Tom Le Paine, Sergio Gómez Colmenarejo, Ziyu Wang +8
Humans are experts at high-fidelity imitation -- closely mimicking a demonstration, often in one attempt. Humans use this ability to quickly solve a task instance, and to bootstrap…
GR00T N1: An Open Foundation Model for Generalist Humanoid Robots
NVIDIA, :, Johan Bjorck +40
General-purpose robots need a versatile body and an intelligent mind. Recent advancements in humanoid robots have shown great promise as a hardware platform for building generalist…
Critic Regularized Regression
Ziyu Wang, Alexander Novikov, Konrad Zolna +8
Offline reinforcement learning (RL), also known as batch RL, offers the prospect of policy optimization from large pre-recorded datasets without online environment interaction. It…
FLARE: Robot Learning with Implicit World Modeling
Ruijie Zheng, Jing Wang, Scott Reed +18
We introduce uture tent presentation Alignment (), a novel framework that integrates predictive latent world modeling into rob…
RoboCat: A Self-Improving Generalist Agent for Robotic Manipulation
Konstantinos Bousmalis, Giulia Vezzani, Dushyant Rao +36
The ability to leverage heterogeneous robotic experience from different robots and tasks to quickly master novel skills and embodiments has the potential to transform robot learnin…
Few-shot Autoregressive Density Estimation: Towards Learning to Learn Distributions
Scott Reed, Yutian Chen, Thomas Paine +5
Deep autoregressive models have shown state-of-the-art performance in density estimation for natural images on large-scale datasets such as ImageNet. However, such models require m…
Task-Relevant Adversarial Imitation Learning
Konrad Zolna, Scott Reed, Alexander Novikov +6
We show that a critical vulnerability in adversarial imitation is the tendency of discriminator networks to learn spurious associations between visual features and expert labels. W…
Sample Efficient Adaptive Text-to-Speech
Yutian Chen, Yannis Assael, Brendan Shillingford +11
We present a meta-learning approach for adaptive text-to-speech (TTS) with few data. During training, we learn a multi-speaker model using a shared conditional WaveNet core and ind…
Learning What and Where to Draw
Scott Reed, Zeynep Akata, Santosh Mohan +3
Generative Adversarial Networks (GANs) have recently demonstrated the capability to synthesize compelling real-world images, such as room interiors, album covers, manga, faces, bir…
Genie: Generative Interactive Environments
Jake Bruce, Michael Dennis, Ashley Edwards +22
We introduce Genie, the first generative interactive environment trained in an unsupervised manner from unlabelled Internet videos. The model can be prompted to generate an endless…
RoboTTT: Context Scaling for Robot Policies
Yunfan Jiang, Yevgen Chebotar, Ruijie Zheng +8
The paper introduces RoboTTT, a robot policy that uses test-time training to handle up to 8,000 timesteps of visual‑motor context, enabling one‑shot imitation from video, on‑the‑fl…
Generative Adversarial Text to Image Synthesis
Scott Reed, Zeynep Akata, Xinchen Yan +3
Automatic synthesis of realistic images from text would be interesting and useful, but current AI systems are still far from this goal. However, in recent years generic and powerfu…
Weakly-supervised Disentangling with Recurrent Transformations for 3D View Synthesis
Jimei Yang, Scott Reed, Ming-Hsuan Yang +1
An important problem for both graphics and vision is to synthesize novel views of a 3D object from a single image. This is particularly challenging due to the partial observability…
Semi-supervised reward learning for offline reinforcement learning
Ksenia Konyushkova, Konrad Zolna, Yusuf Aytar +4
In offline reinforcement learning (RL) agents are trained using a logged dataset. It appears to be the most natural route to attack real-life applications because in domains such a…
Offline Learning from Demonstrations and Unlabeled Experience
Konrad Zolna, Alexander Novikov, Ksenia Konyushkova +6
Behavior cloning (BC) is often practical for robot learning because it allows a policy to be trained offline without rewards, by supervised learning on expert demonstrations. Howev…
World Simulation with Video Foundation Models for Physical AI
NVIDIA, :, Arslan Ali +87
We introduce [Cosmos-Predict2.5], the latest generation of the Cosmos World Foundation Models for Physical AI. Built on a flow-based architecture, [Cosmos-Predict2.5] unifies Text2…
Going Deeper with Convolutions
Christian Szegedy, Wei Liu, Yangqing Jia +6
We propose a deep convolutional neural network architecture codenamed "Inception", which was responsible for setting the new state of the art for classification and detection in th…
SSD: Single Shot MultiBox Detector
Wei Liu, Dragomir Anguelov, Dumitru Erhan +4
We present a method for detecting objects in images using a single deep neural network. Our approach, named SSD, discretizes the output space of bounding boxes into a set of defaul…
Sim-and-Real Co-Training: A Simple Recipe for Vision-Based Robotic Manipulation
Abhiram Maddukuri, Zhenyu Jiang, Lawrence Yunliang Chen +12
Large real-world robot datasets hold great potential to train generalist robot models, but scaling real-world human data collection is time-consuming and resource-intensive. Simula…
DreamGen: Unlocking Generalization in Robot Learning through Video World Models
Joel Jang, Seonghyeon Ye, Zongyu Lin +25
We introduce DreamGen, a simple yet highly effective 4-stage pipeline for training robot policies that generalize across behaviors and environments through neural trajectories - sy…
A Generalist Agent
Scott Reed, Konrad Zolna, Emilio Parisotto +17
Inspired by progress in large-scale language modeling, we apply a similar approach towards building a single generalist agent beyond the realm of text outputs. The agent, which we…
Learning Deep Representations of Fine-grained Visual Descriptions
Scott Reed, Zeynep Akata, Bernt Schiele +1
State-of-the-art methods for zero-shot visual recognition formulate learning as a joint embedding problem of images and side information. In these formulations the current best com…
Parallel Multiscale Autoregressive Density Estimation
Scott Reed, Aäron van den Oord, Nal Kalchbrenner +4
PixelCNN achieves state-of-the-art results in density estimation for natural images. Although training is fast, inference is costly, requiring one network evaluation per pixel; O(N…
ScanComplete: Large-Scale Scene Completion and Semantic Segmentation for 3D Scans
Angela Dai, Daniel Ritchie, Martin Bokeloh +3
We introduce ScanComplete, a novel data-driven approach for taking an incomplete 3D scan of a scene as input and predicting a complete 3D model along with per-voxel semantic labels…
Biologically Inspired Nanomaterials: A Conference Report
Melik Demirel, Atul Parikh, Vincent Crespi +1
The understanding of the nanoscale physical properties of biomolecules and biomaterials will ultimately promote the research in the biological sciences. In this review, we focused…
Evaluation of Output Embeddings for Fine-Grained Image Classification
Zeynep Akata, Scott Reed, Daniel Walter +2
Image classification has advanced significantly in recent years with the availability of large-scale image sets. However, fine-grained classification remains a major challenge due…
QLIP: Text-Aligned Visual Tokenization Unifies Auto-Regressive Multimodal Understanding and Generation
Yue Zhao, Fuzhao Xue, Scott Reed +6
We introduce Quantized Language-Image Pretraining (QLIP), a visual tokenization method that combines state-of-the-art reconstruction quality with state-of-the-art zero-shot image u…
Robust Imitation of Diverse Behaviors
Ziyu Wang, Josh Merel, Scott Reed +3
Deep generative models have recently shown great promise in imitation learning for motor control. Given enough data, even supervised approaches can do one-shot imitation learning;…
Training Deep Neural Networks on Noisy Labels with Bootstrapping
Scott Reed, Honglak Lee, Dragomir Anguelov +3
Current state-of-the-art deep learning systems for visual object recognition and detection use purely supervised training with regularization such as dropout to avoid overfitting.…