Publications (19)
Thinking While Moving: Deep Reinforcement Learning with Concurrent Control
Ted Xiao, Eric Jang, Dmitry Kalashnikov +4
We study reinforcement learning in settings where sampling an action from the policy must be done concurrently with the time evolution of the controlled system, such as when a robo…
End-to-End Learning of Semantic Grasping
Eric Jang, Sudheendra Vijayanarasimhan, Peter Pastor +2
We consider the task of semantic robotic grasping, in which a robot picks up an object of a user-specified class using only monocular images. Inspired by the two-stream hypothesis…
Time-Contrastive Networks: Self-Supervised Learning from Video
Pierre Sermanet, Corey Lynch, Yevgen Chebotar +4
We propose a self-supervised approach for learning representations and robotic behaviors entirely from unlabeled videos recorded from multiple viewpoints, and study how this repres…
RetinaGAN: An Object-aware Approach to Sim-to-Real Transfer
Daniel Ho, Kanishka Rao, Zhuo Xu +3
The success of deep reinforcement learning (RL) and imitation learning (IL) in vision-based robotic manipulation typically hinges on the expense of large scale data collection. Wit…
Watch, Try, Learn: Meta-Learning from Demonstrations and Reward
Allan Zhou, Eric Jang, Daniel Kappler +7
Imitation learning allows agents to learn complex behaviors from demonstrations. However, learning a complex vision-based task may require an impractical number of demonstrations.…
Categorical Reparameterization with Gumbel-Softmax
Eric Jang, Shixiang Gu, Ben Poole
Categorical variables are a natural choice for representing discrete structure in the world. However, stochastic neural networks rarely use categorical latent variables due to the…
Deep Reinforcement Learning for Vision-Based Robotic Grasping: A Simulated Comparative Evaluation of Off-Policy Methods
Deirdre Quillen, Eric Jang, Ofir Nachum +3
In this paper, we explore deep reinforcement learning algorithms for vision-based robotic grasping. Model-free deep reinforcement learning (RL) has been successfully applied to a r…
Practical Imitation Learning in the Real World via Task Consistency Loss
Mohi Khansari, Daniel Ho, Yuqing Du +6
Recent work in visual end-to-end learning for robotics has shown the promise of imitation learning across a variety of tasks. Such approaches are expensive both because they requir…
Do As I Can, Not As I Say: Grounding Language in Robotic Affordances
Michael Ahn, Anthony Brohan, Noah Brown +42
Large language models can encode a wealth of semantic knowledge about the world. Such knowledge could be extremely useful to robots aiming to act upon high-level, temporally extend…
QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation
Dmitry Kalashnikov, Alex Irpan, Peter Pastor +8
In this paper, we study the problem of learning vision-based dynamic manipulation skills using a scalable reinforcement learning approach. We study this problem in the context of g…
Sim2Real View Invariant Visual Servoing by Recurrent Control
Fereshteh Sadeghi, Alexander Toshev, Eric Jang +1
Humans are remarkably proficient at controlling their limbs and tools from a wide range of viewpoints and angles, even in the presence of optical distortions. In robotics, this abi…
BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learning
Eric Jang, Alex Irpan, Mohi Khansari +5
In this paper, we study the problem of enabling a vision-based robotic manipulation system to generalize to novel tasks, a long-standing challenge in robot learning. We approach th…
AW-Opt: Learning Robotic Skills with Imitation and Reinforcement at Scale
Yao Lu, Karol Hausman, Yevgen Chebotar +8
Robotic skills can be learned via imitation learning (IL) using user-provided demonstrations, or via reinforcement learning (RL) using large amountsof autonomously collected experi…
Grasp2Vec: Learning Object Representations from Self-Supervised Grasping
Eric Jang, Coline Devin, Vincent Vanhoucke +1
Well structured visual representations can make robot learning faster and can improve generalization. In this paper, we study how we can acquire effective object-centric representa…
Bayesian Imitation Learning for End-to-End Mobile Manipulation
Yuqing Du, Daniel Ho, Alexander A. Alemi +2
In this work we investigate and demonstrate benefits of a Bayesian approach to imitation learning from multiple sensor inputs, as applied to the task of opening office doors with a…
Scalable Multi-Task Imitation Learning with Autonomous Improvement
Avi Singh, Eric Jang, Alexander Irpan +5
While robot learning has demonstrated promising results for enabling robots to automatically acquire new skills, a critical challenge in deploying learning-based systems is scale:…
Meta-Learning Requires Meta-Augmentation
Janarthanan Rajendran, Alex Irpan, Eric Jang
Meta-learning algorithms aim to learn two components: a model that predicts targets for a task, and a base learner that quickly updates that model when given examples from a new ta…
WAIC, but Why? Generative Ensembles for Robust Anomaly Detection
Hyunsun Choi, Eric Jang, Alexander A. Alemi
Machine learning models encounter Out-of-Distribution (OoD) errors when the data seen at test time are generated from a different stochastic generator than the one used to generate…
Multi-Game Decision Transformers
Kuang-Huei Lee, Ofir Nachum, Mengjiao Yang +8
A longstanding goal of the field of AI is a method for learning a highly capable, generalist agent from diverse experience. In the subfields of vision and language, this was largel…