713 citations · 1.6k across the 19 of their papers we have counts for
20 papers
Locate 3D: Real-World Object Localization via Self-Supervised Learning in 3D
Sergio Arnaud, Paul McVay, Ada Martin +19
We present LOCATE 3D, a model for localizing objects in 3D scenes from referring expressions like "the small coffee table between the sofa and the lamp." LOCATE 3D sets a new state…
DexterityGen: Foundation Controller for Unprecedented Dexterity
Zhao-Heng Yin, Changhao Wang, Luis Pineda +11
Teaching robots dexterous manipulation skills, such as tool use, presents a significant challenge. Current approaches can be broadly categorized into two strategies: human teleoper…
Sparsh: Self-supervised touch representations for vision-based tactile sensing
Carolina Higuera, Akash Sharma, Chaithanya Krishna Bodduluri +8
In this work, we introduce general purpose touch representations for the increasingly accessible class of vision-based tactile sensors. Such sensors have led to many recent advance…
Neural feels with neural fields: Visuo-tactile perception for in-hand manipulation
Sudharshan Suresh, Haozhi Qi, Tingfan Wu +9
To achieve human-level dexterity, robots must infer spatial awareness from multimodal sensing to reason over contact interactions. During in-hand manipulation of novel objects, suc…
Habitat 3.0: A Co-Habitat for Humans, Avatars and Robots
Xavier Puig, Eric Undersander, Andrew Szot +20
We present Habitat 3.0: a simulation platform for studying collaborative human-robot tasks in home environments. Habitat 3.0 offers contributions across three dimensions: (1) Accur…
What do we learn from a large-scale study of pre-trained visual representations in sim and real environments?
Sneha Silwal, Karmesh Yadav, Tingfan Wu +10
We present a large empirical investigation on the use of pre-trained visual representations (PVRs) for training downstream policies that execute real-world tasks. Our study involve…
Deep RL at Scale: Sorting Waste in Office Buildings with a Fleet of Mobile Manipulators
Alexander Herzog, Kanishka Rao, Karol Hausman +37
We describe a system for deep reinforcement learning of robotic manipulation skills applied to a large-scale real-world task: sorting recyclables and trash in office buildings. Rea…
USA-Net: Unified Semantic and Affordance Representations for Robot Memory
Benjamin Bolte, Austin Wang, Jimmy Yang +3
In order for robots to follow open-ended instructions like "go open the brown cabinet over the sink", they require an understanding of both the scene geometry and the semantics of…
How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learned
Julian Ibarz, Jie Tan, Chelsea Finn +3
Deep reinforcement learning (RL) has emerged as a promising approach for autonomously acquiring complex behaviors from low level sensor observations. Although a large portion of de…
Action Image Representation: Learning Scalable Deep Grasping Policies with Zero Real World Data
Mohi Khansari, Daniel Kappler, Jianlan Luo +2
This paper introduces Action Image, a new grasp proposal representation that allows learning an end-to-end deep-grasping policy. Our model achieves grasp success on re…
Quantile QT-Opt for Risk-Aware Vision-Based Robotic Grasping
Cristian Bodnar, Adrian Li, Karol Hausman +2
The distributional perspective on reinforcement learning (RL) has given rise to a series of successful Q-learning algorithms, resulting in state-of-the-art performance in arcade ga…
Watch, Try, Learn: Meta-Learning from Demonstrations and Reward
Allan Zhou, Eric Jang, Daniel Kappler +7
Imitation learning allows agents to learn complex behaviors from demonstrations. However, learning a complex vision-based task may require an impractical number of demonstrations.…
Learning Probabilistic Multi-Modal Actor Models for Vision-Based Robotic Grasping
Mengyuan Yan, Adrian Li, Mrinal Kalakrishnan +1
Many previous works approach vision-based robotic grasping by training a value network that evaluates grasp proposals. These approaches require an optimization process at run-time…
Sim-to-Real via Sim-to-Sim: Data-efficient Robotic Grasping via Randomized-to-Canonical Adaptation Networks
Stephen James, Paul Wohlhart, Mrinal Kalakrishnan +6
Real world data, especially in the domain of robotics, is notoriously costly to collect. One way to circumvent this can be to leverage the power of simulation to produce large amou…
QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation
Dmitry Kalashnikov, Alex Irpan, Peter Pastor +8
In this paper, we study the problem of learning vision-based dynamic manipulation skills using a scalable reinforcement learning approach. We study this problem in the context of g…
Multi-Task Domain Adaptation for Deep Learning of Instance Grasping from Simulation
Kuan Fang, Yunfei Bai, Stefan Hinterstoisser +2
Learning-based approaches to robotic manipulation are limited by the scalability of data collection and accessibility of labels. In this paper, we present a multi-task domain adapt…
Using Simulation and Domain Adaptation to Improve Efficiency of Deep Robotic Grasping
Konstantinos Bousmalis, Alex Irpan, Paul Wohlhart +9
Instrumenting and collecting annotated visual grasping datasets to train modern machine learning algorithms can be extremely time-consuming and expensive. An appealing alternative…
Collective Robot Reinforcement Learning with Distributed Asynchronous Guided Policy Search
Ali Yahya, Adrian Li, Mrinal Kalakrishnan +2
In principle, reinforcement learning and policy search methods can enable robots to learn highly complex and general skills that may allow them to function amid the complexity and…
Path Integral Guided Policy Search
Yevgen Chebotar, Mrinal Kalakrishnan, Ali Yahya +3
We present a policy search method for learning complex feedback control policies that map from high-dimensional sensory inputs to motor torques, for manipulation tasks with discont…
Probabilistic Object Tracking using a Range Camera
Manuel Wüthrich, Peter Pastor, Mrinal Kalakrishnan +2
We address the problem of tracking the 6-DoF pose of an object while it is being manipulated by a human or a robot. We use a dynamic Bayesian network to perform inference and compu…