2.4k citations · 8.4k across the 129 of their papers we have counts for
30 papers · 1 filter
Learning to Navigate Using Mid-Level Visual Priors
Alexander Sax, Jeffrey O. Zhang, Bradley Emi +4
How much does having visual priors about the world (e.g. the fact that the world is 3D) assist in learning to perform downstream motor tasks (e.g. navigating a complex environment)…
Leveraging Pretrained Image Classifiers for Language-Based Segmentation
David Golub, Ahmed El-Kishky, Roberto Martín-Martín
Current semantic segmentation models cannot easily generalize to new object classes unseen during train time: they require additional annotated images and retraining. We propose a…
KETO: Learning Keypoint Representations for Tool Manipulation
Zengyi Qin, Kuan Fang, Yuke Zhu +2
We aim to develop an algorithm for robots to manipulate novel objects as tools for completing different task goals. An efficient and informative representation would facilitate the…
HRL4IN: Hierarchical Reinforcement Learning for Interactive Navigation with Mobile Manipulators
Chengshu Li, Fei Xia, Roberto Martin-Martin +1
Most common navigation tasks in human environments require auxiliary arm interactions, e.g. opening doors, pressing buttons and pushing obstacles away. This type of navigation task…
6-PACK: Category-level 6D Pose Tracker with Anchor-Based Keypoints
Chen Wang, Roberto Martín-Martín, Danfei Xu +5
We present 6-PACK, a deep learning approach to category-level 6D object pose tracking on RGB-D data. Our method tracks in real-time novel object instances of known object categorie…
SURREAL-System: Fully-Integrated Stack for Distributed Deep Reinforcement Learning
Linxi Fan, Yuke Zhu, Jiren Zhu +6
We present an overview of SURREAL-System, a reproducible, flexible, and scalable framework for distributed reinforcement learning (RL). The framework consists of a stack of four la…
3D Scene Graph: A Structure for Unified Semantics, 3D Space, and Camera
Iro Armeni, Zhi-Yang He, JunYoung Gwak +4
A comprehensive semantic understanding of a scene is important for many applications - but in what space should diverse semantic information (e.g., objects, scene categories, mater…
Causal Induction from Visual Observations for Goal Directed Tasks
Suraj Nair, Yuke Zhu, Silvio Savarese +1
Causal reasoning has been an indispensable capability for humans and other intelligent animals to interact with the physical world. In this work, we propose to endow an artificial…
Interactive Gibson Benchmark (iGibson 0.5): A Benchmark for Interactive Navigation in Cluttered Environments
Fei Xia, William B. Shen, Chengshu Li +6
We present Interactive Gibson Benchmark, the first comprehensive benchmark for training and evaluating Interactive Navigation: robot navigation strategies where physical interactio…
Dynamics Learning with Cascaded Variational Inference for Multi-Step Manipulation
Kuan Fang, Yuke Zhu, Animesh Garg +2
The fundamental challenge of planning for multi-step manipulation is to find effective and plausible action sequences that lead to the task goal. We present Cascaded Variational In…
JRDB: A Dataset and Benchmark of Egocentric Robot Visual Perception of Humans in Built Environments
Roberto Martín-Martín, Mihir Patel, Hamid Rezatofighi +5
We present JRDB, a novel egocentric dataset collected from our social mobile manipulator JackRabbot. The dataset includes 64 minutes of annotated multimodal sensor data including s…
Regression Planning Networks
Danfei Xu, Roberto Martín-Martín, De-An Huang +3
Recent learning-to-plan methods have shown promising results on planning directly from observation space. Yet, their ability to plan for long-horizon tasks is limited by the accura…
AC-Teach: A Bayesian Actor-Critic Method for Policy Learning with an Ensemble of Suboptimal Teachers
Andrey Kurenkov, Ajay Mandlekar, Roberto Martin-Martin +2
The exploration mechanism used by a Deep Reinforcement Learning (RL) agent plays a key role in determining its sample efficiency. Thus, improving over random exploration is crucial…
Variable Impedance Control in End-Effector Space: An Action Space for Reinforcement Learning in Contact-Rich Tasks
Roberto Martín-Martín, Michelle A. Lee, Rachel Gardner +3
Reinforcement Learning (RL) of contact-rich manipulation tasks has yielded impressive results in recent years. While many studies in RL focus on varying the observation space or re…
Situational Fusion of Visual Representation for Visual Navigation
Bokui Shen, Danfei Xu, Yuke Zhu +3
A complex visual navigation task puts an agent in different situations which call for a diverse range of visual perception abilities. For example, to "go to the nearest chair", the…
Continuous Relaxation of Symbolic Planner for One-Shot Imitation Learning
De-An Huang, Danfei Xu, Yuke Zhu +4
We address one-shot imitation learning, where the goal is to execute a previously unseen task based on a single demonstration. While there has been exciting progress in this direct…
Making Sense of Vision and Touch: Learning Multimodal Representations for Contact-Rich Tasks
Michelle A. Lee, Yuke Zhu, Peter Zachares +6
Contact-rich manipulation tasks in unstructured environments often require both haptic and visual feedback. It is non-trivial to manually design a robot controller that combines th…
Social-BiGAT: Multimodal Trajectory Forecasting using Bicycle-GAN and Graph Attention Networks
Vineet Kosaraju, Amir Sadeghian, Roberto Martín-Martín +3
Predicting the future trajectories of multiple interacting agents in a scene has become an increasingly important problem for many different applications ranging from control of au…
Improving Social Awareness Through DANTE: A Deep Affinity Network for Clustering Conversational Interactants
Mason Swofford, John Charles Peruzzi, Nathan Tsoi +4
We propose a data-driven approach to detect conversational groups by identifying spatial arrangements typical of these focused social encounters. Our approach uses a novel Deep Aff…
Time-Varying Interaction Estimation Using Ensemble Methods
Brandon Oselio, Amir Sadeghian, Silvio Savarese +1
Directed information (DI) is a useful tool to explore time-directed interactions in multivariate data. However, as originally formulated DI is not well suited to interactions that…
Deep Local Trajectory Replanning and Control for Robot Navigation
Ashwini Pokle, Roberto Martín-Martín, Patrick Goebel +8
We present a navigation system that combines ideas from hierarchical planning and machine learning. The system uses a traditional global planner to compute optimal paths towards a…
Which Tasks Should Be Learned Together in Multi-task Learning?
Trevor Standley, Amir R. Zamir, Dawn Chen +3
Many computer vision applications require solving multiple tasks in real-time. A neural network can be trained to solve multiple tasks simultaneously using multi-task learning. Thi…
4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks
Christopher Choy, JunYoung Gwak, Silvio Savarese
In many robotics and VR/AR applications, 3D-videos are readily-available sources of input (a continuous sequence of depth images, or LIDAR scans). However, those 3D-videos are proc…
Machine Vision for Natural Gas Methane Emissions Detection Using an Infrared Camera
Jingfan Wang, Lyne P. Tchapmi, Arvind P. Ravikumara +5
It is crucial to reduce natural gas methane emissions, which can potentially offset the climate benefits of replacing coal with gas. Optical gas imaging (OGI) is a widely-used meth…