93 citations · 706 across the 49 of their papers we have counts for
51 papers
RL-ViGen: A Reinforcement Learning Benchmark for Visual Generalization
Zhecheng Yuan, Sizhe Yang, Pu Hua +3
Visual Reinforcement Learning (Visual RL), coupled with high-dimensional observations, has consistently confronted the long-standing challenge of out-of-distribution generalization…
ContactArt: Learning 3D Interaction Priors for Category-level Articulated Object and Hand Poses Estimation
Zehao Zhu, Jiashun Wang, Yuzhe Qin +3
We propose a new dataset and a novel approach to learning hand-object interaction priors for hand and articulated object pose estimation. We first collect a dataset using visual te…
MoDem: Accelerating Visual Model-Based Reinforcement Learning with Demonstrations
Nicklas Hansen, Yixin Lin, Hao Su +3
Poor sample efficiency continues to be the primary challenge for deployment of deep Reinforcement Learning (RL) algorithms for real-world applications, and in particular for visuo-…
Policy Adaptation from Foundation Model Feedback
Yuying Ge, Annabella Macaluso, Li Erran Li +2
Recent progress on vision-language foundation models have brought significant advancement to building general-purpose robots. By using the pre-trained models to encode the scene an…
On Pre-Training for Visuo-Motor Control: Revisiting a Learning-from-Scratch Baseline
Nicklas Hansen, Zhecheng Yuan, Yanjie Ze +5
In this paper, we examine the effectiveness of pre-training for visuo-motor control tasks. We revisit a simple Learning-from-Scratch (LfS) baseline that incorporates data augmentat…
DexPoint: Generalizable Point Cloud Reinforcement Learning for Sim-to-Real Dexterous Manipulation
Yuzhe Qin, Binghao Huang, Zhao-Heng Yin +2
We propose a sim-to-real framework for dexterous manipulation which can generalize to new objects of the same category in the real world. The key of our framework is to train the m…
Visual Reinforcement Learning with Self-Supervised 3D Representations
Yanjie Ze, Nicklas Hansen, Yinbo Chen +2
A prominent approach to visual Reinforcement Learning (RL) is to learn an internal state representation using self-supervised methods, which has the potential benefit of improved s…
Graph Inverse Reinforcement Learning from Diverse Videos
Sateesh Kumar, Jonathan Zamora, Nicklas Hansen +2
Research on Inverse Reinforcement Learning (IRL) from third-person videos has shown encouraging results on removing the need for manual reward design for robotic tasks. However, mo…
From One Hand to Multiple Hands: Imitation Learning for Dexterous Manipulation from Single-Camera Teleoperation
Yuzhe Qin, Hao Su, Xiaolong Wang
We propose to perform imitation learning for dexterous manipulation with multi-finger robot hand from human demonstrations, and transfer the policy to the real robot hand. We intro…
Learning Generalizable Dexterous Manipulation from Human Grasp Affordance
Yueh-Hua Wu, Jiashun Wang, Xiaolong Wang
Dexterous manipulation with a multi-finger hand is one of the most challenging problems in robotics. While recent progress in imitation learning has largely improved the sample eff…
Temporal Difference Learning for Model Predictive Control
Nicklas Hansen, Xiaolong Wang, Hao Su
Data-driven model predictive control has two key advantages over model-free methods: a potential for improved sample efficiency through model learning, and better performance as co…
Look Closer: Bridging Egocentric and Third-Person Views with Transformers for Robotic Manipulation
Rishabh Jangir, Nicklas Hansen, Sambaran Ghosal +2
Learning to solve precision-based manipulation tasks from visual feedback using Reinforcement Learning (RL) could drastically reduce the engineering efforts required by traditional…
Learning Continuous Environment Fields via Implicit Functions
Xueting Li, Shalini De Mello, Xiaolong Wang +3
We propose a novel scene representation that encodes reaching distance -- the distance between any position in the scene to a goal along a feasible trajectory. We demonstrate that…
Multi-Person 3D Motion Prediction with Multi-Range Transformers
Jiashun Wang, Huazhe Xu, Medhini Narasimhan +1
We propose a novel framework for multi-person 3D motion trajectory prediction. Our key observation is that a human's action and behaviors may highly depend on the other persons aro…
Online Adaptation for Implicit Object Tracking and Shape Reconstruction in the Wild
Jianglong Ye, Yuntao Chen, Naiyan Wang +1
Tracking and reconstructing 3D objects from cluttered scenes are the key components for computer vision, robotics and autonomous driving systems. While recent progress in implicit…
Video Autoencoder: self-supervised disentanglement of static 3D structure and motion
Zihang Lai, Sifei Liu, Alexei A. Efros +1
A video autoencoder is proposed for learning disentan- gled representations of 3D structure and camera pose from videos in a self-supervised manner. Relying on temporal continuity…
Vision-Guided Quadrupedal Locomotion in the Wild with Multi-Modal Delay Randomization
Chieko Sarah Imai, Minghao Zhang, Yuchen Zhang +4
Developing robust vision-guided controllers for quadrupedal robots in complex environments, with various obstacles, dynamical surroundings and uneven terrains, is very challenging.…
DexMV: Imitation Learning for Dexterous Manipulation from Human Videos
Yuzhe Qin, Yueh-Hua Wu, Shaowei Liu +4
While significant progress has been made on understanding hand-object interactions in computer vision, it is still very challenging for robots to perform complex dexterous manipula…
Learning Vision-Guided Quadrupedal Locomotion End-to-End with Cross-Modal Transformers
Ruihan Yang, Minghao Zhang, Nicklas Hansen +2
We propose to address quadrupedal locomotion tasks using Reinforcement Learning (RL) with a Transformer-based model that learns to combine proprioceptive information and high-dimen…
Test-Time Personalization with a Transformer for Human Pose Estimation
Yizhuo Li, Miao Hao, Zonglin Di +2
We propose to personalize a human pose estimator given a set of test images of a person without using any manual annotations. While there is a significant advancement in human pose…
Stabilizing Deep Q-Learning with ConvNets and Vision Transformers under Data Augmentation
Nicklas Hansen, Hao Su, Xiaolong Wang
While agents trained by Reinforcement Learning (RL) can solve increasingly challenging tasks directly from visual observations, generalizing learned skills to novel environments re…
Semi-Supervised 3D Hand-Object Poses Estimation with Interactions in Time
Shaowei Liu, Hanwen Jiang, Jiarui Xu +2
Estimating 3D hand and object pose from a single image is an extremely challenging problem: hands and objects are often self-occluded during interactions, and the 3D annotations ar…
Single RGB-D Camera Teleoperation for General Robotic Manipulation
Quan Vuong, Yuzhe Qin, Runlin Guo +3
We propose a teleoperation system that uses a single RGB-D camera as the human motion capture device. Our system can perform general manipulation tasks such as cloth folding, hamme…
DAIR: Disentangled Attention Intrinsic Regularization for Safe and Efficient Bimanual Manipulation
Minghao Zhang, Pingcheng Jian, Yi Wu +2
We address the problem of safely solving complex bimanual robot manipulation tasks with sparse rewards. Such challenging tasks can be decomposed into sub-tasks that are accomplisha…