16 citations · 37 across the 17 of their papers we have counts for
21 papers
AnyTask: an Automated Task and Data Generation Framework for Advancing Sim-to-Real Policy Learning
Ran Gong, Xiaohan Zhang, Jinghuan Shang +11
Generalist robot learning remains constrained by data: large-scale, diverse, and high-quality interaction data are expensive to collect in the real world. While simulation has beco…
Sceniris: A Fast Procedural Scene Generation Framework
Jinghuan Shang, Harsh Patel, Ran Gong +1
Synthetic 3D scenes are essential for developing Physical AI and generative models. Existing procedural generation methods often have low output throughput, creating a significant…
Data-Efficient Multitask DAgger
Haotian Fu, Ran Gong, Xiaohan Zhang +3
Generalist robot policies that can perform many tasks typically require extensive expert data or simulations for training. In this work, we propose a novel Data-Efficient multitask…
Accelerating Residual Reinforcement Learning with Uncertainty Estimation
Lakshita Dodeja, Karl Schmeckpeper, Shivam Vats +4
Residual Reinforcement Learning (RL) is a popular approach for adapting pretrained policies by learning a lightweight residual policy that provides corrective actions. While Residu…
Real-is-Sim: Bridging the Sim-to-Real Gap with a Dynamic Digital Twin
Jad Abou-Chakra, Lingfeng Sun, Krishan Rana +5
We introduce real-is-sim, a new approach to integrating simulation into behavior cloning pipelines. In contrast to real-only methods, which lack the ability to safely test policies…
On-Robot Reinforcement Learning with Goal-Contrastive Rewards
Ondrej Biza, Thomas Weng, Lingfeng Sun +6
Reinforcement Learning (RL) has the potential to enable robots to learn from their own actions in the real world. Unfortunately, RL can be prohibitively expensive, in terms of on-r…
Theia: Distilling Diverse Vision Foundation Models for Robot Learning
Jinghuan Shang, Karl Schmeckpeper, Brandon B. May +4
Vision-based robot policy learning, which maps visual inputs to actions, necessitates a holistic understanding of diverse visual tasks beyond single-task needs like classification…
Imagination Policy: Using Generative Point Cloud Models for Learning Manipulation Policies
Haojie Huang, Karl Schmeckpeper, Dian Wang +6
Humans can imagine goal states during planning and perform actions to match those goals. In this work, we propose Imagination Policy, a novel multi-task key-frame policy network fo…
EFEM: Equivariant Neural Field Expectation Maximization for 3D Object Segmentation Without Scene Supervision
Jiahui Lei, Congyue Deng, Karl Schmeckpeper +2
We introduce Equivariant Neural Field Expectation Maximization (EFEM), a simple, effective, and robust geometric algorithm that can segment objects in 3D scenes without annotations…
Semantic keypoint-based pose estimation from single RGB frames
Karl Schmeckpeper, Philip R. Osteen, Yufu Wang +6
This paper presents an approach to estimating the continuous 6-DoF pose of an object from a single RGB image. The approach combines semantic keypoints predicted by a convolutional…
Cross-modal Map Learning for Vision and Language Navigation
Georgios Georgakis, Karl Schmeckpeper, Karan Wanchoo +4
We consider the problem of Vision-and-Language Navigation (VLN). The majority of current methods for VLN are trained end-to-end using either unstructured memory such as LSTM, or us…
Uncertainty-driven Planner for Exploration and Navigation
Georgios Georgakis, Bernadette Bucher, Anton Arapin +3
We consider the problems of exploration and point-goal navigation in previously unseen environments, where the spatial complexity of indoor scenes and partial observability constit…
Bridge Data: Boosting Generalization of Robotic Skills with Cross-Domain Datasets
Frederik Ebert, Yanlai Yang, Karl Schmeckpeper +5
Robot learning holds the promise of learning policies that generalize broadly. However, such generalization requires sufficiently diverse datasets of the task of interest, which ca…
Learning to Map for Active Semantic Goal Navigation
Georgios Georgakis, Bernadette Bucher, Karl Schmeckpeper +2
We consider the problem of object goal navigation in unseen environments. Solving this problem requires learning of contextual semantic priors, a challenging endeavour given the sp…
Object-centric Video Prediction without Annotation
Karl Schmeckpeper, Georgios Georgakis, Kostas Daniilidis
In order to interact with the world, agents must be able to predict the results of the world's dynamics. A natural approach to learn about these dynamics is through video predictio…
Deformable Linear Object Prediction Using Locally Linear Latent Dynamics
Wenbo Zhang, Karl Schmeckpeper, Pratik Chaudhari +1
We propose a framework for deformable linear object prediction. Prediction of deformable objects (e.g., rope) is challenging due to their non-linear dynamics and infinite-dimension…
Reinforcement Learning with Videos: Combining Offline Observations with Interaction
Karl Schmeckpeper, Oleh Rybkin, Kostas Daniilidis +2
Reinforcement learning is a powerful framework for robots to acquire skills from experience, but often requires a substantial amount of online data collection. As a result, it is d…
An Adversarial Objective for Scalable Exploration
Bernadette Bucher, Karl Schmeckpeper, Nikolai Matni +1
Model-based curiosity combines active learning approaches to optimal sampling with the information gain based incentives for exploration presented in the curiosity literature. Exis…
Reactive Navigation in Partially Familiar Planar Environments Using Semantic Perceptual Feedback
Vasileios Vasilopoulos, Georgios Pavlakos, Karl Schmeckpeper +2
This paper solves the planar navigation problem by recourse to an online reactive scheme that exploits recent advances in SLAM and visual object recognition to recast prior geometr…
Learning Predictive Models From Observation and Interaction
Karl Schmeckpeper, Annie Xie, Oleh Rybkin +4
Learning predictive models from interaction with the world allows an agent, such as a robot, to learn about how the world works, and then use this learned model to plan coordinated…
RoboNet: Large-Scale Multi-Robot Learning
Sudeep Dasari, Frederik Ebert, Stephen Tian +6
Robot learning has emerged as a promising tool for taming the complexity and diversity of the real world. Methods based on high-capacity models, such as deep networks, hold the pro…