254 citations · 1.4k across the 100 of their papers we have counts for
103 papers
RePLan: Robotic Replanning with Perception and Language Models
Marta Skreta, Zihan Zhou, Jia Lin Yuan +3
Advancements in large language models (LLMs) have demonstrated their potential in facilitating high-level reasoning, logical reasoning and robotics planning. Recently, LLMs have al…
ORGANA: A Robotic Assistant for Automated Chemistry Experimentation and Characterization
Kourosh Darvish, Marta Skreta, Yuchi Zhao +9
Chemistry experiments can be resource- and labor-intensive, often requiring manual tasks like polishing electrodes in electrochemistry. Traditional lab automation infrastructure fa…
Fast-Grasp'D: Dexterous Multi-finger Grasp Generation Through Differentiable Simulation
Dylan Turpin, Tao Zhong, Shutong Zhang +8
Multi-finger grasping relies on high quality training data, which is hard to obtain: human data is hard to transfer and synthetic data relies on simplifying assumptions that reduce…
Self-Supervised Learning of Action Affordances as Interaction Modes
Liquan Wang, Nikita Dvornik, Rafael Dubeau +2
When humans perform a task with an articulated object, they interact with the object only in a handful of ways, while the space of all possible interactions is nearly endless. This…
SlotDiffusion: Object-Centric Generative Modeling with Diffusion Models
Ziyi Wu, Jingyu Hu, Wuyue Lu +2
Object-centric learning aims to represent visual data with a set of object entities (a.k.a. slots), providing structured representations that enable systematic generalization. Leve…
Learning Achievement Structure for Structured Exploration in Domains with Sparse Reward
Zihan Zhou, Animesh Garg
We propose Structured Exploration with Achievements (SEA), a multi-stage reinforcement learning algorithm designed for achievement-based environments, a particular type of environm…
StepFormer: Self-supervised Step Discovery and Localization in Instructional Videos
Nikita Dvornik, Isma Hadji, Ran Zhang +4
Instructional videos are an important resource to learn procedural tasks from human demonstrations. However, the instruction steps in such videos are typically short and sparse, wi…
Errors are Useful Prompts: Instruction Guided Task Programming with Verifier-Assisted Iterative Prompting
Marta Skreta, Naruki Yoshikawa, Sebastian Arellano-Rubach +6
Generating low-level robot task plans from high-level natural language instructions remains a challenging problem. Although large language models have shown promising results in ge…
-Patching: A Framework for Rapid Adaptation of Pre-trained Convolutional Networks without Base Performance Loss
Chaitanya Devaguptapu, Samarth Sinha, K J Joseph +2
Models pre-trained on large-scale datasets are often fine-tuned to support newer tasks and datasets that arrive over time. This process necessitates storing copies of the model ove…
MVTrans: Multi-View Perception of Transparent Objects
Yi Ru Wang, Yuchi Zhao, Haoping Xu +4
Transparent object perception is a crucial skill for applications such as robot manipulation in household and laboratory settings. Existing methods utilize RGB-D or stereo inputs t…
Orbit: A Unified Simulation Framework for Interactive Robot Learning Environments
Mayank Mittal, Calvin Yu, Qinxi Yu +12
We present Orbit, a unified and modular framework for robot learning powered by NVIDIA Isaac Sim. It offers a modular design to easily and efficiently create robotic environments w…
Offline Policy Optimization in RL with Variance Regularizaton
Riashat Islam, Samarth Sinha, Homanga Bharadhwaj +6
Learning policies from fixed offline datasets is a key challenge to scale up reinforcement learning (RL) algorithms towards practical applications. This is often because off-policy…
Chemistry Lab Automation via Constrained Task and Motion Planning
Naruki Yoshikawa, Andrew Zou Li, Kourosh Darvish +6
Chemists need to perform many laborious and time-consuming experiments in the lab to discover and understand the properties of new materials. To support and accelerate this process…
NeurIPS 2022 Competition: Driving SMARTS
Amir Rasouli, Randy Goebel, Matthew E. Taylor +15
Driving SMARTS is a regular competition designed to tackle problems caused by the distribution shift in dynamic interaction contexts that are prevalent in real-world autonomous dri…
nerf2nerf: Pairwise Registration of Neural Radiance Fields
Lily Goli, Daniel Rebain, Sara Sabour +2
We introduce a technique for pairwise registration of neural fields that extends classical optimization-based local registration (i.e. ICP) to operate on Neural Radiance Fields (Ne…
Breaking Bad: A Dataset for Geometric Fracture and Reassembly
Silvia Sellán, Yun-Chun Chen, Ziyi Wu +2
We introduce Breaking Bad, a large-scale dataset of fractured objects. Our dataset consists of over one million fractured objects simulated from ten thousand base models. The fract…
MoCoDA: Model-based Counterfactual Data Augmentation
Silviu Pitis, Elliot Creager, Ajay Mandlekar +1
The number of states in a dynamic process is exponential in the number of objects, making reinforcement learning (RL) difficult in complex, multi-object domains. For agents to scal…
See, Plan, Predict: Language-guided Cognitive Planning with Video Prediction
Maria Attarian, Advaya Gupta, Ziyi Zhou +3
Cognitive planning is the structural decomposition of complex tasks into a sequence of future behaviors. In the computational setting, performing cognitive planning entails groundi…
SlotFormer: Unsupervised Visual Dynamics Simulation with Object-Centric Models
Ziyi Wu, Nikita Dvornik, Klaus Greff +2
Understanding dynamics from visual observations is a challenging problem that requires disentangling individual objects from the scene and learning their interactions. While recent…
ProgPrompt: Generating Situated Robot Task Plans using Large Language Models
Ishika Singh, Valts Blukis, Arsalan Mousavian +6
Task planning can require defining myriad domain knowledge about the world in which a robot needs to act. To ameliorate that effort, large language models (LLMs) can be used to sco…
Grasp'D: Differentiable Contact-rich Grasp Synthesis for Multi-fingered Hands
Dylan Turpin, Liquan Wang, Eric Heiden +5
The study of hand-object interaction requires generating viable grasp poses for high-dimensional multi-finger models, often relying on analytic grasp synthesis which tends to produ…
Neural Motion Fields: Encoding Grasp Trajectories as Implicit Value Functions
Yun-Chun Chen, Adithyavairavan Murali, Balakumar Sundaralingam +3
The pipeline of current robotic pick-and-place methods typically consists of several stages: grasp pose detection, finding inverse kinematic solutions for the detected poses, plann…
SMPL: Simulated Industrial Manufacturing and Process Control Learning Environments
Mohan Zhang, Xiaozhou Wang, Benjamin Decardi-Nelson +9
Traditional biological and pharmaceutical manufacturing plants are controlled by human workers or pre-defined thresholds. Modernized factories have advanced process control algorit…
Neural Shape Mating: Self-Supervised Object Assembly with Adversarial Shape Priors
Yun-Chun Chen, Haoda Li, Dylan Turpin +2
Learning to autonomously assemble shapes is a crucial skill for many robotic applications. While the majority of existing part assembly methods focus on correctly posing semantic p…