24 citations · 29 across the 22 of their papers we have counts for
24 papers
Self-Supervised Keyframe Discovery for Horizon-Invariant Behavior Cloning
Prabin Kumar Rath, Omkar Patil, Nakul Gopalan
Behavior cloning (BC) in non-Markovian environments is a challenging problem because policies have to reason over contextual information over long horizons. Existing policy archite…
Monkey See, Can Monkey Do? A Benchmark for Evaluating Robot Skill Learning by Observation
Weiwei Gu, Anmol Gupta, Anant Sah +5
Learning from Observation (LfO) is a fundamental robotic capability that replicates how humans and animals socially learn from each other. Beyond its biological parallels, this mod…
Spec2Cov: An Agentic Framework for Code Coverage Closure of Digital Hardware Designs
Sean Lowe, Elias Hilaneh, Alma Babbit +3
Hardware verification is one of the most challenging stages of the hardware design process, requiring significant time and resources to ensure a design is fully validated and produ…
Meanings and Measurements: Multi-Agent Probabilistic Grounding for Vision-Language Navigation
Swagat Padhan, Lakshya Jain, Bhavya Minesh Shah +3
Robots collaborating with humans must convert natural language goals into actionable, physically grounded decisions. For example, executing a command such as "go two meters to the…
StageCraft: Execution Aware Mitigation of Distractor and Obstruction Failures in VLA Models
Kartikay Milind Pangaonkar, Prabin Rath, Omkar Patil +1
Large scale pre-training on text and image data along with diverse robot demonstrations has helped Vision Language Action models (VLAs) to generalize to novel tasks, objects and sc…
You've Got a Golden Ticket: Improving Generative Robot Policies With A Single Noise Vector
Omkar Patil, Ondrej Biza, Thomas Weng +9
Generative robot policies trained on demonstrations using behavior cloning often learn actions that are sub-optimal or misaligned with respect to the downstream task. Policy improv…
PokeNet: Learning Kinematic Models of Articulated Objects from Human Observations
Anmol Gupta, Weiwei Gu, Omkar Patil +2
Articulation modeling enables robots to learn joint parameters of articulated objects for effective manipulation which can then be used downstream for skill learning or planning. E…
Factorizing Diffusion Policies for Observation Modality Prioritization
Omkar Patil, Prabin Rath, Kartikay Pangaonkar +2
Diffusion models have been extensively leveraged for learning robot skills from demonstrations. These policies are conditioned on several observational modalities such as proprioce…
Learning Sequential Kinematic Models from Demonstrations for Multi-Jointed Articulated Objects
Anmol Gupta, Weiwei Gu, Omkar Patil +2
As robots become more generalized and deployed in diverse environments, they must interact with complex objects, many with multiple independent joints or degrees of freedom (DoF) r…
Compositional Instruction Following with Language Models and Reinforcement Learning
Vanya Cohen, Geraud Nangue Tasse, Nakul Gopalan +4
Combining reinforcement learning with language grounding is challenging as the agent needs to explore the environment while simultaneously learning multiple language-conditioned ta…
Composing Diffusion Policies for Few-shot Learning of Movement Trajectories
Omkar Patil, Anant Sah, Nakul Gopalan
Humans can perform various combinations of physical skills without having to relearn skills from scratch every single time. For example, we can swing a bat when walking without hav…
XMoP: Whole-Body Control Policy for Zero-shot Cross-Embodiment Neural Motion Planning
Prabin Kumar Rath, Nakul Gopalan
Classical manipulator motion planners work across different robot embodiments. However they plan on a pre-specified static environment representation, and are not scalable to unsee…
Continual Robot Skill and Task Learning via Dialogue
Weiwei Gu, Suresh Kondepudi, Anmol Gupta +2
Interactive robot learning is a challenging problem as the robot is present with human users who expect the robot to learn novel skills to solve novel tasks perpetually with sample…
Interactive Visual Task Learning for Robots
Weiwei Gu, Anant Sah, Nakul Gopalan
We present a framework for robots to learn novel visual concepts and tasks via in-situ linguistic interactions with human users. Previous approaches have either used large pre-trai…
Improved Inference of Human Intent by Combining Plan Recognition and Language Feedback
Ifrah Idrees, Tian Yun, Naveen Sharma +4
Conversational assistive robots can aid people, especially those with cognitive impairments, to accomplish various tasks such as cooking meals, performing exercises, or operating m…
Language-Conditioned Change-point Detection to Identify Sub-Tasks in Robotics Domains
Divyanshu Raj, Chitta Baral, Nakul Gopalan
In this work, we present an approach to identify sub-tasks within a demonstrated robot trajectory using language instructions. We identify these sub-tasks using language provided d…
Efficient Exploration via First-Person Behavior Cloning Assisted Rapidly-Exploring Random Trees
Max Zuo, Logan Schick, Matthew Gombolay +1
Modern day computer games have extremely large state and action spaces. To detect bugs in these games' models, human testers play the games repeatedly to explore the game and find…
Learning to Follow Language Instructions with Compositional Policies
Vanya Cohen, Geraud Nangue Tasse, Nakul Gopalan +3
We propose a framework that learns to execute natural language instructions in an environment consisting of goal-reaching tasks that share components of their task descriptions. Ou…
Natural Language Specification of Reinforcement Learning Policies through Differentiable Decision Trees
Pradyumna Tambwekar, Andrew Silva, Nakul Gopalan +1
Human-AI policy specification is a novel procedure we define in which humans can collaboratively warm-start a robot's reinforcement learning policy. This procedure is comprised of…
Robot Object Retrieval with Contextual Natural Language Queries
Thao Nguyen, Nakul Gopalan, Roma Patel +3
Natural language object retrieval is a highly useful yet challenging task for robots in human-centric environments. Previous work has primarily focused on commands specifying the d…
Grounding Language Attributes to Objects using Bayesian Eigenobjects
Vanya Cohen, Benjamin Burchfiel, Thao Nguyen +3
We develop a system to disambiguate object instances within the same class based on simple physical descriptions. The system takes as input a natural language phrase and a depth im…
Mitigating Planner Overfitting in Model-Based Reinforcement Learning
Dilip Arumugam, David Abel, Kavosh Asadi +5
An agent with an inaccurate model of its environment faces a difficult choice: it can ignore the errors in its model and act in the real world in whatever way it determines is opti…
A Tale of Two DRAGGNs: A Hybrid Approach for Interpreting Action-Oriented and Goal-Oriented Instructions
Siddharth Karamcheti, Edward C. Williams, Dilip Arumugam +4
Robots operating alongside humans in diverse, stochastic environments must be able to accurately interpret natural language commands. These instructions often fall into one of two…
Accurately and Efficiently Interpreting Human-Robot Instructions of Varying Granularities
Dilip Arumugam, Siddharth Karamcheti, Nakul Gopalan +2
Humans can ground natural language commands to tasks at both abstract and fine-grained levels of specificity. For instance, a human forklift operator can be instructed to perform a…