Publications (15)
Temporal Grounding Graphs for Language Understanding with Accrued Visual-Linguistic Context
Rohan Paul, Andrei Barbu, Sue Felshin +2
A robot's ability to understand or ground natural language instructions is fundamentally tied to its knowledge about the surrounding world. We present an approach to grounding natu…
Uncertainty-aware Active Learning of NeRF-based Object Models for Robot Manipulators using Visual and Re-orientation Actions
Saptarshi Dasgupta, Akshat Gupta, Shreshth Tuli +1
Manipulating unseen objects is challenging without a 3D representation, as objects generally have occluded surfaces. This requires physical interaction with objects to build their…
Sketch-Plan-Generalize: Learning and Planning with Neuro-Symbolic Programmatic Representations for Inductive Spatial Concepts
Namasivayam Kalithasan, Sachit Sachdeva, Himanshu Gaurav Singh +7
Effective human-robot collaboration requires the ability to learn personalized concepts from a limited number of demonstrations, while exhibiting inductive generalization, hierarch…
Learning to Recover from Plan Execution Errors during Robot Manipulation: A Neuro-symbolic Approach
Namasivayam Kalithasan, Arnav Tuli, Vishal Bindal +3
Automatically detecting and recovering from failures is an important but challenging problem for autonomous robots. Most of the recent work on learning to plan from demonstrations…
Learning Neuro-symbolic Programs for Language Guided Robot Manipulation
Namasivayam Kalithasan, Himanshu Singh, Vishal Bindal +5
Given a natural language instruction and an input scene, our goal is to train a model to output a manipulation program that can be executed by the robot. Prior approaches for this…
PhyPlan: Generalizable and Rapid Physical Task Planning with Physics Informed Skill Networks for Robot Manipulators
Mudit Chopra, Abhinav Barnawal, Harshil Vagadia +4
Given the task of positioning a ball-like object to a goal region beyond direct reach, humans can often throw, slide, or rebound objects against the wall to attain the goal. Howeve…
ToolNet: Using Commonsense Generalization for Predicting Tool Use for Robot Plan Synthesis
Rajas Bansal, Shreshth Tuli, Rohan Paul +1
A robot working in a physical environment (like home or factory) needs to learn to use various available tools for accomplishing different tasks, for instance, a mop for cleaning a…
PhyPlan: Compositional and Adaptive Physical Task Reasoning with Physics-Informed Skill Networks for Robot Manipulators
Harshil Vagadia, Mudit Chopra, Abhinav Barnawal +4
Given the task of positioning a ball-like object to a goal region beyond direct reach, humans can often throw, slide, or rebound objects against the wall to attain the goal. Howeve…
TANGO: Commonsense Generalization in Predicting Tool Interactions for Mobile Manipulators
Shreshth Tuli, Rajas Bansal, Rohan Paul +1
Robots assisting us in factories or homes must learn to make use of objects as tools to perform tasks, e.g., a tray for carrying objects. We consider the problem of learning common…
GoalNet: Inferring Conjunctive Goal Predicates from Human Plan Demonstrations for Robot Instruction Following
Shreya Sharma, Jigyasa Gupta, Shreshth Tuli +2
Our goal is to enable a robot to learn how to sequence its actions to perform tasks specified as natural language instructions, given successful demonstrations from a human partner…
GTR: Generalized Grounded Temporal Reasoning for Robot Instruction Following by Combining Large Pre-trained Models
Riya Arora, Niveditha Narendranath, Aman Tambi +3
Consider the scenario where a human cleans a table and a robot observing the scene is instructed with the task "Remove the cloth using which I wiped the table". Instruction followi…
ToolTango: Common sense Generalization in Predicting Sequential Tool Interactions for Robot Plan Synthesis
Shreshth Tuli, Rajas Bansal, Rohan Paul +1
Robots assisting us in environments such as factories or homes must learn to make use of objects as tools to perform tasks, for instance using a tray to carry objects. We consider…
C2F-Space: Coarse-to-Fine Space Grounding for Spatial Instructions using Vision-Language Models
Nayoung Oh, Dohyun Kim, Junhyeong Bang +2
Space grounding refers to localizing a set of spatial references described in natural language instructions. Traditional methods often fail to account for complex reasoning -- such…
Open source software for automatic subregional assessment of knee cartilage degradation using quantitative T2 relaxometry and deep learning
Kevin A. Thomas, Dominik KrzemiÅski, Åukasz KidziÅski +7
Objective: We evaluate a fully-automated femoral cartilage segmentation model for measuring T2 relaxation values and longitudinal changes using multi-echo spin echo (MESE) MRI. We…
Multi-facet Universal Schema
Rohan Paul, Haw-Shiuan Chang, Andrew McCallum
Universal schema (USchema) assumes that two sentence patterns that share the same entity pairs are similar to each other. This assumption is widely adopted for solving various type…