5 papers
Language as a Sensor: Calibrated Spatial Belief Estimation in 3D Scenes from Natural Language
Aryan Naveen, Jason Xinyu Liu, Luca Carlone +1
Robots deployed in human-centric environments routinely receive natural-language descriptions of spatial information ("I left my backpack on the table") that reference parts of the…
LEGS-POMDP: Language and Gesture-Guided Object Search in Partially Observable Environments
Ivy Xiao He, Stefanie Tellex, Jason Xinyu Liu
To assist humans in open-world environments, robots must interpret ambiguous instructions to locate desired objects. Foundation model-based approaches excel at multimodal grounding…
Pragmatic Embodied Spoken Instruction Following in Human-Robot Collaboration with Theory of Mind
Lance Ying, Xinyi Li, Shivam Aarya +6
Spoken language instructions are ubiquitous in agent collaboration. However, in real-world human-robot collaboration, following human spoken instructions can be challenging due to…
λ: A Benchmark for Data-Efficiency in Long-Horizon Indoor Mobile Manipulation Robotics
Ahmed Jaafar, Shreyas Sundara Raman, Sudarshan Harithas +7
Learning to execute long-horizon mobile manipulation tasks is crucial for advancing robotics in household and workplace settings. However, current approaches are typically data-ine…
Learning Efficient and Robust Language-conditioned Manipulation using Textual-Visual Relevancy and Equivariant Language Mapping
Mingxi Jia, Haojie Huang, Zhewen Zhang +7
Controlling robots through natural language is pivotal for enhancing human-robot collaboration and synthesizing complex robot behaviors. Recent works that are trained on large robo…