18 papers
Sensing Which Modality Matters: Evidence-Gated Regularization for Robust VLA Policies
Yue Yang, Diego Romeres, Chiori Hori +3
Vision-Language-Action (VLA) policies fuse multimodal sensory inputs, but training on limited and homogeneous robot demonstrations encourages spurious inter-sensor correlations rat…
SplatCtrl: Perception-Action Coupling via Gaussian Scene Representations and Reactive Robot Control
Siddarth Jain, Ho Jin Choi
Robotic manipulators excel in structured environments but face substantial challenges in unstructured and dynamic settings. This paper presents SplatCtrl, a unified framework for r…
FurnitureVLA: Learning Long-Horizon Bimanual Furniture Assembly with Vision-Language-Action Model
Chenyang Ma, Yue Yang, Radu Corcodel +4
Current work on robot furniture assembly mostly focuses on toy-scale settings or single-arm manipulation. We introduce FurnitureVLA, the first systematic study of real-scale bimanu…
VOLT: Vision and Language Trajectory Segmentation for Faster-than-Demonstration Policies
Robert Ramirez Sanchez, Daniel J. Evans, Dylan P. Losey +1
Humans often take longer to demonstrate a task than a robot would need to execute it. Rather than learning to replicate the demonstration at the same pace, many industrial and prac…
Robot Confirmation Generation and Action Planning Using Long-context Q-Former Integrated with Multimodal LLM
Chiori Hori, Yoshiki Masuyama, Siddarth Jain +4
Human-robot collaboration towards a shared goal requires robots to understand human action and interaction with the surrounding environment. This paper focuses on human-robot inter…
LLMPhy: Parameter-Identifiable Physical Reasoning Combining Large Language Models and Physics Engines
Anoop Cherian, Radu Corcodel, Siddarth Jain +1
Most learning-based approaches to complex physical reasoning sidestep the crucial problem of parameter identification (e.g., mass, friction) that governs scene dynamics, despite it…
Open Human-Robot Collaboration using Decentralized Inverse Reinforcement Learning
Prasanth Sengadu Suresh, Siddarth Jain, Prashant Doshi +1
The growing interest in human-robot collaboration (HRC), where humans and robots cooperate towards shared goals, has seen significant advancements over the past decade. While previ…
DECAF: a Discrete-Event based Collaborative Human-Robot Framework for Furniture Assembly
Giulio Giacomuzzo, Matteo Terreran, Siddarth Jain +1
This paper proposes a task planning framework for collaborative Human-Robot scenarios, specifically focused on assembling complex systems such as furniture. The human is characteri…
Autonomous Robotic Assembly: From Part Singulation to Precise Assembly
Kei Ota, Devesh K. Jha, Siddarth Jain +5
Imagine a robot that can assemble a functional product from the individual parts presented in any configuration to the robot. Designing such a robotic system is a complex problem w…
Interactive Planning Using Large Language Models for Partially Observable Robotics Tasks
Lingfeng Sun, Devesh K. Jha, Chiori Hori +5
Designing robotic agents to perform open vocabulary tasks has been the long-standing goal in robotics and AI. Recently, Large Language Models (LLMs) have achieved impressive result…
EARL: Eye-on-Hand Reinforcement Learner for Dynamic Grasping with Active Pose Estimation
Baichuan Huang, Jingjin Yu, Siddarth Jain
In this paper, we explore the dynamic grasping of moving objects through active pose tracking and reinforcement learning for hand-eye coordination systems. Most existing vision-bas…
Style-transfer based Speech and Audio-visual Scene Understanding for Robot Action Sequence Acquisition from Videos
Chiori Hori, Puyuan Peng, David Harwath +7
To realize human-robot collaboration, robots need to execute actions for new tasks according to human instructions given finite prior knowledge. Human experts can share their knowl…
Learning Generalizable Pivoting Skills
Xiang Zhang, Siddarth Jain, Baichuan Huang +2
The skill of pivoting an object with a robotic system is challenging for the external forces that act on the system, mainly given by contact interaction. The complexity increases w…
Task-Directed Exploration in Continuous POMDPs for Robotic Manipulation of Articulated Objects
Aidan Curtis, Leslie Kaelbling, Siddarth Jain
Representing and reasoning about uncertainty is crucial for autonomous agents acting in partially observable environments with noisy sensors. Partially observable Markov decision p…
Generalizable Human-Robot Collaborative Assembly Using Imitation Learning and Force Control
Devesh K. Jha, Siddarth Jain, Diego Romeres +2
Robots have been steadily increasing their presence in our daily lives, where they can work along with humans to provide assistance in various tasks on industry floors, in offices,…
Design of Adaptive Compliance Controllers for Safe Robotic Assembly
Devesh K. Jha, Diego Romeres, Siddarth Jain +2
Insertion operations are a critical element of most robotic assembly operation, and peg-in-hole (PiH) insertion is one of the most widely studied tasks in the industrial and academ…
Learning to Synthesize Volumetric Meshes from Vision-based Tactile Imprints
Xinghao Zhu, Siddarth Jain, Masayoshi Tomizuka +1
Vision-based tactile sensors typically utilize a deformable elastomer and a camera mounted above to provide high-resolution image observations of contacts. Obtaining accurate volum…
InSeGAN: A Generative Approach to Segmenting Identical Instances in Depth Images
Anoop Cherian, Goncalo Dias Pais, Siddarth Jain +2
In this paper, we present InSeGAN, an unsupervised 3D generative adversarial network (GAN) for segmenting (nearly) identical instances of rigid objects in depth images. Using an an…