103 citations · 192 across the 31 of their papers we have counts for
35 papers
Prompted Policy Search: Reinforcement Learning through Linguistic and Numerical Reasoning in LLMs
Yifan Zhou, Sachin Grover, Mohamed El Mistiri +7
Reinforcement Learning (RL) traditionally relies on scalar reward signals, limiting its ability to leverage the rich semantic knowledge often available in real-world tasks. In cont…
TwinTrack: Bridging Vision and Contact Physics for Real-Time Tracking of Unknown Objects in Contact-Rich Scenes
Wen Yang, Zhixian Xie, Yiting Wang +4
Real-time tracking of previously unseen, highly dynamic objects in contact-rich scenes, such as during dexterous in-hand manipulation, remains a major challenge. Pure vision-based…
SAS-Prompt: Large Language Models as Numerical Optimizers for Robot Self-Improvement
Heni Ben Amor, Laura Graesser, Atil Iscen +7
We demonstrate the ability of large language models (LLMs) to perform iterative self-improvement of robot policies. An important insight of this paper is that LLMs have a built-in…
RoboMD: Uncovering Robot Vulnerabilities through Semantic Potential Fields
Som Sagar, Jiafei Duan, Sreevishakh Vasudevan +4
Robot manipulation policies, while central to the promise of physical AI, are highly vulnerable in the presence of external variations in the real world. Diagnosing these vulnerabi…
Repairing Neural Networks for Safety in Robotic Systems using Predictive Models
Keyvan Majd, Geoffrey Clark, Georgios Fainekos +1
This paper introduces a new method for safety-aware robot learning, focusing on repairing policies using predictive models. Our method combines behavioral cloning with neural netwo…
A Comparison of Imitation Learning Algorithms for Bimanual Manipulation
Michael Drolet, Simon Stepputtis, Siva Kailas +4
Amidst the wide popularity of imitation learning algorithms in robotics, their properties regarding hyperparameter sensitivity, ease of training, data efficiency, and performance h…
Achieving Human Level Competitive Robot Table Tennis
David B. D'Ambrosio, Saminda Abeyruwan, Laura Graesser +24
Achieving human-level speed and performance on real world tasks is a north star for the robotics research community. This work takes a step towards that goal and presents the first…
Task Success is not Enough: Investigating the Use of Video-Language Models as Behavior Critics for Catching Undesirable Agent Behaviors
Lin Guan, Yifan Zhou, Denis Liu +3
Large-scale generative models are shown to be useful for sampling meaningful candidate solutions, yet they often overlook task constraints and user preferences. Their full power is…
Probabilistic Differentiable Filters Enable Ubiquitous Robot Control with Smartwatches
Fabian C Weigend, Xiao Liu, Heni Ben Amor
Ubiquitous robot control and human-robot collaboration using smart devices poses a challenging problem primarily due to strict accuracy requirements and sparse information. This pa…
Open X-Embodiment: Robotic Learning Datasets and RT-X Models
Embodiment Collaboration, Abby O'Neill, Abdul Rehman +291
Large, high-capacity models trained on diverse datasets have shown remarkable successes on efficiently tackling downstream applications. In domains from NLP to Computer Vision, thi…
Enhancing State Estimation in Robots: A Data-Driven Approach with Differentiable Ensemble Kalman Filters
Xiao Liu, Geoffrey Clark, Joseph Campbell +2
This paper introduces a novel state estimation framework for robots using differentiable ensemble Kalman filters (DEnKF). DEnKF is a reformulation of the traditional ensemble Kalma…
Learning Soft Robot Dynamics using Differentiable Kalman Filters and Spatio-Temporal Embeddings
Xiao Liu, Shuhei Ikemoto, Yuhei Yoshimitsu +1
This paper introduces a novel approach for modeling the dynamics of soft robots, utilizing a differentiable filter architecture. The proposed approach enables end-to-end training t…
Certifiably-correct Control Policies for Safe Learning and Adaptation in Assistive Robotics
Keyvan Majd, Geoffrey Clark, Tanmay Khandait +4
Guaranteeing safety in human-centric applications is critical in robot learning as the learned policies may demonstrate unsafe behaviors in formerly unseen scenarios. We present a…
Safe Robot Learning in Assistive Devices through Neural Network Repair
Keyvan Majd, Geoffrey Clark, Tanmay Khandait +4
Assistive robotic devices are a particularly promising field of application for neural networks (NN) due to the need for personalization and hard-to-model human-machine interaction…
Modularity through Attention: Efficient Training and Transfer of Language-Conditioned Policies for Robot Manipulation
Yifan Zhou, Shubham Sonawani, Mariano Phielipp +2
Language-conditioned policies allow robots to interpret and execute human instructions. Learning such policies requires a substantial investment with regards to time and compute re…
A System for Imitation Learning of Contact-Rich Bimanual Manipulation Policies
Simon Stepputtis, Maryam Bandari, Stefan Schaal +1
In this paper, we discuss a framework for teaching bimanual manipulation tasks by imitation. To this end, we present a system and algorithms for learning compliant and contact-rich…
Local Repair of Neural Networks Using Optimization
Keyvan Majd, Siyu Zhou, Heni Ben Amor +2
In this paper, we propose a framework to repair a pre-trained feed-forward neural network (NN) to satisfy a set of properties. We formulate the properties as a set of predicates th…
Multimodal Data Fusion for Power-On-and-Go Robotic Systems in Retail
Shubham Sonawani, Kailas Maneparambil, Heni Ben Amor
Robotic systems for retail have gained a lot of attention due to the labor-intensive nature of such business environments. Many tasks have the potential to be automated via intelli…
Learning Predictive Models for Ergonomic Control of Prosthetic Devices
Geoffrey Clark, Joseph Campbell, Heni Ben Amor
We present Model-Predictive Interaction Primitives -- a robot learning framework for assistive motion in human-machine collaboration tasks which explicitly accounts for biomechanic…
Language-Conditioned Imitation Learning for Robot Manipulation Tasks
Simon Stepputtis, Joseph Campbell, Mariano Phielipp +3
Imitation learning is a popular approach for teaching motor skills to robots. However, most approaches focus on extracting policy parameters from execution traces alone (i.e., moti…
DeepCrashTest: Turning Dashcam Videos into Virtual Crash Tests for Automated Driving Systems
Sai Krishna Bashetty, Heni Ben Amor, Georgios Fainekos
The goal of this paper is to generate simulations with real-world collision scenarios for training and testing autonomous vehicles. We use numerous dashcam crash videos uploaded on…
Assistive Relative Pose Estimation for On-orbit Assembly using Convolutional Neural Networks
Shubham Sonawani, Ryan Alimo, Renaud Detry +3
Accurate real-time pose estimation of spacecraft or object in space is a key capability necessary for on-orbit spacecraft servicing and assembly tasks. Pose estimation of objects i…
Foveated Haptic Gaze
Bijan Fakhri, Troy McDaniel, Heni Ben Amor +3
As digital worlds become ubiquitous via video games, simulations, virtual and augmented reality, people with disabilities who cannot access those worlds are becoming increasingly d…
Clone Swarms: Learning to Predict and Control Multi-Robot Systems by Imitation
Siyu Zhou, Mariano Phielipp, Jorge A. Sefair +2
In this paper, we propose SwarmNet -- a neural network architecture that can learn to predict and imitate the behavior of an observed swarm of agents in a centralized manner. Teste…