90 citations · 175 across the 26 of their papers we have counts for
26 papers
DreamControl-v2: Simpler and Scalable Autonomous Humanoid Skills via Trainable Guided Diffusion Priors
Sudarshan Harithas, Sangkyung Kwak, Pushkal Katara +6
Developing robust autonomous loco-manipulation skills for humanoids remains an open problem in robotics. While RL has been applied successfully to legged locomotion, applying it to…
DreamControl: Human-Inspired Whole-Body Humanoid Control for Scene Interaction via Guided Diffusion
Dvij Kalaria, Sudarshan S Harithas, Pushkal Katara +7
We introduce DreamControl, a novel methodology for learning autonomous whole-body humanoid skills. DreamControl leverages the strengths of diffusion models and Reinforcement Learni…
MatMamba: A Matryoshka State Space Model
Abhinav Shukla, Sai Vemprala, Aditya Kusupati +1
State Space Models (SSMs) like Mamba2 are a promising alternative to Transformers, with faster theoretical training and inference times -- especially for long context lengths. Rece…
Logically Constrained Robotics Transformers for Enhanced Perception-Action Planning
Parv Kapoor, Sai Vemprala, Ashish Kapoor
With the advent of large foundation model based planning, there is a dire need to ensure their output aligns with the stakeholder's intent. When these models are deployed in the re…
GRID: A Platform for General Robot Intelligence Development
Sai Vemprala, Shuhang Chen, Abhinav Shukla +2
Developing machine intelligence abilities in robots and autonomous systems is an expensive and time consuming process. Existing solutions are tailored to specific applications and…
EvDNeRF: Reconstructing Event Data with Dynamic Neural Radiance Fields
Anish Bhattacharya, Ratnesh Madaan, Fernando Cladera +7
We present EvDNeRF, a pipeline for generating event data and training an event-based dynamic NeRF, for the purpose of faithfully reconstructing eventstreams on scenes with rigid an…
ChatGPT for Robotics: Design Principles and Model Abilities
Sai Vemprala, Rogerio Bonatti, Arthur Bucker +1
This paper presents an experimental study regarding the use of OpenAI's ChatGPT for robotics applications. We outline a strategy that combines design principles for prompt engineer…
Is Imitation All You Need? Generalized Decision-Making with Dual-Phase Training
Yao Wei, Yanchao Sun, Ruijie Zheng +7
We introduce DualMind, a generalist agent designed to tackle various decision-making tasks that addresses challenges posed by current methods, such as overfitting behaviors and dep…
ConBaT: Control Barrier Transformer for Safe Policy Learning
Yue Meng, Sai Vemprala, Rogerio Bonatti +2
Large-scale self-supervised models have recently revolutionized our ability to perform a variety of tasks within the vision and language domains. However, using such models for aut…
Masked Autoencoders for Egocentric Video Understanding @ Ego4D Challenge 2022
Jiachen Lei, Shuang Ma, Zhongjie Ba +3
In this report, we present our approach and empirical results of applying masked autoencoders in two egocentric video understanding tasks, namely, Object State Change Classificatio…
Learning Modular Simulations for Homogeneous Systems
Jayesh K. Gupta, Sai Vemprala, Ashish Kapoor
Complex systems are often decomposed into modular subsystems for engineering tractability. Although various equation based white-box modeling techniques make use of such structure,…
PACT: Perception-Action Causal Transformer for Autoregressive Robotics Pre-Training
Rogerio Bonatti, Sai Vemprala, Shuang Ma +3
Robotics has long been a field riddled with complex systems architectures whose modules and connections, whether traditional or learning-based, require significant human expertise…
Learning to Simulate Realistic LiDARs
Benoit Guillard, Sai Vemprala, Jayesh K. Gupta +4
Simulating realistic sensors is a challenging part in data generation for autonomous systems, often involving carefully handcrafted sensor design, scene properties, and physics mod…
LATTE: LAnguage Trajectory TransformEr
Arthur Bucker, Luis Figueredo, Sami Haddadin +4
Natural language is one of the most intuitive ways to express human intent. However, translating instructions and commands towards robotic motion generation and deployment in the r…
Missingness Bias in Model Debugging
Saachi Jain, Hadi Salman, Eric Wong +4
Missingness, or the absence of features from an input, is a concept fundamental to many model debugging tools. However, in computer vision, pixels cannot simply be removed from an…
COMPASS: Contrastive Multimodal Pretraining for Autonomous Systems
Shuang Ma, Sai Vemprala, Wenshan Wang +4
Learning representations that generalize across tasks and domains is challenging yet necessary for autonomous systems. Although task-driven approaches are appealing, designing mode…
CausalCity: Complex Simulations with Agency for Causal Discovery and Reasoning
Daniel McDuff, Yale Song, Jiyoung Lee +7
The ability to perform causal and counterfactual reasoning are central properties of human intelligence. Decision-making systems that can perform these types of reasoning have the…
3DB: A Framework for Debugging Computer Vision Models
Guillaume Leclerc, Hadi Salman, Andrew Ilyas +9
We introduce 3DB: an extendable, unified framework for testing and debugging vision models using photorealistic simulation. We demonstrate, through a wide range of use cases, that…
Representation Learning for Event-based Visuomotor Policies
Sai Vemprala, Sami Mian, Ashish Kapoor
Event-based cameras are dynamic vision sensors that provide asynchronous measurements of changes in per-pixel brightness at a microsecond level. This makes them significantly faste…
Unadversarial Examples: Designing Objects for Robust Vision
Hadi Salman, Andrew Ilyas, Logan Engstrom +3
We study a class of realistic computer vision settings wherein one can influence the design of the objects being recognized. We develop a framework that leverages this capability t…
Adversarial Attacks on Optimization based Planners
Sai Vemprala, Ashish Kapoor
Trajectory planning is a key piece in the algorithmic architecture of a robot. Trajectory planners typically use iterative optimization schemes for generating smooth trajectories t…
AirSim Drone Racing Lab
Ratnesh Madaan, Nicholas Gyde, Sai Vemprala +7
Autonomous drone racing is a challenging research problem at the intersection of computer vision, planning, state estimation, and control. We introduce AirSim Drone Racing Lab, a s…
Safety Considerations in Deep Control Policies with Safety Barrier Certificates Under Uncertainty
Tom Hirshberg, Sai Vemprala, Ashish Kapoor
Recent advances in Deep Machine Learning have shown promise in solving complex perception and control loops via methods such as reinforcement and imitation learning. However, guara…
Collaborative Localization for Micro Aerial Vehicles
Sai Vemprala, Srikanth Saripalli
In this paper, we present a framework for performing collaborative localization for groups of micro aerial vehicles (MAV) that use vision based sensing. The vehicles are each assum…