171 citations · 937 across the 54 of their papers we have counts for
59 papers
Pretrained Embeddings as a Behavior Specification Mechanism
Parv Kapoor, Abigail Hammer, Ashish Kapoor +2
We propose an approach to formally specifying the behavioral properties of systems that rely on a perception model for interactions with the physical world. The key idea is to intr…
ChatGPT for Robotics: Design Principles and Model Abilities
Sai Vemprala, Rogerio Bonatti, Arthur Bucker +1
This paper presents an experimental study regarding the use of OpenAI's ChatGPT for robotics applications. We outline a strategy that combines design principles for prompt engineer…
Is Imitation All You Need? Generalized Decision-Making with Dual-Phase Training
Yao Wei, Yanchao Sun, Ruijie Zheng +7
We introduce DualMind, a generalist agent designed to tackle various decision-making tasks that addresses challenges posed by current methods, such as overfitting behaviors and dep…
ConBaT: Control Barrier Transformer for Safe Policy Learning
Yue Meng, Sai Vemprala, Rogerio Bonatti +2
Large-scale self-supervised models have recently revolutionized our ability to perform a variety of tasks within the vision and language domains. However, using such models for aut…
SMART: Self-supervised Multi-task pretrAining with contRol Transformers
Yanchao Sun, Shuang Ma, Ratnesh Madaan +3
Self-supervised pretraining has been extensively studied in language and vision domains, where a unified model can be easily adapted to various downstream tasks by pretraining repr…
ClimaX: A foundation model for weather and climate
Tung Nguyen, Johannes Brandstetter, Ashish Kapoor +2
Most state-of-the-art approaches for weather and climate modeling are based on physics-informed numerical models of the atmosphere. These approaches aim to model the non-linear dyn…
Masked Autoencoders for Egocentric Video Understanding @ Ego4D Challenge 2022
Jiachen Lei, Shuang Ma, Zhongjie Ba +3
In this report, we present our approach and empirical results of applying masked autoencoders in two egocentric video understanding tasks, namely, Object State Change Classificatio…
Learning Modular Simulations for Homogeneous Systems
Jayesh K. Gupta, Sai Vemprala, Ashish Kapoor
Complex systems are often decomposed into modular subsystems for engineering tractability. Although various equation based white-box modeling techniques make use of such structure,…
PACT: Perception-Action Causal Transformer for Autoregressive Robotics Pre-Training
Rogerio Bonatti, Sai Vemprala, Shuang Ma +3
Robotics has long been a field riddled with complex systems architectures whose modules and connections, whether traditional or learning-based, require significant human expertise…
Learning to Simulate Realistic LiDARs
Benoit Guillard, Sai Vemprala, Jayesh K. Gupta +4
Simulating realistic sensors is a challenging part in data generation for autonomous systems, often involving carefully handcrafted sensor design, scene properties, and physics mod…
LATTE: LAnguage Trajectory TransformEr
Arthur Bucker, Luis Figueredo, Sami Haddadin +4
Natural language is one of the most intuitive ways to express human intent. However, translating instructions and commands towards robotic motion generation and deployment in the r…
Sample-efficient Safe Learning for Online Nonlinear Control with Control Barrier Functions
Wenhao Luo, Wen Sun, Ashish Kapoor
Reinforcement Learning (RL) and continuous nonlinear control have been successfully deployed in multiple domains of complicated sequential decision-making tasks. However, given the…
Reshaping Robot Trajectories Using Natural Language Commands: A Study of Multi-Modal Data Alignment Using Transformers
Arthur Bucker, Luis Figueredo, Sami Haddadin +3
Natural language is the most intuitive medium for us to interact with other people when expressing commands and instructions. However, using language is seldom an easy task when hu…
COMPASS: Contrastive Multimodal Pretraining for Autonomous Systems
Shuang Ma, Sai Vemprala, Wenshan Wang +4
Learning representations that generalize across tasks and domains is challenging yet necessary for autonomous systems. Although task-driven approaches are appealing, designing mode…
Quantum Algorithms for Reinforcement Learning with a Generative Model
Daochen Wang, Aarthi Sundaram, Robin Kothari +2
Reinforcement learning studies how an agent should interact with an environment to maximize its cumulative reward. A standard way to study this question abstractly is to ask how ma…
CausalCity: Complex Simulations with Agency for Causal Discovery and Reasoning
Daniel McDuff, Yale Song, Jiyoung Lee +7
The ability to perform causal and counterfactual reasoning are central properties of human intelligence. Decision-making systems that can perform these types of reasoning have the…
3DB: A Framework for Debugging Computer Vision Models
Guillaume Leclerc, Hadi Salman, Andrew Ilyas +9
We introduce 3DB: an extendable, unified framework for testing and debugging vision models using photorealistic simulation. We demonstrate, through a wide range of use cases, that…
Representation Learning for Event-based Visuomotor Policies
Sai Vemprala, Sami Mian, Ashish Kapoor
Event-based cameras are dynamic vision sensors that provide asynchronous measurements of changes in per-pixel brightness at a microsecond level. This makes them significantly faste…
Unadversarial Examples: Designing Objects for Robust Vision
Hadi Salman, Andrew Ilyas, Logan Engstrom +3
We study a class of realistic computer vision settings wherein one can influence the design of the objects being recognized. We develop a framework that leverages this capability t…
Helping Reduce Environmental Impact of Aviation with Machine Learning
Ashish Kapoor
Commercial aviation is one of the biggest contributors towards climate change. We propose to reduce environmental impact of aviation by considering solutions that would reduce the…
Adversarial Attacks on Optimization based Planners
Sai Vemprala, Ashish Kapoor
Trajectory planning is a key piece in the algorithmic architecture of a robot. Trajectory planners typically use iterative optimization schemes for generating smooth trajectories t…
Do Adversarially Robust ImageNet Models Transfer Better?
Hadi Salman, Andrew Ilyas, Logan Engstrom +2
Transfer learning is a widely-used paradigm in deep learning, where models pre-trained on standard datasets can be efficiently adapted to downstream tasks. Typically, better pre-tr…
AirSim Drone Racing Lab
Ratnesh Madaan, Nicholas Gyde, Sai Vemprala +7
Autonomous drone racing is a challenging research problem at the intersection of computer vision, planning, state estimation, and control. We introduce AirSim Drone Racing Lab, a s…
Safety Considerations in Deep Control Policies with Safety Barrier Certificates Under Uncertainty
Tom Hirshberg, Sai Vemprala, Ashish Kapoor
Recent advances in Deep Machine Learning have shown promise in solving complex perception and control loops via methods such as reinforcement and imitation learning. However, guara…