43 citations · 69 across the 18 of their papers we have counts for
20 papers
Update-Free On-Policy Steering via Verifiers
Maria Attarian, Ian Vyse, Jasper Gerigk +7
In recent years, Behavior Cloning (BC) has become one of the most prevalent methods for learning manipulation from human demonstrations. Despite their successes, BC policies are of…
Gemini Robotics 1.5: Pushing the Frontier of Generalist Robots with Advanced Embodied Reasoning, Thinking, and Motion Transfer
Gemini Robotics Team, Abbas Abdolmaleki, Saminda Abeyruwan +169
General-purpose robots need a deep understanding of the physical world, advanced reasoning, and general and dexterous control. This report introduces the latest generation of the G…
Gemini Robotics: Bringing AI into the Physical World
Gemini Robotics Team, Saminda Abeyruwan, Joshua Ainslie +115
Recent advancements in large multimodal models have led to the emergence of remarkable generalist capabilities in digital domains, yet their translation to physical agents such as…
Predictive Red Teaming: Breaking Policies Without Breaking Robots
Anirudha Majumdar, Mohit Sharma, Dmitry Kalashnikov +3
Visuomotor policies trained via imitation learning are capable of performing challenging manipulation tasks, but are often extremely brittle to lighting, visual distractors, and ob…
Learning the RoPEs: Better 2D and 3D Position Encodings with STRING
Connor Schenck, Isaac Reid, Mithun George Jacob +19
We introduce STRING: Separable Translationally Invariant Position Encodings. STRING extends Rotary Position Encodings, a recently proposed and widely used algorithm in large langua…
Mobility VLA: Multimodal Instruction Navigation with Long-Context VLMs and Topological Graphs
Hao-Tien Lewis Chiang, Zhuo Xu, Zipeng Fu +19
An elusive goal in navigation research is to build an intelligent agent that can understand multimodal instructions including natural language and image, and perform useful navigat…
Modeling the Real World with High-Density Visual Particle Dynamics
William F. Whitney, Jacob Varley, Deepali Jain +3
We present High-Density Visual Particle Dynamics (HD-VPD), a learned world model that can emulate the physical dynamics of real scenes by processing massive latent point clouds con…
Embodied AI with Two Arms: Zero-shot Learning, Safety and Modularity
Jake Varley, Sumeet Singh, Deepali Jain +5
We present an embodied AI system which receives open-ended natural language instructions from a human, and controls two arms to collaboratively accomplish potentially long-horizon…
Revisiting Energy Based Models as Policies: Ranking Noise Contrastive Estimation and Interpolating Energy Models
Sumeet Singh, Stephen Tu, Vikas Sindhwani
A crucial design decision for any robot learning pipeline is the choice of policy representation: what type of model should be used to generate the next set of robot actions? Owing…
Robots That Ask For Help: Uncertainty Alignment for Large Language Model Planners
Allen Z. Ren, Anushri Dixit, Alexandra Bodrova +11
Large language models (LLMs) exhibit a wide range of promising capabilities -- from step-by-step planning to commonsense reasoning -- that may provide utility for robots, but remai…
Agile Catching with Whole-Body MPC and Blackbox Policy Learning
Saminda Abeyruwan, Alex Bewley, Nicholas M. Boffi +9
We address a benchmark task in agile robotics: catching objects thrown at high-speed. This is a challenging task that involves tracking, intercepting, and cradling a thrown object…
RoboPianist: Dexterous Piano Playing with Deep Reinforcement Learning
Kevin Zakka, Philipp Wu, Laura Smith +8
Replicating human-like dexterity in robot hands represents one of the largest open problems in robotics. Reinforcement learning is a promising approach that has achieved impressive…
Mnemosyne: Learning to Train Transformers with Transformers
Deepali Jain, Krzysztof Marcin Choromanski, Avinava Dubey +4
In this work, we propose a new class of learnable optimizers, called \textit{Mnemosyne}. It is based on the novel spatio-temporal low-rank implicit attention Transformers that can…
Single-Level Differentiable Contact Simulation
Simon Le Cleac'h, Mac Schwager, Zachary Manchester +3
We present a differentiable formulation of rigid-body contact dynamics for objects and robots represented as compositions of convex primitives. Existing optimization-based approach…
Robotic Table Wiping via Reinforcement Learning and Whole-body Trajectory Optimization
Thomas Lew, Sumeet Singh, Mario Prats +11
We propose a framework to enable multipurpose assistive mobile robots to autonomously wipe tables to clean spills and crumbs. This problem is challenging, as it requires planning w…
Learning Model Predictive Controllers with Real-Time Attention for Real-World Navigation
Xuesu Xiao, Tingnan Zhang, Krzysztof Choromanski +14
Despite decades of research, existing navigation systems still face real-world challenges when deployed in the wild, e.g., in cluttered home environments or in human-occupied publi…
Multiscale Sensor Fusion and Continuous Control with Neural CDEs
Sumeet Singh, Francis McCann Ramirez, Jacob Varley +2
Though robot learning is often formulated in terms of discrete-time Markov decision processes (MDPs), physical robots require near-continuous multiscale feedback control. Machines…
Optimizing Trajectories with Closed-Loop Dynamic SQP
Sumeet Singh, Jean-Jacques Slotine, Vikas Sindhwani
Indirect trajectory optimization methods such as Differential Dynamic Programming (DDP) have found considerable success when only planning under dynamic feasibility constraints. Me…
Trajectory Optimization with Optimization-Based Dynamics
Taylor A. Howell, Simon Le Cleac'h, Sumeet Singh +3
We present a framework for bi-level trajectory optimization in which a system's dynamics are encoded as the solution to a constrained optimization problem and smooth gradients of t…
Risk-sensitive Inverse Reinforcement Learning via Semi- and Non-Parametric Methods
Sumeet Singh, Jonathan Lacotte, Anirudha Majumdar +1
The literature on Inverse Reinforcement Learning (IRL) typically assumes that humans take actions in order to minimize the expected value of a cost function, i.e., that humans are…