18 citations · 108 across the 24 of their papers we have counts for
25 papers
Gemini Robotics 1.5: Pushing the Frontier of Generalist Robots with Advanced Embodied Reasoning, Thinking, and Motion Transfer
Gemini Robotics Team, Abbas Abdolmaleki, Saminda Abeyruwan +169
General-purpose robots need a deep understanding of the physical world, advanced reasoning, and general and dexterous control. This report introduces the latest generation of the G…
Gemini Robotics: Bringing AI into the Physical World
Gemini Robotics Team, Saminda Abeyruwan, Joshua Ainslie +115
Recent advancements in large multimodal models have led to the emergence of remarkable generalist capabilities in digital domains, yet their translation to physical agents such as…
Learning the RoPEs: Better 2D and 3D Position Encodings with STRING
Connor Schenck, Isaac Reid, Mithun George Jacob +19
We introduce STRING: Separable Translationally Invariant Position Encodings. STRING extends Rotary Position Encodings, a recently proposed and widely used algorithm in large langua…
Linear Transformer Topological Masking with Graph Random Features
Isaac Reid, Kumar Avinava Dubey, Deepali Jain +12
When training transformers on graph-structured data, incorporating information about the underlying topology is crucial for good performance. Topological masking, a type of relativ…
Achieving Human Level Competitive Robot Table Tennis
David B. D'Ambrosio, Saminda Abeyruwan, Laura Graesser +24
Achieving human-level speed and performance on real world tasks is a north star for the robotics research community. This work takes a step towards that goal and presents the first…
Modeling the Real World with High-Density Visual Particle Dynamics
William F. Whitney, Jacob Varley, Deepali Jain +3
We present High-Density Visual Particle Dynamics (HD-VPD), a learned world model that can emulate the physical dynamics of real scenes by processing massive latent point clouds con…
Structured Unrestricted-Rank Matrices for Parameter Efficient Fine-tuning
Arijit Sehanobish, Avinava Dubey, Krzysztof Choromanski +4
Recent efforts to scale Transformer models have demonstrated rapid progress across a wide range of tasks (Wei et al., 2022). However, fine-tuning these models for downstream tasks…
Embodied AI with Two Arms: Zero-shot Learning, Safety and Modularity
Jake Varley, Sumeet Singh, Deepali Jain +5
We present an embodied AI system which receives open-ended natural language instructions from a human, and controls two arms to collaboratively accomplish potentially long-horizon…
SARA-RT: Scaling up Robotics Transformers with Self-Adaptive Robust Attention
Isabel Leal, Krzysztof Choromanski, Deepali Jain +11
We present Self-Adaptive Robust Attention for Robotics Transformers (SARA-RT): a new paradigm for addressing the emerging challenge of scaling up Robotics Transformers (RT) for on-…
Robotic Table Tennis: A Case Study into a High Speed Learning System
David B. D'Ambrosio, Jonathan Abelian, Saminda Abeyruwan +32
We present a deep-dive into a real-world robotic learning system that, in previous work, was shown to be capable of hundreds of table tennis rallies with a human and has the abilit…
Agile Catching with Whole-Body MPC and Blackbox Policy Learning
Saminda Abeyruwan, Alex Bewley, Nicholas M. Boffi +9
We address a benchmark task in agile robotics: catching objects thrown at high-speed. This is a challenging task that involves tracking, intercepting, and cradling a thrown object…
Barkour: Benchmarking Animal-level Agility with Quadruped Robots
Ken Caluwaerts, Atil Iscen, J. Chase Kew +41
Animals have evolved various agile locomotion strategies, such as sprinting, leaping, and jumping. There is a growing interest in developing legged robots that move like their biol…
Mnemosyne: Learning to Train Transformers with Transformers
Deepali Jain, Krzysztof Marcin Choromanski, Avinava Dubey +4
In this work, we propose a new class of learnable optimizers, called \textit{Mnemosyne}. It is based on the novel spatio-temporal low-rank implicit attention Transformers that can…
Efficient Graph Field Integrators Meet Point Clouds
Krzysztof Choromanski, Arijit Sehanobish, Han Lin +13
We present two new classes of algorithms for efficient field integration on graphs encoding point clouds. The first class, SeparatorFactorization(SF), leverages the bounded genus o…
Implicit Two-Tower Policies
Yunfan Zhao, Qingkai Pan, Krzysztof Choromanski +2
We present a new class of structured reinforcement learning policy-architectures, Implicit Two-Tower (ITT) policies, where the actions are chosen based on the attention scores of t…
i-Sim2Real: Reinforcement Learning of Robotic Policies in Tight Human-Robot Interaction Loops
Saminda Abeyruwan, Laura Graesser, David B. D'Ambrosio +6
Sim-to-real transfer is a powerful paradigm for robotic reinforcement learning. The ability to train policies in simulation enables safe exploration and large-scale data collection…
Hybrid Random Features
Krzysztof Choromanski, Haoxian Chen, Han Lin +10
We propose a new class of random feature methods for linearizing softmax and Gaussian kernels called hybrid random features (HRFs) that automatically adapt the quality of kernel es…
Unlocking Pixels for Reinforcement Learning via Implicit Attention
Krzysztof Marcin Choromanski, Deepali Jain, Wenhao Yu +9
There has recently been significant interest in training reinforcement learning (RL) agents in vision-based environments. This poses many challenges, such as high dimensionality an…
ES-ENAS: Efficient Evolutionary Optimization for Large Hybrid Search Spaces
Xingyou Song, Krzysztof Choromanski, Jack Parker-Holder +8
In this paper, we approach the problem of optimizing blackbox functions over large hybrid search spaces consisting of both combinatorial and continuous parameters. We demonstrate t…
Disentangled Planning and Control in Vision Based Robotics via Reward Machines
Alberto Camacho, Jacob Varley, Deepali Jain +2
In this work we augment a Deep Q-Learning agent with a Reward Machine (DQRM) to increase speed of learning vision-based policies for robot tasks, and overcome some of the limitatio…
From Pixels to Legs: Hierarchical Learning of Quadruped Locomotion
Deepali Jain, Atil Iscen, Ken Caluwaerts
Legged robots navigating crowded scenes and complex terrains in the real world are required to execute dynamic leg movements while processing visual input for obstacle avoidance an…
Learning Agile Locomotion Skills with a Mentor
Atil Iscen, George Yu, Alejandro Escontrela +3
Developing agile behaviors for legged robots remains a challenging problem. While deep reinforcement learning is a promising approach, learning truly agile behaviors typically requ…
Reinforcement Learning with Chromatic Networks for Compact Architecture Search
Xingyou Song, Krzysztof Choromanski, Jack Parker-Holder +6
We present a neural architecture search algorithm to construct compact reinforcement learning (RL) policies, by combining ENAS and ES in a highly scalable and intuitive way. By def…
Hierarchical Reinforcement Learning for Quadruped Locomotion
Deepali Jain, Atil Iscen, Ken Caluwaerts
Legged locomotion is a challenging task for learning algorithms, especially when the task requires a diverse set of primitive behaviors. To solve these problems, we introduce a hie…