166 citations · 714 across the 35 of their papers we have counts for
35 papers
Gemini Robotics 1.5: Pushing the Frontier of Generalist Robots with Advanced Embodied Reasoning, Thinking, and Motion Transfer
Gemini Robotics Team, Abbas Abdolmaleki, Saminda Abeyruwan +169
General-purpose robots need a deep understanding of the physical world, advanced reasoning, and general and dexterous control. This report introduces the latest generation of the G…
Splatting Physical Scenes: End-to-End Real-to-Sim from Imperfect Robot Data
Ben Moran, Mauro Comi, Arunkumar Byravan +4
Creating accurate, physical simulations directly from real-world robot motion holds great value for safe, scalable, and affordable robot learning, yet remains exceptionally challen…
Motion Tracking with Muscles: Predictive Control of a Parametric Musculoskeletal Canine Model
Vittorio La Barbera, Steven Bohez, Leonard Hasenclever +2
We introduce a novel musculoskeletal model of a dog, procedurally generated from accurate 3D muscle meshes. Accompanying this model is a motion capture-based locomotion task compat…
Gemini Robotics: Bringing AI into the Physical World
Gemini Robotics Team, Saminda Abeyruwan, Joshua Ainslie +115
Recent advancements in large multimodal models have led to the emergence of remarkable generalist capabilities in digital domains, yet their translation to physical agents such as…
Proc4Gem: Foundation models for physical agency through procedural generation
Yixin Lin, Jan Humplik, Sandy H. Huang +18
In robot learning, it is common to either ignore the environment semantics, focusing on tasks like whole-body control which only require reasoning about robot-environment contacts,…
Diffusion Augmented Agents: A Framework for Efficient Exploration and Transfer Learning
Norman Di Palo, Leonard Hasenclever, Jan Humplik +1
We introduce Diffusion Augmented Agents (DAAG), a novel framework that leverages large language models, vision language models, and diffusion models to improve sample efficiency an…
Learning Robot Soccer from Egocentric Vision with Deep Reinforcement Learning
Dhruva Tirumala, Markus Wulfmeier, Ben Moran +13
We apply multi-agent deep reinforcement learning (RL) to train end-to-end robot soccer policies with fully onboard computation and sensing via egocentric RGB vision. This setting r…
Learning to Learn Faster from Human Feedback with Language Model Predictive Control
Jacky Liang, Fei Xia, Wenhao Yu +47
Large language models (LLMs) have been shown to exhibit a wide range of capabilities, such as writing robot code from language commands -- enabling non-experts to direct robot beha…
Replay across Experiments: A Natural Extension of Off-Policy RL
Dhruva Tirumala, Thomas Lampe, Jose Enrique Chen +9
Replaying data is a principal mechanism underlying the stability and data efficiency of off-policy reinforcement learning (RL). We present an effective yet simple framework to exte…
Towards A Unified Agent with Foundation Models
Norman Di Palo, Arunkumar Byravan, Leonard Hasenclever +3
Language Models and Vision Language Models have recently demonstrated unprecedented capabilities in terms of understanding human intentions, reasoning, scene understanding, and pla…
Language to Rewards for Robotic Skill Synthesis
Wenhao Yu, Nimrod Gileadi, Chuyuan Fu +17
Large language models (LLMs) have demonstrated exciting progress in acquiring diverse new capabilities through in-context learning, ranging from logical reasoning to code-writing.…
A Generalist Dynamics Model for Control
Ingmar Schubert, Jingwei Zhang, Jake Bruce +7
We investigate the use of transformer sequence models as dynamics models (TDMs) for control. We find that TDMs exhibit strong generalization capabilities to unseen environments, bo…
Learning Agile Soccer Skills for a Bipedal Robot with Deep Reinforcement Learning
Tuomas Haarnoja, Ben Moran, Guy Lever +25
We investigate whether Deep Reinforcement Learning (Deep RL) is able to synthesize sophisticated and safe movement skills for a low-cost, miniature humanoid robot that can be compo…
Leveraging Jumpy Models for Planning and Fast Learning in Robotic Domains
Jingwei Zhang, Jost Tobias Springenberg, Arunkumar Byravan +5
In this paper we study the problem of learning multi-step dynamics prediction models (jumpy models) from unlabeled experience and their utility for fast inference of (high-level) p…
NeRF2Real: Sim2real Transfer of Vision-guided Bipedal Motion Skills using Neural Radiance Fields
Arunkumar Byravan, Jan Humplik, Leonard Hasenclever +8
We present a system for applying sim2real approaches to "in the wild" scenes with realistic visuals, and to policies which rely on active perception using RGB cameras. Given a shor…
Imitate and Repurpose: Learning Reusable Robot Movement Skills From Human and Animal Behaviors
Steven Bohez, Saran Tunyasuvunakool, Philemon Brakel +18
We investigate the use of prior knowledge of human and animal movement to learn reusable locomotion skills for real legged robots. Our approach builds upon previous work on imitati…
Learning Transferable Motor Skills with Hierarchical Latent Mixture Policies
Dushyant Rao, Fereshteh Sadeghi, Leonard Hasenclever +8
For robots operating in the real world, it is desirable to learn reusable behaviours that can effectively be transferred and adapted to numerous tasks and scenarios. We propose an…
Learning Coordinated Terrain-Adaptive Locomotion by Imitating a Centroidal Dynamics Planner
Philemon Brakel, Steven Bohez, Leonard Hasenclever +2
Dynamic quadruped locomotion over challenging terrains with precise foot placements is a hard problem for both optimal control methods and Reinforcement Learning (RL). Non-linear s…
Evaluating model-based planning and planner amortization for continuous control
Arunkumar Byravan, Leonard Hasenclever, Piotr Trochim +8
There is a widespread intuition that model-based control methods should be able to surpass the data efficiency of model-free approaches. In this paper we attempt to evaluate this i…
Learning Dynamics Models for Model Predictive Agents
Michael Lutter, Leonard Hasenclever, Arunkumar Byravan +5
Model-Based Reinforcement Learning involves learning a \textit{dynamics model} from data, and then using this model to optimise behaviour, most often with an online \textit{planner…
From Motor Control to Team Play in Simulated Humanoid Football
Siqi Liu, Guy Lever, Zhe Wang +19
Intelligent behaviour in the physical world exhibits structure at multiple spatial and temporal scales. Although movements are ultimately executed at the level of instantaneous mus…
Behavior Priors for Efficient Reinforcement Learning
Dhruva Tirumala, Alexandre Galashov, Hyeonwoo Noh +8
As we deploy reinforcement learning agents to solve increasingly challenging problems, methods that allow us to inject prior knowledge about the structure of the world and effectiv…
Importance Weighted Policy Learning and Adaptation
Alexandre Galashov, Jakub Sygnowski, Guillaume Desjardins +5
The ability to exploit prior experience to solve novel problems rapidly is a hallmark of biological learning systems and of great practical importance for artificial ones. In the m…
A Distributional View on Multi-Objective Policy Optimization
Abbas Abdolmaleki, Sandy H. Huang, Leonard Hasenclever +7
Many real-world problems require trading off multiple competing objectives. However, these objectives are often in different units and/or scales, which can make it challenging for…