Publications (15)
Diffusion Augmented Agents: A Framework for Efficient Exploration and Transfer Learning
Norman Di Palo, Leonard Hasenclever, Jan Humplik +1
We introduce Diffusion Augmented Agents (DAAG), a novel framework that leverages large language models, vision language models, and diffusion models to improve sample efficiency an…
Learning to Learn Faster from Human Feedback with Language Model Predictive Control
Jacky Liang, Fei Xia, Wenhao Yu +47
Large language models (LLMs) have been shown to exhibit a wide range of capabilities, such as writing robot code from language commands -- enabling non-experts to direct robot beha…
Gemini Robotics: Bringing AI into the Physical World
Gemini Robotics Team, Saminda Abeyruwan, Joshua Ainslie +115
Recent advancements in large multimodal models have led to the emergence of remarkable generalist capabilities in digital domains, yet their translation to physical agents such as…
NeRF2Real: Sim2real Transfer of Vision-guided Bipedal Motion Skills using Neural Radiance Fields
Arunkumar Byravan, Jan Humplik, Leonard Hasenclever +8
We present a system for applying sim2real approaches to "in the wild" scenes with realistic visuals, and to policies which rely on active perception using RGB cameras. Given a shor…
Importance Weighted Policy Learning and Adaptation
Alexandre Galashov, Jakub Sygnowski, Guillaume Desjardins +5
The ability to exploit prior experience to solve novel problems rapidly is a hallmark of biological learning systems and of great practical importance for artificial ones. In the m…
Learning Robot Soccer from Egocentric Vision with Deep Reinforcement Learning
Dhruva Tirumala, Markus Wulfmeier, Ben Moran +13
We apply multi-agent deep reinforcement learning (RL) to train end-to-end robot soccer policies with fully onboard computation and sensing via egocentric RGB vision. This setting r…
Proc4Gem: Foundation models for physical agency through procedural generation
Yixin Lin, Jan Humplik, Sandy H. Huang +18
In robot learning, it is common to either ignore the environment semantics, focusing on tasks like whole-body control which only require reasoning about robot-environment contacts,…
Forgetting and Imbalance in Robot Lifelong Learning with Off-policy Data
Wenxuan Zhou, Steven Bohez, Jan Humplik +5
Robots will experience non-stationary environment dynamics throughout their lifetime: the robot dynamics can change due to wear and tear, or its surroundings may change over time.…
Meta reinforcement learning as task inference
Jan Humplik, Alexandre Galashov, Leonard Hasenclever +3
Humans achieve efficient learning by relying on prior knowledge about the structure of naturally occurring tasks. There is considerable interest in designing reinforcement learning…
Learning Agile Soccer Skills for a Bipedal Robot with Deep Reinforcement Learning
Tuomas Haarnoja, Ben Moran, Guy Lever +25
We investigate whether Deep Reinforcement Learning (Deep RL) is able to synthesize sophisticated and safe movement skills for a low-cost, miniature humanoid robot that can be compo…
SkillS: Adaptive Skill Sequencing for Efficient Temporally-Extended Exploration
Giulia Vezzani, Dhruva Tirumala, Markus Wulfmeier +13
The ability to effectively reuse prior knowledge is a key requirement when building general and flexible Reinforcement Learning (RL) agents. Skill reuse is one of the most common a…
Semiparametric energy-based probabilistic models
Jan Humplik, GaÅ¡per TkaÄik
Probabilistic models can be defined by an energy function, where the probability of each state is proportional to the exponential of the state's negative energy. This paper conside…
Inferring couplings in networks across order-disorder phase transitions
Vudtiwat Ngampruetikorn, Vedant Sachdeva, Johanna Torrence +3
Statistical inference is central to many scientific endeavors, yet how it works remains unresolved. Answering this requires a quantitative understanding of the intrinsic interplay…
Language to Rewards for Robotic Skill Synthesis
Wenhao Yu, Nimrod Gileadi, Chuyuan Fu +17
Large language models (LLMs) have demonstrated exciting progress in acquiring diverse new capabilities through in-context learning, ranging from logical reasoning to code-writing.…
Imitate and Repurpose: Learning Reusable Robot Movement Skills From Human and Animal Behaviors
Steven Bohez, Saran Tunyasuvunakool, Philemon Brakel +18
We investigate the use of prior knowledge of human and animal movement to learn reusable locomotion skills for real legged robots. Our approach builds upon previous work on imitati…