Publications (33)
Whole-Body Model-Predictive Control of Legged Robots with MuJoCo
John Z. Zhang, Taylor A. Howell, Zeji Yi +6
We demonstrate the surprising real-world effectiveness of a very simple approach to whole-body model-predictive control (MPC) of quadruped and humanoid robots: the iterative LQR (i…
Modelling Generalized Forces with Reinforcement Learning for Sim-to-Real Transfer
Rae Jeong, Jackie Kay, Francesco Romano +6
Learning robotic control policies in the real world gives rise to challenges in data efficiency, safety, and controlling the initial condition of the system. On the other hand, sim…
DeepMind Control Suite
Yuval Tassa, Yotam Doron, Alistair Muldal +9
The DeepMind Control Suite is a set of continuous control tasks with a standardised structure and interpretable rewards, intended to serve as performance benchmarks for reinforceme…
Relative Entropy Regularized Policy Iteration
Abbas Abdolmaleki, Jost Tobias Springenberg, Jonas Degrave +5
We present an off-policy actor-critic algorithm for Reinforcement Learning (RL) that combines ideas from gradient-free optimization via stochastic search with learned action-value…
Continuous control with deep reinforcement learning
Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel +5
We adapt the ideas underlying the success of Deep Q-Learning to the continuous action domain. We present an actor-critic, model-free algorithm based on the deterministic policy gra…
dm_control: Software and Tasks for Continuous Control
Yuval Tassa, Saran Tunyasuvunakool, Alistair Muldal +8
The dm_control software package is a collection of Python libraries and task suites for reinforcement learning agents in an articulated-body simulation. A MuJoCo wrapper provides c…
Maximum a Posteriori Policy Optimisation
Abbas Abdolmaleki, Jost Tobias Springenberg, Yuval Tassa +3
We introduce a new algorithm for reinforcement learning called Maximum aposteriori Policy Optimisation (MPO) based on coordinate ascent on a relative entropy objective. We show tha…
Motion Tracking with Muscles: Predictive Control of a Parametric Musculoskeletal Canine Model
Vittorio La Barbera, Steven Bohez, Leonard Hasenclever +2
We introduce a novel musculoskeletal model of a dog, procedurally generated from accurate 3D muscle meshes. Accompanying this model is a motion capture-based locomotion task compat…
OstrichRL: A Musculoskeletal Ostrich Simulation to Study Bio-mechanical Locomotion
Vittorio La Barbera, Fabio Pardo, Yuval Tassa +4
Muscle-actuated control is a research topic that spans multiple domains, including biomechanics, neuroscience, reinforcement learning, robotics, and graphics. This type of control…
Learning Dynamics Models for Model Predictive Agents
Michael Lutter, Leonard Hasenclever, Arunkumar Byravan +5
Model-Based Reinforcement Learning involves learning a \textit{dynamics model} from data, and then using this model to optimise behaviour, most often with an online \textit{planner…
Data-efficient Deep Reinforcement Learning for Dexterous Manipulation
Ivaylo Popov, Nicolas Heess, Timothy Lillicrap +7
Deep learning and reinforcement learning methods have recently been used to solve a variety of problems in continuous control domains. An obvious application of these techniques is…
Catch & Carry: Reusable Neural Controllers for Vision-Guided Whole-Body Tasks
Josh Merel, Saran Tunyasuvunakool, Arun Ahuja +6
We address the longstanding challenge of producing flexible, realistic humanoid character controllers that can perform diverse whole-body tasks involving object interactions. This…
Learning Awareness Models
Brandon Amos, Laurent Dinh, Serkan Cabi +7
We consider the setting of an agent with a fixed body interacting with an unknown and uncertain external world. We show that models trained to predict proprioceptive information ab…
From Motor Control to Team Play in Simulated Humanoid Football
Siqi Liu, Guy Lever, Zhe Wang +19
Intelligent behaviour in the physical world exhibits structure at multiple spatial and temporal scales. Although movements are ultimately executed at the level of instantaneous mus…
Deep neuroethology of a virtual rodent
Josh Merel, Diego Aldarondo, Jesse Marshall +3
Parallel developments in neuroscience and deep learning have led to mutually productive exchanges, pushing our understanding of real and artificial neural networks in sensory and c…
Evaluating model-based planning and planner amortization for continuous control
Arunkumar Byravan, Leonard Hasenclever, Piotr Trochim +8
There is a widespread intuition that model-based control methods should be able to surpass the data efficiency of model-free approaches. In this paper we attempt to evaluate this i…
Language to Rewards for Robotic Skill Synthesis
Wenhao Yu, Nimrod Gileadi, Chuyuan Fu +17
Large language models (LLMs) have demonstrated exciting progress in acquiring diverse new capabilities through in-context learning, ranging from logical reasoning to code-writing.…
Learning human behaviors from motion capture by adversarial imitation
Josh Merel, Yuval Tassa, Dhruva TB +5
Rapid progress in deep reinforcement learning has made it increasingly feasible to train controllers for high-dimensional humanoid bodies. However, methods that use pure reinforcem…
Imitate and Repurpose: Learning Reusable Robot Movement Skills From Human and Animal Behaviors
Steven Bohez, Saran Tunyasuvunakool, Philemon Brakel +18
We investigate the use of prior knowledge of human and animal movement to learn reusable locomotion skills for real legged robots. Our approach builds upon previous work on imitati…
Safe Exploration in Continuous Action Spaces
Gal Dalal, Krishnamurthy Dvijotham, Matej Vecerik +3
We address the problem of deploying a reinforcement learning (RL) agent on a physical system such as a datacenter cooling unit or robot, where critical constraints must never be vi…
Learning Continuous Control Policies by Stochastic Value Gradients
Nicolas Heess, Greg Wayne, David Silver +3
We present a unified framework for learning continuous control policies using backpropagation. It supports stochastic control by treating stochasticity in the Bellman equation as a…
Emergence of Locomotion Behaviours in Rich Environments
Nicolas Heess, Dhruva TB, Srinivasan Sriram +9
The reinforcement learning paradigm allows, in principle, for complex behaviours to be learned directly from simple reward signals. In practice, however, it is common to carefully…
MuJoCo Playground
Kevin Zakka, Baruch Tabanpour, Qiayuan Liao +10
We introduce MuJoCo Playground, a fully open-source framework for robot learning built with MJX, with the express goal of streamlining simulation, training, and sim-to-real transfe…
Learning and Transfer of Modulated Locomotor Controllers
Nicolas Heess, Greg Wayne, Yuval Tassa +3
We study a novel architecture and training procedure for locomotion tasks. A high-frequency, low-level "spinal" network with access to proprioceptive sensors learns sensorimotor pr…
Predictive Sampling: Real-time Behaviour Synthesis with MuJoCo
Taylor Howell, Nimrod Gileadi, Saran Tunyasuvunakool +3
We introduce MuJoCo MPC (MJPC), an open-source, interactive application and software framework for real-time predictive control, based on MuJoCo physics. MJPC allows the user to ea…
Achieving Human Level Competitive Robot Table Tennis
David B. D'Ambrosio, Saminda Abeyruwan, Laura Graesser +24
Achieving human-level speed and performance on real world tasks is a north star for the robotics research community. This work takes a step towards that goal and presents the first…
Learning Agile Soccer Skills for a Bipedal Robot with Deep Reinforcement Learning
Tuomas Haarnoja, Ben Moran, Guy Lever +25
We investigate whether Deep Reinforcement Learning (Deep RL) is able to synthesize sophisticated and safe movement skills for a low-cost, miniature humanoid robot that can be compo…
Learning Gentle Object Manipulation with Curiosity-Driven Deep Reinforcement Learning
Sandy H. Huang, Martina Zambelli, Jackie Kay +4
Robots must know how to be gentle when they need to interact with fragile objects, or when the robot itself is prone to wear and tear. We propose an approach that enables deep rein…
Attend, Infer, Repeat: Fast Scene Understanding with Generative Models
S. M. Ali Eslami, Nicolas Heess, Theophane Weber +4
We present a framework for efficient inference in structured image models that explicitly reason about objects. We achieve this by performing probabilistic inference using a recurr…
Efficient Online Learning of Contact Force Models for Connector Insertion
Kevin Tracy, Zachary Manchester, Ajinkya Jain +4
Contact-rich manipulation tasks with stiff frictional elements like connector insertion are difficult to model with rigid-body simulators. In this work, we propose a new approach f…
Local Search for Policy Iteration in Continuous Control
Jost Tobias Springenberg, Nicolas Heess, Daniel Mankowitz +10
We present an algorithm for local, regularized, policy improvement in reinforcement learning (RL) that allows us to formulate model-based and model-free variants in a single framew…
The Trajectory Bundle Method: Unifying Sequential-Convex Programming and Sampling-Based Trajectory Optimization
Kevin Tracy, John Z. Zhang, Jon Arrizabalaga +4
We present a unified framework for solving trajectory optimization problems in a derivative-free manner through the use of sequential convex programming. Traditionally, nonconvex o…
RoboPianist: Dexterous Piano Playing with Deep Reinforcement Learning
Kevin Zakka, Philipp Wu, Laura Smith +8
Replicating human-like dexterity in robot hands represents one of the largest open problems in robotics. Reinforcement learning is a promising approach that has achieved impressive…