Emergence of Locomotion Behaviours in Rich Environments

Abstract

The reinforcement learning paradigm allows, in principle, for complex behaviours to be learned directly from simple reward signals. In practice, however, it is common to carefully hand-design the reward function to encourage a particular solution, or to derive it from demonstration data. In this paper explore how a rich environment can help to promote the learning of complex behavior. Specifically, we train agents in diverse environmental contexts, and find that this encourages the emergence of robust behaviours that perform well across a suite of tasks. We demonstrate this principle for locomotion -- behaviours that are known for their sensitivity to the choice of reward. We train several simulated bodies on a diverse set of challenging terrains and obstacles, using a simple reward function based on forward progress. Using a novel scalable variant of policy gradient reinforcement learning, our agents learn to run, jump, crouch and turn as required by the environment without explicit reward-based guidance. A visual depiction of highlights of the learned behavior can be viewed following https://youtu.be/hx_bgoTF7bs .

References in corpus (7)

• Continuous control with deep reinforcement learning
• High-Dimensional Continuous Control Using Generalized Advantage Estimation
• End-to-End Training of Deep Visuomotor Policies
• Benchmarking Deep Reinforcement Learning for Continuous Control
• Massively Parallel Methods for Deep Reinforcement Learning
• Learning and Transfer of Modulated Locomotor Controllers
• Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updates

Cited by in corpus (189)

• A Brief Survey of Deep Reinforcement Learning
• Learning agile and dynamic motor skills for legged robots
• DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skills
• A Survey and Critique of Multiagent Deep Reinforcement Learning
• How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learned
• Distributed Prioritized Experience Replay
• Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximation
• Emergent Tool Use From Multi-Agent Autocurricula
• Embodied Intelligence via Learning and Evolution
• Distributional Soft Actor-Critic: Off-Policy Reinforcement Learning for Addressing Value Estimation Errors
• Distributed Distributional Deterministic Policy Gradients
• Actor-Attention-Critic for Multi-Agent Reinforcement Learning
• Character Controllers Using Motion VAEs
• Learning by Playing - Solving Sparse Reward Tasks from Scratch
• Benchmarking Model-Based Reinforcement Learning
• Maximum a Posteriori Policy Optimisation
• dm_control: Software and Tasks for Continuous Control
• Applying deep reinforcement learning to active flow control in turbulent conditions
• Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning
• Simple random search provides a competitive approach to reinforcement learning
• Autonomous Unmanned Aerial Vehicle Navigation using Reinforcement Learning: A Systematic Review
• AutoSlim: Towards One-Shot Architecture Search for Channel Numbers
• Emergent Complexity via Multi-Agent Competition
• Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutions
• Sim-to-Real: Learning Agile Locomotion For Quadruped Robots
• Reinforcement and Imitation Learning for Diverse Visuomotor Skills
• Critic Regularized Regression
• Learning to Walk in the Real World with Minimal Human Effort
• Non-Smooth Newton Methods for Deformable Multi-Body Dynamics
• Exploring Model-based Planning with Policy Networks
• Dynamics-Aware Unsupervised Discovery of Skills
• A Survey of Deep Network Solutions for Learning Control in Robotics: From Reinforcement to Imitation
• Measuring and modeling the motor system with machine learning
• Hierarchical visuomotor control of humanoids
• Iterative Reinforcement Learning Based Design of Dynamic Locomotion Skills for Cassie
• Meta reinforcement learning as task inference
• Policy Transfer with Strategy Optimization
• Why Does Hierarchy (Sometimes) Work So Well in Reinforcement Learning?
• An empirical investigation of the challenges of real-world reinforcement learning
• Enhanced POET: Open-Ended Reinforcement Learning through Unbounded Invention of Learning Challenges and their Solutions
• RoboTurk: A Crowdsourcing Platform for Robotic Skill Learning through Imitation
• Relative Entropy Regularized Policy Iteration
• Transfer in Deep Reinforcement Learning Using Successor Features and Generalised Policy Improvement
• DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Frames
• Towards Optimally Decentralized Multi-Robot Collision Avoidance via Deep Reinforcement Learning
• Unicorn: Continual Learning with a Universal, Off-policy Agent
• Recurrent Deterministic Policy Gradient Method for Bipedal Locomotion on Rough Terrain Challenge
• Neural Graph Evolution: Towards Efficient Automatic Robot Design
• Deterministic Implementations for Reproducibility in Deep Reinforcement Learning
• Adaptive Prior Selection for Repertoire-based Online Adaptation in Robotics
• Quantum imaginary time evolution steered by reinforcement learning
• Self-supervised Learning of Image Embedding for Continuous Control
• TensorFlow Agents: Efficient Batched Reinforcement Learning in TensorFlow
• Mix&Match - Agent Curricula for Reinforcement Learning
• Data Efficient Reinforcement Learning for Legged Robots
• Learning Vision-Guided Quadrupedal Locomotion End-to-End with Cross-Modal Transformers
• Trust-PCL: An Off-Policy Trust Region Method for Continuous Control
• One Policy to Control Them All: Shared Modular Policies for Agent-Agnostic Control
• FuzzerGym: A Competitive Framework for Fuzzing and Learning
• Learning to Explore with Meta-Policy Gradient
• Generalization in Dexterous Manipulation via Geometry-Aware Multi-Task Learning
• TriFinger: An Open-Source Robot for Learning Dexterity
• Reinforcement Learning with Competitive Ensembles of Information-Constrained Primitives
• Feedback Control For Cassie With Deep Reinforcement Learning
• Whole-Body Control of a Mobile Manipulator using End-to-End Reinforcement Learning
• Exploiting Hierarchy for Learning and Transfer in KL-regularized RL
• GPU-Accelerated Robotic Simulation for Distributed Reinforcement Learning
• Reactive Stepping for Humanoid Robots using Reinforcement Learning: Application to Standing Push Recovery on the Exoskeleton Atalante
• A hybrid learning method for system identification and optimal control
• Strategies for Using Proximal Policy Optimization in Mobile Puzzle Games
• Sequence Modeling of Temporal Credit Assignment for Episodic Reinforcement Learning
• Where Did My Optimum Go?: An Empirical Analysis of Gradient Descent Optimization in Policy Gradient Methods
• Learning to Generate Pointing Gestures in Situated Embodied Conversational Agents
• Reward-Conditioned Policies
• Modelling Generalized Forces with Reinforcement Learning for Sim-to-Real Transfer
• Generation of ice states through deep reinforcement learning
• Using a thousand optimization tasks to learn hyperparameter search strategies
• Tasks, stability, architecture, and compute: Training more effective learned optimizers, and using them to train themselves
• Behavior Priors for Efficient Reinforcement Learning
• Development of collective behavior in newborn artificial agents
• Asynchronous Methods for Model-Based Reinforcement Learning
• Human-Inspired Multi-Agent Navigation using Knowledge Distillation
• Hedging using reinforcement learning: Contextual $k$-Armed Bandit versus $Q$-learning
• Orchestrating the Development Lifecycle of Machine Learning-Based IoT Applications: A Taxonomy and Survey
• Reinforcement Learning Testbed for Power-Consumption Optimization
• Ecological Reinforcement Learning
• Evolving the Behavior of Machines: From Micro to Macroevolution
• Time Limits in Reinforcement Learning
• The Garden of Forking Paths: Towards Multi-Future Trajectory Prediction
• Deep neuroethology of a virtual rodent
• A Deep Reinforcement Learning Framework for Eco-driving in Connected and Automated Hybrid Electric Vehicles
• Evaluating Agents without Rewards
• Divide-and-Conquer Reinforcement Learning
• Towards General and Autonomous Learning of Core Skills: A Case Study in Locomotion
• Physics-Based Dexterous Manipulations with Estimated Hand Poses and Residual Reinforcement Learning
• Dynamic Experience Replay
• Learning Awareness Models
• AI-MOLE: Autonomous Iterative Motion Learning for Unknown Nonlinear Dynamics with Extensive Experimental Validation
• Comparing Deep Reinforcement Learning and Evolutionary Methods in Continuous Control
• Curriculum in Gradient-Based Meta-Reinforcement Learning
• Human-like machine learning: limitations and suggestions
• Arena: A General Evaluation Platform and Building Toolkit for Multi-Agent Intelligence
• Automating Vehicles by Deep Reinforcement Learning using Task Separation with Hill Climbing
• Offline Meta-Reinforcement Learning with Online Self-Supervision
• The Benchmark Lottery
• Fast and Efficient Locomotion via Learned Gait Transitions
• Guided Curriculum Learning for Walking Over Complex Terrain
• Deep Reinforcement Learning to Acquire Navigation Skills for Wheel-Legged Robots in Complex Environments
• On Entropy Regularized Path Integral Control for Trajectory Optimization
• Zero-Shot Terrain Generalization for Visual Locomotion Policies
• Hindsight Trust Region Policy Optimization
• Learning to Sit: Synthesizing Human-Chair Interactions via Hierarchical Control
• Marathon Environments: Multi-Agent Continuous Control Benchmarks in a Modern Video Game Engine
• Distributed Heuristic Multi-Agent Path Finding with Communication
• Synthesizing Long-Term 3D Human Motion and Interaction in 3D Scenes
• Cascade Attribute Learning Network
• Physically Embedded Planning Problems: New Challenges for Reinforcement Learning
• Imagined Value Gradients: Model-Based Policy Optimization with Transferable Latent Dynamics Models
• Run, skeleton, run: skeletal model in a physics-based simulation
• Reinforcement Learning with Adaptive Curriculum Dynamics Randomization for Fault-Tolerant Robot Control
• Deep Reinforcement Learning in Fluid Mechanics: a promising method for both Active Flow Control and Shape Optimization
• ALLSTEPS: Curriculum-driven Learning of Stepping Stone Skills
• Fiber: A Platform for Efficient Development and Distributed Training for Reinforcement Learning and Population-Based Methods
• PPMC RL Training Algorithm: Rough Terrain Intelligent Robots through Reinforcement Learning
• Balance Between Efficient and Effective Learning: Dense2Sparse Reward Shaping for Robot Manipulation with Environment Uncertainty
• Learning to Design RNA
• Robust Domain Randomised Reinforcement Learning through Peer-to-Peer Distillation
• Combining Benefits from Trajectory Optimization and Deep Reinforcement Learning
• Model-free and Bayesian Ensembling Model-based Deep Reinforcement Learning for Particle Accelerator Control Demonstrated on the FERMI FEL
• Dynamics and Domain Randomized Gait Modulation with Bezier Curves for Sim-to-Real Legged Locomotion
• Thalamocortical motor circuit insights for more robust hierarchical control of complex sequences
• Learning Generalizable Locomotion Skills with Hierarchical Reinforcement Learning
• HJB Optimal Feedback Control with Deep Differential Value Functions and Action Constraints
• An Actor-Critic-Attention Mechanism for Deep Reinforcement Learning in Multi-view Environments
• Bottom-Up Meta-Policy Search
• Competitive Experience Replay
• Cross-Domain Imitation Learning via Optimal Transport
• Explicitly Encouraging Low Fractional Dimensional Trajectories Via Reinforcement Learning
• Using Deep Reinforcement Learning to Learn High-Level Policies on the ATRIAS Biped
• Bridging the Imitation Gap by Adaptive Insubordination
• Augmented Random Search for Quadcopter Control: An alternative to Reinforcement Learning
• Simple Sensor Intentions for Exploration
• Learning Diverse Policies with Soft Self-Generated Guidance
• Snowflake: Scaling GNNs to High-Dimensional Continuous Control via Parameter Freezing
• From Recognition to Prediction: Analysis of Human Action and Trajectory Prediction in Video
• Distributed Deep Reinforcement Learning: An Overview
• Deep Learned Path Planning via Randomized Reward-Linked-Goals and Potential Space Applications
• Learning 6DoF Grasping Using Reward-Consistent Demonstration
• Adaptive Smoothing Path Integral Control
• Importance Weighted Evolution Strategies
• Competitiveness of MAP-Elites against Proximal Policy Optimization on locomotion tasks in deterministic simulations
• Beyond Tabula-Rasa: a Modular Reinforcement Learning Approach for Physically Embedded 3D Sokoban
• NavigationNet: A Large-scale Interactive Indoor Navigation Dataset
• SeRO: Self-Supervised Reinforcement Learning for Recovery from Out-of-Distribution Situations
• Mission schedule of agile satellites based on Proximal Policy Optimization Algorithm
• Partially Connected Automated Vehicle Cooperative Control Strategy with a Deep Reinforcement Learning Approach
• Critic PI2: Master Continuous Planning via Policy Improvement with Path Integrals and Deep Actor-Critic Reinforcement Learning
• Discovering Generalizable Skills via Automated Generation of Diverse Tasks
• PFPN: Continuous Control of Physically Simulated Characters using Particle Filtering Policy Network
• Learning Agile Locomotion Skills with a Mentor
• Hierarchical Skills for Efficient Exploration
• SURREAL-System: Fully-Integrated Stack for Distributed Deep Reinforcement Learning
• Multi-intersection Traffic Optimisation: A Benchmark Dataset and a Strong Baseline
• Design of AoI-Aware 5G Uplink Scheduler UsingReinforcement Learning
• High-Level Perceptual Similarity is Enabled by Learning Diverse Tasks
• Evaluating model-based planning and planner amortization for continuous control
• Discovering Diverse Athletic Jumping Strategies
• Metric-Based Imitation Learning Between Two Dissimilar Anthropomorphic Robotic Arms
• Direct Random Search for Fine Tuning of Deep Reinforcement Learning Policies
• Vision-Guided Quadrupedal Locomotion in the Wild with Multi-Modal Delay Randomization
• Efficient Reinforcement Learning Development with RLzoo
• Containerized Distributed Value-Based Multi-Agent Reinforcement Learning
• Efficient Hyperparameter Optimization for Physics-based Character Animation
• Training Transition Policies via Distribution Matching for Complex Tasks
• Proximal Policy Optimization with Mixed Distributed Training
• Parallelized Reverse Curriculum Generation
• Learning and Exploring Motor Skills with Spacetime Bounds
• Homotopy Based Reinforcement Learning with Maximum Entropy for Autonomous Air Combat
• Biased Estimates of Advantages over Path Ensembles
• Recognition and Synthesis of Object Transport Motion
• Adaptive Policy Transfer in Reinforcement Learning
• Quadruped Locomotion on Non-Rigid Terrain using Reinforcement Learning
• Learning a Skill-sequence-dependent Policy for Long-horizon Manipulation Tasks
• Discrete linear-complexity reinforcement learning in continuous action spaces for Q-learning algorithms
• Learning Transferable Concepts in Deep Reinforcement Learning
• Region Growing Curriculum Generation for Reinforcement Learning
• Lifetime policy reuse and the importance of task capacity
• Zero Shot Learning on Simulated Robots
• Learning walk and trot from the same objective using different types of exploration