Learning to Locomote: Understanding How Environment Design Matters for Deep Reinforcement Learning

Abstract

Learning to locomote is one of the most common tasks in physics-based animation and deep reinforcement learning (RL). A learned policy is the product of the problem to be solved, as embodied by the RL environment, and the RL algorithm. While enormous attention has been devoted to RL algorithms, much less is known about the impact of design choices for the RL environment. In this paper, we show that environment design matters in significant ways and document how it can contribute to the brittle nature of many RL results. Specifically, we examine choices related to state representations, initial state distributions, reward structure, control frequency, episode termination procedures, curriculum usage, the action space, and the torque limits. We aim to stimulate discussion around such choices, which in practice strongly impact the success of RL when applied to continuous-action control problems of interest to animation, such as learning to locomote.

arXiv commentPresented at The 13th Annual ACM SIGGRAPH Conference on Motion, Interaction and Games

References in corpus (13)

• Addressing Function Approximation Error in Actor-Critic Methods
• DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skills
• Deep Reinforcement Learning that Matters
• Curriculum Learning for Reinforcement Learning Domains: A Framework and Survey
• Simple random search provides a competitive approach to reinforcement learning
• Learning Symmetric and Low-energy Locomotion
• Reverse Curriculum Generation for Reinforcement Learning
• Learning Locomotion Skills Using DeepRL: Does the Choice of Action Space Matter?
• Assessing Generalization in Deep Reinforcement Learning
• What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Study
• A Dissection of Overfitting and Generalization in Continuous Reinforcement Learning
• On Inductive Biases in Deep Reinforcement Learning
• Control Frequency Adaptation via Action Persistence in Batch Reinforcement Learning

Cited by in corpus (7)

• Concurrent Training of a Control Policy and a State Estimator for Dynamic and Robust Legged Locomotion
• Observation Space Matters: Benchmark and Optimization Algorithm
• Reinforcement Learning with Adaptive Curriculum Dynamics Randomization for Fault-Tolerant Robot Control
• A Survey on Reinforcement Learning Methods in Character Animation
• Towards Automatic Actor-Critic Solutions to Continuous Control
• PFPN: Continuous Control of Physically Simulated Characters using Particle Filtering Policy Network
• In-situ Value-aligned Human-Robot Interactions with Physical Constraints