Learning Agile Soccer Skills for a Bipedal Robot with Deep Reinforcement Learning

Abstract

We investigate whether Deep Reinforcement Learning (Deep RL) is able to synthesize sophisticated and safe movement skills for a low-cost, miniature humanoid robot that can be composed into complex behavioral strategies in dynamic environments. We used Deep RL to train a humanoid robot with 20 actuated joints to play a simplified one-versus-one (1v1) soccer game. The resulting agent exhibits robust and dynamic movement skills such as rapid fall recovery, walking, turning, kicking and more; and it transitions between them in a smooth, stable, and efficient manner. The agent's locomotion and tactical behavior adapts to specific game contexts in a way that would be impractical to manually design. The agent also developed a basic strategic understanding of the game, and learned, for instance, to anticipate ball movements and to block opponent shots. Our agent was trained in simulation and transferred to real robots zero-shot. We found that a combination of sufficiently high-frequency control, targeted dynamics randomization, and perturbations during training in simulation enabled good-quality transfer. Although the robots are inherently fragile, basic regularization of the behavior during training led the robots to learn safe and effective movements while still performing in a dynamic and agile way -- well beyond what is intuitively expected from the robot. Indeed, in experiments, they walked 181% faster, turned 302% faster, took 63% less time to get up, and kicked a ball 34% faster than a scripted baseline, while efficiently combining the skills to achieve the longer term objectives.

Project website: https://sites.google.com/view/op3-soccer

References in corpus (25)

• Learning agile and dynamic motor skills for legged robots
• Learning Quadrupedal Locomotion over Challenging Terrain
• DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skills
• Learning robust perceptive locomotion for quadrupedal robots in the wild
• How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learned
• A Survey of Multi-Objective Sequential Decision-Making
• Emergent Tool Use From Multi-Agent Autocurricula
• Learning by Playing - Solving Sparse Reward Tasks from Scratch
• A Distributional Perspective on Reinforcement Learning
• Maximum a Posteriori Policy Optimisation
• dm_control: Software and Tasks for Continuous Control
• Concurrent Training of a Control Policy and a State Estimator for Dynamic and Robust Legged Locomotion
• Emergent Complexity via Multi-Agent Competition
• A Unified Game-Theoretic Approach to Multiagent Reinforcement Learning
• Learning to Walk in the Real World with Minimal Human Effort
• Emergent Coordination Through Competition
• Open-ended Learning in Symmetric Zero-sum Games
• DayDreamer: World Models for Physical Robot Learning
• Responsive Safety in Reinforcement Learning by PID Lagrangian Methods
• Legged Locomotion in Challenging Terrains using Egocentric Vision
• A Distributional View on Multi-Objective Policy Optimization
• Information asymmetry in KL-regularized RL
• Behavior Priors for Efficient Reinforcement Learning
• Towards General and Autonomous Learning of Core Skills: A Case Study in Locomotion
• Data-efficient Hindsight Off-policy Option Learning

Cited by in corpus (14)

• HumanMimic: Learning Natural Locomotion and Transitions for Humanoid Robot via Wasserstein Adversarial Imitation
• A roadmap for AI in robotics
• Learning coordinated badminton skills for legged manipulators
• Deep Reinforcement Learning for Bipedal Locomotion: A Brief Survey
• Learning Quadrotor Control From Visual Features Using Differentiable Simulation
• Learn to Teach: Sample-Efficient Privileged Learning for Humanoid Locomotion over Diverse Terrains
• Visual-Policy Learning through Multi-Camera View to Single-Camera View Knowledge Distillation for Robot Manipulation Tasks
• PhysicsFC: Learning User-Controlled Skills for a Physics-Based Football Player Controller
• Mechanical Intelligence-Aware Curriculum Reinforcement Learning for Humanoids with Parallel Actuation
• Intercepting Unauthorized Aerial Robots in Controlled Airspace Using Reinforcement Learning
• Demystifying Reward Design in Reinforcement Learning for Upper Extremity Interaction: Practical Guidelines for Biomechanical Simulations in HCI
• Body Design and Gait Generation of Chair-Type Asymmetrical Tripedal Low-rigidity Robot
• Towards Miniature Humanoid Tele-Loco-Manipulation Using Virtual Reality and Reinforcement Learning
• ForceGrip: Reference-Free Curriculum Learning for Realistic Grip Force Control in VR Hand Manipulation