DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skills

Abstract

A longstanding goal in character animation is to combine data-driven specification of behavior with a system that can execute a similar behavior in a physical simulation, thus enabling realistic responses to perturbations and environmental variation. We show that well-known reinforcement learning (RL) methods can be adapted to learn robust control policies capable of imitating a broad range of example motion clips, while also learning complex recoveries, adapting to changes in morphology, and accomplishing user-specified goals. Our method handles keyframed motions, highly-dynamic actions such as motion-captured flips and spins, and retargeted motions. By combining a motion-imitation objective with a task objective, we can train characters that react intelligently in interactive settings, e.g., by walking in a desired direction or throwing a ball at a user-specified target. This approach thus combines the convenience and motion quality of using motion clips to define the desired style and appearance, with the flexibility and generality afforded by RL methods and physics-based animation. We further explore a number of methods for integrating multiple clips into the learning process to develop multi-skilled agents capable of performing a rich repertoire of diverse skills. We demonstrate results using multiple characters (human, Atlas robot, bipedal dinosaur, dragon) and a large variety of skills, including locomotion, acrobatics, and martial arts.

References in corpus (12)

• Trust Region Policy Optimization
• High-Dimensional Continuous Control Using Generalized Advantage Estimation
• Benchmarking Deep Reinforcement Learning for Continuous Control
• Emergence of Locomotion Behaviours in Rich Environments
• VIME: Variational Information Maximizing Exploration
• Generative Adversarial Imitation Learning
• Unifying Count-Based Exploration and Intrinsic Motivation
• Distral: Robust Multitask Reinforcement Learning
• Learning human behaviors from motion capture by adversarial imitation
• EPOpt: Learning Robust Neural Network Policies Using Model Ensembles
• Learning Locomotion Skills Using DeepRL: Does the Choice of Action Space Matter?
• Learning and Transfer of Modulated Locomotor Controllers

Cited by in corpus (176)

• Learning robust perceptive locomotion for quadrupedal robots in the wild
• How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learned
• AMP: Adversarial Motion Priors for Stylized Physics-Based Character Control
• Character Controllers Using Motion VAEs
• Robust Motion In-betweening
• Reward Constrained Policy Optimization
• ASE: Large-Scale Reusable Adversarial Skill Embeddings for Physically Simulated Characters
• Multi-expert learning of adaptive legged locomotion
• Learning Agile Soccer Skills for a Bipedal Robot with Deep Reinforcement Learning
• Recent advances in applying deep reinforcement learning for flow control: perspectives and future directions
• QuestSim: Human Motion Tracking from Sparse Sensors with Simulated Avatars
• DTC: Deep Tracking Control
• Edge Federation: Towards an Integrated Service Provisioning Model
• Towards Expedited Impedance Tuning of a Robotic Prosthesis for Personalized Gait Assistance by Reinforcement Learning Control
• ControlVAE: Model-Based Learning of Generative Controllers for Physics-Based Characters
• Learning Montezuma's Revenge from a Single Demonstration
• Learning Free Gait Transition for Quadruped Robots via Phase-Guided Controller
• Synthesis of Biologically Realistic Human Motion Using Joint Torque Actuation
• GANimator: Neural Motion Synthesis from a Single Sequence
• Transflower: probabilistic autoregressive dance generation with multimodal attention
• A Survey on Deep Learning for Skeleton-Based Human Animation
• Hierarchical visuomotor control of humanoids
• Iterative Reinforcement Learning Based Design of Dynamic Locomotion Skills for Cassie
• CALM: Conditional Adversarial Latent Models for Directable Virtual Characters
• Policy Transfer with Strategy Optimization
• Rhythm is a Dancer: Music-Driven Motion Synthesis with Global Structure
• Reinforcement Learning Control of a Biomechanical Model of the Upper Extremity
• Making Efficient Use of Demonstrations to Solve Hard Exploration Problems
• Closed-Loop Control of Direct Ink Writing via Reinforcement Learning
• CARL: Controllable Agent with Reinforcement Learning for Quadruped Locomotion
• RL + Model-based Control: Using On-demand Optimal Control to Learn Versatile Legged Locomotion
• Reinforcement Learning Applications
• Learning to Locomote: Understanding How Environment Design Matters for Deep Reinforcement Learning
• C$\cdot$ASE: Learning Conditional Adversarial Skill Embeddings for Physics-based Characters
• LOGAN: Unpaired Shape Transform in Latent Overcomplete Space
• Primal Wasserstein Imitation Learning
• Dynamics-Regulated Kinematic Policy for Egocentric Pose Estimation
• Learning Deformable Object Manipulation from Expert Demonstrations
• HumanMimic: Learning Natural Locomotion and Transitions for Humanoid Robot via Wasserstein Adversarial Imitation
• Residual Force Control for Agile Human Behavior Imitation and Extended Motion Synthesis
• Model-Based Real-Time Motion Tracking using Dynamical Inverse Kinematics on SO(3)
• Goal-conditioned Imitation Learning
• QuestEnvSim: Environment-Aware Simulated Motion Tracking from Sparse Sensors
• PADL: Language-Directed Physics-Based Character Control
• A GAN-Like Approach for Physics-Based Imitation Learning and Interactive Character Control
• Where Do You Think You're Going?: Inferring Beliefs about Dynamics from Behavior
• Enhancing Efficiency and Propulsion in Bio-mimetic Robotic Fish through End-to-End Deep Reinforcement Learning
• Torque-based Deep Reinforcement Learning for Task-and-Robot Agnostic Learning on Bipedal Robots Using Sim-to-Real Transfer
• Probabilistic Character Motion Synthesis using a Hierarchical Deep Latent Variable Model
• Backplay: "Man muss immer umkehren"
• Benchmarking Potential Based Rewards for Learning Humanoid Locomotion
• Variational Dynamic for Self-Supervised Exploration in Deep Reinforcement Learning
• AdaptNet: Policy Adaptation for Physics-Based Character Control
• Imitating and Finetuning Model Predictive Control for Robust and Symmetric Quadrupedal Locomotion
• Deep Reinforcement Learning for Human-Like Driving Policies in Collision Avoidance Tasks of Self-Driving Cars
• Generative Tweening: Long-term Inbetweening of 3D Human Motions
• Provably Efficient Imitation Learning from Observation Alone
• Motion In-Betweening with Phase Manifolds
• Reactive Stepping for Humanoid Robots using Reinforcement Learning: Application to Standing Push Recovery on the Exoskeleton Atalante
• Physical Interaction: Reconstructing Hand-object Interactions with Physics
• Learning to Generate Pointing Gestures in Situated Embodied Conversational Agents
• PDP: Physics-Based Character Animation via Diffusion Policy
• Generating Emotive Gaits for Virtual Agents Using Affect-Based Autoregression
• Deep Reinforcement Learning for Bipedal Locomotion: A Brief Survey
• Recent Advances in Leveraging Human Guidance for Sequential Decision-Making Tasks
• Generating Upper-Body Motion for Real-Time Characters Making their Way through Dynamic Environments
• Learning Navigation Subroutines from Egocentric Videos
• Observation Space Matters: Benchmark and Optimization Algorithm
• Learning Object Manipulation Skills via Approximate State Estimation from Real Videos
• Learning to Brachiate via Simplified Model Imitation
• On Multi-Agent Learning in Team Sports Games
• Reinforcement Learning for Robust Parameterized Locomotion Control of Bipedal Robots
• Automata Guided Reinforcement Learning With Demonstrations
• Deep reinforcement learning driven inspection and maintenance planning under incomplete information and constraints
• Constrained Reinforcement Learning Has Zero Duality Gap
• Physics-Based Dexterous Manipulations with Estimated Hand Poses and Residual Reinforcement Learning
• Ego-Pose Estimation and Forecasting as Real-Time PD Control
• A Survey of Behavior Learning Applications in Robotics -- State of the Art and Perspectives
• WalkTheDog: Cross-Morphology Motion Alignment via Phase Manifolds
• Smooth Exploration for Robotic Reinforcement Learning
• Demonstration-Guided Reinforcement Learning with Learned Skills
• Bidirectional GaitNet: A Bidirectional Prediction Model of Human Gait and Anatomical Conditions
• Learning Humanoid Robot Motions Through Deep Neural Networks
• Diffuse-CLoC: Guided Diffusion for Physics-based Character Look-ahead Control
• Circus ANYmal: A Quadruped Learning Dexterous Manipulation with Its Limbs
• Learning-based Initialization of Trajectory Optimization for Path-following Problems of Redundant Manipulators
• Fast and Efficient Locomotion via Learned Gait Transitions
• Learning to Sit: Synthesizing Human-Chair Interactions via Hierarchical Control
• MAAIP: Multi-Agent Adversarial Interaction Priors for imitation from fighting demonstrations for physics-based characters
• Local Search for Policy Iteration in Continuous Control
• Synchronize Dual Hands for Physics-Based Dexterous Guitar Playing
• Universal Value Density Estimation for Imitation Learning and Goal-Conditioned Reinforcement Learning
• A Two-part Transformer Network for Controllable Motion Synthesis
• Task-Oriented Hand Motion Retargeting for Dexterous Manipulation Imitation
• Model-based Behavioral Cloning with Future Image Similarity Learning
• Marathon Environments: Multi-Agent Continuous Control Benchmarks in a Modern Video Game Engine
• O2A: One-shot Observational learning with Action vectors
• Causal Question Answering with Reinforcement Learning
• An Adaptable Approach to Learn Realistic Legged Locomotion without Examples
• Physically Embedded Planning Problems: New Challenges for Reinforcement Learning
• Towards Robust Direction Invariance in Character Animation
• Teach Biped Robots to Walk via Gait Principles and Reinforcement Learning with Adversarial Critics
• Reward Constrained Interactive Recommendation with Natural Language Feedback
• SuperPADL: Scaling Language-Directed Physics-Based Control with Progressive Supervised Distillation
• PARC: Physics-based Augmentation with Reinforcement Learning for Character Controllers
• Physics-based Scene Layout Generation from Human Motion
• Learning to Walk and Fly with Adversarial Motion Priors
• Adaptive Tracking of a Single-Rigid-Body Character in Various Environments
• Functionality-Driven Musculature Retargeting
• How Much Do Unstated Problem Constraints Limit Deep Robotic Reinforcement Learning?
• Imitation Learning for Human Pose Prediction
• Human-in-the-Loop Methods for Data-Driven and Reinforcement Learning Systems
• Signs of Language: Embodied Sign Language Fingerspelling Acquisition from Demonstrations for Human-Robot Interaction
• The Importance and the Limitations of Sim2Real for Robotic Manipulation in Precision Agriculture
• Dynamic Future Net: Diversified Human Motion Generation
• Contextual Reinforcement Learning of Visuo-tactile Multi-fingered Grasping Policies
• Kinematics-Guided Reinforcement Learning for Object-Aware 3D Ego-Pose Estimation
• PhysicsFC: Learning User-Controlled Skills for a Physics-Based Football Player Controller
• Sequential Motion Planning for Bipedal Somersault via Flywheel SLIP and Momentum Transmission with Task Space Control
• AMOR: Adaptive Character Control through Multi-Objective Reinforcement Learning
• Learning to Run with Potential-Based Reward Shaping and Demonstrations from Video Data
• MobILE: Model-Based Imitation Learning From Observation Alone
• Optical Non-Line-of-Sight Physics-based 3D Human Pose Estimation
• Discovering Fatigued Movements for Virtual Character Animation
• Bridging the Imitation Gap by Adaptive Insubordination
• Implicit Behavioral Cloning
• Real-Time Locomotion on Soft Grounds With Dynamic Footprints
• Environment-aware Motion Matching
• Solving Challenging Control Problems Using Two-Staged Deep Reinforcement Learning
• Learning Humanoid Robot Running Skills through Proximal Policy Optimization
• Tensor-based Cooperative Control for Large Scale Multi-intersection Traffic Signal Using Deep Reinforcement Learning and Imitation Learning
• Reinforcement Learning and Control of a Lower Extremity Exoskeleton for Squat Assistance
• Transition Motion Tensor: A Data-Driven Approach for Versatile and Controllable Agents in Physically Simulated Environments
• Demystifying Reward Design in Reinforcement Learning for Upper Extremity Interaction: Practical Guidelines for Biomechanical Simulations in HCI
• HATSUKI : An anime character like robot figure platform with anime-style expressions and imitation learning based action generation
• SimPoE: Simulated Character Control for 3D Human Pose Estimation
• Constrained Exploration and Recovery from Experience Shaping
• Motion Keyframe Interpolation for Any Human Skeleton via Temporally Consistent Point Cloud Sampling and Reconstruction
• Fast State Stabilization using Deep Reinforcement Learning for Measurement-based Quantum Feedback Control
• Data-Driven Reinforcement Learning for Virtual Character Animation Control
• BundleMoCap: Efficient, Robust and Smooth Motion Capture from Sparse Multiview Videos
• Learning to Ball: Composing Policies for Long-Horizon Basketball Moves
• Hierarchical Style-based Networks for Motion Synthesis
• Discovering Diverse Athletic Jumping Strategies
• How does the structure embedded in learning policy affect learning quadruped locomotion?
• School of hard knocks: Curriculum analysis for Pommerman with a fixed computational budget
• Learning Human-like Locomotion Based on Biological Actuation and Rewards
• C-3PO: Cyclic-Three-Phase Optimization for Human-Robot Motion Retargeting based on Reinforcement Learning
• PFPN: Continuous Control of Physically Simulated Characters using Particle Filtering Policy Network
• Learning Agile Locomotion Skills with a Mentor
• Enhancing Navigation Efficiency of Quadruped Robots via Leveraging Personal Transportation Platforms
• Convolutional Humanoid Animation via Deformation
• Manipulator-Independent Representations for Visual Imitation
• Anatomically Detailed Simulation of Human Torso
• Metric-Based Imitation Learning Between Two Dissimilar Anthropomorphic Robotic Arms
• ForceGrip: Reference-Free Curriculum Learning for Realistic Grip Force Control in VR Hand Manipulation
• Quadruped Locomotion on Non-Rigid Terrain using Reinforcement Learning
• Learning from Interventions using Hierarchical Policies for Safe Learning
• On the Guaranteed Almost Equivalence between Imitation Learning from Observation and Demonstration
• Complex Skill Acquisition Through Simple Skill Imitation Learning
• Distilling a Hierarchical Policy for Planning and Control via Representation and Reinforcement Learning
• NEARL: Non-Explicit Action Reinforcement Learning for Robotic Control
• Deep Reinforcement Learning for High Level Character Control
• MaTIC: a Mathematical Theory of Inferential Communication
• GRP Model for Sensorimotor Learning
• Learning to Manipulate Amorphous Materials
• Learning Goal-Oriented Visual Dialog Agents: Imitating and Surpassing Analytic Experts
• Modelling Human Kinetics and Kinematics during Walking using Reinforcement Learning
• Integration of Imitation Learning using GAIL and Reinforcement Learning using Task-achievement Rewards via Probabilistic Graphical Model
• Efficient Hyperparameter Optimization for Physics-based Character Animation
• Advances, challenges, and opportunities for legged robots
• Deep Generative Modelling of Human Reach-and-Place Action
• Direct Random Search for Fine Tuning of Deep Reinforcement Learning Policies
• Vision-Guided Quadrupedal Locomotion in the Wild with Multi-Modal Delay Randomization
• Injective State-Image Mapping facilitates Visual Adversarial Imitation Learning
• Learning Bipedal Robot Locomotion from Human Movement