Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization

Abstract

Reinforcement learning can acquire complex behaviors from high-level specifications. However, defining a cost function that can be optimized effectively and encodes the correct task is challenging in practice. We explore how inverse optimal control (IOC) can be used to learn behaviors from demonstrations, with applications to torque control of high-dimensional robotic systems. Our method addresses two key challenges in inverse optimal control: first, the need for informative features and effective regularization to impose structure on the cost, and second, the difficulty of learning the cost function under unknown dynamics for high-dimensional continuous systems. To address the former challenge, we present an algorithm capable of learning arbitrary nonlinear cost functions, such as neural networks, without meticulous feature engineering. To address the latter challenge, we formulate an efficient sample-based approximation for MaxEnt IOC. We evaluate our method on a series of simulated tasks and real-world robotic manipulation problems, demonstrating substantial improvement over prior methods both in terms of task complexity and sample efficiency.

arXiv commentInternational Conference on Machine Learning (ICML), 2016, to appear

Cited by in corpus (148)

• On the Opportunities and Risks of Foundation Models
• How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learned
• Deep Reinforcement Learning: An Overview
• Deep reinforcement learning from human preferences
• Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards
• An Algorithmic Perspective on Imitation Learning
• Generative Adversarial Imitation Learning
• One-Shot Visual Imitation Learning via Meta-Learning
• A Survey of Deep RL and IL for Autonomous Driving Policy Learning
• One-Shot Imitation Learning
• A Connection between Generative Adversarial Networks, Inverse Reinforcement Learning, and Energy-Based Models
• Third-Person Imitation Learning
• Variational Discriminator Bottleneck: Improving Imitation Learning, Inverse RL, and GANs by Constraining Information Flow
• InfoGAIL: Interpretable Imitation Learning from Visual Demonstrations
• Scalable agent alignment via reward modeling: a research direction
• Data-efficient Deep Reinforcement Learning for Dexterous Manipulation
• Reinforcement and Imitation Learning for Diverse Visuomotor Skills
• A Review of Robot Learning for Manipulation: Challenges, Representations, and Algorithms
• Generative Adversarial Imitation from Observation
• Efficient Exploration via State Marginal Matching
• Deeply AggreVaTeD: Differentiable Imitation Learning for Sequential Prediction
• Learning Plannable Representations with Causal InfoGAN
• Universal Planning Networks
• A Survey of Deep Network Solutions for Learning Control in Robotics: From Reinforcement to Imitation
• When Humans Aren't Optimal: Robots that Collaborate with Risk-Aware Humans
• Reward-rational (implicit) choice: A unifying formalism for reward learning
• Off-road Autonomous Vehicles Traversability Analysis and Trajectory Planning Based on Deep Inverse Reinforcement Learning
• Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperation
• SQIL: Imitation Learning via Reinforcement Learning with Sparse Rewards
• Scaling data-driven robotics with reward sketching and batch reinforcement learning
• The MineRL 2019 Competition on Sample Efficient Reinforcement Learning using Human Priors
• Safe Imitation Learning via Fast Bayesian Reward Inference from Preferences
• Path Integral Networks: End-to-End Differentiable Optimal Control
• Primal Wasserstein Imitation Learning
• Reinforcement Learning for Robotic Manipulation using Simulated Locomotion Demonstrations
• Few-Shot Goal Inference for Visuomotor Learning and Planning
• Multi-Modal Imitation Learning from Unstructured Demonstrations using Generative Adversarial Nets
• Imitation from Observation: Learning to Imitate Behaviors from Raw Video via Context Translation
• Recurrent Deterministic Policy Gradient Method for Bipedal Locomotion on Rough Terrain Challenge
• Goal-conditioned Imitation Learning
• Deep Reinforcement Learning from Policy-Dependent Human Feedback
• Better-than-Demonstrator Imitation Learning via Automatically-Ranked Demonstrations
• Toward Diverse Text Generation with Inverse Reinforcement Learning
• Where Do You Think You're Going?: Inferring Beliefs about Dynamics from Behavior
• Supervising strong learners by amplifying weak experts
• Longitudinal Face Modeling via Temporal Deep Restricted Boltzmann Machines
• Learning Self-Imitating Diverse Policies
• Inverse reinforcement learning for video games
• Virtual-Taobao: Virtualizing Real-world Online Retail Environment for Reinforcement Learning
• Paraphrase Generation with Deep Reinforcement Learning
• Feature Expansive Reward Learning: Rethinking Human Input
• Learning Temporal Strategic Relationships using Generative Adversarial Imitation Learning
• Wasserstein Distance guided Adversarial Imitation Learning with Reward Shape Exploration
• Automated Lane Change Strategy using Proximal Policy Optimization-based Deep Reinforcement Learning
• Robot eye-hand coordination learning by watching human demonstrations: a task function approximation approach
• Calibration of Human Driving Behavior and Preference Using Naturalistic Traffic Data
• Reinforcement Learning on Web Interfaces Using Workflow-Guided Exploration
• Revisiting Maximum Entropy Inverse Reinforcement Learning: New Perspectives and Algorithms
• Recent Advances in Leveraging Human Guidance for Sequential Decision-Making Tasks
• Multi-task Maximum Entropy Inverse Reinforcement Learning
• Cycle-of-Learning for Autonomous Systems from Human Interaction
• Synthetic Observational Health Data with GANs: from slow adoption to a boom in medical research and ultimately digital twins?
• Energy-Based Imitation Learning
• Imitation Learning: Progress, Taxonomies and Challenges
• Generative Adversarial Imitation Learning for End-to-End Autonomous Driving on Urban Environments
• Quantifying Differences in Reward Functions
• Grounded Relational Inference: Domain Knowledge Driven Explainable Autonomous Driving
• Incorporating Human Domain Knowledge into Large Scale Cost Function Learning
• Model-Based Inverse Reinforcement Learning from Visual Demonstrations
• Offline Meta-Reinforcement Learning with Online Self-Supervision
• Example-Driven Model-Based Reinforcement Learning for Solving Long-Horizon Visuomotor Tasks
• Learning Deep Mean Field Games for Modeling Large Population Behavior
• Learning Reward Models for Cooperative Trajectory Planning with Inverse Reinforcement Learning and Monte Carlo Tree Search
• OffWorld Gym: open-access physical robotics environment for real-world reinforcement learning benchmark and research
• Optimal Path Planning of Autonomous Marine Vehicles in Stochastic Dynamic Ocean Flows using a GPU-Accelerated Algorithm
• Energy-Based Sequence GANs for Recommendation and Their Connection to Imitation Learning
• Adversarial Soft Advantage Fitting: Imitation Learning without Policy Optimization
• Learning Self-Correctable Policies and Value Functions from Demonstrations with Negative Sampling
• Lifelong Robotic Reinforcement Learning by Retaining Experiences
• Augmenting GAIL with BC for sample efficient imitation learning
• Inverse Constrained Reinforcement Learning
• Learning A Single Network for Scale-Arbitrary Super-Resolution
• Adversarial Imitation via Variational Inverse Reinforcement Learning
• Decision Making for Autonomous Driving via Augmented Adversarial Inverse Reinforcement Learning
• Autonomous Assessment of Demonstration Sufficiency via Bayesian Inverse Reinforcement Learning
• Visual Forecasting by Imitating Dynamics in Natural Sequences
• Injection Optimization at Particle Accelerators via Reinforcement Learning: From Simulation to Real-World Application
• Calibration of Shared Equilibria in General Sum Partially Observable Markov Games
• Model-based Behavioral Cloning with Future Image Similarity Learning
• Learning "What-if" Explanations for Sequential Decision-Making
• Weakly-Supervised Reinforcement Learning for Controllable Behavior
• Density Matching Reward Learning
• AliExpress Learning-To-Rank: Maximizing Online Model Performance without Going Online
• Reinforcement Learning with Videos: Combining Offline Observations with Interaction
• CrowdTransfer: Enabling Crowd Knowledge Transfer in AIoT Community
• PsiPhi-Learning: Reinforcement Learning with Demonstrations using Successor Features and Inverse Temporal Difference Learning
• Visual Adversarial Imitation Learning using Variational Models
• Deep Learning in Robotics: A Review of Recent Research
• Seeing All the Angles: Learning Multiview Manipulation Policies for Contact-Rich Tasks from Demonstrations
• VILD: Variational Imitation Learning with Diverse-quality Demonstrations
• Inverse Reinforcement Learning with Conditional Choice Probabilities
• Learning Multi-Task Transferable Rewards via Variational Inverse Reinforcement Learning
• Learning from Ambiguous Demonstrations with Self-Explanation Guided Reinforcement Learning
• A Pragmatic Look at Deep Imitation Learning
• Human-guided Robot Behavior Learning: A GAN-assisted Preference-based Reinforcement Learning Approach
• Stealing Deep Reinforcement Learning Models for Fun and Profit
• Generative Adversarial Imitation Learning with Neural Networks: Global Optimality and Convergence Rate
• What Would You Do? Acting by Learning to Predict
• Imitation Learning from Pixel-Level Demonstrations by HashReward
• On Value Discrepancy of Imitation Learning
• MobILE: Model-Based Imitation Learning From Observation Alone
• Language Conditioned Imitation Learning over Unstructured Data
• Curriculum Design for Teaching via Demonstrations: Theory and Applications
• Hybrid Reinforcement Learning with Expert State Sequences
• Inverse Quadratic Optimal Control for Discrete-Time Linear Systems
• Generalized Maximum Causal Entropy for Inverse Reinforcement Learning
• Learning a Decision Module by Imitating Driver's Control Behaviors
• Follow the Object: Curriculum Learning for Manipulation Tasks with Imagined Goals
• Regularized Inverse Reinforcement Learning
• Inferring and Conveying Intentionality: Beyond Numerical Rewards to Logical Intentions
• Smooth Imitation Learning via Smooth Costs and Smooth Policies
• Learning Navigation Costs from Demonstration with Semantic Observations
• Learning Equality Constraints for Motion Planning on Manifolds
• Fitting a Linear Control Policy to Demonstrations with a Kalman Constraint
• Bellman Gradient Iteration for Inverse Reinforcement Learning
• Inverse Reinforce Learning with Nonparametric Behavior Clustering
• Imitation Learning for Fashion Style Based on Hierarchical Multimodal Representation
• SQUIRL: Robust and Efficient Learning from Video Demonstration of Long-Horizon Robotic Manipulation Tasks
• Continuous-Time Inverse Quadratic Optimal Control Problem
• Deep PQR: Solving Inverse Reinforcement Learning using Anchor Actions
• Learning to Optimize via Wasserstein Deep Inverse Optimal Control
• Provably Efficient Generative Adversarial Imitation Learning for Online and Offline Setting with Linear Function Approximation
• Task Transfer by Preference-Based Cost Learning
• Motion Reasoning for Goal-Based Imitation Learning
• Curriculum Offline Imitation Learning
• Objective-aware Traffic Simulation via Inverse Reinforcement Learning
• Cost Functions for Robot Motion Style
• Inverse Reinforcement Learning in a Continuous State Space with Formal Guarantees
• Generative adversarial training of product of policies for robust and adaptive movement primitives
• Automatic Curricula via Expert Demonstrations
• Imitation Learning by Reinforcement Learning
• Learning Data-Driven Objectives to Optimize Interactive Systems
• Training Transition Policies via Distribution Matching for Complex Tasks
• Versatile Inverse Reinforcement Learning via Cumulative Rewards
• Sample Efficient Imitation Learning via Reward Function Trained in Advance
• Integration of Imitation Learning using GAIL and Reinforcement Learning using Task-achievement Rewards via Probabilistic Graphical Model
• Variational Reward Estimator Bottleneck: Learning Robust Reward Estimator for Multi-Domain Task-Oriented Dialog
• Stochastic Inverse Reinforcement Learning