State Alignment-based Imitation Learning

Abstract

Consider an imitation learning problem that the imitator and the expert have different dynamics models. Most of the current imitation learning methods fail because they focus on imitating actions. We propose a novel state alignment-based imitation learning method to train the imitator to follow the state sequences in expert demonstrations as much as possible. The state alignment comes from both local and global perspectives and we combine them into a reinforcement learning framework by a regularized policy update objective. We show the superiority of our method on standard imitation learning settings and imitation learning settings where the expert and imitator have different dynamics models.

References in corpus (15)

• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Trust Region Policy Optimization
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• Deep Q-learning from Demonstrations
• Data-Efficient Hierarchical Reinforcement Learning
• Generative Adversarial Imitation from Observation
• Efficient Exploration via State Marginal Matching
• Observe and Look Further: Achieving Consistent Performance on Atari
• Near-Optimal Representation Learning for Hierarchical Reinforcement Learning
• Coordinated Multi-Agent Imitation Learning
• Imitating Latent Policies from Observation
• Imitation Learning as $f$-Divergence Minimization
• Wasserstein Adversarial Imitation Learning
• Sample-Efficient Imitation Learning via Generative Adversarial Nets
• Provably Efficient Imitation Learning from Observation Alone

Cited by in corpus (13)

• Transfer Learning in Deep Reinforcement Learning: A Survey
• Primal Wasserstein Imitation Learning
• Energy-Based Imitation Learning
• XIRL: Cross-embodiment Inverse Reinforcement Learning
• Strictly Batch Imitation Learning by Energy-based Distribution Matching
• Correspondence learning between morphologically different robots via task demonstrations
• f-IRL: Inverse Reinforcement Learning via State Marginal Matching
• Estimating Q(s,s') with Deep Deterministic Dynamics Gradients
• Robust Inverse Reinforcement Learning under Transition Dynamics Mismatch
• Cross-Domain Imitation Learning via Optimal Transport
• Unsupervised Domain Adaptation with Dynamics-Aware Rewards in Reinforcement Learning
• Learning Time-Invariant Reward Functions through Model-Based Inverse Reinforcement Learning
• HILONet: Hierarchical Imitation Learning from Non-Aligned Observations