Domain-Adversarial and Conditional State Space Model for Imitation Learning
arXiv:2001.11628
Abstract
State representation learning (SRL) in partially observable Markov decision processes has been studied to learn abstract features of data useful for robot control tasks. For SRL, acquiring domain-agnostic states is essential for achieving efficient imitation learning. Without these states, imitation learning is hampered by domain-dependent information useless for control. However, existing methods fail to remove such disturbances from the states when the data from experts and agents show large domain shifts. To overcome this issue, we propose a domain-adversarial and conditional state space model (DAC-SSM) that enables control systems to obtain domain-agnostic and task- and dynamics-aware states. DAC-SSM jointly optimizes the state inference, observation reconstruction, forward dynamics, and reward models. To remove domain-dependent information from the states, the model is trained with domain discriminators in an adversarial manner, and the reconstruction is conditioned on domain labels. We experimentally evaluated the model predictive control performance via imitation learning for continuous control of sparse reward tasks in simulators and compared it with the performance of the existing SRL method. The agents from DAC-SSM achieved performance comparable to experts and more than twice the baselines. We conclude domain-agnostic states are essential for imitation learning that has large domain shifts and can be obtained using DAC-SSM.
arXiv commentPublished at IROS 2020
References in corpus (17)
- Deep Domain Confusion: Maximizing for Domain Invariance
- Domain Separation Networks
- DeepMind Control Suite
- Recurrent World Models Facilitate Policy Evolution
- Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review
- Learning Latent Dynamics for Planning from Pixels
- State Representation Learning for Control: An Overview
- Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models
- Learning human behaviors from motion capture by adversarial imitation
- Third-Person Imitation Learning
- Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learning
- Variational Inference MPC for Bayesian Model-based Reinforcement Learning
- Directed-Info GAIL: Learning Hierarchical Policies from Unsegmented Demonstrations using Directed Information
- Learning Dexterous Manipulation for a Soft Robotic Hand from Human Demonstration
- Multi-objective Model-based Policy Search for Data-efficient Learning with Sparse Rewards
- PlaNet of the Bayesians: Reconsidering and Improving Deep Planning Network by Incorporating Bayesian Inference
- To Follow or not to Follow: Selective Imitation Learning from Observations