Implicit Behavioral Cloning
arXiv:2109.00137
Abstract
We find that across a wide range of robot policy learning scenarios, treating supervised policy learning with an implicit model generally performs better, on average, than commonly used explicit models. We present extensive experiments on this finding, and we provide both intuitive insight and theoretical arguments distinguishing the properties of implicit models compared to their explicit counterparts, particularly with respect to approximating complex, potentially discontinuous and multi-valued (set-valued) functions. On robotic policy learning tasks we show that implicit behavioral cloning policies with energy-based models (EBM) often outperform common explicit (Mean Square Error, or Mixture Density) behavioral cloning policies, including on tasks with high-dimensional action spaces and visual image inputs. We find these policies provide competitive results or outperform state-of-the-art offline reinforcement learning methods on the challenging human-expert tasks from the D4RL benchmark suite, despite using no reward information. In the real world, robots with implicit policies can learn complex and remarkably subtle behaviors on contact-rich tasks from human demonstrations, including tasks with high combinatorial complexity and tasks requiring 1mm precision.
References in corpus (19)
- DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skills
- Recurrent World Models Facilitate Policy Evolution
- Reinforcement Learning with Deep Energy-Based Policies
- AMP: Adversarial Motion Priors for Stylized Physics-Based Character Control
- D4RL: Datasets for Deep Data-Driven Reinforcement Learning
- Deep Equilibrium Models
- How Neural Networks Extrapolate: From Feedforward to Graph Neural Networks
- How to Train Your Energy-Based Models
- Imitation Learning via Off-Policy Distribution Matching
- Autoregressive Energy Machines
- On the Sample Complexity of Stability Constrained Imitation Learning
- Residual Energy-Based Models for Text Generation
- ContactNets: Learning Discontinuous Contact Dynamics with Smooth, Implicit Representations
- Energy-Based Imitation Learning
- Concept Learning with Energy-Based Models
- Provable Representation Learning for Imitation with Contrastive Fourier Features
- S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learning
- A very accurate method to approximate discontinuous functions with a finite number of discontinuities
- Gradient penalty from a maximum margin perspective