Lipschitzness Is All You Need To Tame Off-policy Generative Adversarial Imitation Learning

Abstract

Despite the recent success of reinforcement learning in various domains, these approaches remain, for the most part, deterringly sensitive to hyper-parameters and are often riddled with essential engineering feats allowing their success. We consider the case of off-policy generative adversarial imitation learning, and perform an in-depth review, qualitative and quantitative, of the method. We show that forcing the learned reward function to be local Lipschitz-continuous is a sine qua non condition for the method to perform well. We then study the effects of this necessary condition and provide several theoretical results involving the local Lipschitzness of the state-value function. We complement these guarantees with empirical evidence attesting to the strong positive effect that the consistent satisfaction of the Lipschitzness constraint on the reward has on imitation performance. Finally, we tackle a generic pessimistic reward preconditioning add-on spawning a large class of reward shaping methods, which makes the base method it is plugged into provably more robust, as shown in several additional theoretical guarantees. We then discuss these through a fine-grained lens and share our insights. Crucially, the guarantees derived and reported in this work are valid for any reward satisfying the Lipschitzness condition, nothing is specific to imitation. As such, these may be of independent interest.

Accepted for publication in Machine Learning 2022

References in corpus (22)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Energy-based Generative Adversarial Network
• Solving Rubik's Cube with a Robot Hand
• On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
• Learning to Navigate in Complex Environments
• Towards Principled Methods for Training Generative Adversarial Networks
• Reinforcement Learning with Unsupervised Auxiliary Tasks
• Connecting Generative Adversarial Networks and Actor-Critic Methods
• Learning to Walk in the Real World with Minimal Human Effort
• Loss is its own Reward: Self-Supervision for Reinforcement Learning
• Measuring the Algorithmic Efficiency of Neural Networks
• An LSTM-Based Autonomous Driving Model Using Waymo Open Dataset
• Waymo Driverless Car Data Analysis and Driving Modeling using CNN and LSTM
• Towards Characterizing Divergence in Deep Q-Learning
• Understanding Multi-Step Deep Reinforcement Learning: A Systematic Study of the DQN Target
• A New Algorithm for Non-stationary Contextual Bandits: Efficient, Optimal, and Parameter-free
• Diagnosing Bottlenecks in Deep Q-learning Algorithms
• Observational Learning by Reinforcement Learning
• Random Expert Distillation: Imitation Learning via Expert Policy Support Estimation
• Positive-Unlabeled Reward Learning
• Online Learning in Markov Decision Processes with Adversarially Chosen Transition Probability Distributions

Cited by in corpus (3)

• What Matters for Adversarial Imitation Learning?
• Visual Adversarial Imitation Learning using Variational Models
• Smooth Imitation Learning via Smooth Costs and Smooth Policies