Variational Discriminator Bottleneck: Improving Imitation Learning, Inverse RL, and GANs by Constraining Information Flow

Abstract

Adversarial learning methods have been proposed for a wide range of applications, but the training of adversarial models can be notoriously unstable. Effectively balancing the performance of the generator and discriminator is critical, since a discriminator that achieves very high accuracy will produce relatively uninformative gradients. In this work, we propose a simple and general technique to constrain information flow in the discriminator by means of an information bottleneck. By enforcing a constraint on the mutual information between the observations and the discriminator's internal representation, we can effectively modulate the discriminator's accuracy and maintain useful and informative gradients. We demonstrate that our proposed variational discriminator bottleneck (VDB) leads to significant improvements across three distinct application areas for adversarial learning algorithms. Our primary evaluation studies the applicability of the VDB to imitation learning of dynamic continuous control skills, such as running. We show that our method can learn such skills directly from \emph{raw} video demonstrations, substantially outperforming prior adversarial imitation learning methods. The VDB can also be combined with adversarial inverse reinforcement learning to learn parsimonious reward functions that can be transferred and re-optimized in new settings. Finally, we demonstrate that VDB can train GANs more effectively for image generation, improving upon a number of prior stabilization methods.

References in corpus (16)

• Auto-Encoding Variational Bayes
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Spectral Normalization for Generative Adversarial Networks
• Trust Region Policy Optimization
• Improved Techniques for Training GANs
• Energy-based Generative Adversarial Network
• Generating Videos with Scene Dynamics
• Which Training Methods for GANs do actually Converge?
• BEGAN: Boundary Equilibrium Generative Adversarial Networks
• Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization
• Deep Variational Information Bottleneck
• Adversarial Autoencoders
• Mode Regularized Generative Adversarial Networks
• A Connection between Generative Adversarial Networks, Inverse Reinforcement Learning, and Energy-Based Models
• Amortised MAP Inference for Image Super-resolution
• Learning human behaviors from motion capture by adversarial imitation

Cited by in corpus (29)

• Multimodal Information Bottleneck: Learning Minimal Sufficient Unimodal and Multimodal Representations
• Adversarial Graph Augmentation to Improve Graph Contrastive Learning
• Graph Information Bottleneck
• IB-GAN: Disentangled Representation Learning with Information Bottleneck Generative Adversarial Networks
• Simple and Effective VAE Training with Calibrated Decoders
• Feature Quantization Improves GAN Training
• Bias-Eliminated Semantic Refinement for Any-Shot Learning
• The Convex Information Bottleneck Lagrangian
• Compensation for undefined behaviors during robot task execution by switching controllers depending on embedded dynamics in RNN
• Wasserstein Distance guided Adversarial Imitation Learning with Reward Shape Exploration
• Lipschitzness Is All You Need To Tame Off-policy Generative Adversarial Imitation Learning
• Learning the Redundancy-free Features for Generalized Zero-Shot Object Recognition
• Learning to Learn with Variational Information Bottleneck for Domain Generalization
• Training Generative Adversarial Networks Via Turing Test
• Positive-Unlabeled Reward Learning
• Neuron Campaign for Initialization Guided by Information Bottleneck Theory
• The MAGICAL Benchmark for Robust Imitation
• Multi-Task Variational Information Bottleneck
• MetaInfoNet: Learning Task-Guided Information for Sample Reweighting
• VILD: Variational Imitation Learning with Diverse-quality Demonstrations
• Preventing Imitation Learning with Adversarial Policy Ensembles
• Off-Policy Adversarial Inverse Reinforcement Learning
• Imitation Learning from Pixel-Level Demonstrations by HashReward
• Inclusive GAN: Improving Data and Minority Coverage in Generative Models
• To Beta or Not To Beta: Information Bottleneck for DigitaL Image Forensics
• Learning Compact Reward for Image Captioning
• Towards Learning to Imitate from a Single Video Demonstration
• Mature GAIL: Imitation Learning for Low-level and High-dimensional Input using Global Encoder and Cost Transformation
• NEARL: Non-Explicit Action Reinforcement Learning for Robotic Control