Variational Empowerment as Representation Learning for Goal-Based Reinforcement Learning

Abstract

Learning to reach goal states and learning diverse skills through mutual information (MI) maximization have been proposed as principled frameworks for self-supervised reinforcement learning, allowing agents to acquire broadly applicable multitask policies with minimal reward engineering. Starting from a simple observation that the standard goal-conditioned RL (GCRL) is encapsulated by the optimization objective of variational empowerment, we discuss how GCRL and MI-based RL can be generalized into a single family of methods, which we name variational GCRL (VGCRL), interpreting variational MI maximization, or variational empowerment, as representation learning methods that acquire functionally-aware state representations for goal reaching. This novel perspective allows us to: (1) derive simple but unexplored variants of GCRL to study how adding small representation capacity can already expand its capabilities; (2) investigate how discriminator function capacity and smoothness determine the quality of discovered skills, or latent goals, through modifying latent dimensionality and applying spectral normalization; (3) adapt techniques such as hindsight experience replay (HER) from GCRL to MI-based RL; and lastly, (4) propose a novel evaluation metric, named latent goal reaching (LGR), for comparing empowerment algorithms with different choices of latent dimensionality and discriminator parameterization. Through principled mathematical derivations and careful experimental studies, our work lays a novel foundation from which to evaluate, analyze, and develop representation learning techniques in goal-based RL.

arXiv commentAccepted at International Conference on Machine Learning (ICML) 2021

References in corpus (17)

• Playing Atari with Deep Reinforcement Learning
• Trust Region Policy Optimization
• Soft Actor-Critic Algorithms and Applications
• QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation
• Hindsight Experience Replay
• Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning
• Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables
• Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Research
• Stochastic Neural Networks for Hierarchical Reinforcement Learning
• Provably Efficient Maximum Entropy Exploration
• Efficient Exploration via State Marginal Matching
• Skew-Fit: State-Covering Self-Supervised Reinforcement Learning
• Near-Optimal Representation Learning for Hierarchical Reinforcement Learning
• Unsupervised Meta-Learning for Reinforcement Learning
• Unsupervised Control Through Non-Parametric Discriminative Rewards
• Learning Latent Plans from Play
• Particle Value Functions