SOAC: The Soft Option Actor-Critic Architecture

Abstract

The option framework has shown great promise by automatically extracting temporally-extended sub-tasks from a long-horizon task. Methods have been proposed for concurrently learning low-level intra-option policies and high-level option selection policy. However, existing methods typically suffer from two major challenges: ineffective exploration and unstable updates. In this paper, we present a novel and stable off-policy approach that builds on the maximum entropy model to address these challenges. Our approach introduces an information-theoretical intrinsic reward for encouraging the identification of diverse and effective options. Meanwhile, we utilize a probability inference model to simplify the optimization problem as fitting optimal trajectories. Experimental results demonstrate that our approach significantly outperforms prior on-policy and off-policy methods in a range of Mujoco benchmark tasks while still providing benefits for transfer learning. In these tasks, our approach learns a diverse set of options, each of whose state-action space has strong coherence.

References in corpus (20)

• Adam: A Method for Stochastic Optimization
• Playing Atari with Deep Reinforcement Learning
• Addressing Function Approximation Error in Actor-Critic Methods
• Soft Actor-Critic Algorithms and Applications
• Dota 2 with Large Scale Deep Reinforcement Learning
• Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation
• Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review
• Deep Variational Information Bottleneck
• Optimal control as a graphical model inference problem
• Soft Actor-Critic for Discrete Action Settings
• Equivalence Between Policy Gradients and Soft Q-Learning
• Dynamics-Aware Unsupervised Discovery of Skills
• Learning Parameterized Skills
• Multi-Level Discovery of Deep Options
• InfoBot: Transfer and Exploration via the Information Bottleneck
• Hierarchical Reinforcement Learning with Advantage-Based Auxiliary Rewards
• Learning Nearly Decomposable Value Functions Via Communication Minimization
• Reinforcement Learning with Competitive Ensembles of Information-Constrained Primitives
• Using Options and Covariance Testing for Long Horizon Off-Policy Policy Evaluation
• Hierarchical Reinforcement Learning via Advantage-Weighted Information Maximization