Count-Based Temperature Scheduling for Maximum Entropy Reinforcement Learning

Abstract

Maximum Entropy Reinforcement Learning (MaxEnt RL) algorithms such as Soft Q-Learning (SQL) and Soft Actor-Critic trade off reward and policy entropy, which has the potential to improve training stability and robustness. Most MaxEnt RL methods, however, use a constant tradeoff coefficient (temperature), contrary to the intuition that the temperature should be high early in training to avoid overfitting to noisy value estimates and decrease later in training as we increasingly trust high value estimates to truly lead to good rewards. Moreover, our confidence in value estimates is state-dependent, increasing every time we use more evidence to update an estimate. In this paper, we present a simple state-based temperature scheduling approach, and instantiate it for SQL as Count-Based Soft Q-Learning (CBSQL). We evaluate our approach on a toy domain as well as in several Atari 2600 domains and show promising results.

References in corpus (12)

• Trust Region Policy Optimization
• Soft Actor-Critic Algorithms and Applications
• Dueling Network Architectures for Deep Reinforcement Learning
• Noisy Networks for Exploration
• Rainbow: Combining Improvements in Deep Reinforcement Learning
• Reinforcement Learning with Deep Energy-Based Policies
• Deep Reinforcement Learning that Matters
• Count-Based Exploration with Neural Density Models
• Bridging the Gap Between Value and Policy Based Reinforcement Learning
• Soft Actor-Critic for Discrete Action Settings
• Maximum Entropy RL (Provably) Solves Some Robust RL Problems
• Approximate Inference and Stochastic Optimal Control