ACL-QL: Adaptive Conservative Level in Q-Learning for Offline Reinforcement Learning

Abstract

Offline Reinforcement Learning (RL), which operates solely on static datasets without further interactions with the environment, provides an appealing alternative to learning a safe and promising control policy. The prevailing methods typically learn a conservative policy to mitigate the problem of Q-value overestimation, but it is prone to overdo it, leading to an overly conservative policy. Moreover, they optimize all samples equally with fixed constraints, lacking the nuanced ability to control conservative levels in a fine-grained manner. Consequently, this limitation results in a performance decline. To address the above two challenges in a united way, we propose a framework, Adaptive Conservative Level in Q-Learning (ACL-QL), which limits the Q-values in a mild range and enables adaptive control on the conservative level over each state-action pair, i.e., lifting the Q-values more for good transitions and less for bad transitions. We theoretically analyze the conditions under which the conservative level of the learned Q-function can be limited in a mild range and how to optimize each transition adaptively. Motivated by the theoretical analysis, we propose a novel algorithm, ACL-QL, which uses two learnable adaptive weight functions to control the conservative level over each transition. Subsequently, we design a monotonicity loss and surrogate losses to train the adaptive weight functions, Q-function, and policy network alternatively. We evaluate ACL-QL on the commonly used D4RL benchmark and conduct extensive ablation studies to illustrate the effectiveness and state-of-the-art performance compared to existing offline DRL baselines.

19 pages, 4 figures, IEEE Transactions on Neural Networks and Learning Systems (2024)

References in corpus (32)

• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Deep Learning using Rectified Linear Units (ReLU)
• Addressing Function Approximation Error in Actor-Critic Methods
• Prioritized Experience Replay
• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• Solving Rubik's Cube with a Robot Hand
• OpenAI Gym
• Conservative Q-Learning for Offline Reinforcement Learning
• Decision Transformer: Reinforcement Learning via Sequence Modeling
• Deep Exploration via Bootstrapped DQN
• Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction
• A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems
• D4RL: Datasets for Deep Data-Driven Reinforcement Learning
• Off-Policy Deep Reinforcement Learning without Exploration
• Behavior Regularized Offline Reinforcement Learning
• MOPO: Model-based Offline Policy Optimization
• Reproducibility of Benchmarked Deep Reinforcement Learning Tasks for Continuous Control
• Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning
• A Minimalist Approach to Offline Reinforcement Learning
• MOReL : Model-Based Offline Reinforcement Learning
• Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog
• Offline Reinforcement Learning with Implicit Q-Learning
• Feature Control as Intrinsic Motivation for Hierarchical Reinforcement Learning
• An Optimistic Perspective on Offline Reinforcement Learning
• Critic Regularized Regression
• Pseudo-Rehearsal: Achieving Deep Reinforcement Learning without Catastrophic Forgetting
• AlgaeDICE: Policy Gradient from Arbitrary Experience
• Towards Characterizing Divergence in Deep Q-Learning
• Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimization
• BAIL: Best-Action Imitation Learning for Batch Deep Reinforcement Learning
• Conservative Offline Distributional Reinforcement Learning
• Demystifying Reinforcement Learning in Time-Varying Systems