Counterfactual experience augmented off-policy reinforcement learning

Abstract

Reinforcement learning control algorithms face significant challenges due to out-of-distribution and inefficient exploration problems. While model-based reinforcement learning enhances the agent's reasoning and planning capabilities by constructing virtual environments, training such virtual environments can be very complex. In order to build an efficient inference model and enhance the representativeness of learning data, we propose the Counterfactual Experience Augmentation (CEA) algorithm. CEA leverages variational autoencoders to model the dynamic patterns of state transitions and introduces randomness to model non-stationarity. This approach focuses on expanding the learning data in the experience pool through counterfactual inference and performs exceptionally well in environments that follow the bisimulation assumption. Environments with bisimulation properties are usually represented by discrete observation and action spaces, we propose a sampling method based on maximum kernel density estimation entropy to extend CEA to various environments. By providing reward signals for counterfactual state transitions based on real information, CEA constructs a complete counterfactual experience to alleviate the out-of-distribution problem of the learning data, and outperforms general SOTA algorithms in environments with difference properties. Finally, we discuss the similarities, differences and properties of generated counterfactual experiences and real experiences. The code is available at https://github.com/Aegis1863/CEA.

Accepted by Neurocomputing, https://doi.org/10.1016/j.neucom.2025.130017

References in corpus (14)

• Prioritized Experience Replay
• Soft Actor-Critic Algorithms and Applications
• CURL: Contrastive Unsupervised Representations for Reinforcement Learning
• Soft Actor-Critic for Discrete Action Settings
• Multi-agent Reinforcement Learning for Cooperative Lane Changing of Connected and Autonomous Vehicles in Mixed Traffic
• Reinforcement Learning in Non-Stationary Environments
• Generative Adversarial Imitation from Observation
• Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learning
• Automatic Data Augmentation for Generalization in Deep Reinforcement Learning
• Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning
• Sample-Efficient Reinforcement Learning via Counterfactual-Based Data Augmentation
• Quantifying the Impact of Non-Stationarity in Reinforcement Learning-Based Traffic Signal Control
• Conditional Image Generation by Conditioning Variational Auto-Encoders
• Enhancing Autonomous Vehicle Training with Language Model Integration and Critical Scenario Generation