Using Offline Data to Speed Up Reinforcement Learning in Procedurally Generated Environments

Abstract

One of the key challenges of Reinforcement Learning (RL) is the ability of agents to generalise their learned policy to unseen settings. Moreover, training RL agents requires large numbers of interactions with the environment. Motivated by the recent success of Offline RL and Imitation Learning (IL), we conduct a study to investigate whether agents can leverage offline data in the form of trajectories to improve the sample-efficiency in procedurally generated environments. We consider two settings of using IL from offline data for RL: (1) pre-training a policy before online RL training and (2) concurrently training a policy with online RL and IL from offline data. We analyse the impact of the quality (optimality of trajectories) and diversity (number of trajectories and covered level) of available offline trajectories on the effectiveness of both approaches. Across four well-known sparse reward tasks in the MiniGrid environment, we find that using IL for pre-training and concurrently during online RL training both consistently improve the sample-efficiency while converging to optimal policies. Furthermore, we show that pre-training a policy from as few as two trajectories can make the difference between learning an optimal policy at the end of online training and not learning at all. Our findings motivate the widespread adoption of IL for pre-training and concurrent IL in procedurally generated environments whenever offline trajectories are available or can be generated.

Initially presented at the Adaptive and Learning Agents Workshop (ALA) at the AAMAS conference 2023; the current extended version was accepted at Neurocomputing journal

References in corpus (14)

• Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards
• An Algorithmic Perspective on Imitation Learning
• Offline Reinforcement Learning with Implicit Q-Learning
• BeBold: Exploration Beyond the Boundary of Explored Regions
• Pretraining in Deep Reinforcement Learning: A Survey
• Measuring Sample Efficiency and Generalization in Reinforcement Learning Benchmarks: NeurIPS 2020 Procgen Benchmark
• I'm sorry Dave, I'm afraid I can't do that, Deep Q-learning from forbidden action
• Hybrid RL: Using Both Offline and Online Data Can Make RL Efficient
• Decoupled Reinforcement Learning to Stabilise Intrinsically-Motivated Exploration
• Policy Expansion for Bridging Offline-to-Online Reinforcement Learning
• Boosting Offline Reinforcement Learning via Data Rebalancing
• Leveraging Fully Observable Policies for Learning under Partial Observability
• Curriculum Offline Imitation Learning
• Can Active Sampling Reduce Causal Confusion in Offline Reinforcement Learning?