Utilizing Skipped Frames in Action Repeats via Pseudo-Actions

Abstract

In many deep reinforcement learning settings, when an agent takes an action, it repeats the same action a predefined number of times without observing the states until the next action-decision point. This technique of action repetition has several merits in training the agent, but the data between action-decision points (i.e., intermediate frames) are, in effect, discarded. Since the amount of training data is inversely proportional to the interval of action repeats, they can have a negative impact on the sample efficiency of training. In this paper, we propose a simple but effective approach to alleviate to this problem by introducing the concept of pseudo-actions. The key idea of our method is making the transition between action-decision points usable as training data by considering pseudo-actions. Pseudo-actions for continuous control tasks are obtained as the average of the action sequence straddling an action-decision point. For discrete control tasks, pseudo-actions are computed from learned action embeddings. This method can be combined with any model-free reinforcement learning algorithm that involves the learning of Q-functions. We demonstrate the effectiveness of our approach on both continuous and discrete control tasks in OpenAI Gym.

arXiv commentDeep Reinforcement Learning Workshop, NeurIPS 2020

References in corpus (12)

• Playing Atari with Deep Reinforcement Learning
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Soft Actor-Critic Algorithms and Applications
• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• Noisy Networks for Exploration
• DeepMind Control Suite
• Rainbow: Combining Improvements in Deep Reinforcement Learning
• Learning Latent Dynamics for Planning from Pixels
• Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels
• Learning to Repeat: Fine Grained Action Repetition for Deep Reinforcement Learning
• Multi-task Deep Reinforcement Learning with PopArt
• On Inductive Biases in Deep Reinforcement Learning