Learning to Repeat: Fine Grained Action Repetition for Deep Reinforcement Learning

Abstract

Reinforcement Learning algorithms can learn complex behavioral patterns for sequential decision making tasks wherein an agent interacts with an environment and acquires feedback in the form of rewards sampled from it. Traditionally, such algorithms make decisions, i.e., select actions to execute, at every single time step of the agent-environment interactions. In this paper, we propose a novel framework, Fine Grained Action Repetition (FiGAR), which enables the agent to decide the action as well as the time scale of repeating it. FiGAR can be used for improving any Deep Reinforcement Learning algorithm which maintains an explicit policy estimate by enabling temporal abstractions in the action space. We empirically demonstrate the efficacy of our framework by showing performance improvements on top of three policy search algorithms in different domains: Asynchronous Advantage Actor Critic in the Atari 2600 domain, Trust Region Policy Optimization in Mujoco domain and Deep Deterministic Policy Gradients in the TORCS car racing domain.

24 pages

Cited by in corpus (14)

• Deep Reinforcement Learning: An Overview
• A Survey of Deep Reinforcement Learning in Video Games
• Automated Reinforcement Learning (AutoRL): A Survey and Open Problems
• Multi-Level Discovery of Deep Options
• The Atari Grand Challenge Dataset
• Learning to Factor Policies and Action-Value Functions: Factored Action Space Representations for Deep Reinforcement learning
• An Analysis of Frame-skipping in Reinforcement Learning
• Learning to Multi-Task by Active Sampling
• TempoRL: Learning When to Act
• TAAC: Temporally Abstract Actor-Critic for Continuous Control
• Reinforcement Learning with Structured Hierarchical Grammar Representations of Actions
• Temporally-Extended ε-Greedy Exploration
• Learning to Mix n-Step Returns: Generalizing lambda-Returns for Deep Reinforcement Learning
• Utilizing Skipped Frames in Action Repeats via Pseudo-Actions