Discovery of Useful Questions as Auxiliary Tasks

Abstract

Arguably, intelligent agents ought to be able to discover their own questions so that in learning answers for them they learn unanticipated useful knowledge and skills; this departs from the focus in much of machine learning on agents learning answers to externally defined questions. We present a novel method for a reinforcement learning (RL) agent to discover questions formulated as general value functions or GVFs, a fairly rich form of knowledge representation. Specifically, our method uses non-myopic meta-gradients to learn GVF-questions such that learning answers to them, as an auxiliary task, induces useful representations for the main task faced by the RL agent. We demonstrate that auxiliary tasks based on the discovered GVFs are sufficient, on their own, to build representations that support main task learning, and that they do so better than popular hand-designed auxiliary tasks from the literature. Furthermore, we show, in the context of Atari 2600 videogames, how such auxiliary tasks, meta-learned alongside the main task, can improve the data efficiency of an actor-critic agent.

Cited by in corpus (26)

• Discovering Reinforcement Learning Algorithms
• Meta-Gradient Reinforcement Learning with an Objective Discovered Online
• A Self-Tuning Actor-Critic Algorithm
• Auxiliary-task Based Deep Reinforcement Learning for Participant Selection Problem in Mobile Crowdsourcing
• General Value Function Networks
• Online Meta-Critic Learning for Off-Policy Actor-Critic Methods
• Return-Based Contrastive Representation Learning for Reinforcement Learning
• Meta-learning curiosity algorithms
• Which Mutual-Information Representation Learning Objectives are Sufficient for Control?
• Balancing Constraints and Rewards with Meta-Gradient D4PG
• Finding online neural update rules by learning to remember
• Reinforcement Learning, Bit by Bit
• Credit Assignment with Meta-Policy Gradient for Multi-Agent Reinforcement Learning
• Discovery of Options via Meta-Learned Subgoals
• A Deeper Look at Discounting Mismatch in Actor-Critic Algorithms
• What Can Learned Intrinsic Rewards Capture?
• Learning Retrospective Knowledge with Reverse Reinforcement Learning
• VIABLE: Fast Adaptation via Backpropagating Learned Loss
• Value-driven Hindsight Modelling
• Beyond Exponentially Discounted Sum: Automatic Learning of Return Function
• Neural Auto-Curricula
• Learning to Reweight Imaginary Transitions for Model-Based Reinforcement Learning
• Learning State Representations from Random Deep Action-conditional Predictions
• Work in Progress: Temporally Extended Auxiliary Tasks
• Parameter-free Gradient Temporal Difference Learning
• Finding Useful Predictions by Meta-gradient Descent to Improve Decision-making