Meta-learning curiosity algorithms

Abstract

We hypothesize that curiosity is a mechanism found by evolution that encourages meaningful exploration early in an agent's life in order to expose it to experiences that enable it to obtain high rewards over the course of its lifetime. We formulate the problem of generating curious behavior as one of meta-learning: an outer loop will search over a space of curiosity mechanisms that dynamically adapt the agent's reward signal, and an inner loop will perform standard reinforcement learning using the adapted reward signal. However, current meta-RL methods based on transferring neural network weights have only generalized between very similar tasks. To broaden the generalization, we instead propose to meta-learn algorithms: pieces of code similar to those designed by humans in ML papers. Our rich language of programs combines neural networks with other building blocks such as buffers, nearest-neighbor modules and custom loss functions. We demonstrate the effectiveness of the approach empirically, finding two novel curiosity algorithms that perform on par or better than human-designed published curiosity algorithms in domains as disparate as grid navigation with image inputs, acrobot, lunar lander, ant and hopper.

arXiv commentPublished in ICLR 2020

References in corpus (26)

• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Neural Architecture Search with Reinforcement Learning
• Neural Architecture Search: A Survey
• Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation
• Searching for Activation Functions
• PathNet: Evolution Channels Gradient Descent in Super Neural Networks
• Noisy Networks for Exploration
• Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning
• #Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning
• Exploration by Random Network Distillation
• Neural Optimizer Search with Reinforcement Learning
• Actor-Mimic: Deep Multitask and Transfer Reinforcement Learning
• Stochastic Neural Networks for Hierarchical Reinforcement Learning
• Meta-Gradient Reinforcement Learning
• Meta-Reinforcement Learning of Structured Exploration Strategies
• Evolved Policy Gradients
• Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutions
• Gotta Learn Fast: A New Benchmark for Generalization in RL
• Some Considerations on Learning to Explore via Meta-Reinforcement Learning
• EX2: Exploration with Exemplar Models for Deep Reinforcement Learning
• Discovery of Useful Questions as Auxiliary Tasks
• Computational Theories of Curiosity-Driven Learning
• Evolving Rewards to Automate Reinforcement Learning
• Benchmarking Bonus-Based Exploration Methods on the Arcade Learning Environment
• Learning Compositional Neural Programs with Recursive Tree Search and Planning
• World Discovery Models

Cited by in corpus (7)

• AutoML-Zero: Evolving Machine Learning Algorithms From Scratch
• Discovering Reinforcement Learning Algorithms
• Improving Generalization in Meta Reinforcement Learning using Learned Objectives
• A Comprehensive Overview and Survey of Recent Advances in Meta-Learning
• Meta Learning Backpropagation And Improving It
• A Brief Summary of Interactions Between Meta-Learning and Self-Supervised Learning
• Bootstrapped Meta-Learning