Continuous Adaptation via Meta-Learning in Nonstationary and Competitive Environments

Abstract

Ability to continuously learn and adapt from limited experience in nonstationary environments is an important milestone on the path towards general intelligence. In this paper, we cast the problem of continuous adaptation into the learning-to-learn framework. We develop a simple gradient-based meta-learning algorithm suitable for adaptation in dynamically changing and adversarial scenarios. Additionally, we design a new multi-agent competitive environment, RoboSumo, and define iterated adaptation games for testing various aspects of continuous adaptation strategies. We demonstrate that meta-learning enables significantly more efficient adaptation than reactive baselines in the few-shot regime. Our experiments with a population of agents that learn and compete suggest that meta-learners are the fittest.

Published as a conference paper at ICLR 2018

References in corpus (6)

• Theoretical Models of Learning to Learn
• Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments
• RL${}^2$: Fast Reinforcement Learning via Slow Reinforcement Learning
• Learning to reinforcement learn
• Emergent Complexity via Multi-Agent Competition
• Contextual Explanation Networks

Cited by in corpus (55)

• Generalizing from a Few Examples: A Survey on Few-Shot Learning
• Assessing Generalization in Deep Reinforcement Learning
• Learning to Adapt in Dynamic, Real-World Environments Through Meta-Reinforcement Learning
• Functional Regularisation for Continual Learning with Gaussian Processes
• Small Sample Learning in Big Data Era
• Unsupervised Meta-Learning for Reinforcement Learning
• Deep Online Learning via Meta-Learning: Continual Adaptation for Model-Based RL
• Orthogonal Gradient Descent for Continual Learning
• RMM: Reinforced Memory Management for Class-Incremental Learning
• Provable Guarantees for Gradient-Based Meta-Learning
• Edge-labeling Graph Neural Network for Few-shot Learning
• One Policy to Control Them All: Shared Modular Policies for Agent-Agnostic Control
• Meta Continual Learning
• Task Augmentation by Rotating for Meta-Learning
• Generalization in Transfer Learning
• Multi-Agent Collaboration via Reward Attribution Decomposition
• Open Problems in Cooperative AI
• Regularization Matters in Policy Optimization
• Scalable Centralized Deep Multi-Agent Reinforcement Learning via Policy Gradients
• Concurrent Meta Reinforcement Learning
• Ecological Reinforcement Learning
• A Policy Gradient Algorithm for Learning to Learn in Multiagent Reinforcement Learning
• Neural Replicator Dynamics
• DiCE: The Infinitely Differentiable Monte-Carlo Estimator
• Model-Based Opponent Modeling
• Meta Reinforcement Learning with Task Embedding and Shared Policy
• Towards Safe Policy Improvement for Non-Stationary MDPs
• Deep Reinforcement Learning for Dynamic Urban Transportation Problems
• Optimizing for the Future in Non-Stationary MDPs
• Online Structured Meta-learning
• Domain-Aware Dynamic Networks
• Continuous Coordination As a Realistic Scenario for Lifelong Learning
• Continual Model-Based Reinforcement Learning with Hypernetworks
• How Fine-Tuning Allows for Effective Meta-Learning
• Machine Learning for Massive Industrial Internet of Things
• Real-Time Object Tracking via Meta-Learning: Efficient Model Adaptation and One-Shot Channel Pruning
• Self-organization of action hierarchy and compositionality by reinforcement learning with recurrent neural networks
• A Channel Coding Benchmark for Meta-Learning
• Multi-Agent Deep Reinforcement Learning with Adaptive Policies
• Decoder Choice Network for Meta-Learning
• Generalized Hidden Parameter MDPs Transferable Model-based RL in a Handful of Trials
• Learning Complex Multi-Agent Policies in Presence of an Adversary
• Reinforcement Learning In Two Player Zero Sum Simultaneous Action Games
• Sparse Meta Networks for Sequential Adaptation and its Application to Adaptive Language Modelling
• Interactive Agent Modeling by Learning to Probe
• Learning to Model Opponent Learning
• Unifying Gradient Estimators for Meta-Reinforcement Learning via Off-Policy Evaluation
• Variable-Shot Adaptation for Online Meta-Learning
• Time-Variant Variational Transfer for Value Functions
• L2E: Learning to Exploit Your Opponent
• Agent Probing Interaction Policies
• Learning Shared Dynamics with Meta-World Models
• On Memory Mechanism in Multi-Agent Reinforcement Learning
• Dynamic Regret Analysis for Online Meta-Learning
• Neural Auto-Curricula