Continuous Adaptation via Meta-Learning in Nonstationary and Competitive Environments
arXiv:1710.03641
Abstract
Ability to continuously learn and adapt from limited experience in nonstationary environments is an important milestone on the path towards general intelligence. In this paper, we cast the problem of continuous adaptation into the learning-to-learn framework. We develop a simple gradient-based meta-learning algorithm suitable for adaptation in dynamically changing and adversarial scenarios. Additionally, we design a new multi-agent competitive environment, RoboSumo, and define iterated adaptation games for testing various aspects of continuous adaptation strategies. We demonstrate that meta-learning enables significantly more efficient adaptation than reactive baselines in the few-shot regime. Our experiments with a population of agents that learn and compete suggest that meta-learners are the fittest.
arXiv commentPublished as a conference paper at ICLR 2018
References in corpus (14)
- Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
- Prototypical Networks for Few-shot Learning
- High-Dimensional Continuous Control Using Generalized Advantage Estimation
- Theoretical Models of Learning to Learn
- Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments
- Matching Networks for One Shot Learning
- RL: Fast Reinforcement Learning via Slow Reinforcement Learning
- Deep Reinforcement Learning for Dialogue Generation
- Learning to reinforcement learn
- Multiagent Bidirectionally-Coordinated Nets: Emergence of Human-level Coordination in Learning to Play StarCraft Combat Games
- Learning to Optimize
- Towards a Neural Statistician
- Emergent Complexity via Multi-Agent Competition
- Contextual Explanation Networks
Cited by in corpus (56)
- Generalizing from a Few Examples: A Survey on Few-Shot Learning
- Deep Reinforcement Learning: An Overview
- Assessing Generalization in Deep Reinforcement Learning
- Learning to Adapt in Dynamic, Real-World Environments Through Meta-Reinforcement Learning
- Functional Regularisation for Continual Learning with Gaussian Processes
- Small Sample Learning in Big Data Era
- Unsupervised Meta-Learning for Reinforcement Learning
- Deep Online Learning via Meta-Learning: Continual Adaptation for Model-Based RL
- Orthogonal Gradient Descent for Continual Learning
- RMM: Reinforced Memory Management for Class-Incremental Learning
- Provable Guarantees for Gradient-Based Meta-Learning
- Edge-labeling Graph Neural Network for Few-shot Learning
- One Policy to Control Them All: Shared Modular Policies for Agent-Agnostic Control
- Meta Continual Learning
- Task Augmentation by Rotating for Meta-Learning
- Generalization in Transfer Learning
- Multi-Agent Collaboration via Reward Attribution Decomposition
- Regularization Matters in Policy Optimization
- Open Problems in Cooperative AI
- Scalable Centralized Deep Multi-Agent Reinforcement Learning via Policy Gradients
- Concurrent Meta Reinforcement Learning
- Ecological Reinforcement Learning
- A Policy Gradient Algorithm for Learning to Learn in Multiagent Reinforcement Learning
- Neural Replicator Dynamics
- DiCE: The Infinitely Differentiable Monte-Carlo Estimator
- Meta Reinforcement Learning with Task Embedding and Shared Policy
- Model-Based Opponent Modeling
- Towards Safe Policy Improvement for Non-Stationary MDPs
- Deep Reinforcement Learning for Dynamic Urban Transportation Problems
- Online Structured Meta-learning
- Optimizing for the Future in Non-Stationary MDPs
- Continuous Coordination As a Realistic Scenario for Lifelong Learning
- Domain-Aware Dynamic Networks
- Continual Model-Based Reinforcement Learning with Hypernetworks
- How Fine-Tuning Allows for Effective Meta-Learning
- Machine Learning for Massive Industrial Internet of Things
- Real-Time Object Tracking via Meta-Learning: Efficient Model Adaptation and One-Shot Channel Pruning
- Self-organization of action hierarchy and compositionality by reinforcement learning with recurrent neural networks
- A Channel Coding Benchmark for Meta-Learning
- Generalized Hidden Parameter MDPs Transferable Model-based RL in a Handful of Trials
- Decoder Choice Network for Meta-Learning
- Multi-Agent Deep Reinforcement Learning with Adaptive Policies
- Learning Complex Multi-Agent Policies in Presence of an Adversary
- Learning to Model Opponent Learning
- Reinforcement Learning In Two Player Zero Sum Simultaneous Action Games
- Variable-Shot Adaptation for Online Meta-Learning
- Interactive Agent Modeling by Learning to Probe
- Sparse Meta Networks for Sequential Adaptation and its Application to Adaptive Language Modelling
- Unifying Gradient Estimators for Meta-Reinforcement Learning via Off-Policy Evaluation
- Dynamic Regret Analysis for Online Meta-Learning
- On Memory Mechanism in Multi-Agent Reinforcement Learning
- Agent Probing Interaction Policies
- L2E: Learning to Exploit Your Opponent
- Learning Shared Dynamics with Meta-World Models
- Neural Auto-Curricula
- Time-Variant Variational Transfer for Value Functions