Meta-trained agents implement Bayes-optimal agents
arXiv:2010.11223
Abstract
Memory-based meta-learning is a powerful technique to build agents that adapt fast to any task within a target distribution. A previous theoretical study has argued that this remarkable performance is because the meta-training protocol incentivises agents to behave Bayes-optimally. We empirically investigate this claim on a number of prediction and bandit tasks. Inspired by ideas from theoretical computer science, we show that meta-learned and Bayes-optimal agents not only behave alike, but they even share a similar computational structure, in the sense that one agent system can approximately simulate the other. Furthermore, we show that Bayes-optimal agents are fixed points of the meta-learning dynamics. Our results suggest that memory-based meta-learning might serve as a general technique for numerically approximating Bayes-optimal agents - that is, even for task distributions for which we currently don't possess tractable models.
Published at 34th Conference on Neural Information Processing Systems (NeurIPS 2020), Vancouver, Canada
References in corpus (16)
- Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
- SmoothGrad: removing noise by adding noise
- Human-level performance in first-person multiplayer games with population-based deep reinforcement learning
- IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
- Not Just a Black Box: Learning Important Features Through Propagating Activation Differences
- Computational Mechanics: Pattern and Prediction, Structure and Simplicity
- Understanding the Role of Individual Units in a Deep Neural Network
- Learning to reinforcement learn
- Grad-CAM: Why did you say that?
- Towards a Neural Statistician
- Graying the black box: Understanding DQNs
- From deep learning to mechanistic understanding in neuroscience: the structure of retinal prediction
- Meta-learning of Sequential Strategies
- Metalearned Neural Memory
- How recurrent networks implement contextual processing in sentiment analysis
- Meta-learners' learning dynamics are unlike learners'