The RL Perceptron: Generalisation Dynamics of Policy Learning in High Dimensions

Abstract

Reinforcement learning (RL) algorithms have proven transformative in a range of domains. To tackle real-world domains, these systems often use neural networks to learn policies directly from pixels or other high-dimensional sensory input. By contrast, much theory of RL has focused on discrete state spaces or worst-case analysis, and fundamental questions remain about the dynamics of policy learning in high-dimensional settings. Here, we propose a solvable high-dimensional model of RL that can capture a variety of learning protocols, and derive its typical dynamics as a set of closed-form ordinary differential equations (ODEs). We derive optimal schedules for the learning rates and task difficulty - analogous to annealing schemes and curricula during training in RL - and show that the model exhibits rich behaviour, including delayed learning under sparse rewards; a variety of learning regimes depending on reward baselines; and a speed-accuracy trade-off driven by reward stringency. Experiments on variants of the Procgen game "Bossfight" and Arcade Learning Environment game "Pong" also show such a speed-accuracy trade-off in practice. Together, these results take a step towards closing the gap between theory and practice in high-dimensional RL.

11 pages, 8 figures, Submitted to Physical Review X

References in corpus (13)

• Machine learning and the physical sciences
• A Mean Field View of the Landscape of Two-Layers Neural Networks
• A Survey of Zero-shot Generalisation in Deep Reinforcement Learning
• The simplest model of jamming
• Modelling the influence of data structure on learning in neural networks: the hidden manifold model
• On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transport
• Typology of phase transitions in Bayesian inference problems
• Theory of spike timing based neural classifiers
• Disordered Systems Insights on Computational Hardness
• Probing transfer learning with a model of synthetic correlated datasets
• Learning curves for the multi-class teacher-student perceptron
• Phase Transitions in Transfer Learning for High-Dimensional Perceptrons
• An Analytical Theory of Curriculum Learning in Teacher-Student Networks