Maximum diffusion reinforcement learning

Abstract

Robots and animals both experience the world through their bodies and senses. Their embodiment constrains their experiences, ensuring they unfold continuously in space and time. As a result, the experiences of embodied agents are intrinsically correlated. Correlations create fundamental challenges for machine learning, as most techniques rely on the assumption that data are independent and identically distributed. In reinforcement learning, where data are directly collected from an agent's sequential experiences, violations of this assumption are often unavoidable. Here, we derive a method that overcomes this issue by exploiting the statistical mechanics of ergodic processes, which we term maximum diffusion reinforcement learning. By decorrelating agent experiences, our approach provably enables single-shot learning in continuous deployments over the course of individual task attempts. Moreover, we prove our approach generalizes well-known maximum entropy techniques, and robustly exceeds state-of-the-art performance across popular benchmarks. Our results at the nexus of physics, learning, and control form a foundation for transparent and reliable decision-making in embodied reinforcement learning agents.

The PDF file contains the collated main text and supplementary information. For supplementary movies, see https://www.youtube.com/playlist?list=PLO5AGPa3klrCTSO-t7HZsVNQinHXFQmn9

References in corpus (8)

• SciPy 1.0--Fundamental Algorithms for Scientific Computing in Python
• How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learned
• A Survey of Zero-shot Generalisation in Deep Reinforcement Learning
• Maximum Caliber: a general variational principle for dynamical systems
• Low rattling: A predictive principle for self-organization in active collectives
• Active Learning in Robotics: A Review of Control Principles
• Ergodic properties of heterogeneous diffusion processes in a potential well
• State Entropy Maximization with Random Encoders for Efficient Exploration

Cited by in corpus (1)

• Real-Time Reinforcement Learning for Dynamic Tasks with a Parallel Soft Robot